Chris and I worked today on analyzing the performance issues of Deme
and Symsys. Here is a summary:

General speed:

1. Is the virtualized server slower at serving pages? Yes. Chris
compared his machine with symsys.stanford.edu and was 15-30% faster. I
compared my machine to deme.stanford.edu and was 30-50% faster. Part
could be due to existing traffic on the server, we'd have to
temporarily disable visitor access to the server to test that.

2. Can we improve speed by utilizing more postgresql memory? No, we've
already allocated several times the size of the database, and
analyzing at a random query did not show it was using the hard drive.

3. Are permissions slowing us down? Yes. Disabling permissions (or
logging in as an admin) speeds up the site substantially (on the order
of about 5x improvement).

4. Can we improve the way we do permission queries? Yes, there are
sometimes redundancies in our permission queries, for example the
symsysgroup viewer could be made significantly faster. I made some
improvements in the permission queries for the sidebar which made my
test page (list view proposal) twice as fast. One surprising thing I
found was that for permission filter queries, we actually spend about
25% of the time *constructing* the query in Django, when we really
should spend all of the time in the database.

Caching:

5. Can we improve speed by caching SQL queries? Yes, but it's not
built into postgresql (unlike mysql I believe). There are third party
projects to do this at the database level or the Django level. For
example, see http://code.google.com/p/pqc/ . However, I don't believe
there is an easy way to deal with invalidation.

6. Can we improve speed by caching web pages? Yes, but we need to
figure out a cache invalidation policy that works correctly. This is
not easy if we don't know in advance exactly what changes to the
database would affect any given page (whereas it is easy on a more
statically defined website).

7. Can we improve the way we cache permissions? Same issue as caching
web pages (it's not easy to know when to invalidate), but at least
it's easier to reason about

Avoiding unnecessary calculations:

8. Most of the slow down people experience will just be the sidebar on
every page (especially related items and action notices, which most
items have). Solution: only load sidebar components using ajax.

Action points: write sidebar components to be lazily loaded with ajax,
rewrite problematic pages like symsysgroup viewer, test deme on
cloud-hosted server to see if speed improvements hold