So that Rob doesn't feel like I'm picking on him, I'm going to pick on Tony
and Ari as well :-)
I've done some comparative shopping using a few servers (and probably have
the record for the most separate http servers ever running on one machine
at the same time :-)
The Victims were:
MDMA, release 0.1 pre-alpha
CERN, release 3.0pre6
NCSA, release 1.3
PLEXUS, release 2.2.1
Although I haven't had enough time to do a full set of benchmarks, the results
are still kind of interesting. These tests were run using web-whacker.
[In these tests, the data set was kept small enough to fit entirely in main
memory- this is because the test server only has 32Mb, (an order of magnitude
less than the target system). I hope to do more realistic tests next week,
using a true log based frequency table.]
[When I refer to a 'loader' I'm talking about an artificial load generator
designed to simulate other processes running on the same system. The dummy
load I use is an infinite loop of "sed -e 's;a;b;g' </usr/dict/words >/dev/null"
]
1500 repetitions, avg 20 queries per second, exponential distribution
---------------------------------------------------------------------
NONE ONE TWO FOUR EIGHT TWELVE LOADERS
------------------------------------------------------------------
MDMA 0.10 0.12 0.36 0.54 2.66 15.65
CERN 11.4 25.08 52.28 -- -- --
NCSA 47.16 76.34 106.91 -- -- --
PLEXUS did not finish (daemon crashed)
-------------------------------------------------------------------
Plexus had difficulties even when the query rates were halved. Plexus was
the only daemon to fail castastrophically in this test.
When the tests were run for more trials, the times for CERN and NCSA continued
to grow, which is as expected. The same was observed in MDMA 12 loader case.
I'm going to do another table of load tolerance at lower query rates next.
Simon
p.s.
Important note: When you've finished running your benchmarks on a
machine with a bunch of loads, make sure you kill the loads before
you try compiling anything :-)