Can your PHP/MySQL CMS handle a Slashdotting?

[Gayle]

Introduction
A little while ago I was curious about how well Mambo would stand up against Post-Nuke under a Slashdotting. The results of that little test can be found here but I decided to refine and expand it a little which is what lead to this test. This benchmark is designed to test nothing other than how the CMSs perform under certain specific load conditions on my machine, nothing in here is meant to say X is better than Y except within the very specific boundaries of the benchmark.

The scripts
I scouted around the net looking for data on exactly how bad the Slashdot effect really was, this article inidated that they got hit with around 280 hits per minute on a fairly popular article so I went with a figure of 300. I whipped up a little bash script to run on the client that generated 5 page requests every second (using ab2, the Apache benchmark) for 2 minutes and stored all the data in files. Another little script on the server dumped the average load to file every 10 seconds.

The machines
The "server" for this little test was an Athlon 1800XP with 512MB RAM. OS was SuSE 9.1 Pro and apache 2.0.49-prefork and MySQL 4.0.18 with a 10MB query cache. Free memory at the start of the test was around 420MB and the server was rebooted between test runs to make sure everyone started from the same base level. Client and server were directly linked by a cat5 crossover cable in order to take transfer time out of the equation as much as possible, no-one was going to fill 100MBit/s on this test.

The configuration
All CMSs were installed with the default config then any speed related options were turned on. Drupal and e107 were the only two to support page cacheing which gave them a significant advantage.

The results
First off, the load comparism

Please note that the scale on this graph is logarithmic. I had to do it that way otherwise Xaraya pushed the others way down the bottom to the point where nothing was visible.

As you can see Xaraya immediately leaps out as causing a much higher load than the others, it wouldn't take long for this to bring the server to it's knees. At the other end of the scale, e107 puts in a particularly impressive performance not even making the server try at all. As expected, the two CMSs with page caches performed the best but Xoops was not far behind. As a side note, Post-Nuke performed vastly better in this test than my previous one, this seems to be down to enabling the MySQL query cache.

Peak load

The peak load (another logarithmic scale) shows more clearly how high the load was pushed during the test run. e107 stands out even more clearly here as the star performer.

Slowest page time
So, exactly how slow did things get? Let's take a look...

Once again, this is a logarithmic scale. e107 never exceeded 0.1 seconds per page load. Xaraya went way over the infamous magic figure of 10 seconds and peaked at a little over 42 seconds. The honours for the non-cached CMSs were shared by Mambo and Xoops, both at 0.74 seconds.

Average page load time

This one is NOT logarithmic, take heed. Xaraya was obviously slowest while e107 was far and away the best performer followed by Drupal. Of the non-cached CMSs Mambo took the honours at 0.26s, less than half that of Post-Nuke.

Conclusion
It seems obvious that e107 and Drupal were greatly helped by their page cacheing, implementing this functionality would probably be a good move for others to consider. Mambo and Xoops performed fairly similarly with Mambo generating a slightly higher load but serving the pages a little faster. Post-Nuke worked at around double the load and double the average page time compared to Mambo/Xoops While Xaraya really didn't take the Slashdotting very well at all.

I want to stress again that this is how they perform on my system under the conditions I generated. These numbers do not mean anything outside of this benchmark. Xaraya does not suck nor is e107 the be all and end all of Free PHP/MySQL CMSs.

[Mambo.ie]

To the non technically minded (and I include myself) this seems a vote for Mambo to include page caching.

I have come across this before in a French Blogging system called Spip and to me it seems to me pages which are created dynamically from database information say, are stored on the server until they are requested and then the server checks if they should be regenerated and either does so or serves the existing page if there is no change to the data.

Is this a fair synopsis?

[Gayle]

Originally Posted by TheGreek

To the non technically minded (and I include myself) this seems a vote for Mambo to include page caching.

Something I've experimented with before (with some good results). If you check this post Robert checked in some cacheing code to CVS today so it will be nice to see how that affects things .

I have come across this before in a French Blogging system called Spip and to me it seems to me pages which are created dynamically from database information say, are stored on the server until they are requested and then the server checks if they should be regenerated and either does so or serves the existing page if there is no change to the data.

Is this a fair synopsis?

It's a fair synopsis of one form of cacheing, the one I (and I think the core team) prefer is more modular. If you cache just whole pages then anything dynamic on those pages will cause them to be regenerated all the time. If you cache page elements seperately then you can assemble the completed page partly from cache and partly dynamically. Hope that makes sense

[Gayle]

I reran the test with the latest CVS (i.e. with cacheing hacked in) and while I'm not going to redo the graphs I do have some figures.

Processor load halved (peak of .84 instead of 1.4, slowest page time halved (.36 from .74) and average page time was .16 instead of .26 so down by a little over a third. Nice results for a quick hack Robert

:-) :-) :-)

Cool :-)

[Grizzletooth]

I was a little curious about the 300 hit/minute number being used to estimate the Slashdot Effect, and did some additional Google'ing to see if there were any references to more popular events, and with any more recent data. I found the following article about an earthquake website (ok, the US Geological Survey website):

http://pasadena.wr.usgs.gov/office/stans/slashdot.html (1999)

In summary, they found that within 15 minutes of the earthquake, they received a peak hit rate of 67 hits per second. Over 4,000 hits per minute! Just to be clear, this is a site that is known for earthquake data, and was not specifically tied to any Slashdot post. But it gives another idea of what "a lot" of hits really means.

Another article: http://www.komar.org/xmas/2003/stats...ot_effect.html (2003)
mentions that they received over 69,000 hits to a Slashdot linked article in 1 hour. Over 1,000 hits per minute.

Finally, http://www.geology.smu.edu/~dpa-www/attention_span/ (2003)
which related to the Space Shuttle Columbia disaster, summarized that they saw peak hit rates approaching 2,000 per minute on a Slashdot linked article.

-Grizzletooth

[Gayle]

Originally Posted by Grizzletooth

In summary, they found that within 15 minutes of the earthquake, they received a peak hit rate of 67 hits per second. Over 4,000 hits per minute! Just to be clear, this is a site that is known for earthquake data, and was not specifically tied to any Slashdot post. But it gives another idea of what "a lot" of hits really means.

I wasn't trying to stress test them though, all I wanted was to simulate a Slashdotting.

http://www.komar.org/xmas/2003/stats...ot_effect.html (2003)
mentions that they received over 69,000 hits to a Slashdot linked article in 1 hour. Over 1,000 hits per minute.

Correct me if I'm wrong but a hit is not the same a page impression. Every image, stylesheet, javascript file etc. is a hit so an HTML page with a single stylesheet and one image would count as 900 hits on my test. I don't have the webserver stats to hand but by my reckoning the Mambo install generates 12 hits per page impression which means 3600 hits per minute.

[Grizzletooth]

I agree they CAN be different. But so often people who are talking about logs and hit rates fail to differentiate, I am not sure which to assume in any of the 3 cases that I quoted.

-Grizzletooth

Originally Posted by Gayle

I wasn't trying to stress test them though, all I wanted was to simulate a Slashdotting.

Correct me if I'm wrong but a hit is not the same a page impression. Every image, stylesheet, javascript file etc. is a hit so an HTML page with a single stylesheet and one image would count as 900 hits on my test. I don't have the webserver stats to hand but by my reckoning the Mambo install generates 12 hits per page impression which means 3600 hits per minute.

[Ce]

Originally Posted by TheGreek

To the non technically minded (and I include myself) this seems a vote for Mambo to include page caching.

I have come across this before in a French Blogging system called Spip and to me it seems to me pages which are created dynamically from database information say, are stored on the server until they are requested and then the server checks if they should be regenerated and either does so or serves the existing page if there is no change to the data.

Is this a fair synopsis?

Hi,

just stumbled across your post after months of absence form the forums... I use mambo for my personal website and SPIP (which is a complete CMS, not a blogging system) for the website of my organisation.

Your synopsis is ok - in fact, what SPIP does is comparing your request for a page against a CACHE writeable folder, check if it exists and, if it exists, if it is 'younger' than a value you specify. If both checks are ok, it serves the cached page - if they are not, it generates the page on the fly.

You can read more (in English) here: http://www.spip.net/en_article2144.h...echerche=cache.

IMHO, I would strongly recommend a caching system in the next version of Mambo - to be honest, it was one of the two key factors leading me to choose SPIP for my work website, the second factor being the long discussed issue of XHTML standards.

[niceguyeddie]

Hi there, I'd be interested in getting the bash script that you used with this test specifically for Xaraya. Since you mention that you did not have the template cache turned on for Xaraya, and likewise would also assume (although assumptions are often wrong) that you did not have our cache manager turned on either. If you could send it to admin@dinerminor.com that would be great. At the very least I can verify the results which would give a few more things to work on.

[Gayle]

Originally Posted by niceguyeddie

Hi there, I'd be interested in getting the bash script that you used with this test specifically for Xaraya. Since you mention that you did not have the template cache turned on for Xaraya, and likewise would also assume (although assumptions are often wrong) that you did not have our cache manager turned on either. If you could send it to admin@dinerminor.com that would be great. At the very least I can verify the results which would give a few more things to work on.

Sure, I'll send you a copy. I'd also be interested in running it again on my test setup if I missed some options. In fact, what would be perfect is if you could email me a properly optimised config (mysqldump would be great) that I can just import. That way I'd know I haven't missed anything

[project3e]

Hi, I was wondering what version of Postnuke you are using and if you have compression (gzip, I think, but not sure) turned on in the settings? Thanks for the great work!

[jsb]

Thanks for including Xaraya in your testing!

Originally Posted by Gayle

I'd also be interested in running it again on my test setup if I missed some options.

Quick steps for prepping a Xaraya installation for a slashdotting:

1) Make sure "cache templates" is enabled (default is yes, so problably is already) - Admin panel > themes > Modify Config: Cache templates: checked

2) Activate xarcachemanager - Admin panel > modules > View All: xarcachemanager (click Install link)

3) Enable page and block level output caching - Admin panel > xarcachemanager > Modify Config: Enable Output Caching System, Enable Page Output Caching, Enable Block Output Caching: all checked, click Update xarCacheManager Configuration button

You can tweak and tune the output caching with the Admin tools to your liking, but the defaults would be a good start for load handling.

In Xaraya's current beta form, setting these options won't put xaraya in the running for top performer, but it should allow you to present the data without the logarithmic scale formatting.

The xaraya output caching system itself in young and has some important design changes that are being worked out now. This, along with other optimization that is being worked on should mean better results from Xaraya 1.0, so don't write the project off as doomed to be dead last forever, just yet.

- Jonn

P.S. Have you considered running PHP with a bytecode caching extension like APC or Turck MMCache? I'd be very interested in the results of all running under this type of PHP environment.

APC:
http://pecl.php.net/package-info.php?package=APC

Turck MMCache:
http://turck-mmcache.sourceforge.net/

[rkeays]

Xaraya without caching:

Start of function trace
Time Memory
0.0015 29648 -> xarmain()
0.0016 29720 -> xarcoreinit()
0.0926 302192 -> xarrequestgetinfo()
0.1279 309464 -> xarpageiscached()
0.1395 353056 -> xarmodfunc()
0.1421 353336 -> articles_user_main()
...
5.7131 5049496 -> xarcurrenterrortype()
5.7476 5012376 -> xartpl_renderpage()
7.0017 5050272 -> xdebug_stop_trace()
End of function trace

Xaraya with caching:

Start of function trace
Time Memory
0.0017 29656 -> xarmain()
0.0019 29728 -> xarcoreinit()
0.0941 302200 -> xarrequestgetinfo()
0.1007 303192 -> xartplsetpagetitle()
0.1297 309288 -> xarcache_init()
0.1300 309472 -> xarpageiscached()
...
0.1365 309640 -> xarcore__shutdown_handler()
End of function trace

(note, much of the initial 7s page load would have been due to the xdebug trace output)

[Gayle]

Originally Posted by jsb

P.S. Have you considered running PHP with a bytecode caching extension like APC or Turck MMCache? I'd be very interested in the results of all running under this type of PHP environment.

I tested three different bytecode caches a while ago and found Turck MMcache to be the only one worth the effort. Unfortunately it doesn't work with apache MPM-prefork which is what I'm currently using

[Gayle]

Originally Posted by project3e

Hi, I was wondering what version of Postnuke you are using and if you have compression (gzip, I think, but not sure) turned on in the settings? Thanks for the great work!

Not at home so can't remember but it's the current download (.726?). Gzip compression was on.

[intel352]

nice article. just wanted to add, Xoops supports cacheing of blocks and modules, you can edit the module cache settings per-module via General Preferences.

i wonder if that will help the results any? i'm a xoops user, big fan of the CMS, and i'm quite proud that it placed so well with no cacheing enabled :-)

oh, and i assume GZip was enabled as well?

thanks for returning such comprehensive reports, this was one of the better tests that i've seen so far.

[lolo irie]

Hi,

Very nice and interesting post. Congrats !

As member of the "new" e107 Dev Team, I'm of course very happy with these results.

I just would like to add, performances are not the only thing to check before to choose a CMS, but of course can be the most important point for site with a lot of visitors...
Like I ever explain, the best is to test different CMS on a test server and compare usability, features, performances and flexability before to choose the system to use for your site.

I discovered Mambo too late to be able to use it, but I know it's a very powerful CMS too and hope to the Mambo project everything fine for the future.

Lolo Irie (http://etalkers.org)
e107 Project (http://e107.org)

[stingrey]

Its great to read the response from different CMS developers to this article and great to see the spirit of the OS community at work.

Nice to see that people are using it as a way to gauge how to improve their own developments.


All the best to all those non-mambo people stopping in from different CMS's, especially developers and lets all continue to push the boundaries


May the Music never stop and Dance never end

[webmedic]

If it would not be to much trouble I would like ot get the scripts used in the testing also.

Thanks

bah@webmedic.net

I have another cms type app I would like ot test it out on ot compare with these tests.