Mambo/Post-Nuke load comparism

[Gayle]

So I was bored and I decided to take a look atexactly how well Mambo would handle a Slashdotting in comparism to a Nuke. I whipped up a little script to simulate the massed hordes of Slashdot in a kinda basic way...

gayle@athene:~/bin> cat slashdot.sh
#!/bin/bash
cat /proc/loadavg | awk ' { print $1 }' >> ${1}
for i in `seq 1 12`;
do
        for j in `seq 1 10`;
        do
                /usr/sbin/ab2 -n 5 -c 5 http://athene/~gayle/${1}/ &
                sleep 1
        done
        cat /proc/loadavg | awk ' { print $1 }' >> ${1}
done

What it does is launch the Apache benchmark every second with 5 simultaneous connections every second for 2 minutes and every 10 seconds it logs the current load average. This traffic is roughly what Slashdot generates on a popular story so it seemed a reasonable way to do it. Observant readers will note that the benchmark is designed to run on the same machine that hosts the sites, there is a reason for this. Firstly, I don't have 2 decent machines to run the test with and my **** one is currently an Exchange server so no bash scripting on that one. Second, this machine has an Athlon 1800XP and 512MB RAM. The processor/RAM usage of the benchmark is both negligible and constant for both tests. It also takes bandwidth out of the equation which is handy .

The server components are not optimised in any way, nor are the CMS scripts. Both are as they come when first installed.

Now for the pretty picture...



Ain't it pretty? The Post-Nuke is the latest version and Mambo is the latest 4.5 CVS, as you can see Mambo took it rather better...

When we started there was a little over 300MB RAM free and server load of effectively nothing. Post-Nuke (PN) ramped up the load very fast until around 45 when it had managed to eat all available RAM. The OS started paging to disk which is why I think the end of the PN graph is a little 'weird' (I think it stopped my little slashdot script while it was paging, that would account for PN catching up a little). Mambo didn't even use 100% processor time for most of the run but PN was at 100% from the start. The only thing on the PN page that wasn't on the Mambo one was a poll, hardly enough to make a difference especially when you take into account that Mambo had the usual default articles which PN doesn't.

So, if you're gonna be Slashdotted you don't want to be using a Nuke!

I did also run this on the 4.6 CVS from about a week or so ago and it peaked at around 9, the current 4.6 CVS shoots to around 75 but it's hardly fair to put them on the graph, being CVS it could be down to 3 by tomorrow

I do notice that ADODB has been patched in this week though...

For reference, I also did the test with the default Apache page as a kinda base level. Problem is, the system load didn't rise above .48 the whole time so it was kinda pointless

[Arthur]

Very well done (although I knew it all the time... ). I will announce this at Mamboportal.com tonight. Thanks Gayle!

PS: Can you maybe attach a CSV file with the data? This could be usefull for some own presentations...

Gayle,

Would you also mind posting OS & version, MySQL version, PHP version and Apache version.

Many thanks.

[Gayle]

Originally Posted by Lil Devil

Gayle,

Would you also mind posting OS & version, MySQL version, PHP version and Apache version.

Many thanks.

Suse 9.1 Pro, Apache 2.0.40, MySQL 4.0.18, and PHP 4.3.4. All stock Suse stuff so probably got a few backports of patches etc in there.

The main reason for not including all the kind of stuff you normally see in benchmarks is because I just wanted to see how well Mambo could stand a hammering in relation to a Nuke. I am intrigued by this idea though and have been thinking it over a little more

What I may do is set up a partition with a well optimised webserver/MySQL config and nothing else. That would give me a standard base to work from without wrecking my normal setup because I may do this again but slightly more scientifically and with a wider range of CMSs.

This one started as a bit of fun but even my basic setup certainly shows that Mambo is way ahead of Post-Nuke in terms of resource use per page impression.

Might have to chart memory usage as well, that could be another interesting comparism.

[Gayle]

Originally Posted by Arthur

PS: Can you maybe attach a CSV file with the data? This could be usefull for some own presentations...

See my previous post, I'll probably do this again in the next week but more stingently and with a wider choice of CMSs. If I do I'll post all data files with it.

[edit]Just read your bit on Mamboportal, I should make it clear that it's not a 60% load but a load average of 60 (i.e. 60 processes in the queue waiting to do their thing). That's waaaaaay overloaded, we're talking up to 40 seconds per page load WITHOUT having any download time.[/edit]

It would also be useful to get the Apachebench results plotted as well e.g. time taken. And then re-run the tests with Mambo 4.5.1 gzip on/off, and stats enabled/disabled.

I am just trying to mod your script for FreeBSD, but FreeBSD doesn't support /proc/loadavg so I am going to have to do something with uptime to get a result.

It would also be good to get a chart plotting how many pages the CMS can serve in say 10 secs.

[grutkowski]

Gayle - Great timeing!

I just joined this thread:
http://forum.mamboserver.com/showthr...2697#post62697

and really didn't buy into the results that were being tossed around.

Great to have some more scientific / real-world results!

As always, you rock gayle!

gr.

[mmx]

It would be interesting to see a comparison between Mambo and Xaraya. Both are modular enough where the features sets could be mixed and matched more closely. Although the schemas are different the content models have similar goals. I would think its a better choice for comparison purposes as we run down the yellow brick road toward 5.0 because the underlying feature sets have many similarities.

[mmx]

Anyone who visited your site earlier when the forums were still there knows this for a fact. There would be 1000-1200 users online at times and you could hit the refresh button to watch the count jump up and down as the load changed during peak periods.

Originally Posted by Arthur

Very well done (although I knew it all the time... ). I will announce this at Mamboportal.com tonight. Thanks Gayle!

PS: Can you maybe attach a CSV file with the data? This could be usefull for some own presentations...

[Gayle]

Originally Posted by mmx

It would be interesting to see a comparison between Mambo and Xaraya. Both are modular enough where the features sets could be mixed and matched more closely. Although the schemas are different the content models have similar goals. I would think its a better choice for comparison purposes as we run down the yellow brick road toward 5.0 because the underlying feature sets have many similarities.

I think I can get about 5 CMSs on the graph before it gets cluttered, assuming I do Xayaya, Mambo, and Post-Nuke that leaves me with 2 others. Any suggestions?

typo3 and ezPublish ;-)

But don't ask me for any help in setting them up LOL

[Cheese]

:X typo3, thats one huge cms

[MasterChief]

Typo3 can be hard to set up. Drupal wouldn't be bad as a replacement. I expect it would be similar to 4.5.

[MasterChief]

Gayle, any chance of negotiating with the opensourcecms guys to run this script on all of their demos? I could probably help out with the data presentation if you need it (text as well as a graphic plot would be useful).

[silexian]

Originally Posted by Gayle

I think I can get about 5 CMSs on the graph before it gets cluttered, assuming I do Xayaya, Mambo, and Post-Nuke that leaves me with 2 others. Any suggestions?

I suggest Xoops and NPDS, instead of EZ Publish and Typo3 , because they aren't comparable with mambo (they're more powerfull (**) but evidently they're bloatware)...

we should eliminate nuke-like CMS because they are so bloat !

Loran

(*) in the same category of Mambo : Xoops, NPDS, MD-Pro, e107, Xaraya, BolinOS,

(**) more powerfull/complete : typo3, EZ Publish, MD-Pro (?), Midgard, WebGUI,...

[spignataro]

i suggest a compasion between tons of CMS....

Xoops Expublish, Matrix CMS, php-nuke, post nuke, etc..

[tonyskyday]

Originally Posted by eddieajau

Gayle, any chance of negotiating with the opensourcecms guys to run this script on all of their demos? I could probably help out with the data presentation if you need it (text as well as a graphic plot would be useful).

Steven,
What about this? ^^^^^^

[spignataro]

be nice if i could read the whole post....lol....id love to help out with if at all possible then

[Gayle]

While it would be nice to run the test on all of them it needs to be in a controlled environment which the OpensourceCMS server is not. It takes a fair amount of time to download and configure them all on a dialup connection so for now at least I think I'll have to keep it to a reasonable number.

Currently the plan of action is:

1. Configure a test environment optimised for PHP/MySQL web apps. Install TurckMMcache and enable query cacheing on MySQL. Maybe tweak Apache, not sure on that yet.

2. Install and configure the CMSs to be as similar as possible.

3. Configure another machine to act as the client.

4. Reconfigure client script to not measure server load and to output all ab2 data into files rather than just discarding.

5. Write server script to measure load at 10 second intervals and timestamp each measurement (it may get delayed if the load is too high).

6. Make pretty graphs.

7. Write up article for OpensourceCMS.com (if they're interested).

8. Submit link to Slashdot

9. ???

10. Profit???

Andrew: Help with data presentation would be most appreciated. I can't figure out how to get gnuplot to play so the piccie above was a screen grab from OOo, that was the best I could come up with. I am certain there are MUCH easier and better ways to do it (and the planned test will produce a much wider range of data).

Not sure how to measure memory usage, I think adding a call to memory_get_usage() on the index.php will tell me what I need to know but I don't know how accurate that will be. Any ideas?

[chanh]

Gayle,

That would be wonderful feature article on our site!

Thanks