April 05, 2006
It Just Worked Out That Way
Just to clear one thing up - munu is not a commercial service. It's not just non-profit, it's non-revenue. That may change in the future, but at the moment we have a gross profit of zilch on an annual turnover of nothing.
Now, on the subject of web spiders: By default, Movable Type 2.6 creates CGI links for the comments and trackbacks for every post.
We have over 160,000 posts here at munu. (Just in MT 2.6.)
Back in November when we moved servers, we had about 140,000 posts. As it happens, when you dump and reload a MySQL database it loses all its indexing statistics. And with no statistics, it simply doesn't use the indexes.
So along came a spider, and decided to index all 280,000 CGI pages against a database with no working indexes causing a near meltdown of a pair of dual-processor 2.8GHz servers.
Web spiders are now banned from our CGI directory.
Posted by: Pixy Misa at 10:10 AM | Comments (2) | Add Comment | Trackbacks (Suck)
Posted by: Susie at April 05, 2006 12:03 PM (a0oF7)
Processing 0.01, elapsed 0.0044 seconds.
16 queries taking 0.0028 seconds, 23 records returned.
Page size 8 kb.
Powered by Minx 0.8 beta.