July 12, 2006
Got It!
Minx is now pie-ified. That's reduced the overhead per page from about 160ms to something like 6ms. Since a page fetched from the cache takes about 2ms to process, and an individual entry page about 12ms, that makes the whole thing just a little bit zippier.
Need to do some more bug testing and performance testing, but it seems stable in terms of speed and memory after coughing up 60,000 pages. With 10 threads, it uses 11MB of memory, though its virtual memory footprint is 114MB. Not entirely sure why it is allocating all that memory and never using it, but since the only real problem that causes is that I can't have more than about 350 threads running in any one Minx instance (119MB real, 2935MB virtual), I can probably live with it. And that only applies on 32-bit platforms anyway.
One thing I'm not doing right now is running with Psyco. Even in the worst case (cached pages), it gives a performance boost of 20%. But it also leaks memory like Netscape 4.5.
Posted by: Pixy Misa at 06:31 PM | Comments (4) | Add Comment | Trackbacks (Suck)
1
Memory leaks are up there with "windows of opportunity" as the worst debugging headaches there are. It's tempting to deal with a memory leak by making periodic reboots part of the normal operating instructions, but that really isn't the right answer.
Posted by: Steven Den Beste at July 13, 2006 12:20 PM (+rSRq)
Processing 0.01, elapsed 0.01 seconds.
16 queries taking 0.0067 seconds, 23 records returned.
Page size 8 kb.
Powered by Minx 0.8 beta.