I've been hot on the track of my here-again/gone-again memory leak since about 1 AM. It didn't show up during my torture test, but when I changed the test to find out how many realistic pages I could generate per second - as opposed to 5MB monsters - it was back in spades.

First I suspected that it might be the thread-local data, which can persist between requests. But a bit of debugging showed that this data wasn't growing.

And a bit more debugging showed that the leak still occurred when I disconnected the entire Minx module from the dispatch code and just returned a static string.

And a bit more debugging showed that the leak even occurred with the CherryPy "Hello, World!" example.

If the tools.sessions option was set in the config file.

So then - a bug in the CherryPy session tracking code!

...

Not quite. Well, possibly a small one, but I'm not certain of that.

By default, CherryPy sessions last for 60 minutes, and (also by default) it can be another 5 minutes before the background cleanup task removes them.

Meanwhile, I was hitting CherryPy with anything up to 150 requests per second. And since I was using wget for the testing, rather than a browser, I was throwing the session cookies away.

So I had something like 500,000 sessions active at any time. And CherryPy appears to use about 300 bytes per session.

I didn't notice this with my torture test because that was generating less than 1000 requests an hour. 300kB of overhead is hard to spot on a 70MB process (10 threads, 5MB pages). 150MB of overhead on a 25MB process (100 threads, 100kB pages) is rather more noticeable.

There's still a problem with Minx slowing down over time. The page generation time in my current test has slowed by 75μs over the last 50,000 pages. I had chalked this trend up to the memory leak, but there may be another problem I need to address.

Well, at least I've made progress. I haven't fixed anything - except my test bed - but I have ascertained that the bug doesn't exist.

And I also know that CherryPy can cope with 500,000 simultaneous sessions...

Posted by: Pixy Misa at 04:35 AM | Comments (87) | Add Comment | Trackbacks (Suck)