December 25, 2006

Life

Dear Santa

Please could I have for Christmas:

  • A laser printer.
  • A rubber ducky.
  • And a winning lottery ticket.

...

Ooh, thanks Santa!*

* Okay, so it was not a very winning lottery ticket...

Posted by: Pixy Misa at 07:57 PM | Comments (387) | Add Comment | Trackbacks (Suck)

Geek

Irony

IronPython is a version of Python written in C# for the .Net platform. It's intended as an embedded scripting language for .Net applications, but can also be used to quickly assemble applications from components written in other .Net languages. Since I'm currently working in both Python and .Net, this is of interest to me, so I downloaded IronPython* and ran my little benchmark.

And then I went and had dinner, and read a book, and read some blogs, and had some ice cream, and then came and posted this. Because IronPython has a leetle performance problem relative to CPython**; or rather, it lacks at least one very important optimisation from CPython. See if you can spot what it is:

SystemCPUClockPythonLoopStringScanTotal
AmeliaCore Duo1.66GHz2.5 (Win)1.2433.1581.0335.434
AmeliaCore Duo1.66GHz2.5 (Win)+Psyco0.0370.4831.1901.710
AmeliaCore Duo1.66GHzIronPython 1.010.6982236.3192.0452239.062

Source code for the benchmark can be found in my earlier post.

* Again; I had it on my old notebook, but that got reformatted and reinstalled at least four times, and in any case I no longer have it.

** The standard version of Python, which is written in C. more...

Posted by: Pixy Misa at 01:57 AM | Comments (37) | Add Comment | Trackbacks (Suck)

Geek

Windows Vista: The Paranoid Schizophrenic Operating System That's Coming Soon To A PC Near You If You're Not Careful

Analysis by Peter Gutmann.

Posted by: Pixy Misa at 01:12 AM | Comments (1279) | Add Comment | Trackbacks (Suck)

Life

Merry Christmas Eveyone!

And stuff.

Posted by: Pixy Misa at 12:11 AM | Comments (38) | Add Comment | Trackbacks (Suck)

December 23, 2006

Geek

Fate Worse Than Death

Just rescued Akane from a load average of over 800, without rebooting.

Spam flood. Double spam flood, in fact - blog comments and incoming email.

Fluffy wasn't running, and Apache took over the machine, and then exim took it the rest of the way. By the time I logged in it was using nearly 100% of swap.

Posted by: Pixy Misa at 03:07 PM | Comments (92) | Add Comment | Trackbacks (Suck)

Geek

Calculate Calculate...

Half a petabyte a month? Eek.

Posted by: Pixy Misa at 05:14 AM | Comments (27) | Add Comment | Trackbacks (Suck)

December 22, 2006

Anime

Yays!

Haruhi-sama!

Hope they do a good job.

Steven:

There's something very strange about the visitor counter at the top. When I visit it with IE, right now, it said 000816. Revisiting it a few minutes later it said 000823. But when I visit it using Firefox, it said 172600. A bit later it said 172661.

Yes indeed.

Posted by: Pixy Misa at 11:43 PM | Comments (730) | Add Comment | Trackbacks (Suck)

Geek

I've Got You Under My Thumb

For some time I've been running a mix of SpamAssassin on the server and Thunderbird junk mail controls and message filters to keep my email under control. I had SpamAssassin just marking email, not deleting it, and then went into my Junk folder every two or three days to see if there was anything that had been misdirected that I needed to attend to.

Over the last few months, the frequency of my visits to the Junk folder have increased to every day, then to two or three times a day, then every two or three hours. Even though I have filters set to auto-delete the obvious crap, I'm getting something like 2000 emails a day, 99% of them spam.

No more.

Today I set SpamAssassin to auto-delete at 9, and set the value for BAYES_99 to 7. (I'm tempted to bump that up to 8.8.)

Unfortunately, since I monitor email for all of mu.nu - bounces, abuse reports, that sort of thing - I need to receive things that set off a lot of normal spam flags. I don't want to, but if I don't keep an eye on things then sooner or later I'll be hearing from our hosting company. (The PHP spam incident I refer to below was one that I caught and fixed without anyone at SoftLayer being troubled because I saw the sudden increase in bounce messages.)

Fortunately the Bayesian filtering in SpamAssassin seems to be very reliable, and a 99% rating plus two points worth of other spammy traits will now relieve me of ever seeing the message. I've seen valid messages get Bayes ratings of 40 or so, but that won't get zapped unless you're doing a lot of other things wrong.

Seems to be working. I'm still getting the occasional spam through to Thunderbird - at a setting of 9, SpamAssassin doesn't catch those blasted PHYA pump-and-dump spams - but most of it is then gets eaten by my filter rules.

And a blessed peace descended upon Pixy Central...

Posted by: Pixy Misa at 03:11 PM | Comments (37) | Add Comment | Trackbacks (Suck)

Life

Bah

DMCA takedowns and curt legal letters,
Government agents tracing overwrought netters,
Servers I've paid for that reject my pings,
Just a few of my least favourite things.

Spammers who sneak in through PHP errors,
Hackers from Turkey spreading their brand of terrors,
Disk drives that die when they're put to the test -
It's Christmastime, guys! Won't you give it a rest?!

Posted by: Pixy Misa at 09:59 AM | Comments (35) | Add Comment | Trackbacks (Suck)

December 21, 2006

Geek

Fun With Windows Networking

One of the best things about Windows networking is that when a Windows PC loses a network connection - say, you unplug the cable - it can no longer see itself.

This matters to me more than most because I run VMWare Server on my notebook, and when I'm not connected to my wireless network I can't connect to my virtual Linux server either. Even though it's running on the same box.

So what I did was this: I installed the Microsoft Loopback Adaptor, and bridged it to my wireless network. So now, with or without wireless, I can still reach my server.

The only problem is, Windows now tells me that my wireless is disconnected. And if you go into the wireless networks screen, right below where it says "Not connected", it has a Disconnect button to disconnect the unconnected network.

Apart from that, it works fine.

I was having trouble mounting the Samba share from the virtual Linux server back to Windows, but that was because Plesk doesn't (or at least, didn't) automatically apply its firewall rules after a reboot.* Quite possible something I did wrong, but certainly confusing.

Plesk also has a nicely confusing way of managing your Samba shares; it adds an include file to the standard smb.conf, puts the shares it's managing in there, and ignores what's defined in the main file. That might actually be the best way to handle things, but it ain't obvious when you run into it for the first time.

I also set up a PPTP VPN to SoftLayer from my home network. That lets me manage the servers even if the regular public network at the hosting company is down (the VPN goes through a dedicated Cogent link); it lets me run backups without touching by 4TB/month bandwidth allowance; and it lets me do things like check on CPU temperatures and voltage levels in the servers. And do both soft and hard resets, if I should feel the urge. Without even opening a browser window!

* Yes, I have a complete web hosting management system installed on a virtual machine on my notebook.

You mean to say you don't?

Posted by: Pixy Misa at 01:25 AM | Comments (36) | Add Comment | Trackbacks (Suck)

December 19, 2006

Rant

I Got A Grub

Pray, does it boot?

Nnnnot really.

WELL, IT'S HARDLY A BLOODY BOOT LOADER, IS IT?!!???!!?

Look, if you go to my brother's computer shop in Padstow, he'll replace the operating system for you.

Okay, so after all that, it just sits there saying GRUB. Remarkable boot loader, GRUB. Beautiful plumage.

I don't think it liked booting from a software mirror for some reason. FC5 was perfectly happy with it. CentOS 4.4, on the other hand, promptly joined the bleedin' choir invisible.

Excuse me, this is irrelevant, isn't it?

Yeah, well it's not easy to pad these Python files out to 150 lines, you know.

See also: This.

Posted by: Pixy Misa at 04:32 PM | Comments (20) | Add Comment | Trackbacks (Suck)

Geek

Aaarrgh!

Installing the new servers at PPoE.

I brought the FC5_x64 DVD with me instead of FC5_i386, but what the heck, the machines are 64 bit and it's officially supported by the software I want to install, so roll with it.

So I install FC5, and then I install the software I want to install, and it gets allll the way through downloading and installing itself and falls over with three dozen version conflicts in various RPMs.

So now I'm reinstalling CentOS 4.4, which is the only other Linux I have on me, but which I know definitely works. Unfortunately, all I have is the CDs, so I have to sit here swapping them back and forth between the machines.

The best part is when, two minutes into CD #4, it asks for CD #1 again. When CD #1 is in another machine.

Pfui.

Posted by: Pixy Misa at 04:11 PM | Comments (630) | Add Comment | Trackbacks (Suck)

Anime

Shananana

Yeah, Shakugan no Shana sucked me right in too. Watched the whole thing in four days.

Posted by: Pixy Misa at 02:04 PM | Comments (122) | Add Comment | Trackbacks (Suck)

Geek

It's A Conspiracy!!!

Went back to my PPoE to install some new machines that didn't arrive before I left. I thought I'd seize the opportunity to run my Python benchmark on them.

...

They're 2GHz Athlon 64 X2s. The results are Exactly. The. Bloody. Same.

I guess that 2GHz Athlons are the baseline for decent performance, for either desktops or servers. But I got my first one in 2003! Okay, so the new ones are 64-bit and dual core (and about 40% cheaper, from memory), but still...

Posted by: Pixy Misa at 02:02 PM | Comments (73) | Add Comment | Trackbacks (Suck)

Anime

There Is No God But Chiyo-chan

And Tadakichi-san is her prophet.

And trusty steed.

Posted by: Pixy Misa at 01:25 PM | Comments (996) | Add Comment | Trackbacks (Suck)

December 18, 2006

Anime

Keiichi Is A Wuss!!

Get a grip? It takes fifteen volumes of manga for him to get up the courage to even touch Belldandy.

Posted by: Pixy Misa at 04:20 PM | Comments (36) | Add Comment | Trackbacks (Suck)

Geek

So There I Was

Looking for an open-source BitTorrent tracker written in Python that I could hook into Minx...

Doh!

Yeah. That one might work.

Update: The art of helpful commenting:

# code duplication because ow.
I know the feeling.

Update: Yuggech.

Okay, so it's been a while since I installed the standard BitTorrent package on Linux.

The last time I did so was 4.0.something. The current release is 5.0.3.

4.0.x required Python 2.2 or later.

As for 5.0.3:

* Install Python, version 2.3 or later. (2.4 recommended) -- Fine. Got that.
* Install wxWidgets, version 2.6 or later, in the Unicode flavor. -- I'm not installing wxWidgets on a server just to make BitTorrent happy! I hope that this is only actually required for the GUI apps.
* Install wxPython, version 2.6 or later, in the Unicode flavor. -- Sod off.
* Install GTK, version 2.6 or later. -- Bugger that.
* Install Twisted, version 2 or later. -- No.
* Install PyCrypto, version 2 or later. -- Now that is actually a darn good idea.
* Install Psyco. -- Psyco is good. Requiring Psyco is very very bad.
* Install Zope.Interface. -- Sod off!
Fortunately, bittorrent.com provides a download site for previous versions. Version 4.4.0 (from January this year) appears to be the last release before they went dependency-crazy. It uses Twisted if you have it installed, but falls back to its own code if Twisted is not available. That's fine.

So 4.4.0 it is. 4.4 seems to be the end of a stable release, followed by a completely new release with these features:

* completely new UI -- Don't need it or want it.
* smart download/queuing behavior -- Don't need it.
* smart seeding behavior -- Don't need it.
* torrent priority system -- Don't need it.
* detailed progress bar and "piece bar" progress bar -- Don't need it.
* better save location management ("incomplete" and "finished" locations) -- Completely irrelevant.
* automatic bandwidth management - Hmm.
* download rate control -- Don't need it.
* transfer rate graphs -- Don't need it.
* choose which files to download first from a torrent -- Don't need it.
* better error handling and reporting -- Hmm.
* fast extensions (see bittorrent.org) -- Useful, yes, but not here.
* torrent "title" support (see bittorrent.org) -- There's nothing on bittorrent.org!
* multiple tracker support -- Actually, that might be something I could use.
* encryption support -- Don't need it.
* Zeroconf ("Rendezvous") local discovery -- Don't need it.
* sparse files -- Don't need it.
* threaded Disk IO -- Don't need it.
* translation update system -- Don't need it.
* now using wxWidgets instead of GTK on Windows, GUI is now native and should be a lot more stable -- Don't need it, don't want it, just make it go away!
* removed support for Python 2.2 -- Whatev.
I guess I can use 4.4.0 and then libtorrent if I need to.

Thanks, BitTorrent peeps, for at least keeping 4.4 around.

Update: libtorrent won't compile. Somewhere, deep in the jungle, a library is missing a .h file. Or something. Ick.

Update: The other thing I want to note, is that for something so popular, BitTorrent is incredibly underdocumented. The only place that the options to the tracker module are documented are in the program, and the only way to find them is by running the program, because they're all in translation files, and if you don't have all the crazy dependencies installed you can't run the program.

Blah.

Here's to statically-linked binaries. Yeah, I know they suck too. But they suck differently.

Posted by: Pixy Misa at 01:40 PM | Comments (44) | Add Comment | Trackbacks (Suck)

Geek

Just The Thing

32-bit signed int measured in microseconds?

Just the thing for timing a long-running benchmark.

Sigh.

Posted by: Pixy Misa at 08:51 AM | Comments (78) | Add Comment | Trackbacks (Suck)

Geek

Isolated Out Of Existence

I've been hot on the track of my here-again/gone-again memory leak since about 1 AM. It didn't show up during my torture test, but when I changed the test to find out how many realistic pages I could generate per second - as opposed to 5MB monsters - it was back in spades.

First I suspected that it might be the thread-local data, which can persist between requests. But a bit of debugging showed that this data wasn't growing.

And a bit more debugging showed that the leak still occurred when I disconnected the entire Minx module from the dispatch code and just returned a static string.

And a bit more debugging showed that the leak even occurred with the CherryPy "Hello, World!" example.

If the tools.sessions option was set in the config file.

So then - a bug in the CherryPy session tracking code!

...

Not quite. Well, possibly a small one, but I'm not certain of that.

By default, CherryPy sessions last for 60 minutes, and (also by default) it can be another 5 minutes before the background cleanup task removes them.

Meanwhile, I was hitting CherryPy with anything up to 150 requests per second. And since I was using wget for the testing, rather than a browser, I was throwing the session cookies away.

So I had something like 500,000 sessions active at any time. And CherryPy appears to use about 300 bytes per session.

I didn't notice this with my torture test because that was generating less than 1000 requests an hour. 300kB of overhead is hard to spot on a 70MB process (10 threads, 5MB pages). 150MB of overhead on a 25MB process (100 threads, 100kB pages) is rather more noticeable.

There's still a problem with Minx slowing down over time. The page generation time in my current test has slowed by 75μs over the last 50,000 pages. I had chalked this trend up to the memory leak, but there may be another problem I need to address.

Well, at least I've made progress. I haven't fixed anything - except my test bed - but I have ascertained that the bug doesn't exist.

And I also know that CherryPy can cope with 500,000 simultaneous sessions...

Posted by: Pixy Misa at 04:35 AM | Comments (87) | Add Comment | Trackbacks (Suck)

December 17, 2006

Geek

Hah

I just finished replacing the SQL-based page caching in Minx with memcached. Which was actually dead easy, and works, and speeds things up not at all... Though memcached is putting about one-third the load on the server as MySQL in the same test.

Currently I'm seeing a time per page of 7.0 ms and 87 pages per second. The system load is about 60% Python, 2.3% memcached, 1.1% MySQL, and 33% idle. I can't seem to push it much higher than that with my current test framework.

Which tells me that:

1. It's plenty fast.
2. memcached uses no CPU, in the broad scheme of things.

The live deployment will have multiple instances of Minx per server, and load-balance across them as well as across the servers themselves. And the servers will be about 3.5 times as fast (comparing the Xeon 3060 to the Athlon XP 2800+).

Multiply it out and I'll get, hmm, about 465 pages per second.

As for the 7ms of overhead, well, even cached pages get passed through the template engine (because there are template tags that take effect after the page is generated) and it still has to check your login against the database (to see if you have any preference settings that would require a custom page to be presented) and so on. I'll try to keep the overhead down, of course, but that's not too terrible, considering.

Now I'll leave it to soak. Python is using 85MB at the moment (with 50 threads), but seems to be creeping up a bit. One odd thing with CherryPy is that the virtual space is much larger - currently 650MB. Which limits me to about 300 threads total per instance.

The other thing I noticed is that memcached was developed by the people at LiveJournal. LiveJournal has since been acquired by SixApart. SixApart are the Movable Type people. I originally started writing Minx just so I could get off Movable Type.

Well, I was amused.

Update: Hmm. 96MB now. Admittedly, it reached 70MB with 10 threads after a long soak, and I was expecting it to be on the order of 350MB this time.

As long as it stops growing eventually.

Update: 121MB and 8.2ms per page at the 130,000 page mark. That smells like a leak, but a pretty small one.

Posted by: Pixy Misa at 07:42 PM | Comments (39) | Add Comment | Trackbacks (Suck)

Anime

Expectations

Steven on Kamichu!

I have never seen a series which was as uneven as this one has been. My earlier impression remains in force: those behind the series never actually made their minds up as to exactly what kind of story they were trying to tell. As a result, it feels episodic, in the worst sense of that word.
It's a slice-of-life series. It's just a series of tales from Yurie's first year as a god.

That's what I expected after seeing the first episode - I don't know why, but I immediately concluded it was going to be that sort of show. And it was, so I was not disappointed. Or at least, far less disappointed, as even given that understanding the show doesn't quite live up to its early promise. (Also, I'm not nearly as allergic to middle-school angst as Steven.)

However, given that the early promise was of a TV series of the quality of Spirited Away, I'm not going to hold it against them that they didn't consistently achieve that.

I give it a 5/5, and rate it as the best show of 2005.

Oh, and the cat epsiode was silly, but Yurie-neko redeems all.

Posted by: Pixy Misa at 01:28 PM | Comments (229) | Add Comment | Trackbacks (Suck)

Blog

Oops

Deleting spam, and I zapped some non-spam. Sorry about that.

Posted by: Pixy Misa at 01:20 PM | Comments (1030) | Add Comment | Trackbacks (Suck)

Geek

All My Servers Are The Same Speed!

I was going to run some little benchmarks to compare the speed of my web servers (Akane, Nabiki and Ranma) and my development servers at home (Naga and Martina).

But the results weren't very interesting, because they're all the same.

Akane and Nabiki are Opteron 170s: dual-core 64-bit 2.0GHz.
Ranma is an Athlon XP 3000+: single-core 32-bit 2.16GHz.
Martina is an Athlon XP 2800+: single-core 32-bit 2.08GHz.
Naga is an Athlon 64 3200+: single-core 64-bit 2.0GHz.

Naga is the only one running a 64-bit kernel, and hence a 64-bit Python.

I also checked up on Namo, the little Celeron box I got to run The Jawa Report when we were getting DDoSed. Kei and Yuri seem to be down right now... Uh, which is bad. They should be alive until the end of the month.

After making my benchmark self-timing, so that I can run it on Windows, I can add:

Lina: Pentium 4 2.6GHz
Amelia: Core Duo 1.66GHz
Haruhi: Core 2 Duo 2.4GHz

I'm not going to bother running tests on Sylphiel and Kyon, my Linux virtual machines, because the times vary by +/- 20% due to the clock problems I mentioned earlier.

SystemCPUClockPythonLoopStringScanTotal
NagaAthlon 642.0GHz2.5/64-bit1.7372.0301.3375.103
AkaneOpteron2.0GHz2.51.8872.7330.8805.500
MartinaAthlon XP2.08GHz2.51.8172.7930.8675.447
RanmaAthlon XP2.16GHz2.51.7602.6970.8405.297
LinaPentium 42.6GHz2.5 (Win)2.0385.0580.8757.971
HaruhiCore 2 Duo2.4GHz2.5 (Win)0.6441.9330.4773.053
AmeliaCore Duo1.66GHz2.5 (Win)1.2433.1581.0335.434
NamoCeleron1.7GHz2.4.33.0474.8931.9609.900

You can see why all the servers for the New Site are going to be Core 2 Duos. I'll probably migrate mu.nu to Core 2 Duo servers some time in '07 as well - the monthly charge is the same. I just have to wait until SoftLayer are offering their double-memory deal again.

And here's the reason why I'm going to stick with the 32-bit kernel for the application servers:

SystemCPUClockPythonLoopStringScanTotal
HaruhiCore 2 Duo2.4GHz2.5 (Win)+Psyco0.0120.2730.5540.839

Psyco is a JIT compiler for Python. While it doesn't always improve performace, when it does, the advantage can be huge, and you can tell it to compile only specific functions if you need to.

Psyco produces 32-bit code. It doesn't work at all on 64-bit Python, and if you run a 32-bit Python on a 64-bit kernel, the overhead of the compatibility layer makes Psyco slower than the standard interpreter.

more...

Posted by: Pixy Misa at 11:54 AM | Comments (570) | Add Comment | Trackbacks (Suck)

December 16, 2006

Geek

Playing With Plesk

I've been playing with Plesk a bit lately. mu.nu has been running on CPanel for the past three years, and I've come to rather dislike it, not least because they still haven't managed to get it to work with Apache 2.x.

Plesk seems to be pretty good - though admittedly I don't have 250 users on it yet. It's certainly prettier and better organised. CPanel with the Xcontroller theme isn't bad at all, but that does nothing for the administration system that lives behind CPanel - WHM - which is what I get to use. "Disorganised" is too kind a word for WHM, unless you were to prefix it with "hopelessly".

I bought a license for Plesk to try it out; I probably didn't need to do that, since as it happens you can download it for free as long as you only want one domain and one email address. I have it installed on Martina and Sylphiel that way; my license key languishes unused. And I got it installed on Ranma as well.

Both my bought license and Ranma's monthly license are for 30 domains. One thing about CPanel - it costs a bundle, but there are no limits to the number of users you can load onto it. Plesk comes in 30, 100 and 300 domain licenses as well as the even more expensive unlimited version.

And, as I looked at putting my site on Ranma, I counted the domains. I set up ranma.mu.nu as a valid web site, so that Host-Tracker has something to track. I have seven pixymisa domains (com, net, org, nu, biz, info, and name*), four domains pointing to this blog (ambientirony.com/net/mu.nu and ai.mu.nu), then there's animeclips.mu.nu, and whatever my short-lived anime downloads blog was.** So that's 17 out of 30 gone right away.

Well, as it turns out, it's a bit better than that. It's not 30 domains, exactly. Subdomains don't count. Neither do add-on domains - domains that point to the same site as an existing one. And since subdomains seem to be able to have their own FTP and web user logins, and mu.nu is of course just one domain, and most of the domains here that aren't mu.nu are actually add-ons to mu.nu subdomains, I could probably host the entire thing sprawling mass of Munuvia on a single 30 domain Plesk license.

Or maybe not, since I can't work out how to point an add-on domain to a subdomain.

But it certainly is pretty.

And while CPanel has not, after three years, managed to support Apache 2.x***, Plesk in the same amount of time finally managed to incorporate Awstats.

On the yet-another hand, their SpamAssassin module is an extra $50.

Meh.

* Uh... they were cheap.

** I'd revive it, except that it was largely a front-end to my Broadcast Machine installation, and I ran into some nasty bugs in Broadcast Machine about the same time the lead developer called it quits. (I think that's what happened. There hasn't been even a patch for it issued in six months.) I'll probably revive it anyway, now that I have another 1.5 terabytes a month to throw around and don't need BitTorrent so much. Minx has some features for doing Broadcast Machine style sites (if anyone remembers what it looked like), but doesn't do torrents, and won't for some time.

*** Okay, it appears that they recently added Apache 2.0 support to the "EDGE" release, essentially their public beta. Of course, Apache is now on 2.2.

Posted by: Pixy Misa at 01:10 AM | Comments (56) | Add Comment | Trackbacks (Suck)

December 15, 2006

Anime

A Bag Of Lolis

Chocotto Sister.

Posted by: Pixy Misa at 09:25 PM | Comments (42) | Add Comment | Trackbacks (Suck)

<< Page 3 >>

Processing 0.26, elapsed 0.7553 seconds.
49 queries taking 0.5284 seconds, 1603 records returned.
Page size 1217 kb.
Powered by Minx 0.8 beta.