Who Writes This Brap?

(Yeah, that's a typo, but I decided to leave it.)

As I've noted before, Movable Type is robust and rich in features, but really really slow.

I've wondered if it's something to do with using Berkeley DB. Some people on the MT support forum have claimed this is the case, but they have not impressed me as particularly knowledgable. Berkeley itself is very fast anyway, so it would have to be a problem with MT's use of it rather than Berkeley itself - though that is quite possible.

Anyway, there's a CGI script available in MT to convert from a Berkeley DB to MySQL or PostgreSQL. Two things to note: It only comes with the upgrade package, and not with the "full" version (minor oops), and it doesn't work (big oops).

Well, if you run it as the docs suggest, as a CGI program from your web browser, it will happily create the necessary tables in your SQL database and report that its work is done. Without, mind you, copying any of the data across.

If instead you run it from the command line, it will copy all your data, only very very slowly. It zips through the first part - I'm not sure exactly what that consisted of, but at least it was quick - but when it starts processing the entries it slows to a crawl. Crawwwwwwwwwl.

Once it's finished, which should be within the next 10 or 12 hours (seriously!), I'll rerun my little stress-test.

Oh, yes: I've written a template which exports your entire Movable Type system in a nice convenient format. Just the ticket if you want to move off MT and onto a more modern and efficient sytem. (Cough Minx cough.)

It takes two hours to run on mu.nu. That's on a lightly-loaded Athlon XP 2500+ with 1GB of memory.

Pfft. But at least it's better than mt-db2sql.cgi.

Posted by: Pixy Misa at 08:24 PM

Comments

1 4% done...

Posted by: Pixy Misa at April 17, 2004 08:31 PM (+S1Ft)

2 6%...

Posted by: Pixy Misa at April 17, 2004 08:43 PM (+S1Ft)

3 10% and counting...

Posted by: Pixy Misa at April 17, 2004 08:57 PM (+S1Ft)

4 15%

Posted by: Pixy Misa at April 17, 2004 09:27 PM (+S1Ft)

5 25%

Posted by: Pixy Misa at April 17, 2004 10:18 PM (+S1Ft)

6 About 45% - seems the speed has picked up a little.

Posted by: Pixy Misa at April 17, 2004 11:49 PM (+S1Ft)

7 Oh dear - I hadn't realised MuNuvia was running on Berkley DB. I don't think the problem is with the Berkley system per se but I noticed a definite jump in performance since my move to the new server (and a MySQL backend) something in the region of 30% - 40% faster on rebuilds, comments etc.

Posted by: Rob at April 17, 2004 11:53 PM (BWDMP)

8 Yeah. But I haven't been able to get a definitive answer from anyone about whether MySQL really is faster than Berkeley - except from people who didn't know what they were talking about.

At least - assuming this conversion works - I'll have some hard performance data.

50%...

Posted by: Pixy Misa at April 18, 2004 12:34 AM (+S1Ft)

9 I've run 5GB Berkeley databases and they worked just fine, so I really don't know what MT's problem is.

It has to be either lousy design or lousy code. There's absolutely no reason for performance like this.

Posted by: Pixy Misa at April 18, 2004 12:36 AM (+S1Ft)

10 75%

Assuming there isn't another stage after this one.

Posted by: Pixy Misa at April 18, 2004 01:27 AM (+S1Ft)

11 It depends on the Berkeley version and the mySQL version. I've found mySQL runs somewhat faster, unless you are moving from the latest Berkeley to an older mySQL. Also depends on how the server is set up, etc. etc.
If it is a large (for a blog) database (say over 25 megabytes), I've found mySQL significantly faster. (All of the above is comment on MT blogs, specifically.) I've done about 15 conversions for people, so it's not a huge sample, just a bit better than a single conversion...

Posted by: Kathy K at April 18, 2004 01:48 AM (NqZv3)

12 Thanks for the info, Kathy. Yes, it's large (for, as you say, a blog) - over 160MB.

Another couple of hours and I'll be able to run my test, and we shall see what we shall see. (Though I'm still wondering what the hell it's doing that the conversion takes so long.)

Posted by: Pixy Misa at April 18, 2004 02:15 AM (+S1Ft)

13 Done copying data from Berkeley DB to your SQL database! All went well.

real 456m4.953s
user 7m51.430s
sys 0m48.630s

So, they put a sleep() call in there just to amuse themselves? Or what?

Posted by: Pixy Misa at April 18, 2004 03:44 AM (+S1Ft)

14 Well, my test is off and running. So far I can state confidently that MT running with MySQL is less than 50 times faster than MT with Berkeley DB. Which means that it is still too slow, but we will have to wait to find out whether it is faster, and by how much.

Posted by: Pixy Misa at April 18, 2004 03:53 AM (+S1Ft)

15 Sainted Henry Fnord! Will you look at that!

(As soon as Trickle finishes trickling and buggers off...)

Oh, too late, it's gone.

Okay, two things:

One, my rebuild script runs (I think) 14 times faster with MySQL than with Berkeley DB. Now that is worth a little pain.

Two, it used 880MB of memory. Eeek!

Posted by: Pixy Misa at April 18, 2004 04:07 AM (+S1Ft)

16 I'm just confirming the timing by running my test on the live database again.

Posted by: Pixy Misa at April 18, 2004 04:12 AM (+S1Ft)

Processing 0.0, elapsed 0.0071 seconds.
18 queries taking 0.0048 seconds, 24 records returned.
Page size 10 kb.
Powered by Minx 0.8 beta.