Okay, folks, here's how it works.

MT is robust and has lots of good features, but it's as slow as a dead dog. For example, on an Athlon XP 2800+, with both the code and the database stored in a ram disk, it takes around 28 seconds of CPU time to add a three-word post to Ambient Irony. I haven't yet had time to profile MT to work out what the hell it's doing with all that time. Part of the problem is MT's insistence on static files and its brute-force approach: Whenever you update a post or add a comment, MT will rebuild not only the individual entry, but also the main index, the archive for whatever categories that post is in, and the archives for that day, week and month, and the archives for the previous day, week and month, and, if they exist, the archives for the following day, week and month.

And MT is slow at just rebuilding a single entry, so when it has to run around rebuilding everything in sight, it's really slow.

Without fixing the code, the only way I can see to cure this is to give it less work to do. One way to do this is to disable any archives you don't use. If you don't use categories, make sure that category archives are turned off.

The next thing you can do is to make the templates simpler. And here's where my little trick comes in.

If you have your blog set up like me, the appearance of the posts in the main index is exactly the same as in the monthly archives and the category archives. You could use a sub-template to keep the common data, but there's an even sneakier thing you can do:

Step 1.

Create a new template. Call it something like "Individual Include Template". Copy the section of your main template that deals with the individual entries into this new template. If you are using the default MT template, the result will be this:

<div class="blogbody">

<a name="<$MTEntryID pad="1"$>"></a>
<h3 class="title"><$MTEntryTitle$></h3>

<$MTEntryBody$>

<MTEntryIfExtended>
<span class="extended"><a href="<$MTEntryPermalink$>#more">Continue reading "<$MTEntryTitle$>"</a></span><br />
</MTEntryIfExtended>

<div class="posted">Posted by <$MTEntryAuthor$> at <a href="<$MTEntryPermalink$>"><$MTEntryDate format="%X"$></a>
<MTEntryIfAllowComments>
| <a href="<$MTCGIPath$><$MTCommentScript$>?entry_id=<$MTEntryID$>" onclick="OpenComments(this.href); return false">Comments (<$MTEntryCommentCount$>)</a>
</MTEntryIfAllowComments>
<MTEntryIfAllowPings>
| <a href="<$MTCGIPath$><$MTTrackbackScript$>?__mode=view&entry_id=<$MTEntryID$>" onclick="OpenTrackback(this.href); return false">TrackBack (<$MTEntryTrackbackCount$>)</a>
</MTEntryIfAllowPings>
</div>

</div>

Save the new template.

Step 2.

Now go into the Archiving page in Weblog Config. Click on the Add New button at the bottom, and create a new Individual Entry archive using the template you just created.

You will need to give the files created by this template a new and unique name. This is what the Archive File Template field is for. You will need to use MT tags here to generate the name. A good, simple name might be

<$MTEntryID>.i

Now, save your archive settings and do a full rebuild of your blog to generate all those little .i files, because in the next step you're going to need them.

Step 3.

Go into your Main Index template and (after making a backup copy), cut out all the code you put into your new "Individual Include Template" and replace it with this:

<?php  include 'archives/<$MTEntryID>.i'; ?>

(Assuming that your archive directory is called "archives", which it probably is.)

Save your changes, but don't rebuild yet. Go into the Preferences page in Weblog Config, and scroll down to where it says File extension for archive files This is probably set to html. You will need to change it to php.

And in the Templates page, select your Main Index template and change the output file to index.php.

Save the changes, and now go and rebuild your main index. If you got it right, it will now rebuild without having to process the details of the individual entries. The entries will instead be read automatically from their individual .i files whenever someone visits your blog.

Repeat step 3 for your daily, weekly, monthly and category archives, whichever ones you may be using, except that for these you will not need to specify the "archives" directory as it will look there automatically.

Step 4. (Optional)

Somewhere near the top of your main index, add the following code:

<?php

$timeparts = explode(' ',microtime());
$starttime = $timeparts[1].substr($timeparts[0],1); ?>

And in a suitable place near the end, add this:

<div class="sidetitle">
Render Time
</div>

<div class="side">
<?php
$timeparts = explode(' ',microtime());
$endtime = $timeparts[1].substr($timeparts[0],1);
echo bcsub($endtime,$starttime,6)?> seconds<br>
</div>

This will show you how long PHP takes to put all the individual entries together to build your page. A fairly typical result for Ambient Irony, which has 25 entries on the main page, is 0.005888 seconds. Which I must say is satisfyingly quick.

Having done all of that, is MT now zoomily fast? Alas, no. It will probably halve the time required to add a post or leave a comment, but it's still dog slow compared to what it should be capable of.

ExpressionEngine is quite a bit faster at processing posts and comments - at least ten times faster - but it's between thirty and a hundred times slower at delivering the pages. That's the main reason I decided not to go down that route.

This is basically how Minx works, except that the Minx template engine is about a hundred times faster than Movable Type's. It's just a pity that it's not finished yet.

Update!

I have now disabled the rebuilding of archive and index pages when someone leaves a comment on Ambient Irony. This has reduced the comment processing time down to around 4 seconds. Still slow, but much less slow.

Posted by: Pixy Misa at 11:58 AM | Comments (50) | Add Comment | Trackbacks (Suck)

1 It is a bit too geeky for a small brained marsupial like me to understand ... thanks for thinking about us.

Posted by: Kang A. Roo @ at April 05, 2004 03:12 PM

2 Nodding my head as if I understand.

And what sort of fee is there to have someone else do this?

Posted by: Jennifer @ at April 05, 2004 04:38 PM

3 I've noticed that not all the site have php extensions. What is the procedure for switching to the php extension? How will that affect links etc back to our moan page? How can I mess up the process at least one or two times?

Posted by: Madfish Willie @ at April 05, 2004 04:56 PM

4 D'oh!... Duh... nevermind.

Posted by: Madfish Willie @ at April 05, 2004 04:59 PM

5 Jennifer, your best bet is to ask me in two weeks time how Minx is coming along.

Either it will be doing something useful by then, or it won't be ready before Christmas, and I'll be looking at interim solutions again.

Posted by: Pixy Misa @ at April 05, 2004 05:18 PM

6 Have I mentioned before that you have the patience of a saint, btw?

Posted by: Jennifer @ at April 05, 2004 05:20 PM

7 Yes, but not recently.

Posted by: Pixy Misa @ at April 05, 2004 05:22 PM

8 I'm scared to do it! I'm afraid I'll break everyone's blog! Someone hold me.

Oh, and Willie?? You wrote:
How will that affect links etc back to our moan page

Paging Dr. Freud!!

Heh.

Posted by: Emma @ at April 06, 2004 01:11 AM

9 If everyone did this, I could turn off the rebuilding of archives on comments, and that would speed things up enormously.

I'm still trying to work out how to do that selectively.

Posted by: Pixy Misa @ at April 06, 2004 01:45 AM

10 The flesh is willing, but the mind is weak. I do volunteer to let someone else do it for me, though.

Posted by: Jennifer @ at April 06, 2004 03:39 AM

11 I'm willing. I'll give it a try tonight (my tonight). I'll email you first.

Posted by: Pixy Misa @ at April 06, 2004 03:51 AM

12 I'm with Jen. I read your post and just scratched my head and thought "Wha???" I'll do whatever I need to help speed up the system, but don't want to trip everyone else up. I'm already feeling guilty for all the time Madfish Willie devoted to making my page "pretty". ::tries to pull her own weight out the door::

Posted by: Alex @ at April 06, 2004 04:05 AM

13 Ouch... Must, figure, out, instructions... (insert agonized scream here)
If it helps, I'm in too, but I fear messing with my template much until I actually translate the HTML and CSS into english I really understand...

Posted by: Tim from Backstage @ at April 06, 2004 10:27 AM

14 Naturally, my solution to this is to scream "Paul!"...lol.

And...I don't even HAVE a moan page. Unless you count it when what I post makes ya moan...in pain.

There should be a beverage alert on that comment...rotfl.

Posted by: Stevie @ at April 06, 2004 11:50 AM

15 Well, maybe tomorrow night. Zzzzzz....

Posted by: Pixy Misa @ at April 06, 2004 03:37 PM

16 Hows about if I do the same thing but with SSI? I've always found Apache includes to be faster than PHP ones although as all .htm are being parsed as PHP it's kind of a moot point. (they are all being parsed as PHP aren't they?)

Last time I looked the problem isn't directly with MT, it's with Perl. No support for multiple threads you see and no easy way to do Asyncronous processing... I was rather hoping they'd have ditched Perl in v 3 and started again with somehting a little more robust but never mind ... minx here we come.

Posted by: Rob @ at April 06, 2004 04:08 PM

17 SSI will work too. I've only set up Snoozebuttondreams to parse html as PHP; for the other sites I've actually configured MT to generate .php files. (Like, doing stuff properly! )

The problem isn't Perl - Perl is darn fast at text processing and supports threads (as of version 5.005, but since revised and upgraded.) The problem is that MT is a pure-CGI program (it could launch background tasks to do some of the work, but doesn't), and that there is something seriously wrong with the MT template engine. Every other template engine in wide use is at least an order of magnitude faster than MT, whether it's written in Perl or PHP or Python or shell scripts.

Posted by: Pixy Misa @ at April 06, 2004 04:21 PM

18 *eyes glazing over*

Posted by: Jennifer @ at April 06, 2004 04:32 PM

19 Oh, and SSI may be faster than PHP inlcudes, but I doubt that the difference will be significant. PHP can pull in 25 files in 5 to 6 milliseconds; that's fast enough for all but really extreme cases.

Minx supports both SSI and PHP includes. Insofar as it supports anything...

Posted by: Pixy Misa @ at April 06, 2004 04:38 PM

20 in that case might be worth looking for a global server cnogi thingy that will try to find a .php file if a .htm file returns a 404 - think of all those busted permalinks!!!

Posted by: Rob @ at April 06, 2004 05:11 PM

21 Yeah, that's not a bad idea.

(Cnogi?)

Posted by: Pixy Misa @ at April 06, 2004 05:24 PM

22 mine's just showing the post date(not part of the entry template) and no entries underneath when I follow the instructions. (i restored from by template backup)

i saw the include code in the source of my page.
do I have to turn something on for php to work?

Posted by: SpaceMonkey @ at April 06, 2004 08:36 PM

23 Willie? Have you done this? If so, will you do me, too?

Moan, indeed. [snoooooooort]

Posted by: Emma @ at April 06, 2004 10:17 PM

24 Hmmm... I'll need access to your box to do you.

Posted by: Madfish Willie @ at April 07, 2004 12:40 AM

25 Actually, no I haven't done it... yet. I've got to work out the tweaks for all these sites I've been messing up lately. I just finished a total redesign for Sir John of Argghhh!!! and have to upload some new images for his banners and I have scripty thingy that is acting up. After I do that, I can do my site first... or maybe the MadLab would be a better place to mess up. Then I can do you! WooHoo!!1!

Posted by: Madfish Willie @ at April 07, 2004 12:44 AM

26 Well, I'll just take a number, then. LOL! Whatta busy boy! ;D

Posted by: Emma @ at April 07, 2004 02:13 AM

27 Can I just ask if anyone except Rob and Pixy have a frigging clue as to what they are talking about? Are aliens coming? Is it something about the lottery numbers? Is it in English? ;-)

Posted by: Simon @ at April 07, 2004 03:36 AM

28 I'm trying to do this, but my site is not rebuilding. How long should it take to rebuild a standard index template? (Mine is still pretty basic -- I just want to add my old blogroll and some categories). If nothing happens after 10 minutes, should I try again??

Posted by: John Lanius @ at April 07, 2004 04:05 AM

29 I'm assuming that MuNubies are exempt from this for at least a couple of weeks.

As far as the speed problem is concerned, I only got DSL a few months ago: I blogged for over three quarters of a year using *dialup.* I'm basically a quasi-Mennonite, here. I keep a nail file by my computer and practice my breathing while the machine does its thang.

Posted by: Little Miss Attila @ at April 07, 2004 06:14 AM

30 The Barkeep said "Hmmm... I'll need access to your box to do you." I think he's been spending too much time with Harvey....

Posted by: Susie @ at April 07, 2004 06:27 AM

31 Um, I should add:

Unless you understand what all this is about, don't do it. You will probably end up breaking everything.

Posted by: Pixy Misa @ at April 07, 2004 06:38 AM

32 John, it shouldn't take very long at all for you to rebuild. Less than a minute, anyway.

Posted by: Pixy Misa @ at April 07, 2004 06:40 AM

33 Right, york notes version (for Si and others)

The more code you have in your blog templates, the more work MT has to do when you post / rebuild becasue it has to "read" (parse) all this code to find out where to put the bloggy bits. These bloggy bits are the sections of a page that change with every entry - title, category, entry, extended entry, etc. The rest of your page is always the same so there is no point in having MT read it.

What Pixy's technique does is reduce the amount of code that MT has to read by seperating the bloggy bits into a new file. That new file is then "included" (think of it as a automatic copy and paste) into the right place in your templates.

The upshot - less code means faster rebuilds.
The down side? None!

w00t!

Oh and Cnogi is the worst spelling of config I've ever managed - I AM THE TYPO KING!

Posted by: Rob @ at April 07, 2004 12:18 PM

34 Oh and if anyone would like assistance in implementing this then by all means send me an email and I'll be happy to help

robert@xset.co.uk

Posted by: Rob @ at April 07, 2004 12:26 PM

35 Unless you understand what all this is about, don't do it. You will probably end up breaking everything.

That settles it. I am definitely better off in the Don't Do It camp (but I've no objection to anyone else doing it on my site at their convenience.)

Posted by: Debbye @ at April 08, 2004 04:31 AM

36 OK, I got the individual entry template working. I'm generating the MTentry.i files ( looked at them, code looks exactly like it does in my main index file) but the 'php include' command is not includin', the php include is in the source of the index, any advice?

Posted by: SpaceMonkey @ at April 08, 2004 05:30 AM

37 Oops, I missed a step. In templates, you have to change the output file for your Main Index from index.html to index.php.

I've done this for you.

Posted by: Pixy Misa @ at April 08, 2004 11:03 AM

38 Hey, now it works!

Posted by: SpaceMonkey @ at April 08, 2004 01:28 PM

39 Ok, it turns out that using Apache includes doesn't work as easily as I'd hoped ... you still have to change the file extension to .shtml and so the advantages are lost.
I can't seem to get the XBitHack on to work as I can't CHMod HTML files to be executable so it looks like I'm stuck with the PHP include.

Pixy, any plans on that global apache handler to parse all HTML as PHP? Apparently it can cause quite a large increase in system resources

Posted by: Rob @ at April 08, 2004 03:13 PM

40 Rob, I'm giving it serious consideration. Since we seem to be heading in the direction of using PHP for everything anyway, and we have plenty of CPU, it probably won't hurt. Much.

Posted by: Pixy Misa @ at April 08, 2004 11:06 PM

Hide Comments | Add Comment