June 13, 2006

Cool

Whoa!

Also Woo!

Got home, plugged the 360 into the new TV (a Samsung LA27T51B), loaded Oblivion and -

Niiiiice.

I was playing it on my computer monitor before. That's a 19" Samsung multifunction, with DVI/VGA/Component/S-Video/Composite/TV inputs, and it's a very nice computer monitor. But it's only 19", and it's 4:3, not widescreen. Effectively (if I can still extract square roots in my head correctly) it's about 17" for a widescreen image.

The new set is 27", which doesn't sound that much larger, but it is.

And the picture quality, even on component input, is superb.

Let's see what my notebook makes of it...

Hmm. Works - almost.

At 1280x768, I get a perfect picture. But it's not a 1280x768 screen, it's 1366x768. 16:9, and bugger sensible resolutions. So if I run it in that mode, everything will be about 6% wider than it should be.

My notebook actually recognises the resolution, and offers an option for 1360x768 (which is what the manual says it should do). But either the notebook or the monitor (and I hope it's the notebook) gets the timing wrong, and the image starts about one fifth of the way across the screen; the leftmost fifth is blank.

1280x720 would get rescaled, but would have the proportions right. But the monitor comes up Mode not supported. Bah. The manual says that it supports all modes between 640x480 and 1360x768, but that don't seem to work so well.

Hmm.

Okay, let's see what we can adjust on the monitor. Fiddle, fiddle, fiddle. Damn. Even at the maximum settings, I can't get the picture quite right. Well, let's try the Auto Adjust, just for laughs.

Hmm. Close, but there's still grey bars down the left and right, which will get kind of annoying. If I could get rid of...

Oi!

What?

Your wallpaper is 1280x1024, right?

Yes?

And this monitor is?

1360 by... oh. Right.

Okay, now that it's working perfectly, let's fire up some widescren video, like, oh, the Kamichu! opening.

...

Okay. I'm sold.

Posted by: Pixy Misa at 08:47 PM | Comments (8) | Add Comment | Trackbacks (Suck)

Geek

Where The Time Goes

Timings:

Standard PythonPython + Psyco
No Comments0.260.22
Inline Comments0.920.68
Sanitised Comments1.310.92

For last 500 entries on Munuviana. All times in seconds. Offer void where prohibited by federal, state or local laws, regulations or institutional policy. Offer ends June 30, 2006. Benchmarked on a Pentium D 820 running Centos 4.2. Python 2.4.2 compiled with GCC 3.4.4. Mileage may vary. Contents may ship during shrinking. Do not eat iPod shuffle. Sanitisation of comments only guarantees valid HTML. The content is your own problem.

Posted by: Pixy Misa at 04:58 PM | Comments (5) | Add Comment | Trackbacks (Suck)

Rant

Well, Crap

DVI does not support PC function.

It's still got a VGA input, but that's just stupid.

Posted by: Pixy Misa at 02:03 PM | Comments (2) | Add Comment | Trackbacks (Suck)

Life

Wait Training

They say that which does not kill us, makes us stronger and it must be true, because this morning I tucked a 27" TV under my arm and carried it home.

Okay, so it's an LCD and only weighs 12kg. Pfft.

Posted by: Pixy Misa at 11:39 AM | Comments (2) | Add Comment | Trackbacks (Suck)

Anime

Galaxy Express 754

MagiPoka episode 16. (That is, the second half of episode 8.)

Both funny and moving. This is turning out to be quite a good series.

Posted by: Pixy Misa at 02:10 AM | Comments (5) | Add Comment | Trackbacks (Suck)

June 12, 2006

Life

The Karma Kittens Come Home To Roost

My dishwasher just blew up.

I see a week of take-away in my future.

Posted by: Pixy Misa at 02:49 PM | Comments (9) | Add Comment | Trackbacks (Suck)

Geek

Fun With Templates

Minx is currently parasitic* on the MT database and user interface, but it has a template system all its own.

The pages are now fully templated, with no funny stuff going on, so I'll show you what it looks like:

Page Template
<html><head>
<title>Minx Dynamic Pages for Movable Type</title>
<link rel="stylesheet" href="http://ai.mu.nu/styles-site.css" type="text/css" />
<script language="JavaScript" src="http://blog2.mu.nu/cgi/showhide.js"></script>
</head>
<body>
<div id="container"><div id="center"><div class="content">
<center><b>[blog.name]</b></center><p>
[posts:here]
[magic.pager]
[magic.stats]
</div></div></div>
</body></html>
Very simple. You start with a standard HTML/XHTML page, and the Minx tags are marked with [square brackets]. If you have something in your template that's in square brackets that's not a Minx tag - intentionally or unintentionally - it just gets spat out unchanged. (The divs look a bit odd, but that's because I'm using an unmodified MT3 stylesheet for testing.)

We can see three types of tags here:
A simple tag, [blog.name], which just looks up the matching database field for the currently active blog and inserts it into the page at that point.

A here tag, which is used to simplify block processing. The [posts.here] and [comments.here] are the most common examples of this. Without any parameters** they loop through the posts or comments using the default settings for your blog and the default post/comment template, as appropriate.

The post and comment templates look like this:

Post Template
<h2>[post.newdate]</h2><p>
<b>[post.title]</b><p>
[post.text]  
<p class="posted">
Posted by: [post.authorlink] at
<a href="[post.url]">[post.time]</a>
| <a href="#" onClick="ShowHide('cc[post.id]'); return false;">Comments ([post.comments])</a>
| <a href="http://blog2.mu.nu/cgi/mcomment.cgi?post=[post.id]">Add Comment</a>
| Trackbacks (Suck)
</p>   
<div id="cc[post.id]" style="display:none">
[comments:here]
<p class="posted">
<a href="#" onClick="ShowHide('cc[post.id]'); return false;">Hide Comments</a>
| <a href="http://blog2.mu.nu/cgi/mcomment.cgi?post=[post.id]">Add Comment</a>
</p>   
</div>

Comment Template

<div id="c[comment.id]">[comment.text]</div>
<p class="posted">
Posted by: [comment.authorlink] at [comment.datetime] </p>
Again, we have regular HTML, with just a bunch of simple tags to insert the desired values.

Minx is intended to let you have full control of post and comment selection from the template, overriding the default settings, but those tags are more complex to process and don't work yet.

Finally, we have magic tags. We have two examples of this, the pager and the stats. Magic tags do magic stuff that isn't necessarily available using regular template substitution. The pager lets you go to the next/previous page of the blog (a fancier pager is coming); the stats show performance information (and a fancier version of that is coming too).

The central idea is to make as much of the feature set as possible available without making the templates scary. So you don't need to have a complex nested structure of post tags and comment tags. Just put [posts.here] in your page template (with maybe some optional parameters) and [comments:here] in your post template. Set the appropriate settings on your blog options screen and you're in business!

Next up: A user interface...

Update: Minx now pulls its templates dynamically from the MT database. If you create index templates called mxPage, mxPost, and mxComment, Minx will use them to override the default templates. (Which are loaded dynamically as well.)

You can also include templates using the [include template] tag. The template to be included must also begin with "mx", since Minx only looks at those, but you don't actually specify the "mx" when you use the tag. So, for example, you can create a blogroll template called mxBlogroll and include it with the tag [include blogroll]. (It's not case-sensitive.)

Update: Conditional processing with if and ifn. if and ifn can be used with any simple tag to test whether the value of that tag is "true" or not. A tag is true if it is a non-zero number or a non-empty string, otherwise it is false. Any text between [if tag] and [/if] is included if the value of the tag is true. And any text between [ifn tag] and [/ifn] is included only if the value of the tag is false.

You can't nest if tags, though you can nest an ifn inside an if and vice-versa. If you want to get more complicated than that, for now you'll need to use sub-templates with include.

* Or rather, symbiotic.
** And, um, parameters aren't working yet.

Posted by: Pixy Misa at 01:45 PM | Comments (10) | Add Comment | Trackbacks (Suck)

Geek

Whee!

It works!

Rather than writing everything from scratch to start with, I put together what I had and pointed it at our existing Movable Type database (which has a number of changes over the original, admittedly). And it works.

40 milliseconds is a bit depressing, though. I'll have to see if I can speed that up a bit.

Okay, some timings: Last 200 entries at Munuviana, with inline comments: 420ms. Without inline comments: 90ms. Having inline comments causes a lot of extra SQL queries; I don't know if it's that or the text processing that's taking the time.

A normal page only shows 20 entries, and takes around 45ms with comments and 15ms without. It's a bit hard to time that more precisely, which is why I bumped the page size up.

Every post and comment has to be passed through the template engine; every comment is also processed through a SGML parser to strip out unwanted or invalid HTML tags. I'd love to get a complete page out in 10ms, but I don't think I can. Well, if it's cached, then sure, but not if the whole thing needs to be dynamically generated.

<Pause.>

<Goes off to implement caching system.>

...

<Comes back again.>

Retrieved from cache, processing time 0.0 seconds.

Yeah, that'll do.

Only problem is that saving the text up to be cached takes an extra 10ms or so. I need to try the StringIO/cStringIO functions and see if they help.

Update: cStringIO provides no performance advantage at all. Which is good in that I know that the easy way to code it is just as efficient, but bad in that I can't speed it up at all.

Posted by: Pixy Misa at 05:27 AM | Comments (4) | Add Comment | Trackbacks (Suck)

June 11, 2006

Geek

Dum De Dum

I've been working on the blog-code, all the live-long day night...

It's good to be doing development again, rather than running around like a hamster on crack trying to shore up systems under attack by hackers, spammers, and just plain lousy other-people's-code.

What I've mostly been doing so far, though, is installing stuff. Python needed an update, as did Apache. Didn't have PIL installed, or Numpy or Pyrex. Pysco has been patched. Memcached and its prerequisites. Bumped MySQL up to 5.0.22, and PostgreSQL up to 8.1.4. (I may end up just using MySQL, but at this stage I'm keeping my options open.) CherryPy and Django and ReportLab. And PHP 5.1.4... (fixeds MySQL option) And PHP 5.1.4 (fixes MySQL option correctly) And PHP 5.1.4 (what do you mean, you can't find zlib? How should I know where it is?)

Okay never mind PHP 5.1.4. I'll do that later.

To work!

Oh, yes: I'm doing all of this on my notebook. Hooray for VMWare Player!

Posted by: Pixy Misa at 11:01 AM | Comments (4) | Add Comment | Trackbacks (Suck)

June 10, 2006

Anime

Duck And Cover

Theme harems? Okie dokie:


1. Urd, from Oh My Goddess
2. Lum, from Urusei Yatsura
3. Yuri, from Dirty Pair (original version only!)
4. Faye Valentine, from Cowboy Bebop
5. Lina Inverse, from Slayers
6. Tira Misu, from Sorceror Hunters

Maid: Honey Kisaragi, from Cutey Honey (She has a maid outfit, so she qualifies!)

Life will never be dull. Brief, possibly, but never dull. more...

Posted by: Pixy Misa at 11:53 PM | Comments (14) | Add Comment | Trackbacks (Suck)

Anime

Subtitled For The Thinking Impaired

Can anyone point me to a subtitling program that doesn't screw up my files? Subtitle Workshop and DivXLand need not apply. Something that actually works without deleting text or formatting or both would be nice.

Posted by: Pixy Misa at 10:57 PM | Comments (10) | Add Comment | Trackbacks (Suck)

Life

Karma: Piss Poor

Did I torment kitties in a past life or something?

Anyway, my new HDTV has not arrived. Yes, they called me yesterday to tell me I could come and pick it up. No, it's not actually there, as such.

Eh.

Okay, off to set up Apache 2.2 and stuffs.

Update: The other advantage of going for the borrowed word is that I could understand it as well. Context helps, of course.

Posted by: Pixy Misa at 10:57 AM | Comments (7) | Add Comment | Trackbacks (Suck)

June 09, 2006

Geek

You Think You've Got Problems

Not only do I have a denial-of-service attack to worry about, I've got 3000 incoming trackbacks per minute.

And that's after I firewalled off the worst offenders. Don't know what it was before, because it made Apache seize up.

And my notebook, which is where I keep, well, pretty much everything, BSOD'd on me earlier, and is now giving me random Unknown Hard Errors.

Yes, I know. BACKUP NOW. What do you think I'm doing?

Update: Whoops, there it goes again. Okay, time for rsync.

Posted by: Pixy Misa at 10:21 PM | Comments (3) | Add Comment | Trackbacks (Suck)

Life

Blue Tuesday Friday

My baby done left me.
My cat left me too.
Can't get this code to compile
No matter what the hell I do.
I got the Blue Screen of Death Blues.

My server got DDoSed.
My hard disk got fried.
Got everything fixed, then
Movable Type lay down and died.
I got the Blue Screen of Death Blues.
Yeah, I got the Blue Screen of Death Blues.
Should have stayed home in bed -
Now I got the Blue Screen of Death Blues.

Posted by: Pixy Misa at 09:05 PM | Comments (2) | Add Comment | Trackbacks (Suck)

Geek

Trackbacks Are Dead

We are running a very non-standard trackback system here at munu. The standard trackback script is disabled, and our custom one simply logs the trackback request in a text file, taking a small fraction of a second. Another process comes along once a minute, scoops up the log file, filters out the crud, and posts whatever remains. But that happens entirely in the background, and since 99.8% of trackbacks are spam, and it can detect and reject a spam trackback in 50 microseconds, the processing is very, very efficient.

Nevertheless, we are getting enough trackbacks right now to tie up fifty Apache processes. That's over half a gigabyte of memory dedicated to returning 404's to spammers.

Posted by: Pixy Misa at 04:43 PM | Comments (3) | Add Comment | Trackbacks (Suck)

Rant

Overhead, Without Any Fuss...

I was feeling good this morning.

The chaos caused by the DDoS attack was largely over.

I have an interesting new project at work that will have spillover benefits for munu.

My new HDTV has arrived.

So has my copy of Rumble Roses XX.

I'd even had seven hours sleep, which is a record for the month so far.

The only problem left to fix was trackbacks, and that wasn't a huge priority. Trackbacks are nice to have, but 99.8% of them are spam anyway.

Oh yes, and the Zarkster had croaked.

So I settled in at my desk at work -

And then I find, over at munu, without any fuss, the blogs were going out. Every time someone got a comment, the main page of their blog would go blank.

I have absolutely no idea why this was happening.

All I know is that it didn't affect Ace.

Ace was running his own copy of Movable Type. An identical copy (in theory) but still a copy. Using the same database, but still a separate copy of the source code.

It took me a two hours of screaming frustration to find this out, but then I switched everyone over to Ace's copy of MT. And then whacked the server over the head repeatedly. And then it worked.

Posted by: Pixy Misa at 03:07 PM | Comments (4) | Add Comment | Trackbacks (Suck)

Geek

Fiddlicreepi

When tasked with building a huge and complex database application, it is valuable to have already spent half your life doing exactly that. Because then, when faced with a seemingly intractable problem, you can simply cast your mind back to how you solved it last time.

Having said that, multi-master replication still poses problems. Having said that, we're not running a bank here. We can say the order of transactions is not guaranteed. The detail lines are in a different order in Japan as compared to the Netherlands? Doesn't matter. As long as they're all present and correct, and the ordering isn't too badly screwed up (minutes matter; seconds don't), we can get away with it. It's a bit annoying that we need an extra field (the original server number) to guarantee uniqueness on some tables, but that's life.

And for the tables that need to be centrally controlled, well, we centrally control those ones. Makes up 0.01% of transactions and 0.0001% of database operations. No biggie.

Look for it on a website near you, probably around September. I can't divulge the details just yet, but don't worry, you'll know it when you see it.

Posted by: Pixy Misa at 08:30 AM | Comments (4) | Add Comment | Trackbacks (Suck)

June 08, 2006

Geek

Um...

From the O'Reilly book, Ajax Hacks:

For example, if you have ever used Google Maps, the way you can drag outlying regions into your view conveys the impression that you have all of the maps stored locally on your computer, for your effortless manipulation. Imagine how unpopular this application would be if every time you tried to "drag" the map the page disappeared for a few (long) moments while the browser waited for another server response.
Imagine living somewhere other than the United States.

Actually, the screen doesn't go blank; instead you see the wrong map for a while as it downloads the tiles, blip... blip... blip... blip... blip...

The application would be so sluggish that no one would use it.
Yeah.

Posted by: Pixy Misa at 05:55 PM | Comments (3) | Add Comment | Trackbacks (Suck)

June 06, 2006

Geek

Right

50 milliseconds to start Python and load the cgi, cgitb, MySQLdb, os, psyco, sgmllib, string, sys, and time libraries. (It's not currently using psyco, because it has no benefit for such a short program, but I left it in.)

50 milliseconds to connect to MySQL. (CPU time. Elapsed time is roughly the same, I think.)

7 milliseconds (elapsed) to return the 50 most recent matches from Ace of Spades for the word "bush".

10 milliseconds to process the results.

I'm going to set up a miniminx to get rid of 1 & 2. Just by way of experiment.

Posted by: Pixy Misa at 03:27 PM | Comments (8) | Add Comment | Trackbacks (Suck)

Life

Taking Stuff For Granted

Ten hours ago I was intently focused on trimming my search routine down to single-digit milliseconds.

Right now I'm just happy to have a working server.

The DDoS attack came back.

Fortunately, this time I had working NFS and no drive failures. Much faster and less painful that way.

Posted by: Pixy Misa at 04:02 AM | Comments (2) | Add Comment | Trackbacks (Suck)

June 05, 2006

Geek

Huh?

I was forced to kill the MT 2.6 search routine at Munu because it was (a) taking 180MB of memory, (b) taking a couple of minutes, and (c) because of (b) people were clicking on it multiple times until it, essentially, killed the server.

I just rewrote it.

I tested it on Ace's blog, by searching for "Bush". It's currently set to only search the last 500 entries, but for my test I set it to scan the last 5000.

It takes one second. And 19MB of memory. About 350ms for the SQL query and 650ms for the program itself.

It sure ain't optimised. It selects the last 5000 entries, sorts them (because God forbid there should be a useful compound index), yanks the entire result set into a list, scans them for each of the search terms, uses an SGML parser to remove HTML tags, and kicks the result out.

For more reasonable searches, like searching for "test" on the last 500 entries at Munuviana, it takes 35ms for the query and 130ms for the program.

About 120ms of that is start-up: Launching the Python interpreter and loading the seven or eight libraries involved.

Now, I'm not using a template system for this. Still, one-tenth the memory, one hundredth(?) the processing time. I can't be sure about the processing difference, because I can't run the original script right now. I configured Apache with a 100MB memory limit for CGI scripts.

For Minx, the 120ms start-up time wil disappear because the application runs as a multi-threaded server itself, not as individual CGI (or PHP or ASP) scripts. Can only do so much about the query time, but I'll play around with it. And I'll pre-store the excerpts rather than create them on the fly. Well, probably. I might be able to live with an average search time of 45ms.

Hmm.

What if I get MySQL to do the matching? Let's see...

Okay, not good. Hmm.

Ah, there we go. Don't use regexp's unless you need them. "LIKE" is nice and brisk. With a 500-result limit, the Ace/Bush search is 125ms for the SQL query and 300ms for the program. And doing it that way, I could actually page it, so 50 results at a time. Hmm. And when I'm taking the 50-word excerpts, rather than whitespace-split the entire entry, I'll just look at the first 400 bytes.

And let's go back and add that compound index while we're at it...

Okay, now we're cooking. 7ms for the query, 100ms for the search script. Since the resolution of the timer seems to be 10ms, and the search script takes 100ms if you feed it an invalid blog id, that's less than 10ms or so for the actual work.

That'll do.

Posted by: Pixy Misa at 05:47 PM | Comments (3) | Add Comment | Trackbacks (Suck)

Rant

NFS Is A Cow

We hates it!

Okay, the problem(s) this time:

If portsentry is running, you cannot start NFS. Solution: Stop portsentry, start portmap and nfsd, restart portsentry.

If sunrpc is not available as a kernel module, rpc.idmapd will not start. Without rpc.idmapd, you can't use NFSv4. Solution: Don't use NFSv4.

Everything is set up correctly, your mount requests are being accepted according to the syslog, and yet the client always sees "Permission denied". Solution: Add the line

none /proc/fs/nfs nfsd noauto,defaults 0 0
to /etc/fstab and mount /proc/fs/nfs You can take the noauto off once you're happy that everything works.

This, of course, is all explained in great detail in the man pages... err, the online howto's... err, in the kernel mailing list.

Posted by: Pixy Misa at 02:06 PM | Comments (6) | Add Comment | Trackbacks (Suck)

June 03, 2006

Rant

Plus Ça Change

One server taken out by a DoS attack, the other by a disk failure.

BUT I STILL GET SPAMMED.

Posted by: Pixy Misa at 04:04 PM | Comments (7) | Add Comment | Trackbacks (Suck)

Rant

The Best Laid Plans

My plans for the weekend involved reading the new Harry Dresden book, having a big bowl of teriyaki beef don, playing with a kitten, and maybe prototyping some new blogging stuff.

They did not include migrating the whole of munu from one server to another to dodge a DoS attack, much less recovering from a hard disk failure during that migration.

So guess what I actually am doing.

Posted by: Pixy Misa at 12:39 PM | Comments (13) | Add Comment | Trackbacks (Suck)

May 30, 2006

Anime

Bulletin Bulletin Bulletin

Future files - and eventually, past files as well - will appear here.

Posted by: Pixy Misa at 10:26 PM | Comments (6) | Add Comment | Trackbacks (Suck)

<< Page 12 >>

Processing 0.03, elapsed 0.1028 seconds.
49 queries taking 0.0846 seconds, 119 records returned.
Page size 106 kb.
Powered by Minx 0.8 beta.