September 27, 2006

Geek

Well, That Was Fun

Had a little excitement at work today. Apparently several blocks of Sydney's CBD went dark, including, of course, our office. Our elderly UPS valiantly struggled along for nearly a minute before expiring; the blackout itself lasted about an hour and a half.

Then I had to fix pretty much everything. Crashed databases, lost routing tables, failed NFS mounts (the systems didn't neatly reboot in the required order), unending fscks (This volume has not been checked for 562 days* so I'm now going to scan every one of your seven million files and there's nothing you can do to stop me so nyah.), broken RAID sets, misconfigured network cards...

Ugh.

Knoppix was used. It would have been even more not fun without Knoppix.

* Actual number.

Posted by: Pixy Misa at 10:13 PM | Comments (32) | Add Comment | Trackbacks (Suck)

1 I take it that no one was able to log out properly during the 1 minute the servers were still up ...

And of course, every temp fix that could not withstand a reboot blew up at once.


Posted by: Kristopher at September 28, 2006 12:52 AM (O5Ju8)

2 Is this why your site had an odd look to it last night? Or was that something else?

Posted by: Wonderduck at September 28, 2006 01:22 AM (6YRS5)

3 If you saw the forum-looking thing, that was a glitch in the Minx caching system.

Posted by: Pixy Misa at September 28, 2006 12:22 PM (PiXy!)

4 That's what it was, yep. "Dear Forum... I'm a student at a small midwestern college, and I never thought..."

Posted by: Wonderduck at September 28, 2006 12:30 PM (6YRS5)

5 Wonderduck, any college student who never thinks might as well just drop out.

Posted by: triticale at September 29, 2006 12:44 PM (IdwjX)

6 Hmmm... 10 points for triticale for ultimate truthiness, minus several million for completely whiffing on the joke.
To be fair, I don't know where in the world Trit is located, so it's certainly possible that he's not familiar with Penthouse magazine...

Posted by: Wonderduck at September 29, 2006 04:37 PM (CJ5+Y)

Hide Comments | Add Comment






Processing 0.01, elapsed 0.0201 seconds.
16 queries taking 0.0163 seconds, 28 records returned.
Page size 9 kb.
Powered by Minx 0.8 beta.