The billing system is down.

The billing system is down because the billing database is down.

The billing database is down because the database server will not restart.

The database server will not restart because it thinks the shared memory is in use.

It thinks the shared memory is in use because it has a bug related to releasing shared memory segments.

It needed to release the shared memory because it crashed.

It crashed because a user exited while holding a microtransaction lock.

The user exited while holding a microtransaction lock because an index key on another, federated database could not be deleted.

The key could not be deleted because the index is corrupt.

The index is corrupt because the other database crashed.*

The other database crashed because it ran out of disk space.

It ran out of disk space because the database directory was full of backup files.

The directory was full of backup files because if the backup filesystem is not mounted, the backup script writes its files into the source directory.**

The backup filesystem was not mounted because both disks in the RAID set had failed.

The disks in the RAID set had failed because they timed out during reads.

They timed out during reads because there's a bug in the firmware on WD200JB disks.

I don't know why there's a bug in the firmware, perhaps we'll die.

* Even so, this shouldn't happen.
** That one is my fault.

Posted by: Pixy Misa at 04:36 PM | Comments (2) | Add Comment | Trackbacks (Suck)