As promised, the following is a summary of what led up to the crash that kept us down for almost four days last week.

About two weeks ago, I started to notice problems with the What The Christ? server. It was mostly on the administrative side of things at first. After a few days, however, we got a string of random reports from visitors who couldn’t connect. It took a day or two before we realized that the problem was an “Error establishing database connection.” message.

That’s the error that WordPress throws when your server can’t establish a connection to your MySQL database – the point from which content is delivered to the user. It’s usually caused by one of two things: a problem with your WordPress configuration file or a (much bigger) issue on the MySQL side of things.

Configuration issues are easy to fix. Unfortunately, we weren’t facing a configuration issue. No, we were facing a much bigger problem. In fact, Mr. Blue said it best.

“Dude, the database is fucking hosed.”

Don’t ask me how it happened. It could have been our migration to WordPress 3.0.1 a few weeks back. It could have been a plug-in that we were testing. It could have been one of a handful of configuration changes that we made at the host level. Whatever it was, we were looking at a really ugly problem. Our databases are not small, nor had we done as much of the maintenance on them as we should have been doing all along. To be honest, we’d been neglecting them because we were way too busy updating and promoting the site.

Needless to say, we were faced with a tough decision. We could either bring down the site, which would have been bad, or we could have tried to troubleshoot the problem live, which would have (potentially) led to downtime anyway. Obviously, we chose to troubleshoot the problem live. About an hour into the process, the server took a giant shit and dumped our progress. (Hooray, ambiguous Plesk error!) Down we went – and down we stayed. I could detail the laundry list of problems we had trying to bring things back online over the two days that followed but I’d rather not. It was a nightmare.

Unfortunately, that was only the start of our troubles.

Initially, the crash didn’t seem like a big deal. I thought we’d be down for a day at most. If there’s one thing we’ve done “right” from day one, it’s our backup routine. We had a dump of even the most recent posts – both the MySQL data and the associated assets. I thought we were good to go. In fact, it seemed like we were good to go after a couple of days sitting around trying to build a better mousetrap. We eventually decided that the best plan was to wipe the server. I reinstalled everything from scratch and started the process of uploading assets to the server. Meanwhile, Blue opened a case of beer and a book on MySQL.

But then something happened. Something bad.

Early Saturday, the server simply decided to die. Period. Hardware failure.

I’m pretty sure anyone within a six block radius of my house heard me scream, “FUCK!” when it happened. Everything shut down. No SSH. No FTP. A moment later, I was on the phone with our host. Thankfully, they understood my frustration and didn’t hold it against me. Instead, they got right to work trying to figure out what the hell had happened.

Ten hours later, the problem was solved. (In case you’re wondering, we didn’t move things to an interim server in the meantime because we figured that our server would be back up and running before we were even able to get everything uploaded on the temporary box.)

From there, the rebuild was simple… but tedious. I managed to tie up four Comcast lines, three of which belong to friends, for close to nine hours while we uploaded assets and ran a much needed database optimization routine. We also cleaned up the SQL configuration and learned a thing or two about responsible administration. (Thank you, mysqltuner.pl.)

So, there you have it. That’s how it went down – pardon the pun.