Back to the Future

There was a catastrophic hard drive failure on the central database server very early this morning, Donncha was on the ball and switched all the traffic to a recent backup so most things would work while we investigated the hardware failure. This means that an old version of your site was shown for a few hours.

A few minutes ago we restored the up-to-date database and we’re currently syncing it to the backup to get back any posts you might have made during the semi-downtime. Even though we were able to recover everything, we’re looking at ways to make things even more redundant, so if this ever happens again the problems will be measure in seconds or minutes. Like I said in the last post, all of the new hardware hasn’t been brought fully online yet.

Technical details: The ext3 filesystem was badly corrupted but thanks to journalling there was no lost data.

Update: All posts have now been restored.

Published by Matt

In 2002 I started contributing to Open Source software, and life has just gotten better from there. Co-founder of WordPress, founder Automattic.

46 thoughts on “Back to the Future

  1. Thanks for all of your hard work this morning Matt and Donncha. Several hours worth of work over the weekend lost induced a mild state of panic this a.m. but all appears to be back to normal. Thanks again.


  2. Matt, a big thanks to you for restoring back the posts…..and a big hug from me… post were written with a lot of time(though may seem average to some)


  3. You guys did a great job, and even took the time out to answer a few of my emails with some details. I appreciate that level of customer service, it came as a complete surprise on a free service like this.

    You have my loyalty.


  4. Yeah, well looks like something is still wrong elsewhere. I had registered for the account ryanl earlier, written a bit and now it redirects to ben_b. After this happened I was able to register the same user account again but it still takes me to the alternate control panel and doesn’t allow me to have any options whatsoever other than the dashboard.


  5. Pingback: roam,maol
  6. Your work is appreciated.
    On the other hand, I was already thinking “major re-writes” and stuff… and then it all came back… arrrfff.
    Your work, as I said, *is* appreciated. Thanks all.


  7. Gaby, your post was made onto the backup hardware, and we just restored all of those. Your link works now and your post is back safe and sound. πŸ™‚


  8. no, mine is combined with
    my dashboard is frozen solid
    I can see two of my posts.
    my old password works
    and computers are wonderful
    and computers are not as nice as you guys.
    no hurry, I’ll wait a while.
    but since I had to recreate
    I want to keep it, thank you. I used 2 invites so far
    to get this blog… worth it, too!
    thank you.


  9. I’m seeing a post which now has somebody else’s name as the author even though the post was made by me, I can’t seem to get to the Feedback button as it disappears (using Firefox on Mac OS X). Any ideas why this has happened?


  10. great job.. last night, about 9 hours ago… some blogs at works but mine ain’t working? why is that so? is it because i used flock? are there any differences with blogs made from invites and the ones made with flock?


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: