One of our primary databases has been offline for weeks thanks to some hardware problems. Thankfully we've been able to run without it, but the existing hardware was beginning to reach its limits, like driving a car on three wheels. The good news is: it's back. And over the next month we have a lot more hardware coming along spread across the country.
You guys did over a half a million pageviews pageviews yesterday, not counting any admin activity. Whew!
As you may have noticed, photomatt.net has been down along with mullenweg.com, which means that I can’t blog and I’m not getting any email. It’s very rough! There seems to be a hardware problem with the server, so hopefully the people at the DC can get it sorted soon enough and I can get back to falling behind on my email.
In the meantime, if you send me a message it may be a bit before I get it.
There was a funky bug where some servers (including my own mullenweg.com) weren’t getting mail from WordPress.com. Detective Donncha was on the case and tracked it down and all that mail should start going through now.
There was a catastrophic hard drive failure on the central database server very early this morning, Donncha was on the ball and switched all the traffic to a recent backup so most things would work while we investigated the hardware failure. This means that an old version of your site was shown for a few hours.
A few minutes ago we restored the up-to-date database and we’re currently syncing it to the backup to get back any posts you might have made during the semi-downtime. Even though we were able to recover everything, we’re looking at ways to make things even more redundant, so if this ever happens again the problems will be measure in seconds or minutes. Like I said in the last post, all of the new hardware hasn’t been brought fully online yet.
Technical details: The ext3 filesystem was badly corrupted but thanks to journalling there was no lost data.
Update: All posts have now been restored.
For the people who are into that sort of thing:
It’s Labor day so we’ve been working extra hard for you. Here are a few of the latest improvements on WordPress.com:
- Managing links works better.
- You can now use extended entries with the “more” WYSIWYG button.
- You now get less generic profile information on registration.
- Invites are a lot smarter — it prefiles a lot of information for people you invite.
- When you invite someone you’re automatically added to their blogroll on their new blog.
- Feedback form works in IE.
- The cool fade I alluded to before is now in full effect. (Delete some more posts and comments.)
- It should be faster now (and we haven’t even moved to the new servers yet).
All the electricity problems have been solved in the new colo and thanks to the super-human Jason Hoffman everything has been racked and stacked and we’ll begin configuring them this week.
As we boldly go where no blogging service has gone before, it seems we don’t have enough power! As in electricity to physically power all of the new hardware we have. I asked if hamsters on wheels would help things out but they seem to think the power requirements are even beyond that. It’s not an issue, because we’re not one thousandth of one percent to the capacity of our existing servers, but I’m personally looking forward to firing up the new hardware, so I imagine some of you might be anxious too. I will keep everyone updated on this WordPress Power Saga 2005 as it unfolds.