Power-loss post-mortem
The post you've all been waiting for! Why we lost all our power, and why it took us so long to come back up afterwards....
(Warning: it's late and I'm tired/rambly, so this post might be incoherent... go ahead and ask questions...)
Why we lost all our power...
Another customer in the facility accidentally pressed the EPO button, then depressed it, replaced the protective case, and left the building. Intenap all thought it was their UPS systems failing, but then logged into them, saw EPO shutdown notifications, and couldn't find any EPO cases open or pressed, so probably freaked out for a bit thinking there was a short in the walls that triggered the EPO, only to get a confession a day or so later.
EPO, by the way, stands for Emergency Power Off and it's a national fire/electrical requirement for firefighters to be able to press these big red buttons near all exits that turn off all power in the entire data center. This is the second time this has happened to us in the years I've been there. The first time the button was unlabeled and unprotected and some dude thought it opened the door. This time we have no clue why it was pressed... maybe that dude tripped and fell onto it... mystery.
Internap will be putting alarms and tamper-proof indicators on the plastic cages surrounding the EPO buttons now, though, so at least if this happens again in the future they'll know why.
Anyway, moving on....
Why it took us so long to come back up...
Ton of reasons:
Faulty mobos/NICs: We have 9 machines with faulty motherboards with embedded NICs that don't do auto-negotiation properly. They only work with certain switches, so they reboot fine, but then their gigabit network comes up at 100 half duplex or something that doesn't work. To get them back up they need somebody at the NOC to plug them into a compatible switch, let them autonego, then switch them to their real switch. Setting the speed/duplex settings on both the host and/or switch themselves doesn't work.... most annoying. We're getting working dual-port gigabit PCI NICs for those machines, rather than replacing their motherboards.
Database start-up: All but a couple of our machines came back up when the power was restored, less than an hour later, but on our databases we intentionally don't have the database start back up on boot. In normal circumstances, if a single machine dies, it died for a reason and we want to investigate it. Normally that doesn't matter either, because we have 2+ of everything. But when every single database restarts, that leaves us with no alive databases, and we have to manually start them all.
Data validation: We could've just blindly started all the databases and trusted they worked, but we didn't trust them. We ran innodb tablespace checksum checks on everything, and also did backups of a lot of the databases before we even tried to bring them back up. (the act of bringing them back up modifies the tablespace, and we didn't want them messing themselves up worse, so a pre-backup was for paranoia...)
MyISAM vs InnoDB: When you lose power to a MySQL db w/ MyISAM tables, the indexes are generally messed and you need to rebuild. Fortunately almost all our databases are purely InnoDB nowadays, so that wasn't a huge problem. Unfortunately, though, the global DB (which is required even to get the site up in partial mode where some users are available and others aren't) is still like 5% myisam and we just hadn't got around to converting those few remaining tables to innodb yet. So every machine in the global cluster required index rebuilds and data checks. That was annoying. The Chef user cluster was also MyISAM (our last MyISAM user cluster), so rather than trust Chef, we restored from an old Chef backup and replayed binlogs to catch it up. That took some time.
Disk cache issues: We have battery-backed RAID cards with write-back caches. That means the RAID card immediately acknowledges writes and tells the OS (and thus DB) that they're done immediately, before they're on disk. This speeds up the DB. But if you lose power, those writes would normally be lost, which is why we have battery-backups on all the cards, and we even monitor the battery health w/ our automated checks. But unknown to us, the raid cards didn't disable the write caching on the drives themselves.... which is frickin' useless! If the controller is already lying to the OS (but doing it safely!) why should the disks behind the controller also lie, but unsafely, for minimal benefit? Our bad there. We should've had that right. So a couple machines were just gibberish afterwards and had to be restored from backup and had their binlogs replayed to catch the backups up to present.
Binlog syncing: We weren't using the option to sync binlogs to disk, so we lost a small number of transactions right before the power loss in the case of clusters that we had to restore from backup. Regrettably, we won't be able to get those posts or comments back.
Slaves tuned for speed: A lot of slave servers (mostly in the global cluster, since the user clusters are almost all master-master now) were tuned for speed, with unsafe filesystem/sync options that favored speed over reliability. Which is normally okay, since you'd lose one machine, not all of them. But we lost all of them, so restoring them all from good slaves was time-consuming.
Things we're doing to avoid this crap in the future...
We're:
-- getting working NICs in those 9 machines
-- all our DBs have redundant power supplies. we'll be plugging one side into Internap's, and the other side into our own UPS, which itself is plugged into Internap's other power grid. that way if EPO is pressed, we'll have 1-4 minutes to do a clean shutdown. (but if we do the rest of the stuff right, this step isn't even required, including having UPSes... in theory... but the UPSes would be comforting)
-- disable disk caching behind all our RAIDs. (bleh... wanna kick ourselves for this, but also the raid vendors for even defaulting and/or allowing it in a BBU write-back setup) but also testing all existing and new hardware to make sure data makes it to disk and pulling power in the middle of write-heavy operations, then verifying the resulting disk image later with the expected result.
-- finish our MyISAM to InnoDB migration, so we don't have to deal with MyISAM index rebuilds
-- enabling binlog sync options
-- stop tuning slaves for speed. this used to matter, but we don't really do the slave thing as much as we used to, so the gain isn't worth it.
-- user level backup harness. we already have a tool to backup a single user's to a GDBM file, incrementally. (so if we run it a day later on the same file, it only updates the changes) so we plan to wrap a major harness around that tool and be backup up all users, all the time. this means that in the event of a future major corruption/outage, we'll be able to restore user-at-a-time, and even to a different database cluster than the one they came from. this also means we can prioritize recovery based on account status, popularity, attempted activity, etc. (and yes, we'll continue doing system-level backups as well, but it's good to have data in different formats just to be paranoid...)
-- also, we already bought a bunch more disk space that we installed today, so we have more room to do backups for bizarre non-typical reasons, and don't need to compress/shuffle so much stuff to make room sometimes.
On Monday, at 10.14 pm UTC, the Wikimedia server cluster experienced a total power failure, taking down Wikipedia, Wikinews, and all other Wikimedia websites.

The servers are housed in a colocation facility (colo) in  Tampa, Florida, USA. They occupy two racks, with each rack receiving electricity from two independent supplies. However, both supplies have circuit-breakers in them, and both opened at the same time, leading to a total power failure. All computers immediately went down. It's normal for fire safety regulations to prohibit uninterruptible power supplies in colos, with the colo providing its own UPS and generator instead. The circuit breakers were on the computer side of this emergency power system, so none of the computers continued to receive power to survive the breaker trip or shut down safely.

The actual reason why the circuit breakers tripped is currently unknown.

When power was restored, it was discovered that most of the MySQL databases that store the data which makes up Wikipedia et al had been corrupted. The main database and the four slaves had all damaged the data on their hard disk drives beyond the ability of the auto-correction to repair. Only one copy survived safely, on a machine that is used for report generation and maintenance tasks, which remained 31 hours backlogged while catching up after an unusually heavy update load during the previous week. 

Volunteer Wikimedia engineers worked through the night rebuilding the databases from the sole good copy onto the other servers. The Wikipedia database is over 180Gb in size, making the copying process last 1.5 - 2 hours for every server it was performed upon.

Regular back-ups of the database of Wikipedia projects are maintained - the encyclopedia in its entirety was ''not'' at risk. The last database download was made on February 9; all edits since then could only have been laboriously rebuilt from logs and recovered from the damaged database requiring much more time and effort.

Limited read-only service was established late Tuesday afternoon, with editing becoming possible 24 hours after the power failure. Final repairs continue now, as well as upgrades to prevent similar issues in the future. Server-intensive features, such as categories and 'watchlists' that display recent changes to selected articles to registered users, remain disabled to ease the load on the recovering systems.

The process which led to the damage originated with the operating system, disk controllers, or hard drives failing to flush the data correctly. 

If the power to a database server is cut mid-write, the database may be corrupted and unreadable, however the operating system, hardware, and software are designed to make this very unlikely. In a previous incident in 2004 power was also lost to a server but the database was undamaged.

To avoid such damage, each database server saves a copy of an edit to be applied to the database on a separate storage system before making the actual update to the database itself. This so-called 'write-ahead logging' should ensure that in the event of a system crash, the database can be rebuilt from a 'last-good' state by replaying the edits saved in the log. 

Earlier this year popular blogging site LiveJournal suffered a similar power failure when another customer at their colocation facility pressed an Emergency Power Off button, intended for use only by firefighters. The company suffered database corruption similar to that seen at Wikimedia. 

LiveJournal are now fitting UPS to their servers to ensure that they have time to shut down safely in the event of a power failure. Wikimedia was said to be investigating the possibility of fitting similar equipment at the time of this failure.

Several pundits have suggested that the use of another database, such as the proprietary database Oracle or the free PostgreSQL, would have avoided the database corruption seen at the server cluster. A post-mortem of the incident show the failure was in the operating system, or the hardware, or some combination of the two. LiveJournal, which also uses MySQL, reported similar database corruption  after their power cut.

The Wikimedia foundation only allows the use of free software on its systems, and future versions of the Mediawiki software will support the PostgreSQL database.

Users are reminded that during times of system failure or excessive demand, they can still search Wikipedia  using Google. The articles may be viewed using Google's cache.