Panic, Chaos, Doom
Posted on July 21, 2007
12 Comments
Let me try to lay out this series of events that has culminated in way too much time at the office this weekend.
It all began…
Six months ago I converted a development application server (aka “the vault”) to a VMware virtual machine hosted on our dev server. Worked great that way and reduced the heat load in our server room (and the dependency on some really old hardware).
Still harmless…
Wednesday night, the dev server said it lost one of the 4 drives in a RAID 5 array. Not a major issue as I knew it was yet another loosened SATA cable and planned to fix it this weekend.
I purchased MozyPro online backup a few weeks ago and began setting up backups on the office servers. Since I’m moving multiple gigs of data over the internet (initial backups of around 20GB) I’ve been doing one at a time. They take a few days since I throttle the backup speed way low during business hours. The dev server said it finished the backup Friday morning.
MozyPro can’t handle open files when run on Windows 2000 (our dev server is win2k server), so I planned on powering down the virtual servers this weekend to get a backup of them. I was already coming in to do the cable swap, so no big deal.
Then disaster struck.
We had an odd (and unexplained) power issue Friday afternoon in the server room. Must be an issue with my UPS as all machines went down hard. Upon coming back up, the dev server magically discovered that previously “lost” drive and began to rebuild the RAID 5 array.
At some point during the rebuild, another drive went missing. Now 2 of 4 drives are gone and we’re sunk. Busted array. The sky begins to fall.
I spent an hour on the phone with Adaptec’s emergency/paid support. We recovered/rebuilt the array (several times) but couldn’t get to any usable data. Couldn’t seem to get to the right mix of drives.
Those VMware virtual servers are definitely gone. Time to swallow hard and move on with the recovery plan. (and choke back a sob).
I logged into MozyPro to get a restore going. I see 21.9GB of data, but one of the directories I had specifically setup for backups is not listed. A very, very important directory for the company. (Monday’s post may be a current resume!)
I called MozyPro support, but the after hours guys can’t see much beyond what I can see. I’ll have to try again during normal business hours to find out if that missing directory, that was allegedly backed up, can be found. In the meantime, I’ve started the restore of what’s listed.
SQL Server 2005 is a bit confused — it knows about a bunch of databases but those databases no longer have data files (they were on the now defunct RAID array). Tomorrow I need to delete and rebuild all those with the Thursday night off-site backups. Any little tweaks or special permissions will be lost. All of that was documented in our wiki…. but it’s gone too (backing it up had fallen completely through the cracks).
This is why IT guys might have a stiff cocktail in the evening.
Tags: backup, mozy, operations, RAID, server
Possibly Related Posts
Comments
12 Responses to “Panic, Chaos, Doom”
Leave a Reply
Doh! Goes to show that disaster recovery in the real world is never quite as it’s designed. We had our company purchased by another, and the new owners were instructed that there was only a DLT backup of the servers, plus a hot site where all the transactions were mirrored. In their infinite money-saving wisdom, they deemed the DLT as unnecessary, since there was a hot site. However, they didn’t purchase the hot site, and failed to set up a new one until AFTER the melt-down of the primary server… OUCH. Yes, the days following that fiasco were built around a strict regimen of caffeine in the a.m., and cocktails in the p.m.
Good luck with your recovery!
Hey, just a little word of advice on the whole “backup my VMs with MOzy” thing. I would suggest that you not try to do that unless your VM is amazingly small. Mozy’s a data backup tool, not a disk imaging program, and you will have a hell of a time getting 10-20gig files backed up, let alone restored.
or, if you’re hell-bent for glory down this path anyway, perhaps you might make a multi-section zip out of the VM, with more reasonable chunks, and then back THAT up. that way, you can just shut down the VM, rip it to a multi-part zip, and go back to using the VM while mozy backs up the pieces. just a suggestion.
in the meantime, enjoy all those beverages, morning and night!
Bummer dude.
Hope all goes well with your data recovery. Sometimes the way problems stack up in the IT world can be extremely frustrating. When it rains it pours. Just as you are finally implementing a good backup strategy, one of your servers goes down. best of luck.
@Stephen - Ouch indeed! Great story and pretty typical I’d wager.
@Frankie - great points. In hindsight, I need to think more about handling the VMs. The multi-file zip approach makes a lot of sense for complete recovery. Maybe I should’ve also had the mozypro client installed on the VMs as well…
@Nathan - bummer indeed! The timing is just terrible.
oh, one more piece of irony. The virtual server I lost? After converting it from physical to virtual, I kept the box laying around, “just in case.”.
That’s the box I used for my linux bandwidth monitoring project tuesday or wednesday of this week…
Hi Chris,
Good posting.
I have been reading about the online backup industry for a while now.
Online backup is maturing and slowly getting the attention of the general consumer.
One website worth mentioning is the backup review site:
http://www.BackupReview.info
This very informative site, not only posts up to date news and articles from the industry, but also lists about 400 online backup companies and ranks the top 25 on a monthly basis.
The site now covers storage in general, not just online backup.
I enjoy reading your posts. Keep it up!
Chris, How about a 30-day free trial with a REAL business backup company? I’ll arrange if for you. http://www.dataprotection.com I tell people all the time that “free” and “low-cost” services are too expensive for real businesses that are serious about protecting their data. Data Protection Services has been offering online backup for business clients since 1996, and we’ve never had a client that was not able to restore his data!
Hi Susan and Jennifer, thanks for the links and stopping by.