[ahimsa-tech] ahimsa* NFS server OS drive failure

[jeff moe]

jeff moe wrote:
> Press F1 to continue

Bame went to the co-lo and replaced the failed OS drive with a new 80 gig PATA 
drive. Thanks!

I then re-installed debian etch (kvm over IP) on the new drive.

Unfortunately, and to my great surprise, I hadn't backed up the mdadm.conf. I 
really really thought I had (and multiple times at that). `apt-get install 
mdadm` found the RAID-1's but didn't bind them together into the overall RAID-0 
(this is a RAID-10 system). I ran this to get them happy:

mdadm --assemble /dev/md6 /dev/md0 /dev/md1 /dev/md2 /dev/md3 /dev/md4 /dev/md5

For info about how the RAID was originally set up, see this:
http://lists.indymedia.org/pipermail/imc-tech/2006-March/0308-yk.html

I was then able to mount the RAID and then fsck'd it.

I put kernel 2.6.16.29 on there too.

Everything is now ok.   Riseup had some routing problems at the same time due 
to upstream issues, so we got two downtimes for the price of one. :)

Oh, and I guess it's just kind of a warning to pleeeeeeeeeeeeeeeeease backup 
your site all the time. Weekly SQL dumps are made for all sites. If you can't 
find yours, let me know.

Oh.Oh. I found out the site was down by someone in irc saying italy was down. 
Feel free to call me (if you have my phone number) to let me know about these 
issues more quickly. If you want my number (and i have an inkling who you are) 
send me an email and I'll give it to you.

Oh.oh.oh, and on top of all that, the LCD backlight failed on my thinkpad so i 
had to run around town to get a monitor! And then I didn't have a video cable 
for the monitor...and...  sheez. It really comes all at once sometimes. ;)

Let me know if you see any problems.

Thanks to bame, mtoups, and micah for sanity checking the mdadm commands for me. :)

-Jeff