[ahimsa-tech] ahimsa-db1 problems RESOLVED

[jeff moe]

ahimsa-db1 was acting really slow and some people from italy popped in to complain.

I tried to ssh in, but could never get a shell. I was able to log in via the 
console and get a shell, but a `w` hung forever until I killed it. 
ctrl-alt-del never did anything. I tried a `/etc/init.d/mysql stop` and let 
that sit for 30 minutes, but it never shut down. I had also shut down the 
apaches on web3 and web4 in the hopes that it would give some breathing room to 
db1, but it didn't help.

Today was the day it does it's full db backup, and it did not complete.

I decided finally to power cycle it, which scares me. The journal `fsck`s all 
came up fine (and I later did them manually too).

I did a mysqlcheck of each database, and they all came up OK.

It was running kernel 2.6.17.8. I installed linux-2.6.16.29 on there since the 
2.6.16 series seems to be a bit less dodgy to me. I also switched the scheduler 
to deadline, which supposedly is better for databases.

I decided to do a full backup before bringing the webservers back up, just in case.

I also did a `apt-get upgrade`.  This thing is running munin/httpd, so 
potentially, that was part of the problem.

So, in sum, some dynamic ahimsa* websites were slow today and completely down 
for a couple hours. I brought it up real slow cuz a corrupted database would 
really suck. :)

-Jeff