[ahimsa-tech] ahimsa-db1 problems RESOLVED
jeff at indymedia.org
Sun Sep 17 15:24:51 PDT 2006
ahimsa-db1 was acting really slow and some people from italy popped in to complain.
I tried to ssh in, but could never get a shell. I was able to log in via the
console and get a shell, but a `w` hung forever until I killed it.
ctrl-alt-del never did anything. I tried a `/etc/init.d/mysql stop` and let
that sit for 30 minutes, but it never shut down. I had also shut down the
apaches on web3 and web4 in the hopes that it would give some breathing room to
db1, but it didn't help.
Today was the day it does it's full db backup, and it did not complete.
I decided finally to power cycle it, which scares me. The journal `fsck`s all
came up fine (and I later did them manually too).
I did a mysqlcheck of each database, and they all came up OK.
It was running kernel 188.8.131.52. I installed linux-184.108.40.206 on there since the
2.6.16 series seems to be a bit less dodgy to me. I also switched the scheduler
to deadline, which supposedly is better for databases.
I decided to do a full backup before bringing the webservers back up, just in case.
I also did a `apt-get upgrade`. This thing is running munin/httpd, so
potentially, that was part of the problem.
So, in sum, some dynamic ahimsa* websites were slow today and completely down
for a couple hours. I brought it up real slow cuz a corrupted database would
really suck. :)
More information about the ahimsa-tech