[IMC-Docs] kompost is broken: help required

[Garcon du Monde]

hello all,

as many of you will know, docs.indymedia.org has been out of action
since the beginning of the month. unfortunately, there appear to be some
major problems and we are unclear of the best strategy with which to
deal with them. thus, this is a request for help/advice.

BACKGROUND
==========

the host server is: kompost

kompost runs vservers -
* humus - the docs vserver
* peat - an irc vserver
* a couple of others

humus was temporarily transitioned off kompost to another vserver-host
machine at the end of september 2008 while work was done on kompost.
this is the last complete backup of docs.indymedia.org that we have
available.

CURRENT STATUS
==============

kompost stopped working at approximately 01:40 2009-01-01. it appeared
to have been powercycled and was waiting for human input on the console
to come back on again. it is unclear what caused the powercycle.

kompost was thus switched on again properly at approximately 17:30
2009-01-03. there were no obvious problems noted. there was a further
reboot approximately 30 minutes later due to accidentally knocking the
power supply. again, no problems were noted and kompost rebooted fine.

unfortunately, kompost went down again a few hours later, reason unknown.

on 2009-01-08, someone was able to attend the colo and investigate
further. kompost was rebooted with the idea of updating the previous
"backup" of humus so that docs could be kept live while further
work/investigations were carried out. unfortunately, we were unable to
carry out this rsync due to disc errors/file system corruption.

we tried a number of times to migrate humus - including before or after
fsck'ing the partitions. unfortunately, kompost kept on dying during
fsck and we were unable to make any progress.

kompost has now been removed from the colo and is on an adsl line,
meaning we have easier (physical) access to the machine. there appears
to be major filesystem corruption - for example:

 kompost:/var/lib/vservers/humus/var/www/docs.indymedia.org/pub/Local/ImcTorunPublicznoscPdf#
ls -al
 total 5071944192
 drwxrwxr-x     2 www-data   www-data                  4096 Aug 18  2005 .
 drwxr-xr-x  1100 www-data   www-data                 49152 Dec 16 14:31 ..
 -rw-r--r-- 18469 daemon           3353              424175 Sep 16  1930
p.tar.gz
 -r--r--r-- 26633 daemon          30850 3696216204596641654 Sep 20  1922
p.tar.gz,v
 ?----wx-wt 21084 1774315222 1078679978          1047011697 Feb 19  1915
publicznosc.pdf
 -r--r--r--  3400 daemon          46006              262205 Mar 12  1954
publicznosc.pdf,v
 kompost:/var/lib/vservers/humus/var/www/docs.indymedia.org/pub/Local/ImcTorunPublicznoscPdf#
ls -alh
 total 4.8T
 drwxrwxr-x     2 www-data   www-data   4.0K Aug 18  2005 .
 drwxr-xr-x  1100 www-data   www-data    48K Dec 16 14:31 ..
 -rw-r--r-- 18469 daemon           3353 415K Sep 16  1930 p.tar.gz
 -r--r--r-- 26633 daemon          30850 3.3E Sep 20  1922 p.tar.gz,v
 ?----wx-wt 21084 1774315222 1078679978 999M Feb 19  1915 publicznosc.pdf
 -r--r--r--  3400 daemon          46006 257K Mar 12  1954 publicznosc.pdf,v
 kompost:/var/lib/vservers/humus/var/www/docs.indymedia.org/pub/Local/ImcTorunPublicznoscPdf#

and, on the backup:

 gdm at strummer:~$ sudo ls -lah
/var/lib/vservers/kompost/humus/var/www/docs.indymedia.org/pub/Local/ImcTorunPublicznoscPdf
 total 1.4M
 drwxrwxr-x    2 www-data www-data 4.0K 2005-08-18 14:26 .
 drwxr-xr-x 1081 www-data www-data  40K 2008-09-17 09:54 ..
 -rw-r--r--    1 www-data www-data 415K 2005-08-18 14:25 p.tar.gz
 -r--r--r--    1 www-data www-data 416K 2005-08-18 14:25 p.tar.gz,v
 -rw-r--r--    1 www-data www-data 256K 2005-08-18 14:26 publicznosc.pdf
 -r--r--r--    1 www-data www-data 257K 2005-08-18 14:26 publicznosc.pdf,v
 gdm at strummer:~$

so, we need to develop some kind of plan - over the next couple of days,
we will have pretty good physical access to the box, but unfortunately i
 do not have very much time. this is therefore also a request for other
(docs) sysadmins to step in please.


POTENTIAL PLAN?
===============

one person has suggested that we could work out the diff between the
backup and the data on humus, and then copy the undamaged diffs back.
however, it is not clear how to do this easily as previous attempts to
sync the data have resulted in kompost crashing.

we also need to figure out the cause of the problems - is there a fault
with hard discs, with memory, the PSU or the motherboard? this will need
to be figured out, and decided if we can replace them, or whether the
whole server needs to be replaced.

FUTURE
======

docs can be hosted on strummer temporarily. we should also try to find
some money for the people who have been hosting the server as i don't
think that we have ever made a contribution to them (even though they
offered to host for free, it is still a costly undertaking). finally, we
may also need money for more hardware, depending on what the underlying
problem is.


solidarity,

	--gdm



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 835 bytes
Desc: OpenPGP digital signature
Url : http://lists.indymedia.org/pipermail/imc-docs/attachments/20090109/5da5db03/attachment.pgp