[IMC-Docs] [Fwd: Reboot: kompost.indymedia.org 2009-01-08 12:26 System Events]

[Alan Dawson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

hi...

I've had some time to look at kompost today.  Please share this with
anybody who is working  to restore kompost




Kompost

=============
Physical tests
=============

Kompost has had memtest86+ installed

memtest86+ was run

Memory errors were shown

Each memory stick was removed ( there were 2 512MB stick and 2 memory
slots ) and tested independently in each memory slot.

Both sticks showed errors.


The memory hasn't been tested in another machine, nor fresh memory
placed in kompost, so I can't say whether the motherboard has problems,
killing memory, or the  sticks independently went bad.  I also didn't
note the exact errors that the sticks failed on.

At this point the disks were removed from kompost and placed in another
machine.

The disks were mounted and an image of the  decrypted vserver partition
has been taken using dd

this has been mounted  read only as a loop device

the last known good backup of docs.indymedia.org has been restored the
new machine.

The disk partition that both of these directory structures  ( ie humus
filesystem and the backup ) are mounted  on has been mounted read only
to prevent any accidental changes to them.

===============
General method.
===============

I'm not able to repair the filesystem, or comment on whether thats
possible.  The investigations below have been performed with the images
and directories marked read only, to prevent the possibility of further
changes and damage to humus.

With that limitation this is what i've done

As far as I understand Twiki has two important directory areas, "pub"
and "data" .

data contains the twiki pages, and revision history.
pub contains attachments to pages.

data directory
==============

This contains a directory for each subtwiki

I've taken directory listings of each subdirectory of the data directory
from both the backup and humus filesystems


diffs of these directory listings have been manually inspected.

The listings and diffs from all the subtwikis look sensible ( ie have
sensible timestamps and file sizes ) EXCEPT for the Local twiki  ( which
is the largest )


Local Twiki data  directory
- ---------------------------

There are many files which have corrupted attributes.
For example :
58 files are marked as "stale NFS handles"
124 files have timestamps outside the years 2002 - 2009


Being able to repair these is beyond my understanding.

Instead I've presumed that most of the data will not have changed
between the backups and the time that kompost failed, and concentrated
on the data that have time stamps in 2008-09, 2008-10, 2008-11, 2008-12,
2009-01

There are 465 files in the Local directory with timestamps in this range


Of these  only 4 show obvious bad attributes

ellen:~/docs-work/data/humus/Local# grep -v www-data list-some.txt
?--Srwsr-t     1 2076184845          0 2008-09-15 06:47
ImcLillePropositionCharte.txt,v
- -rw-r--r--     1 daemon           2481 2008-09-22 11:54
NottsFeaturePage67.txt
- -rw-r--r--     1 daemon          45056 2008-09-28 04:19
NottsFeaturePage65.txt
p-ws-ws-wt     1      51735 1120358727 2008-12-22 22:19
BcnCriminalitzacioCa.txt,v
- -r--r--r--     1  484769793     274432 2008-12-22 22:27
BcnAntifeixismeCa.txt,v

So it seems possible to restore at least the Twiki pages from backup and
 humus to an almost complete state.


Caveats
========

Using the file system attributes to check for the changed files, after
dealing with filesystem corruption seems problematic.

But I guess that if bits have been shifted randomy due to memory errors
then its likely that file system attributes will be wildly wrong rather
than subtley different ( eg file creation date of 1916 rather than
altering some characters inside a file. )

Also I've not looked at the pub directory yet.  This has a slightly more
complex structure, and more data volume, so maybe there are many more
problems, also the problems may not be just located within the Local twiki.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAklyCdYACgkQbQ0gVPmE0wEXdACgjW0iNoKgLRpkkI8iv4pVIGBi
qmoAoMnSqU3+2MhVhxoG1C8Q6LGbaZsE
=JXTB
-----END PGP SIGNATURE-----