[chavez-tech] Chavez is runing out of disk space

[ng]

Hi

> Well I guess it must reflect which servers are actually serving content at 
> the moment, although there may be inactive mirrors.
> 
> Looks like www.indymedia.org.uk would be the best one to kick off if they 
> don't want to reduce their disk space usage and we can't upgrade.

might be, I double checked it with occam a dns working person, and he
said that my assumption from dig results are correct.

>>> Why is www.indymedia.org.uk hosting so many old videos? Could they be 
>>> encouraged to move them to the new media server?
>> indy.de kids once used a script to clean up their database as well
>> generated media. afair they could clean up with that about a 1/3 of
>> their disk usage. this script simply deletes all hidden media or media
>> only associated with hidden articles, as unfortunately hidden media
>> don't get deleted by mir. I don't know how much indy.uk has to hide
>> media etc. however this script might be worth a try?! ping skep or briks
>> to get more about that.
> 
> I don't have their details, perhaps we could get the UK people to talk to 
> them if we give them an ultimatum to reduce their disk usage or get kicked 
> off? Maybe we could (in future) put a quota on their user account?

hmm there will always be some site which use a lot and some less. I
think a quota would just raise some more problems. activly watching
(like chrisc did :) ) might simply be the best solution.

>> as well indy.de uses a different url (media.de.indymedia.org) for media
>> including. so they can setup a special media mirror and don't need the
>> media on every mirror. this might be also worth a try?!
> 
> Seems like a good idea. How does the video get from the publishing server 
> (when uploaded) to the media server? How hard is it to set up, and is 
> anyone from indy.de available to help the indy.uk crew to do it?

I think they would like to help.
The content is coming by rsync scripts pushing the stuff out. So some
advanced rsync scripts for the different mirrors (like excluding
/images/ and /media/ for the/some non-media mirrors) might be enough.
However it have to be ensured that the media mirrors have enough
bandwith, has media mirroring takes a lot more than the text pages.

>> I think if we could find for org or uk a new mirror the problems might
>> be solved easy. all the other solutions: script to clean up, special
>> media mirror need some work, however the status quo would rest.
> 
> It's not really solving it, as long as these sites continue to grow 
> indefinitely we will continue to run into problems every few years.

yeah for sure.

>> we have a server here in switzerland which is yet quite unused etc. 
>> (BoloBolo) however this server is reserved for non-eu mirros (due to 
>> legal reasons), so nothing for a site hosted on chavez.
> 
> Not even www.indymedia.org?

maybe, I'll do a proposal on the bolobolo-list. However I think it might
be better to have more than one mirror for .org, however this might be a
discussion for .org-techie people. who's that?

>> chris: thanks for donating the box! we are very happy to have a solution 
>> like that! dunno if we could organise some extra money, so we could 
>> migrate to a bigger box? what amount of money would be needed?
> 
> Thanks to Chris C for finding the link on Layered's site. Looks like we 
> could get a P4/2.8 with 160GB for $99/mo or a P4 3.6 with 250GB for 
> $125/mo (no RAID in either case).
> 
> I'm currently paying $78/month, but unfortunately I paid for a year in 
> advance from mid-April for a discount, so we have this server until April 
> whether we like it or not.
>
> I'd be OK with upgrading to e.g. the P4/2.8 in March, and stumping up
> the extra cash, if we can get by with what we have until then.

well i think with some cleanup and tweaking we can reach april without
disk space problems. so fore me afterwards it looks like we need 25$
more a month for the server. i think that might or should be organisable
within the collectives using chavez.


> If we could figure out how to stop rsync from royally fscking up these 
> servers then we would need far fewer mirrors worldwide. It shouldn't be 
> rocket science.

the new rsync 3.0 should be much better. anybody already tried it out?!

http://samba.anu.edu.au/ftp/rsync/rsync-3.0.0pre6-NEWS

see ENHANCEMENTS sections.

I'm not sure if it makes sense to have it only updated on chavez or it
might be better to also update it on all other servers which are
connection to chavez (where possible).

> Web serving could probably also be more efficient. For example, why, on a 
> static mirror, do we use SHTML on every page? Half the content could be 
> served by thttpd instead of Apache if we didn't do that, perhaps all of 
> it, at approximately zero load.
> 
> If not, is there any way we could use a thread worker model in Apache 
> instead of prefork?

or simply using nginx for serving the real static content. I think that
every mir site is using SSI in quite every page. at least ch is heavily
using includes to save update time and diskspace (yeah in reality this
is just a few MB of text files ;) ).

nginx example can be found here:
http://blog.kovyrin.net/2006/05/18/nginx-as-reverse-proxy/

i never used it yet, but wanted it to try out already a lot of times, so
maybe this time ;)

greets ng

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 544 bytes
Desc: OpenPGP digital signature
Url : http://lists.indymedia.org/pipermail/chavez-tech/attachments/20071216/dab8c308/attachment.pgp