[traven-tech] rsync

[jb]

hi there,

here are some scripts that pre-chew rsync' work so that it doesn't bug too
much the NFS server on ahimsa - basically on the server side there is a
script that runs some "find" command to list the files that have been
modified recently; and on the client side, the rsync command is passed
a --file-from= command that makes it read the name of the files it has to
fetch.

what's really cool is that the list of files to be mirrored may very well
be on the server side and rsync will figure it out, so there's no dodgy
wget in the way.

on the server side: something like

  cd ecuador-home-dir
  /usr/bin/find . -follow -cmin -10 -type f -not -path "*/rsync/*" > \
        rsync/last-10

  the result looks like:
        http://ecuador.indymedia.org/rsync/last-10

  this file is produced every 5 minutes.

n the client side:

  ${RSYNC} rsync://${RSYNC_SERVER}/${SITE}/ ${DESTINATION_BASE}/

  with
  
  RSYNC="/usr/local/bin/rsync --timeout=360 -CLrptzvu --files-from=:/rsync/last-
10"
  (except for --files-from, I havent' got a clue what the options mean)

the scripts are here:
   http://cartman.sakeos.net/rsync/last.sh
   http://cartman.sakeos.net/rsync/rsync-imc.sh

at least rsync'ing from ahimsa, they really speed up the mirroring process.
it would probably be cool to have the prod servers produce the files in the
/rsync/ directory.

later'
jb