[mir-coders] Re: [IMC-Tech] cms independent push of multimedia content

[Chris]

Hi

On Sun 10-Oct-2004 at 01:49:54PM +0200, matze wrote:
> 
> wouldn't it be nice to have a (almost) real-time copy of
> the multimedia data of your site? this can be achieved
> with mechanisms like rsync, but with huge multimedia
> archives periodical rsyncs in short periods can result
> in load problems. rsync in each run has to scan the
> whole directory to know if there are new or deleted
> files.

Agreed.

> why not delegate this task to the cms, who knows about
> each addition and deletion? 

Yep, this is how it could work with Mir:

Mir currently writes files to disk, what if instead it
wrote them to a subversion repo (subversion is like cvs,
but it versions dirs also and can run over http), this
subversion repo could be on another machine, it could be
backed up via existing tools for mirroring subversion
repo's.

(Actually Mir could still write files to disk but also
trigger a 'svn add' and 'svn ci' as needed after creating
the files to get them added to the repo).

Then the actual www servers have a cron job that does 'svn
up' every 15mins say. If there is new or modified stuff
they get it straight away, no need to checking huge list
of files :-)

> in order to avoid that each cms has to implement it's
> own push mechanism(s) the synchronization would be done
> by a separate application/script. the cms passes a list
> of new files to the cms (via cmdline, database table, a
> locked file, socket, ...), the push app processes it's
> queue(s) and pushes/deletes the files with the transport
> mechanism of the backup box (ftps, scp, rsync (<=
> concrete files), unison, ...).

Um, my idea workd a bit different from this I think...

> in brief: the cms only need to implement the push script
> interface, the synchronisation with different transport
> mechanisms is handled by a separate app/script that
> hopefully evolves to a reliable and flexible solution.

Yep.

> i'd be happy to code (probably in python) a solution
> like this if ppl find it useful. the idea is draft
> v0.0.1, open for all kinds of input

Yep, good thinking :-)

Another thing to cosider (and these things have been
talked about a bit on irc) is how to have multiple
publishing servers, at the moment with Mir you can have
hot backup publishing servers but not multiple ones
running at once, to have this some (complicated, I expect)
work would be needed on the code base...

One other idea is to have public db dumps like Wikipedia
does, but for this to work the db's would need the
sensitive info (internal comments etc) removing first.

So... are we gonna have a irc meeting at some point to
discuss tis stuff?

Chris