[imc-cms] Next Gen CMS Archeticture

[Occam]

Hi,

>>
>>>
>>> 2) What is the idea behind the grid management tier?  What problems
>>> does it solve that aren't handled by having a distributed database?
>>>
>>
>> MySQL Replication is hard to do over long distant internet  
>> connections. The
>> global character of indymedia makes a very decentral server  
>> network. This
>> decentralisation is a "must have" for us, to get around the  
>> effects of
>> repression against indymedia.
>
> When I looked at that indyarch.pdf, there was a layer of distributed
> MySQL underneath the management tier.  Is this still the plan?
>

Well, i guess its a possible setup for some admins, but also, it  
would be possible to use ICE to keep the databases up-to-date.  
However you develope it, its always possible to run MySQL replication.

I had some pratical problems with a mysql replication setup over long  
distance connections, thats why i dislike it a bit, but thats my  
personal opinion. In a "lan" setup, where you have direct local  
connections between the mysql servers, it works ofcause very well.

>>
>> The genereal idea is much more about fail-over, decentralisation, and
>> autonomity(!) of the different modules of a CMS. It is not just about
>> keeping the data save.
>>
>> We dont have a huge need to keep the site up at 99.9%, but we have  
>> a huge
>> need to be able to move sites/modules from one server to another.
>
> I think that doing this well would be a huge step forward for the
> network.  Has there been any work in this area?
>
> One thing I've been thinking about is creating a way to package for a
> site/module, which would include it's code, the data in the database,
> static files, scripts to set it up, etc.  I think there would be a ton
> of cool benefits if this was handled well:
>
>   * One server could backup the sites/modules of another server and to
> step in the other server goes down.
>   * It would make it easier for server admins to add sites/modules to
> their servers
>   * It could simplify making new sites/modules, since it would provide
> a standard way to run install scripts, handle getting DB info
>


Right, and this also goes very much into the "how should you code the  
cms", like where do you make the class seperation, what files do you  
put where.

For example, in the proof of concept i already have a "local" dir the  
includes the front-end applications, (cherrpy controller). So you can  
write different "apps/modules/controller" that only include what they  
need from the back-end.

Every website would have there own "local" dir, but use one shared  
core codebase (lib/*). Mutil-Site Setup as you might know from sf- 
active, drupal or typo3.

This already adds allot of possible "packedges".

So in the local part, you would have a module for search, publishing  
(video|article...), tag clouds, browsing, all the dynamic operations  
you can hardly make static.

Thats the pkg stuff you could possible do on the front-end site.

The shared/back-end part would be needed on all servers in the  
network, expect the static front-end servers, they might get there  
content via WebDAV.

So my ideas was, you always start the same malandro start up script,  
but depending on the config, you would run:

a) a db tier that is just passing objects from ice into the mysql
b) a publishing server that is creating the content object and  
transporting the media files to there final locations
c) a dynamic front-end for browsing all content objects

This implies that the frontend is written in python, which is still a  
open question and needs more research on the possible PHP options.

If you have ideas of how to structure the config / classes / dir  
structure / to make this as easy and logical als possible, feel free  
to play around with the code and let us know!

>>
>> Also, having very different servers, gives a huge need to a have a  
>> CMS that
>> scales and is flexible.
>
> I only have experience running small to medium sized IMCs, but I don't
> understand this need so well.  The IMC servers that I've worked with
> on all serve multiple IMCs.  If that's the case, then I don't see how
> the point to having a web app span multiple servers.
>
> I do think it's important to be able to have a good system for
> fail-over so the network can handle servers going down.  I also like
> the idea of static servers that to help out with bandwidth issues.
> But I don't understand why the web app itself needs to be split over
> multiple servers.
>
> I could just be misunderstanding though.  I don't have a very clear
> idea what services each part would provide.
>

Well see the examples above. If you have a small server with lets say  
1g memory and some 80g diskspace, you might have a hard time to make  
this a publishing box. Because the publishing server must be able to  
store the content for a while, must be able to handle large video  
files etc pp. Also, you need to distribute the media files to there  
final download server, which for video has to be done with bit  
torrent (see http://v2v.cc).

Well, ofcause this is not a very common "Blog like setup", BUT, we  
have the need for this (video.indymedia.org) and we would like to  
make this possible. So we need this flexible system and modularisation.

Second more common example. We have 10 IMC websites running and need  
a search for all of them. We know a good small lightweight search  
application (http://www.dataparksearch.org/) and have a rather small  
server with 500mhz and 512mb memory and 40g diskspace. This server  
could never run a mysql based search for 10 IMC websites. So we just  
create a ICE interface for data park, so that every new content is  
added to the data park index in "real time". Now, create a couple of  
sub-domains, vhost config and templates for the data park front-end,  
that intigrates will with the existing IMC layout and your done. This  
setup will even scale very well, so you can add more IMCs.

The difference is the way the index is stored. If you depend on the  
MySQL index, it will not scale very well for 10 imc's. Data park is  
already optimized to store and search huge amounts of data in a  
rather small solution.

Ofcause, for a small "Blog" setup of malandro, you can just write a  
small search app/controller/module that is using some ICE interface  
to make a search.

Its just about keeping the big and small setups in mind when writing  
the structure of the CMS, thats all i am asking for :)

>>>
>> We had some talk about how to implement it. I personaly prefer to  
>> put the
>> network layer at the "service" level and not down at the Database.  
>> So more
>> like a RPC Interface with services. ICE is doing this very well.  
>> This also
>> gives you a bit more security, since you can avoid any direct DB  
>> access.
>>
>> I never worked with REST, i dunno if its a possible solution. But,  
>> public
>> "blog" API's like wordpress has, are things we need to provide on the
>> frontend level.
>
> I think REST is pretty awesome.  It's basically just creating an API
> using HTTP, for example to get a list of articles you could do "GET
> /articles/", to fetch a particular article you could do "GET
> /articles/<id>" and to add a new one you would do "POST /articles/".
> The format of the request/response body is often XML or JSON, but it
> can be just about anything.
>
> The nice thing is that you get to leverage infrastructure for the web.
>  HTTPS provides a secure communication channel.  It's very easy to
> create a new service using cherrypy, django, rails, or whatever
> framework you want.  Another cool thing is there's little difference
> from the server's point of view between a web site and a REST service.

Well, i guess we will need a REST/JSON whatever interface anyway, to  
allow fancy flickr like tags on articles.

>>>
>>> 4) I have a lot of experience doing python web development (and  
>>> python
>>> develop in general) and pretty decent PHP experience.  Where can I
>>> help?
>>>
>>
>> Well, check out the proof of concept "malandro" code, its in  
>> python using
>> ice. If you have trouble with the setup, come over to #cms on
>> irc.indymedia.org or drop me a email. We also have a dev-server  
>> for malandro
>> with a working running setup you can play with.
>>
>> We made the proof of concept code as a base and a start and to give
>> developers something to play with, so they can see the idea.
>>
>> So, your very welcome to check it out :)
>>
>> occam
>>
>
> I got it running and it seems very cool.  The parts that excite me are
> less the actual CMS and more about building an infastructure around
> it.  I really want to work on that packaging idea above.  I'd also be
> interested it working on modules, especially modules that could be
> shared by different CMSs.
>

Good. I hope this email gives you some more ideas of what and how to  
implement the packaging for malandro.

Feel free to give imc-cms all your input and ideas you have, there  
are many developers on the list already, just do the technobubble  
talk - ppl will understand.

If you already have actual changes for malandro, let us know too and  
send some patches or code pieces. The current code is made for  
playing around with it!

Thanks allot for having a look into all this!

We will work on the documentation of the ideas and the proof-of- 
concept a bit more, and give everything a bigger presentation in the  
next months. Its about time to get all the developers interested! :)

BTW, if your somewhere near Berlin by the end of the year, i will be  
at the CCC Congress, maybe we can meet there.

   occam