Talk:Cluster Performance

Moodle can be deployed in a cluster as described in various forum threads and docs pages however these are hacks and not exactly performant or well designed. The main reason for this is that while Moodle allows for basic separation of deployment stack layers (database and application) but nothing further. The aim of this page is to evaluate the particular challenges to a true highly available load balanced Moodle cluster and any development tasks required to get past these.

This is mostly from my experience with moodle.org but I think that although moodle.org faces different problems to most education institution deployments, the core of the issues are the same. Moodle can currently be balanced between servers as long as servers are in sync. Sync can be accomplished in multiple ways, such as DRBD for storage, or just rsync. The problem is that every server must be in sync at all times with all files. Most cluster filesystems perform badly or have too much latency for this to be viable. A better alternative would be decentralised storage via a CDN or remote bucket like Amazon S3. Other issues mostly revolve around database access. Sharding a DB is possible but unlikely with MySQL and requires a proxy for PostgreSQL. Moodle also lacks support for read slaves or any other system of reducing load on a single server. Lastly, while the Moodle 2.4 MUC should help with this, frontend caching support is minimal at best and systems such as Varnish or Pound cannot work effectively on Moodle.

Suggestions are as follows:

• Rewrite the File API to be more modular for backends such as filesystem, S3, and bitcask
• Allow read-slaving or data-partitioning awareness in Moodle
• Support heavy use of etags in http cache headers

Additionally, having a tiered caching architecture can massively reduce load on databases as well as PHP which may make queries fast enough to get around problems with FastCGI timeouts, allowing for the use of NginX or Lighttpd webservers with php-fpm instead of Apache.

I'm open to any thoughts from others who have worked on large Moodle deployments or cluster web serving setups but I think these are the most critical issues for the moment, particularly the File API.

---

Thanks for your comments, my comments on them: --Dan Poltawski 08:40, 13 September 2012 (WST)

Re: Filesystem.

You are right, the filesystem is the hardest part of Moodle to horizontally scale, it is a single point of failure. Shared and network filesystems are a pain. Its always been the a pain on Moodle deployments i've worked on. We've generally stuck with NFS, but tried all sorts of things for replication, from rsync, to custom zfs transfers to drbd + OCFS2. I agree that support for an s3 backend would be nice to have.

However, this goes down my list of prorities, as:

• In my experience, filesystem performance is not in itself a performance bottleneck. Especially if I take your suggesiton as 'user uploaded content', rather than system generated.
• A bucket works well for things like user uploaded content (I assume, if it allows access control as strictly as we do), but not for other moodledata uses, like language files, preprocessed themes/js etc
• S3 is a way to outsource the problem, sure. But its still fairly expensive for Moodle sites. For example, a high school I used to host had a moodledata directory of about 800GB. (Costs for storing that data right now $1200/y + maybe 200GB a month of bandwidth = $360)

Seperating out 'user uploaded content' files from 'system created' files is something worth thinking about, and itll be interesting to see where we are when MUC lands.

Re: Sharding.

A core function of Moodle is to grade 100 quizes. To do this efficiently we need to JOIN on quiz, grades, user, enrolments, user enrolments tables (guess). We JOIN because its the most efficent way to get this information from the database. This makes sharding close to a non-starter. (Note that we used to have far-less JOINing going on and that itself became a performance problem because of the latency introduced by the amount of database queries.)

Re: etags, "frontend caching support is minimal at best and systems such as Varnish or Pound cannot work effectively on Moodle"

Please be more specific. We already 'support heavy use of etags'. Point to requests which you believe are inadequate. I'm sure there are gaps, but its likely things are not cacheable because they are not cacheable (they are dynamic or require authentication).

Re: tiered caching architecture

Nice buzzword. Please be more specific.