Talk:Cluster Performance

Moodle can be deployed in a cluster as described in various forum threads and docs pages however these are hacks and not exactly performant or well designed. The main reason for this is that while Moodle allows for basic separation of deployment stack layers (database and application) but nothing further. The aim of this page is to evaluate the particular challenges to a true highly available load balanced Moodle cluster and any development tasks required to get past these.

This is mostly from my experience with moodle.org but I think that although moodle.org faces different problems to most education institution deployments, the core of the issues are the same. Moodle can currently be balanced between servers as long as servers are in sync. Sync can be accomplished in multiple ways, such as DRBD for storage, or just rsync. The problem is that every server must be in sync at all times with all files. Most cluster filesystems perform badly or have too much latency for this to be viable. A better alternative would be decentralised storage via a CDN or remote bucket like Amazon S3. Other issues mostly revolve around database access. Sharding a DB is possible but unlikely with MySQL and requires a proxy for PostgreSQL. Moodle also lacks support for read slaves or any other system of reducing load on a single server. Lastly, while the Moodle 2.4 MUC should help with this, frontend caching support is minimal at best and systems such as Varnish or Pound cannot work effectively on Moodle.

Suggestions are as follows:

• Rewrite the File API to be more modular for backends such as filesystem, S3, and bitcask
• Allow read-slaving or data-partitioning awareness in Moodle
• Support heavy use of etags in http cache headers
• Standard location in wwwroot for plugins and language packs

Additionally, having a tiered caching architecture can massively reduce load on databases as well as PHP which may make queries fast enough to get around problems with FastCGI timeouts, allowing for the use of NginX or Lighttpd webservers with php-fpm instead of Apache.

I'm open to any thoughts from others who have worked on large Moodle deployments or cluster web serving setups but I think these are the most critical issues for the moment, particularly the File API.

---

Thanks for your comments, my comments on them: --Dan Poltawski 08:40, 13 September 2012 (WST)

Re: Filesystem.

You are right, the filesystem is the hardest part of Moodle to horizontally scale, it is a single point of failure. Shared and network filesystems are a pain. Its always been the a pain on Moodle deployments i've worked on. We've generally stuck with NFS, but tried all sorts of things for replication, from rsync, to custom zfs transfers to drbd + OCFS2. I agree that support for an s3 backend would be nice to have.

However, this goes down my list of prorities, as:

• In my experience, filesystem performance is not in itself a performance bottleneck. Especially if I take your suggesiton as 'user uploaded content', rather than system generated.
• A bucket works well for things like user uploaded content (I assume, if it allows access control as strictly as we do), but not for other moodledata uses, like language files, preprocessed themes/js etc
• S3 is a way to outsource the problem, sure. But its still fairly expensive for Moodle sites. For example, a high school I used to host had a moodledata directory of about 800GB. (Costs for storing that data right now $1200/y + maybe 200GB a month of bandwidth = $360)

Seperating out 'user uploaded content' files from 'system created' files is something worth thinking about, and itll be interesting to see where we are when MUC lands.

I think with the filesystem, I look at the data dir as being explicitly user uploaded files. Caching stuff belongs in the MUC, sessions belong in a DB, and IMO language files belong in the wwwroot instead. Good permissions should make a standard single location for plugins and language packs perfectly safe while allowing for user uploads to be located wherever and the data directory deprecated unless the user specifically WANTS it for their user uploads.

You're correct that filesystem performance is not actually a bottleneck. It's more the overhead for maintenance and sync when using a cluster filesystem (I tried GlusterFS, OCFS2, and CEPH along with DRBD+XFS) and found none of them work very well. NFS is ok but relies on a single location being ok, and I know most people won't bother with HA failover just for this. Having files offsite though (whether CDN, S3, or whatever else) sorta gets rid of this problem by just making it irrelevant in the first place. If there is a separate location for files, it can be handled in multiple ways that don't rely on webservers having sync, and I think that's the important thing here.

Actually, I think most people do bother with *failover*. They just don't bother with dual masters. DRDB works perfectly sufficiently as a failover solution, it just has a delay whilst you promote the slave in a disaster scenario. --Dan Poltawski 23:03, 15 September 2012 (WST)

Re: Sharding.

A core function of Moodle is to grade 100 quizes. To do this efficiently we need to JOIN on quiz, grades, user, enrolments, user enrolments tables (guess). We JOIN because its the most efficent way to get this information from the database. This makes sharding close to a non-starter. (Note that we used to have far-less JOINing going on and that itself became a performance problem because of the latency introduced by the amount of database queries.)

There are a lot of ways to shard. Workloads that rely on large JOINs are extremely capable on databases like Greenplum which are drop ins for PgSQL, or even just doing proxy-level sharding via PgPool or PL/Proxy. There are test tickets for this currently in the tracker too. I don't think letting Moodle be aware of sharding is a good idea because of the complicated overhead but read slaving would definitely be something to think about. While moodle.org would only use one particular setup (leaning towards PgBouncer and PL/Proxy together), the multiple options are something best evaluated by a particular administrator, and I think all bare equal weight as far as support goes.

Well, i'm glad you think moodle shouldn't be aware of a sharding solution. I know nothing about greenplum, but one google search makes me think its probably too fresh (reports suggest it doesn't support unique indexes across shards). Unfortunately I don't think read slaving would be a magic bullet either, because almost every moodle request needs to do a write. In which case, can we really trust a read slave? What if an earlier read later does a write based on that? Unfortunately this is one of the ways where moodle doesn't fit into the standard 'scaling a web application' form factor. With regard to moodle.org, from my observations its not constrained in any regard where a read slave would help anyway. --Dan Poltawski 23:03, 15 September 2012 (WST)

Re: etags, "frontend caching support is minimal at best and systems such as Varnish or Pound cannot work effectively on Moodle"

Please be more specific. We already 'support heavy use of etags'. Point to requests which you believe are inadequate. I'm sure there are gaps, but its likely things are not cacheable because they are not cacheable (they are dynamic or require authentication).

I'm talking specifically about dynamic content that can be returned without re-querying it from the database. This would be more useful for course information and forums I think but Varnish allows for fairly explicit access controls via HTTP headers to prevent unauthorized access to precached data.

The 'content' which you are talking about is explicitly the sort of problem with MUC is looking to solve. But i'm afraid etags and caching headers are a red herring here, they are are useless in this scenario. The actual pages are completely dynamic, you could turn from a teacher to a student between two requests, or you could be unrolled in that time. This content is dynamic. Moodles permissions structure is complex, so two interface elements available in the first request may not be available in the second. --Dan Poltawski 23:03, 15 September 2012 (WST)

Re: tiered caching architecture

Nice buzzword. Please be more specific.

I'm not sure if this is in the scope of the MUC or not but I mean something similar to Perl's CHI caching modules. I think the Zend caching framework does something similar too. The basic gist is multiple caching backends searched for either specific things or globally in a set order of priorities. An example would be rendered HTML snippets or small database query results being stored in local memory for PHP, things that are accessed more frequently but only by a few nodes in a localhost Redis or Memcached DB, and larger datasets in a networked Memcached DB. Basically, a caching hierarchy depending on what the data is and how it is accessed. Not entirely sure how helpful this would be for Moodle but if a particular node in a Moodle cluster is getting specific traffic, it would make sense to only cache that information locally to that node.

See previous response, what is achievable I think is in MUC. But with regard to snippets, its nearly impossible, because even something as simple as generating a listing of users is problematic, because teacher A may have the permission to view ID numbers, but teacher B does not. Thus we need to dymically generate the content for both teachers. --Dan Poltawski 23:03, 15 September 2012 (WST)