Server cluster

This page is going to describe some basic information related to server clustering...

Requirements

• database server - ACID compliant, for example PostgreSQL and MariaDB
• main server that is able to share dataroot - locking support recommended, for example NFS
• load balancer - for example Nginx
• cluster nodes - web servers
• Memcached server for shared caches

Note: this guide is not intended for Windows OS or any other Microsoft technologies.

Initial installation

1. Perform standard CLI installation on the main server using shared database and dataroot directory.
2. Setup web servers on cluster nodes - use local dirroot and shared database and dataroot.
3. Configure load balancing.

Related config.php settings

$CFG->wwwroot

It must be the same on all nodes, it must be the public facing URL. It cannot be dynamic.

$CFG->sslproxy

Enable if you have https:// wwwroot but the SSL is done by load-balancer instead of web server.

Enable Secure cookies only to make your site really secure, without it cookie stealing is still trivial.

$CFG->reverseproxy

Enable if your nodes are accessed via different URL. Please note that it is not compatible with $CFG->loginhttps.

$CFG->dirroot

It is strongly recommended that $CFG->dirroot (which is automatically set via realpath(config.php)) contains the same path on all nodes. It does not need to point to the same shared directory though. The reason is that some some low level code may use the dirroot value for cache invalidation.

The simplest solution is to have the same directory structure on each cluster node and synchronise these during each upgrade.

The dirroot should be always read only for apache process because otherwise built in plugin installation and uninstallation would get the nodes out of sync.

$CFG->dataroot

This MUST be a shared directory where each cluster node is accessing the files directly. It must be very reliable, administrators cannot manipulate files directly.

Locking support is not required, if any code tries to use file locks in dataroot outside of cachedir or muc directory it is a bug.

$CFG->tempdir

It is recommended to use separate ram disks on each node. Scripts may use this directory during one request only. The contents of this directory may be deleted if there is no pending HTTP request, otherwise delete only files with older timestamps.

Always purge this directory when starting cluster node, you may get some notices or temporary errors if you purge this directory while other requests are active.

$CFG->cachedir

This directory MUST be shared by all cluster nodes. Locking is required.

$CFG->localcachedir

The difference from $CFG->cachedir is that the directory does not have to be shared by all cluster nodes, the file contents do not change. Use local fast filesystem on each cluster node.

Performance recommendations

1. Use OPcache extension on all cluster nodes.
2. Set $CFG->localcachedir to fast local filesystem on each node.
3. Set $CFG->tempdir to fast local filesystem on each node.
4. Use one big central memcached server for all shared caches that support it.
5. Use local memcached instances on cluster nodes for local caches that support it.
6. Store user sessions in one shared memcached server. (See Session_handling for details)
7. Use fast local directory for dirroot on each cluster node.
8. Use dynamic cluster node management.
9. Use transparent proxy servers.

Upgrade procedure

1. Disable load balancer.
2. Upgrade dirroot files on master server.
3. Perform CLI upgrade.
4. Manually reset all caches in cluster nodes - restart web server, delete localcachedir and temp directories, restart memcached, etc. Or delete all cluster nodes and create nodes from a new template.
5. Enable load balancer.

Step by step guide for server clustering in Moodle 2.6

From hardware and performance forum thread https://moodle.org/mod/forum/discuss.php?d=251547

• 1 Determine your specific need for caches. This involves the concepts of shared cache or local cache, disk based cache or server based cache or protocol specific cache like Memcached or MongoDB.
  • 1.1 If you have a slow shared disk, you might benefit from a local disk cache.
  • 1.2 If you have only a few users on your Moodle site, you might not want to set up a Memcached service, even if the acceleration that Memcached provides, is very significant for the user experience.
  • 1.3 If you want to do anything in your power for accelerating the site, you might consider MongoDB.
• 2 Configure the caches, test them and verify them. There is room for improvement in the examples for cache configuration. This is probably the most difficult step.
• 3 Install the cache support on server level. This may involve installing php modules or config for php modules.
• 4 Create a cache instance in MUC here: example.com/cache/admin.php?action=addstore&plugin=memcached
  • 4.1 Give the cache instance some arbitrary name.
  • 4.2 All other fields have a question mark that can be clicked on for excellent help that tells you what to fill in, and the syntax (very important).
  • 4.3 The result should be a Configured Store Instance, with the name of your choice.
• 5 The final step is to deploy the created cache instances by Editing Mappings for e.g. Language string cache in the list of Known cache definitions.
  • 5.1 While experimenting with this, I have found it a life saver to open a separate browser window, where the default setting is selected, so I just need to click on the Save button to revert the setting, just in case I lose contact with the site.
  • 5.2 Select the nam of your configured store instance from the dropdown list and click on the Save button.
  • 5.3 Test the new cache setting. If you lose contact with the site or it behaves weirdly, try using the trick in step 1. In case of emergency you might remove the cache config file (muc/config.php) in the folder pointed to by $CFG->dataroot . It seems that Moodle writes a new default config file if it is missing.

See also

• Performance recommendations
• Caching
• dev:Server clustering improvements proposal
• Hardware and performance forum thread [1]
• How to Cluster Moodle on Multiple Servers for High Availability and Scalability [2]