Server cluster

This page is going to describe some basic information related to server clustering...

Requirements

database server - ACID compliant, for example PostgreSQL and MariaDB
main server that is able to share dataroot - locking support recommended, for example NFS
load balancer - for example Nginx
cluster nodes - web servers
Redis (or Memcached) server for shared MUC caches

Note: this guide is not intended for Windows OS or any other Microsoft technologies.

Initial installation

Perform standard CLI installation on the main server using shared database and dataroot directory.
Setup web servers on cluster nodes - use local dirroot and shared database and dataroot.
Configure load balancing.

Related config.php settings

$CFG->wwwroot

It must be the same on all nodes, it must be the public facing URL. It cannot be dynamic.

$CFG->sslproxy

Enable if you have https:// wwwroot but the SSL is done by load-balancer instead of web server.

Please note that it is not compatible with $CFG->loginhttps. This is because in order for loginhttps to work, we need to know the original request, and whether it was http or https. This allows us to only provide the login form over ssl. The original request is lost when made through a "reverse" proxy / load balancer, so we cannot determine what protocol the request was made in. So we can't provide most of moodle over http and the login page over https.

Enable Secure cookies only to make your site really secure, without it cookie stealing is still trivial.

$CFG->reverseproxy

Enable if your nodes are accessed via different URL.

Please note that it is not compatible with $CFG->loginhttps. This is because in order for loginhttps to work, we need to know the original request, and whether it was http or https. This allows us to only provide the login form over ssl. The original request is lost when made through a "reverse" proxy / load balancer, so we cannot determine what protocol the request was made in. So we can't provide most of moodle over http and the login page over https.

$CFG->dirroot

It is strongly recommended that $CFG->dirroot (which is automatically set via realpath(config.php)) contains the same path on all nodes. It does not need to point to the same shared directory though. The reason is that some some low level code may use the dirroot value for cache invalidation.

The simplest solution is to have the same directory structure on each cluster node and synchronise these during each upgrade.

The dirroot should be always read only for apache process because otherwise built in plugin installation and uninstallation would get the nodes out of sync.

$CFG->dataroot

This MUST be a shared directory where each cluster node is accessing the files directly. It must be very reliable, administrators cannot manipulate files directly.

Locking support is not required, if any code tries to use file locks in dataroot outside of cachedir or muc directory it is a bug.

$CFG->alternative_file_system_class

By default all uploaded content files are de-duplicated are stored inside [$CFG->dataroot]/filedir. But you can replace this is an alternative file store, like a cloud store such as AWS S3.

$CFG->alternative_file_system_class = '\tool_objectfs\s3_file_system';

https://github.com/catalyst/moodle-tool_objectfs/

NOTE: Even if you use an alternate filesystem you MUST still have a shared $CFG->dataroot.

$CFG->tempdir

This directory MUST be shared by all cluster nodes. Locking is required.

$CFG->cachedir

This directory MUST be shared by all cluster nodes. Locking is required.

$CFG->localcachedir

The difference from $CFG->cachedir is that the directory does not have to be shared by all cluster nodes, the file contents do not change. Use local fast filesystem on each cluster node.

The nature of this path is very slow moving files that stay for a long time and are written once and never overwritten.

$CFG->localrequestdir

By default this is created inside /tmp or whatever sys_get_temp_dir() returns and is generally already local disk and not shared with other nodes. This directory holds temporary folders and files which only exist for the duration of a single http request. If you do change this use local fast filesystem on each cluster node.

$CFG->backuptempdir

Server clusters MUST use shared filesystem for backuptempdir! This directory gets very large and fast bursts of IO but files do not generally live here for any length of time. They are write heavy and do not have much contention.

$CFG->alternative_component_cache

This is used to cache a very low level representation of the components which Moodle uses. Because it is so low level it is before other higher level API's are available. It is always loaded on every request so putting this onto local disk is much better.

Typically the $CFG->alternative_component_cache = '/local/cache/dir/core_component.php' would point to local node cache directory.

NOTE: Before upgrade the administrator MUST have to manually execute following on each node:

$ php admin/cli/alternative_component_cache.php --rebuild

Alternatively you could put the cache file directly into dirroot and distribute it together with new PHP source files. Yet another possibility would be to purge all local caches on all nodes before upgrade or simply change the script name in config.php on all nodes before upgrade. Or this could be built into new containers as part of your CI / CD deployment.

Implemented in Moodle 2.6, see MDL-40475.

Performance recommendations

Use OPcache extension on all cluster nodes.
Set $CFG->localcachedir to fast local filesystem on each node.
Use one big central memcached server for all shared caches that support it.
Use local memcached instances on cluster nodes for local caches that support it.
Store user sessions in one shared server e.g. Redis. (See Session handling for details)
Use fast local directory for dirroot on each cluster node.
Use dynamic cluster node management.
Use transparent proxy servers.

Upgrade procedure

Disable load balancer.
Upgrade dirroot files on master server.
Perform CLI upgrade.
Manually reset all caches in cluster nodes - restart web server, delete localcachedir and temp directories, restart memcached, etc. Or delete all cluster nodes and create nodes from a new template.
Enable load balancer.

Step by step guide for server clustering in Moodle 2.6

From hardware and performance forum thread https://moodle.org/mod/forum/discuss.php?d=251547

1 Determine your specific need for caches. This involves the concepts of shared cache or local cache, disk based cache or server based cache or protocol specific cache like Memcached or MongoDB.
- 1.1 If you have a slow shared disk, you might benefit from a local disk cache.
- 1.2 If you have only a few users on your Moodle site, you might not want to set up a Memcached service, even if the acceleration that Memcached provides, is very significant for the user experience.
- 1.3 If you want to do anything in your power for accelerating the site, you might consider MongoDB.
2 Configure the caches, test them and verify them. There is room for improvement in the examples for cache configuration. This is probably the most difficult step.
3 Install the cache support on server level. This may involve installing php modules or config for php modules.
4 Create a cache instance in MUC here: example.com/cache/admin.php?action=addstore&plugin=memcached
- 4.1 Give the cache instance some arbitrary name.
- 4.2 All other fields have a question mark that can be clicked on for excellent help that tells you what to fill in, and the syntax (very important).
- 4.3 The result should be a Configured Store Instance, with the name of your choice.
5 The final step is to deploy the created cache instances by Editing Mappings for e.g. Language string cache in the list of Known cache definitions.
- 5.1 While experimenting with this, I have found it a life saver to open a separate browser window, where the default setting is selected, so I just need to click on the Save button to revert the setting, just in case I lose contact with the site.
- 5.2 Select the name of your configured store instance from the dropdown list and click on the Save button.
- 5.3 Test the new cache setting. If you lose contact with the site or it behaves weirdly, try using the trick in step 1. In case of emergency you might remove the cache config file (muc/config.php) in the folder pointed to by $CFG->dataroot . It seems that Moodle writes a new default config file if it is missing.

Documentation