https://docs.moodle.org/36/en/api.php?action=feedcontributions&user=Andrewnicols&feedformat=atomMoodleDocs - User contributions [en]2024-03-28T10:44:31ZUser contributionsMediaWiki 1.39.6https://docs.moodle.org/36/en/index.php?title=Installing_MSSQL_for_PHP&diff=125313Installing MSSQL for PHP2016-09-14T01:11:02Z<p>Andrewnicols: Remove unsupported versions of PHP from docs (3.1 requires PHP 5.6)</p>
<hr />
<div>{{Installing Moodle}}== Introduction ==<br />
<br />
This short manual is suitable if you are trying to run Moodle using the SQL*Server (MSSQL) RDBMS. Steps detailed below must be performed '''before''' installing Moodle itself.<br />
<br />
Some of this may also apply if you wish to access an MSSQL server for external db authentication/enrollment. <br />
<br />
First of all, minimum required version of MSSQL has been stabilised to MSSQL 2005 (v.9).<br />
<br />
While PHP comes with one, more or less, standard extension (mssql) that provides access to MSSQL databases, early we found some hard limits on it. Basically such default extension has some limits that prevent us to use it at all (you can find more info about these problems [[Development:XMLDB problems#MSSQL, PHP, UTF-8 and UCS-2|here]]).<br />
<br />
So, in order to allow PHP (i.e. Moodle) to access to MSSQL DBs properly we have to install a '''mssql extension alternative''' to save us from the problems related above. See the sections below for details about the various options.<br />
<br />
== Installation overview ==<br />
<br />
1. Get MSSQL Server installed and running. ([http://www.microsoft.com/sql/editions/express/default.mspx A free limited version, SQL Server Express Edition] is available for testing.)<br />
:Make sure that you choose mixed authentication (Windows and local accounts) to keep things simpler later. You'll be asked to define the "sa" account password (it's the default System Administrator account which has full access to all databases by default).<br />
<br />
2. Make sure MS SQL Server can accept incoming TCP/IP connections on port 1433 (the standard one).<br />
:You might need to explicitly allow this in your Windows firewall (see the Control Panel). You may also need to edit options in the :'''SQL Server Configuration Manager''' -> '''Network Configuration''' -> '''Protocols''' -> '''TCP/IP enabled'''<br />
<br />
3. Open the "SQL Server Management Studio" and create a new empty database. If you are using the "sa" account then you don't need to do anything else here.<br />
<br />
<span id="Configuration"></span>4. Configure these settings in your created (and still empty) database:<br />
Configure these settings in your created (and still empty) database:<br />
:*Use a case sensitive collation, such as Latin1_General_CS_AS.<br />
:*ANSI NULLS Enabled = True (ALTER DATABASE xxxx SET ANSI_NULLS ON)<br />
:*Quoted Identifiers Enabled = True (ALTER DATABASE xxxx SET QUOTED_IDENTIFIER ON)<br />
:*Is Read Committed transaction ON = True (ALTER DATABASE xxxx SET READ_COMMITTED_SNAPSHOT ON)<br />
:** In older versions this is not settable via the DB properties. To set READ_COMMITTED_SNAPSHOT, there must be no active connections to the database except for the connection executing the ALTER command. If you are viewing the DB in the Server Management Studio, disconnect from any servers in the "Object Explorer" (right-click > Disconnect), then create a "New Query" and run the ALTER command. See http://msdn.microsoft.com/en-us/library/bb522682.aspx for details.<br />
:** If your DB name starts with a number, you may need to put quotes around the DB name in the query.<br />
<br />
<br />
5. Get PHP installed with a web server. Unless you want to do it under IIS or some other way, the packages on the [http://download.moodle.org Moodle download page] are a good solution.<br />
<br />
6. Choose one of the following specific sections for your server to install the '''mssql extension alternative''' installed and running properly on your PHP box.<br />
<br />
7. Set the following settings in your php.ini file<br />
:* mssql.textlimit = 20971520<br />
:* mssql.textsize = 20971520<br />
<br />
8. With all this properly configured, you can continue with a [[Installing Moodle|standard Moodle installation]].<br />
<br />
== Microsoft Drivers for SQL Server for PHP ==<br />
<br />
In July 2008 Microsoft [http://social.msdn.microsoft.com/forums/en-US/sqldriverforphp/thread/a10e5202-9e41-4ff8-a33e-fbcc7b951be2/ released] a new SQL Server Driver for PHP. This is a PHP extension that allows PHP scripts to read and write data on Microsoft SQL Server databases and it overcomes the problems with the native SQL Server extension that was previously bundled with PHP.<br />
<br />
When using [[IIS]] it is strongly recommended to use the official Microsoft PHP installer from http://php.iis.net/, it should include the latest version of necessary drivers and it also simplifies future upgrades and configuration.<br />
<br />
For Windows servers with [[Apache]] see http://www.microsoft.com/en-us/download/details.aspx?id=20098.<br />
<br />
To know more about how to run Moodle with these drivers go to [[Using the Microsoft SQL Server Driver for PHP]].<br />
<br />
== Using FreeTDS on Windows ==<br />
<br />
<p class="note">'''Important Note 1:''' Due to some previous bugs it's highly recommendable to use PHP >= 5.2.6 and FreeTDS 0.82 + post-release patches ([http://tracker.moodle.org/browse/MDL-14725 more info]).</p><br />
<br />
If your web server is on Windows, use '''php_dblib.dll'''. Despite the name, it's FreeTDS compiled for Windows. (Go to this page for information on [https://docs.moodle.org/en/FreeTDS Using FreeTDS for Unix].) <br />
<br />
Originally we were using the DLLs available at [http://kromann.info/article.php?Id=11062598797760000 Frank Kromann's site], but they are outdated (using old versions of FreeTDS) and that has caused [http://tracker.moodle.org/browse/MDL-14725 some problems] in the past.<br />
<br />
So, right now, the recommended way to use FreeTDS under Windows is to use PHP 5.2.x following the following instructions:<br />
<br />
1. Download the appropriate copy of php_dblib.dll from the list below, and save it into your /PHP/ext directory.<br />
<br />
{| class="table table-striped table-bordered"<br />
|-<br />
! PHP version !! [http://www.iis-aid.com/articles/my_word/difference_between_php_thread_safe_and_non_thread_safe_binaries Thread Safe] !! FreeTDS version !! Download URL<br />
|-<br />
| PHP 5.6.x (vc11-unkown architecture) || Both || 0.95 || You can find some (unverified/untested) drivers in [https://moodle.org/mod/forum/discuss.php?d=232844#p1297504 this forum post], courtesy of Josh Urbain.<br />
|-<br />
| colspan="4" | Thanks to [http://remote-learner.net/ Remote-Learner]] (Moodle [http://moodle.com/partners/ Partner]) and specially to Bryan Williams, donating one Visual C++ 6.0 Pro license to Moodle. Thanks to Trevor Johnson and his builds of the dblib extensions. Thanks to Daniele, Doug, Luis, Sean and many others by their collaboration in MDL-14725. Thanks to Frediano Ziglio and James K. Lowden from [http://freetds.org freetds.org] by their support. Thanks to [[User:Alastair Hole|Alastair Hole]] and Matt Rusiniak for providing the PHP 5.3 builds of the libraries. Thanks to Enyby by providing the PHP 5.4 builds of the libraries. Thanks to David Aylmer and Matt Rusiniak for providing the PHP 5.5 builds of the libraries. Thanks!<br />
|}<br />
<br />
(alternatively here you can find some [[Development:Compiling FreeTDS under Windows|instructions to build those freetds extensions under win32]] yourself)<br />
<br />
<br />
2. FreeTDS requires the .NET Framework v1.1 to be installed. You can [http://www.microsoft.com/downloads/details.aspx?FamilyID=262d25e3-f589-4842-8157-034d1e7cf3a3&DisplayLang=en download it from the Microsoft website] along with its [http://www.microsoft.com/downloads/details.aspx?FamilyID=a8f5654f-088e-40b2-bbdb-a83353618b38&DisplayLang=en service pack]. Alternatively, if you do not wish to install this framework, you can [http://kromann.info/ms-libs/msvcr71.dll download the required DLL] from Frank's site, and save it into your /PHP root directory.<br />
<br />
<br />
3. Edit your /PHP/php.ini file and add this line:<br />
<br />
extension=php_dblib.dll <br />
<br />
Make sure that any lines referring to the php_mssql.dll extension are DISABLED (commented out).<br />
<br />
<br />
4. When the PHP engine loads the FreeTDS extension it needs to be passed certain infiormation in order to be able to connect to your Moodle database. To retrieve this information FreeTDS looks for a file called '''freetds.conf''' in the root folder of the server that PHP installed on (e.g. C:\).<br />
<br />
<br />
'''freetds.conf''' should have the following structure:<br />
<br />
[global]<br />
host = xxx.xxx.xxx.xxx (host name or ip of the MSSQL server)<br />
port = 1433<br />
client charset = UTF-8<br />
tds version = 8.0<br />
text size = 20971520<br />
<br />
<br />
If you want to connect to a particular [http://msdn.microsoft.com/en-us/library/aa174516(SQL.80).aspx instance] of MSSQL you should specify the instance name:<br />
<br />
[global]<br />
host = xxx.xxx.xxx.xxx (host name or ip of the MSSQL server)<br />
instance = xxx (instance name, e.g. INST2)<br />
port = 1433<br />
client charset = UTF-8<br />
tds version = 8.0<br />
text size = 20971520<br />
<br />
<br />
'''Notes:'''<br />
*You can configure FreeTDS to look for the freetds.conf file in any directory that you want - you don't have to use C:\. To do this create a SYSTEM environment variable called '''FREETDS''' and point it to the directory where you have installed the freetds.conf file. If you do not set this environment variable FreeTDS will look for the freetds.conf file in the C:\ folder, which is the default. One possible benefit of setting the FREETDS environment variable and using a different installation directory for freetds.conf is that C:\ is very predictable to a hacker that knows anything about FreeTDS and that is the first place that he would look if he wanted to compromise your system. So, using a different installation directory would just make your system stronger. See the FreeTDS [http://www.freetds.org/userguide/envvar.htm Setting the environment variables] documentation for more information about this FREETDS environment variable.<br />
<br />
*Alternatively, you can [[Development:Compiling FreeTDS under Windows|recompile]] the FreeTDS extension yourself and change the default location to your preferred location at compile time. Then it is not necessary to create any environment variable. You must just ensure that freetds.conf is in the same folder that you specify when you compile php_dblib.dll.<br />
<br />
*MSSQL is usually installed with port 1433 as the default. However, if the port was changed on your server when you installed MSSQL then you need to specify the correct port number.<br />
<br />
<br />
5. Your Moodle '''config.php''' should include lines like these:<br />
<br />
<code php><br />
$CFG->dbtype = 'mssql'; // Required<br />
$CFG->dbhost = 'localhost'; // assuming MS SQL is on the same server, otherwise use an IP<br />
$CFG->dbname = 'moodle'; // or whatever you called the database you created<br />
$CFG->dbuser = 'yourusername'; // I usually use the 'sa' account (dbowner perms are enough)<br />
$CFG->dbpass = 'yourpassword';<br />
$CFG->dbpersist = false;<br />
$CFG->prefix = 'mdl_'; //Prefix, you can change it, but NEVER leave it blank.<br />
</code><br />
<br />
If you don't have a config.php file yet, it can be generated as normal from the Moodle installer. Alternatively you can use the config-dist.php file that comes with the Moodle package to create your own config.php file.<br />
<br />
<br />
6. Restart or start your web server. If Moodle still cannot communicate with the database server, please turn display_startup_errors to "On" in your /PHP/php.ini file, then restart the web server and check for any errors that may indicate incorrect DLL versions or missing dependencies. These error reports, turned off by default in PHP, can be vital in locating a problem with new extension installations.<br />
<br />
<br />
7. Database conection test, try this PHP script, just put in a text file called test.php change ('localhost', 'db_user', 'db_password') to suite your setup, and load from local host (http://localhost/test.php)...<br />
<br />
<code php><br />
<?php<br />
$link = mssql_connect('localhost', 'db_user', 'db_password');<br />
if(!$link) {<br />
echo'Could not connect';<br />
die('Could not connect: ' . mssql_error());<br />
}<br />
echo'Successful connection';<br />
mssql_close($link);<br />
?><br />
</code><br />
<br />
8. Install Moodle as usual. Good luck!<br />
<br />
<br />
=== Troubleshooting ===<br />
If you encounter some problems you can try:<br />
*check that you have DotNet framework 1.1 installed (later version are installed on Vista, but you could need this specific one)<br /><br />
*enable TCP/IP for MSSQL: SQL Server 2005 Network Configuration -> Protocols for MSSQLSERVER -> TCP/IP (Enable) -> Properties -> Ip Addresses -> 127.0.0.1 (Active+Enable)<br /><br />
*make sure the SQL Server Browser service is running SQL Server 2005 Network Configuration -> SQL Server Services<br /><br />
*if you are using SQL Server 2005 and you have the error ''4004: Unicode data in a Unicode-only collation or ntext data cannot be sent to clients using DB-Library (such as ISQL) or ODBC version 3.7 or earlier'', try the ODBTP method (next chapter). The SQL Server complaining that it doesn't support pure Unicode via TDS or older versions of ODBC. Microsoft has deprecated DB-Library a long ago, in favor of ODBC, OLE DB, or SQL Native Client. Many new features of SQL 2005 aren't accessible via DB-Library so if you need them, you could have to switch away from tools based on TDS and DB-Library :(<br />
<br />
== FreeTDS on Linux (on Ubuntu by compiling an mssql.so extension) ==<br />
This is a good read to [http://www.robert-gonzalez.com/2009/02/18/building-the-php-ms-sql-server-extension-from-source-on-ubuntu-810/ building a FreeTDS based mssql extension for apache on Ubuntu]. Do note that [http://www.freetds.org/news.html freeTDS] 0.91 was recently released, you can find latest versions [http://freetds.sourceforge.net/ here]. <br />
<br />
Note: the freetds.conf file you use should have "text size = 20971520" as mentioned in the FreeTDS on Windows section otherwise you might see sessions logging out or worse apache segmentation faults. Also see [[FreeTDS]].<br />
<br />
Note2: Please note that updating php tends to remove the freetds configurations. You might need to reconfigure freetds after an upgrade<br />
<br />
== Using FreeTDS on Debian Lenny ==<br />
I found the following solution using:<br />
* PHP Version 5.2.6-1+lenny9<br />
* Microsoft SQL Server Enterprise Edition, version: 9.00.4053.00<br />
<pre>apt-get install libsybdb5 freetds-common php5-sybase<br />
/etc/init.d/apache2 restart</pre><br />
At the end of the process, if all goes fine, you will find in the mssql section of phpinfo();<br />
<br />
{| class="nicetable"<br />
|-<br />
! MSSQL Support<br />
! enabled<br />
|-<br />
| Library version <br />
| FreeTDS <br />
|}<br />
<br />
Once FreeTDS is correctly installed, don not forget to set it up following explanations in https://docs.moodle.org/en/FreeTDS<br />
<br />
== See also ==<br />
* [[Errors FAQ]]<br />
* Using Moodle [http://moodle.org/mod/forum/view.php?id=28 Installation problems forum]<br />
<br />
[[Category:XMLDB]]<br />
[[Category:DB]]<br />
[[Category:SQL databases]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Publish_as_LTI_tool&diff=123508Publish as LTI tool2016-05-16T00:18:48Z<p>Andrewnicols: Update name of LTI provider enrolment plugin.</p>
<hr />
<div>{{Enrolment}}{{New features}}The 'Publish as LTI tool' enrolment plugin, together with the LTI authentication plugin, allows remote users on a different site (known as an LTI consumer) to access selected courses and activities. In other words, Moodle functions as an LTI tool provider. Grades are sent back to the remote system.<br />
<br />
==Enabling 'Publish as LTI tool' at site level==<br />
<br />
An administrator can enable the 'Publish as LTI tool' for use across the site:<br />
<br />
# Go to ''Site administration > Plugins > Authentication > Manage authentication'' and enable LTI<br />
# Go to ''Site administration > Plugins > Enrolments > Manage enrol plugins'' and enable 'Shared external tool'<br />
<br />
It is recommended that the site administration setting 'Allow frame embedding' is enabled so that tools are displayed within a frame rather than in a new window.<br />
<br />
==Sharing access to a course or activity==<br />
<br />
# Go to the course and in ''Course administration > Users > Enrolment methods'' add 'Shared external tool' as an enrolment method<br />
# In 'Tool to be provided' select the course or activity to be shared<br />
# Click the 'Add method' button<br />
# Go to ''Course administration > Shared external tools'' and make note of the URL and secret for the LTI consumer<br />
<br />
The LTI consumer can be another Moodle site or any other LTI-consumer-compliant LMS, such as Sakai.<br />
<br />
[[es:Herramienta publicar como LTI]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Server_cluster&diff=119558Server cluster2015-08-07T07:35:06Z<p>Andrewnicols: /* Performance recommendations */</p>
<hr />
<div>This page is going to describe some basic information related to server clustering...<br />
<br />
==Requirements==<br />
<br />
* database server - ACID compliant, for example PostgreSQL and MariaDB<br />
* main server that is able to share dataroot - locking support recommended, for example NFS<br />
* load balancer - for example Nginx<br />
* cluster nodes - web servers<br />
* Memcached server for shared caches<br />
<br />
Note: this guide is not intended for Windows OS or any other Microsoft technologies.<br />
<br />
==Initial installation==<br />
<br />
# Perform standard CLI installation on the main server using shared database and dataroot directory.<br />
# Setup web servers on cluster nodes - use local dirroot and shared database and dataroot.<br />
# Configure load balancing.<br />
<br />
==Related config.php settings==<br />
<br />
===$CFG->wwwroot===<br />
It must be the same on all nodes, it must be the public facing URL. It cannot be dynamic.<br />
<br />
===$CFG->sslproxy===<br />
Enable if you have https:// wwwroot but the SSL is done by load-balancer instead of web server.<br />
<br />
Enable ''Secure cookies only'' to make your site really secure, without it cookie stealing is still trivial.<br />
<br />
===$CFG->reverseproxy===<br />
Enable if your nodes are accessed via different URL. Please note that it is not compatible with $CFG->loginhttps.<br />
<br />
===$CFG->dirroot===<br />
It is strongly recommended that $CFG->dirroot (which is automatically set via realpath(config.php)) contains the same path on all nodes. It does not need to point to the same shared directory though. The reason is that some some low level code may use the dirroot value for cache invalidation.<br />
<br />
The simplest solution is to have the same directory structure on each cluster node and synchronise these during each upgrade.<br />
<br />
The dirroot should be always read only for apache process because otherwise built in plugin installation and uninstallation would get the nodes out of sync.<br />
<br />
===$CFG->dataroot===<br />
<br />
This '''MUST''' be a shared directory where each cluster node is accessing the files directly. It must be very reliable, administrators cannot manipulate files directly.<br />
<br />
Locking support is not required, if any code tries to use file locks in dataroot outside of cachedir or muc directory it is a bug.<br />
<br />
===$CFG->tempdir===<br />
<br />
This directory '''MUST''' be shared by all cluster nodes. Locking is required.<br />
<br />
===$CFG->cachedir===<br />
<br />
This directory '''MUST''' be shared by all cluster nodes. Locking is required.<br />
<br />
===$CFG->localcachedir===<br />
<br />
The difference from $CFG->cachedir is that the directory does not have to be shared by all cluster nodes, the file contents do not change. Use local fast filesystem on each cluster node.<br />
<br />
==Performance recommendations==<br />
<br />
#Use OPcache extension on all cluster nodes.<br />
#Set $CFG->localcachedir to fast local filesystem on each node.<br />
#Use one big central memcached server for all shared caches that support it.<br />
#Use local memcached instances on cluster nodes for local caches that support it.<br />
#Store user sessions in one shared memcached server. (See [[Session_handling]] for details)<br />
#Use fast local directory for dirroot on each cluster node.<br />
#Use dynamic cluster node management.<br />
#Use transparent proxy servers.<br />
<br />
==Upgrade procedure==<br />
<br />
#Disable load balancer.<br />
#Upgrade dirroot files on master server.<br />
#Perform CLI upgrade.<br />
#Manually reset all caches in cluster nodes - restart web server, delete localcachedir and temp directories, restart memcached, etc. Or delete all cluster nodes and create nodes from a new template.<br />
#Enable load balancer.<br />
<br />
==Step by step guide for server clustering in Moodle 2.6==<br />
From hardware and performance forum thread [https://moodle.org/mod/forum/discuss.php?d=251547 https://moodle.org/mod/forum/discuss.php?d=251547]<br />
<br />
* 1 Determine your specific need for caches. This involves the concepts of shared cache or local cache, disk based cache or server based cache or protocol specific cache like Memcached or MongoDB. <br />
** 1.1 If you have a slow shared disk, you might benefit from a local disk cache.<br />
** 1.2 If you have only a few users on your Moodle site, you might not want to set up a Memcached service, even if the acceleration that Memcached provides, is very significant for the user experience.<br />
** 1.3 If you want to do anything in your power for accelerating the site, you might consider MongoDB.<br />
* 2 Configure the caches, test them and verify them. There is room for improvement in the examples for cache configuration. This is probably the most difficult step.<br />
* 3 Install the cache support on server level. This may involve installing php modules or config for php modules.<br />
* 4 Create a cache instance in MUC here: example.com/cache/admin.php?action=addstore&plugin=memcached<br />
** 4.1 Give the cache instance some arbitrary name.<br />
** 4.2 All other fields have a question mark that can be clicked on for excellent help that tells you what to fill in, and the syntax (very important).<br />
** 4.3 The result should be a Configured Store Instance, with the name of your choice.<br />
* 5 The final step is to deploy the created cache instances by Editing Mappings for e.g. Language string cache in the list of Known cache definitions.<br />
** 5.1 While experimenting with this, I have found it a life saver to open a separate browser window, where the default setting is selected, so I just need to click on the Save button to revert the setting, just in case I lose contact with the site.<br />
** 5.2 Select the nam of your configured store instance from the dropdown list and click on the Save button. <br />
** 5.3 Test the new cache setting. If you lose contact with the site or it behaves weirdly, try using the trick in step 1. In case of emergency you might remove the cache config file (muc/config.php) in the folder pointed to by $CFG->dataroot . It seems that Moodle writes a new default config file if it is missing.<br />
<br />
==See also==<br />
*[[Performance recommendations]]<br />
*[[Caching]]<br />
*[[:dev:Server clustering improvements proposal]]<br />
*Hardware and performance forum thread [https://moodle.org/mod/forum/discuss.php?d=251547https://moodle.org/mod/forum/discuss.php?d=251547]<br />
*How to Cluster Moodle on Multiple Servers for High Availability and Scalability [http://www.severalnines.com/blog/clustering-moodle-multiple-servers-high-availability-scalability]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Server_cluster&diff=119557Server cluster2015-08-07T04:53:49Z<p>Andrewnicols: /* $CFG->tempdir */</p>
<hr />
<div>This page is going to describe some basic information related to server clustering...<br />
<br />
==Requirements==<br />
<br />
* database server - ACID compliant, for example PostgreSQL and MariaDB<br />
* main server that is able to share dataroot - locking support recommended, for example NFS<br />
* load balancer - for example Nginx<br />
* cluster nodes - web servers<br />
* Memcached server for shared caches<br />
<br />
Note: this guide is not intended for Windows OS or any other Microsoft technologies.<br />
<br />
==Initial installation==<br />
<br />
# Perform standard CLI installation on the main server using shared database and dataroot directory.<br />
# Setup web servers on cluster nodes - use local dirroot and shared database and dataroot.<br />
# Configure load balancing.<br />
<br />
==Related config.php settings==<br />
<br />
===$CFG->wwwroot===<br />
It must be the same on all nodes, it must be the public facing URL. It cannot be dynamic.<br />
<br />
===$CFG->sslproxy===<br />
Enable if you have https:// wwwroot but the SSL is done by load-balancer instead of web server.<br />
<br />
Enable ''Secure cookies only'' to make your site really secure, without it cookie stealing is still trivial.<br />
<br />
===$CFG->reverseproxy===<br />
Enable if your nodes are accessed via different URL. Please note that it is not compatible with $CFG->loginhttps.<br />
<br />
===$CFG->dirroot===<br />
It is strongly recommended that $CFG->dirroot (which is automatically set via realpath(config.php)) contains the same path on all nodes. It does not need to point to the same shared directory though. The reason is that some some low level code may use the dirroot value for cache invalidation.<br />
<br />
The simplest solution is to have the same directory structure on each cluster node and synchronise these during each upgrade.<br />
<br />
The dirroot should be always read only for apache process because otherwise built in plugin installation and uninstallation would get the nodes out of sync.<br />
<br />
===$CFG->dataroot===<br />
<br />
This '''MUST''' be a shared directory where each cluster node is accessing the files directly. It must be very reliable, administrators cannot manipulate files directly.<br />
<br />
Locking support is not required, if any code tries to use file locks in dataroot outside of cachedir or muc directory it is a bug.<br />
<br />
===$CFG->tempdir===<br />
<br />
This directory '''MUST''' be shared by all cluster nodes. Locking is required.<br />
<br />
===$CFG->cachedir===<br />
<br />
This directory '''MUST''' be shared by all cluster nodes. Locking is required.<br />
<br />
===$CFG->localcachedir===<br />
<br />
The difference from $CFG->cachedir is that the directory does not have to be shared by all cluster nodes, the file contents do not change. Use local fast filesystem on each cluster node.<br />
<br />
==Performance recommendations==<br />
<br />
#Use OPcache extension on all cluster nodes.<br />
#Set $CFG->localcachedir to fast local filesystem on each node.<br />
#Set $CFG->tempdir to fast local filesystem on each node.<br />
#Use one big central memcached server for all shared caches that support it.<br />
#Use local memcached instances on cluster nodes for local caches that support it.<br />
#Store user sessions in one shared memcached server. (See [[Session_handling]] for details)<br />
#Use fast local directory for dirroot on each cluster node.<br />
#Use dynamic cluster node management.<br />
#Use transparent proxy servers.<br />
<br />
==Upgrade procedure==<br />
<br />
#Disable load balancer.<br />
#Upgrade dirroot files on master server.<br />
#Perform CLI upgrade.<br />
#Manually reset all caches in cluster nodes - restart web server, delete localcachedir and temp directories, restart memcached, etc. Or delete all cluster nodes and create nodes from a new template.<br />
#Enable load balancer.<br />
<br />
==Step by step guide for server clustering in Moodle 2.6==<br />
From hardware and performance forum thread [https://moodle.org/mod/forum/discuss.php?d=251547 https://moodle.org/mod/forum/discuss.php?d=251547]<br />
<br />
* 1 Determine your specific need for caches. This involves the concepts of shared cache or local cache, disk based cache or server based cache or protocol specific cache like Memcached or MongoDB. <br />
** 1.1 If you have a slow shared disk, you might benefit from a local disk cache.<br />
** 1.2 If you have only a few users on your Moodle site, you might not want to set up a Memcached service, even if the acceleration that Memcached provides, is very significant for the user experience.<br />
** 1.3 If you want to do anything in your power for accelerating the site, you might consider MongoDB.<br />
* 2 Configure the caches, test them and verify them. There is room for improvement in the examples for cache configuration. This is probably the most difficult step.<br />
* 3 Install the cache support on server level. This may involve installing php modules or config for php modules.<br />
* 4 Create a cache instance in MUC here: example.com/cache/admin.php?action=addstore&plugin=memcached<br />
** 4.1 Give the cache instance some arbitrary name.<br />
** 4.2 All other fields have a question mark that can be clicked on for excellent help that tells you what to fill in, and the syntax (very important).<br />
** 4.3 The result should be a Configured Store Instance, with the name of your choice.<br />
* 5 The final step is to deploy the created cache instances by Editing Mappings for e.g. Language string cache in the list of Known cache definitions.<br />
** 5.1 While experimenting with this, I have found it a life saver to open a separate browser window, where the default setting is selected, so I just need to click on the Save button to revert the setting, just in case I lose contact with the site.<br />
** 5.2 Select the nam of your configured store instance from the dropdown list and click on the Save button. <br />
** 5.3 Test the new cache setting. If you lose contact with the site or it behaves weirdly, try using the trick in step 1. In case of emergency you might remove the cache config file (muc/config.php) in the folder pointed to by $CFG->dataroot . It seems that Moodle writes a new default config file if it is missing.<br />
<br />
==See also==<br />
*[[Performance recommendations]]<br />
*[[Caching]]<br />
*[[:dev:Server clustering improvements proposal]]<br />
*Hardware and performance forum thread [https://moodle.org/mod/forum/discuss.php?d=251547https://moodle.org/mod/forum/discuss.php?d=251547]<br />
*How to Cluster Moodle on Multiple Servers for High Availability and Scalability [http://www.severalnines.com/blog/clustering-moodle-multiple-servers-high-availability-scalability]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Gradebook&diff=115925Gradebook2014-11-12T01:44:34Z<p>Andrewnicols: Notes on tooltip removed.</p>
<hr />
<div>{{Grades}}<br />
All the grades for each student in a course can be found in the course gradebook, or 'Grader report' in ''Administration > Course administration > Grades''.<br />
<br />
The grader report collects [[Grade_items|items]] that have been graded from the various parts of Moodle that are assessed, and allows you to view and change them as well as sort them out into [[Grade_categories|categories]] and calculate totals in various ways. When you add an assessed item in a Moodle course, the gradebook automatically creates space for the grades it will produce and also adds the grades themselves as they are generated, either by the system or by you.<br />
<br />
The grades displayed are initially displayed as the raw marks from the assessments themselves, so will depend on how you set those up e.g. an essay out of 36 will appear as however many raw marks that student got, not a percentage (although this can be changed later, see below).<br />
<br />
Note that various default options for the gradebook are set at system level by the administrator and can be marked as being overridable by you, or fixed. This means that the options will not always be set up the same way for every user when they see the grader report for the first time.<br />
===Scrolling through the gradebook===<br />
<br />
{{New features}}<br />
The gradebook allows for smooth and stable scrolling horizontally and vertically through grades. It uses the whole window, making it accessible on all platforms.<br />
<br />
{|<br />
| [[File:scrolling28a.png|thumb|500px|Scrolling in all directions]]<br />
| [[File:wholewindow.png|thumb|500px|Using the whole window]]<br />
|}<br />
<br />
[[File:newgradereporta.png|thumb|600px|center]]<br />
<br />
==Display==<br />
<br />
Along the top of the grader report are several rows: first the course, then the category, then the columns for each graded activity (for example: Assignment, Quiz, Lesson). Any activities settings which were left "uncategorised" will appear in the general category which is named after the course by default (any category name can be changed).<br />
<br />
[[Image:gradereportcategories.png|center|thumb|600px]]<br />
<br />
You can add a row showing the range of possible scores by selecting 'Show ranges' in 'My report preferences.<br />
<br />
There are three ways that the categories can be displayed:<br />
<br />
* Grades only - without the category totals column<br />
* Aggregates - Category total column only<br />
* Full view - grades and the aggregates (the totals column for the category) <br />
<br />
Each section has a small icon immediately to the right of its name. Clicking this will cycle through these display modes for that category;<br />
<br />
<br />
[[Image:grades3iconsa.png|thumb|500px|center]]<br />
<br />
===Sorting by columns===<br />
<br />
You can sort by any column. Click the [[Image:iconsort.png]] symbol near the top of a column to sort by that column (1 below). This will change the symbol to a single down arrow. Clicking again will sort lowest-to-highest, changing the symbol to an up arrow. The arrows will toggle between these two states until you click on a different column.<br />
<br />
'''New in 2.8''' You can also access the [[Single view]] by clicking the pencil icon next to the arrows (2 below)<br />
<br />
[[File:iconsgrades.png]]<br />
<br />
You can sort the students by clicking the arrow (1 below), access individual [[User report| user reports]] by clicking the icon (2 below) and similar to the grade items you can access [[Single view]] by clicking the pencil icon (3 below)<br />
<br />
[[File:studentsort.png]]<br />
<br />
===Highlighting scores that are either adequate or unacceptable in red and green===<br />
<br />
Turn editing on and click on the edit icon in the controls cell at the top of the column. You can then (maybe need to 'Show more') see the option to enter a 'grade to pass'. Once set, any grades falling above this will be highlighted in green and any below will be highlighted in red.<br />
<br />
Note that the highlighting will not show if the Grader report is viewed in the editing mode.<br />
<br />
<br />
===Searching and filtering the gradebook===<br />
If you change the course settings Group mode to Visible groups or Separate groups a drop-down menu will appear in the gradebook to allow you to filter your students by groups.<br />
<br />
<br />
It is also possible to search students by first name and last name:<br />
<br />
[[File:gradebooksearch.png]]<br />
<br />
==Editing==<br />
<br />
Note: Editing anything in the gradebook refers to editing the grades '''only''' and none of the available operations bear any relationship to editing the main course page i.e. the appearance of your course page cannot be influenced by anything you do in the gradebook. The "Turn editing on" button functions separately from the main course one, so editing can be on in the gradebook, but simultaneously off when you switch back to course view. This is because editing grades and editing the course page are separate capabilities. Roles such as 'non-editing teacher' may only have one or the other.<br />
<br />
===Altering the grades===<br />
You can click "Turn editing on" at the top right to show an edit icon next to each grade. Clicking on the icon will bring up the editing screen for that grade which will allow you to set the grade, its written feedback and a number of other attributes.<br />
<br />
Alternatively, you can choose "Quick grading" and "Quick feedback" in 'My preferences' to make the report appear with editable boxes containing each grade, so you can change many at once. This capability can be a real time saver, but make sure to save at reasonable intervals lest a pageful of changes be lost.<br />
<br />
Note: If you make changes here, they are then shown highlighted to indicate grades which have been manually changed.<br />
<br />
===Hiding columns or individual grades===<br />
Turning on editing then clicking the "Show show/hide icons" link will give you the familiar show/hide eye icon next to each grade and at the top of each column. For more information, read about [[Grade_hiding|grade hiding]].<br />
<br />
===Recalculating===<br />
If you change any part of an assessment e.g. alter the maximum grade for one of the questions in a quiz, you may find that the columns do not yet reflect the change you have made. Click '''Turn editing on''' twice to force the gradebook to re-check.<br />
<br />
==Gradebook capabilities==<br />
<br />
There is just one gradebook capability, [[Capabilities/gradereport/grader:view|View the grader report]], which is allowed for the default roles of manager, teacher and non-editing teacher.<br />
<br />
==Extending the gradebook==<br />
<br />
The Gradebook can be extended in three main ways:<br />
<br />
===1. [https://moodle.org/plugins/browse.php?list=category&id=9 Grade reports]===<br />
Which are the main way to view and manipulate grades<br />
* [[LAEGrader report|LAE Grader Report]] Alternative to Grader report that scrolls vertically and horizontally without losing student columns or grade item header rows. Lot of additional enhancements.<br />
* [https://moodle.org/plugins/view.php?plugin=gradereport_updfgrades Upload PDF Grader Report] lets you view assignment grades, comments and lateness in a report<br />
<br />
===2. Grade import plugins===<br />
Which allow data to be imported from external sources<br />
<br />
===3. [https://moodle.org/plugins/browse.php?list=category&id=10 Grade export plugins]:===<br />
Which allow you to export grade data for other systems<br />
* [https://moodle.org/plugins/view.php?plugin=gradeexport_pdf PDF document] This is moodle plugin for exporting grades in PDF format. It is developed by using "Moodle PDF library"<br />
* [https://moodle.org/plugins/view.php?plugin=gradeexport_checklist Checklist] This is a grade export plugin which will create an Excel spreadsheet containing all the checkmarks from a single checklist.<br />
<br />
<br />
==See also==<br />
<br />
* Join the discussions about gradebook plugins in the [https://moodle.org/mod/forum/view.php?id=2122 Gradebook forum].<br />
* [http://youtu.be/YkuK2w1lJnk Gradebook 2.8 screencast] from Moodle HQ<br />
*[http://youtu.be/ShGv7lwMfm0 Greg Egan - Introduction to grading and the gradebook - iMoot 2013] Excellent clarification of Grader Report, Categories and Items, Scales, Aggregation (especially helpful for accurate overall grades!), Letter Grades, and Rubrics. Outcomes are not covered.<br />
<br />
[[de:Bewertungen]]<br />
[[es:Libro de calificaciones]]<br />
[[fr:Carnet de notes]]<br />
[[ja:評定表]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113032Caching2014-06-10T06:44:36Z<p>Andrewnicols: Grammatical and typo fixups</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. It is useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve your sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mongodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column select the link for '''Edit mappings'''.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Scenario:<br />
You have have a web server that has a Moodle site as well as other sites.<br />
You also have a Memcache server that is used by several sites including Moodle.<br />
<br />
Memcache has a limited size cache, that when full and requested to store more information frees space by dropping the least used cache entries.<br />
<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
<br />
Solution:<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
<br />
Explanation:<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step-by-step guide to this.<br />
<br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered are some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br />
<br />
If you are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113031Caching2014-06-10T06:40:47Z<p>Andrewnicols: Clarify example</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. It is useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve your sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mongodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column select the link for '''Edit mappings'''.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Scenario:<br />
You have have a web server that has a Moodle site as well as other sites.<br />
You also have a Memcache server that is used by several sites including Moodle.<br />
<br />
Memcache has a limited size cache, that when full and requested to store more information frees space by dropping the least used cache entries.<br />
<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
<br />
Solution:<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
<br />
Explanation:<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113030Caching2014-06-10T06:32:53Z<p>Andrewnicols: Grammatical fixups</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. It is useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve your sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mongodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column select the link for '''Edit mappings'''.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Imagine if you will that you have have a web server that has a Moodle site as well as other sites. You also have a Memcache server that is used by several sites including Moodle.<br /><br />
Memcache is a limited size cache, that when full and requested to store more information frees space by dropping least used cache entries.<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113028Caching2014-06-10T06:29:24Z<p>Andrewnicols: Grammatical and typo fixups</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. It is useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve your sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mongodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column you should see a link to'''Edit mappings''', click this.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Imagine if you will that you have have a web server that has a Moodle site as well as other sites. You also have a Memcache server that is used by several sites including Moodle.<br /><br />
Memcache is a limited size cache, that when full and requested to store more information frees space by dropping least used cache entries.<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113027Caching2014-06-10T05:47:44Z<p>Andrewnicols: Grammatical fixups</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. It is useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve you sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mogodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column you should see a link to'''Edit mappings''', click this.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Imagine if you will that you have have a web server that has a Moodle site as well as other sites. You also have a Memcache server that is used by several sites including Moodle.<br /><br />
Memcache is a limited size cache, that when full and requested to store more information frees space by dropping least used cache entries.<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113026Caching2014-06-10T05:36:40Z<p>Andrewnicols: Grammatical fixups</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it. A cache by nature is volatile and any information that is absolutely mission critical should be a more permanent data store likely the database.<br />
<br />
Nonetheless there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. Its useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve you sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mogodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column you should see a link to'''Edit mappings''', click this.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Imagine if you will that you have have a web server that has a Moodle site as well as other sites. You also have a Memcache server that is used by several sites including Moodle.<br /><br />
Memcache is a limited size cache, that when full and requested to store more information frees space by dropping least used cache entries.<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Caching&diff=113025Caching2014-06-10T05:26:30Z<p>Andrewnicols: missing word</p>
<hr />
<div>{{Performance}}<br />
<br />
A cache is a collection of processed data that is kept on hand and re-used in order to avoid costly repeated database queries.<br />
<br />
Moodle 2.4 saw the implementation of MUC, the Moodle Universal Cache. This new system allows certain functions of Moodle (eg string fetching) take advantage of different installed cache services (eg files, ram, memcached).<br />
<br />
In future versions of Moodle we will continue expanding the number of Moodle functions that use MUC, which will continue improving performance, but you can already start using it to improve your site.<br />
<br />
==General approach to performance testing==<br />
<br />
Here is the general strategy you should be taking:<br />
<br />
# Build a test environment that is as close to your real production instance as possible (eg hardware, software, networking, etc)<br />
# Make sure to remove as many uncontrolled variables as you can from this environment (eg other services)<br />
# Use a tool to place a realistic, but simulated and repeatable load upon you server. (eg jmeter or selenium).<br />
# Decide on a way to measure performance of the server by capturing data (ram, load, time taken, etc)<br />
# Run your load and measure a baseline performance result.<br />
# Change one variable at a time, and re-run the load to see if performance gets better or worse. Repeat as necessary.<br />
# When you discover settings that result in a consistent performance improvement, apply to your production site.<br />
<br />
==Cache configuration in Moodle==<br />
<br />
Since Moodle 2.4, Moodle has provided a caching plugin framework to give administrators the ability to control where Moodle stores cached data. For most Moodle sites the default configuration should be sufficient and it is not necessary to change the configuration. For larger Moodle sites with multiple servers, administrators may wish to use memcached, mongodb or other systems to store cache data. The cache plugin screen provides administrators with the ability to configure what cache data is stored where.<br /><br />
Caching in Moodle is controlled by what is known as the Moodle Universal Cache. Commonly referred to MUC.<br />
<br />
This document explains briefly what MUC is before proceeding into detail about the concepts and configuration options it offers.<br />
<br />
==The basic cache concepts in Moodle==<br />
Caching in Moodle isn't as complex as it first appears. A little background knowledge will go a long way in understanding how cache configuration works.<br />
<br />
===Cache types===<br />
Lets start with cache types (sometimes referred to as mode). There are three basic types of caches in Moodle.<br />
<br />
The first is the application cache. This is by far the most commonly used cache type in code. Its information is shared by all users and its data persists between requests. Information stored here is usually cached for one of two reasons, either it is required information for the majority of requests and saves us a one of more database interactions or it is information that is accessed less frequently but is resource intensive to generate.<br /><br />
By default this information is stored in an organised structure within your Moodle data directory.<br />
<br />
The second cache type is the session cache. This is just like the PHP session that you will already be familiar with, in fact it uses the PHP session by default. You may be wondering why we have this cache type at all, but the answer is simple. MUC provides a managed means of storing, and removing information that is required between requests. It offers developers a framework to use rather than having to re-invent the wheel and ensures that we have access to a controlled means of managing the cache as required.<br /><br />
Its important to note that this isn't a frequently used cache type as by default session cache data is stored in the PHP session and the PHP session is stored in the database. Uses of the session cache type are limited to small datasets as we don't want to bloat sessions and thus put strain on the database.<br />
<br />
The third and final type is the request cache. Data stored in this cache type only persists for the lifetime of the request. If you're a PHP developer think of it like a managed static variable.<br /><br />
This is by far the least used of the three cache types, uses are often limited to information that will be accessed several times within the same request, usually by more than area of code.<br />
Cached information is stored in memory by default.<br />
<br />
==== Cache types and multiple-server systems ====<br />
<br />
If you have a system with multiple front-end web servers, the application cache must be shared between the servers. In other words, you cannot use fast local storage for the application cache, but must use shared storage or some other form of shared cache such as a shared memcache.<br />
<br />
The same applies to session cache, unless you use a 'sticky sessions' mechanism to ensure that within a session, users always access the same front-end server.<br />
<br />
===Cache backends===<br />
Cache backends are where data actually gets stored. These include things like the file system, php session, Memcached, and memory.<br /><br />
By default just file system, php session, and memory are used within Moodle.<br /><br />
We don't require that a site has access to any other systems such a Memcached. Instead that is something you are responsible for installing and configuring yourself.<br /><br />
When cache backends are mentioned think of systems outside of Moodle that can be used to store data. The MongoDB server, the Memcache server and similiar "server" applications.<br />
<br />
===Cache stores===<br />
Cache stores are a plugin type within Moodle. They facilitate connecting Moodle to the cache backends discussed above.<br /><br />
Moodle ships with the three defaults mentioned above as well as Memcache, Memcached, and MongoDB.<br /><br />
You can find other cache store plugins in the [https://moodle.org/plugins/browse.php?list=category&id=48 plugins database].<br /><br />
The code for these is located within cache/stores in your Moodle directory root.<br />
<br />
Within Moodle you can configure as many cache stores as your architecture requires. If you have several Memcache servers for instance you can create an cache store instance for each.<br /><br />
Moodle by default contains three cache store instances that gets when you've made no other configuration.<br />
* A file store instance is created which gets used for all application caches. It stores its data in your moodledata directory.<br />
* A session store instance is created which gets used for all session caches. It stores its data in the PHP session, which by default is stored if your database.<br />
* A static memory store instance is created which gets used for all request cache types. Data exists in memory for just the lifetime of a request.<br />
<br />
===Caches: what happens in code===<br />
Caches are created in code and are used by the developer to store data they see a need to cache.<br /><br />
Lets keep this section nice and short because perhaps you are not a developer. There is one very important point you must know about.<br /><br />
The developer does not get any say in where the data gets cached. They must specify the following information when creating a cache to use.<br />
# The type of cache they require.<br />
# The area of code this cache will belong to (the API if you will).<br />
# The name of the cache, something they make up to describe in one word what the cache stores.<br />
<br />
There are several optional requirements and settings they can specify as well, but don't worry about that at this point.<br /><br />
The important point is that they can't choose which cache backend to use, they can only choose the type of cache they want from the three detailed above.<br />
<br />
===How it ties together===<br />
<br />
This is best described in relation to roles played in an organisation.<br />
<br />
# The system administrator installs the cache backends you wish to use. Memcache, XCache, APC and so on.<br />Moodle doesn't know about these, they are outside of Moodle's scope and purely the responsibility of your system administrator.<br />
# The Moodle administrator creates a cache store instance in Moodle for each backend the site will make use of.<br />There can be one or more cache stores instances for each backend. Some backends like Memcached have settings to create separated spaces within one backend.<br />
# The developer has created caches in code and is using them to store data.<br />He doesn't know anything about how you will use your caches, he just creates a "cache" and tells Moodle what type it is best for it.<br />
# The Moodle administrator creates a mapping between a cache store instance and a cache.<br />That mapping tells Moodle to use the backend you specify to store the data the developer wants cached.<br />
<br />
In addition to that you can take things further still.<br />
* You can map many caches to a single cache store instance.<br />
* You can map multiple cache store instances to a single cache with priority (primary ... final)<br />
* You can map a cache store instance to be the default store used for all caches of a specific type that don't otherwise have specific mappings.<br />
<br />
If this is the first time you are reading about the Moodle Universal Cache this probably sounds pretty complex but don't worry it will be discussed in better detail as we work through how to configure the caching in Moodle.<br />
<br />
==Advanced concepts==<br />
These concepts are things that most sites will not need to know or concern themselves about.<br />
<br />
You should only start looking here if you are looking to maximise performance on large sites running over clustered services with shared cache backends, or on multi-site architecure again where information is being shared between sites.<br />
<br />
===Locking===<br />
<br />
The idea of locking is nothing new, it is the process of controlling access in order to avoid concurrency issues.<br />
<br />
MUC has a second type of plugin, a cache lock plugin that gets used when caches require it. To date no caches have required it, a cache by nature is volatlie and any information that is absolutely mission critical should be a more permanent data store likely the database.<br /><br />
None the less there is a locking system that cache definitions can require within their options and that will be applied when interacting with a cache store instance.<br />
<br />
===Sharing===<br />
<br />
Every bit of data that gets stored within a cache has a calculated unique key associated with it.<br /><br />
By default part of that key is the site identifier making any content stored in the cache specific to the site that stored it. For most sites this is exactly what you want.<br /><br />
However in some situations its beneficial to allow mutiple sites, or somehow linked sites to share cached data.<br /><br />
Of course not all caches can be shared, however some certainly can and by sharing you can further reduce load and increase performance by maximising resource use.<br />
<br />
This is an advanced feature, if you choose to configure sharing please do so carefully.<br />
<br />
To make use of sharing you need to first configure identical cache store instances in the sites you want to share information, and then on each site set the sharing for the cache to the same value.<br />
<br />
; Sites with the same site ID : This is the default, it allows for sites with the same site ID to share cached information. It is the most restrictive but is going to work for all caches. All other options carry an element of risk in that you have to ensure the information in the cache is applicable to all sites that will be accessing it.<br />
; Sites running the same version : All sites accessing the backend that have the same Moodle version can share the information this cache has stored in the cache store.<br />
; Custom key : For this you manually enter a key to use for sharing. You must then enter the exact same key into the other sites you want to share information.<br />
; Everyone : The cached data is accessible to all other sites accessing the data. This option puts the ball entirely in the Moodle administrators court.<br />
<br />
As an example if you had several Moodle sites all the same version running on server with APC installed you could decide to map the language cache to the APC store and configure sharing for all sites running the same version.<br /><br />
The language cache for sites on the same version is safe to share, it is used on practically every page, and APC is extremely fast. These three points may result in a nice wee performance boost for your sites.<br />
<br />
==The cache configuration screen==<br />
<br />
The cache configuration screen is your one stop shop for configuring caching in Moodle.<br /><br />
It gives you an overview of how caching is currently configured for your site and it provides links to all of the actions you can perform to configure caching to your specific needs.<br />
<br />
===Accessing the cache configuration screen===<br />
<br />
[[Image:caching-27-01-configuration-screen.png|thumb|500px|The cache configuration screen in Moodle 2.6]]<br />
<br />
The cache configuration screen can only be accessed by users with the ''moodle/site:config'' capability. By default this is only admins.<br /><br />
Once logged in the configuration screen can be found in the Settings block under '''Site Administration > Plugins > Caching > Configuration'''.<br />
<br />
===Installed cache stores===<br />
<br />
[[Image:caching-27-02-installed-cache-stores.png|thumb|500px|Installed cache stores screenshot]]<br />
<br />
This is showing you a list of cache store plugins that you have installed.<br /><br />
For each plugin you can quickly see whether it is ready to be used (any php requirements have been met), how many store instances already exist on this site, the cache types that this store can be used for, what features it supports (advanced) and any actions you can perform relating to this store.<br />
<br />
Often the only action available is to create a new store instance.<br /><br />
Most stores support having multiple instances, however not all as you will see that that the session cache and static request cache do not. For those two stores it does not make sense to have multiple instances.<br />
<br />
===Configured store instances===<br />
<br />
[[Image:caching-27-03-configured-store-instances.png|thumb|500px|Configured store instances screenshot]]<br />
<br />
Here you get a list of the cache store instances on this site.<br />
<br />
; '''Store name''' : The name given to this cache store instance when it is created so that you can recognise it. It can be anything you want and is only used so that you can identify the store instance.<br />
; '''Plugin''' : The cache store plugin of which this is an instance of.<br />
; '''Ready''' : A tick gets shown when all PHP requirements have been met as well as any connection or set-up requirements have been verified.<br />
; '''Store mappings''' : The number of caches this store instance has been mapped to explicitly. Does not include any uses through default mappings (discussed below).<br />
; '''Modes''' : The modes that this cache store instance can serve.<br />
; '''Supports''' : ''Advanced''. The features supported by this cache store instance.<br />
; '''Locking mechanism''' : ''Advanced''. The locking mechanism this cache store instance will make use of. We've not discussed this yet, but read on and you'll find information on it.<br />
; '''Actions''' : Any actions that can be performed against this cache store instance.<br />
<br />
===Known cache definitions===<br />
<br />
[[Image:caching-27-04-known-cache-definitions.png|thumb|500px|Known cache definitions screenshot]]<br />
<br />
The idea of a cache definition hasn't been discussed here yet. It is something controlled by the developer. When they create a cache they can do so in two ways, the first is by creating a cache definition. This is essentially telling Moodle about the cache they've created. The second way is to create an Adhoc cache. Developers are always encouraged to use the first method. Only caches with a definition can be mapped and further configured by the admin. Adhoc caches will make use of default settings only.<br /><br />
Typically Adhoc caches are only permitted in situations where the cache is small and configuring it beyond defaults would provide no benefit to administrators. Really its like saying you the administrator doesn't need to concern yourself with it.<br />
<br />
For each cache shown here you get the following information:<br />
<br />
; '''Definition''' : A concise description of this cache.<br />
; '''Mode''' : The cache type this cache is designed for.<br />
; '''Component''' : The code component the cache is associated with.<br />
; '''Area''' : The area of code this cache is serving within the component.<br />
; '''Store mappings''' : The store or stores that will be used for this cache.<br />
; '''Sharing''' : How is sharing configured for this site.<br />
; '''Actions''' : Any actions that can be performed on the cache. Typically you can edit the cache store instance mappings, edit sharing, and purge the cache.<br />
<br />
You'll also find at the bottom of this table a link title "Rescan definitions". Clicking this link will cause Moodle to go off an check all core components, and installed plugins looking for changes in the cache definitions.<br /><br />
This happens by default during upgrade, and if a new cache definition is encountered. However should you find yourself looking for a cache that isn't there this may be worth a try.<br /><br />
It is also handy for developers as it allows them to quickly apply changes when working with caches. Its useful when tweaking cache definitions to find what works best.<br />
<br />
Information on specific cache definitions can be found on the [[Cache definitions]] page.<br />
<br />
===Summary of cache lock instances===<br />
<br />
[[Image:caching-27-05-summary-of-cache-lock-instances.png|thumb|500px|Summary of cache lock instances screenshot]]<br />
<br />
As mentioned above cache locking is an advanced concept in MUC.<br /><br />
The table here shows information on the configured locking mechanisms available to MUC. By default just a single locking mechanism is available, file locking.<br />
At present there are no caches that make use of this and as such I won't discuss it further here.<br />
<br />
===Stores used when no mapping is present===<br />
<br />
[[Image:caching-27-06-stores-used-when-no-mapping-is-present.png|thumb|500px|Mapping of default stores screenshot]]<br />
<br />
This table quickly shows which cache store instances are going to be used for cache types if there are specific mappings in place.<br /><br />
Of simply this shows the default cache store instances.<br />
<br />
At the bottom you will notice there is a link "Edit mappings" that takes you to a page where you can configure this.<br />
<br />
==Adding cache store instances==<br />
<br />
The default configuration is going to work for all sites, however you may be able to improve you sites performance by making use of various caching backends and techniques. The first thing you are going to want to do is add cache store instances configured to connect to/use the cache backends you've set up.<br />
<br />
===File cache===<br />
<br />
[[Image:caching-27-07-add-file-cache-store.png|thumb|300px|Adding a file cache store screenshot]]<br />
<br />
When on the cache configuration screen within the ''Installed cache stores'' table you should be able to see the File cache plugin, click ''`Add instance`'' to start the process of adding a file cache store instance.<br />
<br />
When creating a file cache there is in fact only one required param, the store name. The store name is used to identify the file store instance in the configuration interface and must be unique to the site.<br />
It can be anything you want, but we would advice making it something that describes you intended use of the file store.<br />
<br />
The following properties can also be specified, customising where the file cache will be located, and how it operates.<br />
<br />
; '''Cache path''' : Allows you to specify a directory to use when storing cache data on the file system. Of course the user the webserver is running as must have read/write access to this directory. By default (blank) the Moodledata directory will be used.<br />
; '''Auto create directory''' : If enabled when the cache is initialised if the specified directory does not exist Moodle will create it. If this is specified and the directory does not exist the the cache will be deemed not ready and will not be used.<br />
; '''Single directory store''' : By default the file store will create a subdirectory structure to store data in. The first 3 characters of the data key will be used as a directory. This is useful in avoiding file system limits it the file system has a maximum number of files per directory. By enabling this option the file cache will not use subdirectories for storage of data. This leads to a flat structure but one that is more likely hit file system limits. Use with care.<br />
; '''Prescan directory''' : One of the features the file cache provides is to prescan the storage directory when the cache is first used. This leads to faster checks of files at the expense of an in-depth read.<br />
<br />
The file cache store is the default store used for application caches and by default the moodledata directory gets used for the cache.<br />
File access can be a taxing resource in times of increased load and the following are some ideas about configuring alternative file stores in order to improve performance.<br />
<br />
First up is there a faster file system available for use on your server.<br /><br />
Perhaps you've an SSD installed but are not using it for your moodledata directory because space is a premium.<br /><br />
You should consider creating a directory or small partition on your SSD and creating a file store to use that instead of your Moodle data directory.<br />
<br />
Next you've not got a faster drive available for use, but you do have plenty of free space.<br /><br />
Something that may be worth giving a shot would be to create a small partition on the drive you've got installed that uses a performance orientated file system.<br /><br />
Many linux installations these days for example use EXT4, a nice file system but one that has overheads due to the likes of journalling.<br /><br />
Creating a partition and using a file system that has been optimised for performance may give you that little boost you are looking for. Remember caches are designed to be volatile and choosing a file system for a cache is different decision to choosing a file system for your server.<br /><br />
<br />
Finally if you're ready to go to lengths and have an abundance of memory on your server you could consider creating a ramdisk/tmpfs and pointing a file store at that.<br />
Purely based in memory, it is volatile exactly like the cache is, and file system performance just isn't going to get much better than this.<br /><br />
Of course you will be limited in space and you are essentially taking that resource away from your server.<br />
<br />
Please remember with all of these ideas that they are just ideas.<br /><br />
What ever you choose - test, test, test, be sure of the decision you make.<br />
<br />
===Memcache===<br />
<br />
[[Image:caching-27-08-add-memcache-store.png|thumb|300px|Add Memcache store screenshot]]<br />
<br />
Before you can add a Memcache store instance you must first have a Memcached server you can access and have the Memcache php extension installed and enabled on your web server.<br />
<br />
Like the file store you must provide a the store name. It is used to identify the store instance in the configuration interface and must be unique to the site.<br /><br />
For a Memcache store you must also enter the Memcache server, or servers you wish it to make use of.<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
Optionally you can also specify a key prefix to use. What you enter here will prefixed to all keys before accessing the server and can be used to effectively partition the Memcache space in a recognisable way. This can be handy if you have a management tool for you Memcached server that you use to inspect what is stored there.<br />
<br />
===Memcached===<br />
<br />
[[Image:caching-27-09-add-memcached-store.png|thumb|300px|Add Memcached store screenshot]]<br />
<br />
Like the Memcache store you must first have a Memcached server you can access and have the Memcached php extension installed and enabled on your server.<br />
<br />
Also like the Memcache store there are two required parameters in configuring a Memcached store.<br />
<br />
; '''Store name''' : It is used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Servers''' : The servers you wish this cache store use. See below for details.<br />
<br />
Servers should be added one per line and each line can contain 1 to 3 properties separated by colons.<br />
# The URL or IP address of the server (required)<br />
# The port the server is listening on (optional)<br />
# The weight to give this server (optional)<br />
<br />
For example, if you had two Memcached instances running on your server, one configured for the default port, and one configured for 11212 you would use the following:<br />
<code><br />
127.0.0.1<br />
127.0.0.1:11212<br />
</code><br />
<br />
There are also several optional parameters you can set when creating a Memcached store.<br />
<br />
; '''Use compression''' : Defaults to true, but can be disabled if you wish.<br />
; '''Use serialiser''' : Allows your to select which serialiser gets used when communicating with the Memcache server. By default the Memcached extension and PHP only provide one serialised, however there is a couple of others available for installation if you go looking for them. One for example is the igbinary found at https://github.com/igbinary/igbinary.<br />
; '''Prefix key''' : Allows you to set some characters that will be prefixed to all keys before interacting with the server.<br />
; '''Hash method''' : The hash method provided by the Memcached extension is used by default here. However you can select to use an alternative if you wish. http://www.php.net/manual/en/memcached.constants.php provides a little information on the options available. Please note if you wish to you can also override the default hash function PHP uses within your php.ini.<br />
; '''Buffer writes''' : Disabled by default, and for good reason. Turning on buffered writes will minimise interaction with the Memcached server by buffering io operations. The downside to this is that on a system with any concurrency there is a good chance multiple requests will end up generating the data because no one had pushed it to the Memcached server when they first requested it. Enabling this can be advantageous for caches that are only accessed in capability controlled areas for example where multiple interaction is taking a toll on network resources or such. But that is definitely on the extreme tweaking end of the scale.<br />
<br />
===MongoDB===<br />
<br />
[[Image:caching-27-10-add-mongodb-store.png|thumb|300px|Add MongoDB store screenshot]]<br />
<br />
MongoDB is an open source document orientated NoSQL database. Check out their website www.mogodb.org for more information.<br />
<br />
; '''Store name''' : Used to identify the store instance in the configuration interface and must be unique to the site.<br />
; '''Server''' : This is the connection string for the server you want to use. Multiple servers can be specified using a comma-separated list.<br />
; '''Database''' : The name of the database to make use of.<br />
; '''Username''' : The username to use when making a connection.<br />
; '''Password''' : The password of the user being used for the connection.<br />
; '''Replica set''' : The name of the replica set to connect to. If this is given the master will be determined by using the ismaster database command on the seeds, so the driver may end up connecting to a server that was not even listed.<br />
; '''Use''' : If enabled the usesafe option will be used during insert, get, and remove operations. If you've specified a replica set this will be forced on anyway.<br />
; '''Use safe value''' : You can choose to provide a specific value for use safe. This will determine the number of servers that operations must be completed on before they are deemed to have been completed.<br />
; '''Use extended keys''' : If enabled full key sets will be used when working with the plugin. This isn't used internally yet but would allow you to easily search and investigate the MongoDB plugin manually if you so choose. Turning this on will add an small overhead so should only be done if you require it.<br />
<br />
==Mapping a cache to a store instance==<br />
<br />
[[Image:caching-27-11-store-mapping.png|thumb|300px|Cache definition store mapping screenshot]]<br />
<br />
Mapping a store instance to a cache tells Moodle to use that store instance when the cache is interacted with. This allows the Moodle administrator to control where information gets stored and to most importantly optimise performance of your site by making the most of the resources available to your site.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Known cache definitions'' table and within it find the cache you'd like to map.<br />
In the actions column you should see a link to'''Edit mappings''', click this.<br />
<br />
The screen you are presented allows you to map one or more cache store instances to be used by this cache.<br />
<br />
If no stores are mapped for the cache then the default stores are used. Have a look at the section below for information on changing the default stores.<br />
<br />
If a single store instance is mapped to the cache the following occurs:<br />
; ''Getting data from the cache'' : Moodle asks the cache to get the data. The cache attempts to get it from the store. If the store has it it gives it to the cache, and the cache gives it to Moodle so that it can use the data. If the store doesn't have it the a fail is returned and Moodle will have to generate the data and will most likely then send it to the cache.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, and the cache will give it to the cache store.<br />
<br />
If multiple store instances are mapped to the cache the following occurs:<br />
; ''Getting data from a store'' : Moodle asks the cache to get the data. The cache attempts to get it from the first store. If the first store has it then it returns the data to the cache and the cache returns it to Moodle. If the first store doesn't have the data then it attempts to get the data from the second store. If the second store has it it returns it to the first store that then stores it itself before returning it to the cache. If it doesn't then the next store is used. This continue until either the data is found or there are no more stores to check.<br />
; ''Storing data in the cache'' : Moodle will ask the cache to store some data, the cache will give it to every mapped cache store for storage.<br />
<br />
The main advantage to assigning multiple stores is that you can introduce cache redundancy. Of course this introduces an overhead so it should only be used when actually required.<br />
The following is an example of when mapping multiple stores can provide an advantage:<br />
<pre><br />
Imagine if you will that you have have a web server that has a Moodle site as well as other sites. You also have a Memcache server that is used by several sites including Moodle.<br /><br />
Memcache is a limited size cache, that when full and requested to store more information frees space by dropping least used cache entries.<br />
You want to use Memcache for your Moodle site because it is fast, however you are aware that it may introduce more cache misses because it is a heavily used Memcache server.<br />
To get around this you map two stores to caches you wish to use Memcache.<br />
You make Memcache the primary store, and you make the default file store the final cache store.<br />
By doing this you've created redundancy, when something is requested Moodle first tries to get it from Memcache (the fastest store) and if its not there it proceeds to check the file cache.<br />
</pre><br />
<br />
Just a couple more points of interest:<br />
<br />
* Mapping multiple caches will introduce overhead, the more caches mapped the more overhead.<br />
* Consider the cache stores you are mapping to, if data remains there once set then there is no point mapping any further stores after it. This technique is primarily valuable in situations where data is not guaranteed to remain after being set.<br />
* Always test your configuration. Enable the display of performance information and then watch which stores get used when interacting with Moodle in such a way as to trigger the cache.<br />
<br />
==Setting the stores that get used when no mapping is present==<br />
<br />
[[Image:caching-27-12-default-store-mapping.png|thumb|300px|Setting which stores get used when no mapping is present screenshot]]<br />
<br />
This is really setting the default stores that get used for a cache type when there is not a specific mapping that has been made for it.<br />
<br />
To set a mapping first browse to the cache configuration screen.<br /><br />
Proceed to find the ''Stores used when no mapping is present'' table.<br /><br />
After the table you will find a link '''Edit mappings''', click this.<br />
<br />
On the screen you are presented with you can select one store for each cache type to use when a cache of the corresponding type gets initialised and there is not an explicit mapping for it.<br />
<br />
Note that on this interface the drop downs only contain store instances that are suitable for mapping to the type.<br />
Not all instances will necessarily be shown. If you have a store instance you don't see then it is not suitable for '''ALL''' the cache definitions that exist.<br /><br />
You will not be able to make that store instance the default, you will instead need to map it explicitly to each cache you want/can use it for.<br />
<br />
==Configuring caching for your site==<br />
<br />
This is where it really gets tricky, and unfortunately there is no step by step guide to this.<br /><br />
How caching can be best configured for a site comes down entirely to the site in question and the resources available to it.<br />
<br />
What can be offered is some tips and tricks to get you thinking about things and to perhaps introduce ideas that will help you along the way.<br /><br />
If yu are reading this document and you've learnt a thing or two about configuring caching on your site share your learnings by adding to the points here.<br />
<br />
* Plan it. It's a complex thing. Understand your site, understand your system, and really think how users will be using it all.<br />
* If you've got a small site the gains aren't likely to be significant, if you've got a large site getting this right can lead to a substantial boost in performance.<br />
* When looking at cache backends really research the advantages and disadvantages of each. Keep your site in mind when thinking about them. Depending upon your site you may find that no one cache backend is going to meet the entire needs of your site and that you will benefit from having a couple of backends at your disposal.<br />
* Things aren't usually as simple as installing a cache backend and then using it. Pay attention to configuration and try to optimise it for your system. Test it separately and have an understanding of its performance before tell Moodle about it. The cache allows you to shift load off the database and reduce page request processing.<br />If for instance you have Memcache installed but your connection has not been optimised for it you may well find yourself in a losing situation before you even tell Moodle about the Memcache server.<br />
* When considering your default store instances keep in mind that they must operate with data sets of varying sizes and frequency. For a large site really your best bet is to look at each cache definition and map it to a store that is best suited for the data it includes and the frequency of access.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Again when mapping store instances to caches really think about the cache you are mapping and make a decision based upon what you understand of your site and what you know about the cache.<br />Cache definitions have been documented [[Cache definitions]].<br />
* Test your configuration. If you can stress test it even better! If you turn on performance information Moodle will also print cache access information at the bottom of the screen. You can use this to visually check the cache is being used as you expect, and it will give you an indication of where misses etc are occurring.<br />
* Keep an eye on your backend. Moodle doesn't provide a means of monitoring a cache backend and that is certainly something you should keep an eye on. Memcache for instance drops least used data when full to make room for new entries. APC on the other hand stops accepting data when full. Both will impact your performance if full and you're going to encounter misses. However APC when full is horrible, but it is much faster.<br />
<br />
==Other performance testing==<br />
<br />
Two links that might be useful to anyone considering testing performance on their own servers:<br />
<br />
* [http://www.iteachwithmoodle.com/2012/10/12/moodle-performance-testing-how-much-more-horsepower-do-each-new-versions-of-moodle-require/ Moodle performance testing: how much more horsepower do each new versions of Moodle require?]<br />
* [http://www.iteachwithmoodle.com/2012/10/11/how-to-stress-test-your-moodle-server-using-loadstorm/ How to load test your Moodle server using Loadstorm]<br />
<br />
==Other performance advice for load-balanced web servers==<br />
<br />
# In Moodle 2.4 onwards with load-balanced web servers, don't use the default caching option that stores the data in moodledata on a shared network drive. Use memcached instead. See Tim Hunt's article on http://tjhunt.blogspot.de/2013/05/performance-testing-moodle.html<br />
# In Moodle 2.6 onwards make sure you set $CFG->localcachedir to some local directory in config.php (for each node). This will speed up some of the disk caching that happens outside of MUC, such as themes, javascript, libraries etc.<br />
<br />
==More information==<br />
* [[Cache definitions]] Information on the cache definitions found within Moodle.<br />
* [[:dev:Cache API|Cache API]] Details of the Cache API.<br />
* [[:dev:Cache API - Quick reference|Cache API - Quick reference]] A short, code focused page of on the Cache API.<br />
* [[:dev:The Moodle Universal Cache (MUC)|The Moodle Universal Cache (MUC)]] The original cache specification.<br />
<br />
<br />
Cache related forum discussions that may help in understanding MUC:<br />
* [https://moodle.org/mod/forum/discuss.php?d=217195 MUC is here, now what?] <br />
* [https://moodle.org/mod/forum/discuss.php?d=226123 Status of MUC?]<br />
* [https://moodle.org/mod/forum/discuss.php?d=222250 Putting cachedir on local disks in cluster]<br />
* [https://moodle.org/mod/forum/discuss.php?d=232122 moodle cachestore_file]<br />
<br />
<br />
Other:<br />
* [http://jfilip.ca/2013/08/20/moodle-2-4-5-vs-2-5-1-performance-and-muc-apc-cache-store/ Moodle 2.4.5 vs. 2.5.1 performance and MUC APC cache store] blog post by Justin Filip<br />
<br />
<br />
[[de:Caching]]<br />
[[es:Cacheando]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Git_for_Administrators&diff=110503Git for Administrators2014-02-28T00:39:52Z<p>Andrewnicols: Add links to up-to-date and relevant information</p>
<hr />
<div>{{Installing Moodle}}<br />
This page describes how to maintain a copy of Moodle on your production server which can easily be upgraded using Git. If you have customisations of Moodle core code, you are advised to follow the instructions in the [[Development:Quick Git start guide for Moodle development|Quick Git start guide for Moodle development]].<br />
<br />
To get the most of of Git it is worth making the effort to understand its basic concepts - see the See also section below. It can be a bit of a learning curve, especially if you are used to CVS or Subversion. <br />
<br />
== Getting hold of Git (Windows, OSX, Linux and others) ==<br />
<br />
Support for Git was, up until recently, mostly confined to Linux but builds are now available for most popular operating systems:<br />
<br />
* List of downloads from Git site - http://git-scm.com/download<br />
<br />
Once you have downloaded and installed your OS relevant git installation, the git commands in this document should work with your operating system.<br />
<br />
== Obtaining the code from Git ==<br />
<br />
The command line version of Git is discussed here. Graphical clients are little more than wrappers around the command line version, so you should be able to deduce the correct parameters quite easily. <br />
<br />
You can find the official Moodle git repository at git://git.moodle.org/moodle.git (with an official clone at git://github.com/moodle/moodle.git). To initialize your local checkout, use<br />
<pre><br />
$ git clone git://git.moodle.org/moodle.git (1)<br />
$ cd moodle<br />
$ git branch -a (2)<br />
$ git branch --track MOODLE_26_STABLE origin/MOODLE_26_STABLE (3)<br />
$ git checkout MOODLE_26_STABLE (4)<br />
</pre><br />
* The command (1) initializes the new local repository as a clone of the 'upstream' (i.e. the remote server based) moodle.git repository. The upstream repository is called 'origin' by default. It creates a new directory named ''moodle'', where it downloads all the files. This operation can take a while as it is actually getting the entire history of all Moodle versions<br />
* The command (2) lists all available branches.<br />
* Use the command (3) to create a new local branch called MOODLE_26_STABLE and set it to track the remote branch MOODLE_26_STABLE from the upstream repository.<br />
* The command (4) actually switches to the newly created local branch. <br />
<br />
Note that Git has a huge number of options for each command and it's actually possible to do the above process with a single command (left as an exercise!!).<br />
<br />
==Git from behind a firewall==<br />
<br />
Git uses a read-only protocol that may be blocked by your firewall (port 9418). If this is a problem, you can use Github's http version <nowiki>https://github.com/moodle/moodle.git</nowiki>. It's a bit slower, so use the Git protocol if you can.<br />
<br />
== Updating your installation ==<br />
<br />
The Moodle development team performs integration and testing of fixed bugs every Monday and Tuesday. On Wednesday you can install all patches by updating your code. Check the [http://git.moodle.org/gw?p=moodle.git;a=summary shortlog] to see if the official repository has been already updated or not.<br />
<br />
To update your code to the latest version (on the MOODLE_26_STABLE branch) '''all''' you have to do is:<br />
<pre><br />
$ cd /path/to/your/moodle/<br />
$ git pull<br />
</pre><br />
If this is a production site you should still consider the [[Upgrade]] instructions (e.g. take backups).<br />
<br />
== Installing a contributed extension from its Git repository ==<br />
<br />
This is one way to handle adding plugins from other Git repositories into your Moodle repository. Another way is to use Git Submodules. However, at the time of writing, this is one of Git's rougher features and should be regarded as an advanced option. <br />
<br />
For example, let us say we want to install the [[Certificate module]] from its Git repository into our Moodle 2.6.<br />
<pre><br />
$ cd /path/to/your/moodle/<br />
$ cd mod (1)<br />
$ git clone https://github.com/markn86/moodle-mod_certificate.git certificate (2)<br />
$ cd certificate<br />
$ git checkout -b MOODLE_26_STABLE origin/MOODLE_26_STABLE (3)<br />
$ git branch -d master (4)<br />
</pre><br />
The command (1) changes the current directory into the ''mod'' folder of your local Moodle clone. The command (2) creates a new subdirectory ''certificate'' and makes a local clone of vanilla Certificate repository. The command (3) creates a new local branch that will track the remote branch with a Certificate version for Moodle 2.6. The command (4) deletes the ''master'' that was created automatically by git-clone in (2) as we do not want it in this production checkout.<br />
<br />
Note: you should check first the compatibility of a module with your Moodle branch by asking directly to the Maintainer before cloning the repo or - if you want to guess it - by issueing the command below before running the command (3), in order to verify what is available among the branches:<br />
<pre><br />
$ git branch -a<br />
* master<br />
remotes/origin/HEAD -> origin/master<br />
remotes/origin/MOODLE_20_STABLE<br />
remotes/origin/MOODLE_21_STABLE<br />
remotes/origin/MOODLE_22_STABLE<br />
remotes/origin/MOODLE_23_STABLE<br />
remotes/origin/MOODLE_24_STABLE<br />
remotes/origin/MOODLE_25_STABLE<br />
remotes/origin/MOODLE_26_STABLE<br />
remotes/origin/master<br />
</pre><br />
This will avoid an error message when you issue the command (3) against a nonexistent branch, e.g.:<br />
<pre><br />
§ git checkout -b MOODLE_27_STABLE origin/MOODLE_27_STABLE<br />
fatal: git checkout: updating paths is incompatible with switching branches.<br />
Did you intend to checkout 'origin/MOODLE_27_STABLE' which can not be resolved as commit?<br />
</pre><br />
<br />
Now it is wise to put the new directory mod/certificate/ to the list of ignored files of the main Moodle clone, otherwise a status of the main clone will keep reminding you that the new code has not been checked in.<br />
<pre><br />
$ cd /path/to/your/moodle/<br />
$ echo /mod/certificate/ >> .git/info/exclude<br />
</pre><br />
To update your Moodle installation now, you must visit both Git repositories and pull changes from upstream.<br />
<pre><br />
$ cd /path/to/your/moodle/<br />
$ git pull<br />
$ cd mod/certificate<br />
$ git pull<br />
</pre><br />
Writing a shell script with these lines in the root of Moodle installation is a very good idea. Otherwise it is easy to forget what Git repositories are there within the main Moodle repository.<br />
<br />
== See also ==<br />
<br />
; Moodle Docs<br />
* [[Git FAQ]]<br />
* [[Windows installation using Git]]<br />
* [[Git for Mac]]<br />
* [[:dev:Moodle versions]]<br />
* For some screenshots see [[User:Frank_Ralf/Git]] (still work in progress)<br />
* For fixing a Tracker Issue (MDL) / Forking Moodle / CONTRIButing code ... [[User:Sam_Hemelryk/My_Moodle_Git_workflow]]<br />
* [[Moodle_Production_Server_with_GIT|Case study Git + Moodle from Technical University Berlin]]<br />
<br />
; Moodle forum discussions<br />
* [https://moodle.org/mod/forum/discuss.php?d=2551753 Github and Moodle deployment for production]<br />
* [https://moodle.org/mod/forum/discuss.php?d=213695 Got GIT installed on my site- here's how!]<br />
* [http://moodle.org/mod/forum/discuss.php?d=168094 GIT help needed]<br />
* [http://moodle.org/mod/forum/discuss.php?d=165236 Best way to manage CONTRIB code with GIT]<br />
* [http://moodle.org/mod/forum/discuss.php?d=167063 Handy Git tip for tracking 3rd-party modules and plugins]<br />
* [http://moodle.org/mod/forum/discuss.php?d=167730 Moodle Git repositories]<br />
* [http://moodle.org/mod/forum/discuss.php?d=183693 Git and CVS]<br />
* [http://moodle.org/mod/forum/discuss.php?d=208904 GIT for dummies]<br />
* [http://moodle.org/mod/forum/discuss.php?d=211930 Git and upgrading misunderstanding]<br />
* [https://moodle.org/mod/forum/discuss.php?d=231046 Clear git guide for Admins (not developers)]<br />
<br />
; External resources <br />
* [http://thamblings.blogspot.com.au/2013/07/upgrading-moodle-from-git.html Deploying Moodle from git - Blog post from a production experience]<br />
* [http://www.kernel.org/pub/software/scm/git/docs/everyday.html Everyday GIT With 20 Commands Or So]<br />
* [http://gitref.org/ Git Reference]<br />
* [http://progit.org/book/ Pro Git book]<br />
* [http://eigenjoy.com/2008/05/15/git-from-the-bottom-up/ Git from the bottom up]<br />
<br />
[[ja:管理者用Git]]<br />
[[fr:Git_pour_administrateurs]]<br />
[[es:Git para Administradores]]</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Talk:Features_tour&diff=102720Talk:Features tour2013-01-10T09:02:26Z<p>Andrewnicols: </p>
<hr />
<div>==Notes==<br />
<br />
Link to tracker - MDLSITE-2076<br />
<br />
===Examples from other VLEs===<br />
<br />
*Fronter http://uk.fronter.info/product/ <br />
*Frog http://www.frogtrade.com/index.phtml?d=2458882<br />
*Schoology https://www.schoology.com/about.php<br />
*Blackboard http://www.blackboard.com/Platforms/Learn/Products/Blackboard-Learn/Features.aspx<br />
*Claroline http://www.claroline.net/demo-2/?lang=en<br />
*Sakai http://www.sakaiproject.org/learning-management<br />
*Sharepoint http://www.sharepointlms.com/features<br />
<br />
===Examples from Andrew N===<br />
<br />
* http://www.cloudflare.com/features-security<br />
* http://usa.autodesk.com/autocad/features/<br />
* https://www.google.com/intl/en/chrome/browser/features.html#security<br />
* http://www.ubuntu.com/ubuntu/features<br />
* http://www.microsoft.com/exchange/en-us/technologies.aspx<br />
* http://beoplay.com/Products/BeoplayV1 (scroll down the page or use the up/down arrows on your keyboard - this is all HTML5)<br />
* http://www.apple.com/iphone/features/<br />
* https://www.gandi.net/hosting/ (particularly the feature spots at the top on the carousel you can move left/right)<br />
* https://github.com/features/projects (similar kind of idea to the previous)<br />
<br />
===Possible features(not yet in any sequence)===<br />
<br />
*Possibly text and screenshots should be in sections as mentioned on tracker - student/teacher/admin?<br />
<br />
Are you a teacher? In Moodle you can.... (and much more!)<br />
<br />
Are you a student? In Moodle you can.... (and much more!)<br />
<br />
Are you an administrator? In Moodle you can....(and much more!)<br />
<br />
*Forum/wiki/drag and drop upload/embedded searchable media/customisable assignments/workshop/community hub/lesson and book/database and glossary/enrolment methods/activity completion/conditional activities<br />
<br />
===First thoughts (please ignore)===<br />
<br />
Are you a teacher? In Moodle you can...<br />
*Search for and embed multimedia at the click of a button<br />
*Drag your favourite teaching resources straight onto the page<br />
*Get your students learning together on forums and wikis<br />
*Set them individual, group or anonymously marked assignments<br />
*Create workshops for them to peer and self assess<br />
*Direct their learning path with Conditional Activities<br />
<br />
...and much more!<br />
<br />
Are you a student? In Moodle you can...<br />
*Build knowledge with your classmates by adding to glossaries and databases<br />
*Access files from your private area, from Google, Dropbox, Flickr and many other online storage sites<br />
*track your progress through each activity<br />
<br />
...and much more!<br />
<br />
Are you an administrator? In Moodle you can...<br />
*Authenticate and enrol your users in a variety of ways<br />
*Enhance your Moodle with contributed plugins that you can then update right from inside your site<br />
*Sign up to Community Hubs sharing free courses<br />
<br />
...and much more!</div>Andrewnicolshttps://docs.moodle.org/36/en/index.php?title=Capabilities/mod/forum:allowforcesubscribe&diff=102495Capabilities/mod/forum:allowforcesubscribe2012-12-21T19:50:08Z<p>Andrewnicols: Clarify text slightly</p>
<hr />
<div>{{Capabilities}}<br />
*This capability is present since Moodle 2.3.3<br />
*This allows a user to be subscribed automatically to forums where subscription is automatic or forced.<br />
*This capability is allowed for the default roles of teacher, non-editing teacher and student. It is not set for the role of manager.<br />
<br />
The following table summarises the effect this capability has on the various forum subscription methods:<br />
<br />
{| class="wikitable"<br />
! rowspan="2" | Subscription Mode<br />
! colspan="2" | Effect of capability...<br />
|-<br />
! on initial subscription<br />
! on subscription changes<br />
|-<br />
| Force subscription<br />
| Users with the capability are subscribed<br />
| It is not possible to change subscription<br />
|-<br />
| Auto subscription<br />
| Users with the capability are subscribed<br />
| Any user can choose to subscribe or unsubscribe at any point<br />
|-<br />
| Optional subscription<br />
| The capability has no effect<br />
| Any user can choose to subscribe or unsubscribe at any point<br />
|-<br />
| Subscription disabled<br />
| The capability has no effect<br />
| It is not possible to change subscription<br />
|}<br />
<br />
==See also==<br />
<br />
* [[Forum module]]<br />
<br />
[[Category:Capabilities|Forum]]<br />
[[Category:Forum]]</div>Andrewnicols