Performance 2.x

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

Performance 2.x
Project state	Planning
Tracker issue	MDL-23754
Discussion	N/A
Assignee	unassigned

This page lists potential performance improvements that could be implemented in Moodle 2.1 or later. In general we must lower the number of queries on each page, we have to decrease the memory use and we should load less javascript code. There is not much point in detailed benchmarking of current code because the causes of slowness are already known, once we fix following issues the result could be very different.

PAGE level caching and improvements

The goal is to centralise all DB access caching on the current page and simplify page setup steps. The PAGE object should hold current course, activity, course_module, user's groups and enrolment status.

There would be several new methods such as:

$PAGE->init_course($course_or_id)
$PAGE->init_activity_by_cmid($modulename, $cmid)
$PAGE->init_activity_by_id($module, $id)
$PAGE->require_login()
$PAGE->is_enrolled($active)
$PAGE->get_current_group()
$PAGE->get_current_groupmode()

The benefit could be tens of DB operations per page and simplified API resulting in improved security. The backwards compatibility should not be a problem. Please note that $PAGE must not be abused in low level API and cron.

The only potential problem is the sloppy use of $PAGE in cron scripts, it has to be completely eliminated. We could create a fake page class that prints debug info when PAGE accessed from cron script, or it might return only some basic information.

Estimate: relatively easy, few BC problems, 1 week, pre-requirement for navigation cleanup

Navigation cleanup

The goal is fewer DB hits per page, fewer PHP includes resulting in reduced memory use.

We would used PAGE level caching instead of current navigation caches and DB access. The unnecessary includes are usually caused by incorrect coding style in plugins.

Admin user performance could be significantly improved if the admin tree was loaded via ajax because the admin tree is usually necessary only when admin is already in the admin section or on the frontpage. The admin tree was never intended to be constructed on each page in the first place.

The expected benefit is fewer page includes (less memory used) and fewer DB hits on each page.

Estimate: relatively easy, no BC problems, 1 week

Text caching redesign

Current text caching is not very efficient and may actually crate a bottleneck at the database level.

cache_text table was always very problematic, since the introduction of new context specific filtering and more options the hits are even less likely. The cache_filters previously used as a second level cache could benefit from shared memory caching. The proposed solution is to move the caching from the format_text() function to each filter plugin. This would allow much more efficient caching without DB bottlenecks.

Estimate: moderate, no BC problems, 1 week for core; filter caching would be implemented later using new shared memory or static caches.

Javascript performance

Our biggest problems are YUI2 and order of JS initialisation steps. We should switch to official YUI 2in3 – unfortunately YUI 2.9.0 is not supported yet and YUI 3.3.0 does not work with our code. YUI2 code is also not converted to YUI3 everywhere yet – YUI 3.4.0 should include all features we need, but it will be released much later this year

After we switched JS init in footer we started to have problems with event handler registration – it happens too late - for example you select new language and nothing happens because you did it too early. One possible solution is define priorities which would allow us to load all the non-essential stuff such as dock and navigation after the plugin specific code (such as lang menu and other jump menus).

In short we should switch to YUI 2in3, get rid of YUI2 as much as possible and tweak the order JS execution n each page. Benefits: faster page load, better perceived performance, some fast clicking problems solved. It would be nice to have a new JS debug console.

Estimate: difficult, BC problems caused by elimination of YUI2 in main JS scope, few weeks; our release is not aligned with YUI releases, we might have to wait till YUI 3.4.0 to get best results

Static dir caching

At present we are relying on browser level caching for all theme CSS and images, user images and javascript code. This works fine without any server configuration and it is very reliable during Moodle upgrades. If for some reason the browser caching does not work (mobil client, https, incorrect testing setup) the server may get overloaded because the serving of files via PHP is a more expensive than standard Apache file serving.

Solution could be a new static directory where we would mirror all our image,CSS and JS resources – server administrators would have to configure appropriate caching directives manually. This feature would be controlled by a new experimental settings, it might require manual static dir rebuilding and extra server configuration.

The only potential problem is automatic guessing of image extensions, this could be solved by conversion of all theme images to png format.

Estimate: relatively easy, no BC problems expected, 1 week at least

General caching framework

Please note this general caching framework is not a silver bullet, we will need to spend a lot of time on each individual improvement that is going to use it.

We could use different types of caching, we need to pinpoint the exact places and appropriate caching strategies. There has to be a balance otherwise the caching might actually slow the servers. This issue is describing general caching types and a few examples.

Types of caches:

persistent – this is used to cache expensive information which is not changing very often; it can be recreated at any time from existing data; it can be stored in database, filesystem, session or shared memory. Examples: htmlpurifier serializer, language list, tex image cache, modinfo, etc.
request – eliminate repeated queries, it is relatively easy to get the information. Examples: context cache, admin tree caching, PAGE level caching, etc.

Sometimes the caching strategy has to be manually selected by administrator depending on the type of server load.

A few candidates that could use this framework:

list of installed languages
text filtering plugins
context caching
user preferences
definition of roles from USER->access

The major problem is that we do not know what the API should look like, we need to start with implementation and gradually improve the API. I think we should begin with internal code first because we can deal with BC issues there easily.

Estimate: hard, major API changes expected, months

Smaller sessions

Large sessions cause performance problems, we have to find ways to minimise our session data.

Known problems:

bogus calendar stuff
repeated role definitions in each session
user preferences

Solution is to use other caching solutions and optimise the amount of information stored in session.

Estimate: mixed difficulty, some parts can be fixed easily, others may require major refactoring

Load balancing support

Moodle is not optimised to be compatible with advanced load balancing set-ups. Usually the network filesystem access is the bottleneck here. The obvious solution would be to split the dataroot into separate storage, cache and temp areas.

Depends on:

improved caching
no temporary files that span http requests in filesystem or shared memory
separation of caches and temp files from moodledata
elimination of file locking operations

Estimate: hard, several months, depends on cleanup in several areas and changes in API

New logging framework

Current logging is insufficient and may be major bottleneck at the database level. Ideally all logging and subsequent access to log information should be abstracted at PHP level to allow different implementations (separate DB link, filesystem, etc.).

Estimate: hard, API changes + new plugins and reports necessary, months

Compare performance of different databases

Some database might be significantly faster, it would be nice to know what database is best for each different server load type.

General prerequisites

Core developers are not skilled at server configuration or production server optimisations. Large parts of Moodle 2.0 were developed on ageing computers using slow ADSL connections without any real test data. Some refactoring and API changes necessary for future performance improvements were already carried out, but in general the performance did not have high priority during the 2.0dev cycle.

Developers need:

access to data sets similar to existing large production sites.
to know how are production servers configured (load balancing, db configurations, apache configurations, reverse proxing, etc.)
access to large hardware that can simulate complex server loads and both slow & fast client connection.
access to all supported browsers, operating systems and especially mobile devices.
to study available profiling tools and ways how to simulate real life workloads.

Current build-in tools:

basic performance counters in page footers - we need more information and some external logging
data generator script - unmaintained, we need to improve it significantly
xhprof integration

Documentation