Offline Moodle: Incremental Backups

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

Current work on Incremental Backups

One of the first features that on Offline Moodle requires from Moodle is to be able to get updates from a central Moodle in a small, efficient package rather than a large download. The only reliable method for getting the required information in and out of Moodle at the moment is using the backup/restore feature that's built into Moodle and thus Offline Moodle. This works well enough but can easily create large files which will cause problems when a user just wants to update quickly before they go to work for example.

The answer to this is to enhance the backup component to offer smaller incremental updates as well as the standard full backup. To achieve this the Open University (UK) have teamed up with Catalyst to develop this functionality and get it into the core of Moodle. We believe that not only is this functionality a necessary feature for Offline Moodle but that it will benefit many sections of the community particularly those responsible for managing Moodle installations.

You can report a bug at http://tracker.moodle.org log it as part of the 'non core components' under the 'Offline Moodle' component and please include 'Incremental Backup:' in the bug title to help us identify it. Assign it to Colin Chambers.

If you wish to get in contact then please email me at c.chambers@open.ac.uk

Latest Work

If you'd like to check our progress you can download Moodle 1.9 with the latest incremental backup code at http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=shortlog;h=refs/heads/mdl19-incrementalbackup .

Configuring Incrementals

• first install xdelta on your machine - see xdelta.org for details.
• in admin > courses > Incremental Backups, make sure the xdelta path states where xdelta is installed on your machine.
• in admin > courses > Incremental Backups, set whether incremental backups can be created as part of manual backups. see below for more details on configuration.

Generating Incrementals

Manual Generation

This is the easiest way to create incrementals and test them. make sure you have enabled incrementals in manual backups in admin > Courses > Incremental Backups, then just create a course with a little content. Then selecting backup from the course admin menu, make sure you select generate incrementals from the form provided. It's at the bottom of the list of drop down menus about half way down the form. Then generate the backup. - it will generate incrementals for every backup currently stored in the backup directory.

Cron based incrementals

Cron based incrementals is a very similar process to cron based backups, but it allows you to select which courses you want to generate backups and incrementals for, and which modules to include in the backups. Configuration is handled on the Admin > Courses > Incremental Backup page and the course selection is made on the Admin > Courses > Incremental Courses page.

Applying/getting an Incremental

This isn't "easy" with the code currently in Git, - I should have a way for "clients" to easily get an update later this week. each backup has it's own Hash as an ID, so when a client requests a new update, it passes through it's current hash. - this hash is compared to the list of hash's stored on the servers incremental_instance table. - it then checks to see what course this hash is for, and if it has an incremental on file. - if an incremental doesn't exist, it will return a full backup (an incremental might not exist if you specify a small number of backups to keep in the incremental cron settings)

the file get_incremental.php runs on the client eg: /backup/get_incremental.php?hash=b8fbaacc962c450bafe6861d6e70fcd9&id=3

this then matches up with a send_incremental.php file that would sit on the "server"

get_incremental on the client "calls" send_incremental on the server via an http request twice. first time to check if the cnxn works, and if an incremental is available. second time to download the incremental.

If the cURL connection fails, I thought we should put a nice html form that allows users to click on a link to download the incremental - this would allow the browser proxy settings to be used etc if there were firewall issues, and we could also do some work to the server interface so that if the hash they pass isn't known, and we can't match it to a course, then we can provide contact info, or some other method of providing them a full backup for the course they are doing. then the same html form on the client would have an upload form so that once they have downloaded an update, they can upload it to their local moodle and have it installed. We should also used any proxy settings set in admin > config when calling curl.

Once it has "retrieved" an incremental from the server, the client uses xdelta to apply the downloaded incremental to a local copy of the last known backup received from the server. - it then kicks off the restore process. As part of the restore process, the user has the option to "update" an existing course instead of the usual - delete existing data/add to new course type options. - this uses the backup_guids table to make sure that we don't get duplicates of data (currently only partially working with discussion forums.

To Do

Will flesh this todo out a bit more soon.

• Write "helper" to call get_incremental.php to allow a client to get/install an update.
• complete "update course" type for resources and forums, so that we don't duplicate data
• don't keep unchanged incrementals - check the xdelta manifest and if no changes since last backup, delete new backup and incrementals.
• write "pretty form" for when cURL fails
• use proxy settings if set when cURL called
• create client config page to pull out hardcoded vars from get_incremental process

Known Issues

MDL-14585 issues with saving local/settings.php