Automatic updates deployment

Automatic updates deployment
Project state	Specification draft
Tracker issue	MDL-35238 and MDL-35239
Discussion	Automatic updates deployment
Assignee	David Mudrak, Eloy Lafuente

Moodle 2.4

Use cases

Meet Sue, a Moodle administrator. She runs her Moodle at a cheap web hosting company where she gets an FTP access only (and some basic web tool to administer one SQL database). No shell access, no way to use Git etc. She never modifies source codes manually. And she likes how Wordpress deals with updates.

Use case 1

Sue uploads a new minor Moodle version and executes the upgrade (say from 2.4.0 to 2.4.1)
At the "Plugins check" screen, there is an info about some available updates. Prior to continuing with the Moodle upgrade, Sue is recommended to deploy new versions for these plugins.
So she keeps all available plugin updates selected and clicks "Update all selected plugins"
New versions of selected plugins are downloaded and deployed into wwwroot, replacing the current content (some progress bars and info would be nice to have).
The "Plugins check" screen is reload again. Now no available updates are reported - those plugins that were reported before are now marked as "To be upgraded"
Sue presses the "Upgrade Moodle database now" button. Both Moodle core and all plugins are updated.

Use case 2

Sue wants to install a contributed extension into her Moodle.
Using a new admin tool, she can browse the list of available plugins. She gets the basic information about plugins and can read more about them using a direct link to http://moodle.org/plugins registry.
When she finds the plugin she is looking for, she presses the button "Download".
At the next screen, confirms the source URL and the target location. She presses the "Download" button again.
The ZIP file with the plugin is downloaded. Its MD5 content hash is compared with the expected value. The ZIP is unpacked into the correct location inside wwwroot.
The browser is redirected to the /admin so the normal upgrade continues as if Sue uploaded the plugin herself.

Use case 3

On Friday morning, Sue gets a regular email generated by her site that a new weekly build is available for her Moodle.
She likes her site being up-to-date because she knows that up-to-date code is essential for the site security.
There is a simple way to update whole Moodle via the web. Sue does not need to download the whole pack, unpack it locally and upload it via FTP to her server (in older versions this was very boring job that discouraged Sue from doing it). She is also happy that she does not need to compare the remote side with the ZIP contents to look for files that should be now removed (in older versions this was pretty challenging for her so she often found herself having some old files left at the server).

Use case 4

Bob, a friend of Sue, asks her for an assistance with a fresh Moodle installation. They are having a nice weekend in a beautiful village with very poor internet connectivity. Uploading the whole Moodle pack would take ages.
Even in this case, there is no need to deal with the download, unzip and upload procedure - Sue just uploads a single small PHP file to the empty wwwroot of Bob's future server.
Then she visits the script's URL and after providing some details into the form, the script fetches the whole Moodle package, unpacks it and redirects the browser to the /admin so that standard installation can continue.

Technical issues to deal with

In Use case 1 above, there is a new version of Moodle already uploaded to the server. We are in the middle of the upgrade process. It is generally very tricky to depend on any Moodle functionality in this moment. The code that actually fetches the packages from download.moodle.org must be independent of the Moodle itself. Otherwise, the complex chain of includes would lead to problems. In other words, we can't include config.php to fetch and deploy the ZIPs.
For the same reasons, the part of the machinery that actually fetches the files from the download.moodle.org server in Use case 3, must be independent of the rest of Moodle. In other words, Moodle can't overwrite its own codes. Don't try it. Bad thing would happen with Gaia.
In Use case 3, downloading full ~40MB ZIP is not ecological. There might be a way of having "incremental/patch packages" generated for each weekly build that would contain just a subset of files that were modified in that release.
In Use case 3, we need to put the Moodle site into a hard maintenance mode prior to deploying the code so that there is no chance that a random user visits the site meantime.
In Use case 4, we need some sort of authentication so that the actual deployment can't be executed by a random visitor (or a bot) that accesses the site before Sue does it. The script might generate a solid strong passphrase (token) into a text file in wwwroot. So that only people having access to the server via FTP can read the passphrase and copy it into the web form.
Note that we always expect that Sue has its own way of doing full database and moodledata backups prior to any changes. Having an in-built tool to make such backup is out of scope of this project.

Implementation schedule

Episode IV: Use case 1 is implemented - installed plugins can be updated via the web only (MUST-HAVE for Moodle 2.4)
Episode III: Use case 2 is implemented - new plugins can be installed (NICE-TO-HAVE for Moodle 2.4, MUST-HAVE for 2.5)
Episode II: Use case 3 is implemented - Moodle core can be updated (NICE-TO-HAVE for 2.5, MUST-HAVE for 2.6)
Episode I: Use case 4 is implemented - Moodle can be installed from scratch via a bootstrap file (NICE-TO-HAVE for 2.5 or 2.6)

Implementation plan

Required components of the machinery

A standalone tool (script) that is able to fetch a ZIP package from a given URL, compare its MD5 content hash with a given value, unpack it, and deploy the content to a given location inside wwwroot.
Modification of the "Plugins check" and "Plugins overview" admin pages so they provide a way to select available updates and pass required info to the standalone script mentioned above.
Extending the REST API at the download.moodle.org so that it provides all required info (URL to download the ZIP from and the ZIP's MD5 hash).
New admin tool that allows the administrator to browse the list of available plugins (compare to the community finder tool that lets you browse the list of available courses

Overall logic flow

There are two places where a plugin update can be ignited from: 1) Plugins check page (in the middle of the upgrade process) and 2) Plugins overview page. We need to be able to ignite the deployment from both places while still re-use as much functionality as possible.

1. Admin looks at say Plugins overview pgae 2. The page script (/admin/index.php) reads a singleton instance of available_update_deployer. It sets its own URL as the caller's URL (this one that actually accepts some params via HTTP) and the return URL. 3. When displaying the available update info, the singleton is used to generate a widget (single buttom currently) "Install this update". Clicking the button lets the admin to review the request (what plugin, where from etc) and confrm the request. 4. When confirmed, the page redirects to a standalone script /mdeploy.php which actually makes the job. 5. The browser is then redirected back to the caller's return URL.

Documentation