Development:Backup 2.0: Difference between revisions

From MoodleDocs
mNo edit summary
No edit summary
 
(17 intermediate revisions by 6 users not shown)
Line 1: Line 1:
{{Work in progress}}{{Moodle_2.0}}
{{Template:Development:Backup 2.0}}{{Moodle_2.0}}__NOTOC__{{Work in progress}}
== Summary & Objectives ==
The backup & restore functionality has been present since Moodle 1.1 (29 August 2003, more than six years ago!) and, although it has been continuously improved and renewed, the core has remained basically the same.


=== Drop in ideas ===
Many things have evolved since then both in Moodle and PHP-land so this project is an attempt to rework the backup & restore functionality in order to achieve these goals:


* Support incremental backups & restore.
* '''Modernise''' the code base, using actual PHP/DB techniques: OOP, exceptions, temporary DB tables, transactions... aiming to reduce the current ''"spaghetti code"'' in various places.
* Support 1-activity backup.
* '''Unify''' current developments, so all backup alternatives (manual, export, silent, scheduled...) will be executed by the same code base (also for restore/import of course).
* Backup/restore one topic only (not just one activity) --[[User:Samuli Karevaara|Samuli Karevaara]] 21:29, 5 March 2009 (CST)
* '''Improve''' the process, making it more reliable, being able to pre-detect wrong situations, rollback (if possible) to previous states, and apply any possible benefit both for speed (some interim in-memory caches) and memory (better parsing techniques).
* Support anonymisation of personal data on backup.
* Improve '''Security''', so the Moodle permissions structure will be 100% respected and any user data disclosure will be avoided, of course, whilst giving the admins the flexibility to change default behaviours by configuring them in a standard way.
* Separate process and progress. ''(Perhaps [http://cvs.moodle.org/moodle/lib/weblib.php?r1=1.1214&r2=1.1215 these classes I wrote] can help)--[[User:Tim Hunt|Tim Hunt]] 19:41, 1 March 2009 (CST)''
* '''New features''' requested from the Community will be added. Things like fine grained control of various restore parts, the ability to backup not only course structures, but sections or individual activities, anonymised backups and so on will début with Moodle 2.0. See the [[Development:Backup 2.0 requirements|requirements]] section for more info.
* XML format doesn't need to be radical different (IMO).
* Support restore of old (1.9 only?) backups.
* Hook in backup & restore to intercept & support other formats (BB, IMS-CC...)
* One code base both for manual and scheduled backup
* Scheduled restore
* Fileless export/import
* Export/import over mnet
* OOP.
* Secure (don't handle non-allowed data, user matching on restore, salted passwords..)
* Safe (some sort of "rollback" on failure - requires to have everything annotated somewhere - backup_ids or so).
* file-less backups - if you have separate backups, import/export etc.
* Allow to backup one category of courses (at the same time, one zip per course). MDL-17187
* Allow to mark courses to be excluded from scheduled backup manually.
* Avoid anti-timeout output when not running from browser. MDL-17282
* Review related caps, cleaning them and adding missing bits, improving sec. consistency.
* Log all backups (both manual and scheduled). Improve logging in general.
* Separate each module's portion of XML into its own namespace.
* Use the namespaces to allow each module the option of validating it's XML content before processing.


=== General prerequisites ===
== Implementation plan ==


==== [[Development:Backup 2.0 - Improve XML parsing|Improve XML parsing]] ====
META bug in the Tracker: MDL-21432
One of the major bottlenecks in backup and, specially, in restore reliability under Moodle 1.x has been the big amount of memory needed to handle those operations (see bugs like MDL-14302, MDL-15489, MDL-9838... and many others). While the whole XML file (moodle.xml) is parsed in a [http://en.wikipedia.org/wiki/Simple_API_for_XML SAX] way (hence, "streamed" and requiring small amounts of memory),  we use to group some XML contents into "parts" in order to delegate the operations over those "parts" to different plugins (modules, blocks...). And problems arrive when some of those "parts" is big enough and processing them with the [http://php.net/xml_parse_into_struct xml_parse_into_struct()] function and the [http://www.hansanderson.com/php/xml/ xmlize] library uses exaggerated amounts of memory (for example, to process a 12.5MB file requires 311MB of memory, crazy!). So we need to switch to an alternate method to parse those "parts" using much less memory and to build the corresponding in-memory object with one acceptable throughput (speed).
 
[[Category:Backup]]

Latest revision as of 10:07, 25 April 2010

Template:Development:Backup 2.0Template:Moodle 2.0

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.


Summary & Objectives

The backup & restore functionality has been present since Moodle 1.1 (29 August 2003, more than six years ago!) and, although it has been continuously improved and renewed, the core has remained basically the same.

Many things have evolved since then both in Moodle and PHP-land so this project is an attempt to rework the backup & restore functionality in order to achieve these goals:

  • Modernise the code base, using actual PHP/DB techniques: OOP, exceptions, temporary DB tables, transactions... aiming to reduce the current "spaghetti code" in various places.
  • Unify current developments, so all backup alternatives (manual, export, silent, scheduled...) will be executed by the same code base (also for restore/import of course).
  • Improve the process, making it more reliable, being able to pre-detect wrong situations, rollback (if possible) to previous states, and apply any possible benefit both for speed (some interim in-memory caches) and memory (better parsing techniques).
  • Improve Security, so the Moodle permissions structure will be 100% respected and any user data disclosure will be avoided, of course, whilst giving the admins the flexibility to change default behaviours by configuring them in a standard way.
  • New features requested from the Community will be added. Things like fine grained control of various restore parts, the ability to backup not only course structures, but sections or individual activities, anonymised backups and so on will début with Moodle 2.0. See the requirements section for more info.

Implementation plan

META bug in the Tracker: MDL-21432

Retrieved from "https://docs.moodle.org/405/en/index.php?title=Development:Backup_2.0&oldid=71423"
Powered by MediaWiki