Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Backup 2.0 requirements: Difference between revisions

From MoodleDocs
No edit summary
 
(9 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Template:Development:Backup 2.0}}{{Moodle_2.0}}__NOTOC__{{Work in progress}}
{{Template:WillNotMigrate}}{{obsolete}}
 
This page shows the analysis/justification/decision for all the [[Backup 2.0 drop in ideas|drop in ideas]] detected/suggested along the last months mashed up with the main goals of the project. While pinpointing all the requirements of the new Backup 2.0, also we'll be defining the main list of tasks to fulfil them.
This page shows the analysis/justification/decision for all the [[Backup 2.0 drop in ideas|drop in ideas]] detected/suggested along the last months mashed up with the main goals of the project. While pinpointing all the requirements of the new Backup 2.0, also we'll be defining the main list of tasks to fulfil them.


Line 12: Line 13:
'''Support incremental backups & restore'''
'''Support incremental backups & restore'''


This has been a requirement since long time ago (see various discussions in the Forums). Anyway it seems that, at the end, the only way to accomplish this with certain degree of success is to use '''external tools'" to calculate the differences between two backup files and apply them in order to build the necessary final backup file to be restored. The alternative approach, to make the functionality to be available under Moodle would require time/hast stamping near all information and keeping big logs of each backup file in order to selectively know what needs to be exported / imported. Not viable.
This has been a requirement since long time ago (see various discussions in the Forums). Anyway it seems that, at the end, the only way to accomplish this with certain degree of success is to use '''external tools''' to calculate the differences between two backup files and apply them in order to build the necessary final backup file to be restored. The alternative approach, to make the functionality to be available under Moodle would require time/hast stamping near all information and keeping big logs of each backup file in order to selectively know what needs to be exported / imported. Not viable.


So, from the backup / restore perspective, there is '''nothing to do''' but to provide information in one format that makes those "external tools" to be able to handle information properly. With the current changes already performed in the backup format (where information is highly splited, that should be far easier than under Moodle 1.9).
So, from the backup / restore perspective, there is '''nothing to do''' but to provide information in one format that makes those "external tools" to be able to handle information properly. With the current changes already performed in the backup format (where information is highly splited, that should be far easier than under Moodle 1.9).
Line 54: Line 55:


'''Support restore of old (1.9 only?) backups'''
'''Support restore of old (1.9 only?) backups'''
This is a '''must''' requirements and will be developed once Moodle 2.0 <==> 2.0 backup / restore is completely working. It will use the mechanism described at [[Backup 2.0 multiple formats|backup multiple formats]] so the format will be intercepted and converted. In any case it's going to be a really complex task, due to the deep changes performed in Moodle 2 (new modules, files...) and is, by itself, one separated project which codebase won't interfere at all with Moodle 2.0 backup / restore (although will be executed as part of it normally).


== RQ09: ==
== RQ09: ==


'''Hook in backup & restore to intercept & [[Backup 2.0 multiple formats|support other formats]]'''
'''Hook in backup & restore to intercept & [[Backup 2.0 multiple formats|support other formats]]'''
Backup will allow the generation of backup files in different formats. This is supported from core, so once the corresponding backup controller is instantiated and the corresponding format specified, everything (settings, ui, plan, tasks and steps) will be performed accordingly with that backup format.
In the other side, restore will detect as soon as possible the format of the file, performing the necessary conversions to transform the original format detected into one suitable Moodle 2.0 backup format.


== RQ10: ==
== RQ10: ==


'''One code base both for manual and scheduled backup'''
'''One code base both for manual and scheduled backup'''
Process will be unique. This is 100% mandatory. Another thing is the "scheduled" bit of those backups and their "iterative" (multiple) characteristic. Both are out from backup process responsibilities (see [[#RQ01:|RQ01]] above). One script / cron combination will provide such functionalities while using core backup API for each of the target courses when necessary.


== RQ11: ==
== RQ11: ==

Latest revision as of 19:02, 3 May 2022


Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.


Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.


This page shows the analysis/justification/decision for all the drop in ideas detected/suggested along the last months mashed up with the main goals of the project. While pinpointing all the requirements of the new Backup 2.0, also we'll be defining the main list of tasks to fulfil them.

RQ01:

Enable Backup and restore of a Course and it's related meta courses IN ONE HIT to save on adminstration time

Meta courses are out in Moodle 2.0 and their functionality will be, somehow, replaced my the new Cohorts (a.k.a. site-wide groups). In any case, and as a general rule, any backup / restore operation involving more that one course is not backup / restore responsibility (i.e. the functionality is not built in there) but responsibility of newly scripts using the new backup API to do those "multiple operations".

RQ02:

Support incremental backups & restore

This has been a requirement since long time ago (see various discussions in the Forums). Anyway it seems that, at the end, the only way to accomplish this with certain degree of success is to use external tools to calculate the differences between two backup files and apply them in order to build the necessary final backup file to be restored. The alternative approach, to make the functionality to be available under Moodle would require time/hast stamping near all information and keeping big logs of each backup file in order to selectively know what needs to be exported / imported. Not viable.

So, from the backup / restore perspective, there is nothing to do but to provide information in one format that makes those "external tools" to be able to handle information properly. With the current changes already performed in the backup format (where information is highly splited, that should be far easier than under Moodle 1.9).

RQ03:

Support 1-activity backup

Supported!

RQ04:

Support 1-section backup (topic/week)

Supported!

RQ05:

Support anonymisation of personal data on backup

Supported! Right now it's limited to anonymize user information bits (name, email...) and related info (preferences, tags, personal files...) to be exported. There are two points that could be improved:

  • Improvements to the anonymizer (MDL-22158): Make information to "look" better, using random names and so on.
  • Be more intrusive (NOBUG): Also anonymize (replace) file submissions and friends by "template-blank" files.

RQ06:

Separate process and progress

This is one of the primary goals of the new subsystem. Already have two well defined points where visualization can happen (conditionally). The backup / restore UI (MDL-22142 - previous to process). And the output process (MDL-22144 - handled by output singleton). Both used only if necessary and not interfering the process itself at all. Also, logging (MDL-22143) has been separated from process and delegated to proper loggers. Finally, heartbeat information (MDL-22143 - needed to avoid browser timeouts) is also performed transparently by the process and developer doesn't need to do it manually any more.

RQ07:

XML format doesn't need to be radical different

While the original idea (at early Moodle 2.0 planning stages) was to keep the Backup 2.0 format as similar as possible with 1.x format, finally some core changes (new File API...), other requirements (like 1-activity, 1-section backups, see above...) and the need to modernize a bit the old format (uppercase, not using attributes...), have produced this original thoughts to change. So Moodle 2.0 will debut one new format, split along multiple files for easier reference / manipulation.

Note that contents will be basically the same, as far as contents are just "DB dumps". They just will be more organized along different files. See the general architecture page for more information about the new format.

RQ08:

Support restore of old (1.9 only?) backups

This is a must requirements and will be developed once Moodle 2.0 <==> 2.0 backup / restore is completely working. It will use the mechanism described at backup multiple formats so the format will be intercepted and converted. In any case it's going to be a really complex task, due to the deep changes performed in Moodle 2 (new modules, files...) and is, by itself, one separated project which codebase won't interfere at all with Moodle 2.0 backup / restore (although will be executed as part of it normally).

RQ09:

Hook in backup & restore to intercept & support other formats

Backup will allow the generation of backup files in different formats. This is supported from core, so once the corresponding backup controller is instantiated and the corresponding format specified, everything (settings, ui, plan, tasks and steps) will be performed accordingly with that backup format.

In the other side, restore will detect as soon as possible the format of the file, performing the necessary conversions to transform the original format detected into one suitable Moodle 2.0 backup format.

RQ10:

One code base both for manual and scheduled backup

Process will be unique. This is 100% mandatory. Another thing is the "scheduled" bit of those backups and their "iterative" (multiple) characteristic. Both are out from backup process responsibilities (see RQ01 above). One script / cron combination will provide such functionalities while using core backup API for each of the target courses when necessary.

RQ11:

Scheduled restore

RQ12:

Fix restore so that one can also select to restore course settings

RQ13:

Fileless export/import

RQ14:

Export/import over mnet

RQ15:

OOP

RQ16:

Secure

RQ17:

Safe

RQ18:

file-less backups - if you have separate backups, import/export etc.

RQ19:

Allow to backup one category of courses. MDL-17187

RQ20:

Allow to mark courses to be excluded from scheduled backup manually

RQ21:

Avoid anti-timeout output when not running from browser. MDL-17282

RQ22:

Review related caps, cleaning them and adding missing bits, improving sec. consistency

RQ23:

Log all backups (both manual and scheduled). Improve logging in general

RQ24:

Separate each module's portion of XML into its own namespace

RQ25:

Use the namespaces to allow each module the option of validating it's XML content before processing

RQ26:

Roll dates

RQ27:

Backup would need to be called when publish a course on the community hub

RQ28:

Metadata... to be included in backup/restore. Something like Metadata

RQ29:

Allow backup of roles with permissions, assigns and overrides. MDL-17081