Resource module file API migration

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Moodle 2.0

• PROJECT STATE: Proposal
• MAIN TRACKER ISSUE: MDL-16089 Resource module migration
• DISCUSSION AND COMMENTS: not announced on moodle.org yet
• AUTHOR: Petr Škoda (škoďák) and others

Introduction

Current problems

Some general problems:

• plain text and html resource types separate
• embedded images and files in general are stored in course files without appropriate access control
• local course files and Internet links are mixed
• HarvestRoad Hive repository type is not maintained.
• can't move a "whole" web page properly from one course to another (eg course import).

The idea of resource types implemented as subtypes has always been a little awkward and not that helpful, incomplete list of problems follows:

1. sub-plugin APIs are not as well-defined as module APIs
2. refactoring inside the module is made harder
3. subplugins can't uninstall
4. subplugins need to bump the module version to upgrade
5. universal data fields get reused for different things
6. capability in subplugins not supported
7. subplugin upgrade is not integrated into main module upgrade - subplugin upgrade is executed after the main module upgrade (critical)
8. subplugins can not have own lang packs
9. subplugins do not have own log actions
10. instead of reducing code duplication advanced plugins may duplicate a lot more core
11. it is not easy to move unmaintained subplugins into contrib
12. unofficial subplugins not reported during upgrade
13. no support for subplugins in file api

Migration of resource types in Moodle 2.0

Splitting resource into multiple modules

After some discussion, we've decided to split resource into normal modules, simplifying the structure at the same time. The benefits are cleaner, more maintainable code and a simpler Moodle.

mod/resource will still remain and will be used for two purposes - to serve uploaded resource materials and also to redirect old URLs to the new modules. We will add a new table with backup of current resource table with all other information needed for upgrades and URL redirections.

In order to still be able to group the new modules together in the user interface as "resource" plugins, we can mark these as "resource" plugins in the modules table in core, in a field called "archetypes" (later there might be "assignment" in here too).

Resource types

Resource types in Moodle 1.9		New modules in Moodle 2.0
Type	Description	Module	Description
directory	an arbitrary directory of files in course files	mod/folder	a collection of files in separate file area
html	A html page, containing absolute links to media from course files or external links	mod/page	a general text page - html markup, format FORMAT_HTML and files stored in separate file area
text	A plain text page in some format	mod/page	a general text page - no markup, format FORMAT_PLAIN and no files
file	link to file from course files of the same course, files may contain relative links to other files	mod/resource	one file or collection of files, one file is always selected as main file
file	link to other course or general external URL	mod/url	general URL
ims	Files extracted into moddata from an IMS content package obtained from course files or web repository	mod/imspackage	Files in separate file area extracted from an IMS content package obtained from repository
repository	Links to external files in repositories (ie Hive)	mod/url?	(na)

Contrib resource types

There are other third party plugins in contrib that need to be migrated:

• mod/resource/type/digitalnz - browser and add items listed in http://www.digitalnz.org/
• mod/resource/type/globe - browser and add items in Ariadne repositories
• mod/resource/type/jmol - display chemistry structures from a jmol format file
• mod/resource/type/rss - display one RSS feed on a page
• mod/resource/type/slideshow - display a directory of images as a slideshow

Resource-like contrib modules

Separate modules will use similar migration techniques:

• mod/book - displays multiple pages of HTML, print support, table of contents, etc.
• mod/lightboxgallery - http://moodle.greenhead.ac.uk/external/lightbox_gallery/

New resource modules

Resource file (mod/resource)

General resource type with two purposes:

1. display uploaded files
2. redirect legacy resource links to new modules

Primary use cases:

• presentation/download of PDF files and office documents
• zip archive downloads
• presentation of general html files created in external system

Edit UI mockup

Page (mod/page)

Formatted text pages with embedded media files (images, flash, etc.). Very easy to use.

Primary use cases:

• short study materials
• general information
• notices

Most of the features are actually implemented in embedded text editor (TinyMCE).

Edit UI mockup

Folder (mod/folder)

Folder resource module, replaces original directory of files resource type. Files and folders are stored in separate file area.

Primary use cases:

• easy distribution of multiple files

Edit UI mockup

View UI mockup

URL (mod/url)

A URL to an external page or file.

Edit UI mockup

IMS Content Package (mod/imspackage)

IMS Content Package module, designed for easy presentation of standard compliant content packages. This is the recommended format for import of materials created in external systems. See http://www.imsglobal.org/content/packaging/

Primary use cases:

• import of study materials from other systems

Upgrade

Each resource type requires different upgrade code because the files are stored in different places that require special handling. They also have different settings and options with different text encoding in the table fields: reference, alltext and options.

lib/db/upgrade.php

Create new table resource_historic and copy all current records from resource table. Add new fields:

• oldid - original id from resource table
• cmid - original course module id
• newmodule - null means not converted, string value is name of new module
• newid - new instance id referencing id in newmodule table
• migration time

Switching resource types to full modules

Implemented in lib/upgradelib.php function switch_resource_type($resource_historic)

• $resource_historic must already contain new module info (newmodule and newid fields)
• links course_module to newid instead of old resource id
• deletes old record instance
• updates log table

• no need to upgrade gradebook - no grades or outcomes in current resources
• no need to change permissions - the same cmid means the same context
• no need for capability migration - resource does not define any capabilities, only mod/resource:exportresource added in 2.0dev
• old section placement retained, because it is specified in course_modules table
• no need to migrate files, no files in areas yet

Module upgrade steps

mod/resource/db/upgrade.php

1. change resource table structure - we do not need to keep any data because copy of everything is kept in resource_historic table
2. mark all records in resource table as not migrated
3. search for migration candidates in resource_historic table - original type file, link pointing to course file from the same course
4. find out old setting from historic record
5. update resource table record, marks the instance as active too
6. copy linked course file to new file area, mark this file as main file
7. enable #On-demand migration of files if file may contain relative links (whitelist image types and some other binary types)

mod/url/db/install.php

1. search for migration candidates in resource_historic table - type file with URLs outside of current course files
2. find out all new settings from existing historic record
3. create new instance in url table
4. update resource_historic
5.  call switch_resource_type($resource_historic)

mod/folder/db/install.php

1. search for migration candidates in resource_historic table - type directory
2. create new instance in folder table
3. update resource_historic
4. call switch_resource_type($resource_historic)
5. recursively copy the chosen course directory to the folder file area

mod/page/db/install.php

1. search for migration candidates in resource_historic table - type html and text
2. find out all new settings from existing historic record
3. create new instance in page table
4. update resource_historic
5. call switch_resource_type($resource_historic)
6. search for any absolute links in case of html text and copy them to new page file area keeping the same path relative to course files root
7. enable #On-demand migration of files if any copied file above may contain relative links

mod/imspackage/db/install.php

1. search for migration candidates in resource_historic table - type ims
2. find instance setting
3. create new instance in ims table
4. update resource_historic
5. call switch_resource_type($resource_historic)
6. recursively copy moddata files into ims file area
7. copy original backup of package file to another new ims file area

Others

• local files - type file with link starting with #
• Hive repositories - work out the URL of the resource and convert it to a simple URL?
• TODO: add detailed contrib migration info

On-demand migration of files

Some new resource modules may contain relative links to other files originally stored in course files area. We may not be able to detect all these relatively linked files during the migration/upgrade and copy them to the new file area. When browser opens a migrated file (served via $CFG->wwwroot/pluginfile.php) it first constructs absolute url (again starting with $CFG->wwwroot/pluginfile.php) and requests it from server. Now pluginfile.php locates our new resource module, module instance looks for the file in its own new file area and if it does not exists it tries to find the forgotten file in original course files. Any forgotten files are migrated to new file area on-demand.

The on-demand migration will be implemented only in several new modules (mod/resource and mod/page), others either know all required files or there can not be any relative links.

On-demand flag legacyfiles is stored in module instance table, all new instances created in 2.0 have it disabled. Upgraded instances have modedit setting which turns it on/off as needed. Migration/upgrade process tries to detect and problematic resource content and automatically turns on the on-demand migration:

• html files possibly containing images, css, applets and javascript
• flash applets containing relative links to media (eg sounds or images) and data files
• java applets containing relative links to media
• javascript that loads media via relative links (eg imagine an auto-generated slideshow)

We still want to move all these legacy files into the correct resource filearea, let's try to explain it once more ;-)

1. When a browser requests to view one of these legacy HTML files, it needs to load all the JS and Java and Flash it contains.
2. These small programs run and need to access other files, they do not know absolute URLs, so they contruct it from own absolute URL they know and specified relative links.
3. These requests to Moodle will hit the pluginfile.php script
4. The pluginfile.php script checks that the file doesn't exist yet in the resource file area, and if so:
   1. pluginfile.php uses the relative path to copy the original file from the courses area to the resource file area
   2. pluginfile.php updates legacyfile field for the resource to be the current timestamp, indicating that some processing has happened.

In any case the on-demand migration is relatively cheap, we do not need to turn it off for performance reasons - it is just one wasted db query when resource contains incorrect relative links. The cost of migration itself is not significant because it is done only once.

Problem: we don't really know when the resource is completely "finished" (ie every single associated file has been checked), so we can't automatically turn off on-demand migration after this process. That's why the manual post-upgrade script described bellow may help in some situations.

There are cases when we need to know list of files belonging to each resource:

• backup
• import resource instances into other courses

Since we have a timestamp in there we COULD consider a cron which detects completely migrated resources - resource with a lot of some recent active and no on-demand migration for a long time. Alternatively we may use these conditions to propose finished migrations on some admin report or post-upgrade script.

Post-upgrade processing

After the Moodle 2.0 upgrade has completed we take the admin to a special page that:

1. explains the issue of resources that contain links to other files
2. explains how it can be fixed by on-demand migration (see below)
3. presents an option for "automatic parsing" which tries to automatically migrate simpler resources such as HTML files with links to HTML files, images, CSS
4. presents a list of remaining flagged resources as links that need to be visited manually in the browser to be fixed

I imagine this interface might have a small frame on the left with the main pages being loaded in the larger frame on the right. The left frame has controls and info like:

• total number of pages to process
• next and previous buttons
• some button to say "I have completely viewed this resource"

So the admin will have to sit there and go through all these resources and check them off to complete the upgrade. Until this process is finished none of these operations can be guaranteed to work so we might even think about disabling those operations:

• Backup
• Restore
• Import of a course

Normal viewing of resources WILL work fine for teachers and students, because it will just trigger the normal on-demand migration of files.

Candidates for removal from core

List of obsoleted or unmaintained features proposed for removal from core:

• hive SSO
• hive repositories
• IMS repository
• special mod/resource/type/file/localfile.php (win32 only tweak)
• weblinkauth hook from login/index.php
• opening of resources in new windows

hive sso

Located in /sso/hive/*. Unmaintained and obsoleted by auth SSO plugins and Moodle networking. It should be possible to implement new auth/hivesso and maintain it in contrib. Any volunteers?

hive repositories

Unmaintained. Any volunteers?

IMS repository

Obsoleted by regular repositories. Maybe a new core IMS repo plugin.

file/localfile.php

No maintainer. Intended for large files stored on local networks or CDs, obsoleted by faster Internet connections and various offline Moodle projects. Volunteers may create new plugin and maintained it in contrib.

weblinkauth hook

Obsoleted during the auth refactoring in 1.8dev. Volunteers may continue maintaining it as new contrib auth plugin.

Opening resources in customized new windows

This is going to be controversial, not really sure if this is a good idea, please post in the page comments ;-)

Cons:

• teachers are used to it
• enlarges browser screen area

Pros:

• no support in xhtml strict
• not recommended by accessibility experts
• browsers allow users to easily open links in new tabs/windows manually when needed
• removing browser UI elements such as address/menu/status via javascript is considered evil - browser interfaces are not cluttered with useless dead space anymore, screens are larger, general F11 full-screen support
• chrome-less windows give false sense of copy-protection
• new window support was never implemented in directory type

Possible solution:

• keep support for opening of files in normal new windows for local files and external links - prevents loss of navigation
• remove new window detailed options completely - no more browser tricks such as window resize, chrome removal, etc.
• remove new window support from mod/page - more consistency with other modules, improves accessibility compliance

Other ideas

Dealing with caching issues

To have more control over the caching of files, we could use the same mechanism already implemented in the SCORM module, by changing the URL:

1. Internally store all files as itemid=0
2. Add new revision db field into resource table
3. When serving files always construct links with itemid==revision
4. Ignore itemid for resource files in pluginfile.php

See also

• Using the file API
• Repository API
• Portfolio API
• File API
• MDL-14589 - File API Meta issue
• MDL-16089 - Resource module conversion