Resource module file API migration: Difference between revisions
Line 228: | Line 228: | ||
* special mod/resource/type/file/localfile.php (win32 only tweak), intended for large files stored on local networks or CDs | * special mod/resource/type/file/localfile.php (win32 only tweak), intended for large files stored on local networks or CDs | ||
* weblinkauth hook from login/index.php | * weblinkauth hook from login/index.php | ||
* opening of resources in new windows - this could be very controversial, not really sure if it is a good idea | |||
TODO: add sections with more explanation and alternative solutions for sites that are still using these features | TODO: add sections with more explanation and alternative solutions for sites that are still using these features | ||
===hive sso=== | |||
Located in /sso/hive/*. Unmaintained and obsoleted by auth SSO plugins and Moodle networking. It should be possible to implement new auth/hivesso and maintain it in contrib. Any volunteers? | |||
===hive repositories=== | |||
Unmaintained. Any volunteers? | |||
===file/localfile.php=== | |||
No maintainer. Obsoleted by faster Internet connections and various offline Moodle projects. Volunteers may create new plugin and maintained it in contrib. | |||
===weblinkauth hook=== | |||
Obsoleted during the auth refactoring in 1.8dev. Volunteers may continue maintaining it as new contrib auth plugin. | |||
===Opening resources in customized new windows=== | |||
This is going to be controversial, please post in the page comments ;-) | |||
Pros: | |||
* teachers are used to it | |||
* prevents loss of navigation elements for external links and directly opened media | |||
* enlarges browser screen area | |||
* new window support was never implemented in directory type | |||
Cons: | |||
* no support in xhtml strict | |||
* not recommended by accessibility experts | |||
* browsers allow users to easily open links in new tabs/windows manually when needed | |||
* removing browser UI elements such as address/menu/status via javascript is considered ''evil'' - browser interfaces are not cluttered with useless dead space anymore, screens are larger, general F11 full-screen support | |||
* chrome-less windows give false sense of ''copy-protection'' | |||
Possible solution: | |||
* keep support for opening of files in normal new windows for local files and external links - prevents loss of navigation | |||
* remove new window detailed options completely - no more browser tricks such as page resize, UI removal, etc. | |||
* remove new window support from mod/page - more consistency with other modules | |||
=Other ideas= | =Other ideas= |
Revision as of 20:07, 5 June 2009
Moodle 2.0
Please use talk page for comments and proposals, thanks.
Current problems
Some general problems:
- plain text and html resource types separate
- embedded images and files in general are stored in course files without appropriate access control
- local course files and Internet links are mixed
- HarvestRoad Hive repository type is not maintained.
- can't move a "whole" web page properly from one course to another (eg course import).
The idea of resource types implemented as subtypes has always been a little awkward and not that helpful, incomplete list of problems follows:
- sub-plugin APIs are not as well-defined as module APIs
- refactoring inside the module is made harder
- subplugins can't uninstall
- subplugins need to bump the module version to upgrade
- universal data fields get reused for different things
- capability in subplugins not supported
- subplugin upgrade is not integrated into main module upgrade - subplugin upgrade is executed after the main module upgrade (critical)
- subplugins can not have own lang packs
- subplugins do not have own log actions
- instead of reducing code duplication advanced plugins may duplicate a lot more core
- it is not easy to move unmaintained subplugins into contrib
- unofficial subplugins not reported during upgrade
- no support for subplugins in file api
Migration of resource types in Moodle 2.0
Splitting resource into multiple modules
After some discussion, we've decided to split resource into normal modules, simplifying the structure at the same time. The benefits are cleaner, more maintainable code and a simpler Moodle.
mod/resource will still remain and will be used for two purposes - to serve uploaded resource materials and also to redirect old URLs to the new modules. We will add a new table with backup of current resource table with all other information needed for upgrades and URL redirections.
In order to still be able to group the new modules together in the user interface as "resource" plugins, we can mark these as "resource" plugins in the modules table in core, in a field called "archetypes" (later there might be "assignment" in here too).
Resource types
Resource types in Moodle 1.9 | New modules in Moodle 2.0 | ||
---|---|---|---|
Type | Description | Module | Description |
directory | an arbitrary directory of files in course files | mod/folder | a collection of files in separate file area |
html | A html page, containing absolute links to media from course files or external links | mod/page | a general text page - html markup, format FORMAT_HTML and files stored in separate file area |
text | A plain text page in some format | mod/page | a general text page - no markup, format FORMAT_PLAIN and no files |
file | link to file from course files of the same course, files may contain relative links to other files | mod/resource | one file or collection of files, one file is always selected as main file |
file | link to other course or general external URL | mod/url | general URL |
ims | Files extracted into moddata from an IMS resource package obtained from course files or web repository | mod/ims | Files in separate file area extracted from an IMS resource package obtained from repository |
repository | Links to external files in repositories (ie Hive) | mod/url? | (na) |
Contrib resource types
There are other third party plugins in contrib that need to be migrated:
- mod/resource/type/digitalnz - browser and add items listed in http://www.digitalnz.org/
- mod/resource/type/globe - browser and add items in Ariadne repositories
- mod/resource/type/jmol - display chemistry structures from a jmol format file
- mod/resource/type/rss - display one RSS feed on a page
- mod/resource/type/slideshow - display a directory of images as a slideshow
Resource-like contrib modules
Separate modules will use similar migration techniques:
- mod/book - displays multiple pages of HTML, print support, table of contents, etc.
- mod/lightboxgallery - http://moodle.greenhead.ac.uk/external/lightbox_gallery/
New resource modules
The new modules will be:
- mod/page - formatted text pages with embedded files (images, flashm etc.)
- mod/resource - a folder of files, with one file selected as main file
- mod/folder - a folder of files, listed for selection (view/download)
- mod/url - a URL to an external page or file
- mod/ims - an IMS package
mod/page
TODO: add mockups and descriptions of functionality
mod/resource
TODO: add mockups and descriptions of functionality
mod/folder
TODO: add mockups and descriptions of functionality
mod/url
TODO: add mockups and descriptions of functionality
mod/ims
TODO: add mockups and descriptions of functionality
Automatic upgrade
Each resource type requires different upgrade code because the files are stored in different places that require special handling. They also have different settings and options with different text encoding in the table fields: reference, alltext and options.
Resources that can not be completely migrated during upgrade time will be flagged using a new field called legacyfiles:
- 0 = new resource created in Moodle 2
- 1 = upgraded resource that has completed migration
- 2 = upgraded resource that has not started migration yet
- timestamp = last date of a migrated file within this resource
Directory
Recursively copy the chosen course directory to the resource area. No other upgrade steps are required.
A full filemanager will be required to manage the tree of files.
File
Files in the course directory
- Copy the linked file (image, sound, pdf, HTML, etc.) from course area to resource file area
- If the file is HTML or SWF then flag this resource for post-upgrade processing with legacyfiles = 2 (see next section), otherwise flag with legacyfiles = 1
Files referenced by URL
(This includes links to other courses and site files)
This does not require any upgrade. The url is stored in reference field and will continue working.
Page (Web and Text)
- Convert to one plug-in called 'page' (editable via the new editor forms element)
- Flag this resource for post-upgrade processing with legacyfiles = 2 (see next section).
IMS
- Copy files from moddata to new resource file area
- Copy original backup of package file to another new resource area
No other special upgrade steps required.
Repository
Work out the URL of the resource and convert it to a simple URL.
Others
(TODO: Add some docs here for each type)
Post-upgrade processing
After the Moodle 2.0 upgrade has completed we take the admin to a special page that:
- explains the issue of resources that contain links to other files
- explains how it can be fixed by on-demand migration (see below)
- presents an option for "automatic parsing" which tries to automatically migrate simpler resources such as HTML files with links to HTML files, images, CSS
- presents a list of remaining flagged resources as links that need to be visited manually in the browser to be fixed
I imagine this interface might have a small frame on the left with the main pages being loaded in the larger frame on the right. The left frame has controls and info like:
- total number of pages to process
- next and previous buttons
- some button to say "I have completely viewed this resource"
So the admin will have to sit there and go through all these resources and check them off to complete the upgrade. Until this process is finished none of these operations can be guaranteed to work so we might even think about disabling those operations:
- Backup
- Restore
- Import of a course
Normal viewing of resources WILL work fine for teachers and students, because it will just trigger the normal on-demand migration of files.
On-demand migration of files
After the upgraded process we will have resources flagged with legacyfiles = 2 that need migrating, including:
- html files possibly containing images, applets and javascript
- flash applets containing relative links to media (eg sounds or images)
- java applets containing relative links to media
- javascript that loads media via relative links (eg imagine an auto-generated slideshow)
We still want to move all these legacy files into the correct resource filearea, so we do it like this:
- When a browser requests to view one of these "legacyfile" HTML files, it will load all the JS and Java and Flash it contains.
- These programs run, and make relative requests to media if they want to.
- These requests to Moodle will hit the pluginfile.php script
- The pluginfile.php script checks that the file doesn't exist yet in the resource file area, and if so:
- pluginfile.php uses the relative path to copy the original file from the courses area to the resource file area
- pluginfile.php updates legacyfile field for the resource to be the current timestamp, indicating that some processing has happened.
Problem: we don't really know when the resource is completely "finished" (ie every single associated file has been checked), so we can't automatically reset legacyfiles = 1 after this process. That's why the manual post-upgrade script described above is necessary.
Since we have a timestamp in there we COULD consider a cron job to reset the value to 1 once the timestamp has reached some old age (and it's unlikely that any more files need to be migrated for that resource) to prevent too much parsing. Alternatively we could contact the admin or teacher and ask them to confirm migration using the postupgrade script.
Candidates for removal
List of obsoleted or unmaintained features proposed for discontinuation.
- hive repositories
- hive SSO
- special mod/resource/type/file/localfile.php (win32 only tweak), intended for large files stored on local networks or CDs
- weblinkauth hook from login/index.php
- opening of resources in new windows - this could be very controversial, not really sure if it is a good idea
TODO: add sections with more explanation and alternative solutions for sites that are still using these features
hive sso
Located in /sso/hive/*. Unmaintained and obsoleted by auth SSO plugins and Moodle networking. It should be possible to implement new auth/hivesso and maintain it in contrib. Any volunteers?
hive repositories
Unmaintained. Any volunteers?
file/localfile.php
No maintainer. Obsoleted by faster Internet connections and various offline Moodle projects. Volunteers may create new plugin and maintained it in contrib.
weblinkauth hook
Obsoleted during the auth refactoring in 1.8dev. Volunteers may continue maintaining it as new contrib auth plugin.
Opening resources in customized new windows
This is going to be controversial, please post in the page comments ;-)
Pros:
- teachers are used to it
- prevents loss of navigation elements for external links and directly opened media
- enlarges browser screen area
- new window support was never implemented in directory type
Cons:
- no support in xhtml strict
- not recommended by accessibility experts
- browsers allow users to easily open links in new tabs/windows manually when needed
- removing browser UI elements such as address/menu/status via javascript is considered evil - browser interfaces are not cluttered with useless dead space anymore, screens are larger, general F11 full-screen support
- chrome-less windows give false sense of copy-protection
Possible solution:
- keep support for opening of files in normal new windows for local files and external links - prevents loss of navigation
- remove new window detailed options completely - no more browser tricks such as page resize, UI removal, etc.
- remove new window support from mod/page - more consistency with other modules
Other ideas
Dealing with caching issues
To have more control over the caching of files, we could use the same mechanism already implemented in the SCORM module, by changing the URL:
- Internally store all files as itemid=0
- Add new revision db field into resource table
- When serving files always construct links with itemid==revision
- Ignore itemid for resource files in pluginfile.php
See also
- Using the file API
- Repository API
- Portfolio API
- File API
- MDL-14589 - File API Meta issue
- MDL-16089 - Resource module conversion