Improved support for external File content

Revision as of 08:01, 9 November 2011 by Dongsheng Cai (talk | contribs) (File picker walk-through)

Jump to: navigation, search

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on or use the page comments.

File synching
Project state Planning
Tracker issue MDL-28666
Discussion TODO
Assignee Martin Dougiamas

Moodle 2.2

Problems with Files in 2.0

  • It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
  • It is not currently easy to create a simple shared "course repository" for teachers to use

Example use cases that will become possible

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

  1. The teacher uploads the file to their private file area (or other repository).
  2. In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
  3. Replacing the file will automatically mean the linked copies use it.
  4. Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together

  1. One teacher adds a course repository block to the course
  2. Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
  3. The content of the repository can be edited via the "Course repositories" block.
  4. All of the teachers in that course can now see that appear in their filepickers and select files
  5. Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.

  1. This will not be possible, because the assignment will not allow linking to files.
  2. The student is forced to upload a copy of a file and this is protected by the assignment module.

Solution summary

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

  1. Extend Files API with a new "source" concept, with all files now having a "sourcerepositoryid" (the repository instance that the file came from) and a "sourcereference" (the address in that repository of the content of the file).
  2. All files copied into Moodle have a "sourcerepositoryid" of "0", but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file).
  3. Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
  4. Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem browser.
  5. Virtual files are served via pluginfile.php URLs just like normal files:
    1. pluginfile.php uses module callback to determine access (as now), and if it passes then
    2. pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
    3. If the sourcerepository is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
    4. The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
    5. If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
  6. When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
    1. Garbage cleanups
    2. Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.


File picker walk-through

File picker using repository reference SD.png

  1. User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
  2. User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
  3. A. File picker will ask repository plugin for customised repository UI if supported
  4. B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
  5. User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface File+picker+options.png
    • C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
    • C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
  6. D. If "Use the latest version" selected, Repository API will download the file as usual but store it in special area in file pool for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
  7. E. Repository API ask File API to create a file with content hash and UUID. File API stores the UUID and repository instance id in `mdl_files_source` table and returns the file URL which looks the same as any other Moodle file URL.
  8. File picker gives the file URL to TinyMCE
  9. When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
  10. Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

File request walk-through

File Request SD.png

  1. A. User request a file
  2. B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
  3. C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
  4. D. File API passes raw file reference information to repository plugin
  5. E. Repository plugin will firstly check if file available locally
  6. F. If not repository plugin will use file reference information to grab the file
  7. G. Repository API returns file content to File API to serve the file

Database changes

We can create another table for left joining, this requires File API query this table when locating files:

  • New file table `mdl_files_source`
    • `id` primary key
    • `fileid` foreign key of `mdl_files`.`id`
    • `repositoryid`
    • `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.
  • `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
    • `id` primary key
    • `userid` (0 if it's guest),
    • `timeaccess`,
    • `fileid`

File API changes

  • pluginfile.php: when file isn't local file (`sourcerepositoryid` isn't zero), pluginfile.php should ask repository plugin‘s callback to fetch file contents, Repository API decide return the cached copy or fresh contents
  • draftfile.php should do the same as pluginfile.php
  • When call file_prepare_draft_area(), it should keep linking files' repository information.
  • Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
  • new method: file_storage::create_file_from_reference
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
  • Cron: we probably need a Repository API function to cache/update external files

Repository API changes

  • Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE, repository plugin needs to declare what types of files are supported
  • When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, `sourcerepositoryid` will be repoistory instance id, `sourcereference` will be a reference, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference
  • Making repository plugin upgrade and versioning possible, repository plugin may need to update `sourcereference` in `mdl_files` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)
  • When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function
  • Cron will use repository plugin callbacks to clean up cache files

Repository plugins changes

  • Server files: store file parameters in `reference` field
  • Private files: same as server files plugin
  • Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
  • Flickr private: needs flickr secret and token and photo id to locate the files
  • File system: store file path in `reference` field
  • s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.

Filepicker Javascript API for customizing

  • File picker should be able to dislable file references by taking an option
  • Support <object> tag in filepicker container
  • Provide Javascript API to allow plugin communicate with filepicker

File manager to handle virtual files

Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area