Improved support for external File content

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

File synching
Project state	Planning
Tracker issue	TODO
Discussion	TODO
Assignee	Martin Dougiamas

Moodle 2.2

Problems with Files in 2.0

• It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
• It is not currently easy to create a simple shared "course repository" for teachers to use

Example use cases that will become possible

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

1. The teacher uploads the file to their private file area (or other repository).
2. In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
3. Replacing the file will automatically mean the linked copies use it.
4. Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together

1. One teacher adds a course repository block to the course
2. Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
3. The content of the repository can be edited via the "Course repositories" block.
4. All of the teachers in that course can now see that appear in their filepickers and select files
5. Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.

1. This will not be possible, because the assignment will not allow linking to files.
2. The student is forced to upload a copy of a file and this is protected by the assignment module.

Proposed solution summary

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

1. Extend Files API with a new "source" concept, with all files now having a "sourcerepositoryid" (the repository instance that the file came from) and a "sourcereference" (the address in that repository of the content of the file).
2. All files copied into Moodle have a "sourcerepositoryid" of "0", but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file).
3. Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
4. Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem browser.
5. Virtual files are served via pluginfile.php URLs just like normal files:
   1. pluginfile.php uses module callback to determine access (as now), and if it passes then
   2. pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
   3. If the sourcerepository is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
   4. The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
   5. If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
6. When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
   1. Garbage cleanups
   2. Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

Details

Database changes

• `mdl_files` table needs new fields: `sourcerepositoryid` and `sourcereference`. `sourcerepositoryid` default value is 0. `sourcereference` default value is NULL, it can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

• `mdl_files_log` table for files access log, contains fields: `userid` (0 if it's guest), `timestamp`, `fileid`, File API should have a new function to insert records to this table

File API changes

• pluginfile.php: when file isn't local file (`sourcerepositoryid` isn't zero), pluginfile.php should ask repository plugin‘s callback to fetch file contents if not cached
• draftfile.php should do the same as pluginfile.php
• When call file_prepare_draft_area(), it should keep linking files' repository information.
• Adding callback to files API delete function, if the actaully file get deleted, callback needs to check all linked files, and convert them to actually files
• new method: file_storage::create_file_from_reference

public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)

• Cron: we probably need a File API function to pre-cache external files by looking `mdl_files_log` for files never been accessed.

Repository API changes

• Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need one more type FILE_REFERENCE, if repository plugin needs to declare what types of files are supported
• When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, `sourcerepositoryid` will be repoistory instance id, `sourcereference` will be a reference, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

• Cron will use repository plugin callbacks to clean up cache files

• Making repository plugin upgrade and versioning possible, repository plugin may need to update `sourcereference` in `mdl_files` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

• When deleting repository instance, all files imported by this instance will have to be converted to actualy files, this has to be done by File API function

Filepicker Javascript API for customizing

• File picker should be able to dislable file references by taking an option
• Support <object> tag in filepicker container
• Provide Javascript API to allow plugin communicate with filepicker

File manager to handle virutal files

Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area