Development:Repository API

Template:Moodle 2.0This page describes the specification for a future feature, currently being worked on for Moodle 2.0. This spec is STILL UNDER CONSTRUCTION.

See MDL-13766 to track the status of the implementation.

The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions to ask, problems to report or major changes to suggest, please add them to the page comments, or start a discussion in the Repositories forum. We'll endeavour to merge all such suggestions into the main spec before we start development.

Note that parts of this document have been now split off into a separate Development:File_API

Objectives

1. Allow all Moodle users to easily bring content into Moodle from external repositories
2. Provide a consistent interface to any external repository, for any Moodle module

Use cases

Teacher adding an external file as a new resource

1. Teacher wants to add a new resource to a course
2. Teacher clicks the "Choose a resource" button
3. Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
4. Teacher chooses a file in an external repository
5. File is COPIED into Moodle and stored by the resource module
6. File is marked as owned by that user
7. Whenever someone wants to view that file, the resource module controls access (see Development:File API )

Teacher linking to an external file as a new resource (think video repository)

1. Teacher wants to display a file in the repository
2. Teacher clicks the "Choose a resource" button
3. Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
4. Teacher chooses a file in an external repository
5. Link to the file is COPIED into Moodle and stored by the resource module
6. Link is marked as owned by that user
7. Whenever someone wants to follow that link, the resource module controls access (see Development:File API )

Student submitting an assignment

1. Student needs to submit an assignment and presses the "Choose files" button
2. Student sees a "file picker" where they can see files listed on any of several configured repositories (file picker login, file picker browser, file picker search)
3. Student chooses MySpace from the list
4. Student is prompted to enter MySpace username/password (if admin allows it, a checkbox could be there to "remember this for next time" but remember security)
5. Student sees their files in MySpace and chooses one or more
6. Files are copied from MySpace to Moodle
7. Assignment module controls the permissions so that only the Student and assignment graders can see the file (other students would not have permission).

Student attaching an image to a forum

1. Student needs to attach an image and presses the "Choose files" button in the posting screen
2. Student sees a "file picker" where they can see files listed on any of several configured repositories
3. Student chooses Mahara from the list
4. Student is prompted to enter Mahara username/password
5. Student sees their files in Mahara and chooses one image
6. Image is copied to Moodle
7. Image file is attached to forum post by Forum module (by reference)
8. Forum module controls permissions so that anyone who can read that forum can see that file

Student attaching the same image in another forum

1. Student needs to submit an assignment and presses the "Choose files" button
2. Student sees a "file picker" where they can see files listed on any of several configured repositories
3. Student chooses "Local files" from the list and sees all the files they've uploaded before
4. A COPY of the image file is attached to forum post by Forum module
5. Forum module controls access to this file.

Please add more use cases in this same format

Mock screenshots

When you first call up the file picker and choose a repository, you might be asked to log in (if saving of passwords is not allowed):

Browsing files could look something like this:

And you can also search:

General architecture

Each repository plugin (a standard Moodle plugin stored under /repository/xxx) will subclass the standard API and override methods specific to that repository.

As is usual in Moodle, there will be admin settings to disable/enable certain repository plugins as standard, as well as user settings so that users can add their own personal repositories to the standard list (eg Yahoo Briefcase or Google Docs) and to select their default repository.

Once a repository has been used the file will usually be copied into Moodle there and then. However there will also be options to:

• only return the URL to the file if it's desired to keep it external (but this does present security and integrity risks), or
• refresh the local file copy regularly and automatically
• refresh the file manually if desired

Once in Moodle, it is subject to the Development:File API for access control like any other file.

Repository requirements

From the Moodle point of view, each repository is just a hierarchy of nodes.

The repository MUST provide:

1. A URI to download each node (eg file).
2. A list of the nodes (eg files and directories) under a given node (eg directory). This allows Moodle to construct a standard browse interface (much like a standard OS file picker). However some repository plugins may choose to completely override the repository_browse() method and implement their own interface, that's OK, as long as they end up with a URL for the file.

The repository can OPTIONALLY:

1. Require some authentication credentials
2. Provide more metadata about each node (mime type, size, dates, related files, dublin core stuff, etc)
3. Describe a search facility (so that Moodle can construct a search form)
4. Provide copyright and usage rules (or just information about the rules)

Repository plugins

Some plugins I'd like to see developed for the first version are:

• local - very similar to the current course-based file manager, except user-based
• moodle - an interface to another Moodle site, accessed over a secure mnet connection
• jsr170 - an interface that can talk to anything that supports jsr170 (eg Alfresco)
• oki - an OKI emulator allowing us to access things with OKI interfaces,like Fedora
• briefcase - an interface to Yahoo Briefcase
• myspace - an interface to MySpace files (perhaps via this MySpace API)
• googledocs - an interface to Google Docs
• s3 - an interface to Amazon S3
• skydrive - an interface to Microsoft's SkyDrive files
• box - an interface to box.net
• facebook - an interface to Facebook files
• merlot - an interface to the learning materials in Merlot.org
• flickr - an interface to flickr
• youtube - an interface to YouTube
• mahara - an interface to a Mahara installation
• Dspace - a repository from MIT
• DOOR - another popular open source repository
• SMB shares - An interface for windows shares e.g. personal folders on network drives. Would need to link with LDAP as usernames will often be wholly/partially the same as network folder names. This could be done using SAMBA, but would also need to work on windows machines natively. See this block for a linux implementation.
• WebDAV - to access arbitrary external WebDAV servers

Tables

repository

This table contains one entry for every configured external repository instance.

Field	Type	Default	Info
id	int(10)		autoincrementing
repositoryname	varchar 255		A custom name for this repository (non-unique)
repositorytype	varchar(255)		The name of the plugin being used (1 special case: "local" which covers all files uploaded from desktop)
userid	int(10)		The person who created this repository instance
contextid	int(10)		The context that this repository is available to ( = system context for site-wide ones)
username	varchar(255)		username to log in with, if required (almost never!)
password	varchar(255)		password to log in with, if required (almost never!)
data1	text		Other information useful to the plugin
data2	text		Other information useful to the plugin
data3	text		Other information useful to the plugin
data4	text		Other information useful to the plugin
data5	text		Other information useful to the plugin
visible	tinyint(1)	1	Disable/enable a instance
timecreated	int(10)		The time this repository was created
timemodified	int(10)		The last time the repository was modified

File types

The context at which someone is inserting a file may require certain file types (eg uploading a new user profile image is only looking for images).

To support this, the calling code needs to be able to specify the required mimetypes, and the listing code should be able to filter the results based on these mimetypes. Ideally the repository itself can do the filtering for ultimate speed (though not all repositories will support this).

We will have to develop special new mimetypes for Moodle files like backups (application/vnd.moodle.backup) and IMS learning design (application/vnd.moodle.imsld) etc

Technical walkthrough

(See also the functional spec for the Development:Repository_File_Picker )

There are two main cases where the repository API will be used: as part of a Moodleform to add a file and as part of the HTML editor to add a media element into some HTML). We also have to cater for the using Moodleforms without Javascript.

In all of these cases the files will be uploaded to Moodle while using the file picker dialog and stored in a temporary file area owned by the currently active user. It is only AFTER the submission of the entire Moodleform that we will know the full context, itemids to store the file properly, so at this time the file will be copied into the correct filearea.

Case 1: As part of a Moodleform with Javascript

1. Moodle module code calls a "filepicker" moodleform item whenever a file is required, which includes the following information to pass to the File API:

eg $mform->addElement('filepicker', 'uniqueelementid', $fullname, $data)

2. When rendering the form, Moodle will display a read-only filename field with an "Find file" button next to it. There will also be a hidden field to store a file reference later (this is what actually gets used, the filename field is just for users to see something).

3. When the add file button is pressed, the form will be "replaced" in the page by a larger resizeable area containing an AJAX file picker. (After picking the display can be closed). (There could be a user option to make this a popup window instead, if required)

4. The AJAX file picker interface will list all the active repositories as a menu, and list files in one of several formats (like Windows/Mac/Linux): Details, Names, Icons.

5. For each plugin, the AJAX interface will prompt the user to login first (if required) asking the plugin to log in behind the scenes. It'll also ask the plugin to return listing data in response to clicks and searches.

6. Finally, when the user selects a file and clicks the "Select" button, the AJAX interface will trigger a method in the plugin that will fetch the file and call the File Storage API to store the file using the uniqueelementid and the current user info. While this is happening, the interface should show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

7. After a file has finally been selected we will have a file ID which we can pass back to the original Moodle form (to the hidden field named uniqueelementid_formid). The picker can then rename the read-only filename field before it hides itself.

8. Submitting the form will trigger the mform processing for this field, which will check fields, create things in the module etc. Once this has been finally successful the developer must call an mform function to "fix" the info for each file and "move" it into the module file area:

 eg $mform->store_local_file('uniqueelementid', $context, $filearename, $itemid, $filepath);

9. Cron jobs in File Storage api should automatically delete any files in the user's tempfile area that are older than 7 days or move them into a trash can in the user's file area (perhaps).

Case 2: As part of a Moodleform without Javascript

Steps 1-2 are the same as for the case with Javascript.

3. The add file button is a submit button for the form with a different value. When the add file button is pressed,

• the whole form will be submitted to the original location (but with a different submit button value)
• moodleforms get_data() will detect this is a "repository save" and can save the full POST info in the current session tagged with the id of the openfile element, together with the URL to return to
• moodleforms get_data() then redirects the user to a new page showing the main picker interface

4. The file picker interface will have to be a completely new and separate interface from the AJAX one. It could be a long hierarchy listing, or reload a lot.

5. Finally, when the user selects a file and submits using the "Select" button to picker.php, it will trigger a method in the plugin that will fetch the file and call the File API to store the file using the filearea and context information we already had. While this is happening, the interface can show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

6. After this, picker.php will redirect/continue back to the original form page. The form can be constructed as usual, however, when the form is rendered using display() method moodleforms should now look for relevant saved content in the session and use that to override any content in the form (and then delete the saved info in the session).

Steps 8-9 are the same as for the case with Javascript.

Case 3: As part of a HTML editor

The key thing here is a move away from storing any absolute URLs to files in our HTML texts. Instead we'll store relative names.

1. The moodleform for a textarea (HTML editor) will require a path to the filearea associated with this HTML. eg wwwroot/pluginfile.php/13/content/0/. This would have to be the user_draft area if the filearea doesn't exist yet eg wwwroot/draftfile.php/userid/tempfile/uniquelementid

2. All textarea content will also need to have str_replace done on it to replace @@pluginfile@@/somefilenames.jpg in the content to use this path so that it comes up right in the editor. (Note this also needs to be done on every format_text command too when showing this text.)

3. The path parameter also needs to be added to the editor configuration in the current page.

4. Editor plugins can be modified to look for these variables in the editor configuration.

5. When adding an image or other media element, the same AJAX repository picker will show up as a popup div to allow people to pick from any repository and choose files to download. The repository picker is responsible for downloading the file in real-time, storing it as a user temporary file if the filearea doesn't already exist, prefixing the supplied path to the filename and returning a URL back to the dialog text input before closing.

6. On submission, and after the HTML is stored, we might now have a new permanent filearea, so we'll need to update any associated temporary files to make sure they have the proper file area information.

Repository plugins

Each repository plugin is required to contain the following elements:

class repository()

This class implements the interface to a particular repository, for browsing, selecting and updating files. The base class (repository) is defined in /repository/lib.php, while each repository defines an inherited class (eg repository_alfresco) in /repository/repositoryname/repository.class.php

Repositories can redefine any of these methods as required (and in some instances, MUST redefine them):

__construct($repositoryid, $contextid, $options=array())

MUST redefine

Accept necessary parameters, and do initialization of repository.

get_file($url)

Given a URL, download a file from there, save the file in a temporary directory.

get_listing($parent='/', $search=''')

Given a path, and perhaps a search, get a listing of files. In the case of AJAX file picker, this function should return json format Javascript array.

print_login()

MUST redefine

Show the login screen, if required. In the case of AJAX file picker, this function should return json format array which defined the login form.

print_listing

Get a listing from get_listing, print it or return HTML code.

print_search

MUST redefine

Print the search form

@TODO: Think about are we really need it?

store_login($username, $password, $userid=''')

If you do want to cache login details for various repositories, then use this method.

cron()

Defines operations that happen occasionally on cron.

ajax_info()

Return information for creating ajax request

create()

Create an instance

delete()

Delete this instance from `repository` table

hide()

Hide a repository instance from file picker list

set_option()

set options in data1-data5 fields, can be overrided

get_option()

get option list or a specific option from database

has_admin_config

If this plugin need admin settings, please refine this function to return true

admin_config_form

if has_admin_config return true, this function MUST redefine, it will help to build the setting form.

get_option_names

if has_admin_config return true, this function MUST redefine, it will return option names

icon.png

A logo that represents the repository. Ideally square but we should handle all sizes.

See also

• Development:Repository Administration Specification
• Development:Repository Interface for Moodle/Course/User
• Development:Repository Plugins
• Development:Repository How to Create Plugin
• Development:Repository File Picker
• Development:File API
• Development:Portfolio API