View source for Development:Repository API

{{Moodle_2.0}}

The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document.  However, if you have questions to ask, problems to report or major changes to suggest, please add them to the [[Development_talk:Repository_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the further development and fix all kinds of problems.

Note that parts of this document have been now split off into a separate [[Development:File_API]]

==Objectives==

# Allow all Moodle users to easily bring content into Moodle from external repositories
# Provide a consistent interface to any external repository, for any Moodle module

==Use cases==

===Teacher adding an external file as a new resource===

# Teacher wants to add a new resource to a course 
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# File is COPIED into Moodle and stored by the resource module
# File is marked as owned by that user
# Whenever someone wants to view that file, the resource module controls access  (see [[Development:File API]] )

===Teacher linking to an external file as a new resource (think video repository) ===

# Teacher wants to display a file in the repository 
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# Link to the file is COPIED into Moodle and stored by the resource module
# Link is marked as owned by that user
# Whenever someone wants to follow that link, the resource module controls access  (see [[Development:File API]] )

===Student submitting an assignment===
# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories ([https://docs.moodle.org/en/Image:Filepicker_login.jpg file picker login], [https://docs.moodle.org/en/Image:Filepicker_browser.jpg file picker browser], [https://docs.moodle.org/en/Image:Filepicker_search.jpg file picker search])
# Student chooses MySpace from the list
# Student is prompted to enter MySpace username/password (if admin allows it, a checkbox could be there to "remember this for next time" but remember security)
# Student sees their files in MySpace and chooses one or more
# Files are copied from MySpace to Moodle 
# Assignment module controls the permissions so that only the Student and assignment graders can see the file (other students would not have permission).

===Student attaching an image to a forum===
# Student needs to attach an image and presses the "Choose files" button in the posting screen
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses Mahara from the list
# Student is prompted to enter Mahara username/password
# Student sees their files in Mahara and chooses one image
# Image is copied to Moodle 
# Image file is attached to forum post by Forum module (by reference)
# Forum module controls permissions so that anyone who can read that forum can see that file

===Student attaching the same image in another forum===

# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses "Local files" from the list and sees all the files they've permission to use
# A COPY of the image file is attached to forum post by Forum module
# Forum module controls access to this file.



Please add more use cases in this same format

==Mock screenshots==
When you first call up the file picker and choose a repository, you might be asked to log in (if saving of passwords is not allowed):

[[Image:Filepicker_login.jpg]]

Browsing files could look something like this:

[[Image:Filepicker_browser.jpg]]

And you can also search:

[[Image:Filepicker_search.jpg]]

==General architecture==

Each repository plugin (a standard Moodle plugin stored under /repository/xxx) will subclass the standard API and override methods specific to that repository.

As is usual in Moodle, there will be admin settings to disable/enable certain repository plugins as standard, as well as user settings so that users can add their own personal repositories to the standard list (eg [http://briefcase.yahoo.com Yahoo Briefcase] or [http://docs.google.com Google Docs]) and to select their default repository.

Once a repository has been used the file will usually be copied into Moodle there and then.  However there will also be options to:
* only return the URL to the file if it's desired to keep it external (but this does present security and integrity risks), or
* refresh the local file copy regularly and automatically
* refresh the file manually if desired

Once in Moodle, it is subject to the [[Development:File API]] for access control like any other file.

==Repository requirements==

From the Moodle point of view, each repository is just a hierarchy of nodes.

The repository MUST provide:
# A URI to download each node (eg file).
# A list of the nodes (eg files and directories) under a given node (eg directory).  This allows Moodle to construct a standard browse interface (much like a standard OS file picker).

The repository can OPTIONALLY:
# Require some authentication credentials 
# Provide more metadata about each node (mime type, size, dates, related files, dublin core stuff, etc)
# Describe a search facility (so that Moodle can construct a search form)
# Provide copyright and usage rules (or just information about the rules)

==Repository plugins==

Some plugins I'd like to see developed for the first version are:
* box - an interface to [http://box.net box.net]
* mahara - an interface to a Mahara installation
* Server Files - very similar to the current course-based file manager, except user-based
* Remote Moodle - an interface to another Moodle site, accessed over a secure mnet connection
* googledocs - an interface to [http://docs.google.com Google Docs]
* s3 - an interface to [http://www.amazon.com/gp/browse.html?node=16427261 Amazon S3]
* flickr - an interface to [http://flickr.com flickr]
* WebDAV - to access arbitrary external WebDAV servers
* merlot - an interface to the learning materials in [http://www.merlot.org/merlot/materials.htm Merlot.org]
* File System - a plugin to list files on local file system, of course, you can mount remote files to this local directory
* youtube - an interface to [http://youtube.com YouTube]
* jsr170 - an interface that can talk to anything that supports jsr170 (eg [http://www.alfresco.com/ Alfresco])
* oki - an OKI emulator allowing us to access things with OKI interfaces,like [http://www.fedora.info/ Fedora]
* briefcase - an interface to [http://briefcase.yahoo.com/ Yahoo Briefcase]
* myspace - an interface to MySpace files (perhaps via [http://www.programmableweb.com/api/myspace this MySpace API])
* skydrive - an interface to Microsoft's [http://skydrive.live.com/ SkyDrive] files
* Dropbox - an interface to Dropbox files [http://www.dropbox.com]
* facebook - an interface to Facebook files
* [http://www.dspace.org/ Dspace] - a repository from MIT
* DOOR - another popular open source repository
* SMB shares - An interface for windows shares e.g. personal folders on network drives. Would need to link with LDAP as usernames will often be wholly/partially the same as network folder names. This could be done using SAMBA, but would also need to work on windows machines natively. See [http://moodle.org/mod/data/view.php?d=13&rid=991 this block] for a linux implementation.

==Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field''' 
|'''Type''' 
|'''Default''' 
|'''Info''' 

|-
|'''id'''
|int(10)
|
|autoincrementing 

|-
|'''type'''
|varchar(255)
|
|The type of the repository 

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field''' 
|'''Type''' 
|'''Default''' 
|'''Info''' 

|-
|'''id'''
|int(10)
|
|autoincrementing 

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid''' 
|int(10)
| 
|The id of repository type

|-
|'''userid''' 
|int(10)
| 
|The person who created this repository instance

|-
|'''contextid''' 
|int(10)
| 
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
| 
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
| 
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field''' 
|'''Type''' 
|'''Default''' 
|'''Info''' 

|-
|'''id'''
|int(10)
|
|autoincrementing 

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

===File types===

The context at which someone is inserting a file may require certain file types (eg uploading a new user profile image is only looking for images).  

To support this, the calling code needs to be able to specify the required mimetypes, and the listing code should be able to filter the results based on these mimetypes.  Ideally the repository itself can do the filtering for ultimate speed (though not all repositories will support this).

We will have to develop special new mimetypes for Moodle files like backups (application/vnd.moodle.backup) and IMS learning design (application/vnd.moodle.imsld) etc

==Technical walkthrough==

(See also the functional spec for the [[Development:Repository_File_Picker]] )

There are two main cases where the repository API will be used: as part of a Moodleform to add a file and as part of the HTML editor to add a media element into some HTML).  We also have to cater for the using Moodleforms without Javascript.

In all of these cases the files will be uploaded to Moodle while using the file picker dialog and stored in a temporary file area owned by the currently active user.  It is only AFTER the submission of the entire Moodleform that we will know the full context, itemids to store the file properly, so at this time the file will be copied into the correct filearea.

===Case 1: As part of a Moodleform with Javascript===

1. Moodle module code calls a "filepicker" moodleform item whenever a file is required, which includes the following information to pass to the File API:

 eg $mform->addElement('filepicker', 'uniqueelementid', $fullname, $data)
 
2. When rendering the form, Moodle will display a read-only filename field with an '''"Add file"''' button next to it.  There will also be a hidden field to store a file reference later (this is what actually gets used, the filename field is just for users to see something).

3. When the add file button is pressed, the form will be "replaced" in the page by a larger resizeable area containing an AJAX file picker.  (After picking the display can be closed).   (There could be a user option to make this a popup window instead, if required)

4. The AJAX file picker interface will list all the active repositories as a menu, and list files in one of several formats (like Windows/Mac/Linux): Details, Names, Icons.

5. For each plugin, the AJAX interface will prompt the user to login first (if required) asking the plugin to log in behind the scenes.  It'll also ask the plugin to return listing data in response to clicks and searches.  

6. Finally, when the user selects a file and clicks the "Select" button, the AJAX interface will trigger a method in the plugin that will fetch the file and call the File Storage API to '''store''' the file using the '''uniqueelementid''' and the current user info.  While this is happening, the interface should show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.  

7. After a file has finally been selected we will have a file ID which we can pass back to the original Moodle form (to the hidden field named '''uniqueelementid_formid''').  The picker can then rename the read-only filename field before it hides itself.

8. Submitting the form will trigger the mform processing for this field, which will check fields, create things in the module etc.  Once this has been finally successful the developer must call an mform function to "fix" the info for each file and "move" it into the module file area:

  eg $mform->store_local_file('uniqueelementid', $context, $filearename, $itemid, $filepath);

9. Cron jobs in File Storage api should automatically delete any files in the user's tempfile area that are older than 7 days or move them into a trash can in the user's file area (perhaps).

===Case 2: As part of a Moodleform without Javascript===

Steps 1-2 are the same as for the case with Javascript.

3. The add file button is a submit button for the form with a different value.  When the add file button is pressed,
* the whole form will be ''submitted'' to the original location (but with a different submit button value)
* moodleforms get_data() will detect this is a "repository save" and can save the full POST info in the current session tagged with the id of the openfile element, together with the URL to return to
* moodleforms get_data() then redirects the user to a new page showing the main picker interface

4. The file picker interface will have to be a completely new and separate interface from the AJAX one.  It could be a long hierarchy listing, or reload a lot.

5. Finally, when the user selects a file and submits using the "Select" button to picker.php, it will trigger a method in the plugin that will fetch the file and call the [[Development:File_API|File API]] to store the file using the filearea and context information we already had.   While this is happening, the interface can show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

6. After this, picker.php will redirect/continue back to the original form page.  The form can be constructed as usual, however, when the form is rendered using display() method moodleforms should now look for relevant saved content in the session and use that to override any content in the form (and then delete the saved info in the session).

Steps 8-9 are the same as for the case with Javascript.

===Case 3: As part of a HTML editor===

The key thing here is a move away from storing any absolute URLs to files in our HTML texts.  Instead we'll store relative names.

1. The moodleform for a textarea (HTML editor) will require a path to the filearea associated with this HTML.  eg '''wwwroot/pluginfile.php/13/content/0/'''.  This would have to be the user_draft area if the filearea doesn't exist yet  eg '''wwwroot/draftfile.php/userid/tempfile/uniquelementid'''

2. All textarea content will also need to have str_replace done on it to replace @@pluginfile@@/somefilenames.jpg in the content to use this path so that it comes up right in the editor.  (Note this also needs to be done on every format_text command too when showing this text.)

3. The path parameter also needs to be added to the editor configuration in the current page.

4. Editor plugins can be modified to look for these variables in the editor configuration.

5. When adding an image or other media element,  the same AJAX repository picker will show up as a popup div to allow people to pick from any repository and choose files to download.  The repository picker is responsible for downloading the file in real-time, storing it as a user temporary file if the filearea doesn't already exist, prefixing the supplied path to the filename and returning a URL back to the dialog text input before closing.

6. On submission, and after the HTML is stored, we might now have a new permanent filearea, so we'll need to update any associated temporary files to make sure they have the proper file area information.

==Repository plugins==

Each repository plugin is required to contain the following elements:

===class repository()===

This class implements the interface to a particular repository, for browsing, selecting and updating files.  The base class (repository) is defined in /repository/lib.php, while each repository defines an inherited class (eg repository_alfresco) in /repository/repositoryname/repository.class.php

Repositories can redefine any of these methods as required (and in some instances, MUST redefine them):

====__construct($repositoryid, $contextid, $options=array(), $readonly)====

'''MUST''' redefine

Accept necessary parameters, and do initialization of repository.

====get_file($url, $file = '')====

Given a URL, download a file from there, save the file in a temporary directory.

====get_link($info)====
Get the url of external resource

====get_listing($path='/', $page='''''''')====

Given a path, and perhaps a search, get a listing of files. In the case of AJAX file picker, this function should return json format Javascript array.

====search($keyword)====
Search repository by given keyword, it will return an array of the same format of get_listing

====print_login()====

Show the login screen, if required. In the case of AJAX file picker, this function should return json format array which defined the login form.

====print_search==== 

Print the search form, it will return a json string

====get_meta()====
Return information for creating ajax request, it is private function, you don't need to rewrite it.

====create()====
Create an instance

====delete()====
Delete this instance from `repository` table

====hide()====
Hide a repository instance from file picker list

====set_option()====
set options in data1-data5 fields, can be overrided

====get_option()====
get option list or a specific option from database

====get_type_option_names()====
If this plugin needs admin settings, please refine this function to return option names.

====type_config_form()====
If get_type_option_names return non empty array, this function '''MUST''' redefine, it will help to build the setting form.

====get_instance_option_names()====
If plugin instance needs settings, this function will return instance option names.

====instance_config_form()====
If get_instance_option_names return non empty array, this function '''MUST''' redefine, it will help to build the instance setting form.


====supported_filetypes()====
What file types are supported by this repository plugin, it will return an array, the file type name is defined in a [http://freemind.sourceforge.net/wiki/index.php/Main_Page freemind] file in lib/file/file_types.mm

====supported_returntypes()====
The repository plugin could support external link or copying files to moodle. If the plugin support file link only, developer should override this function to return FILE_EXTERNAL, if plugin support copying file only, it should return FILE_INTERNAL, by default, plugin supports both.

====filter()====
Filter file listing to exclude specific file types

===icon.png===

A logo that represents the repository.  Ideally square but we should handle all sizes.


==See also==

* [[Development:Repository Administration Specification]]
* [[Development:Repository Interface for Moodle/Course/User]]
* [[Development:Repository plugins]]
* [[Development:Repository File Picker]]
* [[Development:File API]]
* [[Development:Portfolio API]]
* MDL-13766 and MDL-16543 Repository API Meta issues

[[Category:Repositories]]

[[ja:開発:リポジトリAPI]]