Portfolio API
| Warning: This page describes the original specification for the portfolios feature. The actual implementation currently existing in Moodle may differ from this specification. When adding support for portfolios to your plugins, you can find the page Adding a Portfolio Button to a page useful. |
Moodle 2.0
Overview
The Portfolio API is a core set of interfaces that all Moodle code will/should use so that we can easily publish files to all kinds of external document repository systems.
It's important to remember that portfolios are generally treated as WRITE-ONLY. All we are doing in Moodle is grabbing stuff and pushing it out to somewhere. Management of the files and further combining/reflecting is done through the native interface provided by the portfolio system. Reading of files from a repository is handled by the Repository API.
A typical user story:
- When portfolios are enabled, every page or major piece of content in Moodle has a little "Save" button beside it.
- User clicks one of these buttons
- User is able to choose from a list of configured portfolios (this step will be skipped if there's only one).
- User may be asked to define the format of the captured content (eg pdf, IMS LD, HTML, XML ...)
- User may be asked to define some metadata to go with the captured content (some will be generated automatically).
- The content and metadata is COPIED to the external portfolio system
- User has an option to "Return to the page you left" or "Visit their portfolio".
Note this will be just as useful for teachers as for students.
The formatting possibilities will vary depending on the context of the button and the type of external portfolios. So for example, the "Save" button on the course page would allow the user to capture the whole course in IMS LD or Moodle backup format, which you would not have on a forum page.
Architecture
Here is how it will work:
Plugins and libraries
There will be one type of plugins
- Portfolio (eg Mahara/Elgg/OSP/Facebook/Download) - this will be portfolio/type/xxx
The transport layer (eg mnet/http/scp/cp/dav etc) or clients (eg box.net/flickr) will be written as libraries, to be shared by both repository and portfolio.
Then there will be different formats that plugins will support (and the part of moodle exporting content must support as well), eg IMS, moodle native, mahara native, pdf, encrypted pdf. These will have good libraries supporting them.
Admin
It is important to be allowed to have multiple instances of (some) plugins. The workflow for adding a new one is:
- Admin navigates to portfolio config
- Selects from the list of available portfolio plugins, and clicks 'add a new external portfolio' (some may be disabled if there is an instance already and the plugin doesn't support multiple instances)
- Configure the plugin - select which transport and content types to use if there are multiple supported and installed, urls, authentication keys, etc.
- Set permissions (maybe) - handled by roles.
This is not necessary for every type of portfolio, because many will just require the user to authenticate directly and if we do ever want to retain settings for each user we just use user preferences.
Exporting
- User is viewing a page that calls new portfolio_add_button(). This checks to see if there are any configured portfolio plugin instances, and also (maybe) any permissions related to portfolios, and what the user's portfolio settings are, and then displays either a single 'add to portfolio' button, or a drop down menu of the available systems with the add button.
- When this button is pressed, the user is redirected to portfolio/add.php, with some post data containing the responsible area (activity module or something like course or blog) callback file and callback arguments, as well as optionally some information about what type of content it is.
- On this page, the user is presented with a form to enter metadata about the item, and configure any options. At this point if there are multiple formats available for export (based on the intersection of what the plugin and module support), the user can select which format they want. The plugin and module can both export mform elements for this page. The user can at this point also select to send the data and wait (with a warning it might take awhile), or queue it for processing if it's larger. This is determined by the size of the content to be exported.
- When the user has submitted the form, they are displayed a summary of what they're about to export, with 'confirm' and 'cancel' buttons. Cancel cancels the request, and cleans up any temporary data, and returns the user to where they came from, while confirm goes to the next step.
- At any point, the portfolio plugin might need to take control for a step. For example, facebook or flickr might require the user to log in for the first time and confirm moodle is allowed to access their API.
- When the user has confirmed their summary, a 'portfolio_send' event will be triggered. At this point, one of two things happen.
- If the user has elected to wait, the 'instant' event is fired, and when the caller gets control again, it displays the status to the user.
- If the user has elected to queue, the delayed event is fired and the user is notified.
- The user is given the option to continue to their portfolio, or return to where they were
- When the event is handled (either through the cron or instant event), the following happens:
- The event handler is invoked. This reawakens the transfer and defers control to the caller and then the portfolio to prepare and send the package.
- When this is complete, we return control to the event handler (which, if it's an 'instant' one, will return true to the caller.
Storage
Obviously during this process, state is going to be lost between webserver requests and also between user input and event handling. All of the data is stored in the database, in the form of a serialized (and base64 encoded) representation of the exporter, plugin and caller objects.
Files are also going to be written during the preparation stage of the export, and these are stored in a special portfolio area using the new files api.
Access/Permissions
- The calling code is responsible for performing the permission checks necessary before asking to display any button, but during the export the portfolio code will call a check_permissions function on the caller object.
- I would really like to be able to make some portfolio instances available to some roles but this has fallen out of scope.
Event API
The portfolio code uses the event api to handle queued events and there is one entry point for this that reawakens the transfer objects and resumes the transfer. Additionally, portfolio plugins can subscribe to events like any other part of moodle.
Technical
Abstract Portfolio Baseclass: portfolio_plugin_base
Mixes providing some basic functionality by means of its own functions, with a number of abstract functions plugins must implement, and with some functions that plugins can also optionally override.
See also: Writing_a_Portfolio_Plugin for a full list of all methods you must/can/shouldn't override, as well as associated instructions for what else you need to do to create a new portfolio plugin
Abstract Caller Baseclass : portfolio_caller_base
Whenever somewhere in Moodle wants an 'add to portfolio' button, they must subclass this.
See also: Adding_a_Portfolio_Button_to_a_page for a full list of all methods you must/can/shouldn't override as well as the associated instructions for how to call portfolio_add_button.
Database Tables
The actual information about plugins that are installed is just stored in mdl_config_plugin.
Additionally, as we're configuring instances of plugins, rather than just one config set per plugin, we're not using mdl_config_plugin, but instead our own set of tables:
portfolio_instance:
| Field | Datatype | Comment |
| id | integer | sequence |
| plugin | varchar(50) | name of plugin (should match directory in portfolio/type) |
| name | varchar(255) | name of this plugin instance |
| visible | smallint | 0 or 1 |
portfolio_instance_config:
| Field | Datatype | Comment |
| id | integer | sequence |
| instance | integer | (pseudo)fk to portfolio_instance |
| name | varchar(255) | config name |
| value | text | config value |
portfolio_instance_user:
| Field | Datatype | Comment |
| id | integer | sequence |
| instance | integer | (pseudo)fk to portfolio_instance |
| userid | integer | (pseudo)fk to mdl_user |
| name | varchar(255) | config name |
| value | text | config value |
portfolio_log:
| Field | Datatype | Comment |
| id | integer | sequence |
| userid | integer | (pseudo) fk to mdl_user |
| time | integer | unix timestamp of transfer |
| portfolio | integer | (pseudo) fk to mdl_portfolio_instance |
| caller_class | varchar(150) | name of caller class (used in the case of duplicates to display information) |
| caller_file | varchar(255) | file that contains the definition of caller_class |
| caller_sha1 | varchar(255) | sha1 information of export |
portfolio_tempdata
| Field | Datatype | Comment |
| id | integer | sequence |
| data | text | serialized representation of export data |
| expirytime | integer | time this data (and the transfer) expires (and the record (and associated files)) will be deleted |
| userid | integer | psuedo fk to mdl_user |
All plugins can also implement their own database tables as needed, by creating a db/install.xml and db/upgrade.php inside portfolio/type/xxx/ (See Writing_a_Portfolio_Plugin) for more information.
Portfolio Plugins
- mahara (will be done for the initial implementation)
- download (will be done for the initial implementation)
- box.net (will be done for the initial implementation)
- flickr (Nico has been writing this but it is incomplete)
- googledocs (I think DanP has been writing this)
transport types and formats should be able to be found in a shared location for multiple plugins of both portfolio and repository type to use, but also might be specific to one type of plugin which means that moodle should support looking in multiple locations for these plugins. (eg mahara native format would be in the mahara portfolio plugin, but pdf format will be in a shared library)
Possible Transport Types
- mnet (will be done for the initial implementation as part of the Mahara Portfolio Plugin)
- download (just uses send_file_* functions)
- http
- filesystem (could be local/nfs/samba whatever) (cp)
- ssh based (eg scp - should find and re-use the elgg block code as it deals with using ssh keys nicely)
- webdav
- open social? (http://code.google.com/apis/opensocial/)
Possible Export Formats
(Note that we don't necessarily want more than one of these for the initial implementation)
- implemented now:
- html
- image
- video
- plaintext
- 'file' (fallback)
- possibly implemented in the future
- encrypted pdf
- ims?
- LEAP2A
- moodle native?
- mahara native?
- Dublin Core (Enovation implemented this)
Testing
At this stage, the portfoliolib and button objects have tests, and the callers have tests to check whether their sha1 generation remains consistent appropriately. This includes the implicit testing of constructing the caller objects (which verifies the callback arguments).
The plugins are not currently tested and even if they were we would not be able to test interaction with the remote system.
MNET
This section has moved to MNET_Roadmap
See also MNET_API for the documentation of xmlrpc functions
Duplication of Data
Moodle will keep track of what content it transfers and when. It keeps a sha1 has of the data, so that if the user tries to export the same content, Moodle can warn the user.
It cannot, however, be responsible for how external systems deal with this case. The different plugins can do what they can. For example, mahara will create new files rather than overwrite. The box.net plugin will try very hard to rename files to avoid collisions.
Save points in Moodle
moved to http://tracker.moodle.org/browse/MDL-15758 during development
Still TODO
There are a few things still I have not been able to complete for various reasons (generally reliance on other parts of the system, eg Files API). There are bugs for all of these, but reproduced here for completeness:
- MDL-16406 - waiting on QA (Jerome)
- MDL-16048 - waiting on QA (Nico)
- MDL-16313 - this just didn't get far enough up my list and I'm still not sure how relevant it is.
- MDL-15777 - reliance on Files API - data fields that subclass data_field_file need to be extracted and copied separately into the export area using copy_existing_file - this is essentially done but I still think picture should subclass file. More info in MDL-16493
- MDL-15777 - reliance on Files API - data module can only export as CSV even though plain export also supports ods/xls as those two libraries are not updated to the new Files API. More info in MDL-15911
- MDL-16326 - reliance on Files API - 'file' resource module has not been updated to use Files API, so exporting from this type is not yet implemented (currently HTML and plaintext only)
- MDL-16175 - (currently) unreasonable reliance on exceptions. Especially for queued events, currently if a portfolio transfer is woken up at cron to be completed and an error happens in mnet (eg one remote site is down or misconfigured), the entire cronjob will die as mnet functions call print_error, which calls die(). This essentially means cron will stay broken (for all of moodle) until that transfer expires. This is not really a bug in portfolio code, but it definitely exacerbates an already brittle situation.
Unit Test TODO
Currently there's quite a few tests implemented, but outstanding are tests that rely on the generator to create files for the callers. As the generator gets updated to create this data, the portfolio unit tests will start throwing exceptions in the portfolio_exporter_text->copy_existing_file method so that it will become obvious when this needs to be updated as the tests will start failing (they are currently passing)
Current exhaustive list of export scenarios
mod/assignment
upload single file
This is pretty straightforward. It should display the export icon next to the single file (no large form/button), and the export should respect the mime-based subtypes (_IMAGE, _VIDEO etc)
upload multiple files
This one is a little more complex. You should get an export icon next to individual files, and, additionally, if there is more than one, an export form at the bottom (which will export all files). Exporting multiple files will always stop mime detection and fallback to _FILE format.
online text
Displays the full form at the bottom of the page. Should export as format _HTML.
mod/chat
These should all export as format _HTML, and contain no references back to Moodle (eg user profile images, which won't be able to be seen necessarily)
Session page
Should export entire session.
Report page per session
Should export entire session.
Report page all sessions
Should concatenate all sessions together.
mod/data
Single entry export
Should export as HTML. Any files should be included along with the html. If there is only one field in the entry and it is a file or image, the mimetype should be respected, and the export format should be based on that (eg _IMAGE)
Whole database instance export
Should export as CSV. Files are not included (This is the same as the other CSV export)
mod/forum
Whole discussion
The export format is FILE, attachments come alongside discussion.html - this could be improved later to be HTML if there are no attachments.
Single post
The export format is _HTML.
Single post with attachments
The export format is FILE as it's mixed, and attachments come alongside post.html.
Single attachment
Mimetypes should be respected and the export format should be based on them (eg _IMAGE)
mod/glossary
Single entry export
Should export the entry as HTML.
Whole glossary export
Should export the glossary as CSV - similar to a current glossary export.
mod/resource
HTML resource
Should export as _HTML.
text resource
Should export as _TEXT.
file resource
Not implemented yet. Blocked by FILES API. Should respect the mimetype of the file and export in the appropriate format.
See also
- Core APIs
- Repository API
- File API
- MDL-14591 - Portfolio API Meta issue
