MoodleDocs - User contributions [en]

Enrolment API

2013-02-22T13:45:46Z

CHRISF: Example call to get_enrolled_users() was passing "true" to the $userfields argument

Since Moodle 2.0 there is a new concept of user enrolments, they are fully independent from the roles and capabilities. Capabilities are very often used in combination with enrolment status.

==What is enrolment?==

Enrolled users may fully participate in a course. Active user enrolment allows user to enter course. Only enrolled users may be group members. Grades are stored only for enrolled users.

==Unenrolment==

Unenrolment is irreversible operation that purges user participation information. Full unenrolment is suitable only if you do not need to preserve all course participation information including user grades.

==Enrolment status==

Instead of full unenrolment it is usually better to only ''suspend'' user enrolment. If there are other ways to enter the course (such guest access) it is recommended to remove user roles at the same time.

==Activity participation==

Activity developers decide the enrolment related behaviour of module.

There are some general guidelines:
* Only users with active enrolments should receive notifications.
* Activities should display enrolled users with some capability as participants.
* By default only users with active enrolments should be displayed in reports.
* There should be option to display all enrolled users including suspended enrolments.
* For performance reasons invisible participation data should be purged on unenrolment.
* Contributions visible by other participants should be kept after unenrolment (such as forum posts).

==API functions==

===is_enrolled()===
Is user participating in the course? Returns true for students and teachers, false for administrators and other managers. User enrolments can be either active or suspended, suspended users can not enter the course (unless there is some kind of guest access allowed) or have moodle/course:view capability and are usually hidden in the UI.

<code php>
function is_enrolled(context $context, $user = null, $withcapability = '', $onlyactive = false)
</code>

Good example is choice module where we have one slot for each participant, people that are not enrolled are not allowed to vote <code>is_enrolled($context, $USER, 'mod/choice:choose')</code>. Another example is assignment where users need to be enrolled and have capability to submit assignemnts <code>is_enrolled($this->context, $USER, 'mod/assignment:submit')</code>.

===get_enrolled_users()===

Sometimes you need to know the list of users that can participate in some activity.

<code php>
function get_enrolled_sql(context $context, $withcapability = '', $groupid = 0, $onlyactive = false)
function get_enrolled_users(context $context, $withcapability = '', $groupid = 0, $userfields = 'u.*', $orderby = '', $limitfrom = 0, $limitnum = 0)
function count_enrolled_users(context $context, $withcapability = '', $groupid = 0)
</code>

For example you want to know who is able to summit assignment right now:
<code php>
$submissioncandidates = get_enrolled_users($modcontext, 'mod/assignment:submit');
</code>

The assignment module needs to hold data for all users that are enrolled including the users with suspended enrolments and without any roles. Module developers may decide to purge all user data when user is fully unenrolled.

SQL select from get_enrolled_sql() is often used for performance reasons - you can use it in joins to get specific information for enrolled users only.

==See also==
* [[Enrolment plugins]]
* [[Enrolment usage overview]]
* [[New enrolments in 2.0]]

[[Category:Enrolment]]

Moodle 2.3 release notes

2012-07-06T13:08:20Z

CHRISF:

[[Releases]] > {{FULLPAGENAME}}

Release date: 25th June 2012

Here is [http://tracker.moodle.org/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=project+%3D+mdl+AND+resolution+%3D+fixed+AND+fixVersion+in+%28%222.3%22%29+ORDER+BY+priority+DESC&runQuery=true&clear=true the full list of fixed issues in 2.3].

Many thanks to [http://moodle.org/dev/contributions.php?version=2.3.x everyone that worked on the new features in this release], including those from [http://moodle.com/hq/team Moodle HQ], [http://www.open.ac.uk/ The Open University], [http://www.netspot.com.au/ NetSpot], [http://www.synergy-learning.com/ Synergy Learning], [http://www.luns.net.uk/ Lancaster University Network Services], [http://catalyst.net.nz/ Catalyst IT], [http://www.cvaconsulting.com/ CV&A Consulting] (funded by [http://www.esade.edu ESADE] and [http://www.il3.ub.edu IL3-UB]) and others. More details can be found below under each major feature, or look at the list of bugs fixed above.

A special thanks to our incredible ''''Integration Team'''' from Moodle HQ who worked tirelessly with all developers to review, test and help finish code for inclusion in Moodle core:

* Eloy Lafuente
* Sam Hemelryk
* Dan Poltawski
* Aparup Bannerjee

Finally, thanks to all our testers both within Moodle HQ and from the whole community, who have contributed to producing our most exciting and stable release yet!

===Requirements===

* Minimum browser: Firefox 4, Internet Explorer 8, Safari 5, Google Chrome 11, Opera 9
* Moodle upgrade: Moodle 2.2 or later (if upgrading from earlier versions, you must upgrade to 2.2 as a first step)
* Minimum DB versions: Postgres 8.3, MySQL 5.1.33 (MDL-33984), MSSQL 2005 or Oracle 10.2
* Minimum PHP version: PHP 5.3.2

===Major new features===

====Files usability====

* MDL-31907 - A nicer-looking file picker with fewer clicks.
* Images now display as true thumbnails in the file picker and file manager.
* Other files have pretty icons for most file types.
* Files view can be easily toggled between icons view or a table view with sizes and dates, or a hierarchical list view.
* You can now drag and drop files directly from your desktop straight into file areas!
* File info (eg license information, sizes, dates) can be easily edited and viewed in a popup dialogue.
* Files can be created as "aliases/shortcuts" of other files. This allows you to, for example, use a single file in your private files area multiple times in all your courses. If you update the original file then all the aliases will automatically update!
* Aliases are easily identifiable in the file manager interface.

Credits to Moodle HQ: Marina Glancy, Barbara Ramiro, Martin Dougiamas, David Mudrak, Dongsheng Cai and others.

{|
|-
| [[File:truthumbnailsiconsview.png|thumb|File picker icon view]]
| [[File:tableview.png|thumb|File picker table view]]
| [[File:hierarchicallistview.png|thumb|File picker hierarchical list view]]
|-
| [[File:popupdialogue.png|thumb|File info popup dialogue]]
| [[File:alias.png|thumb|Creating an alias]]
| [[File:shortcut1.png|thumb|Aliases are easily identifiable]]
|}
====Repository improvements====

* MDL-32117 - New [https://docs.moodle.org/23/en/EQUELLA_repository EQUELLA repository] enabling users to make aliases (shortcuts) to EQUELLA files.
* MDL-28666 - If a repository supports it then it's possible to make an alias/shortcut to a file in an external repository. If the file is updated in the repository, then this change is reflected in Moodle. The file remains under Moodle access control however, and the original URL is not usually revealed. In the 2.3 core release, this is supported by private files, server files, file system, Box.net and EQUELLA repositories.
* The repository plugin is now able to take over the whole right-hand pane of the file picker and provide its own searching/browsing interface.
* MDL-31675 - Server files repository now available for database, forum and glossary activities.

Credits to Moodle HQ: Dongsheng Cai, Marina Glancy, Martin Dougiamas, Petr Skoda

====Improvements to course pages ====

* MDL-32476 - Courses can now choose to show sections on individual pages
* MDL-31052 - All AJAX editing on the course pages has been modernised and cleaned up (YUI3). It's on by default now too.
* MDL-31263 - Blocks can be dragged and dropped around the page (again)
* MDL-30617 - An optional new popup [https://docs.moodle.org/23/en/Course_homepage "Activity chooser"] has been added with full introduction, examples and links about each activity or resource module.
* MDL-22504 - You can now drag files straight into the course page and they will be added as resources.
* MDL-31215 - You can edit the name of any activity or resource directly on the course page without entering the settings (works particularly well with drag and drop).
* MDL-32771 - You can now add/remove sections directly from the [https://docs.moodle.org/23/en/Course_homepage course page].

Credits to: Andrew Nicols and Ruslan Kabalin (LUNS) Davo Smith (Synergy Learning), Dan Poltawski and Martin Dougiamas (Moodle HQ)

{|
| [[File:dragandrop.png|thumb|File drag and drop]]
| [[File:activity chooser.png|thumb|Activity chooser]]
|}
====Assignment module====

* MDL-26997 - Complete rewrite of the assignment module from scratch, by [http://www.netspot.com.au/ NetSpot] (Moodle Partner in Australia)
* Assignment subtypes are no longer needed. New plugins are possible for
* Old Assignment module is still installed, and works, but is not necessary

Credits to Netspot: Damyon Wiese, Raymond Wijaya, Minh-Tam Nguyen

* MDL-31731 - New [https://docs.moodle.org/23/en/Marking_guide marking guide] advanced grading method, where a teacher enters a comment per criterion and a mark up to a maximum

Credits to Dan Marsden Catalyst IT (development); Lightwork Team at Massey University (concept); NetSpot Innovation Grant (funding)

====Book module====

* MDL-32709 - The most popular third-party resource module ever, [https://docs.moodle.org/23/en/Book_module Book] finally joins core. Welcome!

Credits to Petr Škoda, Moodle HQ

====Quiz module====

* MDL-3030 - More robust handling of quiz attempts that are not submitted by the deadline.
* MDL-3054 & MDL-11047 - There is now an option for teacher to force students to answer the quiz questions strictly in order. As part of this, the quiz remembers which page the student was last on, and will take them back there when they resume an attempt.

Credits to Tim Hunt, The Open University and Charles Fulton, Lafayette College

====SCORM module====

* MDL-29745 - New [https://docs.moodle.org/23/en/Using_SCORM SCORM graph report] plugin (credits to Ankit Agarwal, Moodle HQ and Dan Marsden, Catalyst IT)
* MDL-32937 - You can now drag a SCORM package straight into the course page and it will be added as SCORM activity (credits to Christopher Tombleson and Dan Marsden, Catalyst IT, and Davo Smith, Synergy Learning)

====Workshop module====

* MDL-26099 - [https://docs.moodle.org/23/en/Workshop_settings Option to make the workshop switch to the assessment phase] automatically after the submissions deadline (including automatic allocation of submissions for assessment)
* MDL-25660 - Workshop submission deadlines are shown in the calendar
* MDL-27508 - Improved support for pagination and filtering workshop submissions by group
* MDL-32638 - Workshop supports file browsing via Server files repository (including improved access control when serving submission files)

Credits to David Mudrak, Moodle HQ

====Available update notifications====

*MDL-20438 - Admins are sent [https://docs.moodle.org/23/en/Notifications notification of any updates available] for core code and for any contributed plugins installed on the site (from the [http://moodle.org/plugins plugins directory]). Admins can also check for available updates using buttons on the notifications and plugins overview pages.

Credits to David Mudrak, Moodle HQ

===Other highlights===

====Google Docs and Picasa plugins requirement====

* MDL-29857 - Due to a change in Google's service, Moodle has switched to a more secure and more user-friendly system for communicating with Google called 'OAuth 2.0'. An administrator must register their site with Google, as described in [https://docs.moodle.org/23/en/Google_OAuth_2.0_setup Google OAuth 2.0 setup] in order to obtain a client ID and secret for use in configuring all Google Docs and Picasa plugins (the Google Docs and Picasa repositories and the Google Docs and Picasa portfolios).

Credits to Dan Poltawski, Moodle HQ

====IMS Common Cartridge 1.1 Export====

* MDL-33079 - In Moodle 2.2 we introduced IMS CC import. Well, now you can also export to that format, via the normal Backup menu. Just look for the IMS CC checkbox.

====Miscellaneous====

* MDL-31121 - Option in [https://docs.moodle.org/23/en/File_module_settings file resource settings] to display file size and/or type on course page (credits to Sam Marshall, The Open University)
* MDL-32009 - Admin option for uninstalling messaging outputs and report of messaging output statuses on plugins overview page
* MDL-29941 - Admin option to enable a [https://docs.moodle.org/23/en/CSS_optimiser CSS optimiser] that analyses and refactors CSS before caching it
* MDL-24419 - [https://docs.moodle.org/23/en/Conditional_activities_settings Conditional activities setting] enabling teachers to restrict access to a course section using similar logic to Conditional Activities (with thanks to the [http://www.unsw.edu.au/ University of New South Wales] for funding and [http://www.netspot.com.au/ NetSpot] for developing this feature and Sam Marshall at the [http://www.open.ac.uk OU] for integrating it to core)
* MDL-26901 - Option to add extra fonts to the [https://docs.moodle.org/23/en/Text_editor TinyMCE HTML editor]
* MDL-31315 - When editing a form, a warning will be displayed if trying to navigate away with content edited
* MDL-33401 - Managers are now allowed the capability to edit blocks (moodle/block:edit) by default for new installs. This does not affect existing installations.
* MDL-32005 - New group and groupings ID number setting for matching groups and groupings against external systems (credits to Andrew Nicols, LUNS)
* MDL-30482 - New [https://docs.moodle.org/23/en/Capabilities/mod/glossary:view view glossary entries capability]
* MDL-31158 - New [https://docs.moodle.org/23/en/Grade_settings recover grades default setting]
* MDL-11378 - New [https://docs.moodle.org/23/en/Messaging_settings SMTP security email setting]

===Security issues===

All security issues that were fixed in 2.2.x and 2.1.x were also fixed in 2.3.

===For developers: API changes===

Abbreviated descriptions of API changes are always kept up to date in the "upgrade.txt" within each plugin area. We do this so that the information is always exactly right for the version of Moodle you are using. Changes in this release:

;Blocks: http://git.moodle.org/gw?p=moodle.git;a=blob;f=blocks/upgrade.txt;hb=master
;Course formats: http://git.moodle.org/gw?p=moodle.git;a=blob;f=course/format/upgrade.txt;hb=master
;Filters: http://git.moodle.org/gw?p=moodle.git;a=blob;f=filter/upgrade.txt;hb=master
;Activity modules: http://git.moodle.org/gw?p=moodle.git;a=blob;f=mod/upgrade.txt;hb=master
;Quiz access rules: http://git.moodle.org/gw?p=moodle.git;a=blob;f=mod/quiz/accessrule/upgrade.txt;hb=master
;Quiz reports: http://git.moodle.org/gw?p=moodle.git;a=blob;f=mod/quiz/report/upgrade.txt;hb=master
;Portfolio plugins: http://git.moodle.org/gw?p=moodle.git;a=blob;f=portfolio/upgrade.txt;hb=master
;Question behaviour plugins: http://git.moodle.org/gw?p=moodle.git;a=blob;f=question/behaviour/upgrade.txt;hb=master
;Question behaviour plugins: http://git.moodle.org/gw?p=moodle.git;a=blob;f=question/format/upgrade.txt;hb=master
;Question types: http://git.moodle.org/gw?p=moodle.git;a=blob;f=question/type/upgrade.txt;hb=master
;Repository plugins: http://git.moodle.org/gw?p=moodle.git;a=blob;f=repository/upgrade.txt;hb=master
;Themes: http://git.moodle.org/gw?p=moodle.git;a=blob;f=theme/upgrade.txt;hb=master

====Core API changes====
* MDL-31902 All xxx_get_participants() functions are removed from core
* As a part of MDL-32471, the signature of send_stored_file() has been [http://git.moodle.org/gw?p=moodle.git;a=commitdiff;h=796495fed29f12e4a81bd406558d8eeffd0e64ac modified]. The last two parameters $filename and $dontdie were replaced with a single array containing additional options for the file serving. The pluginfile callbacks in plugins are supposed to transfer these options from the caller to send_stored_file() - see the note below.
* MDL-28666 Files API changes, added ability to create file reference using file_storage::create_file_from_reference() method, update file record attributes using stored_file class
* MDL-28666 Repository API changes, added new APIs repository::get_file_reference(), repository::get_file_by_reference(), repository::get_reference_details(), repository::send_file(), the new APIs make serving files from external repository possible
* MDL-33446 Make sure that areas in your plugins where you DON'T want content to change have support for file references turned OFF. Some areas in Moodle core that do not support references are assignment submissions, workshop submissions, forum posts, and quiz essay questions - this is so that students are not able to update files from outside Moodle after any due date
* AJAX Flags: MDL-32509 and MDL-32908 The $CFG->enablecourseajax and $USER->ajax fields have been removed. The fields was not widely respected and all 'advanced javascript' should work in a progressive enhancement and accessible way. A site wide flag still exists but may be phased out in the future.

====Plugin API changes====

* As a part of MDL-32471, the API of the plugin function xyz_pluginfile() has been extended. There is a new array parameter passed to these callbacks containing additional options for the file serving. The array should be re-passed to send_stored_file(). The change is pretty trivial - see [http://git.moodle.org/gw?p=moodle.git;a=commitdiff;h=261cbbacc15ef1732a357d689908c91c15e0617a examples].
* Activity modules can now declare support for course drag and drop upload [[Implementing_Course_drag_and_drop_upload_support_in_a_module]]

====Webservice changes====
Few changes could break existing web service clients in 2.3 - untill this version we tried not to break anything. However these changes will make the client's developer life easier, so we prefered to do them now than later. Please take in consideration these improvements and retest your clients:
* [https://docs.moodle.org/dev/Errors_handling_in_web_services Error codes and Warnings]
* All text fields have an additional format field as parameter and return value (MDL-32581)
* Thanks to the increasing number of contributions, we improved our [https://docs.moodle.org/dev/How_to_contribute_a_web_service_function_to_core contributor web service guide]
* From 2.3, all web service functions integrated in master will land (when possible) in supported minor versions (e.g. 2.3.1, 2.3.2...).
* Many [http://tracker.moodle.org/browse/MDL-31253 fixes] and new [http://tracker.moodle.org/browse/MDL-29934 API functions].

====Unit tests====

We have switched completely to using [[PHPUnit]] for all our unit tests now. All existing simpletests have been rewritten, and new tests have been added.

We intend to move towards a completely unit-test-driven development methodology (where the tests are written first!) for significant new code, and we also encourage all developers to implement unit tests covering at least the core features of their code.

Moodle HQ run these tests on an automated basis for all new code submitted for integration, as well as on each weekly release.

==== Community hub changes ====
Some bug fixes and improvements in [http://tracker.moodle.org/browse/MDL-30247 core] and in the [http://tracker.moodle.org/browse/CONTRIB-3348 plugin]. Hub administrators must update their hub to the most recent version regarding CONTRIB-3646.

<noinclude>==See also==

* [https://docs.moodle.org/23/en/Category:New_features User documentation of new features in Moodle 2.3]
* [https://docs.moodle.org/23/en/Upgrading_to_Moodle_2.3 Upgrading to Moodle 2.3] - information for admins who are upgrading from earlier versions
*[[Moodle 2.2 release notes]]

[[Category:Release notes]]
[[Category:Moodle 2.3]]

[[fr:Notes de mise à jour de Moodle 2.3]]
</noinclude>

Talk:Core APIs

2012-02-20T15:07:34Z

CHRISF: /* Some early comments about this list of APIs */

=== Some early comments about this list of APIs ===

* Surely there is one reason for inventing this, completely bypassing the existing components (both for plugins, subplugins and subsystems). But I cannot really imagine it. Why not use direct mapping between components and APIs ? Code is going to use ones and docs are going to use another?
* More yet, I don't get why the list of APIs ignores everything that is provided with plugin abilitites. Aren't "auth" or "enrol" or "reports" valid core APIs? In fact I think that the parts of Moodle currently working under the "plugins umbrella" are the most formal APIs we have in codebase (at least the OOP ones).
* Some of the APIs in the list seem incorrect, inaccurate:
** database API. it should be split, for sure in ddl and dml APIs at least.
** time API. => datetime, surely.
** renderer API => is that really an API
** upgrade API => WTF is that? I'm specially interested on that, as far as it seems I'm the man in charge of documenting it and I don't know what the hell it is (MDL-30971).
** grade and grading APIs => Should we have also group and grouping APIs? (sorry, joking, hehe, but don't buy the difference).
** preference API : set/get/remove_user_preference() functions? Or also config ?
** admin API ?
** user API ?
** install/uninstall API ?
** ...
** ...

So, in summary, I don't get which benefits this new list provides, nor I think we are doing well by keeping apart some APIs because of them being "pluggable", nor I think the APIs are well balanced at all. Just my 2cents (really not destructive, but constructive, although it sounds the opposite, guys).

--[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 05:52, 5 January 2012 (WST)

Hi,

Certainly I agree with Eloy on this one. Why introduce another list of the same API's we already have.<br />
While they look 80% the difference between the code and documentation will cause real confusion for developers learning Moodle, especially in regards to terminology.<br />

I think we need to take great care that we don't come up with new terminology for things within Moodle.<br />
As Eloy noted there are some things in the API list that don't make sense, I've just gone through as part of MDL-30979 and updated the PHP docs for the Output API's. I assume that is what you mean by renderer API, hehe you've confused me with that one?!

Cheers<br />
--[[User:Sam Hemelryk|Sam Hemelryk]] 09:48, 5 January 2012 (WST)

i think i also generally share the sentiments above, so here's my general 'exactly!' for the above.

definitely for ddl and dml being mashed together into database API is sad loss of definition. I think theres certainly room for more planned and in depth discussions about our terminologies. not to mention the phpdoc is still not clearly and somewhat agreeably defined in our documentation project where we're implementing this atm.

(whee first talk page comment by me!) cheers!

[[User:Aparup Banerjee|Aparup Banerjee]] 00:18, 6 January 2012 (WST)

This new list is for documentation purposes only. We need a new list because NO OLD LIST OF APIS EXISTS! There *is* no table of contents that can group all the main functions available for *PLUGIN AUTHORS*. This started as a subset of get_core_subsystems() and has had things added where necessary.

Getting to specifics from above, based on discussion in HQ this morning and my thoughts:

* database -> ddl and dml AGREED, done.
* time -> datetime Is the reason because of PHPs name? We don't use that name anywhere else?
* renderer -> output AGREED. Either is fine for me. "Renderer" was what EU people agreed on some weeks ago and it is a bit better for googling, but whatever.
* upgrade API : It's meant to be everything you can do in upgrade/install (including upgradelib.php and all the tricks we use), and I was thinking it included DDL too. We can leave it until later if required.
* grade and grading APIs => They are completely separate APIs. The names are a little confusing but I think we need to reuse get_core_subsystems() were possible.
* preference API : set/get/remove_user_preference() functions, yes.
* admin API -> Yes, I guess we need to explain settings.php and friends. Thanks.
* user API -> That's hard to identify. I left it out for now as very few plugins needs to create/edit/delete users. or courses.
* install/uninstall API -> Don't know about that. install.xml and what else?

--[[User:Martin Dougiamas|Martin Dougiamas]] 11:28, 6 January 2012 (WST)

Re: a User and a Course API, I think these would be very useful for those of us trying to integrate Moodle with our Student Information Systems. I had a look at Jerome's Web Services API, but I think there have been some fundamental oversights in that implementation. All of those functions require that you know the ID of things like users and courses, when an external system is not going to have that information. It would be useful to be able to do things like:

* <code>$user = get_user(array('username'=>'chrisf'));</code>
* <code>$course = get_course(array('idnumber'=>'CF101'));</code>
* <code>enrol_user(array('username'=>'chrisf'), array('idnumber'=>'CF101') [,rolename])</code>.

The user and course tables contain columns for <code>idnumber</code>, which seems uniquely suited to operations of this type. I'm kind of surprised no-one else seems to be looking for this kind of API.

-- [[User:Chris Fryer|Chris Fryer]] 23:07, 20 February 2012 (WST)

=== About completion and availability (aka condition) ===

Plz, take a look to MDL-30996 (and MDL-30995). IMO we need to clearly state the differences and uses, both between those 2 APIs and with the "course completion" one, that I don't know if should have its own API. This involves rethinking the names given for each API, and their occurrences/explanation both in phpdocs and Docs.

--[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:47, 8 February 2012 (WST)

I agree, we should state differences and uses between completion and condition. Saying so, condition should not be changed to availability, because get_core_subsystem() return condition and it should be similar. I am not sure about course_completion, so no comments.

--[[User:Rajesh Taneja (rajesh)|Rajesh Taneja (rajesh)]]

: But, once again, we decided to have the APIs under the @category umbrella to have them *separated* (and free) from packages. So I don't find any problem naming them properly (and differently from their @package couterparts). They are just two *independent* ways to list/group things, so better if at least one of them is correct.

:--[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 17:19, 9 February 2012 (WST)

=== Some 2 little details about naming ===

* Also, offtopic, it's a bit inconsistent to use "gerunds" for some APIs (rating, grading...), plurals for others (files..) and nouns/verbs (which is the official way, nouns?) for the rest.

* Finally, Docs should match the name of the API completely, or at least offer one redirect from it, so anybody can make https://docs.moodle.org/dev/xxxx_API and arrive to the desired page. Sounds silly but surely it's useful to include jumps between system X and the Docs.

--[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:58, 8 February 2012 (WST)

Web Services

2011-06-30T10:19:01Z

CHRISF: Redirected page to Web services

#REDIRECT [[Web_services]]

File API internals

2010-12-02T12:09:32Z

CHRISF:

{{Work in progress}}
{{Infobox Project
|name = File API
|state = Implemented
|tracker = MDL-14589
|discussion = n/a
|assignee = [[User:Petr Škoda (škoďák)|Petr Škoda (škoďák)]]
}}
{{Moodle 2.0}}

==Objectives==

The goals of the new File API are:

* allow files to be stored within Moodle, as part of the content (as we do now).
* use a consistent and flexible approach for all file handling throughout Moodle.
* give modules control over which users can access a file, using capabilities and other local rules.
* make it easy to determine which parts of Moodle use which files, to simplify operations like backup and restore.
* track where files originally came from.
* avoid redundant storage, when the same file is used twice.
* fully support Unicode file names, irrespective of the capabilities of the underlying file system.

==Overview==

The File API is a set of core interfaces to allow the rest of Moodle to store, serve and manage files. It applies only to files that are part of the Moodle site's content. It is not used for internal files, such as those in the following subdirectories of dataroot: temp, lang, cache, environment, filter, search, sessions, upgradelogs, ...

The API can be subdivided into the following parts:
; File storage
: Low level file storage without access control information. Stores the content of files on disc, with metadata in associated database tables.
; File serving
: Lets users accessing a Moodle site get the files (file.php, draftfile.php, pluginfile.php, userfile.php, etc.)
:* Serve the files on request
:* with appropriate security checks
; File related user interfaces
: Provides the interface for (lib/form/file.php, filemanager.php, filepicker.php and files/index.php, draftfiles.php)
:* Form elements allowing users to select a file using the Repository API, and have it stored within Moodle.
:* UI for users to manage their files, replacing the old course files UI
; File browsing API
: Allows code to browse and optionally manipulate the file areas
:* find information about available files in each area.
:* print links to files.
:* optionally move/rename/copy/delete/etc.

== File API internals ==

=== File storage on disk ===

Files are stored in $CFG->dataroot (also known as moodledata) in the filedir subfolder.

Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.

Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce. Then it will be stored in on disc as moodledata/filedir/08/13/71/081371cb102fa559e81993fddc230c79205232ce.

This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

=== Files table ===

This table contains one entry for each usage of a file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary. It is necessary because some databases have hard limit on index size.

If, for example, the same image is used in a user's profile, and a forum post, then there will be two rows in this table, one for each use of the file, and Moodle will treat the two as separate files, even though the file is only stored once on disc.

{| class="nicetable"
! Field
! Type
! Default
! Info
|-
| '''id'''
| int(10)
| auto-incrementing
| The unique ID for this file.
|-
| '''contenthash'''
| varchar(40)
|
| The sha1 hash of content.
|-
| '''pathnamehash'''
| varchar(40)
|
| The sha1 hash of "/contextid/component/filearea/itemid/filepath/filename.ext" - prevents file duplicates and allows fast lookup. It is necessary because some databases have hard limit on index size.
|-
| '''contextid'''
| int(10)
|
| The context id defined in context table - identifies the instance of plugin owning the file.
|-
| '''component'''
| varchar(50)
|
| Like "mod_forum", "course", "mod_assignment", "backup"
|-
| '''filearea'''
| varchar(50)
|
| Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.
|-
| '''itemid'''
| int(10)
|
| Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)
|-
| filepath
| text
|
| relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this
|-
| filename
| varchar(255)
|
| The full Unicode name of this file (case sensitive)
|-
| '''userid'''
| int(10)
| NULL
| Optional - general user id field - meaning depending on plugin
|-
| filesize
| int(10)
|
| size of file - bytes
|-
| mimetype
| varchar(100)
| NULL
| type of file
|-
| status
| int(10)
|
| general file status flag - will be used for lost or infected files
|-
| source
| text
|
| file source - usually url
|-
| author
| varchar(255)
|
| original author of file, used when importing from other systems
|-
| license
| varchar(255)
|
| license type, empty means site default
|-
| timecreated
| int(10)
|
| The time this file was created
|-
| timemodified
| int(10)
|
| The last time the file was last modified
|}

Indexes:
* non-unique index on (contextid, component, filearea, itemid)
* non-unique index on (contenthash)
* unique index on (pathnamehash).

The plugin type does not need to be specified because it can be derived from the context. Items like blog that do not have their own context will use their own file area inside a suitable context. In this case, the user context.

Entries with filename = '.' represent directories. Directory entries like this are created automatically when a file is added within them.

Note: 'files' plural is used even thought that goes against the [[Database|coding guidelines]] because 'file' is a reserved word in some SQL dialects.

===Implementation of basic operations===

'''Each plugin may directly access only files in own context and areas!'''

Low level access API is defined in ''file_storage'' class which is obtained from <code>get_file_storage()</code>.

====Storing a file====

# Calculate the SHA1 hash of the file contents.
# Check if a file with this SHA1 hash already exists on disc in file directory or file trash. If not, store the file there.
# Add the record for this file to the files table using the low level address

====Reading a file====

# Fetch the record (which includes the SHA1 hash) for the file you want from the files table. You can fetch either all area files or quickly get one file with a specific contenthash.
# Retrieve the contents using the SHA1 hash from the file directory.

====Deleting a file====

# Delete the record from the files table.
# Verify if some other file is still needing the content, if not move the content file into file trash
# Later, admin/cron.php deletes content files from trash directory

== File serving ==

Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server properly (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

=== legacy file.php ===

Serves legacy course files, the file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files are stored in <code>array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>

The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.

=== pluginfile.php ===
All plugins should use this script to serve all files.
* plugins decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* links to these files are constructed on the fly from the relative links stored in database, this means that plugin may link only own files

Absolute file links need to be rewritten if html editing allowed in plugin. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/intro/image.jpg, it is converted back to internal links before saving.

Script parameters are virtual file names, in most cases the parameters match the low level file storage, but they do not have to:

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

==== Assignment example ====

/pluginfile.php/assignmentcontextid/mod_assignment/intro/someimage.jpg
/pluginfile.php/assignmentcontextid/mod_assignment/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/mod_assignment/allsubmissions/groupid/allsubmissionfiles.zip

The last line example of virtual file that should created on the fly, it is not implemented yet.

====scorm example====

/pluginfile.php/scormcontextid/mod_scorm/intro/someimage.jpg
/pluginfile.php/scormcontextid/mod_scorm/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems.

====quiz example====

pluginfile.php/quizcontextid/mod_quiz/intro/niceimage.jpg

====questions example====

This section was out of date. See [[File_storage_conversion_Quiz_and_Questions]] for the latest thinking.

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>

/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>

=== Temporary files ===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* processing by executable files (latex, mimetex)

These files should never use utf-8 file names.

=== Legacy file storage and serving ===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

== File browsing API ==

This is what other parts of Moodle use to access files that they do not own.

=== Class: file_browser ===

=== Class: file_info and subclasses ===

== File related user interfaces ==

All files are obtained through from the file repositories.

=== Formslib fields ===
* file picker
* file manager
* file upload (obsolete, do not use)

=== Integration with the HTML editor ===

Each instance of the HTML editor can be told to store related files in a particular file area.

During editing, files are stored in a draft files area. Then when the form is submitted they are moved into the real file area.

Files are selected using the repository file picker.

=== Legacy file manager ===

Available only for legacy reasons. It is not supposed to be used.

All the contexts, file areas and files now form a single huge tree structure, although each user only has access to certain parts of that tree. The file manager (files/index.php) allow users to browse this tree, and manage files within it, according to the level of permissions they have.

Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

== Backwards compatibility ==

=== Content backwards compatibility ===

This should be preserved as much as possible. This will involve rewriting links in content during the upgrade to 2.0.

Some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Code backwards compatibility===

Other Moodle code (for example plugins) will have to be converted to the new APIs. See [[Using_the_file_API]] for guidance.

It is not possible to provide backwards-compatibility here. For example, the old $CFG->dataroot/$courseid/ will no longer exist, and there is no way to emulate that, so we won't try.

== Upgrade and migration ==

When a site is upgraded to Moodle 2.0, all the files in moodledata will have to be migrated. This is going to be a pain, like DML/DDL was :-(

The upgrade process should be interruptible (like the Unicode upgrade was) so it can be stopped/restarted any time.

=== Migration of content ===

* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new area, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be many outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

=== Moving files to files table and file pool ===

The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.

Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some people use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

== Other issues ==

=== Unicode support in zip format ===

Zip format is an old standard for compressing files. It was created long before Unicode existed, and Unicode support was only recently added. There are several ways used for encoding of non-ASCII characters in path names, but unfortunately it is not very standardised. Most Windows packers use DOS encoding.

Client software:
* Windows built-in compression - bundled with Windows, non-standard DOS encoding only
* WinZip - shareware, Unicode option (since v11.2)
* TotalCommander - shareware, single byte(DOS) encoding only
* 7-Zip - free, Unicode or DOS encoding depending on characters used in file name (since v4.58beta)
* Info-ZIP - free, uses some weird character set conversions

PHP extraction:
* Info-ZIP binary execution - no Unicode support at all, mangles character sets in file names (depends on OS, see docs), files must be copied to temp directory before compression and after extraction
* PclZip PHP library - reads single byte encoded names only, problems with random problems and higher memory usage.
* Zip PHP extension - kind of works in latest PHP versions

Large file support:
PHP running under 32bit operating systems does not support files >2GB (do not expect fix before PHP 6). This might be a potential problem for larger backups.

Tar Alternative:
* tar with gzip compression - easy to implement in PHP + zlib extension (PclTar, Tar from PEAR or custom code)
* no problem with unicode in *nix, Windows again expects DOS encoding :-(
* seems suitable for backup/restore - yay!

Summary:
# added zip processing class that fully hides the underlying library
# using single byte encoding "garbage in/garbage out" approach for encoding of files in zip archives; add new 'zipencoding' string into lang packs (ex: cp852 DOS charset for Czech locale) and use it during extraction (we might support true unicode later when PHP Zip extension does that)

== Not implemented yet ==
* antivirus scanning - this needs a different api because the upload of files is now handled via repository plugins

== See also ==

* [[Using the file API]]
* [[Repository API]]
* [[Portfolio API]]
* [[Resource module file API migration]]
* MDL-14589 - File API Meta issue

{{CategoryDeveloper}}
[[Category:Files]]

[[ja:開発:ファイルAPI]]
[[ru:Development:File_API]]

User:Chris Fryer

2007-07-25T09:22:02Z

CHRISF:

Systems administrator at the [http://clt.lse.ac.uk/ Centre for Learning Technology], London School of Economics.

User talk:Helen Foster

2006-02-07T10:08:51Z

CHRISF: /* Documentation for different versions */

== More hierarchy ==

Hi Helen, I think this wiki is very useful and I would like to thank you for the work you are putting into it. I like your navigation boxes on the module pages. I have a suggestion for extending these: I think it would be nice to have an extra link that allows one to go higher up in the hierarcy to [[Teacher_documentation]]. What do you think? --[[User:Delius|Delius]] 17:48, 26 January 2006 (WST)

:Hi Gustav, thanks for your kind words. Your contributions to the documentation are most welcome. Please feel free to edit [[Template:Quizzes]] or any other template page. You may also wish to consider further use of category pages e.g. [[:Category:Teacher]] for navigation. -- [[User:Helen|Helen]] 18:26, 26 January 2006 (WST)

== Do you monitor all talk pages? ==

Helen, I have posted another question for you on [[Talk:mod/quiz/index]]. Do you get alerted to anything I put on any talk page or should I post on this page if I want you to see it? --[[User:Delius|Delius]] 18:01, 26 January 2006 (WST)

:Hi Gustav, please note that I often check [[Special:Recentchanges|Recent changes]] or the rss feed. --[[User:Helen|Helen]] 18:26, 26 January 2006 (WST)

::Yes, the [[Special:Recentchanges|Recent changes]] page is very useful. You can tell that I have never used a wiki intensively before. Thanks for your tips. --[[User:Gustav|Gustav]] 18:35, 26 January 2006 (WST)

:::Hey, let's discover the wiki way together! :-) --[[User:Helen|Helen]] 00:24, 27 January 2006 (WST)

==Moodle profile==

Helen- I think you should link to your moodle profile like Martin does (assuming you have one, which I assume <grin>). -- D.I. 29Jan06
Wow, you're fast.. .or I'm confused. --D.I. 29Jan06

:Hi D.I. thanks for your suggestion, which [[User:UrsHunkler|someone else]] seem to be following up! ;-) --[[User:Helen|Helen]] 05:00, 30 January 2006 (WST)

== German moodle docs possible? ==

Hello Helen,

thank you very much for these nice and structured doc pages.
When do you plan to launch a german part of the docs.moodle.org pages?
I ask these question because i can translate some of the english doc pages to german.

Greetings Christoph

:Hi Christoph, thanks for your kind offer of translation. Please email ''docs AT moodle DOT org'' so I may get in touch with you easily when the German Moodle documentation is launched. --[[User:Helen|Helen]] 05:17, 30 January 2006 (WST)

== German User Docs ==

Thanks a lot, Helen, for all your work.
I would also like to help with the German version.

Ulrike

:Hi Ulrike, thanks for your kind offer too. MoodleDocs in other languages will happen soon, for sure! --[[User:Helen|Helen]] 08:30, 31 January 2006 (WST)

== Documentation for different versions ==

Helen

Is there a mechanism for specifying which version of Moodle a particular document refers to? If, for instance, a feature is only available in version 1.6, how would a 1.5.3 user know they couldn't follow these instructions?

Equally, if something is done differently in two versions, how would we stop an edit war breaking out between 1.5.3 and 1.6 users?

I appreciate this probably isn't an issue just yet, but it's possible it may become one in later versions.

Cheers,

[[User:CHRISF|CHRISF]] 19:20, 1 February 2006 (WST)

:Hi ChrisF, thanks for highlighting the issue of differences in Moodle versions. Perhaps a [[Template:Moodle 1.6|Moodle 1.6 template]], inserted using the code <code><nowiki>{{Moodle 1.6}}</nowiki></code>, may be used to show features specific to Moodle 1.6. What do you think? --[[User:Helen|Helen]] 04:22, 4 February 2006 (WST)

::Good idea. I think that would clear it up nicely. [[User:CHRISF|CHRISF]] 18:08, 7 February 2006 (WST)

User talk:Helen Foster

2006-02-01T11:20:48Z

CHRISF: Documentation for different versions

== More hierarchy ==

Hi Helen, I think this wiki is very useful and I would like to thank you for the work you are putting into it. I like your navigation boxes on the module pages. I have a suggestion for extending these: I think it would be nice to have an extra link that allows one to go higher up in the hierarcy to [[Teacher_documentation]]. What do you think? --[[User:Delius|Delius]] 17:48, 26 January 2006 (WST)

:Hi Gustav, thanks for your kind words. Your contributions to the documentation are most welcome. Please feel free to edit [[Template:Quizzes]] or any other template page. You may also wish to consider further use of category pages e.g. [[:Category:Teacher]] for navigation. -- [[User:Helen|Helen]] 18:26, 26 January 2006 (WST)

== Do you monitor all talk pages? ==

Helen, I have posted another question for you on [[Talk:mod/quiz/index]]. Do you get alerted to anything I put on any talk page or should I post on this page if I want you to see it? --[[User:Delius|Delius]] 18:01, 26 January 2006 (WST)

:Hi Gustav, please note that I often check [[Special:Recentchanges|Recent changes]] or the rss feed. --[[User:Helen|Helen]] 18:26, 26 January 2006 (WST)

::Yes, the [[Special:Recentchanges|Recent changes]] page is very useful. You can tell that I have never used a wiki intensively before. Thanks for your tips. --[[User:Gustav|Gustav]] 18:35, 26 January 2006 (WST)

:::Hey, let's discover the wiki way together! :-) --[[User:Helen|Helen]] 00:24, 27 January 2006 (WST)

==Moodle profile==

Helen- I think you should link to your moodle profile like Martin does (assuming you have one, which I assume <grin>). -- D.I. 29Jan06
Wow, you're fast.. .or I'm confused. --D.I. 29Jan06

:Hi D.I. thanks for your suggestion, which [[User:UrsHunkler|someone else]] seem to be following up! ;-) --[[User:Helen|Helen]] 05:00, 30 January 2006 (WST)

== German moodle docs possible? ==

Hello Helen,

thank you very much for these nice and structured doc pages.
When do you plan to launch a german part of the docs.moodle.org pages?
I ask these question because i can translate some of the english doc pages to german.

Greetings Christoph

:Hi Christoph, thanks for your kind offer of translation. Please email ''docs AT moodle DOT org'' so I may get in touch with you easily when the German Moodle documentation is launched. --[[User:Helen|Helen]] 05:17, 30 January 2006 (WST)

== German User Docs ==

Thanks a lot, Helen, for all your work.
I would also like to help with the German version.

Ulrike

:Hi Ulrike, thanks for your kind offer too. MoodleDocs in other languages will happen soon, for sure! --[[User:Helen|Helen]] 08:30, 31 January 2006 (WST)

== Documentation for different versions ==

Helen

Is there a mechanism for specifying which version of Moodle a particular document refers to? If, for instance, a feature is only available in version 1.6, how would a 1.5.3 user know they couldn't follow these instructions?

Equally, if something is done differently in two versions, how would we stop an edit war breaking out between 1.5.3 and 1.6 users?

I appreciate thir probably isn't an issue just yet, but it's possible it may become one in later versions.

Cheers,

[[User:CHRISF|CHRISF]] 19:20, 1 February 2006 (WST)