MoodleDocs - User contributions [en]

Grades FAQ

2008-12-03T07:15:02Z

Nicolasconnault:

{{Grades}}
== General ==

===How can I change how grades are displayed?===

Grades may be displayed as as actual grades, as percentages (in reference to the minimum and maximum grades) or as letters.

The default grade display type for the site is set by an administrator in ''Administration > Grades > [[Grade item settings]]''. However, this may be changed at course level.

To change how grades are displayed for particular [[Grade items|grade items]], or category and course summaries (called aggregations):

# Follow the grades link in the course administration block.
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon for the grade item, category total or course total.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button at the bottom of the page.

Alternatively, to change how grades are displayed for the whole course:

# Follow the grades link in the course administration block.
# Select "Course settings" from the gradebook dropdown menu.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button.

===How can I hide entered grades until a specified date?===

To set a "Hidden until" date:

#Access the course gradebook via the grades link in the course administration block.
#Select "Categories and items" from the gradebook dropdown menu.
#Click on the edit icon opposite the activity for which a "Hidden until" date is to be set.
#On the edit grade item page, ensure that advanced settings are displayed. (Click the "Show advanced" button if not.)
#Enable the "Hidden until" setting by unchecking the disable checkbox, then set a date.
#Click the "Save changes" button.

=== Is it possible to show the teachers/administrators' grades in the grader report? ===
Yes, at the site level you can define which roles will appear in the grader report. This can be found in [[General_grade_settings#Graded_Roles|Administration > Grades > General settings]]. Also read [http://moodle.org/mod/forum/discuss.php?d=92612 this discussion] for some more ideas.

===Why can't I change a grade within an assignment after changing it in the gradebook?===

When you edit a grade directly in the gradebook, an "overridden" flag is set, meaning that the grade can no longer be changed from within the assignment.

However, the flag can be removed by turning editing on in the [[Grader report|grader report]], then clicking the [[Grade editing|edit grade]] icon, unchecking the overridden box and saving the changes.

===How do I get groups to show up in the grader report?===

For groups to show up in the grader report, group mode should be set to visible or separate groups in the [[Course settings|course settings]]. This will result in a groups dropdown menu being displayed, enabling a teacher to view the grades of all participants, or only the grades for a selected group.

== Reports ==
=== How do I create my own custom gradebook reports? ===
Here is a [[Development:Gradebook_Report_Tutorial|tutorial]] explaining all the main steps involved.

== Aggregation ==
=== I can't find where to change the aggregation type for my gradebook categories! ===
Each category has an aggregation type, which can be changed through that category's "edit" page. To access that page, you must use one of 2 ways:

1. In the grader report, turn "Editing" on, then click the little "hand" icon next to the category whose aggregation you want to change
2. In the "Edit categories and Items" page (accessible through the "choose an action" menu, top left), you see a tree view of the categories and items in your gradebook. The top category is the course category. Each category also has a "hand" icon, which leads to the category edit page

=== How can I grade some of my activities without the results affecting my students' course total? ===
#Create two [[Grade categories]], one for your "activities still being graded," and one for your "released" activities.
#Ensure that "Aggregate including subcategories" (an advanced option) is unchecked for your top level course grade category.
##Where is this? In gradebook (grader report), in the upper right corner, click the "Turn Editing On" button.
##Click the edit icon next to the "course category" (usually your course name, just above the quiz names and below all the clickable links that were revealed when you turned editing on)
##Then make sure you have the "Show Advanced" option turned on.
#Edit the "activities still being graded" category's "course total" item. (This is one of the categories you created above.)
##Where is this? Look for the edit icon under "category total" that is below this category's name
#Set the "grade type" to "none".
#Tick the "Hidden" checkbox.
#Save your changes.
#Move all your activities being graded in the "activities still being graded" category.
#Move all your activities already graded in the "released" category.

Note: I rewrote this a bit, to help people find where things are. However, this method didn't seem to work for me on Moodle 1.9.

=== My student completed only one activity out of 5, but his course total shows 100%. How do I show a more "progressive" course total? ===
By default, only non-empty grades are aggregated, the others are ignored. However, you can change this setting as well as others that affect the course total, by turning "Editing" on in the grader report, and clicking the "Edit" icon next to the course category (the very top row of the grader report).

You can untick the box "Aggregate only non-empty grades" if you want to show a more "progressive" score for each student. Their empty grades will count as a 0 and will be counted in the course mean/total.

If you prefer to show a sum of points, rather than a percentage, you can change the course category's aggregation method to "Sum of grades".

=== How can I display the average grade for my course categories (not grade categories)? ===
In Moodle 1.9 there is no way to aggregate course totals within each category. The gradebook is course-centered, and there is currently no User Interface for showing grades within an entire course category at once.

=== How can I setup weighted assignments? ===
See [[Using "Weighted Mean of Grades" to weight categories containing assignments]].

== Categories ==
=== How many depths of categories/subcategories can I create? ===
There is no programmatic limit, but there are practical limits. Very deeply nested structures are difficult to manage. 3 levels of categories should be sufficient for most situations. Note that there is always at least one level of categories, since the Course category always encompasses all other categories and grade items, can cannot be deleted.

=== I can't find setting X in the grade category edit page! Where is it? ===
If a setting documented on the [[Grade categories]] page does not appear on your edit page, it may mean that it is set globally in your site. See [[Grade_category_settings#Forcing_settings|Forcing settings]] for more information.

== Outcomes ==
=== I have just upgraded to Moodle 1.9, and I want to set up an outcome item for my course. What are the steps required? ===
# Administration > Grades > General settings > [[General_grade_settings#Enable_outcomes|Enable outcomes]]
#[[Scales#Creating_a_new_scale|Create a scale]]
#Create a course outcome (read the [[Outcomes| outcomes documentation]] for instructions). Assign to it the scale you just created.
#Assign the outcome to your course
#Enter the "Grades" section of your course, from the course administration block
#In the Actions menu (top left), select Edit -> Categories and Items
#Click "Add outcome item"
#Follow the instructions of the [[Outcome items|outcome items documentation]] to create the outcome item

You can now give your students a rating on the outcome dimension you just created. If you created a standard outcome, you will be able to use it in other courses and follow your students' performance across these courses.

===How can I remove an outcome from an activity?===

An outcome can be removed from an activity by deleting it on the gradebook edit categories and items page. This results in the outcomes being deselected on the update activity page.

== Modules ==
=== The activity module (Module name) doesn't support grading. How can I give my students a grade anyway? ===
You can create a [[Grade_items#Manual_grade_items|grade item]] manually in the gradebook. You will have to grade your students through the [[Grader report]] interface (in editing mode).

=== I just graded some of my students using the (Module name) interface, but the results aren't showing up in the grader report. What's going on? ===
Here are some of the possible reasons:

#The corresponding [[Grade items|grade item]] is [[Grade_locking#In_grade_items|locked]], or its parent [[Grade categories|category]] is [[Grade_locking#In_grade_categories|locked]].
#The module code is not using the [[Development:Grades#API_for_communication_with_modules.2Fblocks|gradebook API]] correctly

=== I just created a new assignment with the "Grade" setting set to "No grade", but it still appears in the gradebook ===
The reason is that the gradebook is now the place where both numerical and textual types of feedback are recorded for all activity modules. The word "Grading" in assignment relates only to numerical grades, but the ability to give text feedback remains, and must be recorded in the gradebook. This is why a grade item is created for it. You can hide the grade item if you do not want it to appear in the user reports.

==Weights and extra credits==

===How do I create an assignment for which students can receive a grade higher than the maximum?===
---Documentation yet to be written---

==See also==
*[[Gradebook 1.9 Tutorial]]
*Gradebook Scenarios/Use Cases [https://docs.moodle.org/en/experimental:_gb_tutoring]
*Using Moodle [http://moodle.org/mod/forum/view.php?id=2122 Gradebook forum]

Using Moodle forum discussions:
*[http://moodle.org/mod/forum/discuss.php?d=102609 Can I aggregate only non hidden items?]

[[Category:FAQ]]

[[ca:PMF de les qualificacions]]
[[fr:FAQ des notes]]

Development:Repository plugins

2008-11-13T12:05:57Z

Nicolasconnault: /* Administration APIs */

A guide for developers on how to create a repository plugin.

== Introduction ==
===Prerequisites===
Before starting coding, it is necessary to know how to use repository administration pages and how to use the file picker.

===First steps===
# Create a folder for your plugin in ''moodle/repository/'' e.g. ''moodle/repository/myplugin''
# Create the following files and add them to the plugin folder:
#* ''repository.class.php''
#* ''icon.png'' (the icon displayed in the file picker)
# Create the language file ''repository_myplugin.php'' and add it to the plugin folder, keeping the following folder structure:
#*''moodle/repository/myplugin/lang/en_utf8/repository_myplugin.php'' or ''moodle/lang/en_utf8/repository_myplugin.php''

===Overview===

The 3 different parts to write
# Administration - You can customise the way administrators and users can configure their repositories.
# File picker integration - The core of your plugin, it will manage communication between Moodle and the repository service, and also the file picker display.
# I18n - Internationalization should be done at the same time as you're writing the other parts.

==Administration APIs==

As an example, let's create a Flickr plugin for accessing a public flickr account. The plugin will be called "Flickr Public".

Firstly the skeleton:

<code php>
<?php
/**
* repository_flickr_public class
* Moodle user can access public flickr account
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class repository_flickr_public extends repository {
}
?>
</code>

Then consider the question "What does my plugin do?"

In the Moodle file picker, we want to display some flickr public repositories directly linked to a flickr public account. For example ''My Public Flickr Pictures'', and also ''My Friend's Flickr Pictures''. When the user clicks on one of these repositories, the public pictures are displayed in the file picker.

In order to access to a flickr public account, the plugin needs to know the email address of the Flickr public account owner. So the administrator will need to set an email address for every repository. Let's add an "email address" setting to every repository.

<code php>
//We tell the API that the repositories have specific settings: "email address"
public static function get_instance_option_names() {
return array('email_address');
}

//We add an "email address" text box to the create/edit repository instance Moodle form
public function instance_config_form(&$mform) {
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', get_string('required'), 'required', null, 'client');
}
</code>

So at this moment all our Flickr Public Repositories will have a specific email address. However this is not enough. In order to communicate with Flickr, Moodle needs to know a Flickr API key (http://www.flickr.com/services/api/). This API key is the same for any repository. We could add it with the email address setting but the administrator would have to enter the same API key for every repository. Hopefully the administrator can add settings to the plugin level, impacting all repositories. The code is similar the repository instance settings:
<code php>
//We tell the API that the repositories have general settings: "api_key"
public static function get_type_option_names() {
return array('api_key');
}

//We add an "api key" text box to the create/edit repository plugin Moodle form (also called a Repository type Moodle form)
public function type_config_form(&$mform) {
//the following line is needed in order to retrieve the API key value from the database when Moodle displays the edit form
$api_key = get_config('flickr_public', 'api_key');
$mform->addElement('text', 'api_key', get_string('apikey', 'repository_flickr_public'),
array('value'=>$api_key,'size' => '40'));
$mform->addRule('api_key', get_string('required'), 'required', null, 'client');
}
</code>

Have we finished yet?

Yes! We have created everything necessary for the administration pages. But let's go further. It would be good if the user can enter any "Flickr public account email address" in the file picker. In fact we want to display in the file picker a Flickr Public repository that the Moodle administrator can never delete. Let's add:

<code php>
//this function is only called one time, when the Moodle administrator add the Flickr Public Plugin into the Moodle site.
public static function plugin_init() {
//here we create a default repository instance. The last parameter is 1 in order to set the instance as readonly.
repository_static_function('flickr_public','create', 'flickr_public', 0, get_system_context(),
array('name' => 'default instance','email_address' => null),1);
}
</code>

That's all - the administration part of our Flickr Public plugin is done. For your information, Box.net, Flickr, and Flickr Public all have similar administration APIs.

===Functions===
All of the following functions are optional. If they're not implemented, your plugin will not have manual settings and will have only one instance displayed in the File Picker (The repository API creates this unique instance when the administrator add the plugin).

==== get_instance_option_names====
''This function must be declared static''<br>
Return an array of strings. These strings are setting names. These settings are specific to an instance.
If the function returns an empty array, the API will consider that the plugin displays only one repository in the file picker.
Parent function returns an empty array.

====instance_config_form(&$mform)====
This is for modifying the Moodle form displaying the settings specific to an instance.

For example, to add a required text box called email_address:
<code php>
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', $strrequired, 'required', null, 'client');
</code>

''Note: ''mform'' has by default a name text box (cannot be removed).''

Parent function does nothing.

====get_type_option_names====
''This function must be declared static''<br>
Return an array of string. These strings are setting names. These settings are shared by all instances.
Parent function return an empty array.

====type_config_form(&$mform)====
This is for modifying the Moodle form displaying the plugin settings.
Similar to ''instance_config_form(&$mform)''

====plugin_init()====
''This function must be declared static''<br>
This function is called when the administrator adds the plugin. So unless the administrator deletes the plugin and re-adds it, it should be called only once.
Parent function does nothing.

==File picker APIs==
=== Quick Start ===
'''To be completed'''<br>
First of all, the File Picker using intensively Ajax you will need a easy way to debug. Install firePHP (MDL-16371) and make it works. It will save you a lot of time.

Your first question when you write your plugin specification is 'Does the user need to log-in'?<br>
.....

For most of plugins, you need to establish a connection with the remote repository. This connection can be done into the get_listing(), constructor() function...<br>
.....

You wanna display a list of files once the user is logged<br>
.... get_listing() .....

You wanna retrieve the file that the user selected<br>
....

Optional question that you should ask yourself is 'Does the user can execute a search'<br>
.... search() ....

===Functions you *MUST* override===
====__construct====
You may initialize your plugin here, such as:
# Get options from database
# Get user name and password from HTTP POST

====get_listing($path="")====
This function will return a list of files, the list must be a array like this:
<code php>
$list = array(
'path'=>'/var/repo/',
'manage'=>'http://webmgr.moodle.com',
'list'=> array(
array('title'=>'filename1', 'date'=>'01/01/2009', 'size'=>'10MB', 'source'=>'http://www.moodle.com/dl.rar'),
array('title'=>'folder', 'date'=>'01/01/2009', 'size'=>'0', 'children'=>array())
)
);
</code>
'''The full specification:'''
<code php>
array(
// iframe
'iframe' => (string) the url of the iframe, once this option is set, the right panel in file picker will display a iframe instead tree view or thumbnail view, in this mode, plugin developers could design the layout of repository freely, there is a function named repository_downoload_btn, it will display a download button which can help to move files to moodle file system.
// the current path
'path' => (string) path for the current folder
// dynload tells file picker to fetch list dynamically, when user click
// the folder, it will send a ajax request to server side.
'dynload' => (bool) use dynamic loading,
// will display a link in file picker
'manage' => (string) url of the file manager,
// set to true, the login link will be removed from file picker
'nologin' => (bool) requires login,
// set to true, the search link will be removed from file picker
'nosearch' => (bool) no search link,
// set this option will display a upload form in file picker
// only used in upload plugin currently
'upload' => array( // upload manager
'name' => (string) label of the form element,
'id' => (string) id of the form element
),
// file picker will build a file tree according this
// list
'list' => array(
array( // file
'title' => (string) file name,
'date' => (string) file last modification time, usually userdate(...),
'size' => (int) file size,
'thumbnail' => (string) url to thumbnail for the file,
'thumbnail_width' => (int) the width of the thumbnail image,
'source' => plugin-dependent unique path to the file (id, url, path, etc.),
'url'=> the accessible url of file
),
array( // folder - same as file, but no 'source'.
'title' => (string) folder name,
'path' => (string) path to this folder
'date' => (string) folder last modification time, usually userdate(...),
'size' => 0,
'thumbnail' => (string) url to thumbnail for the folder,
'children' => array( // an empty folder needs to have 'children' defined, but empty.
// content (files and folders)
)
),
)
)
</code>
Dynamically loading
Some repositories contain many files which cannot load in one time, in this case, we need dynamically loading to fetch them step by step, files in subfolder won't be listed until user click the folder in file picker treeview.

As a plug-in developer, if you set dynload flag as '''true''', you should return files and folders (set children as a null array) in current path instead of building the whole file tree.

Example of dynamically loading
See [http://cvs.moodle.org/moodle/repository/alfresco/repository.class.php?view=log Alfresco] plug-in

===Functions you can override===
====print_login====
This function will help to print a login form, for the Ajax file picker, this function will return a
PHP array to define this form.
<code php>
public function print_login(){
if ($this->options['ajax']) {
$user_field->label = get_string('username', 'repository_boxnet').': ';
$user_field->id = 'box_username';
$user_field->type = 'text';
$user_field->name = 'boxusername';
$user_field->value = $ret->username;

$passwd_field->label = get_string('password', 'repository_boxnet').': ';
$passwd_field->id = 'box_password';
$passwd_field->type = 'password';
$passwd_field->name = 'boxpassword';

$ret = array();
$ret['login'] = array($user_field, $passwd_field);
return $ret;
}
}
</code>
This will help to generate a form by file picker which contains user name and password input elements.

If your plugin don't require logging in, you don't need to override it, it will call get_listing to list files automatically by default.
====check_login====
This function will return a boolean value to tell Moodle whether the user has logged in.
By default, this function will return true.
====logout====
When a user clicks the logout button in file picker, this function will be called. You may clean up the session or disconnect the connection with remote server here.
====global_search====
Moodle Repository API supports global search, this function will return a boolean value to tell Moodle if this plugin is ready to search. By default, it will return false - you need to override it to enable global search.
====print_search====
When a user clicks the search button on file picker, this function will be called to return a search form. By default, it will create a form with single search bar - you can override it to create a advanced search form.
====search====
This function will do the searching job. You can obtain the POST parameters from the from the form you created in print_search function
This function will return a file list exactly like the one from get_listing.
====get_file====
When a user clicks the "Get" button to transfer the file, this function will be called. Basically, it will download a file to Moodle - you can override it to modify the file then move it to a better location.
====get_name====
This function will return the name of the repository instance.

== I18n - Internationalization ==
These following strings are required in ''moodle/repository/myplugin/lang/en_utf8/repository_myplugin.php'' or ''moodle/lang/en_utf8/repository_myplugin.php'':

<code php>
$string['configplugin'] = 'Flickr Public configuration';
$string['repositorydesc'] = 'A Flickr public repository';
$string['repositoryname'] = 'Flickr Public';
</code>

== Standard repository plugins ==
This is the functional specification list of the officially supported repository plugins.
For each plugins, the two mains part we are interested in are:
* How do I administrate the plugin? See [[Development:Repository_Administration_Specification| Repository Administration Specification - UC001-3]]
* How do I set up an account for this repository? See [[Development:Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]

=== Functional specifications ===
*[[Development:Box.net Repository Plugin|Box.net Repository Plugin]]
*[[Development:Flickr Repository Plugin|Flickr Repository Plugin]]
*[[Development:Moodle Repository Plugin|Moodle Local Repository Plugin]]
*[[Development:MNET Repository Plugin|Moodle MNET Repository Plugin]]

==See also==

*[[Development:Repository API| Repository API]]
*[[Development:Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
* MDL-16543 - A list of officially supported repository plugins

[[Category:Repositories]]

Grades FAQ

2008-11-10T13:45:21Z

Nicolasconnault: /* Modules */

{{Grades}}
== General ==

===How can I change how grades are displayed?===

Grades may be displayed as as actual grades, as percentages (in reference to the minimum and maximum grades) or as letters.

The default grade display type for the site is set by an administrator in ''Administration > Grades > [[Grade item settings]]''. However, this may be changed at course level.

To change how grades are displayed for particular [[Grade items|grade items]], or category and course summaries (called aggregations):

# Follow the grades link in the course administration block.
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon for the grade item, category total or course total.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button at the bottom of the page.

Alternatively, to change how grades are displayed for the whole course:

# Follow the grades link in the course administration block.
# Select "Course settings" from the gradebook dropdown menu.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button.

===How can I hide entered grades until a specified date?===

To set a "Hidden until" date:

#Access the course gradebook via the grades link in the course administration block.
#Select "Categories and items" from the gradebook dropdown menu.
#Click on the edit icon opposite the activity for which a "Hidden until" date is to be set.
#On the edit grade item page, ensure that advanced settings are displayed. (Click the "Show advanced" button if not.)
#Enable the "Hidden until" setting by unchecking the disable checkbox, then set a date.
#Click the "Save changes" button.

=== Is it possible to show the teachers/administrators' grades in the grader report? ===
Yes, at the site level you can define which roles will appear in the grader report. This can be found in [[General_grade_settings#Graded_Roles|Administration > Grades > General settings]]. Also read [http://moodle.org/mod/forum/discuss.php?d=92612 this discussion] for some more ideas.

===Why can't I change a grade within an assignment after changing it in the gradebook?===

When you edit a grade directly in the gradebook, an "overridden" flag is set, meaning that the grade can no longer be changed from within the assignment.

However, the flag can be removed by turning editing on in the [[Grader report|grader report]], then clicking the [[Grade editing|edit grade]] icon, unchecking the overridden box and saving the changes.

===How do I get groups to show up in the grader report?===

For groups to show up in the grader report, group mode should be set to visible or separate groups in the [[Course settings|course settings]]. This will result in a groups dropdown menu being displayed, enabling a teacher to view the grades of all participants, or only the grades for a selected group.

== Reports ==
=== How do I create my own custom gradebook reports? ===
Here is a [[Development:Gradebook_Report_Tutorial|tutorial]] explaining all the main steps involved.

== Aggregation ==
=== I can't find where to change the aggregation type for my gradebook categories! ===
Each category has an aggregation type, which can be changed through that category's "edit" page. To access that page, you must use one of 2 ways:

1. In the grader report, turn "Editing" on, then click the little "hand" icon next to the category whose aggregation you want to change
2. In the "Edit categories and Items" page (accessible through the "choose an action" menu, top left), you see a tree view of the categories and items in your gradebook. The top category is the course category. Each category also has a "hand" icon, which leads to the category edit page

=== How can I grade some of my activities without the results affecting my students' course total? ===
#Create two [[Grade categories]], one for your "activities still being graded," and one for your "released" activities.
#Ensure that "Aggregate including subcategories" (an advanced option) is unchecked for your top level course grade category.
##Where is this? In gradebook (grader report), in the upper right corner, click the "Turn Editing On" button.
##Click the edit icon next to the "course category" (usually your course name, just above the quiz names and below all the clickable links that were revealed when you turned editing on)
##Then make sure you have the "Show Advanced" option turned on.
#Edit the "activities still being graded" category's "course total" item. (This is one of the categories you created above.)
##Where is this? Look for the edit icon under "category total" that is below this category's name
#Set the "grade type" to "none".
#Tick the "Hidden" checkbox.
#Save your changes.
#Move all your activities being graded in the "activities still being graded" category.
#Move all your activities already graded in the "released" category.

Note: I rewrote this a bit, to help people find where things are. However, this method didn't seem to work for me on Moodle 1.9.

=== My student completed only one activity out of 5, but his course total shows 100%. How do I show a more "progressive" course total? ===
By default, only non-empty grades are aggregated, the others are ignored. However, you can change this setting as well as others that affect the course total, by turning "Editing" on in the grader report, and clicking the "Edit" icon next to the course category (the very top row of the grader report).

You can untick the box "Aggregate only non-empty grades" if you want to show a more "progressive" score for each student. Their empty grades will count as a 0 and will be counted in the course mean/total.

If you prefer to show a sum of points, rather than a percentage, you can change the course category's aggregation method to "Sum of grades".

=== How can I display the average grade for my course categories (not grade categories)? ===
In Moodle 1.9 there is no way to aggregate course totals within each category. The gradebook is course-centered, and there is currently no User Interface for showing grades within an entire course category at once.

=== How can I setup weighted assignments? ===
See [[Using "Weighted Mean of Grades" to weight categories containing assignments]].

== Categories ==
=== How many depths of categories/subcategories can I create? ===
There is no programmatic limit, but there are practical limits. Very deeply nested structures are difficult to manage. 3 levels of categories should be sufficient for most situations. Note that there is always at least one level of categories, since the Course category always encompasses all other categories and grade items, can cannot be deleted.

=== I can't find setting X in the grade category edit page! Where is it? ===
If a setting documented on the [[Grade categories]] page does not appear on your edit page, it may mean that it is set globally in your site. See [[Grade_category_settings#Forcing_settings|Forcing settings]] for more information.

== Outcomes ==
=== I have just upgraded to Moodle 1.9, and I want to set up an outcome item for my course. What are the steps required? ===
#[[General_grade_settings#Enable_outcomes|Administration > Grades > General settings > Enable outcomes]]
#[[Scales#Creating_a_new_scale|Create a scale]]
#Create a course outcome (read the [[Outcomes| outcomes documentation]] for instructions). Assign to it the scale you just created.
#Assign the outcome to your course
#Enter the "Grades" section of your course, from the course administration block
#In the Actions menu (top left), select Edit -> Categories and Items
#Click "Add outcome item"
#Follow the instructions of the [[Outcome items|outcome items documentation]] to create the outcome item

You can now give your students a rating on the outcome dimension you just created. If you created a standard outcome, you will be able to use it in other courses and follow your students' performance across these courses.

== Modules ==
=== The activity module (Module name) doesn't support grading. How can I give my students a grade anyway? ===
You can create a [[Grade_items#Manual_grade_items|grade item]] manually in the gradebook. You will have to grade your students through the [[Grader report]] interface (in editing mode).

=== I just graded some of my students using the (Module name) interface, but the results aren't showing up in the grader report. What's going on? ===
Here are some of the possible reasons:

#The corresponding [[Grade items|grade item]] is [[Grade_locking#In_grade_items|locked]], or its parent [[Grade categories|category]] is [[Grade_locking#In_grade_categories|locked]].
#The module code is not using the [[Development:Grades#API_for_communication_with_modules.2Fblocks|gradebook API]] correctly

=== I just created a new assignment with the "Grade" setting set to "No grade", but it still appears in the gradebook ===
The reason is that the gradebook is now the place where both numerical and textual types of feedback are recorded for all activity modules. The word "Grading" in assignment relates only to numerical grades, but the ability to give text feedback remains, and must be recorded in the gradebook. This is why a grade item is created for it. You can hide the grade item if you do not want it to appear in the user reports.

==See also==
*[[Gradebook 1.9 Tutorial]]
*Using Moodle [http://moodle.org/mod/forum/view.php?id=2122 Gradebook forum]

Using Moodle forum discussions:
*[http://moodle.org/mod/forum/discuss.php?d=102609 Can I aggregate only non hidden items?]

[[Category:FAQ]]

[[ca:PMF de les qualificacions]]
[[fr:FAQ des notes]]

Talk:experimental: SOP multicats

2008-11-10T09:47:48Z

Nicolasconnault:

You don't need to use calculations, in fact. You can simply set the "uncategorised" category's grade type to "None". This way, it will not be counted in the course total. You can also hide this category to prevent it from appearing in the students' report.[[User:Nicolas Connault|Nicolas Connault]] 03:47, 10 November 2008 (CST)

Talk:experimental: SOP multicats

2008-11-10T09:47:37Z

Nicolasconnault: New page: You don't need to use calculations, in fact. You can simply set the "uncategorised" category's grade type to "None". This way, it will not be counted in the course total. You can also hide...

User:Nicolas Connault

2008-11-07T13:44:24Z

Nicolasconnault: /* Beginnings at Moodle HQ */

==Contact Details==
ICQ: 826611
Yahoo : nicolasconnault
MSN: nikozeta@hotmail.com
AIM: NikoZeta
Skype: nicolasconnault
Email: nicolasconnault@gmail.com
Cell: 06 11 84 13 08

==First steps towards Moodle==
Born in 1978 in France, I have 4 brothers and 1 sister. At age 19, I spent 2 years on a mission in England for the [http://www.mormon.org Church of Jesus Christ of Latter-Day Saints]. There I learned much about human nature and saw much misery and suffering, mostly caused by dysfunctional relationships between people, especially in families. I decided I would study psychology and become a family counselor, to help alleviate the hurt I saw all around me.

I returned home to France in 2000, and shortly thereafter met my wife Anne-Marie while chatting on ICQ. We discovered we had similar interests, beliefs and values, and within 2 weeks (!?) I proposed to her over the Internet. I hadn't yet seen a picture of her. To me, physical appearance was not a concern.

She came over to France 6 (long!) months later, and we were married in France (civilly) and in England (for eternity in the Temple). A few months later we moved to England for a job which did not work out as expected, following which we moved to Western Australia in 2001.

==University and Self-teaching==
There I started taking interest in Web Development as a hobby (I had already built a couple of static websites as a youth), and used the many resources available online to teach myself LAMP/WAMP development. When I started my psychology degree (externally at Edith Cowan University), I chose a minor in professional computing, which included Object-Oriented programming, Database design and Web Development, but these added little to the knowledge I had already gained through learning online. The vibrant PHP community was the greatest source of learning for me.

A few years ago I undertook my first serious open source project with two other friends, Christopher Vance and Vickie Comrie (both from the USA). The project was a PHP auction framework, along the lines of Ebay but much more simple, which could be implemented by small companies wanting an auction system. It was a voluntary project for a fairly large american charity. This project hasn't been maintained for years, and I would probably cringe if I looked at the source code now, but the skills I learned while working on it have proven invaluable.

The code for the auction project attracted the attention of a small web development company, Triangle Solutions, who contracted me immediately on a part-time basis, later to become full-time. During the 1.5 years I worked for Triangle, I contributed to many different projects, but the most important was PHP Support Tickets, an application originally written in horrible spaghetti PHP which I was asked to upgrade. My decision was to rewrite it using Object-Oriented principles, of which I knew enough to improve the application at the time. Unfortunately I did not have the time resources available to devote enough time on this project, which greatly hindered its development. I still think it has great potential, but I would certainly refactor it if I had the opportunity to work on it again.

==Beginnings at Moodle HQ==
In December 2006, I applied for the developer position at Moodle HQ in Perth, and was hired on a full-time basis. I started out doing mainly bug-squashing for version 1.8, but was also made responsible for unit testing, an area which had been greatly neglected until then. As soon as 1.8 was stable and released, I started working on the new gradebook internals for 1.9, together with Yu Zhang, Martin Dougiamas and Petr Skoda.

==Other programming interests==
=== Ruby ===
One last note on development: I have recently taken great interest in the Ruby language, although that interest started much earlier than the current Ruby on Rails craze. I find the language extremely elegant, intuitive and fun to write in. It enables me to be much more creative than with PHP or Java. I also enjoy the Rails framework, although it certainly isn't as easy to learn as some people would have you believe. I am currently working on a real estate application (sales and rentals) written on that platform. You can see the front-end for one of my clients at [http://www.glenmarrealty.com.au Glenmar Realty].

=== Test-Driven Development (TDD) ===
Since my early days with Object-Oriented languages, I have been a strong proponent of unit testing, and a keen supporter of the Test-Driven Development approach. Unfortunately it is not always easy to adopt such methodology when working on a project like Moodle, with much legacy and non-OO code. The approach was successfully used when developing the new gradebook for Moodle 1.9, although it could have used better.

In the last 6 months I have given 3 presentations on the subject of Unit Testing. One was at the [http://elearning.lse.ac.uk/blogs/clt/?p=251 2007 UK Moodle Moot], and the other two were given at two different branches of the BCS, [http://nottmderby.bcs.org/events08-feb.htm Derby-Nottingham] and [http://www.herts.bcs.org/past.htm Hertfordshire]. [http://moodlemoot.org/file.php/4/moddata/assignment/1/438/presentation.ppt My slides] are available at these sites, but there are minor differences between them, since I try to improve my presentation each time.

==Psychology==
On the topic of psychology: during my external studies, I became more and more disillusioned with the traditional counseling practice. I felt that, although the intentions of most professional psychologists were noble, their skills were often exercised within an ideological framework of manufactured needs. This meant that often, their contributions to the well-being of the human race were more imagined than real.

My interests diverged towards community development, group processes, interpersonal relationships, child development and the learning process. I became aware that people tended to get better emotionally when they had a strong support network, and that non-professional assistance in the form of support groups was often more effective than prohibitory and extensive therapy sessions.

I also became more and more disenchanted with the educational practices of the universities in Australia. They seemed to be based on ancient traditions and to ignore the very principles they were teaching. I especially abhorred the lecture medium of teaching, which was one reason why I chose to study externally. However I did a few units on campus and was glad to see some tutors trying to change their delivery methods to something more informal and participative. Overall I was extremely disappointed in the appalling quality of external teaching. It was extremely difficult to connect with other students, and you only got help if you persistently sought for it. Feedback was as rare (and precious) as gold. BlackBoard was inadequate in many ways, and most tutors didn't even use it. The drop-out rates from external courses was so great that it was very difficult to extract that figure from university staff. I wouldn't be surprised if only around 10% of all external students across all Australian universities actually stuck to their entire course, not to mention those who stay but fail.

My project for the honours degree (which I have delayed for next year) was to compare the teaching methods of a German university with those of Edith Cowan University, and to compare the measurable results. The project for my doctorate was to develop an online, open source software suite that would enable external students to connect with each other as a vibrant community, in order to facilitate self-motivated learning and participation. This was a major factor in me accepting the job with Moodle, which represented exactly what I had in mind.

Well, this is a rather lengthy description, but those that have read all of it now know a lot about me, and will be better able to understand the person behind the posts and the code.

Talk:Grades FAQ

2008-11-07T13:39:29Z

Nicolasconnault:

This FAQ asks begins answering the question '''Why is the new gradebook so complicated?'''

There has been considerable discussion about usability as Moodle has grown, including questioning whether adding new features necessarily requires a more complicated interface. Some have argued (including this author) that more field testing, carefully chosen defaults, and progressive revealing of features (including an "advanced" button) can make it much easier for new users while making it possible to make available features selectivity available as needed. During the Beta process, progress was made on this, but additional improvements are probably quite possible.

An obvious example would have been to retain the <1.9 interface as a legacy option, but built on top of the new code. The counter-argument is to make all options available to every user so that they know that the feature is there, but this has a high cost in terms of usability as a product becomes feature-rich. --[[User:Gary Anderson|Gary Anderson]] 10:59, 29 March 2008 (CDT)

:We couldn't retain the <1.9 interface, we would have had to re-code it entirely. The main reason for developing the new gradebook was internal, to enable long-term scalability, extensibility and consistency. We don't expect end users to understand or be interested in what goes on behind the User Interface, in the database, in the backend code, but it is true that usability has suffered as a result. The good news is that our efforts are now entirely focused on usability instead of these other important considerations. [[User:Nicolas Connault|Nicolas Connault]] 07:39, 7 November 2008 (CST)

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T10:41:50Z

Nicolasconnault: /* Advanced features */

{{Moodle 2.0}}

One of the goals of Moodle 2.0 is to improve the '''usability''' of the gradebook introduced in 1.9.

The purpose of this page is to present a list of proposed improvements, and gather feedback on these implementations before they make it into this important release. These proposed improvements are based on bug reports and forum posts that have been coming in since the 1.9 release.

We would really like your feedback on:
# the changes already proposed here
# any issues that you feel are not yet addressed

Please join the discussions and post your suggestions in the [http://moodle.org/mod/forum/view.php?f=397 Gradebook forum].

==Common frustrations==
Below are some of the most often reported usability issues in the 1.9 gradebook, which we are trying to address. Several of these are grouped as sub-tasks under MDL-16913.

===Assigning weights to categories and grade items===
This is the most common frustration. Weights are used extensively in many institutions, and were used in 1.8. However, they are difficult to understand in 1.9, and difficult to set up. This issue is discussed extensively in MDL-15680.

===Moving items to categories===
Currently, the [[Edit categories and items]] page lets you move only one grade item or category at a time. It takes two page refreshes and two mouse clicks per move. This is very tedious and time-consuming when many items have to be moved around. MDL-13775 and MDL-12502 address this problem.

===Removing the ''overridden'' attribute of individual grades===
When grades are imported into an existing gradebook, or when grades are manually edited in the grader report, these grades become [[Grade_editing#Overridden|overridden]], which prevents linked activity modules from updating this grade. Removing this attribute is very time-consuming, since one must enter the edit page of each individual grade, untick the checkbox and submit the form. This issue is discussed in [http://moodle.org/mod/forum/discuss.php?d=109636 this forum thread].

===Viewing the overall contribution of each category and item to the course aggregation===
The tracker issue MDL-13777 reports the difficulty in seeing the contribution of each grade item and category to the course total. Calculations and weights may apply, which are not visible in any of the current reports except in each individual category or item's edit page.

==Patch for Edit Categories and Items page==
Following is a proposed patch to the '''Edit categories and items''' page in the 1.9 gradebook. It addresses many usability issues while remaining simple. The following features are currently implemented:

===Simple features===
[[Image:gradebook_categories_simple.png|600px]]
*Items and categories are displayed in a table instead of a list. This improves readability and will make drag and drop easier to implement.
*All advanced features are hidden by default, so that the interface by default is almost identical to the original
*An additional column has checkboxes for grade items. All selected items can then be moved to one of the existing grade categories for the current course. Javascript "Select all/Select None" links replace these checkboxes for categories, to make it faster to select or de-select items.

===Advanced features===
[[Image:gradebook_categories_advanced.png|600px]]

*[[Category aggregation|Grade category aggregation type]]: changing this reloads the page, because the grade item weights/extra credits only apply to some of these aggregation types.
*Weight or extra credit: depending on the parent category's aggregation type, each category and grade item may have an input field or a checkbox for weight or extra credit. All these can be edited, then submitted with one click.
*Grade range is displayed for information purposes, but cannot be edited through this interface (it is most often controlled by the linked activity)
*[[Grade_categories#Aggregate_only_non-empty_grades|Aggregate only non-empty grades]]
*[[Grade_categories#Aggregate_including_sub-categories|Aggregate including sub-categories]]
*[[Grade_categories#Include_outcomes_in_aggregation|Include outcomes in aggregation]]
*[[Grade_categories#Drop_the_lowest|Drop the lowest]]
*[[Grade_categories#Keep_the_highest|Keep the highest]]
*[[Grade items|Multiplicator]]
*[[Grade_items|Offset]]

All these settings can be changed then submitted with only one page reload. Just be mindful that changing a category's aggregation type will reload the page and you will lose any non-submitted changes you made to the other settings.

File:gradebook categories advanced.png

2008-11-04T10:41:22Z

Nicolasconnault: The new "Edit categories and items" interface, with advanced settings displayed.

The new "Edit categories and items" interface, with advanced settings displayed.

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T10:40:51Z

Nicolasconnault: /* Simple features */

{{Moodle 2.0}}

One of the goals of Moodle 2.0 is to improve the '''usability''' of the gradebook introduced in 1.9.

The purpose of this page is to present a list of proposed improvements, and gather feedback on these implementations before they make it into this important release. These proposed improvements are based on bug reports and forum posts that have been coming in since the 1.9 release.

We would really like your feedback on:
# the changes already proposed here
# any issues that you feel are not yet addressed

Please join the discussions and post your suggestions in the [http://moodle.org/mod/forum/view.php?f=397 Gradebook forum].

==Common frustrations==
Below are some of the most often reported usability issues in the 1.9 gradebook, which we are trying to address. Several of these are grouped as sub-tasks under MDL-16913.

===Assigning weights to categories and grade items===
This is the most common frustration. Weights are used extensively in many institutions, and were used in 1.8. However, they are difficult to understand in 1.9, and difficult to set up. This issue is discussed extensively in MDL-15680.

===Moving items to categories===
Currently, the [[Edit categories and items]] page lets you move only one grade item or category at a time. It takes two page refreshes and two mouse clicks per move. This is very tedious and time-consuming when many items have to be moved around. MDL-13775 and MDL-12502 address this problem.

===Removing the ''overridden'' attribute of individual grades===
When grades are imported into an existing gradebook, or when grades are manually edited in the grader report, these grades become [[Grade_editing#Overridden|overridden]], which prevents linked activity modules from updating this grade. Removing this attribute is very time-consuming, since one must enter the edit page of each individual grade, untick the checkbox and submit the form. This issue is discussed in [http://moodle.org/mod/forum/discuss.php?d=109636 this forum thread].

===Viewing the overall contribution of each category and item to the course aggregation===
The tracker issue MDL-13777 reports the difficulty in seeing the contribution of each grade item and category to the course total. Calculations and weights may apply, which are not visible in any of the current reports except in each individual category or item's edit page.

==Patch for Edit Categories and Items page==
Following is a proposed patch to the '''Edit categories and items''' page in the 1.9 gradebook. It addresses many usability issues while remaining simple. The following features are currently implemented:

===Simple features===
[[Image:gradebook_categories_simple.png|600px]]
*Items and categories are displayed in a table instead of a list. This improves readability and will make drag and drop easier to implement.
*All advanced features are hidden by default, so that the interface by default is almost identical to the original
*An additional column has checkboxes for grade items. All selected items can then be moved to one of the existing grade categories for the current course. Javascript "Select all/Select None" links replace these checkboxes for categories, to make it faster to select or de-select items.

===Advanced features===
[[Image:gradebook_categories_advanced.png]]
*[[Category aggregation|Grade category aggregation type]]: changing this reloads the page, because the grade item weights/extra credits only apply to some of these aggregation types.
*Weight or extra credit: depending on the parent category's aggregation type, each category and grade item may have an input field or a checkbox for weight or extra credit. All these can be edited, then submitted with one click.
*Grade range is displayed for information purposes, but cannot be edited through this interface (it is most often controlled by the linked activity)
*[[Grade_categories#Aggregate_only_non-empty_grades|Aggregate only non-empty grades]]
*[[Grade_categories#Aggregate_including_sub-categories|Aggregate including sub-categories]]
*[[Grade_categories#Include_outcomes_in_aggregation|Include outcomes in aggregation]]
*[[Grade_categories#Drop_the_lowest|Drop the lowest]]
*[[Grade_categories#Keep_the_highest|Keep the highest]]
*[[Grade items|Multiplicator]]
*[[Grade_items|Offset]]

All these settings can be changed then submitted with only one page reload. Just be mindful that changing a category's aggregation type will reload the page and you will lose any non-submitted changes you made to the other settings.

File:gradebook categories simple.png

2008-11-04T10:39:51Z

Nicolasconnault: The new "Edit categories and items" interface, with advanced settings hidden.

The new "Edit categories and items" interface, with advanced settings hidden.

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T10:39:10Z

Nicolasconnault: /* Patch for Edit Categories and Items page */

{{Moodle 2.0}}

One of the goals of Moodle 2.0 is to improve the '''usability''' of the gradebook introduced in 1.9.

The purpose of this page is to present a list of proposed improvements, and gather feedback on these implementations before they make it into this important release. These proposed improvements are based on bug reports and forum posts that have been coming in since the 1.9 release.

We would really like your feedback on:
# the changes already proposed here
# any issues that you feel are not yet addressed

Please join the discussions and post your suggestions in the [http://moodle.org/mod/forum/view.php?f=397 Gradebook forum].

==Common frustrations==
Below are some of the most often reported usability issues in the 1.9 gradebook, which we are trying to address. Several of these are grouped as sub-tasks under MDL-16913.

===Assigning weights to categories and grade items===
This is the most common frustration. Weights are used extensively in many institutions, and were used in 1.8. However, they are difficult to understand in 1.9, and difficult to set up. This issue is discussed extensively in MDL-15680.

===Moving items to categories===
Currently, the [[Edit categories and items]] page lets you move only one grade item or category at a time. It takes two page refreshes and two mouse clicks per move. This is very tedious and time-consuming when many items have to be moved around. MDL-13775 and MDL-12502 address this problem.

===Removing the ''overridden'' attribute of individual grades===
When grades are imported into an existing gradebook, or when grades are manually edited in the grader report, these grades become [[Grade_editing#Overridden|overridden]], which prevents linked activity modules from updating this grade. Removing this attribute is very time-consuming, since one must enter the edit page of each individual grade, untick the checkbox and submit the form. This issue is discussed in [http://moodle.org/mod/forum/discuss.php?d=109636 this forum thread].

===Viewing the overall contribution of each category and item to the course aggregation===
The tracker issue MDL-13777 reports the difficulty in seeing the contribution of each grade item and category to the course total. Calculations and weights may apply, which are not visible in any of the current reports except in each individual category or item's edit page.

==Patch for Edit Categories and Items page==
Following is a proposed patch to the '''Edit categories and items''' page in the 1.9 gradebook. It addresses many usability issues while remaining simple. The following features are currently implemented:

===Simple features===
[[Image:gradebook_categories_simple.png]]
*Items and categories are displayed in a table instead of a list. This improves readability and will make drag and drop easier to implement.
*All advanced features are hidden by default, so that the interface by default is almost identical to the original
*An additional column has checkboxes for grade items. All selected items can then be moved to one of the existing grade categories for the current course. Javascript "Select all/Select None" links replace these checkboxes for categories, to make it faster to select or de-select items.

===Advanced features===
[[Image:gradebook_categories_advanced.png]]
*[[Category aggregation|Grade category aggregation type]]: changing this reloads the page, because the grade item weights/extra credits only apply to some of these aggregation types.
*Weight or extra credit: depending on the parent category's aggregation type, each category and grade item may have an input field or a checkbox for weight or extra credit. All these can be edited, then submitted with one click.
*Grade range is displayed for information purposes, but cannot be edited through this interface (it is most often controlled by the linked activity)
*[[Grade_categories#Aggregate_only_non-empty_grades|Aggregate only non-empty grades]]
*[[Grade_categories#Aggregate_including_sub-categories|Aggregate including sub-categories]]
*[[Grade_categories#Include_outcomes_in_aggregation|Include outcomes in aggregation]]
*[[Grade_categories#Drop_the_lowest|Drop the lowest]]
*[[Grade_categories#Keep_the_highest|Keep the highest]]
*[[Grade items|Multiplicator]]
*[[Grade_items|Offset]]

All these settings can be changed then submitted with only one page reload. Just be mindful that changing a category's aggregation type will reload the page and you will lose any non-submitted changes you made to the other settings.

Grades FAQ

2008-11-04T10:37:03Z

Nicolasconnault: I can't find setting X in the grade category edit page! Where is it?

{{Grades}}
== General ==

===How can I change how grades are displayed?===

Grades may be displayed as as actual grades, as percentages (in reference to the minimum and maximum grades) or as letters.

The default grade display type for the site is set by an administrator in ''Administration > Grades > [[Grade item settings]]''. However, this may be changed at course level.

To change how grades are displayed for particular [[Grade items|grade items]], or category and course summaries (called aggregations):

# Follow the grades link in the course administration block.
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon for the grade item, category total or course total.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button at the bottom of the page.

Alternatively, to change how grades are displayed for the whole course:

# Follow the grades link in the course administration block.
# Select "Course settings" from the gradebook dropdown menu.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button.

===How can I hide entered grades until a specified date?===

To set a "Hidden until" date:

#Access the course gradebook via the grades link in the course administration block.
#Select "Categories and items" from the gradebook dropdown menu.
#Click on the edit icon opposite the activity for which a "Hidden until" date is to be set.
#On the edit grade item page, ensure that advanced settings are displayed. (Click the "Show advanced" button if not.)
#Enable the "Hidden until" setting by unchecking the disable checkbox, then set a date.
#Click the "Save changes" button.

=== Is it possible to show the teachers/administrators' grades in the grader report? ===
Yes, at the site level you can define which roles will appear in the grader report. This can be found in [[General_grade_settings#Graded_Roles|Administration > Grades > General settings]]. Also read [http://moodle.org/mod/forum/discuss.php?d=92612 this discussion] for some more ideas.

===Why can't I change a grade within an assignment after changing it in the gradebook?===

When you edit a grade directly in the gradebook, an "overridden" flag is set, meaning that the grade can no longer be changed from within the assignment.

However, the flag can be removed by turning editing on in the [[Grader report|grader report]], then clicking the [[Grade editing|edit grade]] icon, unchecking the overridden box and saving the changes.

===How do I get groups to show up in the grader report?===

For groups to show up in the grader report, group mode should be set to visible or separate groups in the [[Course settings|course settings]]. This will result in a groups dropdown menu being displayed, enabling a teacher to view the grades of all participants, or only the grades for a selected group.

== Reports ==
=== How do I create my own custom gradebook reports? ===
Here is a [[Development:Gradebook_Report_Tutorial|tutorial]] explaining all the main steps involved.

== Aggregation ==
=== I can't find where to change the aggregation type for my gradebook categories! ===
Each category has an aggregation type, which can be changed through that category's "edit" page. To access that page, you must use one of 2 ways:

1. In the grader report, turn "Editing" on, then click the little "hand" icon next to the category whose aggregation you want to change
2. In the "Edit categories and Items" page (accessible through the "choose an action" menu, top left), you see a tree view of the categories and items in your gradebook. The top category is the course category. Each category also has a "hand" icon, which leads to the category edit page

=== How can I grade some of my activities without the results affecting my students' course total? ===
#Create two [[Grade categories]], one for your "activities still being graded," and one for your "released" activities.
#Ensure that "Aggregate including subcategories" (an advanced option) is unchecked for your top level course grade category.
##Where is this? In gradebook (grader report), in the upper right corner, click the "Turn Editing On" button.
##Click the edit icon next to the "course category" (usually your course name, just above the quiz names and below all the clickable links that were revealed when you turned editing on)
##Then make sure you have the "Show Advanced" option turned on.
#Edit the "activities still being graded" category's "course total" item. (This is one of the categories you created above.)
##Where is this? Look for the edit icon under "category total" that is below this category's name
#Set the "grade type" to "none".
#Tick the "Hidden" checkbox.
#Save your changes.
#Move all your activities being graded in the "activities still being graded" category.
#Move all your activities already graded in the "released" category.

Note: I rewrote this a bit, to help people find where things are. However, this method didn't seem to work for me on Moodle 1.9.

=== My student completed only one activity out of 5, but his course total shows 100%. How do I show a more "progressive" course total? ===
By default, only non-empty grades are aggregated, the others are ignored. However, you can change this setting as well as others that affect the course total, by turning "Editing" on in the grader report, and clicking the "Edit" icon next to the course category (the very top row of the grader report).

You can untick the box "Aggregate only non-empty grades" if you want to show a more "progressive" score for each student. Their empty grades will count as a 0 and will be counted in the course mean/total.

If you prefer to show a sum of points, rather than a percentage, you can change the course category's aggregation method to "Sum of grades".

=== How can I display the average grade for my course categories (not grade categories)? ===
In Moodle 1.9 there is no way to aggregate course totals within each category. The gradebook is course-centered, and there is currently no User Interface for showing grades within an entire course category at once.

=== How can I setup weighted assignments? ===
See [[Using "Weighted Mean of Grades" to weight categories containing assignments]].

== Categories ==
=== How many depths of categories/subcategories can I create? ===
There is no programmatic limit, but there are practical limits. Very deeply nested structures are difficult to manage. 3 levels of categories should be sufficient for most situations. Note that there is always at least one level of categories, since the Course category always encompasses all other categories and grade items, can cannot be deleted.

=== I can't find setting X in the grade category edit page! Where is it? ===
If a setting documented on the [[Grade categories]] page does not appear on your edit page, it may mean that it is set globally in your site. See [[Grade_category_settings#Forcing_settings|Forcing settings]] for more information.

== Outcomes ==
=== I have just upgraded to Moodle 1.9, and I want to set up an outcome item for my course. What are the steps required? ===
#[[General_grade_settings#Enable_outcomes|Administration > Grades > General settings > Enable outcomes]]
#[[Scales#Creating_a_new_scale|Create a scale]]
#Create a course outcome (read the [[Outcomes| outcomes documentation]] for instructions). Assign to it the scale you just created.
#Assign the outcome to your course
#Enter the "Grades" section of your course, from the course administration block
#In the Actions menu (top left), select Edit -> Categories and Items
#Click "Add outcome item"
#Follow the instructions of the [[Outcome items|outcome items documentation]] to create the outcome item

You can now give your students a rating on the outcome dimension you just created. If you created a standard outcome, you will be able to use it in other courses and follow your students' performance across these courses.

== Modules ==
=== The activity module (Module name) doesn't support grading. How can I give my students a grade anyway? ===
You can create a [[Grade_items#Manual_grade_items|grade item]] manually in the gradebook. You will have to grade your students through the [[Grader report]] interface (in editing mode).

=== I just graded some of my students using the (Module name) interface, but the results aren't showing up in the grader report. What's going on? ===
Here are some of the possible reasons:

#The corresponding [[Grade items|grade item]] is [[Grade_locking#In_grade_items|locked]], or its parent [[Grade categories|category]] is [[Grade_locking#In_grade_categories|locked]].
#The module code is not using the [[Development:Grades#API_for_communication_with_modules.2Fblocks|gradebook API]] correctly

==See also==
*[[Gradebook 1.9 Tutorial]]
*Using Moodle [http://moodle.org/mod/forum/view.php?id=2122 Gradebook forum]

Using Moodle forum discussions:
*[http://moodle.org/mod/forum/discuss.php?d=102609 Can I aggregate only non hidden items?]

[[Category:FAQ]]

[[ca:PMF de les qualificacions]]
[[fr:FAQ des notes]]

Grade categories

2008-11-04T10:34:38Z

Nicolasconnault: Link to Grade_category_settings#Forcing_settings

{{Grades}}
[[Image:Edit grade category.png|thumb|Editing a grade category]]Grades can be organised into grade categories.
A grade category has its own aggregated grade which is calculated from its grade items. There is no limit to the level of nesting of categories (a category may belong to another category). However, each grade item may belong to only one category. Also, all grade items and categories belong to at least one, permanent category: [[Edit_categories_and_items#Top_category|the course category]].

==Adding a grade category==
To add a grade category:
# Select "Categories and items" from the gradebook dropdown menu.
# Click the "Add category" button near the bottom of the page.
# Give the grade category a meaningful name.
# Select grade category settings as appropriate. Advanced settings may be made available by clicking the "Show advanced" button.
# Click the "Save changes" button.

==Editing a grade category==
To edit a grade category:
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon opposite the grade category you wish to edit.
# After editing the grade category, click the "Save changes" button.

==Settings==
If any of the following do not appear in your page, it may mean that they are set globally in your site. See [[Grade_category_settings#Forcing_settings|Forcing settings]] for more information.

=== Aggregation ===
See [[Category aggregation]] for a detailed explanation.

=== Aggregate only non-empty grades ===
Non-existent grades are either treated as minimal grades or not included in the aggregation. For example, an assignment graded between 0 and 100 for which only half the students have been graded will either count the non-graded submissions as 0 (option switched off) or will ignore them (option switched on).

Important: An empty grade is simply a missing gradebook entry, and could mean different things. For example, it could be a participant who hasn't yet submitted an assignment, an assignment submission not yet graded by the teacher, or a grade that has been manually deleted by the gradebook administrator. Caution in interpreting these "empty grades" is thus advised.

=== Aggregate including sub-categories ===
The aggregation is usually done only with immediate children, it is also possible to aggregate grades in all subcategories excluding other aggregated grades.

=== Include outcomes in aggregation ===
Including outcomes in aggregation may not lead to the desired overall grade, so you have the option to include or leave them out.

=== Drop the lowest ===
If set, this option will drop the X lowest grades, X being the selected value for this option.

=== Keep the highest ===
If set, this option will only retain the X highest grades, X being the selected value for this option.

=== Aggregation view ===
Each category can be displayed in three ways: Full mode (aggregated column and grade item columns), the aggregated column only, or the grade items alone.

==See also==

*[[Grade items]]
*[[Edit categories and items]]
*[[Grade category settings]] - for administrators
*[http://www.youtube.com/watch?v=sUslTuZPu6A Video showing the effects of the grade category settings]
*Using Moodle [http://moodle.org/mod/forum/discuss.php?d=91632 Grade categories and weights 1.8 to 1.9?] forum discussion

[[ca:grade/edit/tree/category]]
[[fr:Catégories d'évaluation]]
[[cs:Kategorie známek]]

Grades FAQ

2008-11-04T10:30:25Z

Nicolasconnault: How can I setup weighted assignments?

{{Grades}}
== General ==

===How can I change how grades are displayed?===

Grades may be displayed as as actual grades, as percentages (in reference to the minimum and maximum grades) or as letters.

The default grade display type for the site is set by an administrator in ''Administration > Grades > [[Grade item settings]]''. However, this may be changed at course level.

To change how grades are displayed for particular [[Grade items|grade items]], or category and course summaries (called aggregations):

# Follow the grades link in the course administration block.
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon for the grade item, category total or course total.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button at the bottom of the page.

Alternatively, to change how grades are displayed for the whole course:

# Follow the grades link in the course administration block.
# Select "Course settings" from the gradebook dropdown menu.
# From the Grade display type menu, select real (for actual grades), percentage or letter.
# Click the "Save changes" button.

===How can I hide entered grades until a specified date?===

To set a "Hidden until" date:

#Access the course gradebook via the grades link in the course administration block.
#Select "Categories and items" from the gradebook dropdown menu.
#Click on the edit icon opposite the activity for which a "Hidden until" date is to be set.
#On the edit grade item page, ensure that advanced settings are displayed. (Click the "Show advanced" button if not.)
#Enable the "Hidden until" setting by unchecking the disable checkbox, then set a date.
#Click the "Save changes" button.

=== Is it possible to show the teachers/administrators' grades in the grader report? ===
Yes, at the site level you can define which roles will appear in the grader report. This can be found in [[General_grade_settings#Graded_Roles|Administration > Grades > General settings]]. Also read [http://moodle.org/mod/forum/discuss.php?d=92612 this discussion] for some more ideas.

===Why can't I change a grade within an assignment after changing it in the gradebook?===

When you edit a grade directly in the gradebook, an "overridden" flag is set, meaning that the grade can no longer be changed from within the assignment.

However, the flag can be removed by turning editing on in the [[Grader report|grader report]], then clicking the [[Grade editing|edit grade]] icon, unchecking the overridden box and saving the changes.

===How do I get groups to show up in the grader report?===

For groups to show up in the grader report, group mode should be set to visible or separate groups in the [[Course settings|course settings]]. This will result in a groups dropdown menu being displayed, enabling a teacher to view the grades of all participants, or only the grades for a selected group.

== Reports ==
=== How do I create my own custom gradebook reports? ===
Here is a [[Development:Gradebook_Report_Tutorial|tutorial]] explaining all the main steps involved.

== Aggregation ==
=== I can't find where to change the aggregation type for my gradebook categories! ===
Each category has an aggregation type, which can be changed through that category's "edit" page. To access that page, you must use one of 2 ways:

1. In the grader report, turn "Editing" on, then click the little "hand" icon next to the category whose aggregation you want to change
2. In the "Edit categories and Items" page (accessible through the "choose an action" menu, top left), you see a tree view of the categories and items in your gradebook. The top category is the course category. Each category also has a "hand" icon, which leads to the category edit page

=== How can I grade some of my activities without the results affecting my students' course total? ===
#Create two [[Grade categories]], one for your "activities still being graded," and one for your "released" activities.
#Ensure that "Aggregate including subcategories" (an advanced option) is unchecked for your top level course grade category.
##Where is this? In gradebook (grader report), in the upper right corner, click the "Turn Editing On" button.
##Click the edit icon next to the "course category" (usually your course name, just above the quiz names and below all the clickable links that were revealed when you turned editing on)
##Then make sure you have the "Show Advanced" option turned on.
#Edit the "activities still being graded" category's "course total" item. (This is one of the categories you created above.)
##Where is this? Look for the edit icon under "category total" that is below this category's name
#Set the "grade type" to "none".
#Tick the "Hidden" checkbox.
#Save your changes.
#Move all your activities being graded in the "activities still being graded" category.
#Move all your activities already graded in the "released" category.

Note: I rewrote this a bit, to help people find where things are. However, this method didn't seem to work for me on Moodle 1.9.

=== My student completed only one activity out of 5, but his course total shows 100%. How do I show a more "progressive" course total? ===
By default, only non-empty grades are aggregated, the others are ignored. However, you can change this setting as well as others that affect the course total, by turning "Editing" on in the grader report, and clicking the "Edit" icon next to the course category (the very top row of the grader report).

You can untick the box "Aggregate only non-empty grades" if you want to show a more "progressive" score for each student. Their empty grades will count as a 0 and will be counted in the course mean/total.

If you prefer to show a sum of points, rather than a percentage, you can change the course category's aggregation method to "Sum of grades".

=== How can I display the average grade for my course categories (not grade categories)? ===
In Moodle 1.9 there is no way to aggregate course totals within each category. The gradebook is course-centered, and there is currently no User Interface for showing grades within an entire course category at once.

=== How can I setup weighted assignments? ===
See [[Using "Weighted Mean of Grades" to weight categories containing assignments]].

== Categories ==
=== How many depths of categories/subcategories can I create? ===
There is no programmatic limit, but there are practical limits. Very deeply nested structures are difficult to manage. 3 levels of categories should be sufficient for most situations. Note that there is always at least one level of categories, since the Course category always encompasses all other categories and grade items, can cannot be deleted.

== Outcomes ==
=== I have just upgraded to Moodle 1.9, and I want to set up an outcome item for my course. What are the steps required? ===
#[[General_grade_settings#Enable_outcomes|Administration > Grades > General settings > Enable outcomes]]
#[[Scales#Creating_a_new_scale|Create a scale]]
#Create a course outcome (read the [[Outcomes| outcomes documentation]] for instructions). Assign to it the scale you just created.
#Assign the outcome to your course
#Enter the "Grades" section of your course, from the course administration block
#In the Actions menu (top left), select Edit -> Categories and Items
#Click "Add outcome item"
#Follow the instructions of the [[Outcome items|outcome items documentation]] to create the outcome item

You can now give your students a rating on the outcome dimension you just created. If you created a standard outcome, you will be able to use it in other courses and follow your students' performance across these courses.

== Modules ==
=== The activity module (Module name) doesn't support grading. How can I give my students a grade anyway? ===
You can create a [[Grade_items#Manual_grade_items|grade item]] manually in the gradebook. You will have to grade your students through the [[Grader report]] interface (in editing mode).

=== I just graded some of my students using the (Module name) interface, but the results aren't showing up in the grader report. What's going on? ===
Here are some of the possible reasons:

#The corresponding [[Grade items|grade item]] is [[Grade_locking#In_grade_items|locked]], or its parent [[Grade categories|category]] is [[Grade_locking#In_grade_categories|locked]].
#The module code is not using the [[Development:Grades#API_for_communication_with_modules.2Fblocks|gradebook API]] correctly

==See also==
*[[Gradebook 1.9 Tutorial]]
*Using Moodle [http://moodle.org/mod/forum/view.php?id=2122 Gradebook forum]

Using Moodle forum discussions:
*[http://moodle.org/mod/forum/discuss.php?d=102609 Can I aggregate only non hidden items?]

[[Category:FAQ]]

[[ca:PMF de les qualificacions]]
[[fr:FAQ des notes]]

Using "Weighted Mean of Grades" to weight categories containing assignments

2008-11-04T10:27:14Z

Nicolasconnault:

{{Grades}}
The first feature of the new grade book that some may want to be able to master is the ability to 'clump' assignments into categories and then have each category be assigned a different weight in the overall final grade. (Note - The non-technical term 'clump' is being used to avoid the term aggregate in the hopes of preventing confusion given the terminology used in the new grade book)

Example (how some courses could be evaluated using categories)

Attendance & Participation 40%,
Quizzes 20%,
Tests (Mid-term & Final) 20%,
Final Projects 20%,

First, create the categories to be used. This can be done after assignments have been created but we found it easier to create the categories first. For this short doc two categories were utilized (Reading and Writing).

In order to be able to weight categories, select "Weighted Mean of Grades" as the Aggregation method. You can also select the weight the category will be given here. In this case Reading is being weighted at 10% and Writing at 90%. These figures can be adjusted after creating the categories by selecting Categories and Items from the pulldown window from within the Grader.

Here are the details of the two categories created.

[[Image:Reading_10Percent.jpg]]
[[Image:Writing_90Percent.jpg]]

For this short doc four Assignments (offline activities) were created. Assign each a category at the bottom of the page when creating the assignment. (they can be assigned later as well from the grade book but since the categories have already been created it is just as easy to assign the category when creating the assignment)

[[Image:Giving_Assignment_A_Category.jpg]]

(Note - We noticed some irregular behavior with the ID Number when leaving it blank and letting Moodle generate it automatically so we decided it is better to assign an ID number here. 1234 was used for simplicity in this example.)

So after creating the 4 assignments this is what it should look like (roughly) from the front page of your course.

[[Image:Four_assignments.jpg]]

After grading the assignments you should be able to view the grade book and see that the categories are being weighted as desired in the overall Aggregation Course Total.

[[Image:WeightedMeanGradeReport2.jpg]]

Explanation for the student at the top of the list - Reading is 10% thus the 100% score in the reading category results in a 10% contribution to the overall Aggregation Course Total and Writing is set at 90% thus the 10% score results in a 9% contribution to the overall Aggregation Course Total.

Grade categories

2008-11-04T10:17:28Z

Nicolasconnault: /* Settings */

{{Grades}}
[[Image:Edit grade category.png|thumb|Editing a grade category]]Grades can be organised into grade categories.
A grade category has its own aggregated grade which is calculated from its grade items. There is no limit to the level of nesting of categories (a category may belong to another category). However, each grade item may belong to only one category. Also, all grade items and categories belong to at least one, permanent category: [[Edit_categories_and_items#Top_category|the course category]].

==Adding a grade category==
To add a grade category:
# Select "Categories and items" from the gradebook dropdown menu.
# Click the "Add category" button near the bottom of the page.
# Give the grade category a meaningful name.
# Select grade category settings as appropriate. Advanced settings may be made available by clicking the "Show advanced" button.
# Click the "Save changes" button.

==Editing a grade category==
To edit a grade category:
# Select "Categories and items" from the gradebook dropdown menu.
# Click the edit icon opposite the grade category you wish to edit.
# After editing the grade category, click the "Save changes" button.

==Settings==
=== Aggregation ===
See [[Category aggregation]] for a detailed explanation.

=== Aggregate only non-empty grades ===
Non-existent grades are either treated as minimal grades or not included in the aggregation. For example, an assignment graded between 0 and 100 for which only half the students have been graded will either count the non-graded submissions as 0 (option switched off) or will ignore them (option switched on).

Important: An empty grade is simply a missing gradebook entry, and could mean different things. For example, it could be a participant who hasn't yet submitted an assignment, an assignment submission not yet graded by the teacher, or a grade that has been manually deleted by the gradebook administrator. Caution in interpreting these "empty grades" is thus advised.

=== Aggregate including sub-categories ===
The aggregation is usually done only with immediate children, it is also possible to aggregate grades in all subcategories excluding other aggregated grades.

=== Include outcomes in aggregation ===
Including outcomes in aggregation may not lead to the desired overall grade, so you have the option to include or leave them out.

=== Drop the lowest ===
If set, this option will drop the X lowest grades, X being the selected value for this option.

=== Keep the highest ===
If set, this option will only retain the X highest grades, X being the selected value for this option.

=== Aggregation view ===
Each category can be displayed in three ways: Full mode (aggregated column and grade item columns), the aggregated column only, or the grade items alone.

==See also==

*[[Grade items]]
*[[Edit categories and items]]
*[[Grade category settings]] - for administrators
*[http://www.youtube.com/watch?v=sUslTuZPu6A Video showing the effects of the grade category settings]
*Using Moodle [http://moodle.org/mod/forum/discuss.php?d=91632 Grade categories and weights 1.8 to 1.9?] forum discussion

[[ca:grade/edit/tree/category]]
[[fr:Catégories d'évaluation]]
[[cs:Kategorie známek]]

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T09:25:39Z

Nicolasconnault:

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T09:05:44Z

Nicolasconnault:

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T09:05:03Z

Nicolasconnault: common frustrations

Development:Gradebook interface improvements for Moodle 2.0

2008-11-04T08:17:33Z

Nicolasconnault: Introduction

Grades

2008-10-13T05:49:55Z

Nicolasconnault: Reverted edits by Koltyn (Talk); changed back to last version by Tsala

{{Grades}}<p class="note">'''Note:''' This page, together with the pages listed in the block on the right, describe the gradebook in Moodle 1.9 onwards. For documentation on the gradebook in Moodle prior to 1.9, see [[Grades pre-1.9]].</p>

==Introduction==

The concepts of ''grades'' and of ''gradebook'' have been completely revisited in Moodle 1.9. Although these words are used in earlier versions, important differences are documented here in order to avoid misconceptions.

The two central ideas of grading in Moodle 1.9 are:

#'''Grades''' are scores attributed to participants in a Moodle course
#The '''gradebook''' is a repository of these grades: modules push their grades to it, but the gradebook doesn't push anything back to the modules

The three building blocks of the Gradebook in Moodle 1.9 are

*The [[Grade_categories|grade category]]
*The [[Grade_items|grade item]]
*The grade (see above)

As an overview:
*A grade category groups grade items together, and has settings for affecting these grade items
*A grade item stores a grade for each course participant, and has settings for affecting these grades
*A grade has settings for affecting how it is displayed to the users, as well as [[Grade locking|locking]] and [[Grade hiding|hiding]] functions.

Grades can be [[Grade_calculations|calculated]], [[Grade_categories#Aggregation|aggregated]] and [[Grader_report#Display|displayed]] in a variety of ways, the many settings having been designed to suit the needs of a great variety of organisations.

Many activities in Moodle, such as [[Assignment module|assignments]], [[Forum module|forums]] and [[Quiz module|quizzes]] may be given grades. Grades may have numerical values, or words/phrases from a [[Scales|scale or rating system]].

Grades can also be used as [[Outcomes|outcomes]] and as arbitrary text attributed to each participant in a course.

==Grades pushed by modules==
When activity modules produce grades, they use the [[Development:Grades#API_for_communication_with_modules.2Fblocks|gradebook public API]] to push (or send) their grades to the gradebook. These grades are then stored in database tables that are independent of the modules. The grades are still kept in the module database tables, and the gradebook will never access or modify these original grades.

The gradebook, however, provides administrators and teachers with tools for changing the ways in which grades are calculated, aggregated and displayed, as well as [[grade/edit/tree/grade|means to change the grades manually]] (a manual edit of a grade automatically locks the grade in the gradebook, so that the module which originally created the grade can no longer update that grade in the gradebook until the grade is unlocked).

==Settings affecting grades==
Being the smallest unit in the gradebook, the grade is affected by many settings at different levels. Here is a list of these levels, in hierarchical order:

*[[General_grade_settings|Site-wide general settings]]
*[[Grade_category_settings|Site-wide grade category settings]]
*[[Grade_item_settings|Site-wide grade item settings]]
*[[Gradebook_report_settings|Gradebook report settings]]
*[[Gradebook_course_settings|Course settings]]
*[[Grade_categories|Category settings]]
*[[Grade_items|Grade item settings]]
*[[grade/edit/tree/grade|Grade settings]]

==Outcomes==

[[Outcomes]] are specific descriptions of what a student is expected to be able to do or understand at the completion of an activity or course. An activity might have more than one outcome, and each may have a grade against it (usually on a [[Scales|scale]]).

==Gradebook reports==

The gradebook includes a variety of reports, available via the grades link in each [[Course administration block|course administration block]]:

* [[Grader report]] - The main teacher view of a course gradebook. The "[[Grade preferences|My report preferences]]" tab in the grader report enables teachers to change how the grader report is displayed.
* [[Outcomes report]]
* [[Overview report]]
* [[User report]]

==Grades organisation==

Teachers may organise grades into [[Grade categories|grade categories]], [[Grade import|import]] and/or [[Grade export|export]] grades, and make [[Grade calculations|grade calculations]].

Symbols to represent ranges of grades may be set as [[Grade letters|grade letters]].

Administrators may control the appearance of the gradebook site-wide by adjusting settings available via the grades link in the site administration block:

*[[General grade settings]]
*[[Grade category settings]]
*[[Grade item settings]]
*[[Gradebook report settings]]

==See also==

*Using Moodle [http://moodle.org/mod/forum/view.php?id=2122 Gradebook forum]

Video tutorials:
*[http://www.youtube.com/watch?v=YeUy-_kbvqQ Basic Moodle Gradebook howto]
*[http://www.youtube.com/watch?v=5hrLNbifiGQ Gradebook reports]
*[http://www.youtube.com/watch?v=lXEefYe3qdk How to use the grade item settings and grade letters at admin level]
*[http://www.youtube.com/watch?v=sUslTuZPu6A Grade category settings]
*[http://www.youtube.com/watch?v=EB58W3KePBc How to set up the gradebook]
*[http://www.youtube.com/watch?v=PmkEGfvjj9U How to use outcomes in Moodle]
*[http://www.youtube.com/watch?v=yZcbN_7p2zI How to export grades from the gradebook]
*[http://www.youtube.com/watch?v=p6zWwJGb9TA How to use gradebook site settings and defaults]
*[http://www.youtube.com/watch?v=WKUGyzAXcyA How to set up calculations in the gradebook (basic)]
*[http://www.youtube.com/watch?v=VBEj8mmu8lM How to set up calculations in the gradebook (advanced)]
*[http://www.youtube.com/watch?v=jWPUEqdhI4A How to change the display of grades in the gradebook]

[[ca:Qualificacions]]
[[cs:Známky]]
[[eu:Kalifikazioak]]
[[fr:Notes]]

Development:Calculated question type

2008-10-10T14:25:52Z

Nicolasconnault: Flagging this article as Obsolete design

{{obsolete_design}}
{{Questiontype developer docs}}
{{Calculated question dev docs}}
The person who knows most about this question type is probably [http://moodle.org/user/view.php?id=112682&course=5 Pierre Pichet].

==Database tables==

===quiz_calculated===
The '''quiz_calculated''' table is an extension to the quiz_questions table by the calculated questiontype. However, it would be more suitable to change that to be an extension of the quiz_answers table, which, from a data perspective, is already possible, since an answer id is stored in the answer field. The questiontype code would need some changes to take this into account, however.

;id :int(10) unsigned NOT NULL auto_increment,
;question :int(10) unsigned NOT NULL default '0',
;answer :int(10) unsigned NOT NULL default '0',
;tolerance :varchar(20) NOT NULL default '0.0',
;tolerancetype :int(10) NOT NULL default '1',
;correctanswerlength :int(10) NOT NULL default '2',
;correctanswerformat :int(10) NOT NULL default '2',

===quiz_numerical_units===
The calculated question type shares the [[Numerical_question_developer_docs#quiz_numerical_units|quiz_numerical_units table]] with the [[Numerical_question_developer_docs|numerical question type]].

==Response storage==
The calculated questiontype inherits its methods create_session_and_responses(), restore_session_and_responses() and save_session_and_responses() from the abstract dataset dependent questiontype. This questiontype serializes the information of the chosen dataset together with the (single) response value into a string of the format datasetXX-RESPONSE, where dataset is the actual string "dataset", XX is the number of the chosen dataset item and RESPONSE is the value that was received from the submitted form.

==Question->options==

==State->options==
==See also==
*[[Calculated question creation developper docs]]
*[[Calculated question development|Developer notes on calculated question development]]

[[Category:Developer|Calculated question type]]
[[Category:Quiz]]

Student projects/Secure RSS feeds

2008-10-10T14:23:40Z

Nicolasconnault:

{{obsolete_design}}

<p class="note">'''Note''': This page outlines ideas for the "Secure RSS feeds" project. If you have any comments or suggestions, please add them to the [[Talk:Student projects/Secure RSS Feeds|page comments]].''</p>

{{Moodle 1.9}}
== Status ==

Release candidate. All main objectives complete.

==Summary==
Secure RSS feeds is a project about making the RSS feeds published by Moodle secure so that only desired people can access the feeds. More details here.[http://code.google.com/soc/2008/moodle/appinfo.html?csaid=3141B6C0C1823EA1]

Typical RSS URL will look like: “<nowiki>http://domain/moodle/rss/file.php/contextid/</nowiki><u>[[#hashkey|hash_key]]</u>user_id/modulename/instance/any/other/params/module/wants/rss.xml”.

Where [[#hashkey|hash_key]] – special hash-string used to identify user.

User is identified by comparing part [[#hashkey|hash_key]] with the real hash value of user_id + user_private_key(from DB) + modulename + instance(from URL) concatenation.

If someone stole one private feed URL, he won’t be able to use it for reading other private feeds.

==Security==

# <div id="hashkey"> Hash-key is a hash value from user_id, user_private_key, modulename (and other information, which is used to identify RSS feed) concatenation. </div>
# If hash-key is not specified, consider user as guest.
# In current version of spec, hashes are not additionally salted.
# User private keys are tied to context id's.
# There is an option to force https:// for all RSS feeds

==Core functions==

===rss_auth()===
'''rss_auth($hash_key, $user_id, $course_id, $context_id, $module, $instance, $info )'''

* ''$hash_key'' - long hash-like string from URL.
* ''$user_id'' - user id from URL
* ''$course_id'' - the id of the course this feeds belongs to
* ''$context_id'' - the id of the context this feeds belongs to
* ''$module'' - module name or course module object this feeds belongs to
* ''$instance'' - instance id. Could be blogid, forumid etc
* ''$info'' - additonal information, which is used to accurately identify RSS feed. Can be array.

Authenticates user by hash-string in URL, sets up $USER and other necessary stuff(done by calling Moodle core function require_user_key_login()). Checks if the user can access particular course and module.
Function terminates with error if user doesn't have access to course\module.

===rss_get_url_key()===
'''rss_get_url_key( $userid, $contextid, $modulename, $instance, $info)'''

* ''$user'' - user id.
* ''$contextid'' - the id of the context this feeds belongs to
* ''$modulename'' - module name this feeds belongs to
* ''$instance'' - instance id. Could be blogid, forumid etc
* ''$info'' - additonal information, which is used to accurately identify RSS feed. Can be array.

Function returns long hash-like string, which can be used later to access specific RSS feed. Used when printing links.

===RSS feed generation===

'''modulename_rss_newstuff($instance, $time,&$cache, $info)'''
This function checks if there is something new in module since $time

'''modulename_rss_generate_feed($instance, $context, $info, $cache)'''
This function generates and returns XML rss contents

==Changes in RSS feed subsystem==
* No more Cron jobs for RSS feeds.
* All feeds are generated on the fly (i.e. no cached .xml files)

Most of the times nothing changes in the feed - we do not have to send the actual feed content, we can just send HTTP 304 Not Modified header. And because no actual content is sent, this allows us to skip loading all capabilities, identifying users etc - improve performance.

It may be convenient to prefetch some data in rss_newstuff(), that's why $cache is used.
However, it duplicates rcache functionality a bit, so I'm thinking about removing it.

But there is a problem, when rss_new_stuff() result depends on what capabilities user has.
In this situation during newstuff() check we assume that user has all the necessary capabilites. If there are no changes since last feed fetching - send 304 Not Modified. Otherwise, do the real check during feed content generation.

==Database tables==
Fields added to existing tables.

===course===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|rsstype
|int(1)
|0
|0 - disabled. 1 - recent activity rss
|-
|rssarticles
|int(2)
|0
|number of recent articles in RSS feed
|}

===assigment===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|rsstype
|int(1)
|0
|0 - disabled. 1 - assignment submissions rss
|-
|rssarticles
|int(2)
|0
|number of recent articles in RSS feed
|}

==Interface mockups==

===RSS links on Course page===
[[Image:blocks.gif]]

===Calendar RSS links===
[[Image:calendar_rss.gif]]

[[Image:calblockrss.gif]]

===Recent activity RSS feed preferences page===

[[Image:activityrsspref.gif]]

==Tasks and Timeline==

* Further develop spec, get feedback, feel out implementation ✔
* Implement core functions - 1-2w ✔
* Secure existing RSS feeds in Moodle 1w ✔
*# Forums ✔
*# Blogs ✔
*# Database module ✔
*# Glossary ✔
* Add option to force HTTPS for RSS feeds ✔
* Add RSS to other areas of Moodle.
*# Calendar(Upcoming events) 1-2w ✔
*# Recent Activity 1-2w ✔
*# Assigments submitted 1w ✔
*# Messaging 1w ✔
* Upgrade whole RSS subsystem. 1-3w
*# Each module should have own function, that checks if there are any changes. ✔
*# Use ETag and If-Modified-Since headers. ✔
*# Generate RSS content on the fly(no cache files, no rss cron jobs) ✔
*# ContextId ✔
*# file.php (stub code) ✔
* Optional tasks - 1.5w
*# Give user an ability to reset his private keys ✔
*# Recent activity feed for "My courses" ✔
* Extensive debugging - 1w
* End-term evaluation

== Glossary ==

{| border="1" cellpadding="2" cellspacing="0"
|'''Term'''
|'''Definition'''
|-
| Hash value (also called a "digest" or a "checksum")
| A concise representation of the longer message or document from which it was computed. The message digest is a sort of "digital fingerprint" of the larger document.
|-
| RSS feed
| A family of Web feed formats used to publish all kind of frequently updated content, usually blog entries, news headlines, and podcasts. RSS proved to be very convenient and easy-to-use, fast–to-implement technology, which makes users more productive and saves a lot of time.
|-
| user_private_key
| unique hash-like string used for user identification. Stored in database.

|}

== See also ==

*[[GSOC/2008]]
* [[Student projects]]
*[http://moodle.org/mod/forum/discuss.php?d=96026 Project discussion thread]
*[http://tracker.moodle.org/browse/MDL-15122 Tracker issue related to project]

[[Category:Project]]
[[Category:Developer|Feeds]]
[[Category:Feeds]]

Student projects/Secure RSS feeds

2008-10-10T14:23:20Z

Nicolasconnault: Flagging this article as Obsolete design

{{obsolete_design}}
{{Moodle 1.9}}
<p class="note">'''Note''': This page outlines ideas for the "Secure RSS feeds" project. If you have any comments or suggestions, please add them to the [[Talk:Student projects/Secure RSS Feeds|page comments]].''</p>

== Status ==

Release candidate. All main objectives complete.

==Summary==
Secure RSS feeds is a project about making the RSS feeds published by Moodle secure so that only desired people can access the feeds. More details here.[http://code.google.com/soc/2008/moodle/appinfo.html?csaid=3141B6C0C1823EA1]

Typical RSS URL will look like: “<nowiki>http://domain/moodle/rss/file.php/contextid/</nowiki><u>[[#hashkey|hash_key]]</u>user_id/modulename/instance/any/other/params/module/wants/rss.xml”.

Where [[#hashkey|hash_key]] – special hash-string used to identify user.

User is identified by comparing part [[#hashkey|hash_key]] with the real hash value of user_id + user_private_key(from DB) + modulename + instance(from URL) concatenation.

If someone stole one private feed URL, he won’t be able to use it for reading other private feeds.

==Security==

# <div id="hashkey"> Hash-key is a hash value from user_id, user_private_key, modulename (and other information, which is used to identify RSS feed) concatenation. </div>
# If hash-key is not specified, consider user as guest.
# In current version of spec, hashes are not additionally salted.
# User private keys are tied to context id's.
# There is an option to force https:// for all RSS feeds

==Core functions==

===rss_auth()===
'''rss_auth($hash_key, $user_id, $course_id, $context_id, $module, $instance, $info )'''

* ''$hash_key'' - long hash-like string from URL.
* ''$user_id'' - user id from URL
* ''$course_id'' - the id of the course this feeds belongs to
* ''$context_id'' - the id of the context this feeds belongs to
* ''$module'' - module name or course module object this feeds belongs to
* ''$instance'' - instance id. Could be blogid, forumid etc
* ''$info'' - additonal information, which is used to accurately identify RSS feed. Can be array.

Authenticates user by hash-string in URL, sets up $USER and other necessary stuff(done by calling Moodle core function require_user_key_login()). Checks if the user can access particular course and module.
Function terminates with error if user doesn't have access to course\module.

===rss_get_url_key()===
'''rss_get_url_key( $userid, $contextid, $modulename, $instance, $info)'''

* ''$user'' - user id.
* ''$contextid'' - the id of the context this feeds belongs to
* ''$modulename'' - module name this feeds belongs to
* ''$instance'' - instance id. Could be blogid, forumid etc
* ''$info'' - additonal information, which is used to accurately identify RSS feed. Can be array.

Function returns long hash-like string, which can be used later to access specific RSS feed. Used when printing links.

===RSS feed generation===

'''modulename_rss_newstuff($instance, $time,&$cache, $info)'''
This function checks if there is something new in module since $time

'''modulename_rss_generate_feed($instance, $context, $info, $cache)'''
This function generates and returns XML rss contents

==Changes in RSS feed subsystem==
* No more Cron jobs for RSS feeds.
* All feeds are generated on the fly (i.e. no cached .xml files)

Most of the times nothing changes in the feed - we do not have to send the actual feed content, we can just send HTTP 304 Not Modified header. And because no actual content is sent, this allows us to skip loading all capabilities, identifying users etc - improve performance.

It may be convenient to prefetch some data in rss_newstuff(), that's why $cache is used.
However, it duplicates rcache functionality a bit, so I'm thinking about removing it.

But there is a problem, when rss_new_stuff() result depends on what capabilities user has.
In this situation during newstuff() check we assume that user has all the necessary capabilites. If there are no changes since last feed fetching - send 304 Not Modified. Otherwise, do the real check during feed content generation.

==Database tables==
Fields added to existing tables.

===course===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|rsstype
|int(1)
|0
|0 - disabled. 1 - recent activity rss
|-
|rssarticles
|int(2)
|0
|number of recent articles in RSS feed
|}

===assigment===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|rsstype
|int(1)
|0
|0 - disabled. 1 - assignment submissions rss
|-
|rssarticles
|int(2)
|0
|number of recent articles in RSS feed
|}

==Interface mockups==

===RSS links on Course page===
[[Image:blocks.gif]]

===Calendar RSS links===
[[Image:calendar_rss.gif]]

[[Image:calblockrss.gif]]

===Recent activity RSS feed preferences page===

[[Image:activityrsspref.gif]]

==Tasks and Timeline==

* Further develop spec, get feedback, feel out implementation ✔
* Implement core functions - 1-2w ✔
* Secure existing RSS feeds in Moodle 1w ✔
*# Forums ✔
*# Blogs ✔
*# Database module ✔
*# Glossary ✔
* Add option to force HTTPS for RSS feeds ✔
* Add RSS to other areas of Moodle.
*# Calendar(Upcoming events) 1-2w ✔
*# Recent Activity 1-2w ✔
*# Assigments submitted 1w ✔
*# Messaging 1w ✔
* Upgrade whole RSS subsystem. 1-3w
*# Each module should have own function, that checks if there are any changes. ✔
*# Use ETag and If-Modified-Since headers. ✔
*# Generate RSS content on the fly(no cache files, no rss cron jobs) ✔
*# ContextId ✔
*# file.php (stub code) ✔
* Optional tasks - 1.5w
*# Give user an ability to reset his private keys ✔
*# Recent activity feed for "My courses" ✔
* Extensive debugging - 1w
* End-term evaluation

== Glossary ==

{| border="1" cellpadding="2" cellspacing="0"
|'''Term'''
|'''Definition'''
|-
| Hash value (also called a "digest" or a "checksum")
| A concise representation of the longer message or document from which it was computed. The message digest is a sort of "digital fingerprint" of the larger document.
|-
| RSS feed
| A family of Web feed formats used to publish all kind of frequently updated content, usually blog entries, news headlines, and podcasts. RSS proved to be very convenient and easy-to-use, fast–to-implement technology, which makes users more productive and saves a lot of time.
|-
| user_private_key
| unique hash-like string used for user identification. Stored in database.

|}

== See also ==

*[[GSOC/2008]]
* [[Student projects]]
*[http://moodle.org/mod/forum/discuss.php?d=96026 Project discussion thread]
*[http://tracker.moodle.org/browse/MDL-15122 Tracker issue related to project]

[[Category:Project]]
[[Category:Developer|Feeds]]
[[Category:Feeds]]

Development:Events

2008-10-10T14:22:17Z

Nicolasconnault: Flagging this article as Obsolete design

{{obsolete_design}}
{{Moodle 1.9}}
The Events API is a new core system in Moodle to allow better communication between modules. It's based on modules triggering new events with attached data, and the other modules handling those events with custom functions.

==Overview==

We'll be using the example of a grade being posted from a module into the [[Development:Grades|new gradebook in Moodle 1.9]], but there are obviously all kinds of events possible.

===Triggering an event===

Whenever a grade is created or changed by a module, it should “tell” the system about it (in addition to any local working storage it uses). So, using the quiz as an example, we first define an object as follows:

$eventdata = new object();
$eventdata->itemid = $grade_item->id;
$eventdata->userid = $USER->id;
$eventdata->gradevalue = $currentvalue;

Then we post the object as an event and forget about it:

events_trigger('grade_updated', $eventdata);

===Handling an event===

Modules can define an events.php in their db directory which defines events they want to be notified about, and describes which of their functions or class methods should be notified. For example, an export plugin could register something like:

$handlers = array (
'grade_updated' => array (
'handlerfile' => '/grade/export/banner/lib.php',
'handlerfunction' => 'banner_handle_grade_test', // argument to call_user_func(), could be an array
'schedule' => 'cron'
)
);
These are parsed during install / upgrade and stored in a simple database table.

Then, when a grade_updated event happens, all the registered functions for that event will be called something like this (but with more error handling):

include_once($CFG->dirroot.$handlers['grade_updated']['handlerfile']);
call_user_func($handlers['grade_updated']['handlerfunction'], $eventdata);

All plugins in Moodle have access to this and can this easily “hook in” to 'grade_updated' events (and of course any other events).

==Database structure==

There are 3 core tables for events. Note that if a handler is queued, and yet to be processed or processing failed, then all subsequent calls on that handler must be queued.

===events_handlers===

This table is for storing which components requests what type of event, and the location of the responsible handlers. For example, the grade book can register 'grade_added' event with a function add_grade() that should be called any time an 'grade_added' event is triggered by a module.

These entries are created by parsing events.php files in all the modules, and can be rebuilt any time (during an upgrade, say).

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Info'''
|-
|id
|int(10)
|auto increment identifier
|-
|eventname
|varchar(255)
|name of the event, e.g. 'grade_updated'
|-
|handlermodule
|varchar(255)
|e.g. moodle, mod/forum, block/rss_client
|-
|handlerfile
|varchar(255)
|path to the file of the function, eg /grade/export/lib.php
|-
|handlerfunction
|text
|serialized string or array describing function, suitable to be passed to '''call_user_func()'''
|-
|schedule
|varchar(255)
|'cron' or 'instant'.
|-
|status
|int(10)
|number of failed attempts to process this handler
|}

===events_queue===

This table is for storing queued events. It stores only one copy of the eventdata here, and entries from this table are being references by the events_queue_handlers table.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Info'''
|-
|id
|int(10)
|auto increment identifier
|-
|eventdata
|longtext
|serialized version of the data object passed to the event handler.
|-
|stackdump
|text
|serialized debug_backtrace showing where the event was fired from
|-
|userid
|int(10)
|$USER->id when the event was fired
|-
|timecreated
|int(10)
|time stamp of the first time this was added
|}

===events_queue_handlers===

This is the list of queued handlers for processing. The event object is retrieved from the events_queue table. When no further reference is made to the events_queue table, the corresponding entry in the events_queue table should be deleted. Entry should get deleted (?) after a successful event processing by the specified handler. The status field keeps track of failures, after it gets to a certain number (eg 10?) it should trigger an "event failed" event (that could result in admin being emailed etc, or perhaps even the originating module taking care of it or rolling something back etc).

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Info'''
|-
|id
|int(10)
|auto increment identifier
|-
|queuedeventid
|int(10)
|foreign key id corresponding to the id of the event_queues table
|-
|handlerid
|int(10)
|foreign key id corresponding to the id of the event_handlers table
|-
|status
|int(10)
|number of failed attempts to process this handler
|-
|errormessage
|text
|if an error happened last time we tried to process this event, record it here.
|-
|timemodified
|int(10)
|time stamp of the last attempt to run this from the queue
|}

==Standards for naming events==

All event names should follow a consistent naming pattern, such as modulename_noun_verb

==Events which exist==

===Users===
* user_updated
* password_changed

===Courses===
* course_updated
* course_deleted
* category_updated
* category_deleted

===Groups===
* group_deleted
* grouping_deleted
* group_user_added
* group_user_removed

==Events wishlist==

List of events which it would be nice to have. Please add to this list if what you want is not shown here.

* user_created (for example to handle custom emails. this is commonly desired: e.g. send a custom email to a related person (teacher, boss, etc.) based on some institution-specific logic)
* user_enrolled_in_course
* user_unenrolled_from_course
* course_created

== See also ==

* [http://moodle.org/mod/forum/discuss.php?d=69103 General Developer Forum thread for discussing this proposal].
* [[:Development:Grades]]

[[Category:Developer|Events]]
[[Category:Grades]]

Template:obsolete design

2008-10-10T14:18:13Z

Nicolasconnault:

<div class="notice metadata" id="stub" style="clear:both;"><p class="note">''This article is a design document which is <strong>no longer in use</strong>, the code it describes having been written and added to the Moodle codebase, or simply considered then abandoned. The information it contains is likely to be <strong>out of date</strong>, especially API and Database Schema specifications.''</p></div>
<includeonly>[[Category:Obsolete_Design]]</includeonly>
<noinclude>This template will categorize articles that include it into [[:Category:Obsolete_Design]].
</noinclude>

Development:Obsolete - Moodle forms library

2008-10-10T14:17:34Z

Nicolasconnault:

==OBSOLETE INFORMATION==
{{obsolete_design}}
'''This was a proposal for discussion that was not used - you can see info about the development of the formslib [[Development:lib/formslib.php|here]]'''.

The OU has developed quite a nice in-house library to simplify creating Moodle editing forms. We would like to contribute it back to the community, and then as part of the accessibility work we are comissioning, get all Moodle forms modified to use it. However, before we can contribute it back, we need to get consensus withing the community that it is right for Moodle, and we need to tidy up the code a bit.

This document outlines how the library should work after cleaning up, so the community can see whether they like it, and OU developers know what is involved in the cleanup.

==What the system will be capable of==

The library allows developers to create editing forms by creating a high-level representation of the form as a data-structure in memory, configuring all necessary options, and setting initial values of fields, and then the library generates the actual HTML of the form to include in the page.

We are not planning to change significantly the way that Moodle editing forms look and function. However, by putting all the HTML (and JavaScript, CSS, ...) generation in one place, we make it much easier to make systematic improvements to accessibility, usability, client-side validation, etc. in future. Indeed, the OU's code already generates HTML that is a bit more accessible that most Moodle forms.

By allowing developers to think at a higher level, we make their life easier, in the same way that datalib saves them from worrying about the details of SQL most of the time.

The general way the library will work is that it will assume sensible defaults for everything to reduce the amount that has to be typed. So for a lot of input fields, you will only have to specify

# the field type (e.g. text, think <input type="" ... />, However the hope would be to move towards higher-level types, such as integer, path, url, as in require_param, etc.)
# the field name (think <input name="" ... />).

Everything else that is needed will be derived from these using sensible defaults. We need label text? We'll look up the field name in the lang file (each form will specify which lang file to use). We need a help link? Well, use /lang/en_utf8/help$langfile/$name.html. We need a field size? Each type will have a sensible default.

However, if you want to override any of these defaults, you can by setting that option explicitly. Because there will be lots of options, and normally you will only need to change a few, the library will use a named attribute style, so example code would look like:

<pre>
$field = new text_field('fieldname');
$field->set_set('size', 30);
$field->set_set('label', 'lablestring');

// or

$field = new text_field('fieldname');
$field->set_set(array('size' => 30, 'label' => 'lablestring'));

// or

$field = new text_field('fieldname', array(
'size' => 30,
'label' => 'lablestring'
));
</pre>

In this example, 'lablestring' would again get looked up in the lang file automatically.

For this situation, where there are lots of options available, but most people only need to change a few, I think this style of API works better than PHP function calls with lots of optional parameters. The options available for each field type will be documented in the comment where that class is defined.

The library is designed to make it easy to add new field types, or new options to existing field types. Indeed, that is how the library has evolved so far: a bit at a time as needed.

New field types are just new PHP classes, and they are likely to be subclasses of a base class that only change a few things. They don't even have to be defined in the central form library. If you need an specialist field type only within one (e.g. contrib) module, you can create a new field-type subclass in that module code and use it in that modules forms without having to touch core code. And if we later want to move that field type into core, it is trivial to move the class definition.

Since we will have to tidy up the code anyway, all the function/class/method/option names in the API are potentially changable.

==PHP API==

Each thing in the API will be a PHP class, these basically fall into three categories: the whole form, groupings of form fields, and individual field types. Note that the groupings are currently only used for functional reasons, like showing or hiding groups of elements. Logical groupings, that would correspond to <fieldset> tags for accessibility, are not included yet, but could be added.

I am just going to list all the possible method calls. I will assume that it is clear what they do from their names (and I am running out of time).

===The whole form===

There is a class to represent a form, and most of the time you will be able to do everything you want to do by calling methods on this class, and you don't have to worry about the rest of the API.

$mf = new moodle_form($langfile);

$field = &$mf->add($fieldtype, $fieldname, $fieldoptions, $insertbefore);
$field = &$mf->add_item($fieldobject, $insertbefore); // $insertbefore is optional. By default things are inserted at the end of the form.
$field = &$mf->remove($fieldname);
$field = &$mf->get($fieldname);

These let you add or remove things from the form. The field you have just done stuff to is returned, but most of the time you will not bother to do anything with the return value. These method work with groups, etc. not just fields.

$mf->set_action($url);
$mf->set_init_html_editor(); // There was a reason why the library can't guess this from the list of fields.
$mf->set_submit_caption($langstringid);
$mf->set_hidden($fieldname, $value);
$mf->set_value($fieldname, $value); // The initial value for this field. (or use the $form attribute on show()).
$mf->set_error($fieldname, $message); // Error message to be displayed in red next to this field if you are doing server-side validation.

$mf->show($action, $form);

Show the form. $form is an object holding the default value of each field that has not already been set using set_value().

$mf->add_xhtml($htmlphpcode, $insertbefore);
$mf->set_xhtml_param($name, $value);

You can insert arbitrary bits of HTML or PHP code into the form using this. When outputting the form, the library does '''eval('?>'.$htmlphpcode.'<?php ');''' in a place where $form is in scope, and the variables passed in via set_xhtml_param are available from an array as $xmlform[$name].

$new_mf = moodle_form::loadfile($xmlfilename);
$new_mf = moodle_form::loadstring($xmlstring);

Create a whole form using the XML syntax mentioned below.

===Fields===

The library also has classes representing the different field types, which you can use if you want.

$field = new xxxx_field($name, $options_array); // where xxxx is one of the field types below.
$field->set($optionname, $optionvalue);
$field->set($optionarray); // Array of $optionname => $optionvalue pairs.
$optionvalue = $field->get($optionname);

$field->set_value($value); // Sets the initial value of this field.

$field->output($form); // Print this form element.

Normally you won't call output() directly. You will call $mf->output, which will call $field->output on each field for you, but you can use this yourself if you just want to print one form control anywhere you like, not as part of a form.

===Groups===

Groups (which will probably get renamed to conditionalshow, but I don't want to change all the examples just now) allow you to show or hide a collection of fields, depending on the setting of another field.

groups have basically the same methods as fields, but $name is optional. $name has no function, unless you need to identify the group later for a call the $mf->get($groupname);

There are a few extra methods they have over fields:

$field = &$group->add($fieldtype, $fieldname, $fieldoptions, $insertbefore);
$field = &$group->add_item($fieldobject, $insertbefore);
$field = &$group->remove($fieldname);
$field = &$group->get($fieldname);

$group->set_condition($conditionfield, $conditionvalue);

The elements within the group will only be visible when the field $conditionfield elsewhere in the form has value $conditionvalue. This works best when $conditionfield is a dropdown.

Do we also want a conditionalenable group that enables/disables a bunch of fields, rather than showing or hiding them?

===Multiples===

This lets you do things like enter multiple usernames, as in the 'working example' below. You would also use it on the editing form for multiple choice questions, to enter any number of answers with matching grades.

Again, they support the same methods as fields, and also:

$field = &$multiple->add($fieldtype, $fieldname, $fieldoptions, $insertbefore);
$field = &$multiple->add_item($fieldobject, $insertbefore);
$field = &$multiple->remove($fieldname);
$field = &$multiple->get($fieldname);

$multiple->set_max($number);

For accessibility reasons, multiples work by printing a certain number of copies into the HTML, and these are then shown or hidden by the JavaScript. This is for accessibility reasons, and so the forms work without JavaScript. max is the number of copies that are printed. It defaults to 10.

==Field types initially supported==

All field will support the options:

'''lable''' the field lable. This is a string that is looked up in the language file. Or, if the string starts with an '=' character, then this is trimmed off, and the rest of the string is used literally. Defaults to name.

'''help''' the name of the help file to link to. Defaults to name. Setting to '' removes the help icon.

'''helpalt''' the langstring to use for the tooltip of the help icon.

'''optional''' if set to 'yes', then adds a checkbox before the field to enable or diable it. The checkbox's name is name is constructed by taking the field name and appending 'enable'. If you want another name, set this option to the checkbox name, instead of 'yes'.

'''needcapability''' this is mainly aimed at 1.7 roles and permissions. Will hide this field unless the user has the named capability. At first, this will only recognise the values admin, teacher, student, noneditingteacher, guest, and translate these into the obvious isadmin() type calls.

More options will probably get added in future.

===display===

Just diplay a value with a label, the value can't be edited.

===text===

Text input box.

Supports the additional options:

'''required''' a regexp that is used for client-side validation against that regexp.

'''size''' as in <input size="" ... />

===file===

File upload box.

===date===

Date, like quiz open and close dates.

===html===

The standard HTML editor, or just a text area, depending on the settings. (This calls print_textarea).

===dropdown===

A dropdown menu. This field has the additional methods:

$dropdown->add_option($value, $label);
$dropdown->remove_option($value);

$label is optional. By default: if $value is an integer, use that integer as the label, otherwise look $value up in the langfile.

Dropdown supports the additional option:

'''default''' which option to select by default.

===radio===

A set of linked radio buttons, which are defined in the same way as dropdown menu options.

===multiselect===

A list box where you can select multiple options, which are defined in the same way as dropdown menu options.

Actually, I think we should change the implementation of this to use a set of checkboxes when there are fewer than about a dozen options, and automatically switch to useing a list box when there are more than that. And maybe add an optional parameter to force the listbox/checkbox decision one way or the other.

===yesno===

A dropdown menu with just the two options 'yes' and 'no'.

===user===

A flashy text box where you can enter user's names, and it does AJAX stuff to help you auto-complete them.

===visible===

This generates the standard "Visible to students" field that appears on add/update module forms.

===groupmode===

This generates the standard "Group mode" field that appears on add/update module forms.

==XML form definition format==

This provides the quickest way to create most forms. It should be clear how this translates into the PHP API calls defined above. XML elements $mf->add() calls. XML attributes correspond either to required information, or to set_opt calls. Form filds, like dropdowns, that need extra information get it from child elements. See the examples below for what it looks like.

==A working example==

Here is an example of a form produced with the current (un-cleaned-up) version of the OU's library. The form looks like this:

[[Image:Formproposal_Add_newsfeed_form.png]]

There is some OU-specific stuff here, like presentation and authids, and the fact that we reveal $user->username to teachers. I am assuming you can filter that out.

More interestingly, notice that

* The required field Name has not been filled it, so its label is red, and the create button is disabled. Below, you will see that anything typed into this field will actually be validated against a regular expression.
* The user field type does cool AJAX to save you typing the whole name.
* After you have added one user as a poster, a second box appeared where we could type a second user name. And when we finish typing here, a third box will appear.

The XML defining the form looks like this:

<pre>
<editform langfile='block_newsfeed'>
<text name='location'/>
<text name='name' required='\S'/>
<text name='pres' required='^([0-9]{2}[a-zA-Z]?)?$'/>
<html name='summary'/>
<dropdown name='type'>
<option value='internal'>type_internal</option>
<option value='external'>type_external</option>
</dropdown >
<group requiredname="type" requiredvalue="external">
<item type='text' name='url'/>
</group>
<group requiredname="type" requiredvalue="internal">
<date name='startdate'/>
<dropdown name='public'>
<option value='1'>access_public</option>
<option value='0'>access_private</option>
</dropdown >
<text name='defaultauthid'/>
<multiple>
<text name='optionalauthids' required='^([A-Z0-9]+)?$'/>
</multiple>
<multiple>
<user name='posters'/>
</multiple>
<multiple>
<user name='approvers'/>
</multiple>
</group>
</editform>
</pre>

The PHP code that creates the form definition ($xf), and the object with all the current values ($form) and sets all the default values looks like this:

<pre>
$xf=xml_form::load_file('editfeed.xml');
$xf->set_init_html_editor(true);
$form=new stdClass;
$newsfeedid=optional_param('newsfeedid',0,PARAM_INT);
if($newsfeedid) {
$nf=feed_system::$inst->get_feed($newsfeedid);
$form->location=$nf->get_folder()->get_path();
$xf->set_hidden('newsfeedid',$newsfeedid);

$form->name=$nf->get_name();
$form->pres=$nf->get_pres();
if($form->pres==null) {
$form->pres='';
}
$form->summary=$nf->get_summary();
if(is_a($nf,'external_news_feed')) {
$form->type='external';
$form->url=$form->get_url();
} else {
$form->type='internal';
$form->startdate=$nf->get_start_date();
$form->public=$nf->is_public();
$form->defaultauthid=$nf->get_default_authid();
xml_form::set_multiple($form,'optionalauthids',$nf->get_optional_authids());
xml_form::set_multiple($form,'posters',$nf->get_poster_usernames());
xml_form::set_multiple($form,'approvers',$nf->get_approver_usernames());
$form->newsfeedid=$newsfeedid;
}

} else {
$folderid=required_param('folderid',PARAM_INT);
$lf=feed_system::$inst->get_location_folder($folderid,null);
$nf=null;
$xf->set_submit_caption(get_string('create'));
$form->location=$lf->get_path();
$xf->set_hidden('newsfeedid',0);
}

// ... print_header and other boring bits.

print_simple_box_start('center');
$xf->show(basename(__FILE__), $form);
print_simple_box_end();
</pre>

==Examples of converting existing Moodle forms==

These are four example forms that Marting Dougiamas asked to see how we would handle.

===Add resource -> Link to a file or web site===

[[Image:Formproposal_Add link resource.png]]

I have taken the liberty of changing the way the window options work on this form. Instead of having a show/hide window settings button, and when that is turned on showing all the options for both same window and new window, I have changed it to be a same window/popup window dropdown, and depending on the setting there, showing or hiding the particular set of options. It would also be possible to use the library to generate the existing form.

This form contains some very specific bits, namely the location field and the parameters sections. For now I have kept the existing code to generate these bits. If fields like the location field were used in other places, we could add a url field type. I don't see any future in generalising the parameters bit, though the code to generate it could be cleaned up. I just copied and pasted the existing code, and made the minimal changes.

I've used a multiselect for the window options. That will require a small change to the response processing code.

<pre>
$mf = moodle_form::loadstring('
<editform langfile="resource">
<text name="name" help=""/>
<htmlarea name="summary"/>
<xhtml label="location"><![CDATA[
echo "<input type=\"text\" name=\"reference\" size=\"90\" value=\"$form->reference\" alt=\"reference\" /><br />";
button_to_popup_window ("/files/index.php?id=$form->course&choose=form.reference", "coursefiles", $xmlform['strchooseafile'], 500, 750, $xmlform['strchooseafile']);
echo "<input type=\"button\" name=\"searchbutton\" value=\"$xmlform['strsearch'] ...\" ".
"onclick=\"return window.open('$CFG->resource_websearch', 'websearch', 'menubar=1,location=1,directories=1,toolbar=1,scrollbars,resizable,width=800,height=600');\" />\n";
if ($CFG->resource_allowlocalfiles) {
button_to_popup_window ("/mod/resource/type/file/localfile.php?choose=form.reference",
"localfiles", get_string('localfilechoose', 'resource'), 400, 600,
get_string('localfilechoose', 'resource'));
}
]]></xhtml>
<url name="location" showuploadbutton="1" showsearchbutton="1" help=""/>
<dropdown name="windowpopup">
<option value='0'>pagewindow</option>
<option value='1'>newwindow</option>
</dropdown>
<group requiredname="newwindow" requiredvalue="0">
<hidden name="hframepage" value="0"/>
<multiselect name="framepage" help="">
<option value="1">frameifpossible</option>
</multiselect>
</group>
<group requiredname="newwindow" requiredvalue="1">
<multiselect name="windowoptions">
</multiselect>
<integer name="resource_popupwidth" label="resource_popupwidth"/>
<integer name="resource_popupheight" label="resource_popupheight"/>
</group>
<yesno name="parameters"/>
<group requiredname="newwindow" requiredvalue="1">
<xhtml><![CDATA[
<table align="center">

<tr>
<td align="center"><?php print_string("parameter", "resource") ?></td>
<td align="center"><?php print_string("variablename", "resource") ?></td>
</tr>

<?php

for ($i=0; $i < $xmlform['maxparameters']; $i++) {
echo "<tr>\n";
echo "<td valign=\"top\">\n";
echo "<select name=\"parameter$i\">\n";
echo "<option value=\"-\">-- ".get_string('chooseparameter', 'resource')." --</option>\n";
foreach ($xmlform['parameters'] as $field=>$fieldarr) {
if ($fieldarr['value'] === "optgroup") {
echo "<optgroup label=\"{$fieldarr['langstr']}\">\n";
} elseif ($fieldarr['value'] === "/optgroup") {
echo "</optgroup>\n";
} else {
echo "<option value=\"$field\"";
if ($xmlform['alltextfield'][$i]['parameter'] == $field) {
echo " selected=\"selected\"";
}
echo ">{$fieldarr['langstr']}</option>\n";
}
}
echo "</select>\n";
echo "</td>\n";
echo "<td valign=\"top\">\n";
echo "<input type=\"text\" name=\"parse$i\" value=\"{$xmlform['alltextfield'][$i]['parse']}\" alt=\"parameter$i\"/>\n";
echo "</td>\n";
echo "</tr>\n";
}

?>
</table>
]]></xhtml>
</group>
')
$mf->set_xhtml_param('strchooseafile', get_string(...));
$mf->set_xhtml_param('strsearch', get_string(...));

$mf->set_xhtml_param('maxparameters', $this->maxparameters);
$mf->set_xhtml_param('parameters', $this->parameters);
$mf->set_xhtml_param('alltextfield', $alltextfield); // Assuming that this code is after the end of setup() in mod\resource\type\file\resource.class.php

$winopt = $mf->get('windowoptions');
foreach ($RESOURCE_WINDOW_OPTIONS as $optionname) {
$defaultvalue = "resource_popup$optionname";
$form->$optionname = $CFG->$defaultvalue;
if ($optionname != 'height' && $optionname != 'width') {
$winopt->add_option($optionname, "str$optionname");
}
}
</pre>

===Add quiz===

[[Image:Formproposal_Add quiz form.png]]

To convert this form, we need to do something special for the "students may review" bit. You could either invent an new <optiongrid> type, which would be quite easy to implement. That is the option used below. Alternatively, you could use the feature for including arbitrary HTML and PHP in the form, and keep the existing code for generating this part of the form. Since the existing layout breaks when you make the browser window narrow, as in the screenshot, I would be inclined to redo it.

<pre>
$mf = moodle_form::loadstring('
<editform langfile="quiz" help="">
<text name="name"/>
<htmlarea name="introduction"/>
<date name="available" label="quizopen" optional="yes"/>
<date name="due" label="quizclose" optional="yes"/>
<real name="timelimit" lableafter="minutes" helpalt="quiztimer" optional="yes"/>
<dropdown name="questionsperpage">
<option value='0'>unlimited</option>

</dropdown>
<yesno name="shufflequestions"/>



<optiongrid name="reviewoptions">
<col name="responses"/>
<col name="scores"/>
<col name="feedback"/>
<col name="answers"/>
<row name="immediately" lable="reviewimmediately"/>
<row name="open" lable="reviewopen"/>
<row name="closed" lable="reviewclosed"/>
</optiongrid>



<groupmode/>
<visible/>
')
$questionsperpage =& $mf->get('questionsperpage');
for ($i = 1; $i <= 50; $i += 1) {
$questionsperpage->add_option($i);
}
</pre>

This does not take into account the settings on http://''example.com''/moodle/admin/module.php?module=quiz, which lets the admin move certain quiz options to be hidden behind an "advanced" option. To implement this you would need to add a show/hide advanced control (I would do this as a <yesno name="showadvanced"/>, and an empty advanced group, then use some PHP code like

<pre>
$fix = 0;
$advgroup = $mf->get_field('advanced');

if ($CFG->quiz_fix_timelimit) {
$item = &$mf->remove('timelimit');
$advgroup->add($item)
$fix = 1;
}
// ... and so on, for all the other options. Or you could try to be clever
// and do this as a loop over an array of field names ... then
if (!$fix) {
$form->remove('showadvanced');
$form->remove('advanced');
}
</pre>

===Add database===

[[Image:Formproposal_Add_database_form.png]]

I won't type out full code for this example most of it is simple, and it should be clear how to do it given the above examples, just comment on a couple of things:

'''Entries required before viewing''' I assume this is only indented because the label is so long that having it all lined up would break the table layout. I think our code using CSS for layout just word-wraps long labels and keeps everything aligned, which I think is better.

'''Allow posts to be rated?''' I would put the disabled controls in a group that only appears if the checkbox is checked, so that those controls show and hide, rather than enabling or disabling. However, if we want to keep this the same as it is now, you could implemnt the conditionalenable group type.

===Manage groups for a course===

This sort of form is currently beyond what the form library was designed to produce. I would leave this as hand-coded HTML. In time, the form library may gain some AJAX field types for selecting students, and such like, that could usefully be within a redesigned version of this form. Since the form field types just output HTML and Javascript, it should be possible to use them within hand-crafted forms.

[[Image:Formproposal_Groups_form.png]]

==Questions==

Do we need a syntax like lable='langfile:langstring' for using lang strings from other lang files where necessary? '''yes'''

Where does this live. I think the main library file that people have to include should be lib/formlib.php, and that is all anyone needs to include. However, to we break each field type into its own PHP file, perhaps in a lib/formlib directory, to make it easier to add new field types in future? '''Probably all in one file'''

Some form fields will need bits of CSS and JavaScript to work. Do we add the CSS to the standard theme, and combine all the javascript into a single library somewhere, or do we break it up into the individual field types, and recombine it dynamically at runtime? I think I favour breaking up the PHP code, but keeping all the JS and CSS in one place. '''Should use YUI were possible. We will still need some custom JavaScript and CSS though. Where?'''

Do we use this for vaidation as well? '''Yes.''' Currently Moodle PHP files that do forms tend to look like:

<pre>
$var1 = require/optional_param(...);
// etc. for all the other form variables.

if (data_submitted && confirm_sesskey()) {
// Try to process submitted data.
if (/*processing ok*/)
// Redirect away
} else {
// Set initial form values.
}

// Lots of code to output the form HTML. Of maybe include(somthing.html);
</pre>

The proposal is to change this to something like

<pre>
$mf = new moodle_form();
// Setup $mf with the form structure

$form = new stdClass;
if ($mf->validate_submitted_data($form)) { // Pass by reference to get data back.
// Try to process submitted data in $form
if (/*processing ok*/)
// Redirect away
} else {
// Set initial values in $form
}

$mf->display($form)
</pre>

[[Category:Developer|Obsolete - Moodle forms library]]

Category:Obsolete Design

2008-10-10T14:13:47Z

Nicolasconnault: New page: These articles were created to assist developers in the creation of new Moodle features. Once the code is written and added to CVS, these articles are less and less used and updated, so th...

These articles were created to assist developers in the creation of new Moodle features. Once the code is written and added to CVS, these articles are less and less used and updated, so their content should not be trusted, especially their API and DB schema specifications.

Development:Grades

2008-10-10T14:11:56Z

Nicolasconnault: Flagging this article as Obsolete design

[http://moodle.org/mod/forum/discuss.php?d=69223 There is an ongoing discussion about this spec here]
{{obsolete_design}}
== Executive Summary ==

The gradebook mechanisms must be rebuilt to:

# '''Improve performance and scalability''' - All grades from throughout the system will be pushed to a central system of tables. This means reports based on grades can be generated much faster, and the gradebook has ultimate control over the content.
# '''Improve flexibility''' - All aspects of the new gradebook will use simple plugin structures, namely: exports, imports and displays/reports. It is expected that the community will be very active in producing [[Development:Gradebook Report Tutorial |special-purpose reports]] analysing the basic grade data in new ways, for example, or writing plugins to transfer grades to student information systems.
# '''Allow rubrics for outcomes (aka standards,competencies,goals)''' - As well as numerical grades, each grading item can consist of a number of scores made on a rubric against a standard outcome statement. These can be automatically converted to a numerical grade if desired or just shown as is.
# '''Allow arbitrary columns and derived columns''' - Arbitrary columns of data can be added (either manually or via import). Columns can also be automatically filled based on formulas.
# '''Implement limited public API''' - Activities may use this API to send grades/outcomes to gradebook and find out the final grades.

== Glossary ==

Here are some terms used in the gradebook, both in the development and the user interface. Using these terms in discussions about the gradebook will help to reduce confusion.

{| border="1" cellpadding="2" cellspacing="0"
|'''Term'''
|'''Definition'''
|-
|Activity
|An instance of an activity module [[Category:Modules|Module]] (e.g. a single quiz, assignment etc...)
|-
|Calculation
|A formula used to calculate grades, based (optionally) on other grade items. Not the same as [[Calculated_question_type|Calculated question types]].
|-
|Category
|A set of Grade Items. A Category also has its own aggregated Grade which is calculated from its Grade Items. There is no limit to the level of nesting of Categories (a Category may belong to another Category). However, each Grade Item may belong to only one Category.
|-
|Course completion
|The concept of meeting certain criteria for completing a course. In the context of the gradebook, this means a set of grades that must be reached, or a number of outcomes/competencies to complete/master.
|-
|Grade
|A Grade is a single assessment. It may be a number or an item on a scale (possibly tied to an Outcome). Raw grade value is the numerical or scale grade from activity. Final grade is the grade reported in gradebook.
|-
|[[Gradebook|Gradebook]]
|A central location in Moodle where students' Grades are stored and displayed. Teachers can keep track of their students' progress and organise which set of Grades their students will be able to see. Students see their own Grades.
|-
|Grade Item
|A "column" of Grades. It can be created from a specific Activity or other module, calculated from other Grade Items, or entered manually.
|-
|[[Development:Grades#Locked_grades|Grade Locks]]
|See linked section of this page
|-
|History
|The gradebook has its own type of log, which keeps a History of all changes made to grades.
|-
|[[Development:Outcomes|Outcome]]
|[[Development:Outcomes|Outcomes]] are specific descriptions of what a person is expected to be able to do or understand at the completion of an activity or course. An activity might have more than one outcome, and each may have a grade against it (usually on a scale). Other terms for Outcomes are ''Competencies'' and ''Goals''. See some [[Development:Outcomes_examples|Examples]].
|-
|[[Scales|Scale]]
|A scale is a set of responses from which the teacher can choose one. eg Very cool, Cool, Fairly cool, Not very cool, Not cool
|-
|Letter Grades
|Special representation of grade values similar to scales. Letters are configured in course contexts or above and are defined by lower boundary. eg A (above 90 %), B (above 80 %), C (above 70 %), D (above 50 %), F (above 0 %)

|}

== Database structures ==
=== grade_items ===

This table keeps information about gradeable items (ie columns). If an activity (eg an assignment or quiz) has multiple grade_items associated with it (eg several outcomes and numerical grade), then there will be a corresponding multiple number of rows in this table.

idnumber is a tag unique inside a course identifying the grade item, useful for identifying data in exports and for referring to the grade item in calculations. It is the same as the idnumber in course_modules.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|'''id'''
|int(10)
|
|autoincrementing
|-
|'''courseid'''
|int(10)
|
|The course this item is part of
|-
|'''categoryid'''
|int(10)
|<center>NULL</center>
|the category group this item belongs to
|-
|itemname
|varchar(255)
|<center>NULL</center>
|The name of this item (pushed in by the module or entered by user)
|-
|'''itemtype'''
|varchar(30)
|
|'mod', 'blocks', 'manual', 'course', 'category' etc
|-
|itemmodule
|varchar(30)
|<center>NULL</center>
|'forum', 'quiz', 'csv', etc
|-
|iteminstance
|int(10)
|<center>NULL</center>
|id of the item module
|-
|itemnumber
|int(10)
|<center>NULL</center>
|Can be used to distinguish multiple grades for an activity
|-
|iteminfo
|text
|<center>NULL</center>
|Info and notes about this item XXX
|-
|'''idnumber'''
|varchar(255)
|<center>NULL</center>
|Arbitrary idnumber provided by the module responsible (optional and course unique)
|-
|calculation
|text
|<center>NULL</center>
|Spreadsheet-type formula used to process the raw grades into final grades
|-
|'''gradetype'''
|int(4)
|<center>1</center>
|0 = none, 1 = value, 2 = scale, 3 = text
|-
|grademax
|float(10,5)
|<center>100</center>
|What is the maximum allowable grade?
|-
|grademin
|float(10,5)
|<center>0</center>
|What is the minimum allowable grade?
|-
|'''scaleid'''
|int(10)
|<center>NULL</center>
|If this grade is based on a scale, which one is it?
|-
|'''outcomeid'''
|int(10)
|<center>NULL</center>
|If this is outcome item, which outcome is it?
|-
|gradepass
|float(10,5)
|<center>0</center>
|What grade is needed to pass? grademin <= gradepass <= grademax
|-
|multfactor
|float(10,5)
|<center>1.0</center>
|Multiply all raw grades from activities by this
|-
|plusfactor
|float(10,5)
|<center>0.0</center>
|Add this to all raw grades from activities by this
|-
|aggregationcoef
|float(10,5)
|<center>0.0</center>
|Weight applied to all grades in this grade item during aggregation with other grade items.
|-
|sortorder
|int(10)
|<center>0</center>
|Sorting order of the columns (pre-order walk of the grading tree)
|-
|display
|int(10)
|<center>0</center>
|Display as real grades, percentages (in reference to the minimum and maximum grades) or letters (A, B, C etc..), or course default (0)
|-
|hidden
|int(10)
|<center>0</center>
|1 is hidden, 1 is hide always, > 1 is a date to hide until (prevents viewing of all user grades)
|-
|'''locked'''
|int(10)
|<center>0</center>
|0 is not locked, > 0 is a date when was item locked (no final grade or grade_item updates possible)
|-
|'''locktime'''
|int(10)
|<center>0</center>
|0 no auto locking, > 0 is a date to lock grade item and final grades after automatically
|-
|deleted
|int(10)
|<center>0</center>
|1 means the associated module instance has been deleted
|-
|'''needsupdate'''
|int(10)
|<center>0</center>
|If this flag is set, then the whole column will be recalculated. If set in course item, some other item needs recalculation. Calculated and category items are recalculated together with any other items.
|-
|timecreated
|int(10)
|
|The first time this grade_item was created
|-
|timemodified
|int(10)
|
|The last time this grade_item was modified
|}

=== grade_categories ===

This table keeps information about categories, used for grouping items. An associated grade_item will be maintained for each category to store the aggregate data.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|'''id'''
|int(10)
|
|autoincrementing
|-
|'''courseid'''
|int(10)
|
|The course this grade category is part of
|-
|'''parent'''
|int(10)
|<center>NULL</center>
|Parent grade_category (hierarchical)
|-
|depth
|int(10)
|<center>0</center>
|How deep is this category from the highest level (1,2,3)
|-
|path
|varchar(255)
|
|Shows the path as /1/2/3/
|-
|fullname
|varchar(255)
|
|The name of this grade category
|-
|aggregation
|int(10)
|<center>0</center>
|A constant pointing to one of the predefined aggregation strategies (none, mean,median,sum, etc)
|-
|keephigh
|int(10)
|<center>0</center>
|Keep only the X highest items
|-
|droplow
|int(10)
|<center>0</center>
|Drop the X lowest items
|-
|aggregateonlygraded
|int(1)
|<center>0</center>
|Aggregate only existing grades
|-
|aggregateoutcomes
|int(1)
|<center>0</center>
|Aggregate otcomes together with normal items
|-
|aggregatesubcats
|int(1)
|<center>0</center>
|Aggregate only items placed directly in category or all items in subcategories excluding the subcategory totals
|-
|timecreated
|int(10)
|
|The first time this grade_category was created
|-
|timemodified
|int(10)
|
|The last time this grade_category was modified
|}

=== grade_grades ===

This table keeps individual grades for each user and each item. The raw grade is exactly as imported or submitted by modules. The rawgrademax/min and rawscaleid are stored here to record the values at the time the grade was stored, because teachers might change this for an activity! All the results are normalised/resampled/calculated for the finalgrade, which is relative to the max/min/scaleid values stored in the grade_item. The finalgrade field is effectively a cache and values are rebuilt whenever raw values or the grade_item changes.

Note that the finalgrade for a scale-based item may be non-integer! It needs to be rounded on display.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''
|-
|'''id'''
|int(10)
|
|autoincrementing
|-
|'''itemid'''
|int(10)
|
|The item this grade belongs to
|-
|'''userid'''
|int(10)
|
|The user who this grade is for
|-
|rawgrade
|float(11,10)
|<center>NULL</center>
|The raw grade that came into the system
|-
|rawgrademax
|float(11,10)
|<center>100</center>
|The maximum allowable grade when this was created
|-
|rawgrademin
|float(11,10)
|<center>0</center>
|The minimum allowable grade when this was created
|-
|'''rawscaleid'''
|int(10)
|<center>NULL</center>
|If this grade is based on a scale, which one was it?
|-
|'''usermodified'''
|int(10)
|<center>NULL</center>
|the userid of the person who last modified the raw grade value
|-
|finalgrade
|float(11,10)
|<center>NULL</center>
|The final grade (cached) after all calculations are made
|-
|hidden
|int(10)
|<center>0</center>
|0 is not hidden, 1 is hide always, > 1 is a date to hide until
|-
|'''locked'''
|int(10)
|<center>0</center>
|0 is not locked, > 0 when was the grade locked
|-
|'''locktime'''
|int(10)
|<center>0</center>
|0 is never, > 0 is a date to lock the final grade after automatically
|-
|exported
|int(10)
|<center>0</center>
|0 is not exported, > 0 is the last exported date
|-
|excluded
|int(10)
|<center>0</center>
|grade excluded from aggregation, > 0 is the last exported date
|-
|overridden
|int(10)
|<center>0</center>
|0 is not overridden, > 0 is the last overridden date
|-
|feedback
|text
|<center>NULL</center>
|Manual feedback from the teacher. Could be a code like 'mi'.
|-
|feedbackformat
|int(10)
|<center>0</center>
|Text format for feedback
|-
|information
|text
|<center>NULL</center>
|not sued yet (Further information like forum rating distribution 4/5/7/0/1 ?)
|-
|informationformat
|int(10)
|<center>0</center>
|Text format for information
|-
|timecreated
|int(10)
|
|temporary hack - the date of submission in activity if any, new field expected in 2.0
|-
|timemodified
|int(10)
|
|temporary hack - the date of grading in activity or date of manual grading in gradebook, new field expected in 2.0
|}

=== grade_outcomes ===

This table describes the outcomes used in the system. An outcome is a statement tied to a rubric scale from low to high, such as “Not met, Borderline, Met” (stored as 0,1 or 2). For more info about these see [[Development:Outcomes]].

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''courseid'''
|int(10)
|<center>NULL</center>
|Mostly these are defined site wide ie NULL

|-
|shortname
|varchar(255)
|
|The short name or code for this outcome statement

|-
|fullname
|text
|
|The full description of the outcome (usually 1 sentence)

|-
|'''scaleid'''
|int(10)
|
|The recommended scale for this outcome.
|-
|description
|text
|<center>NULL</center>
|The full description of the outcome (usually 1 sentence)
|-
|timecreated
|int(10)
|
|the time this outcome was first created

|-
|timemodified
|int(10)
|
|the time this outcome was last updated

|-
|'''usermodified'''
|int(10)
|<center>NULL</center>
|the userid of the person who last modified this outcome
|}

=== grade_outcomes_courses ===
An intersection table used to make standard outcomes available to courses.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''courseid'''
|int(10)
|
|The id of the course being assigned the outcome

|-
|'''outcomeid'''
|int(10)
|
|The id of the outcome being assigned to the course
|}

=== grade_import_newitem ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|itemname
|varchar(255)
|
|*TODO* Document

|-
|importcode
|int(12)
|
|*TODO* Document
|}

=== grade_import_values ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''itemid'''
|int(10)
|NULL
|*TODO* Document

|-
|newgradeitem
|int(10)
|NULL
|*TODO* Document

|-
|'''userid'''
|int(10)
|
|*TODO* Document

|-
|finalgrade
|float(10,5)
|NULL
|*TODO* Document

|-
|feedback
|text
|NULL
|*TODO* Document

|-
|importcode
|int(12)
|
|*TODO* Document
|}

=== History tables ===

These table keep track of changes to most of the grade tables. Using these it should be possible to reconstruct the grades at any point in time in the past, or to audit grade changes over time. It should be quicker to use these tables for that, rather than storing this information in the main Moodle log. The following tables are set up for that purpose:

#grade_categories_history
#grade_grades_history
#grade_items_history
#grade_outcomes_history

Each of them has exactly the same DB structure as their matching table (e.g. grade_categories), with 3 extra fields:

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|action
|int(10)
|0
|The action that lead to the change being recorded (insert, update, delete)

|-
|'''oldid'''
|int(10)
|
|The id of the record being changed or inserted (PK of the main table, not the history table)

|-
|source
|varchar(255)
|NULL
|The URL from which the action originated
|}

=== grade_letters ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|contextid
|int(10)
|
|What contextid does this letter apply to (from levels CONTEXT_SYSTEM, CONTEXT_COURSECAT or CONTEXT_COURSE)

|-
|lowerboundary
|float(10,5)
|
|The lower boundary of the letter. Its upper boundary is the lower boundary of the next highest letter, unless there is none above, in which case it's grademax for that grade_item.

|-
|letter
|varchar(255)
|
|The display value of the letter. Can be any character or string of characters (OK, A, 10% etc..)
|}

== Overview of module communication ==

Modules usually store raw grades internally and pass them into gradebook every time they change. Gradebook may also request activities to resend the grades.

The gradebook is designed to be as separate as possible from the code of activities - modules do not read grade tables or use internal gradebook API.

Originally it was planned to use new events API, but in the end it was decided to use minimal API consisting of several function in lib/gradelib.php and each mod/xxx/lib.php

=== Backward compatibility with Moodle 1.8 and earlier ===

Function grade_grab_legacy_grades($courseid) may be used to request transfer of grades from legacy or 3rd party activities which were not yet converted to new grade API. This function is not called automatically.

Modules are responsible to push existing grades into gradebook during upgrade.

==API for communication with modules/blocks==

Modules may use only functions from lib/gradelib.php which are marked as public. This API may be extended in later 1.9.x release. Activities should access/update only own grades.

===grade_get_grades()===

grade_get_grades($courseid, $itemtype, $itemmodule, $iteminstance, $userid_or_ids=0)

Returns grading information for given activity - optionally with users grades. Manual, course or category items can not be queried.

===grade_get_outcomes()===

grade_get_outcomes($courseid, $itemtype, $itemmodule, $iteminstance,$userid=0)

Returns list of outcomes used in course together with current outcomes for this user.

===grade_is_locked()===

eg grade_is_locked($courseid, $itemtype, $itemmodule, $iteminstance, $itemnumber, $userid=NULL)

This function will tell a module whether a grade (or grade_item if $userid is not given) is currently locked or not. If it's locked to the current user then the module can print a nice message or prevent editing in the module. If no $userid is given, the method will always return the grade_item's locked state. If a $userid is given, the method will first check the grade_item's locked state (the column). If it is locked, the method will return true no matter the locked state of the specific grade being checked. If unlocked, it will return the locked state of the specific grade.
([http://moodle.org/mod/forum/discuss.php?d=69223#p311329 info])

===grade_update()===

grade_update($source, $courseid, $itemtype, $itemmodule, $iteminstance, $itemnumber, $grades=NULL, $itemdetails=NULL)

Submit new or update grade; update/create grade_item definition. Grade must have userid specified, rawgrade and feedback with format are optional. rawgrade NULL means 'Not graded', missing property or key means do not change existing. Only following grade item properties can be changed 'itemname', 'idnumber', 'gradetype', 'grademax', 'grademin', 'scaleid', 'multfactor', 'plusfactor', 'deleted'.

===grade_update_outcomes()===

grade_update_outcomes($source, $courseid, $itemtype, $itemmodule, $iteminstance, $userid, $data)

Updates outcomes of a given user. Manual outcomes cannot be updated.

== Private gradebook API ==
Private API is used by gradebook plugins and core Moodle code, it may change in 2.0. Most of the interesting classes and functions are in lib/gradelib.php, grade/lib.php and grade/report/lib.php.

===grade_regrade_final_grades()===

grade_regrade_final_grades($courseid=NULL, $userid=NULL, $updated_item=NULL)

Updates all grade_grades->finalgrade records for each grade_item matching the given attributes. The search is further restricted, so that only grade_items that have needs_update == true or that use calculation are retrieved and used for the update. The function returns the number of grade_items updated (NOT the same as the number of grades_grades updated!).

===grade_verify_idnumber()===

grade_verify_idnumber($idnumber, $grade_item = null, $cm = null, $gradeitem)

Verify new value of idnumber - checks for uniqueness of new idnubmers, existing are kept intact.

===remove_course_grades()===

remove_course_grades($courseid, $showfeedback)

Remove all grade related course data - history is kept

TODO: add description of the other methods and classes + simple usage examples + querylib.php description

== Dealing with multiple grades ==

Modules usually produce only one grade item per activity. Optionally one or more outcomes may be attached to activities.

Some activities may need to aggregate multiple ratings or attempts before sending them into the gradebook. Activities can not send variable number of items.

If the gradebook receives multiple grade items from a module, then they are automatically grouped together in a unique grade category (with the same name as the module instance). See [[Development:Outcomes]] for more details.

TODO: this may still be changed

== Calculated grade items ==

Categories or manual items maybe calculated using spreadsheet-like formulas. Formulas may reference other items from the same course only using Id numbers in double square brackets.

eg: <nowiki>= MEAN([[quiz121]], [[quizend]]) + [[assignmentAXC]] + 20.0</nowiki>

== Regrading / updating of final grades ==

TODO: describe needsupdate flag and incremental updates

== Adjustment of raw grades ==
Grade_item contains optional rules for adjusting the raw grade before it is cached into a final grade. These rules are processed BEFORE the calculation discussed above. Scale is never changed. Multfactor and plusfactor may be used to alter raw grades coming from activities, but it is recommended to use formulas instead.

== Displaying the grades to ordinary participants ==

The module takes responsibility for displaying grades within the module (to a student, say). It is recommended to use the real final grades obtained using grade_get_grades() functionBecause guidebook might force hiding, override grade, etc.

For full display of grades in a whole course say, the student uses the same link as teachers use to access the gradebook. However, due to their different permissions they will only have access to specific reports. By default this is the ''User report'' report which only shows their own grades and has very few configuration options.

== Locked grades ==

Both whole columns and individual grades can be locked in the gradebook, via the ''locked'' field. Teachers may want to do this to prevent further changes from the modules, or from other teachers. When a grade is locked, any changes that might affect that grade are ignored. When the graded is unlocked, activities are asked to resend the latest grades.

In the main GUIs the lock toggling will be achieved by clicking on a little padlock icon beside each entry or column.

There is also an option to lock grade or item after some specified date.

== Overridden grades ==

Grades can be manually modified (overridden) in the gradebook. When this is done the entered value is always used instead of the aggregated, calculated or activity grade.

== Logging ==

All grading related changes maybe logged in history tables.

== Security Issues ==

For security an option to force SSL for the gradebook might be good.

==Overall grade==

Each course has exactly one course grade item. It may be used for this purpose now. Other course completion criteria will be implemented in 2.0.

== Report plugins ==
All the main interface of the gradebook are implemented as report plugins. Each plugin is fully responsible for page layout, there are some handy functions in grade/lib.php. They can even define their own capabilities and extra tables if the core tables are not enough, as they'll have a full /grade/report/xxxx/db directory.

Each report defines one capability to allow people to see that report, so that admins have control over who can see what reports. For example, the participant interface can be a totally separate report plugin.

This allows for the widest flexibility and safety in how grades are presented.

=== Default teacher interface ===
This interface will be what teachers see by default, and will subsume everything the current interface (in Moodle 1.8) does.

Some snippets of functionality:
{{Moodle 1.9}}
Overall it's a grid, with participant names down one side and grade items along the top.

Columns will be able to be collapsed together by grouping them into categories. Grades for categories can be calculated via various means.

“Eye-cons” on the columns and checkboxes by every grade (this bit possibly controlled with a switch) allow hiding by category, by column, by individual grade.

Textual notes can be added to each grade for more info. These show up to participants as well.

A groups menu allows the teacher to switch between showing EACH of the groups they have access to, or ALL the groups they have access to.

All grade items will link to modulepath/grade.php?id=44 which will work out what the current person should be allowed to see and either redirect them to the correct page or just show them immediately. This copes with situations like the quiz, say, where we want editing teachers to go to the detailed reports there while participants just see their own grade or whatever the quiz is set to show.

User preference to SWITCH between showing raw grades, percentage grades, or both, or grade letters (A/B/C etc).

Settings for grade letters not only define the transformation from percentage to grades, but also the transformation from letters to grades (in case the teacher edits some of the letter grades).

Categories are shown above the headings for each column. Clicking for more info on a category will just show the category with a summary column showing total/average for just that category (PLUS the summary column for the whole course).

All columns should be sortable up/down.

At the bottom of each column is a row with the mean course score. If in groups mode, then add ANOTHER row with just the group mean. Add the number of grades used in brackets. eg 56% (11). When the report is paged, these means are still for the whole course/group (not the page!)

{{Moodle 2.0}}

Teachers can type “straight into” the grid using AJAX or fallback to forms. No popup menus for values.

Later on we can support customisable shorthand codes to make data entry quick (eg type 'ab' for absent, or 'nge' for not good enough).

See [http://test.moodle.com/grade/report/grader/index.php?id=2 the test site] for a live demo of this report.

=== Default participant interface ===
This interface will be what participants see by default:

Some snippets of functionality:

*Invert the grid to show one item per row, with the total/average at the bottom.
*Use second/third columns to show categories.
*Include ranking score in another column.
*Show feedback
*Show percentage
*No editing functionality.

See [http://test.moodle.com/grade/report/user/index.php?id=2 the test site] for a live demo of this report.

=== Outcomes report ===
This simple informational report displays all the outcomes used by the course, with the following information:

*Outcome name
*Overall average: If the outcome is used by more than one activity, this shows you the mean across all these activities in the current course
*Site-wide: Yes or No: A site-wide outcome is automatically made available to all courses.
*Activities: A list of links to the activities in the current course that use each outcome. One row per activity (table splits here)
*Average: For each activity using the outcome, the average score is shown.
*Number of grades: For each activity using the outcome, the number of grades is shown (non-graded participants are ignored)

See [http://test.moodle.com/grade/report/outcomes/index.php?id=2 the test site] for a live demo of this report.

=== Overview report ===
Another basic report, showing a participant's course averages in each of the courses in which s/he has received grades.

See [http://test.moodle.com/grade/report/overview/index.php the test site] for a live demo of this report.

== Export plugins ==

The API for these is extremely simple. Each export plugin should occupy a directory under /grade/export/xyz and needs to provide only an index.php file as a the primary interface. This file just accepts a 'courseid' parameter.

== Import plugins ==

Each import plugin should occupy a directory under /grade/import/xyz and needs to provide only an index.php file as a the primary interface. This file just accepts a 'courseid' parameter.

The index.php will show an interface for further options and selections.

Import plugin must validate data before starting the import operation, if some parts of import fail the user must be notified.

Some sample import plugins are:

===Import from CSV===

Accepts an upload of (or URL to) a CSV file. Multiple options describe how to process the file, which columns to add etc. The imported grades always override current grades.

===Import from XML===

Accepts an upload of (or URL to) an XML file with this kind of format (from OU).

<results batch="[someuniqueimportnumber]">
<result>
<state>['new' or 'regrade']</state>
<assignment>[idnumber]</assignment>
<student>[studentid]</student>
<score>[score]</score>
</result>
<result>
<state>['new' or 'regrade']</state>
<assignment>[idnumber]</assignment>
<student>[studentid]</student>
<score>[score]</score>
</result>
[...]
</results>

== Capabilities and Permissions ==

* moodle/grade:view - view own grades or grades of other user if used in CONTEXT_USER

* moodle/grade:viewall - view grades of all users

* moodle/grade:viewhidden - see grades that are marked as hidden for the owner

* moodle/grade:hide - be able to hide/unhide cells, items or categories

* moodle/grade:lock - be able to lock cells, items or categories

* moodle/grade:unlock - be able to unlock cells, items or categories

* moodle/grade:manage - manage grade items and categories in gradebook (create, edit, lock, hide, delete, etc.)

* moodle/grade:import - general import grades, requires separate permission for each plugin

* moodle/grade:export - export grades, requires separate permission for each plugin

* gradereport/grader:view - can view the grader report

* gradeimport/csv:view - can view/use the csv import plugin

* gradeexport/csv:view - can view/use the csv export plugin

* moodle:site/accessallgroups

== Development Tasks and Tracking ==

See [http://tracker.moodle.org/browse/MDL-9137 MDL-9137] for the full details.

==Updating module code==
Module authors must implement new gradebook API and add upgrade code for migration of old grades into new gradebook. Fortunately the needed changes are not big.

Steps:
*add xxx_update_grades() function into mod/xxx/lib.php
*add xxx_grade_item_update() function into mod/xxx/lib.php
*patch xxx_update_instance(), xxx_add_instance() and xxx_delete_instance() to call xxx_grade_item_update()
*patch all places of code that change grade values to call xxx_update_grades()
*patch code that displays grades to students to use final grades from the gradebook

There are many examples in official modules, assignment has the most advanced implementation.

== Ideas for the future ==
{{Moodle 2.0}}
TODO: add new meta issue into tracker

*option to aggregate including/excluding hidden grades - needs db changes
*option to rollback all changes during import operation if anything fails
*performance improvements
*conditional activities
*course completion criteria
*better public API for modules
*better API for gradebook plugins
*better state tracking in export plugins
*ajax reports
*specialised reports
*submission and marking date tracking db changes
*calculation formula improvements
*historical views
*individual graph of grades (time vs %). Bar graph, lineal graph. Add (or not) the maximum posible; line of 0 (=minimum), 25, 50 (=median), 75 and 100 (=max) percentils of the group.
*

== See also ==

* [http://moodle.org/mod/forum/discuss.php?d=69223&mode=3 Gradebook Development ideas] forum discussion
* Using Moodle [http://moodle.org/mod/forum/discuss.php?d=51107 New gradebook for Moodle] forum discussion
* [[Development:Gradebook Report Tutorial]]

[[Category:Developer|Grades]]
[[Category:Grades]]

Template:obsolete design

2008-10-10T14:11:11Z

Nicolasconnault: Adding obsolete design article template

<div class="notice metadata" id="stub" style="clear:both;"><p class="note">''This article is a design document which is <strong>no longer in use</strong>, the code it describes having been written and added to the Moodle codebase. The information it contains is likely to be <strong>out of date</strong>, especially API and Database Schema specifications.''</p></div>
<includeonly>[[Category:Obsolete_Design]]</includeonly>
<noinclude>This template will categorize articles that include it into [[:Category:Obsolete_Design]].
</noinclude>

Category aggregation

2008-10-10T13:01:18Z

Nicolasconnault: Adding info about extra credits for sum of grades aggregation

{{Grades}}

== Overview ==
This menu lets you choose the aggregation strategy that will be used to calculate each participant's overall grade for a [[Grade categories|grade category]]. The different options are explained below.

The grades are first converted to percentage values (interval from 0 to 1), then aggregated using one of the strategies below and finally converted to the associated category item's range (between Minimum grade and Maximum grade).

Important: An empty grade is simply a missing gradebook entry, and could mean different things. For example, it could be a participant who hasn't yet submitted an assignment, an assignment submission not yet graded by the teacher, or a grade that has been manually deleted by the gradebook administrator. Caution in interpreting these "empty grades" is thus advised.

== Aggregation strategies ==

=== Mean of grades ===
The sum of all grades divided by the total number of grades.
A1 70/100, A2 20/80, A3 10/10, category max 100:
(0.7 + 0.25 + 1.0)/3 = 0.65 --> 65/100
=== Weighted mean ===
Each grade item can be given a weight, which is then used in the arithmetic mean aggregation to influence the importance of each item in the overall mean.
A1 70/100 weight 10, A2 20/80 weight 5, A3 10/10 weight 3, category max 100:
(0.7*10 + 0.25*5 + 1.0*3)/18 = 0.625 --> 62.5/100
=== Simple weighted mean ===
The difference from Weighted mean is that weight is calculated as Maximum grade - Minimum grade for each item. 100 point assignment has weight 100, 10 point assignment has weight 10.
A1 70/100, A2 20/80, A3 10/10, category max 100:
(0.7*100 + 0.25*80 + 1.0*10)/190 = 0.526 --> 52.6/100
=== Mean of grades (with extra credits) ===
Arithmetic mean with a twist. An old, now unsupported aggregation strategy provided here only for backward compatibility with old activities.
=== Median of grades ===
The middle grade (or the mean of the two middle grades) when grades are arranged in order of size. The advantage over the mean is that it is not affected by outliers (grades which are uncommonly far from the mean).
A1 70/100, A2 20/80, A3 10/10, category max 100:
0.7 + 0.25 + 1.0 --> 0.25 --> 25/100
=== Smallest grade ===
The result is the smallest grade after normalisation. It is usually used in combination with Aggregate only non-empty grades.
A1 70/100, A2 20/80, A3 10/10, category max 100:
min(0.7 + 0.25 + 1.0) = 0.25 --> 25/100
=== Highest grade ===
The result is the highest grade after normalisation.
A1 70/100, A2 20/80, A3 10/10, category max 100:
max(0.7 + 0.25 + 1.0) = 1.0 --> 100/100
=== Mode of grades ===
The mode is the grade that occurs the most frequently. It is more often used for non-numerical grades. The advantage over the mean is that it is not affected by outliers (grades which are uncommonly far from the mean). However it loses its meaning once there is more than one most frequently occurring grade (only one is kept), or when all the grades are different from each other.
A1 70/100, A2 35/50, A3 20/80, A4 10/10, A5 7/10 category max 100:
mode(0.7; 0.7; 0.25; 1.0; 0.7) = 0.7 --> 70/100
=== Sum of grades ===
The sum of all grade values. Scale grades are ignored. This is the only type that does not convert the grades to percentages internally. The Maximum grade of associated category item is calculated automatically as a sum of maximums from all aggregated items.
A1 70/100, A2 20/80, A3 10/10:
70 + 20 + 10 = 100/190

When the "Sum of grades" aggregation strategy is used, a grade item can act as Extra credit for the category. This means that the grade item's maximum grade will not be added to the category total's maximum grade, but the item's grade will. Following is an example:

* Item 1 is graded 0-100
* Item 2 is graded 0-75
* Item 1 has the "Act as extra credit" checkbox ticked, Item 2 doesn't.
* Both items belong to Category 1, which has "Sum of grades" as its aggregation strategy
* Category 1's total will be graded 0-75
* A student gets graded 20 on Item 1 and 70 on Item 2
* The student's total for Category 1 will be 75/75 (20+70 = 90 but Item 1 only acts as extra credit, so it brings the total to its maximum)

[[ca:Agregació de les categories]]
[[fr:Tendance centrale de la catégorie]]

Quality assurance

2008-09-22T07:41:21Z

Nicolasconnault: /* Ideas/Notes */

{{stub}}[[Category:Quality Assurance]]
=Introduction=
==What is QA==
Here few documents related to QA/Testing on Wikipedia: [http://en.wikipedia.org/wiki/Software_quality_assurance], [http://en.wikipedia.org/wiki/Software_testing]

==QA Objective==
Main objective: is a major version right for release?

A major version is right when (TBD):
* blocker bug number < 1%
* critical bug number < 5%
* major bug number < 20%
* test cases have been written/updated for new features
* all test cases are marked as ''Passed''

A successful QA cycle assures that we have tested all major Moodle functionalities.

==Participate to QA for your benefit==
The document described the Moodle QA. Any participation is welcome. Please use the tab "page comments" in order to participate. A good Moodle QA will be a great benefit for Moodle project but also for any entity using Moodle.

= QA cycle =
We use Jira as Test Case Management System, see the specific chapter at the bottom of this page.

== Setting a QA cycle ==
We will need to write into Jira the test cases the first time, then update them for any new QA cycle. In fact it would be good that people writing/updating the functional specs, update the test cases in the same time. It would not take so long for them. Another possibility: create a test case status "''Cannot be run''". So when a tester cannot run a test case because the feature changed, the QA manager is aware that this specific test case needs to be updated.

== Running a QA cycle ==
QA participants will choose into Jira the feature they want to test (status set to "''Not run''" or "''to retest''").
* If the tester finds a blocker/critical/major bug, he sets the test case to "''Failed''". Then he writes a bug issue and links the test case to the bug issue. "''Failed''" test cases will be set to "''to retest''" by the bug fixer.
* If the bug is minor and the feature is working, the tester still writes a bug issue and still links the test case to the bug issue. However he sets the test case to "''Passed''".

== Validate a QA cycle ==
The Test Case management system (Jira) needs to display:
* how many test cases are "''failed''","''passed''","''not run''","''to retest''" for a specific version/component. (test case issues)
* how many critical/major bugs for a specific version/component (bug issues)
<br>From these results the QA manager valid the version for releasing.

=Draft/Ideas/Opinion/Requirements/Notes=
==Need to be identified==
QA Cost (in persons-hours) for:
* writting test cases (the first time)
* preparing a QA cycle (update test cases, get a new QA system ready for testing)
* running all test cases

When should a QA cycle be run?<br>
What function specs/Use Cases have we got?<br>
Who can write test cases?<br>
Who can run test cases?<br>
Do we need test data?

==Ideas/Notes==
* We would need to make a clear statement in the Moodle docs in order to make a difference between: User Manuals, Function Specifications, Technical Specifications, Requirements, ... and many existing mixed document. (The moodle docs are a powerful source of information, however at the current time Google is your best friend for finding information.)
* There is a lack of detailed functional specifications and Use Cases => at this moment only developers and experimented users would be able to write test cases with missing use cases.
* I need to study Quality Assurance into open source world: how do the other projects manage their QA department?
* Moodle doesn't have deadline. It is released when it will be ready. However take care about no-end QA phase.
* there are many experimented users who can test Moodle.
* some people/companies using Moodle probably have already written test plans, test cases, maybe even use cases.
* need documentation on how to write a test case and how to run test cases. Need also document for QA manager (setting a QA cycle, read the result, managing the QA cycle)
* need to link Moodledocs QA documents with Moodle docs tracker documents, organise an easy-to-read Quality Engineering section in Moodledocs (include Code Testing as well).
* we could create a testing program as Netbeans [http://qa.netbeans.org/processes/cat/65/index.html]
* create a Moodle package including test data for people testing in local
* I don't think we need smoke tests as we don't really build anything in PHP.

==Jira as Test Case Management System==
The main reason to use Jira as Test Case Management System is that the Moodle community is familiar with. It will also be easy to link test case issues to bug issues.
* The search tool of Jira will provide the QA results.
* we will create a main issue containing thousand of sub-tasks. These subtasks will be the test cases. Then for every new QA cycle we will clone this main issue. Need to identify how to change subtask version quickly/easily.
* re-write the ''Jira as a Test Case Management Software documentation'' [https://docs.moodle.org/en/Jira_as_a_Test_Case_Management_Software]

==Other QA from popular open-source projects==
* Netbeans wrote specification tests [http://wiki.netbeans.org/TestSpecifications]. They have a QA program [http://qa.netbeans.org/processes/cat/65/index.html]
* Firefox use an integrated testcase management and QA tool called Litmus [https://litmus.mozilla.org/]. An example of a test case: [https://litmus.mozilla.org/show_test.cgi?id=5036]. They've got more than 7000 test cases.
* OpenOffice has a QA page [http://qa.openoffice.org/ooQAReloaded/ooQA-ManualTesting.html]. I found it a bit a hassle to navigate into these pages. Make some succinct documentation for Moodle QA tester in order to start quickly and easily.

Development:Admin settings

2008-09-18T12:04:50Z

Nicolasconnault: Fixing typos

Moodle's configuration is stored in a mixture of the config, config_plugins, and a few other tables. These settings are edited through the administration screens, which can be accessed by going to the .../admin/index.php URL on your moodle site, or using the Administration block that appears to administrators of the Moodle front page. This page explains how the code for displaying and editing of these settings works.

==Where to find the code==

This is explained further below, but in summary:
* The library code is all in lib/adminlib.php.
* The definition of the all the parts of the admin tree is in admin/settings/* some of which call out to plugins to see if they have settings they want to add.
* The editing and saving of settings is managed by admin/settings.php, admin/upgradesettings.php, and admin/search.php.
* The administration blocks that appear on the front page and on most of the admin screens is in blocks/admin_tree and blocks/admin_bookmarks.

In my experience most of this code is pretty easy to understand and work with, so here I will focus on giving an overview. For details, see the code.

==The building blocks==

All the settings are arranged into a tree structure. This tree structure is represented in memory as a tree of PHP objects.

At the root of the tree is an '''admin_root''' object.

That has children that are '''admin_category'''s.

Admin categories contain other categories, '''admin_settingpage'''s, and '''admin_externalpage'''s.

Settings pages contain individual '''admin_setting'''s.

admin_setting is a base class with lots of subclasses like '''admin_setting_configtext''', '''admin_setting_configcheckbox''', and so on. If you need to, you can create new subclasses.

External pages are for things that do not fit into the normal settings structure. For example the global assign roles page, or the page for managing activity modules.

==How the tree is built==

When Moodle needs the admin tree, it calls admin_get_root in lib/adminlib.php, which
# creates a global $ADMIN object which is an instance of admin_root.
# does require_once admin/settings/top.php, which adds the top level categories to $ADMIN.
# does require_once on all the other files in admin/settings to add more specific settings pages and the settings themselves. Some of these settings files additionally make calls out to various types of plugins. For example
#* admin/settings/plugins.php gives activity modules, blocks, question types, ... a chance to add admin settings.
# adds the admin reports to the tree.

As an optimisation, before building each bit of the tree, some capability checks are performed, and bits of the tree are skipped if the current user does not have permission to access them.

Exactly how different types of plugin should add their settings to the tree should be documented in the [[Development:Developer_documentation#Make_a_new_plugin|instructions for writing that sort of plugin]].

==Individual settings==

Let us look at a simple example: [http://cvs.moodle.org/moodle/mod/forum/settings.php?view=markup mod/forum/settings.php]. This is included by admin/settings/plugins.php, which has already created $settings, which is an admin_settingpage that we can add to. The file contains lots of lines that look a bit like:

$settings->add(new admin_setting_configcheckbox('forum_replytouser', get_string('replytouser', 'forum'),
get_string('configreplytouser', 'forum'), 1));

What this means is that to our settings page, we are adding a checkbox setting. To understand this in more detail, we need to know what arguments the constructor for an admin_setting takes. The definition of the constructor is:

function admin_setting($name, $visiblename, $description, $defaultsetting) {

So, $name here is 'forum_replytouser'. This means that this setting is stored in the database in the row of the config table where name='forum_replytouser', and is accessible as $CFG->forum_replytouser.

$visiblename is get_string('replytouser', 'forum'), this is the label that is put in front of setting on the admin screen.

$description is get_string('configreplytouser', 'forum'), this is a short bit of text displayed underneath the setting to explain it further.

$defaultsetting is the default value for this setting. This value is used when Moodle is installed. For simple settings like checkboxes and text fields, this is a simple value. For some more complicated settings, this is an array.

Let as now look at a more complicated example, from mod/quiz/settingstree.php:

$quizsettings->add(new admin_setting_text_with_advanced('quiz/timelimit',
get_string('timelimit', 'quiz'), get_string('configtimelimit', 'quiz'),
array('value' => '0', 'fix' => false), PARAM_INT));

This shows two new things:

$name here is 'quiz/timelimit'. This is more complicated than a simple setting name. It means that this setting is stored in the config_plugins table, in the row where plugin='quiz' and name='timelimit'. It is not accessible through $CFG, to get it you can use get_config('quiz', 'timelimit').

And this example shows a $defaultsetting that is an array.

Normally, if you want a particular sort of setting, the easiest way is to look around the admin screens of your Moodle site, and find a setting like the one you want. Then go and copy the code and edit it. Therefore, we do not include a complete list of setting types here.

==External pages==

admin_externalpages represent screens of settings that do not fall into the standard pattern of admin_settings. The admin_externalpage object in the settings tree holds the URL of a PHP page that controls various settings.

In that PHP page, near the start you need to call the function admin_externalpage_setup($pagename), then, instead of the usual print_header and print_footer functions, you use admin_externalpage_print_header() and admin_externalpage_print_footer() functions. This ensures that your page appears with the administration blocks and appropriate navigation.

Note that there are some subclasses of admin_externalpage, for example admin_page_managemods. In a lot of cases, these subclasses only exist to override the search method so this page can be found by appropriate searches.

Once again, to understand this in more depth, your best approach is to look at how some of the external pages in Moodle work.

==See also==

* [[Development:Modules|adding settings for actvitiy modules]]
* [[Development:Admin_reports#How_your_report_gets_included_in_the_admin_tree|adding admin reports to the tree]]
* [[Development:Repository_plugins#APIs_for_Administration|configuration of repository plugins]]
* [[Development:Filters#Adding_a_settings_screen|configuration for filter]]
* [[Development:Developer_documentation|Other developer documentation]]

[[Category:Developer]]

Development:Portfolio API

2008-09-16T18:00:35Z

Nicolasconnault: /* mod/assignmnet */

{{Moodle_2.0}}This page describes the specification for a future feature, currently being worked on for [[Roadmap|Moodle 2.0]]. This spec is STILL UNDER CONSTRUCTION.

==Overview==

The Portfolio API is a core set of interfaces that all Moodle code will/should use so that we can easily publish files to all kinds of external document repository systems.

It's important to remember that portfolios are generally treated as WRITE-ONLY. All we are doing in Moodle is grabbing stuff and pushing it out to somewhere. Management of the files and further combining/reflecting is done through the native interface provided by the portfolio system. Reading of files from a repository is handled by the [[Development:Repository API|Repository API]].

A typical user story:

# When portfolios are enabled, every page or major piece of content in Moodle has a little "Save" button beside it.
# User clicks one of these buttons
# User is able to choose from a list of configured portfolios (this step will be skipped if there's only one).
# User may be asked to define the format of the captured content (eg pdf, IMS LD, HTML, XML ...)
# User may be asked to define some metadata to go with the captured content (some will be generated automatically).
# The content and metadata is COPIED to the external portfolio system
# User has an option to "Return to the page you left" or "Visit their portfolio".

Note this will be just as useful for teachers as for students.

The formatting possibilities will vary depending on the context of the button and the type of external portfolios. So for example, the "Save" button on the course page would allow the user to capture the whole course in IMS LD or Moodle backup format, which you would not have on a forum page.

==Architecture==

Here is how it will work:

===Plugins and libraries===

There will be one type of plugins

# Portfolio (eg Mahara/Elgg/OSP/Facebook/Download) - this will be portfolio/type/xxx

The transport layer (eg mnet/http/scp/cp/dav etc) or clients (eg box.net/flickr) will be written as libraries, to be shared by both repository and portfolio.

Then there will be different formats that plugins will support (and the part of moodle exporting content must support as well), eg IMS, moodle native, mahara native, pdf, encrypted pdf. These will have good libraries supporting them.

===Admin===

It is important to be allowed to have multiple instances of (some) plugins. The workflow for adding a new one is:

*Admin navigates to portfolio config
*Selects from the list of available portfolio plugins, and clicks 'add a new external portfolio' (some may be disabled if there is an instance already and the plugin doesn't support multiple instances)
*Configure the plugin - select which transport and content types to use if there are multiple supported and installed, urls, authentication keys, etc.
*Set permissions (maybe) - handled by roles.

This is not necessary for every type of portfolio, because many will just require the user to authenticate directly and if we do ever want to retain settings for each user we just use user preferences.

===Exporting===

*User is viewing a page that calls new portfolio_add_button(). This checks to see if there are any configured portfolio plugin instances, and also (maybe) any permissions related to portfolios, and what the user's portfolio settings are, and then displays either a single 'add to portfolio' button, or a drop down menu of the available systems with the add button.
*When this button is pressed, the user is redirected to portfolio/add.php, with some post data containing the responsible area (activity module or something like course or blog) callback file and callback arguments, as well as optionally some information about what type of content it is.
*On this page, the user is presented with a form to enter metadata about the item, and configure any options. At this point if there are multiple formats available for export (based on the intersection of what the plugin and module support), the user can select which format they want. The plugin and module can both export mform elements for this page. The user can at this point also select to send the data and wait (with a warning it might take awhile), or queue it for processing if it's larger. This is determined by the size of the content to be exported.
*When the user has submitted the form, they are displayed a summary of what they're about to export, with 'confirm' and 'cancel' buttons. Cancel cancels the request, and cleans up any temporary data, and returns the user to where they came from, while confirm goes to the next step.
*At any point, the portfolio plugin might need to take control for a step. For example, facebook or flickr might require the user to log in for the first time and confirm moodle is allowed to access their API.
*When the user has confirmed their summary, a 'portfolio_send' event will be triggered. At this point, one of two things happen.
# If the user has elected to wait, the 'instant' event is fired, and when the caller gets control again, it displays the status to the user.
# If the user has elected to queue, the delayed event is fired and the user is notified.
*The user is given the option to continue to their portfolio, or return to where they were
*When the event is handled (either through the cron or instant event), the following happens:
*The event handler is invoked. This reawakens the transfer and defers control to the caller and then the portfolio to prepare and send the package.
*When this is complete, we return control to the event handler (which, if it's an 'instant' one, will return true to the caller.

===Storage===

Obviously during this process, state is going to be lost between webserver requests and also between user input and event handling. All of the data is stored in the database, in the form of a serialized (and base64 encoded) representation of the exporter, plugin and caller objects.

Files are also going to be written during the preparation stage of the export, and these are stored in a special portfolio area using the new files api.

===Access/Permissions===
* The calling code is responsible for performing the permission checks necessary before asking to display any button, but during the export the portfolio code will call a check_permissions function on the caller object.
* I would really like to be able to make some portfolio instances available to some roles but this has fallen out of scope.

===Event API===

The portfolio code uses the event api to handle queued events and there is one entry point for this that reawakens the transfer objects and resumes the transfer. Additionally, portfolio plugins can subscribe to events like any other part of moodle.

==Technical==

===Abstract Portfolio Baseclass: portfolio_plugin_base===

Mixes providing some basic functionality by means of its own functions, with a number of abstract functions plugins must implement, and with some functions that plugins can also optionally override.

See also: [[Development:Writing_a_Portfolio_Plugin]] for a full list of all methods you must/can/shouldn't override, as well as associated instructions for what else you need to do to create a new portfolio plugin

===Abstract Caller Baseclass : portfolio_caller_base===

Whenever somewhere in Moodle wants an 'add to portfolio' button, they must subclass this.

See also: [[Development:Adding_a_Portfolio_Button_to_a_page]] for a full list of all methods you must/can/shouldn't override as well as the associated instructions for how to call portfolio_add_button.

===Database Tables===

The actual information about plugins that are installed is just stored in mdl_config_plugin.

Additionally, as we're configuring instances of plugins, rather than just one config set per plugin, we're not using mdl_config_plugin, but instead our own set of tables:

'''portfolio_instance:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|plugin
|varchar(50)
|name of plugin (should match directory in portfolio/type)
|-
|name
|varchar(255)
|name of this plugin instance
|-
|visible
|smallint
|0 or 1
|}

'''portfolio_instance_config:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|instance
|integer
|(pseudo)fk to portfolio_instance
|-
It cannot, however, be responsible for how external systems deal with this case. The different plugins can do what they can. For example, mahara will create new files rather than overwrite. The box.net plugin will try very hard to rename files to avoid collisions.
|name
|varchar(255)
|config name
|-
|value
|text
|config value
|}

'''portfolio_instance_user:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|instance
|integer
|(pseudo)fk to portfolio_instance
|-
|userid
|integer
|(pseudo)fk to mdl_user
|-
|name
|varchar(255)
|config name
|-
|value
|text
|config value
|}

'''portfolio_log:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|userid
|integer
|(pseudo) fk to mdl_user
|-
|time
|integer
|unix timestamp of transfer
|-
|portfolio
|integer
|(pseudo) fk to mdl_portfolio_instance
|-
|caller_class
|varchar(150)
|name of caller class (used in the case of duplicates to display information)
|-
|caller_file
|varchar(255)
|file that contains the definition of caller_class
|-
|caller_sha1
|varchar(255)
|sha1 information of export
|}

'''portfolio_tempdata'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|data
|text
|serialized representation of export data
|-
|expirytime
|integer
|time this data (and the transfer) expires (and the record (and associated files)) will be deleted
|-
|userid
|integer
|psuedo fk to mdl_user
|}

All plugins can also implement their own database tables as needed, by creating a db/install.xml and db/upgrade.php inside portfolio/type/xxx/ (See [[Development:Writing_a_Portfolio_Plugin]]) for more information.

===Portfolio Plugins===

#[[Development:Mahara_Portfolio_Plugin|mahara]] (will be done for the initial implementation)
#download (will be done for the initial implementation)
#box.net (will be done for the initial implementation)
#flickr (Nico has been writing this but it is incomplete)
#googledocs (I think DanP has been writing this)

transport types and formats should be able to be found in a shared location for multiple plugins of both portfolio and repository type to use, but also might be specific to one type of plugin which means that moodle should support looking in multiple locations for these plugins. (eg mahara native format would be in the mahara portfolio plugin, but pdf format will be in a shared library)

===Possible Transport Types===

# mnet (will be done for the initial implementation as part of the Mahara Portfolio Plugin)
# download (just uses send_file_* functions)
# http
# filesystem (could be local/nfs/samba whatever) (cp)
# ssh based (eg scp - should find and re-use the elgg block code as it deals with using ssh keys nicely)
# webdav
# open social? (http://code.google.com/apis/opensocial/)

===Possible Export Formats===

(Note that we don't necessarily want more than one of these for the initial implementation)

* implemented now:
# html
# image
# video
# plaintext
# 'file' (fallback)

* possibly implemented in the future
# pdf
# encrypted pdf
# ims?
# leap/piop? (http://wiki.cetis.ac.uk/LEAP_2.0)
# moodle native?
# mahara native?
# Dublin Core (Enovation implemented this)

===Testing===

At this stage, the portfoliolib and button objects have tests, and the callers have tests to check whether their sha1 generation remains consistent appropriately. This includes the implicit testing of constructing the caller objects (which verifies the callback arguments).

The plugins are not currently tested and even if they were we would not be able to test interaction with the remote system.

==MNET==

This section has moved to [[Development:MNET_Roadmap]]

See also [[Development:MNET_API]] for the documentation of xmlrpc functions

==Duplication of Data==

Moodle will keep track of what content it transfers and when. It keeps a sha1 has of the data, so that if the user tries to export the same content, Moodle can warn the user.

It cannot, however, be responsible for how external systems deal with this case. The different plugins can do what they can. For example, mahara will create new files rather than overwrite. The box.net plugin will try very hard to rename files to avoid collisions.

==Save points in Moodle==

moved to http://tracker.moodle.org/browse/MDL-15758 during development

==Still TODO==

There are a few things still I have not been able to complete for various reasons (generally reliance on other parts of the system, eg Files API). There are bugs for all of these, but reproduced here for completeness:

* MDL-16406 - waiting on QA (Jerome)
* MDL-16048 - waiting on QA (Nico)
* MDL-16313 - this just didn't get far enough up my list and I'm still not sure how relevant it is.
* MDL-15777 - reliance on Files API - data fields that subclass data_field_file need to be extracted and copied separately into the export area using copy_existing_file - this is essentially done but I still think picture should subclass file. More info in MDL-16493
* MDL-15777 - reliance on Files API - data module can only export as CSV even though plain export also supports ods/xls as those two libraries are not updated to the new Files API. More info in MDL-15911
* MDL-16326 - reliance on Files API - 'file' resource module has not been updated to use Files API, so exporting from this type is not yet implemented (currently HTML and plaintext only)
* MDL-16175 - (currently) unreasonable reliance on exceptions. Especially for queued events, currently if a portfolio transfer is woken up at cron to be completed and an error happens in mnet (eg one remote site is down or misconfigured), the entire cronjob will die as mnet functions call print_error, which calls die(). This essentially means cron will stay broken (for all of moodle) until that transfer expires. This is not really a bug in portfolio code, but it definitely exacerbates an already brittle situation.

===Unit Test TODO===

Currently there's quite a few tests implemented, but outstanding are tests that rely on the generator to create files for the callers. As the generator gets updated to create this data, the portfolio unit tests will start throwing exceptions in the portfolio_exporter_text->copy_existing_file method so that it will become obvious when this needs to be updated as the tests will start failing (they are currently passing)

==Current exhaustive list of export scenarios==

===mod/assignment===

====upload single file====

This is pretty straightforward. It should display the export icon next to the single file (no large form/button), and the export should respect the mime-based subtypes (_IMAGE, _VIDEO etc)

====upload multiple files====

This one is a little more complex. You should get an export icon next to individual files, and, additionally, if there is more than one, an export form at the bottom (which will export all files). Exporting multiple files will always stop mime detection and fallback to _FILE format.

====online text====

Displays the full form at the bottom of the page. Should export as format _HTML.

===mod/chat===

These should all export as format _HTML, and contain no references back to Moodle (eg user profile images, which won't be able to be seen necessarily)

====Session page====

Should export entire session.

====Report page per session====

Should export entire session.

====Report page all sessions====

Should concatenate all sessions together.

===mod/data===

====Single entry export====

Should export as HTML. Any files should be included along with the html. If there is only one field in the entry and it is a file or image, the mimetype should be respected, and the export format should be based on that (eg _IMAGE)

====Whole database instance export====

Should export as CSV. Files are not included (This is the same as the other CSV export)

===mod/forum===

====Whole discussion====

The export format is FILE, attachments come alongside discussion.html - this could be improved later to be HTML if there are no attachments.

====Single post====

The export format is _HTML.

====Single post with attachments====

The export format is FILE as it's mixed, and attachments come alongside post.html.

====Single attachment====

Mimetypes should be respected and the export format should be based on them (eg _IMAGE)

===mod/glossary===

====Single entry export====

Should export the entry as HTML.

====Whole glossary export====

Should export the glossary as CSV - similar to a current glossary export.

===mod/resource===

====HTML resource====

Should export as _HTML.

====text resource====

Should export as _TEXT.

====file resource====

Not implemented yet. Blocked by FILES API. Should respect the mimetype of the file and export in the appropriate format.

==See also==
* [[Development:Repository API]]
* [[Development:File API]]

Talk:Gradebook 1.9 Tutorial

2008-09-08T12:57:23Z

Nicolasconnault: /* No Item Weight setting for categories */

Strategy 1: Items weighted by their point values

I would call this Strategy "Unweighted" since the weight of each item is 1. I guess this is what Moodle means by "simple weighted"

== No Item Weight setting for categories ==

In my Moodle 1.9.2 I can see no ''Item Weight'' setting for grade categories (it's mentioned in the tutorial). But there is an item called ''Aggregation coefficient''. Are these two identical? <br />--[[User:Daniel Miksik|Daniel Miksik]] 07:16, 5 September 2008 (CDT)
:Yes, aggregation coefficient is synonymous with "weight" in this context.[[User:Nicolas Connault|Nicolas Connault]] 07:57, 8 September 2008 (CDT)

Talk:Gradebook 1.9 Tutorial

2008-09-08T12:54:24Z

Nicolasconnault: /* No Item Weight setting for categories */

Strategy 1: Items weighted by their point values

I would call this Strategy "Unweighted" since the weight of each item is 1. I guess this is what Moodle means by "simple weighted"

== No Item Weight setting for categories ==

In my Moodle 1.9.2 I can see no ''Item Weight'' setting for grade categories (it's mentioned in the tutorial). But there is an item called ''Aggregation coefficient''. Are these two identical? <br />--[[User:Daniel Miksik|Daniel Miksik]] 07:16, 5 September 2008 (CDT)
|Yes, aggregation coefficient is synonymous with "weight" in this context.

Development:Portfolio API

2008-08-18T07:55:12Z

Nicolasconnault: typos

{{Moodle_2.0}}This page describes the specification for a future feature, currently being worked on for [[Roadmap|Moodle 2.0]]. This spec is STILL UNDER CONSTRUCTION.

==Overview==

The Portfolio API is a core set of interfaces that all Moodle code will/should use so that we can easily publish files to all kinds of external document repository systems.

It's important to remember that portfolios are generally treated as WRITE-ONLY. All we are doing in Moodle is grabbing stuff and pushing it out to somewhere. Management of the files and further combining/reflecting is done through the native interface provided by the portfolio system. Reading of files from a repository is handled by the [[Development:Repository API|Repository API]].

A typical user story:

# When portfolios are enabled, every page or major piece of content in Moodle has a little "Save" button beside it.
# User clicks one of these buttons
# User is able to choose from a list of configured portfolios (this step will be skipped if there's only one).
# User may be asked to define the format of the captured content (eg pdf, IMS LD, HTML, XML ...)
# User may be asked to define some metadata to go with the captured content (some will be generated automatically).
# The content and metadata is COPIED to the external portfolio system
# User has an option to "Return to the page you left" or "Visit their portfolio".

Note this will be just as useful for teachers as for students.

The formatting possibilities will vary depending on the context of the button and the type of external portfolios. So for example, the "Save" button on the course page would allow the user to capture the whole course in IMS LD or Moodle backup format, which you would not have on a forum page.

==Architecture==

Here is how it will work:

===Plugins and libraries===

There will be two separate types of plugins

# Portfolio (eg Mahara/Elgg/OSP/Facebook/Download) - this will be portfolio/type/xxx
# Transport (eg mnet/http/scp/cp/dav etc) - these will be reused across different portfolio plugins

Then there will be different formats that plugins will support (and the part of moodle exporting content must support as well), eg IMS, moodle native, mahara native, pdf, encrypted pdf. These will have good libraries supporting them.

The reason for abstracting the transport layer to its own plugin type is that it's pretty likely that the same external system might talk both portfolio and repository.

===Admin===

It is important to be allowed to have multiple instances of (some) plugins. The workflow for adding a new one is:

*Admin navigates to portfolio config
*Selects from the list of available portfolio plugins, and clicks 'add a new external portfolio' (some may be disabled if there is an instance already and the plugin doesn't support multiple instances)
*Configure the plugin - select which transport and content types to use if there are multiple supported and installed, urls, authentication keys, etc.
*Set permissions (maybe) - handled by roles.

This is not necessary for every type of portfolio, because many will just require the user to authenticate directly and if we do ever want to retain settings for each user we just use user preferences.

===Exporting===

*User is viewing a page that calls portfolio_add_button(). This checks to see if there are any configured portfolio plugin instances, and also (maybe) any permissions related to portfolios, and what the user's portfolio settings are, and then displays either a single 'add to portfolio' button, or a drop down menu of the available systems with the add button.
*When this button is pressed, the user is redirected to portfolio/add.php, with some post data containing the responsible area (activity module or something like course or blog) callback file and function.
*On this page, the user is presented with a form to enter metadata about the item, and configure any options. At this point if there are multiple formats available for export (based on the intersection of what the plugin and module support), the user can select which format they want. The plugin and module can both export mform elements for this page. The user can at this point also select to send the data and wait (with a warning it might take awhile), or queue it for processing if it's larger.
*When the user has submitted the form, they are displayed a summary of what they're about to export, with 'confirm' and 'edit' buttons. Edit just relaunches the form, and confirm goes to the next step.
*At this point, the portfolio plugin (or transport plugin) might need to take control for a step. For example, facebook or flickr might require the user to log in for the first time and confirm moodle is allowed to access their API
*When the user has confirmed their summary, a 'portfolio_send' event will be triggered. At this point, one of two things happen.
# If the user has elected to wait, the 'instant' event is fired, and when the caller gets control again, it displays the status to the user.
# If the user has elected to queue, the delayed event is fired and the user is notified.
*The user is given the option to continue to their portfolio, or return to where they were
*
*When the event is handled (either through the cron or instant event), the following happens:
*The event handler is invoked. This is almost certainly going to be a function in lib/portfoliolib.php rather than any handler in the portfolio plugins. The event handler in turn calls the original callback that was passed to portfolio/add.php (module//lib.php or course/lib.php or something), which prepares the data in the format the user selected and returns control to the event handler.
*At this point we have all the metadata we need, and we have the content to send, so it's time to start transport negotiation, which happens in stages. Note that some plugins might not implement all stages, and 3 and 4 might be interchangeable in order, depending on the external system.
#request
#authentication
#content
#metadata (some control is passed to the portfolio plugin for packaging - either to just write out an xml file and zip up the whole package, or send a series of requests)
#completion
*When this is complete, we return control to the event handler (which, if it's an 'instant' one, will return true to the caller, which will be the ajax request)

===Storage===

Obviously during this process, state is going to be lost between webserver requests and also between user input and event handling. Some data will be stored in the user's session (the result of the metadata form, for example) and some in temporary files on the file system. These will be stored in dataroot/temp/portfolio/$userid-$timestamp/

===Access/Permissions===
* Portfolio plugin must be allocatable per context (site/course/group etc)
* How can we assume all users will have an account on the remote system? We could try and silently authenticate them (or at least see if it's possible to) in the background at the first call to portfolio_add_button (and then cache the result in the session)
* Check MNet policy - it assumes everyone with the role can have an account (assuming they_sso_in or whatever it's called is on) - this doesn't scale though, other transport mechanisms and or plugin types might not follow this assumption. I think this really must be defined by the portfolio plugin only, not the transport plugin.

===Event API===

It is probably worth using the event api to handle the sending of data, mostly just because it provides a handy way of detached processing. However, any portfolio_send event will be 'owned' by particular "instances" of plugins - and each plugin that subscribes to portfolio_send event (all of them by definition) will have to check if the event data belongs to any of their instances, which is potentially quite messy.

==Technical==

===Abstract Portfolio Baseclass: portfolio_plugin_base===

Mixes providing some basic functionality by means of its own functions, with a number of abstract functions plugins must implement, and with some functions that plugins can also optionally override.

See also: [[Development:Writing_a_Portfolio_Plugin]] for a full list of all methods you must/can/shouldn't override, as well as associated instructions for what else you need to do to create a new portfolio plugin

===Abstract Caller Baseclass : portfolio_caller_base===

Whenever somewhere in Moodle wants an 'add to portfolio' button, they must subclass this.

See also: [[Development:Adding_a_Portfolio_Button_to_a_page]] for a full list of all methods you must/can/shouldn't override as well as the associated instructions for how to call portfolio_add_button.

===Database Tables===

The actual information about plugins that are installed is just stored in mdl_config.

Additionally, as we're configuring instances of plugins, rather than just one config set per plugin, we're not using mdl_config_plugin, but instead our own set of tables:

'''portfolio_instance:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|plugin
|varchar(50)
|name of plugin (should match directory in portfolio/type)
|-
|name
|varchar(255)
|name of this plugin instance
|-
|visible
|smallint
|0 or 1
|}

'''portfolio_instance_config:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|instance
|integer
|(pseudo)fk to portfolio_instance
|-
|name
|varchar(255)
|config name
|-
|value
|text
|config value
|}

'''portfolio_instance_user:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|instance
|integer
|(pseudo)fk to portfolio_instance
|-
|userid
|integer
|(pseudo)fk to mdl_user
|-
|name
|varchar(255)
|config name
|-
|value
|text
|config value
|}

'''portfolio_log:'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|userid
|integer
|(pseudo) fk to mdl_user
|-
|time
|integer
|unix timestamp of transfer
|-
|portfolio
|integer
|(pseudo) fk to mdl_portfolio_instance
|-
|caller_class
|varchar(150)
|name of caller class (used in the case of duplicates to display information)
|-
|caller_file
|varchar(255)
|file that contains the definition of caller_class
|-
|caller_sha1
|varchar(255)
|sha1 information of export
|}

'''portfolio_tempdata'''

{| border="1" cellpadding="2" cellspacing="0"
|-
|'''Field'''
|'''Datatype'''
|'''Comment'''
|-
|id
|integer
|sequence
|-
|data
|text
|serialized representation of export data
|-
|expirytime
|integer
|time this data (and the transfer) expires (and the record (and associated files)) will be deleted
|}

All plugins can also implement their own database tables as needed, by creating a db/install.xml and db/upgrade.php inside portfolio/type/xxx/ (See [[Development:Writing_a_Portfolio_Plugin]]) for more information.

===Portfolio Plugins===

#[[Development:Mahara_Portfolio_Plugin|mahara]] (will be done for the initial implementation)
#download (should be done for the initial implementation)

transport types and formats should be able to be found in a shared location for multiple plugins of both portfolio and repository type to use, but also might be specific to one type of plugin which means that moodle should support looking in multiple locations for these plugins. (eg mahara native format will be in the mahara portfolio plugin, but pdf format will be in a shared library)

===Possible Transport Types===

# mnet (will be done for the initial implementation as part of the Mahara Portfolio Plugin)
# download (will be done for the initial implementation as part of the Download Portfolio Plugin)
# http
# filesystem (could be local/nfs/samba whatever) (cp)
# ssh based (eg scp - should find and re-use the elgg block code as it deals with using ssh keys nicely)
# webdav
# open social? (http://code.google.com/apis/opensocial/)

===Possible Export Formats===

(Note that we don't necessarily want more than one of these for the initial implementation)

# html
# pdf
# encrypted pdf
# ims?
# leap/piop? (http://wiki.cetis.ac.uk/LEAP_2.0) -- looking increasingly possible Mahara will speak this, so hopefully for the intial implementation
# moodle native?
# mahara native?
# Dublin Core (Enovation implemented this)

===Testing===

Nico to help (see Development Approach for TDD-related approach)

==MNET==

This section has moved to [[Development:MNET_Roadmap]]

==Duplication of Data==

Moodle will keep track of what content it transfers and when. It'll remember the url of the original export location (alphabetisising the url parameters and removing the sessionkey) and a sha1 hash of the data, so that if the user tries to export the same content, Moodle can warn the user (along with a notification of whether it has changed or not).

At this point we can build support into the API for plugins to offer to differentiate between 'replace' and 'add', but won't implement it for any we write for the initial implementation.

==Assumptions==

* Using Moodle's built in Event API

==Save points in Moodle==

moved to http://tracker.moodle.org/browse/MDL-15758 during development

==Outstanding questions==

*Dependance of transport plugins - eg filesystem might depend on http for sending request/finished pings

==Development Approach==
* create two base classes
* create mahara portfolio plugin, relying on mnet
* create portfolio/add.php and portfolio/lib.php
* start adding calls to the portfolio_add_button function to various places in moodle, with their callbacks.
* create a dummy listener in mahara until the mahara side is specified

After a conversation with Nico about testing, we decided the most sensible ordering to start with would be:

# Write portfolio/add.php
# Write *skeleton functions* to support it in portfolio/lib.php and the class structure
# Write tests for those methods that are necessary
# Implement the functions to make the tests pass
# Keep adding module implementations

==See also==
* [[Development:Repository API]]
* [[Development:File API]]

Development:Adding a Portfolio Button to a page

2008-08-14T14:30:18Z

Nicolasconnault: Highlighting PHP code

==Introduction==

Adding an 'Add to Portfolio' button to any page is relatively trivial, there are just two things that you need to do:

==Write a subclass==

You can either subclass portfolio_caller_base for the general case, or portfolio_module_caller_base if you're somewhere inside mod/

This sounds scary, but really it's not! It's a very small class. portfolio_caller_base has abstract functions that you '''must''' override, and some functions that you '''can''' override if you want to do something special. You should call it something like $module_portfolio_caller, or in a more complicated case (say assignment/type/upload, assignment_upload_portfolio_caller)

If you're adding the portfolio button somewhere in a module, it's better to subclass portfolio_module_caller_base, which implements 2 of the below abstract methods for you.

===Methods you must override===

=====__construct=====

When your object is constructed, the contents of whatever callback arguments you passed to portfolio_add_button are passed back to you here in an array, so your chance to set member variables or do whatever you need is in the constructor.

=====get_navigation=====

During the export screens, it's desirable to still have some sensible navigation that logically follows from the place the user was before they started the export process. This function should return components to pass to build_nagivation (extralinks and cm). '''portfolio_module_caller_base implements this for you'''

=====prepare_package=====

prepares the package up before control is passed to the portfolio plugin. You should copy any files (or write out any files) into the temporary directory provided, where they'll be found by the portfolio plugin

See also [[Development:Adding_a_Portfolio_Button_to_a_page#setting portfolio internal]]

=====expected_time=====

You should return a constant here to indicate how long the transfer is expected to take. This should be based on the size of the file. There are three options, PORTFOLIO_TIME_LOW, PORTFOLIO_TIME_MODERATE, and PORTFOLIO_TIME_HIGH.
The first means the user will not be asked if they want to wait for the transfer or not, they will just wait. The second and third mean they'll be given the option (and in the case of the third, advised not to).
The portfolio plugin can override this if it wants (eg in the case of download, they always want to wait for the transfer)

=====check_permissions=====

portfolio/add.php will expect the caller to verify the user is allowed to export the given content. This function should perform any has_capability checks it needs to and return a booelan.

=====get_return_url=====

This is used for redirecting the user in the case of a cancelled export, or at the end of their export, they are offered the option of continuing back to where they were (what this function returns) or on to their portfolio. '''portfolio_module_caller_base implements this for you (but will use mod/modname/view.php)'''

=====display_name (static)=====

A nice language string for displaying the location of this export to the user (this is used to notify the user in case of duplicate exports that originated from different places in moodle (Eg exporting an assignment upload and a forum post attachment that are the same file)

=====get_sha1=====

Return a sha1 of the content being exported - used to detect duplicate exports later.

===Methods you can override===

=====supported_formats (static)=====

The formats this caller can support. At export time, both the plugin and the caller are polled for which formats they can support, and then the intersection is used to determine the export format. In the case that the intersection is greater than 1, the user is asked for their selection.

The available formats you can choose from are in portfolio_supported_formats and are constants PORTFOLIO_FORMAT_XXX. By default, the subclass defines PORTFOLIO_FORMAT_FILE.

=====has_export_config=====

If there's any addition config during the export process (for example, extra metadata), you can override this function to return true. If you do this, you must also override export_config_form and get_export_summary.

=====export_config_form=====

This function is called, and passed a moodle form object by reference to add elements to it.

====export_config_validation====

This follows the exact same format as the validation() function in the moodleform object.

====get_allowed_export_config====

If at any point, your caller is going to use set_export_config, you must implement this function to return an array of allowed config fields. (Note that you can set export time config even if you're not using interactive user config)

====get_export_summary====

If your plugin has overridden has_export_config, you must implement this to display nicely to the user on the confirmation screen. It should return a named array (keys are nice strings to describe the config, values are the config options)

==Call portfolio_add_button in the appropriate place==

Now that you've implemented this class, you just need to add the button. To do this, you require_once("$CFG->libdir/portfoliolib.php"); and call portfolio_add_button. It takes the following parameters:

=====$callbackclass=====

The name of the class you've made that subclassed portfolio_caller_base.

=====$callbackargs=====

An associative array of key=>value pairs you want passed to the constructor of your class. These '''must''' be primitives, as they are added as hidden form fields and cleaned to either PARAM_ALPHAEXT, PARAM_NUMERIC or PARAM_PATH. Weird stuff will happen if they're not compliant.

=====$callbackfile=====

This can be autodetected from the backtrace of where this function was called, but if your class definition isn't in the same file as the caller (eg if your caller is some .php script, but the class is in a lib.php file), you can pass it explicitly here.

=====$fullform=====

whether you want the full form with the dropmenu of available plugins or just a little icon. defaults to true. (using the icon will force a whole screen on the wizard)

=====$return=====

Whether you want the output returned or echoed. Defaults to false (echo)

==Other ways to integrate==

If it's undesirable to add a form, there are a couple of things you can do. For an example, see the export tab of the 'data' module, which has already an export form, but just adds an option to export to portfolio rather than file:

<code php>
require_once($CFG->libdir . '/portfoliolib.php');
if (has_capability('mod/data:exportallentries', get_context_instance(CONTEXT_MODULE, $this->_cm->id))) {
if ($portfoliooptions = portfolio_instance_select(portfolio_instances(),
call_user_func(array('data_portfolio_caller', 'supported_formats')),
'data_portfolio_caller', '', true, true)) {
$mform->addElement('header', 'notice', get_string('portfolionotfile', 'data') . ':');
$portfoliooptions[0] = get_string('none');
ksort($portfoliooptions);
$mform->addElement('select', 'portfolio',
get_string('portfolio', 'portfolio'), $portfoliooptions);
}
}
</code>

This code adds a select option to the existing export form containing the available portfolio instances.

Then in the form handler:
<code php>
if (array_key_exists('portfolio', $formdata) && !empty($formdata['portfolio'])) {
// fake portfolio callback stuff and redirect
$formdata['id'] = $cm->id;
$formdata['exporttype'] = 'csv'; // force for now
$url = portfolio_fake_add_url($formdata['portfolio'], 'data_portfolio_caller',
'/mod/data/lib.php', $formdata);
redirect($url);
}
</code>

The portfolio_fake_add_url function returns the url that you need to redirect to, that would normally be the result of portfolio_add_button form being submitted.

==A few extra notes on the caller base class==

===protected $course===
There is a protected member variable, $course, that subclasses can set (with $this->set('course', $course); ).

portfolio_add_button tries to look for a course object at the point that it's called, by doing a global $COURSE which is hackish but mostly works. This is also used to build the navigation during the export process. If for some reason, your navigation doesn't include the current course, you can set it like this. You shouldn't need to in the majority of cases.

===serialization===
The caller object is stored in the database in serialized form. The portfolio code works around this by loading the class definitions and then serializing and unserializing the objects again. However, if you're storing any real objects in your caller class, you will need to do this as well. See the assignment implementation for how this done (using php5's __wakeup function):

<code php>
public function __wakeup() {
require_once($this->assignmentfile);
$this->assignment = unserialize(serialize($this->assignment));
}
</code>

===setting portfolio internal===
Sometimes during prepare_package, you need to call functions in the libraries to render content as HTML that would normally also call portfolio_add_button. This will result in 'you already have an export active in this session' error - to get around this, do something like

<code php>
define('PORTFOLIO_INTERNAL', true);
modulename_print_some_htmlcontent();
</code>

Development:Adding a Portfolio Button to a page

2008-08-14T14:29:55Z

Nicolasconnault: Highlighting PHP code

==Introduction==

Adding an 'Add to Portfolio' button to any page is relatively trivial, there are just two things that you need to do:

==Write a subclass==

You can either subclass portfolio_caller_base for the general case, or portfolio_module_caller_base if you're somewhere inside mod/

This sounds scary, but really it's not! It's a very small class. portfolio_caller_base has abstract functions that you '''must''' override, and some functions that you '''can''' override if you want to do something special. You should call it something like $module_portfolio_caller, or in a more complicated case (say assignment/type/upload, assignment_upload_portfolio_caller)

If you're adding the portfolio button somewhere in a module, it's better to subclass portfolio_module_caller_base, which implements 2 of the below abstract methods for you.

===Methods you must override===

=====__construct=====

When your object is constructed, the contents of whatever callback arguments you passed to portfolio_add_button are passed back to you here in an array, so your chance to set member variables or do whatever you need is in the constructor.

=====get_navigation=====

During the export screens, it's desirable to still have some sensible navigation that logically follows from the place the user was before they started the export process. This function should return components to pass to build_nagivation (extralinks and cm). '''portfolio_module_caller_base implements this for you'''

=====prepare_package=====

prepares the package up before control is passed to the portfolio plugin. You should copy any files (or write out any files) into the temporary directory provided, where they'll be found by the portfolio plugin

See also [[Development:Adding_a_Portfolio_Button_to_a_page#setting portfolio internal]]

=====expected_time=====

You should return a constant here to indicate how long the transfer is expected to take. This should be based on the size of the file. There are three options, PORTFOLIO_TIME_LOW, PORTFOLIO_TIME_MODERATE, and PORTFOLIO_TIME_HIGH.
The first means the user will not be asked if they want to wait for the transfer or not, they will just wait. The second and third mean they'll be given the option (and in the case of the third, advised not to).
The portfolio plugin can override this if it wants (eg in the case of download, they always want to wait for the transfer)

=====check_permissions=====

portfolio/add.php will expect the caller to verify the user is allowed to export the given content. This function should perform any has_capability checks it needs to and return a booelan.

=====get_return_url=====

This is used for redirecting the user in the case of a cancelled export, or at the end of their export, they are offered the option of continuing back to where they were (what this function returns) or on to their portfolio. '''portfolio_module_caller_base implements this for you (but will use mod/modname/view.php)'''

=====display_name (static)=====

A nice language string for displaying the location of this export to the user (this is used to notify the user in case of duplicate exports that originated from different places in moodle (Eg exporting an assignment upload and a forum post attachment that are the same file)

=====get_sha1=====

Return a sha1 of the content being exported - used to detect duplicate exports later.

===Methods you can override===

=====supported_formats (static)=====

The formats this caller can support. At export time, both the plugin and the caller are polled for which formats they can support, and then the intersection is used to determine the export format. In the case that the intersection is greater than 1, the user is asked for their selection.

The available formats you can choose from are in portfolio_supported_formats and are constants PORTFOLIO_FORMAT_XXX. By default, the subclass defines PORTFOLIO_FORMAT_FILE.

=====has_export_config=====

If there's any addition config during the export process (for example, extra metadata), you can override this function to return true. If you do this, you must also override export_config_form and get_export_summary.

=====export_config_form=====

This function is called, and passed a moodle form object by reference to add elements to it.

====export_config_validation====

This follows the exact same format as the validation() function in the moodleform object.

====get_allowed_export_config====

If at any point, your caller is going to use set_export_config, you must implement this function to return an array of allowed config fields. (Note that you can set export time config even if you're not using interactive user config)

====get_export_summary====

If your plugin has overridden has_export_config, you must implement this to display nicely to the user on the confirmation screen. It should return a named array (keys are nice strings to describe the config, values are the config options)

==Call portfolio_add_button in the appropriate place==

Now that you've implemented this class, you just need to add the button. To do this, you require_once("$CFG->libdir/portfoliolib.php"); and call portfolio_add_button. It takes the following parameters:

=====$callbackclass=====

The name of the class you've made that subclassed portfolio_caller_base.

=====$callbackargs=====

An associative array of key=>value pairs you want passed to the constructor of your class. These '''must''' be primitives, as they are added as hidden form fields and cleaned to either PARAM_ALPHAEXT, PARAM_NUMERIC or PARAM_PATH. Weird stuff will happen if they're not compliant.

=====$callbackfile=====

This can be autodetected from the backtrace of where this function was called, but if your class definition isn't in the same file as the caller (eg if your caller is some .php script, but the class is in a lib.php file), you can pass it explicitly here.

=====$fullform=====

whether you want the full form with the dropmenu of available plugins or just a little icon. defaults to true. (using the icon will force a whole screen on the wizard)

=====$return=====

Whether you want the output returned or echoed. Defaults to false (echo)

==Other ways to integrate==

If it's undesirable to add a form, there are a couple of things you can do. For an example, see the export tab of the 'data' module, which has already an export form, but just adds an option to export to portfolio rather than file:

<code php>
require_once($CFG->libdir . '/portfoliolib.php');
if (has_capability('mod/data:exportallentries', get_context_instance(CONTEXT_MODULE, $this->_cm->id))) {
if ($portfoliooptions = portfolio_instance_select(portfolio_instances(),
call_user_func(array('data_portfolio_caller', 'supported_formats')),
'data_portfolio_caller', '', true, true)) {
$mform->addElement('header', 'notice', get_string('portfolionotfile', 'data') . ':');
$portfoliooptions[0] = get_string('none');
ksort($portfoliooptions);
$mform->addElement('select', 'portfolio',
get_string('portfolio', 'portfolio'), $portfoliooptions);
}
}
</code>

This code adds a select option to the existing export form containing the available portfolio instances.

Then in the form handler:
<code php>
if (array_key_exists('portfolio', $formdata) && !empty($formdata['portfolio'])) {
// fake portfolio callback stuff and redirect
$formdata['id'] = $cm->id;
$formdata['exporttype'] = 'csv'; // force for now
$url = portfolio_fake_add_url($formdata['portfolio'], 'data_portfolio_caller',
'/mod/data/lib.php', $formdata);
redirect($url);
}
</code>

The portfolio_fake_add_url function returns the url that you need to redirect to, that would normally be the result of portfolio_add_button form being submitted.

==A few extra notes on the caller base class==

===protected $course===
There is a protected member variable, $course, that subclasses can set (with $this->set('course', $course); ).

portfolio_add_button tries to look for a course object at the point that it's called, by doing a global $COURSE which is hackish but mostly works. This is also used to build the navigation during the export process. If for some reason, your navigation doesn't include the current course, you can set it like this. You shouldn't need to in the majority of cases.

===serialization===
The caller object is stored in the database in serialized form. The portfolio code works around this by loading the class definitions and then serializing and unserializing the objects again. However, if you're storing any real objects in your caller class, you will need to do this as well. See the assignment implementation for how this done (using php5's __wakeup function):

<code php>
public function __wakeup() {
require_once($this->assignmentfile);
$this->assignment = unserialize(serialize($this->assignment));
}
</code>

===setting portfolio internal===
Sometimes during prepare_package, you need to call functions in the libraries to render content as HTML that would normally also call portfolio_add_button. This will result in 'you already have an export active in this session' error - to get around this, do something like

define('PORTFOLIO_INTERNAL', true);
modulename_print_some_htmlcontent();

Development:Adding a Portfolio Button to a page

2008-08-14T14:28:16Z

Nicolasconnault: Highlighting PHP code

==Introduction==

Adding an 'Add to Portfolio' button to any page is relatively trivial, there are just two things that you need to do:

==Write a subclass==

You can either subclass portfolio_caller_base for the general case, or portfolio_module_caller_base if you're somewhere inside mod/

This sounds scary, but really it's not! It's a very small class. portfolio_caller_base has abstract functions that you '''must''' override, and some functions that you '''can''' override if you want to do something special. You should call it something like $module_portfolio_caller, or in a more complicated case (say assignment/type/upload, assignment_upload_portfolio_caller)

If you're adding the portfolio button somewhere in a module, it's better to subclass portfolio_module_caller_base, which implements 2 of the below abstract methods for you.

===Methods you must override===

=====__construct=====

When your object is constructed, the contents of whatever callback arguments you passed to portfolio_add_button are passed back to you here in an array, so your chance to set member variables or do whatever you need is in the constructor.

=====get_navigation=====

During the export screens, it's desirable to still have some sensible navigation that logically follows from the place the user was before they started the export process. This function should return components to pass to build_nagivation (extralinks and cm). '''portfolio_module_caller_base implements this for you'''

=====prepare_package=====

prepares the package up before control is passed to the portfolio plugin. You should copy any files (or write out any files) into the temporary directory provided, where they'll be found by the portfolio plugin

See also [[Development:Adding_a_Portfolio_Button_to_a_page#setting portfolio internal]]

=====expected_time=====

You should return a constant here to indicate how long the transfer is expected to take. This should be based on the size of the file. There are three options, PORTFOLIO_TIME_LOW, PORTFOLIO_TIME_MODERATE, and PORTFOLIO_TIME_HIGH.
The first means the user will not be asked if they want to wait for the transfer or not, they will just wait. The second and third mean they'll be given the option (and in the case of the third, advised not to).
The portfolio plugin can override this if it wants (eg in the case of download, they always want to wait for the transfer)

=====check_permissions=====

portfolio/add.php will expect the caller to verify the user is allowed to export the given content. This function should perform any has_capability checks it needs to and return a booelan.

=====get_return_url=====

This is used for redirecting the user in the case of a cancelled export, or at the end of their export, they are offered the option of continuing back to where they were (what this function returns) or on to their portfolio. '''portfolio_module_caller_base implements this for you (but will use mod/modname/view.php)'''

=====display_name (static)=====

A nice language string for displaying the location of this export to the user (this is used to notify the user in case of duplicate exports that originated from different places in moodle (Eg exporting an assignment upload and a forum post attachment that are the same file)

=====get_sha1=====

Return a sha1 of the content being exported - used to detect duplicate exports later.

===Methods you can override===

=====supported_formats (static)=====

The formats this caller can support. At export time, both the plugin and the caller are polled for which formats they can support, and then the intersection is used to determine the export format. In the case that the intersection is greater than 1, the user is asked for their selection.

The available formats you can choose from are in portfolio_supported_formats and are constants PORTFOLIO_FORMAT_XXX. By default, the subclass defines PORTFOLIO_FORMAT_FILE.

=====has_export_config=====

If there's any addition config during the export process (for example, extra metadata), you can override this function to return true. If you do this, you must also override export_config_form and get_export_summary.

=====export_config_form=====

This function is called, and passed a moodle form object by reference to add elements to it.

====export_config_validation====

This follows the exact same format as the validation() function in the moodleform object.

====get_allowed_export_config====

If at any point, your caller is going to use set_export_config, you must implement this function to return an array of allowed config fields. (Note that you can set export time config even if you're not using interactive user config)

====get_export_summary====

If your plugin has overridden has_export_config, you must implement this to display nicely to the user on the confirmation screen. It should return a named array (keys are nice strings to describe the config, values are the config options)

==Call portfolio_add_button in the appropriate place==

Now that you've implemented this class, you just need to add the button. To do this, you require_once("$CFG->libdir/portfoliolib.php"); and call portfolio_add_button. It takes the following parameters:

=====$callbackclass=====

The name of the class you've made that subclassed portfolio_caller_base.

=====$callbackargs=====

An associative array of key=>value pairs you want passed to the constructor of your class. These '''must''' be primitives, as they are added as hidden form fields and cleaned to either PARAM_ALPHAEXT, PARAM_NUMERIC or PARAM_PATH. Weird stuff will happen if they're not compliant.

=====$callbackfile=====

This can be autodetected from the backtrace of where this function was called, but if your class definition isn't in the same file as the caller (eg if your caller is some .php script, but the class is in a lib.php file), you can pass it explicitly here.

=====$fullform=====

whether you want the full form with the dropmenu of available plugins or just a little icon. defaults to true. (using the icon will force a whole screen on the wizard)

=====$return=====

Whether you want the output returned or echoed. Defaults to false (echo)

==Other ways to integrate==

If it's undesirable to add a form, there are a couple of things you can do. For an example, see the export tab of the 'data' module, which has already an export form, but just adds an option to export to portfolio rather than file:

<code php>
require_once($CFG->libdir . '/portfoliolib.php');
if (has_capability('mod/data:exportallentries', get_context_instance(CONTEXT_MODULE, $this->_cm->id))) {
if ($portfoliooptions = portfolio_instance_select(portfolio_instances(),
call_user_func(array('data_portfolio_caller', 'supported_formats')),
'data_portfolio_caller', '', true, true)) {
$mform->addElement('header', 'notice', get_string('portfolionotfile', 'data') . ':');
$portfoliooptions[0] = get_string('none');
ksort($portfoliooptions);
$mform->addElement('select', 'portfolio',
get_string('portfolio', 'portfolio'), $portfoliooptions);
}
}
</code>

This code adds a select option to the existing export form containing the available portfolio instances.

Then in the form handler:
<code php>
if (array_key_exists('portfolio', $formdata) && !empty($formdata['portfolio'])) {
// fake portfolio callback stuff and redirect
$formdata['id'] = $cm->id;
$formdata['exporttype'] = 'csv'; // force for now
$url = portfolio_fake_add_url($formdata['portfolio'], 'data_portfolio_caller',
'/mod/data/lib.php', $formdata);
redirect($url);
}
</code>

The portfolio_fake_add_url function returns the url that you need to redirect to, that would normally be the result of portfolio_add_button form being submitted.

==A few extra notes on the caller base class==

===protected $course===
There is a protected member variable, $course, that subclasses can set (with $this->set('course', $course); ).

portfolio_add_button tries to look for a course object at the point that it's called, by doing a global $COURSE which is hackish but mostly works. This is also used to build the navigation during the export process. If for some reason, your navigation doesn't include the current course, you can set it like this. You shouldn't need to in the majority of cases.

===serialization===
The caller object is stored in the database in serialized form. The portfolio code works around this by loading the class definitions and then serializing and unserializing the objects again. However, if you're storing any real objects in your caller class, you will need to do this as well. See the assignment implementation for how this done (using php5's __wakeup function):

public function __wakeup() {
require_once($this->assignmentfile);
$this->assignment = unserialize(serialize($this->assignment));
}

===setting portfolio internal===
Sometimes during prepare_package, you need to call functions in the libraries to render content as HTML that would normally also call portfolio_add_button. This will result in 'you already have an export active in this session' error - to get around this, do something like

define('PORTFOLIO_INTERNAL', true);
modulename_print_some_htmlcontent();

Development:Adding a Portfolio Button to a page

2008-08-14T14:06:17Z

Nicolasconnault: /* get_return_url */

==Introduction==

Adding an 'Add to Portfolio' button to any page is relatively trivial, there are just two things that you need to do:

==Write a subclass==

You can either subclass portfolio_caller_base for the general case, or portfolio_module_caller_base if you're somewhere inside mod/

This sounds scary, but really it's not! It's a very small class. portfolio_caller_base has abstract functions that you '''must''' override, and some functions that you '''can''' override if you want to do something special. You should call it something like $module_portfolio_caller, or in a more complicated case (say assignment/type/upload, assignment_upload_portfolio_caller)

If you're adding the portfolio button somewhere in a module, it's better to subclass portfolio_module_caller_base, which implements 2 of the below abstract methods for you.

===Methods you must override===

=====__construct=====

When your object is constructed, the contents of whatever callback arguments you passed to portfolio_add_button are passed back to you here in an array, so your chance to set member variables or do whatever you need is in the constructor.

=====get_navigation=====

During the export screens, it's desirable to still have some sensible navigation that logically follows from the place the user was before they started the export process. This function should return components to pass to build_nagivation (extralinks and cm). '''portfolio_module_caller_base implements this for you'''

=====prepare_package=====

prepares the package up before control is passed to the portfolio plugin. You should copy any files (or write out any files) into the temporary directory provided, where they'll be found by the portfolio plugin

See also [[Development:Adding_a_Portfolio_Button_to_a_page#setting portfolio internal]]

=====expected_time=====

You should return a constant here to indicate how long the transfer is expected to take. This should be based on the size of the file. There are three options, PORTFOLIO_TIME_LOW, PORTFOLIO_TIME_MODERATE, and PORTFOLIO_TIME_HIGH.
The first means the user will not be asked if they want to wait for the transfer or not, they will just wait. The second and third mean they'll be given the option (and in the case of the third, advised not to).
The portfolio plugin can override this if it wants (eg in the case of download, they always want to wait for the transfer)

=====check_permissions=====

portfolio/add.php will expect the caller to verify the user is allowed to export the given content. This function should perform any has_capability checks it needs to and return a booelan.

=====get_return_url=====

This is used for redirecting the user in the case of a cancelled export, or at the end of their export, they are offered the option of continuing back to where they were (what this function returns) or on to their portfolio. '''portfolio_module_caller_base implements this for you (but will use mod/modname/view.php)'''

=====display_name (static)=====

A nice language string for displaying the location of this export to the user (this is used to notify the user in case of duplicate exports that originated from different places in moodle (Eg exporting an assignment upload and a forum post attachment that are the same file)

=====get_sha1=====

Return a sha1 of the content being exported - used to detect duplicate exports later.

===Methods you can override===

=====supported_formats (static)=====

The formats this caller can support. At export time, both the plugin and the caller are polled for which formats they can support, and then the intersection is used to determine the export format. In the case that the intersection is greater than 1, the user is asked for their selection.

The available formats you can choose from are in portfolio_supported_formats and are constants PORTFOLIO_FORMAT_XXX. By default, the subclass defines PORTFOLIO_FORMAT_FILE.

=====has_export_config=====

If there's any addition config during the export process (for example, extra metadata), you can override this function to return true. If you do this, you must also override export_config_form and get_export_summary.

=====export_config_form=====

This function is called, and passed a moodle form object by reference to add elements to it.

====export_config_validation====

This follows the exact same format as the validation() function in the moodleform object.

====get_allowed_export_config====

If at any point, your caller is going to use set_export_config, you must implement this function to return an array of allowed config fields. (Note that you can set export time config even if you're not using interactive user config)

====get_export_summary====

If your plugin has overridden has_export_config, you must implement this to display nicely to the user on the confirmation screen. It should return a named array (keys are nice strings to describe the config, values are the config options)

==Call portfolio_add_button in the appropriate place==

Now that you've implemented this class, you just need to add the button. To do this, you require_once("$CFG->libdir/portfoliolib.php"); and call portfolio_add_button. It takes the following parameters:

=====$callbackclass=====

The name of the class you've made that subclassed portfolio_caller_base.

=====$callbackargs=====

An associative array of key=>value pairs you want passed to the constructor of your class. These '''must''' be primitives, as they are added as hidden form fields and cleaned to either PARAM_ALPHAEXT, PARAM_NUMERIC or PARAM_PATH. Weird stuff will happen if they're not compliant.

=====$callbackfile=====

This can be autodetected from the backtrace of where this function was called, but if your class definition isn't in the same file as the caller (eg if your caller is some .php script, but the class is in a lib.php file), you can pass it explicitly here.

=====$fullform=====

whether you want the full form with the dropmenu of available plugins or just a little icon. defaults to true. (using the icon will force a whole screen on the wizard)

=====$return=====

Whether you want the output returned or echoed. Defaults to false (echo)

==Other ways to integrate==

If it's undesirable to add a form, there are a couple of things you can do. For an example, see the export tab of the 'data' module, which has already an export form, but just adds an option to export to portfolio rather than file:

require_once($CFG->libdir . '/portfoliolib.php');
if (has_capability('mod/data:exportallentries', get_context_instance(CONTEXT_MODULE, $this->_cm->id))) {
if ($portfoliooptions = portfolio_instance_select(
portfolio_instances(),
call_user_func(array('data_portfolio_caller', 'supported_formats')),
'data_portfolio_caller', '', true, true)) {
$mform->addElement('header', 'notice', get_string('portfolionotfile', 'data') . ':');
$portfoliooptions[0] = get_string('none');
ksort($portfoliooptions);
$mform->addElement('select', 'portfolio', get_string('portfolio', 'portfolio'), $portfoliooptions);
}
}

This code adds a select option to the existing export form containing the available portfolio instances.

Then in the form handler:

if (array_key_exists('portfolio', $formdata) && !empty($formdata['portfolio'])) {
// fake portfolio callback stuff and redirect
$formdata['id'] = $cm->id;
$formdata['exporttype'] = 'csv'; // force for now
$url = portfolio_fake_add_url($formdata['portfolio'], 'data_portfolio_caller', '/mod/data/lib.php', $formdata);
redirect($url);
}

The portfolio_fake_add_url function returns the url that you need to redirect to, that would normally be the result of portfolio_add_button form being submitted.

==A few extra notes on the caller base class==

===protected $course===
There is a protected member variable, $course, that subclasses can set (with $this->set('course', $course); ).

portfolio_add_button tries to look for a course object at the point that it's called, by doing a global $COURSE which is hackish but mostly works. This is also used to build the navigation during the export process. If for some reason, your navigation doesn't include the current course, you can set it like this. You shouldn't need to in the majority of cases.

===serialization===
The caller object is stored in the database in serialized form. The portfolio code works around this by loading the class definitions and then serializing and unserializing the objects again. However, if you're storing any real objects in your caller class, you will need to do this as well. See the assignment implementation for how this done (using php5's __wakeup function):

public function __wakeup() {
require_once($this->assignmentfile);
$this->assignment = unserialize(serialize($this->assignment));
}

===setting portfolio internal===
Sometimes during prepare_package, you need to call functions in the libraries to render content as HTML that would normally also call portfolio_add_button. This will result in 'you already have an export active in this session' error - to get around this, do something like

define('PORTFOLIO_INTERNAL', true);
modulename_print_some_htmlcontent();

Development talk:DB layer 2.0 migration docs

2008-07-22T03:26:25Z

Nicolasconnault: /* Why / when do we '''have''' to use the params array in the "_sql" functions? */

* The developer reading this article MUST read XMLDB Documentation (at least Introduction + three first Developing section pages) otherwise he's going to ask plenty of questions :) It could be very good that the first line of this article be: "In order to understand this article you need to know how works the database abstraction layer in Moodle (previous to 2.0)".

* API link is broken (=> http://php.moodle.org/ "Server not found")

* We know that some changes need to be done on 2.0. But on what?
All these examples concerns the glossary module. It would be good to indicate it first. When I read it first, I didn't really know what was going to be explained in XMLDB changes. In fact all this section is about "Updating your upgrade.php file for Moodle 2.0". Conclusion: we really need a clear first sentence for the all "XMLDB changes" section. So we know what we are going to read. Once I knew what is about, all was very clear.

* If we don't make any change on DDL code we shouldn't make it as a section/block. The DDL section should be a note. If you remove this section don't forget to update: "Changes below are grouped into 3 main blocks" => "Changes below are grouped into 2 main blocks"

* DML section:
this is about to change all DML code, in all Moodle files? I think it miss an introduction line as well.

* Maybe you should use more the word: YOU should do that, YOU do this, YOU ... The developer will feel more guided.

Note: I wrote this comment on first read.

Note 2: I'm new as dev in Moodle, and I've just discover all the power of Moodle XMLDB reading. All the current XMLDB looks awesome, I don't even talk about the coming 2.0 :)

[[User:jerome mouneyrac|jerome mouneyrac]] 22:22, 22 May 2008 (CDT)

== random things ==

1 I added a marker re add_index or somesuch, is it really renamed add_key or is that just a copy/paste error? If it is correct, needs changing to emphasise that the function has really changed.

2 The section with nothing in it (DDL) should probably be removed :)

3 'Golden' and 'iron' makes no sense at all to me, nor did the explanation. Could we rename these in some way?

4 The example using named params should probably be changed to use better names for the params eg fn, ln - if you're using 'param1' and 'param2' you should use use the ordered one.

5 Some of the examples are unnecessarily wordy - for example, why put the array into a variable then call the function? It only has two elements, you can do it in the same line no problem.
:Wordiness in code samples is perfectly OK, Sam, but you're free to be more terse in your implementations ;) [[User:Nicolas Connault|Nicolas Connault]] 06:59, 3 June 2008 (CDT)
::Yes, but my point is that it makes it look bad for people who are having to change. It's like, OMG! this was one line before and now it's four! wtf I have to define a separate array every time I scratch my backside! why are you making my life a misery, you unfeeling b****rds!!! That kind of thing. :) I don't think it makes it easier to understand to show use of a mostly-unnecessary separate variable. [[User:sam marshall|sam marshall]] 06:24, 6 June 2008 (CDT)

6 Wow this new system is a fantastic improvement (except for the new IN function which is kind of nasty)

[[User:sam marshall|sam marshall]] 04:32, 3 June 2008 (CDT)

== Why / when do we '''have''' to use the params array in the "_sql" functions? ==

# I'm not sure I'm clear on what the advantages are in using the "params" array for the "*_sql" functions (e.g. get_records_sql). Also, the text says we '''must''' do it, but I see lots of examples in the already converted code where this hasn't been done. Is it really a '''must do''' or is it more of a '''can do'''?
:You must always use the params array WHEN you have params to pass ;) If the SQL doesn't contain dynamic parameters (subject to SQL injection), we don't need the params array. [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)
:Aha! So the purpose is to help make sure that dynamic data used in SQL searches is cleansed? Is that correct? [[User:Mike Churchward|Mike Churchward]] 21:31, 21 July 2008 (CDT)
::Yes, and to get rid of the troublesome stripslashes().
# The new recordset handling seems odd. We used to use the "rs_fetch_next_record($rs)" function, which as I understood it was optimized so that the entire db records contents weren't stored in the PHP array variable. Now we are replacing it with a "foreach ($rs as $record)". Doesn't that mean that the entire data query results are stored in a PHP variable again?
:The new recordset uses a new feature of PHP5, the Iterator interface. Foreach can then be used on the recordset. (See the bottom of [http://phplens.com/adodb/code.initialization.html the adodb doc] for more info). [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)
:Hmmm... Looks like I need to spend some time learning PHP5's features. So, in PHP5, the "foreach" construct unlocks a classes' interator functions if they exist? [[User:Mike Churchward|Mike Churchward]] 21:31, 21 July 2008 (CDT)
::Correct :) [[User:Nicolas Connault|Nicolas Connault]] 22:24, 21 July 2008 (CDT)

Development talk:DB layer 2.0 migration docs

2008-07-22T03:24:53Z

Nicolasconnault: Answering Mike

* The developer reading this article MUST read XMLDB Documentation (at least Introduction + three first Developing section pages) otherwise he's going to ask plenty of questions :) It could be very good that the first line of this article be: "In order to understand this article you need to know how works the database abstraction layer in Moodle (previous to 2.0)".

* API link is broken (=> http://php.moodle.org/ "Server not found")

* We know that some changes need to be done on 2.0. But on what?
All these examples concerns the glossary module. It would be good to indicate it first. When I read it first, I didn't really know what was going to be explained in XMLDB changes. In fact all this section is about "Updating your upgrade.php file for Moodle 2.0". Conclusion: we really need a clear first sentence for the all "XMLDB changes" section. So we know what we are going to read. Once I knew what is about, all was very clear.

* If we don't make any change on DDL code we shouldn't make it as a section/block. The DDL section should be a note. If you remove this section don't forget to update: "Changes below are grouped into 3 main blocks" => "Changes below are grouped into 2 main blocks"

* DML section:
this is about to change all DML code, in all Moodle files? I think it miss an introduction line as well.

* Maybe you should use more the word: YOU should do that, YOU do this, YOU ... The developer will feel more guided.

Note: I wrote this comment on first read.

Note 2: I'm new as dev in Moodle, and I've just discover all the power of Moodle XMLDB reading. All the current XMLDB looks awesome, I don't even talk about the coming 2.0 :)

[[User:jerome mouneyrac|jerome mouneyrac]] 22:22, 22 May 2008 (CDT)

== random things ==

1 I added a marker re add_index or somesuch, is it really renamed add_key or is that just a copy/paste error? If it is correct, needs changing to emphasise that the function has really changed.

2 The section with nothing in it (DDL) should probably be removed :)

3 'Golden' and 'iron' makes no sense at all to me, nor did the explanation. Could we rename these in some way?

4 The example using named params should probably be changed to use better names for the params eg fn, ln - if you're using 'param1' and 'param2' you should use use the ordered one.

5 Some of the examples are unnecessarily wordy - for example, why put the array into a variable then call the function? It only has two elements, you can do it in the same line no problem.
:Wordiness in code samples is perfectly OK, Sam, but you're free to be more terse in your implementations ;) [[User:Nicolas Connault|Nicolas Connault]] 06:59, 3 June 2008 (CDT)
::Yes, but my point is that it makes it look bad for people who are having to change. It's like, OMG! this was one line before and now it's four! wtf I have to define a separate array every time I scratch my backside! why are you making my life a misery, you unfeeling b****rds!!! That kind of thing. :) I don't think it makes it easier to understand to show use of a mostly-unnecessary separate variable. [[User:sam marshall|sam marshall]] 06:24, 6 June 2008 (CDT)

6 Wow this new system is a fantastic improvement (except for the new IN function which is kind of nasty)

[[User:sam marshall|sam marshall]] 04:32, 3 June 2008 (CDT)

== Why / when do we '''have''' to use the params array in the "_sql" functions? ==

# I'm not sure I'm clear on what the advantages are in using the "params" array for the "*_sql" functions (e.g. get_records_sql). Also, the text says we '''must''' do it, but I see lots of examples in the already converted code where this hasn't been done. Is it really a '''must do''' or is it more of a '''can do'''?
:You must always use the params array WHEN you have params to pass ;) If the SQL doesn't contain dynamic parameters (subject to SQL injection), we don't need the params array. [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)
:Aha! So the purpose is to help make sure that dynamic data used in SQL searches is cleansed? Is that correct? [[User:Mike Churchward|Mike Churchward]] 21:31, 21 July 2008 (CDT)
# The new recordset handling seems odd. We used to use the "rs_fetch_next_record($rs)" function, which as I understood it was optimized so that the entire db records contents weren't stored in the PHP array variable. Now we are replacing it with a "foreach ($rs as $record)". Doesn't that mean that the entire data query results are stored in a PHP variable again?
:The new recordset uses a new feature of PHP5, the Iterator interface. Foreach can then be used on the recordset. (See the bottom of [http://phplens.com/adodb/code.initialization.html the adodb doc] for more info). [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)
:Hmmm... Looks like I need to spend some time learning PHP5's features. So, in PHP5, the "foreach" construct unlocks a classes' interator functions if they exist? [[User:Mike Churchward|Mike Churchward]] 21:31, 21 July 2008 (CDT)
::Correct on both counts :) [[User:Nicolas Connault|Nicolas Connault]] 22:24, 21 July 2008 (CDT)

Development talk:DB layer 2.0 migration docs

2008-07-21T18:57:20Z

Nicolasconnault: Answers to Sam's pertinent questions

* The developer reading this article MUST read XMLDB Documentation (at least Introduction + three first Developing section pages) otherwise he's going to ask plenty of questions :) It could be very good that the first line of this article be: "In order to understand this article you need to know how works the database abstraction layer in Moodle (previous to 2.0)".

* API link is broken (=> http://php.moodle.org/ "Server not found")

* We know that some changes need to be done on 2.0. But on what?
All these examples concerns the glossary module. It would be good to indicate it first. When I read it first, I didn't really know what was going to be explained in XMLDB changes. In fact all this section is about "Updating your upgrade.php file for Moodle 2.0". Conclusion: we really need a clear first sentence for the all "XMLDB changes" section. So we know what we are going to read. Once I knew what is about, all was very clear.

* If we don't make any change on DDL code we shouldn't make it as a section/block. The DDL section should be a note. If you remove this section don't forget to update: "Changes below are grouped into 3 main blocks" => "Changes below are grouped into 2 main blocks"

* DML section:
this is about to change all DML code, in all Moodle files? I think it miss an introduction line as well.

* Maybe you should use more the word: YOU should do that, YOU do this, YOU ... The developer will feel more guided.

Note: I wrote this comment on first read.

Note 2: I'm new as dev in Moodle, and I've just discover all the power of Moodle XMLDB reading. All the current XMLDB looks awesome, I don't even talk about the coming 2.0 :)

[[User:jerome mouneyrac|jerome mouneyrac]] 22:22, 22 May 2008 (CDT)

== random things ==

1 I added a marker re add_index or somesuch, is it really renamed add_key or is that just a copy/paste error? If it is correct, needs changing to emphasise that the function has really changed.

2 The section with nothing in it (DDL) should probably be removed :)

3 'Golden' and 'iron' makes no sense at all to me, nor did the explanation. Could we rename these in some way?

4 The example using named params should probably be changed to use better names for the params eg fn, ln - if you're using 'param1' and 'param2' you should use use the ordered one.

5 Some of the examples are unnecessarily wordy - for example, why put the array into a variable then call the function? It only has two elements, you can do it in the same line no problem.
:Wordiness in code samples is perfectly OK, Sam, but you're free to be more terse in your implementations ;) [[User:Nicolas Connault|Nicolas Connault]] 06:59, 3 June 2008 (CDT)
::Yes, but my point is that it makes it look bad for people who are having to change. It's like, OMG! this was one line before and now it's four! wtf I have to define a separate array every time I scratch my backside! why are you making my life a misery, you unfeeling b****rds!!! That kind of thing. :) I don't think it makes it easier to understand to show use of a mostly-unnecessary separate variable. [[User:sam marshall|sam marshall]] 06:24, 6 June 2008 (CDT)

6 Wow this new system is a fantastic improvement (except for the new IN function which is kind of nasty)

[[User:sam marshall|sam marshall]] 04:32, 3 June 2008 (CDT)

== Why / when do we '''have''' to use the params array in the "_sql" functions? ==

# I'm not sure I'm clear on what the advantages are in using the "params" array for the "*_sql" functions (e.g. get_records_sql). Also, the text says we '''must''' do it, but I see lots of examples in the already converted code where this hasn't been done. Is it really a '''must do''' or is it more of a '''can do'''?
:You must always use the params array WHEN you have params to pass ;) If the SQL doesn't contain dynamic parameters (subject to SQL injection), we don't need the params array. [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)
# The new recordset handling seems odd. We used to use the "rs_fetch_next_record($rs)" function, which as I understood it was optimized so that the entire db records contents weren't stored in the PHP array variable. Now we are replacing it with a "foreach ($rs as $record)". Doesn't that mean that the entire data query results are stored in a PHP variable again?
:The new recordset uses a new feature of PHP5, the Iterator interface. Foreach can then be used on the recordset. (See the bottom of [http://phplens.com/adodb/code.initialization.html the adodb doc] for more info). [[User:Nicolas Connault|Nicolas Connault]] 13:57, 21 July 2008 (CDT)

Development:Unit tests

2008-07-04T12:40:03Z

Nicolasconnault: Instructions for Unit Testing in 2.0

{{Moodle 1.7}}Location: ''Administration > Reports > Unit tests''

The purpose of Unit Tests is to evaluate the individual parts of a program (functions, and methods of classes) to make sure that each element individually does the right thing. Unit Tests can be one of the first steps in a quality control process for developing or tweaking Moodle code. The next steps will involve other forms of testing to ensure that these different parts work together properly.

The unit testing framework is based on the [http://www.lastcraft.com/simple_test.php SimpleTest] framework. It was incorporated into Moodle by Nick Freear and Tim Hunt from [http://www.open.ac.uk/ The Open University].

== Running the unit tests in Moodle ==

=== Running the basic tests ===

# Log in with an admin account.
# Go to the admin screen.
# Click on the '''Reports''' link near the bottom of the page.
# Click on the '''Run the unit tests''' link.
# Wait for the tests to run.

=== Options for running the tests ===

At the bottom of the tests page, there is form that lets you adjust the options used when running the tests.

==== Show passes as well as fails ====

Normally, only details of the tests that have failed are printed. Turning on this options shows details of all the passes too.

==== Show the search for test files ====

The tests to run are found automatically be searching the codebase for files whose names match '''test*.php''' in directories called '''simpletest'''. Turning on this option will print a list of the folders searched and the test files found. This is sometimes useful for debugging.

This option is particularly useful when one of your test files has a syntax error. When this happens, you sometimes just get a blank page with no error message. Turning on the show search option lets you see which test file it was that gave the error. If necessary, you can enable this option manually by adding "showsearch=1" to the end of the URL.

==== Run a thorough test (may be slow) ====

If you turn on this option, then as well as looking for files called '''test*.php''', the search also looks for files called '''slowtest*.php'''.

To be useful, the full test run should find most bugs, but not take too long to complete. So if you have very, very detailed tests of an area of the code, it may be better to select a subset for everday testing, and only use the more detailed tests when a bug is reported, or you are doing new development in that area of the code.

This option is most useful when combined with the next option.

==== Only run tests in ====

Normally, tests from all parts of the codebase are run. However, when you are just doing development of one part of the code, that is a waste of time. You can type the name of a folder (for example '''mod/quiz''') or a particular test file (for example '''lib/simpletest/testdatalib.php''') and then only those tests will be run.

[[Image:RunOnlyTheseTests.png|right]] Instead of typing a path into this box, there is an easier way. Whenever a pass or fail is displayed, the name of the test file is printed. Each section of the path name is a link to run only the tests in that folder or file.

== Writing new tests ==

As an example, suppose we wanted to start writing tests for the functions in the file 'question/editlib.php'.

=== Where to put the tests ===

If you have read the first half of this page and were paying attention, you can probably work out that you should create a folder called '''question/testeditlib''', and create a file in there called something like '''testeditlib.php'''.

The skeleton of this file should look like:

<pre><?php
/**
* Unit tests for (some of) question/editlib.php.
*
* @copyright © 2006 The Open University
* @author T.J.Hunt@open.ac.uk
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
* @package question
*/

/** */
require_once(dirname(__FILE__) . '/../../config.php');

global $CFG;
require_once($CFG->libdir . '/simpletestlib.php'); // Include the test libraries
require_once($CFG->dirroot . '/question/editlib.php'); // Include the code to test

/** This class contains the test cases for the functions in editlib.php. */
class question_editlib_test extends UnitTestCase {
function test_get_default_question_category() {
// Do the test here/
}
}
?></pre>

That is, you have a class called something_test, and in that class you have lots of methods called test_something. Normally, you have one test method for each function you want to test, and you may as well called the test method '''test_name_of_function_being_tested'''.

=== Inside a test function ===

The inside of a test function tyically looks like this:

<pre>function test_get_default_question_category() {
// Set things up in preparation for the test.

// Call the function you want to test.

// Check that the result is what you expected.
}</pre>

For example:

=== Test data ===

TODO

=== setUp and tearDown methods ===

If all your test cases relate to the same area of code, and so need the same set of test data, then you can create a method called <code>setUp()</code> that sets up the test data. If present, this method will be called before each test method. You can write a matching <code>tearDown()</code> method if there is any clean-up that needs to be done after each test case has run.

If you have some test test cases the need one sort of setup, and some other test cases that need a different setup, consider splitting your tests into two separate classes, each with its own <code>setUp()</code> method.

=== Further information ===

The simpletest documentation is at: http://simpletest.sourceforge.net/.

== Changes to your existing code to make it work with unit testing ==

When code is being tested, it gets included from inside one of the simpletest library function. If the code is expecting to be run directly (for example, if it is a view.php or index.php function), you are likely to get errors because that expectation is no longer true.

=== Include paths ===

Includes like

<pre>require_once('../../config.php'); // Won't work.</pre>

won't work. Instead, the more robust option is

<pre>require_once(dirname(__FILE__) . '/../../config.php'); // Do this.</pre>

=== Access to global variables ===

Because your code was included from within a function, you can't access global variables until you have done a global statement.

<pre>require_once(dirname(__FILE__) . '/../../config.php');
require_once($CFG->libdir . '/moodlelib.php'); // Won't work.</pre>

<pre>require_once(dirname(__FILE__) . '/../../config.php');

global $CFG; // You need this.
require_once($CFG->libdir . '/moodlelib.php'); // Will work now.</pre>

== Unit testing in 2.0 ==
{{Moodle 2.0}}
With the Objectification of the Database libraries in Moodle 2.0, new and better approaches to Unit testing can be used. Here is a sample of a simple test case: (in course/simpletest)

require_once($CFG->dirroot . '/course/lib.php');

global $DB;
Mock::generate(get_class($DB), 'mockDB');

class courselib_test extends UnitTestCase {
var $realDB;

function setUp() {
global $DB;
$this->realDB = clone($DB);
$DB = new mockDB();
}

function tearDown() {
global $DB;
$DB = $this->realDB;
}

function testMoveSection() {
global $DB;
$course = new stdClass();
$course->id = 1;

$sections = array();
for ($i = 1; $i < 11; $i++) {
$sections[$i] = new stdClass();
$sections[$i]->id = $i;
$sections[$i]->section = $i - 1;
}

$DB->expectOnce('get_records', array('course_sections', array('course' => $course->id)));
$DB->setReturnValue('get_records', $sections);
$this->assertFalse(move_section($course, 2, 3));
}
}

== Further reading about unit testing ==

The best book I know about unit testing is '''Pragmatic Unit Testing in Java with JUnit'' by Andrew Hunt (no relation) and David Thomas. I know, this book is not called Pragmatic Unit Testing in PHP with SimpleTest. However, it is an excellent book - short, to the point, and very practical. Most of what it says is not specific to Java and JUnit and it is obvious how to apply it in our testing setup.

[[Category:Developer|Unit tests]]
[[Category:Report]]

Development:Repository File Picker

2008-07-04T06:23:57Z

Nicolasconnault: fixing typos

'''Functional Specification Revisions:'''
:0.1 - 30/06/2008 - Jerome Mouneyrac - Draft Version

'''Audience:'''
:Developer/QA tester

'''Status:'''
:not implemented ([http://tracker.moodle.org/browse/MDL-15348 tracker issue])

'''Related Documents:'''
*[[Development:Repository API| Repository API]]
*[[Development:Repository_Plugins| Repository Plugins]]
*[[Development:Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
<br />
==Introduction==
This document is about functional specification for the file picker<br />

==Requirements==
* be able to select a file in a external repository (in order to be associated with an activity, resource, user profil,...)

==User Interface==

=== Javascript enabled ===

==== Moodleform ====
Following a picture of the Moodleform user interface: one button to display or undisplay the file picker, the read-only filename text field, and the ajax file picker.

[[Image:Ajax_file_picker.png]]
<br /><br />

==== HTML editor ====
Following a mock screen of the HTML editor:<br /><br />
[[Image:Htmleditor.png]]
<br /><br />

=== Javascript disabled ===
As there is no HTML editor when javascript is disabled, we look to the Moodleform case only:<br />
on every page when a user wishes to add a file, he clicks on the Add file button. A new page with the file picker is displayed in the same window. The file picker user interface should be similar to the one with Javascript, except that every user action will refresh the entire page. Once the user clicks on Select button, he's redirected to the previous page. The previously entered data (by the user) are still displayed.

===Mock file picker screenshots===
When you first call up the file picker and choose a repository, you might be asked to log in (if saving of passwords is not allowed):

[[Image:Filepicker_login.jpg]]

Browsing files could look something like this:

[[Image:Filepicker_browser.jpg]]

And you can also search:

[[Image:Filepicker_search.jpg]]

==Use Cases==
TODO: please write them
===UC004-1 Select a file from an Moodleform===
====Base scenario====
#...

====Pre conditions====
:- repository plugin is enabled and has been setup in the administration

===UC004-2 Select a file from HTML editor===
===UC004-3 Select a file when javascript is disabled===

Development:File API

2008-07-03T15:04:37Z

Nicolasconnault: Comment on filesize calculation

This page outlines the current thinking about implementing file storage and access in Moodle 2.0. It's a SPECIFICATION UNDER CONSTRUCTION!

The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions, problems to report or major changes to suggest please add them to the [[Development_talk:File_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the main spec before we start development.

==Objectives==

# Allow files to be added directly into Moodle (as we do now)
# Remember where files came from
# Give modules control over the access to files using capabilities and other local rules
# Consistent and simple approach for ALL file handling throughout Moodle

==Overview==

The File API is a core set of interfaces that all Moodle code will use to:
# store files within Moodle
# display files to Moodle users

It applies only to "user" files. It will NOT apply to local files and caches created by Moodle such as these directories in dataroot: temp, lang, cache, environment, filter, rss, search, sessions, upgradelogs etc

The API will be split into several independent parts:
# File serving API
## file.php
## pluginfile.php
## userfile.php
## rssfile.php
# File storage API
## optional access control
## optional repo sync
# File management API
## File browsing
## File linking (editor integration)
## Upload from repository

==File serving API==
Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

===file.php===
Serves course files.

Implements basic file access. Ideally only images and files linked from course sections should be there, no XSS protection required - we expect javascript, sw, etc. there, no way to make it "secure". The access control is not critical any more if we move most of the files into modules

The file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files would be stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'content', 'itemid'=>0)</code>

===pluginfile.php===
(aka modfile.php)
Sends module, block, question files.
* modules decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* only internal links to selected areas are supported - you can link images in summary area, but not the assignment submissions

Absolute file links need to be rewritten if html editing allowed in module. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/summary/image.jpg, it is converted back to internal links before saving.

::Can the distinct file areas supported by one plugin be declared somehow in order add some information about them? For example, I think it can be interesting to declare:
::* assignment_summary:
::** relpath='summary'
::** userdata=false
::** anotherproperty=anothervalue
::* assignment_submission:
::** relpath='submission/@@USERID@@'
::** userdata=false
::** anotherproperty=anothervalue
::* and so on...
::And then, when the editor "receives" one "assignment_summary" areaname, if knows what to show and so on? Also that info could be useful to know, in backup & restore if some fileareas have to be processed or no (userdata=false). Or also, when reconstructing the links (str_replace() above). And will cause to have a well defined list of fileareas by module, instead of coding them in a free way (prone to errors). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 16:35, 28 June 2008 (CDT)

::Something like this will be part of file management API, hardcoding this in file storage would make it less flexible imo [[User:Skodak|Skodak]]

::Yup, yup. Storage doesn't know anything but get/put files (nothing else). It's part of management, absolutely. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:21, 29 June 2008 (CDT)

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

====assignment example====

/pluginfile.php/assignmentcontextid/summary/someimage.jpg
/pluginfile.php/assignmentcontextid/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/extra/allsubmissionfiles.zip

::Uhm... all those files together? What's going to differentiate the "submission" path in the example above from the "summary" path? Is it supposed that the editor, or the filemanager won't allow , for example to pick-up one file from the "submission" area to be used in the summary of one assignment and only the "summary" area will be showed? That means multiple file managers by context and it's against the clean "one file manager per context" agreed below [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 21:28, 26 June 2008 (CDT)
::Yes Eloy, the different areas (summary, submission) etc. have different uses, different access control. There are two types of file manager - the two pane file manager which lists all contexts+areas user may access, and minimalistic manager in html editor which shows only subset of areas from current plugin (because you can not link anything else).

====scorm example====

/pluginfile.php/scormcontextid/summary/someimage.jpg
/pluginfile.php/scormcontextid/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems. The lifetime should be adjustable in module settings.

====quiz example====

pluginfile.php/quizcontextid/summary/niceimage.jpg
pluginfile.php/quizcontextid/report/type/export.ods

====questions example====

pluginfile.php/SYSCONTEXTID/question/questionid/file.jpg

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'filearea'=>'blog', 'itemid'=>$blogentryid)</code>

====backup example====
It would be nice to have some special protection of backup files - new capabilities for backup file download, upload. Backups contain a lot of personal info, we could block restoring of backups from other sites too.

/pluginfile.php/coursecontextid/backup/backupfile.zip

Internally stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'backup', 'itemid'=>0)</code>

===userfile.php===
Personal file storage, intended as an online storage of work in progress like assignments before the submission.
* read/write own files only for now
* option to share with others later
* personal "websites" will not be supported (security)

/userfile.php/userid/dir/dir/filename.ext

===rssfile.php===
Replaces rss/file.php which is kept only for backwards compatibility.
RSS files should not require sessions/cookies, URLs should contain some sort of security token/key.
Internally the files may be stored in database or together with other files.
Performance improvements - we should support both Etag (cool) and Last-Modified (more used), when we receive If-None-Match/If-Modified-Since => 304

/rssfile.php/contextid/any/parameters/module/wants/rss.xml
/rssfile.php/SYSCONTEXTID/blog/userid/rss.xml

Again modules and plugins decide what gets sent to user.

===Temporary files===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* zipping/unzipping
* processing by executable files (latex, mimetex)

Ideally these files should never use utf-8 (which is a major problem for zipping at the moment).
Proposed new sha1 based file storage is not suitable both for performance and technical reasons.

===Legacy file storage and serving===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

==File storage API==
Modules in general work only with local Moodle files. One of the major reason is performance when accessing external repository files. It will be possible to use repositories instead of file uploading and also to keep local files synced with external repository.

File contents are stored in moodledata/filepool indexed using SHA1 hashes instead of file names; file names, relative paths and other metadata will be stored in file(_xxx) database tables. This should be fully abstracted so that modules do not actually know where the files are located. When storing files the content is sent as string or file handle, when reading content it is returned as file handle.

===files table===

This table contains one entry for every file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary.

note: plural used because file is a reserved word

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|
|The sha1 hash of content.

|-
|'''contextid'''
|int(10)
|
|The context id defined in context table - identifies the instance of plugin owning the file.

|-
|filearea
|varchar(50)
|
|Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.

|-
|itemid
|int(10)
|
|Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)

|-
|filepath
|text
|
|relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this

|-
|filename
|varchar(255)
|
|The full Unicode name of this file (case sensitive)

|-
|filesize
|int(10)
|
|size of file - bytes

|-
|mimetype
|varchar(100)
|NULL
|type of file

|-
|'''userid'''
|int(10)
|NULL
|Optional - general user id field - meaning depending on plugin

|-
|timecreated
|int(10)
|
|The time this file was created

|-
|timemodified
|int(10)
|
|The last time the file was modified
|}

index on "contextid, filearea, itemid" and "sha1hash"

Plugin type is not specified because it is derived from contextid, items like blog that do not have own context will use own filearea usually from systemcontextid.

::Perhpas we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:54, 29 June 2008 (CDT)

::Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 18:54, 29 June 2008 (CDT)

=== files_cleanup table ===

This table contains candidates for deletion from the file pool. Files are not deleted immediately, cron uses the files_cleanup table, verifies the file is not used any more and deletes it from pool. Reasons for cron clean-up are performance and prevention of collision - there could be a problem with concurrent uploads and deletes, we will probably need to add some table-based locking during the clean-up.

We might add an extra script that does deep validation of pool area - report missing files, report orphaned files, content not matching the sha1 filename, etc. - this would very very time consuming.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|

|}

=== files_metadata table ===

This table contains extra metadata about files. Repositories could provide this, or it could be manually edited in the local copy.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''name'''
|varchar(255)
|
|The name of extra metadata

|-
|value
|text
|
|Value

|}

===files_acl table===

This table describes optional ACL for file. This is not required in majority of cases, modules usually hardcode the file access logic, course files should not be used much any more.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|The file we are defining access for

|-
|'''contextid'''
|int(10)
|
|The context where this file is being published

|-
|'''capability'''
|text
|
|The capability that is required to see this file.
|}

====acl notes====
* this is missing some concept similar to '''user/group/others''', for example in case of user files typical user can not assign permissions or view them - this becomes useless there
* it is more important to synchronise the availability of file link and the file itself - having link pointing to inaccessible file or file which is accessible when not wanted are both problems
* browser/proxy caching works against us here - "secret" files should not be cached

===files_sync table===

This table contains information on how to synchronise data with repositories. Data would be synchronised from cron.php or on demand from file manager. The sync would be one way only (repository-->local file).

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''repositoryid'''
|int(10)
|
|The repository instance this is associated with, see [[Development:Repository_API]]

|-
|updates
|int(10)
|
|Specifies the update schedule (0 = none, 1 = on demand, other = some period in seconds)

|-
|repositorypath
|text
|
|The full path to the original file on the repository

|-
|timeimportfirst
|int(10)
|
|The first time this file was imported into Moodle

|-
|timeimportlast
|int(10)
|
|The most recent time that this file was imported into Moodle
|}

===File content storage===
Originally the file storage hierarchy contained a lot of metadata including userids, entry ids, filenames, etc. The file content will now be stored separately from file metadata. It must supports utf8 on all platforms.

File storing:
# calculate SHA1 hash of content
# check if file with SHA1 name exists, if not add the file to file pool
# remove SHA1 from list of deleted files if found there
# store file in ''file'' table, use SHA1 as file pool identifier

File reading:
#fetch file record from 'file' table - probably using file id or combination of contextid+instanceid
#fetch content of file

File deleting:
#delete record from ''file'' table, remember file SHA1
#store the deleted SHA1 in deleted files table, do not remove the physical file yet
#wait for cron cleanup script to actually delete the file named SHA1 (proper table locking needed to prevent race conditions when adding/deleting files)

====File pool details====
located in $CFG->dataroot/filepool/, all files can not be stored in one directory due to OS limitations, it uses 3 levels based on first three characters of sha1 hash. It is unlikely that there will be thousands of files with the same first 3 chars in sha1 hash of their content.

This type of storage saves a lot of disk space when storing multiple copies of the same large file. It can also help substantially when synchronising data with external repositories. Another benefit is we can detect inconsistencies in file content.

File read performance is similar to previous code, file write performance will be slower - due to hashing and extra database access.

dataroot
/filepool
/00
/01
...
/'''23'''
/00
/01
...
/'''1e'''
/00
/01
...
/'''2d'''
/231e2dc421be4fcd0172e5afceea3970e2f3d940.jpg
...
/fe
/ff

==File management API==

This section describes following:
#file manager
#integration with html editor
#interactions with repos

===File manager===
Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

Originally there was a single file tree for each course. We need to fully separate each module/block from the course files and there might be also independent file areas in modules (ex: module introduction, content files, submissions, post attachments). File area may be defined as a small tree where we can use relative paths. These file areas are hanging from the branches of the context tree (this needs a picture).

===Integration with htmleditor===
Html editor should be able to browse only relevant files - for example when editing resource introduction only images from the file area of that resource should be available; when editing html resource page only the content area images should be listed.

There are several problems here:
# when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area
# we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile/intro@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.

Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.

===Interactions with repos===
Repositories may serve as a replacement for file uploading. They may be also used to synchronise files between courses. The repo option should be available whenever there is a file upload field, sometimes with extra "keep synchronised" option (this would not make sense for stuff like assignment submissions).

==Upgrade, migration and backwards compatibility==
It is going to be a pain again like DML/DDL ;-)

===Code backwards compatibility===
0% backwards compatibility related to file storage. New objects will be mandatory to use. Old $CFG->dataroot/$courseid/ will be empty, $CFG->dataroot/blog/ too, etc.

===Content backwards compatibility===
Means existing courses should not loose images, flash, etc. Though some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Migration of content===
* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new are, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be many outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

===Moving files to files table and file pool===
The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.
Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some ppl use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

==Backup/restore changes==
File handling in backups needs to be fully rewritten - list of files in xml + pool of sha1 named files with contents. This solves the utf-8 trouble here, yay!!

==Quotas==
File size will be stored in files table, we can use simple queries to find out how much space is used, however this may not be accurate because the sha1 hash based storage eliminates duplicate files.
::If sha1 string is stored in the files table (non-unique), then it can be used to detect duplicates within the files table and only counting their size once. To avoid having to search this often, we could do it periodically and store the filesize with each file record (only the first of the duplicates gets a filesize, the others get 0). [[User:Nicolas Connault|Nicolas Connault]]

*total course files - find out all contexts used in course, query files table with contextid IN ($listofcontexts)
*module files - find module context and calculate space per file area
*user files quota - inside the personal area only, counting all attachments in all mods might take a while
We could also divide the file size by number of instances that are using it, this might be considered more accurate in some scenarios.

==Other==
*antivirus scanning + upload manager rewrite/integration with forms lib
*zip compression and extraction

==Major problems==
List of hard to solve prolbems

===unicode zip support===
Unicode chars in zip files uploaded by teachers - unfortunately there is no 100% solution that will work for anybody because most zip programs do not support unicode, it is usually ''garbage in/garbage out'' which works in some cases only

Latest WinZIP 11.2 and Total Commander seem to support some very limited form of utf-8 encodings of file names. I managed to create a zip file in Windows (Czech locale) and extract them in linux with native PHP zip functions, the only step I needed to add was conversion cp852(DOS charset for Czech locale) -> UTF-8. The native windows zipping did not work for me though, because it does some different borking of charsets.

In any case it seems likely that native PHP support in PHP 5.2.x should be better than current pclzip or infozip binary.

===empty directories===
Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.

===file overwriting===
Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.

== Some little comments to be considered (to avoid forgetting them) ==

* each context will have its own "file manager"
* separate "file manager context" files (FMF) and "internal context" (ICF) files (current modedit files, submissions, attachements...)
* /pluginfile.php/SYSCONTEXTID/{blog|question} and so... will have own FMF too? Or only ICF ?
* Way to copy between contexts
* Links = -1 for them
* Deletion strategy (locks, quarantine status...)
* include support for quotas per user, per course, etc
* upgrade process should be interruptable (like the unicode upgrade) so it can be stopped/restarted any time

==Justin's thinking out loud==

I'm actually working on implementing this along with extending an existing Alfresco integration to work together with the whole File / Repository system and I wanted to get some of my comments and thoughts in here for feedback. Go easy on me. =)

So far I've only got one that I'd like to solicit some feedback on (BTW, if this would be better suited to a forum discussion, let me know):

===Not storing the full ''filepath'' with each entry in the '''file''' table===
*For browsing a directory structure, determining things like child directories or a parent directory given a filepath requires a lot of extraneous coding in PHP. I think it might be better served to create a new '''file_directory''' table, storing only a directory name, and reference to a parent directory record. The benefits here are that we're storing a lot of duplicate text field values in the '''file''' table and browsing through the file picker for local files doesn't require a lot of PHP overhead to calculate links to parent / child directories.
*Given that file permissions are no longer calculated using structured file paths, using the complete, full, path to a given file would most likely never be needed.

*The '''repositorypath''' field in the '''repository_sync''' table still makes sense, though.

*The '''file_directory''' table:

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''parent'''
|int(10)
|
|ID of directory that this record is a child of.

|-
|'''directoryname'''
|varchar(255)
|
|The actual name of this directory.
|}

skodak: filepath is stored in files table - its root is the corresponding filearea, the file manager will use the context tree to find all plugins/courses and ask them to return the list of areas with all those small branches inside it

==See also==

* [[Development:Repository API]]
* [[Development:Portfolio API]]

Development:File API

2008-07-03T14:58:07Z

Nicolasconnault: fixing grammar

This page outlines the current thinking about implementing file storage and access in Moodle 2.0. It's a SPECIFICATION UNDER CONSTRUCTION!

The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions, problems to report or major changes to suggest please add them to the [[Development_talk:File_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the main spec before we start development.

==Objectives==

# Allow files to be added directly into Moodle (as we do now)
# Remember where files came from
# Give modules control over the access to files using capabilities and other local rules
# Consistent and simple approach for ALL file handling throughout Moodle

==Overview==

The File API is a core set of interfaces that all Moodle code will use to:
# store files within Moodle
# display files to Moodle users

It applies only to "user" files. It will NOT apply to local files and caches created by Moodle such as these directories in dataroot: temp, lang, cache, environment, filter, rss, search, sessions, upgradelogs etc

The API will be split into several independent parts:
# File serving API
## file.php
## pluginfile.php
## userfile.php
## rssfile.php
# File storage API
## optional access control
## optional repo sync
# File management API
## File browsing
## File linking (editor integration)
## Upload from repository

==File serving API==
Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

===file.php===
Serves course files.

Implements basic file access. Ideally only images and files linked from course sections should be there, no XSS protection required - we expect javascript, sw, etc. there, no way to make it "secure". The access control is not critical any more if we move most of the files into modules

The file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files would be stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'content', 'itemid'=>0)</code>

===pluginfile.php===
(aka modfile.php)
Sends module, block, question files.
* modules decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* only internal links to selected areas are supported - you can link images in summary area, but not the assignment submissions

Absolute file links need to be rewritten if html editing allowed in module. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/summary/image.jpg, it is converted back to internal links before saving.

::Can the distinct file areas supported by one plugin be declared somehow in order add some information about them? For example, I think it can be interesting to declare:
::* assignment_summary:
::** relpath='summary'
::** userdata=false
::** anotherproperty=anothervalue
::* assignment_submission:
::** relpath='submission/@@USERID@@'
::** userdata=false
::** anotherproperty=anothervalue
::* and so on...
::And then, when the editor "receives" one "assignment_summary" areaname, if knows what to show and so on? Also that info could be useful to know, in backup & restore if some fileareas have to be processed or no (userdata=false). Or also, when reconstructing the links (str_replace() above). And will cause to have a well defined list of fileareas by module, instead of coding them in a free way (prone to errors). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 16:35, 28 June 2008 (CDT)

::Something like this will be part of file management API, hardcoding this in file storage would make it less flexible imo [[User:Skodak|Skodak]]

::Yup, yup. Storage doesn't know anything but get/put files (nothing else). It's part of management, absolutely. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:21, 29 June 2008 (CDT)

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

====assignment example====

/pluginfile.php/assignmentcontextid/summary/someimage.jpg
/pluginfile.php/assignmentcontextid/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/extra/allsubmissionfiles.zip

::Uhm... all those files together? What's going to differentiate the "submission" path in the example above from the "summary" path? Is it supposed that the editor, or the filemanager won't allow , for example to pick-up one file from the "submission" area to be used in the summary of one assignment and only the "summary" area will be showed? That means multiple file managers by context and it's against the clean "one file manager per context" agreed below [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 21:28, 26 June 2008 (CDT)
::Yes Eloy, the different areas (summary, submission) etc. have different uses, different access control. There are two types of file manager - the two pane file manager which lists all contexts+areas user may access, and minimalistic manager in html editor which shows only subset of areas from current plugin (because you can not link anything else).

====scorm example====

/pluginfile.php/scormcontextid/summary/someimage.jpg
/pluginfile.php/scormcontextid/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems. The lifetime should be adjustable in module settings.

====quiz example====

pluginfile.php/quizcontextid/summary/niceimage.jpg
pluginfile.php/quizcontextid/report/type/export.ods

====questions example====

pluginfile.php/SYSCONTEXTID/question/questionid/file.jpg

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'filearea'=>'blog', 'itemid'=>$blogentryid)</code>

====backup example====
It would be nice to have some special protection of backup files - new capabilities for backup file download, upload. Backups contain a lot of personal info, we could block restoring of backups from other sites too.

/pluginfile.php/coursecontextid/backup/backupfile.zip

Internally stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'backup', 'itemid'=>0)</code>

===userfile.php===
Personal file storage, intended as an online storage of work in progress like assignments before the submission.
* read/write own files only for now
* option to share with others later
* personal "websites" will not be supported (security)

/userfile.php/userid/dir/dir/filename.ext

===rssfile.php===
Replaces rss/file.php which is kept only for backwards compatibility.
RSS files should not require sessions/cookies, URLs should contain some sort of security token/key.
Internally the files may be stored in database or together with other files.
Performance improvements - we should support both Etag (cool) and Last-Modified (more used), when we receive If-None-Match/If-Modified-Since => 304

/rssfile.php/contextid/any/parameters/module/wants/rss.xml
/rssfile.php/SYSCONTEXTID/blog/userid/rss.xml

Again modules and plugins decide what gets sent to user.

===Temporary files===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* zipping/unzipping
* processing by executable files (latex, mimetex)

Ideally these files should never use utf-8 (which is a major problem for zipping at the moment).
Proposed new sha1 based file storage is not suitable both for performance and technical reasons.

===Legacy file storage and serving===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

==File storage API==
Modules in general work only with local Moodle files. One of the major reason is performance when accessing external repository files. It will be possible to use repositories instead of file uploading and also to keep local files synced with external repository.

File contents are stored in moodledata/filepool indexed using SHA1 hashes instead of file names; file names, relative paths and other metadata will be stored in file(_xxx) database tables. This should be fully abstracted so that modules do not actually know where the files are located. When storing files the content is sent as string or file handle, when reading content it is returned as file handle.

===files table===

This table contains one entry for every file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary.

note: plural used because file is a reserved word

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|
|The sha1 hash of content.

|-
|'''contextid'''
|int(10)
|
|The context id defined in context table - identifies the instance of plugin owning the file.

|-
|filearea
|varchar(50)
|
|Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.

|-
|itemid
|int(10)
|
|Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)

|-
|filepath
|text
|
|relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this

|-
|filename
|varchar(255)
|
|The full Unicode name of this file (case sensitive)

|-
|filesize
|int(10)
|
|size of file - bytes

|-
|mimetype
|varchar(100)
|NULL
|type of file

|-
|'''userid'''
|int(10)
|NULL
|Optional - general user id field - meaning depending on plugin

|-
|timecreated
|int(10)
|
|The time this file was created

|-
|timemodified
|int(10)
|
|The last time the file was modified
|}

index on "contextid, filearea, itemid" and "sha1hash"

Plugin type is not specified because it is derived from contextid, items like blog that do not have own context will use own filearea usually from systemcontextid.

::Perhpas we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:54, 29 June 2008 (CDT)

::Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 18:54, 29 June 2008 (CDT)

=== files_cleanup table ===

This table contains candidates for deletion from the file pool. Files are not deleted immediately, cron uses the files_cleanup table, verifies the file is not used any more and deletes it from pool. Reasons for cron clean-up are performance and prevention of collision - there could be a problem with concurrent uploads and deletes, we will probably need to add some table-based locking during the clean-up.

We might add an extra script that does deep validation of pool area - report missing files, report orphaned files, content not matching the sha1 filename, etc. - this would very very time consuming.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|

|}

=== files_metadata table ===

This table contains extra metadata about files. Repositories could provide this, or it could be manually edited in the local copy.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''name'''
|varchar(255)
|
|The name of extra metadata

|-
|value
|text
|
|Value

|}

===files_acl table===

This table describes optional ACL for file. This is not required in majority of cases, modules usually hardcode the file access logic, course files should not be used much any more.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|The file we are defining access for

|-
|'''contextid'''
|int(10)
|
|The context where this file is being published

|-
|'''capability'''
|text
|
|The capability that is required to see this file.
|}

====acl notes====
* this is missing some concept similar to '''user/group/others''', for example in case of user files typical user can not assign permissions or view them - this becomes useless there
* it is more important to synchronise the availability of file link and the file itself - having link pointing to inaccessible file or file which is accessible when not wanted are both problems
* browser/proxy caching works against us here - "secret" files should not be cached

===files_sync table===

This table contains information on how to synchronise data with repositories. Data would be synchronised from cron.php or on demand from file manager. The sync would be one way only (repository-->local file).

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''repositoryid'''
|int(10)
|
|The repository instance this is associated with, see [[Development:Repository_API]]

|-
|updates
|int(10)
|
|Specifies the update schedule (0 = none, 1 = on demand, other = some period in seconds)

|-
|repositorypath
|text
|
|The full path to the original file on the repository

|-
|timeimportfirst
|int(10)
|
|The first time this file was imported into Moodle

|-
|timeimportlast
|int(10)
|
|The most recent time that this file was imported into Moodle
|}

===File content storage===
Originally the file storage hierarchy contained a lot of metadata including userids, entry ids, filenames, etc. The file content will now be stored separately from file metadata. It must supports utf8 on all platforms.

File storing:
# calculate SHA1 hash of content
# check if file with SHA1 name exists, if not add the file to file pool
# remove SHA1 from list of deleted files if found there
# store file in ''file'' table, use SHA1 as file pool identifier

File reading:
#fetch file record from 'file' table - probably using file id or combination of contextid+instanceid
#fetch content of file

File deleting:
#delete record from ''file'' table, remember file SHA1
#store the deleted SHA1 in deleted files table, do not remove the physical file yet
#wait for cron cleanup script to actually delete the file named SHA1 (proper table locking needed to prevent race conditions when adding/deleting files)

====File pool details====
located in $CFG->dataroot/filepool/, all files can not be stored in one directory due to OS limitations, it uses 3 levels based on first three characters of sha1 hash. It is unlikely that there will be thousands of files with the same first 3 chars in sha1 hash of their content.

This type of storage saves a lot of disk space when storing multiple copies of the same large file. It can also help substantially when synchronising data with external repositories. Another benefit is we can detect inconsistencies in file content.

File read performance is similar to previous code, file write performance will be slower - due to hashing and extra database access.

dataroot
/filepool
/00
/01
...
/'''23'''
/00
/01
...
/'''1e'''
/00
/01
...
/'''2d'''
/231e2dc421be4fcd0172e5afceea3970e2f3d940.jpg
...
/fe
/ff

==File management API==

This section describes following:
#file manager
#integration with html editor
#interactions with repos

===File manager===
Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

Originally there was a single file tree for each course. We need to fully separate each module/block from the course files and there might be also independent file areas in modules (ex: module introduction, content files, submissions, post attachments). File area may be defined as a small tree where we can use relative paths. These file areas are hanging from the branches of the context tree (this needs a picture).

===Integration with htmleditor===
Html editor should be able to browse only relevant files - for example when editing resource introduction only images from the file area of that resource should be available; when editing html resource page only the content area images should be listed.

There are several problems here:
# when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area
# we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile/intro@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.

Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.

===Interactions with repos===
Repositories may serve as a replacement for file uploading. They may be also used to synchronise files between courses. The repo option should be available whenever there is a file upload field, sometimes with extra "keep synchronised" option (this would not make sense for stuff like assignment submissions).

==Upgrade, migration and backwards compatibility==
It is going to be a pain again like DML/DDL ;-)

===Code backwards compatibility===
0% backwards compatibility related to file storage. New objects will be mandatory to use. Old $CFG->dataroot/$courseid/ will be empty, $CFG->dataroot/blog/ too, etc.

===Content backwards compatibility===
Means existing courses should not loose images, flash, etc. Though some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Migration of content===
* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new are, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be many outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

===Moving files to files table and file pool===
The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.
Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some ppl use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

==Backup/restore changes==
File handling in backups needs to be fully rewritten - list of files in xml + pool of sha1 named files with contents. This solves the utf-8 trouble here, yay!!

==Quotas==
File size will be stored in files table, we can use simple queries to find out how much space is used, however this may not be accurate because the sha1 hash based storage eliminates duplicate files.
*total course files - find out all contexts used in course, query files table with contextid IN ($listofcontexts)
*module files - find module context and calculate space per file area
*user files quota - inside the personal area only, counting all attachments in all mods might take a while
We could also divide the file size by number of instances that are using it, this might be considered more accurate in some scenarios.

==Other==
*antivirus scanning + upload manager rewrite/integration with forms lib
*zip compression and extraction

==Major problems==
List of hard to solve prolbems

===unicode zip support===
Unicode chars in zip files uploaded by teachers - unfortunately there is no 100% solution that will work for anybody because most zip programs do not support unicode, it is usually ''garbage in/garbage out'' which works in some cases only

Latest WinZIP 11.2 and Total Commander seem to support some very limited form of utf-8 encodings of file names. I managed to create a zip file in Windows (Czech locale) and extract them in linux with native PHP zip functions, the only step I needed to add was conversion cp852(DOS charset for Czech locale) -> UTF-8. The native windows zipping did not work for me though, because it does some different borking of charsets.

In any case it seems likely that native PHP support in PHP 5.2.x should be better than current pclzip or infozip binary.

===empty directories===
Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.

===file overwriting===
Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.

== Some little comments to be considered (to avoid forgetting them) ==

* each context will have its own "file manager"
* separate "file manager context" files (FMF) and "internal context" (ICF) files (current modedit files, submissions, attachements...)
* /pluginfile.php/SYSCONTEXTID/{blog|question} and so... will have own FMF too? Or only ICF ?
* Way to copy between contexts
* Links = -1 for them
* Deletion strategy (locks, quarantine status...)
* include support for quotas per user, per course, etc
* upgrade process should be interruptable (like the unicode upgrade) so it can be stopped/restarted any time

==Justin's thinking out loud==

I'm actually working on implementing this along with extending an existing Alfresco integration to work together with the whole File / Repository system and I wanted to get some of my comments and thoughts in here for feedback. Go easy on me. =)

So far I've only got one that I'd like to solicit some feedback on (BTW, if this would be better suited to a forum discussion, let me know):

===Not storing the full ''filepath'' with each entry in the '''file''' table===
*For browsing a directory structure, determining things like child directories or a parent directory given a filepath requires a lot of extraneous coding in PHP. I think it might be better served to create a new '''file_directory''' table, storing only a directory name, and reference to a parent directory record. The benefits here are that we're storing a lot of duplicate text field values in the '''file''' table and browsing through the file picker for local files doesn't require a lot of PHP overhead to calculate links to parent / child directories.
*Given that file permissions are no longer calculated using structured file paths, using the complete, full, path to a given file would most likely never be needed.

*The '''repositorypath''' field in the '''repository_sync''' table still makes sense, though.

*The '''file_directory''' table:

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''parent'''
|int(10)
|
|ID of directory that this record is a child of.

|-
|'''directoryname'''
|varchar(255)
|
|The actual name of this directory.
|}

skodak: filepath is stored in files table - its root is the corresponding filearea, the file manager will use the context tree to find all plugins/courses and ask them to return the list of areas with all those small branches inside it

==See also==

* [[Development:Repository API]]
* [[Development:Portfolio API]]

Development:File API

2008-07-03T14:56:04Z

Nicolasconnault: fixing typos

This page outlines the current thinking about implementing file storage and access in Moodle 2.0. It's a SPECIFICATION UNDER CONSTRUCTION!

The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions, problems to report or major changes to suggest please add them to the [[Development_talk:File_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavour to merge all such suggestions into the main spec before we start development.

==Objectives==

# Allow files to be added directly into Moodle (as we do now)
# Remember where files came from
# Give modules control over the access to files using capabilities and other local rules
# Consistent and simple approach for ALL file handling throughout Moodle

==Overview==

The File API is a core set of interfaces that all Moodle code will use to:
# store files within Moodle
# display files to Moodle users

It applies only to "user" files. It will NOT apply to local files and caches created by Moodle such as these directories in dataroot: temp, lang, cache, environment, filter, rss, search, sessions, upgradelogs etc

The API will be split into several independent parts:
# File serving API
## file.php
## pluginfile.php
## userfile.php
## rssfile.php
# File storage API
## optional access control
## optional repo sync
# File management API
## File browsing
## File linking (editor integration)
## Upload from repository

==File serving API==
Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

===file.php===
Serves course files.

Implements basic file access. Ideally only images and files linked from course sections should be there, no XSS protection required - we expect javascript, sw, etc. there, no way to make it "secure". The access control is not critical any more if we move most of the files into modules

The file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files would be stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'content', 'itemid'=>0)</code>

===pluginfile.php===
(aka modfile.php)
Sends module, block, question files.
* modules decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* only internal links to selected areas are supported - you can link images in summary area, but not the assignment submissions

Absolute file links need to be rewritten if html editing allowed in module. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/summary/image.jpg, it is converted back to internal links before saving.

::Can the distinct file areas supported by one plugin be declared somehow in order add some information about them? For example, I think it can be interesting to declare:
::* assignment_summary:
::** relpath='summary'
::** userdata=false
::** anotherproperty=anothervalue
::* assignment_submission:
::** relpath='submission/@@USERID@@'
::** userdata=false
::** anotherproperty=anothervalue
::* and so on...
::And then, when the editor "receives" one "assignment_summary" areaname, if knows what to show and so on? Also that info could be useful to know, in backup & restore if some fileareas have to be processed or no (userdata=false). Or also, when reconstructing the links (str_replace() above). And will cause to have a well defined list of fileareas by module, instead of coding them in a free way (prone to errors). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 16:35, 28 June 2008 (CDT)

::Something like this will be part of file management API, hardcoding this in file storage would make it less flexible imo [[User:Skodak|Skodak]]

::Yup, yup. Storage doesn't know anything but get/put files (nothing else). It's part of management, absolutely. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:21, 29 June 2008 (CDT)

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

====assignment example====

/pluginfile.php/assignmentcontextid/summary/someimage.jpg
/pluginfile.php/assignmentcontextid/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/extra/allsubmissionfiles.zip

::Uhm... all those files together? What's going to differentiate the "submission" path in the example above from the "summary" path? Is it supposed that the editor, or the filemanager won't allow , for example to pick-up one file from the "submission" area to be used in the summary of one assignment and only the "summary" area will be showed? That means multiple file managers by context and it's against the clean "one file manager per context" agreed below [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 21:28, 26 June 2008 (CDT)
::Yes Eloy, the different areas (summary, submission) etc. have different uses, different access control. There are two types of file manager - the two pane file manager which lists all contexts+areas user may access, and minimalistic manager in html editor which shows only subset of areas from current plugin (because you can not link anything else).

====scorm example====

/pluginfile.php/scormcontextid/summary/someimage.jpg
/pluginfile.php/scormcontextid/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems. The lifetime should be adjustable in module settings.

====quiz example====

pluginfile.php/quizcontextid/summary/niceimage.jpg
pluginfile.php/quizcontextid/report/type/export.ods

====questions example====

pluginfile.php/SYSCONTEXTID/question/questionid/file.jpg

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'filearea'=>'blog', 'itemid'=>$blogentryid)</code>

====backup example====
It would be nice to have some special protection of backup files - new capabilities for backup file download, upload. Backups contain a lot of personal info, we could block restoring of backups from other sites too.

/pluginfile.php/coursecontextid/backup/backupfile.zip

Internally stored in <code>array('contextid'=>$coursecontextid, 'filearea'=>'backup', 'itemid'=>0)</code>

===userfile.php===
Personal file storage, intended as an online storage of work in progress like assignments before the submission.
* read/write own files only for now
* option to share with others later
* personal "websites" will not be supported (security)

/userfile.php/userid/dir/dir/filename.ext

===rssfile.php===
Replaces rss/file.php which is kept only for backwards compatibility.
RSS files should not require sessions/cookies, URLs should contain some sort of security token/key.
Internally the files may be stored in database or together with other files.
Performance improvements - we should support both Etag (cool) and Last-Modified (more used), when we receive If-None-Match/If-Modified-Since => 304

/rssfile.php/contextid/any/parameters/module/wants/rss.xml
/rssfile.php/SYSCONTEXTID/blog/userid/rss.xml

Again modules and plugins decide what gets sent to user.

===Temporary files===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* zipping/unzipping
* processing by executable files (latex, mimetex)

Ideally these files should never use utf-8 (which is a major problem for zipping at the moment).
Proposed new sha1 based file storage is not suitable both for performance and technical reasons.

===Legacy file storage and serving===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

==File storage API==
Modules in general work only with local Moodle files. One of the major reason is performance when accessing external repository files. It will be possible to use repositories instead of file uploading and also to keep local files synced with external repository.

File contents are stored in moodledata/filepool indexed using SHA1 hashes instead of file names; file names, relative paths and other metadata will be stored in file(_xxx) database tables. This should be fully abstracted so that modules do not actually know where the files are located. When storing files the content is sent as string or file handle, when reading content it is returned as file handle.

===files table===

This table contains one entry for every file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary.

note: plural used because file is a reserved word

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|
|The sha1 hash of content.

|-
|'''contextid'''
|int(10)
|
|The context id defined in context table - identifies the instance of plugin owning the file.

|-
|filearea
|varchar(50)
|
|Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.

|-
|itemid
|int(10)
|
|Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)

|-
|filepath
|text
|
|relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this

|-
|filename
|varchar(255)
|
|The full Unicode name of this file (case sensitive)

|-
|filesize
|int(10)
|
|size of file - bytes

|-
|mimetype
|varchar(100)
|NULL
|type of file

|-
|'''userid'''
|int(10)
|NULL
|Optional - general user id field - meaning depending on plugin

|-
|timecreated
|int(10)
|
|The time this file was created

|-
|timemodified
|int(10)
|
|The last time the file was modified
|}

index on "contextid, filearea, itemid" and "sha1hash"

Plugin type is not specified because it is derived from contextid, items like blog that do not have own context will use own filearea usually from systemcontextid.

::Perhpas we could also hash filepath and filename and index by them, to save some text limitations in the DB side (length limits of indexes, not indexable, complex retrieval...). [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 11:54, 29 June 2008 (CDT)

::Also, perhaps we should store finally the plugin type there to save some queries per request, using it to drive to the correct file handling of each plugin. [[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 18:54, 29 June 2008 (CDT)

=== files_cleanup table ===

This table contains candidates for deletion from the file pool. Files are not deleted immediately, cron uses the files_cleanup table, verifies the file is not used any more and deletes it from pool. Reasons for cron clean-up are performance and prevention of collision - there could be a problem with concurrent uploads and deletes, we will probably need to add some table-based locking during the clean-up.

We might add an extra script that does deep validation of pool area - report missing files, report orphaned files, content not matching the sha1 filename, etc. - this would very very time consuming.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|sha1hash
|varchar(40)
|

|}

=== files_metadata table ===

This table contains extra metadata about files. Repositories could provide this, or it could be manually edited in the local copy.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''name'''
|varchar(255)
|
|The name of extra metadata

|-
|value
|text
|
|Value

|}

===files_acl table===

This table describes optional ACL for file. This is not required in majority of cases, modules usually hardcode the file access logic, course files should not be used much any more.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|The file we are defining access for

|-
|'''contextid'''
|int(10)
|
|The context where this file is being published

|-
|'''capability'''
|text
|
|The capability that is required to see this file.
|}

====acl notes====
* this is missing some concept similar to '''user/group/others''', for example in case of user files typical user can not assign permissions or view them - this becomes useless there
* it is more important to synchronise the availability of file link and the file itself - having link pointing to inaccessible file or file which is accessible when not wanted are both problems
* browser/proxy caching works against us here - "secret" files should not be cached

===files_sync table===

This table contains information on how to synchronise data with repositories. Data would be synchronised from cron.php or on demand from file manager. The sync would be one way only (repository-->local file).

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''fileid'''
|int(10)
|
|Id of file.

|-
|'''repositoryid'''
|int(10)
|
|The repository instance this is associated with, see [[Development:Repository_API]]

|-
|updates
|int(10)
|
|Specifies the update schedule (0 = none, 1 = on demand, other = some period in seconds)

|-
|repositorypath
|text
|
|The full path to the original file on the repository

|-
|timeimportfirst
|int(10)
|
|The first time this file was imported into Moodle

|-
|timeimportlast
|int(10)
|
|The most recent time that this file was imported into Moodle
|}

===File content storage===
Originally the file storage hierarchy contained a lot of metadata including userids, entry ids, filenames, etc. The file content will now be stored separately from file metadata. It must supports utf8 on all platforms.

File storing:
# calculate SHA1 hash of content
# check if file with SHA1 name exists, if not add the file to file pool
# remove SHA1 from list of deleted files if found there
# store file in ''file'' table, use SHA1 as file pool identifier

File reading:
#fetch file record from 'file' table - probably using file id or combination of contextid+instanceid
#fetch content of file

File deleting:
#delete record from ''file'' table, remember file SHA1
#store the deleted SHA1 in deleted files table, do not remove the physical file yet
#wait for cron cleanup script to actually delete the file named SHA1 (proper table locking needed to prevent race conditions when adding/deleting files)

====File pool details====
located in $CFG->dataroot/filepool/, all files can not be stored in one directory due to OS limitations, it uses 3 levels based on first three characters of sha1 hash. It is unlikely that there will be thousands of files with the same first 3 chars in sha1 hash of their content.

This type of storage saves a lot of disk space when storing multiple copies of the same large file. It can also help substantially when synchronising data with external repositories. Another benefit is we can detect inconsistencies in file content.

File read performance is similar to previous code, file write performance will be slower - due to hashing and extra database access.

dataroot
/filepool
/00
/01
...
/'''23'''
/00
/01
...
/'''1e'''
/00
/01
...
/'''2d'''
/231e2dc421be4fcd0172e5afceea3970e2f3d940.jpg
...
/fe
/ff

==File management API==

This section describes following:
#file manager
#integration with html editor
#interactions with repos

===File manager===
Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

Originally there was a single file tree for each course. We need to fully separate each module/block from the course files and there might be also independent file areas in modules (ex: module introduction, content files, submissions, post attachments). File area may be defined as a small tree where we can use relative paths. These file areas are hanging from the branches of the context tree (this needs a picture).

===Integration with htmleditor===
Html editor should be able to browse only relevant files - for example when editing resource introduction only images from the file area of that resource should be available; when editing html resource page only the content area images should be listed.

There are several problems here:
# when adding new resource its context does not exist yet, we will have to create some table to handle temporary file storage for adding of new stuff, not easy but should be solvable - maybe we could abuse the course context id or store it temporarily in some special user file area
# we can not use absolute address relinking for pluginfile.php links, instead we can use the absolute links only when editing and before storage convert them to something like @@thispluginfile/intro@@/2112/112/image.jpg before storage. the local links would be converted to full absolute links before display or editing. Not all file areas will support this (ex: linking to assignment submission does not make sense because nobody else may access it anyway). This would allow us to implement image preview in html editor.

Html editor should contain simplified single pane file manager with basic operations only - select file area, browse file area, upload file/copy user file/use repo file, delete. The editor will communicate with modules and core through ajax call to some script specified by module embedding the editor. The callback script would use different logic to construct the tree of files than the File manager, it needs to know only about files that other ppl viewing the resulting html may access.

===Interactions with repos===
Repositories may serve as a replacement for file uploading. They may be also used to synchronise files between courses. The repo option should be available whenever there is a file upload field, sometimes with extra "keep synchronised" option (this would not make sense for stuff like assignment submissions).

==Upgrade, migration and backwards compatibility==
It is going to be a pain again like DML/DDL ;-)

===Code backwards compatibility===
0% backwards compatibility related to file storage. New objects will be mandatory to use. Old $CFG->dataroot/$courseid/ will be empty, $CFG->dataroot/blog/ too, etc.

===Content backwards compatibility===
Means existing courses should not loose images, flash, etc. Though some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Migration of content===
* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new are, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be a lot of outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

===Moving files to files table and file pool===
The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.
Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some ppl use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

==Backup/restore changes==
File handling in backups needs to be fully rewritten - list of files in xml + pool of sha1 named files with contents. This solves the utf-8 trouble here, yay!!

==Quotas==
File size will be stored in files table, we can use simple queries to find out how much space is used, however this may not be accurate because the sha1 hash based storage eliminates duplicate files.
*total course files - find out all contexts used in course, query files table with contextid IN ($listofcontexts)
*module files - find module context and calculate space per file area
*user files quota - inside the personal area only, counting all attachments in all mods might take a while
We could also divide the file size by number of instances that are using it, this might be considered more accurate in some scenarios.

==Other==
*antivirus scanning + upload manager rewrite/integration with forms lib
*zip compression and extraction

==Major problems==
List of hard to solve prolbems

===unicode zip support===
Unicode chars in zip files uploaded by teachers - unfortunately there is no 100% solution that will work for anybody because most zip programs do not support unicode, it is usually ''garbage in/garbage out'' which works in some cases only

Latest WinZIP 11.2 and Total Commander seem to support some very limited form of utf-8 encodings of file names. I managed to create a zip file in Windows (Czech locale) and extract them in linux with native PHP zip functions, the only step I needed to add was conversion cp852(DOS charset for Czech locale) -> UTF-8. The native windows zipping did not work for me though, because it does some different borking of charsets.

In any case it seems likely that native PHP support in PHP 5.2.x should be better than current pclzip or infozip binary.

===empty directories===
Hmm, thinking a bit more about Justin's comment I realised there is no support for empty directories in this proposal. This will require either new table or some hack in files table - maybe we could add files with "." as name and just skip them when iterating directory content.

===file overwriting===
Concept of file overwriting does not exist anymore here, the path+filename are not enforced to be unique - we can not make index because sloppy mssql does not allow indexes larger than 900 bytes :-( We will haev to emulate it somehow and deal with collisions if found.

== Some little comments to be considered (to avoid forgetting them) ==

* each context will have its own "file manager"
* separate "file manager context" files (FMF) and "internal context" (ICF) files (current modedit files, submissions, attachements...)
* /pluginfile.php/SYSCONTEXTID/{blog|question} and so... will have own FMF too? Or only ICF ?
* Way to copy between contexts
* Links = -1 for them
* Deletion strategy (locks, quarantine status...)
* include support for quotas per user, per course, etc
* upgrade process should be interruptable (like the unicode upgrade) so it can be stopped/restarted any time

==Justin's thinking out loud==

I'm actually working on implementing this along with extending an existing Alfresco integration to work together with the whole File / Repository system and I wanted to get some of my comments and thoughts in here for feedback. Go easy on me. =)

So far I've only got one that I'd like to solicit some feedback on (BTW, if this would be better suited to a forum discussion, let me know):

===Not storing the full ''filepath'' with each entry in the '''file''' table===
*For browsing a directory structure, determining things like child directories or a parent directory given a filepath requires a lot of extraneous coding in PHP. I think it might be better served to create a new '''file_directory''' table, storing only a directory name, and reference to a parent directory record. The benefits here are that we're storing a lot of duplicate text field values in the '''file''' table and browsing through the file picker for local files doesn't require a lot of PHP overhead to calculate links to parent / child directories.
*Given that file permissions are no longer calculated using structured file paths, using the complete, full, path to a given file would most likely never be needed.

*The '''repositorypath''' field in the '''repository_sync''' table still makes sense, though.

*The '''file_directory''' table:

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''parent'''
|int(10)
|
|ID of directory that this record is a child of.

|-
|'''directoryname'''
|varchar(255)
|
|The actual name of this directory.
|}

skodak: filepath is stored in files table - its root is the corresponding filearea, the file manager will use the context tree to find all plugins/courses and ask them to return the list of areas with all those small branches inside it

==See also==

* [[Development:Repository API]]
* [[Development:Portfolio API]]