MoodleDocs - User contributions [en]

Profiling PHP

2021-08-11T13:18:19Z

Dongsheng:

PHP has two types of profiler:
* XHProf is a standard designed for capturing profile traces on live servers. It was originally designed by engineers at Facebook and is suitable for production environments.
* [http://www.xdebug.org/docs/profiler XDebug] is well known. and understood, and its trace output is supported by tools like [http://kcachegrind.sourceforge.net/html/Home.html KCachegrind]. It is not recommended for production environments.

==Tideways (replacement for XHProf)==

[https://tideways.com/profiler/downloads Tideways] is a PHP7-compatible replacement for XHProf, and support for this is built into Moodle. ([https://tideways.io/ Tideways] is a large, paid-for service, which helps to identify a range of issues with production servers. Whilst the service itself is a paid-for service, the instrumentation tooling is both Open Source, and free.)

When the XHProf PHP extension is installed, the "Profiling" section is displayed in "Site administration / Development". And when the profiling setting (profilingenabled) is enabled, a new link "Profiling runs" is also displayed in the Development tab. From this section, apart from access to the profiling runs, you can also import .mpr (Moodle profiling) files.

There are two variants of the plugin. You will almost certainly want the tideways_xhprof plugin.

* [https://tideways.com/profiler/downloads tideways] - the original tool, which includes both xhprof instrumentation, and support for the Tideways service. This version is compatible with:
** Moodle 3.0.6
** Moodle 3.1.2
** Moodle 3.2 onwards
* [https://github.com/tideways/php-xhprof-extension tideways_xhprof] - a rewritten implementation of the original which only includes the xhprof instrumentation. Compatible with:
** Moodle 3.1.12
** Moodle 3.2.9
** Moodle 3.3.6
** Moodle 3.4.3
** Moodle 3.5 onwards

For installation details please see https://github.com/tideways/php-xhprof-extension

See
* [[Setting up xhprof on Moodle]]
* http://tjhunt.blogspot.co.uk/2013/05/performance-testing-moodle.html
* https://docs.google.com/presentation/d/1MV4R71UBgPgzM6I9h_yDnYcxIJlgCoLBUDtJJr-TzNI/present#slide=id.i0 presentation about using XHProf

=== Export Xhprof data to plugins ===

Starting at 3.6, Moodle allows for a plugin to handle the Xhprof data and not insert the traces into the database as implemented in (MDL-63031)[https://tracker.moodle.org/browse/MDL-63031]

To disable Moodle from writing traces to the database, add: "$CFG->disableprofilingtodatabase = False;" into config.php

Plugins will need to provide a _store_profiling_data function to handle the data.

==Xdebug==

[http://xdebug.org/docs/profiler Xdebug] is a powerful PHP debugging tool. The first release of Xdebug was in 2002, since that it keeps growing and remains popular among PHP developers. Among its major features are stack and function tracing, code coverage analysis, remote debugging and scripts profiling. As the page topic suggests, we will focus on its profiling feature, which provides developer with detailed information about the script performance, helps identifying which parts of the code are slow. Collected information is being stored in cachegrind compatible file and can be analysed using one of external tools, such as [http://kcachegrind.sourceforge.net/html/Home.html KCachegrind], [http://sourceforge.net/projects/wincachegrind/ WinCacheGrind], [http://code.google.com/p/xdebugtoolkit/ xdebugtoolkit] or web-based analyser [https://github.com/jokkedk/webgrind#readme Webgrind]. Xdebug is simple to install and operate, it does not require code changes.

===Installing Xdebug extension===

The first step that has to be done is installing Xdebug extension. This procedure depends on the OS you are using, but general idea is to obtain Xdebug php extension and specify full path to it using ''zend_extension'' setting in ''php.inf'' file. Official [http://www.xdebug.org/docs/install Xdebug installation documentation] explains its installation on Windows, installing through PEAR/PECL and compiling from source.
Xdebug has been packaged for many Linux distributions, so it can be installed using corresponding package management tool. On Debian or Ubuntu, for example one would need to execute:

# apt-get install php5-xdebug

This will put ''xdebug.so'' in default php modules directory and create ''/etc/php5/conf.d/xdebug.ini'' file with single line:

zend_extension=/usr/lib/php5/20090626/xdebug.so

Note that this is equal to specifying the same line in ''php.inf'', example above just reflects split php configuration.

Once the Xdebug extension is installed and specified in php configuration, one may restart the webserver and information about Xdebug should appear in ''phpinfo()'' function output. If not, make sure that ''zend_extension'' line is not commented out, extension file exists in specified location and refer to webserver logs for more details.

===Configuring Xdebug Profiler===

When Xdebug extension is installed, it is time to configure profiling functionality. (If you are running apache2 as your web server and are using split configuration files such as the default apt-get install mentioned above, these settings should be added to "/etc/php5/apache2/conf.d/xdebug.ini".) There are number of parameters related to profiling, all of them start with ''xdebug.profiler_'' prefix. First of all, profiler should be enabled. There are two ways of doing it. One way is keeping it always enabled, so profiling information will be generated on every page request:

xdebug.profiler_enable=1

Another way is making Xdebug writing profiling information on demand by triggering it with GET/POST or COOKIE variable named ''XDEBUG_PROFILE'':

xdebug.profiler_enable_trigger = 1

This option is preferable for several reasons: profiling data files are relatively large especially on complex scripts and you may run out of disk-space pretty quick, it allows generating profiling information only when you need it, thus it is easier to find the generated file in output directory, finally, on demand profiling can be done on production servers, though it is not recommended. If you have enabled ''xdebug.profiler_enable_trigger'' option, make sure that ''xdebug.profiler_enable'' is disabled, otherwise this will lead to dump file being generated on each request.

Whatever method of profiling enabling you have chosen, profiling data files will be generated on each request (or request with a trigger parameter) in directory specified with ''xdebug.profiler_output_dir'' directive. By default is it set to ''/tmp'', but you may change it to any more suitable location.

xdebug.profiler_output_dir=/tmp

Generated file will be named in accordance with ''xdebug.profiler_output_name'' setting. This setting can handle some specifiers and use them in the file name. Default name pattern is ''cachegrind.out.%p''; in the actual file name %p will be replaced with process ID value. The fill list of specifier can be found [http://www.xdebug.org/docs/all_settings#trace_output_name here]. More intuitive naming is recommended, so that it is easier to find the file you have just generated in the bunch of others:

xdebug.profiler_output_name=cachegrind.out.%R.%t

With this naming pattern, timestamp and script name will be reflected in the file name.

When you are done with configuration, it is time to test it. First of all, the webserver has to be restarted, so that the new config will come into effect. Depending on your operating system and web server, you should be able to restart it with a command similar to:

sudo service apache2 restart

(This should work for apache2 on several flavours of Linux.)

===Profiling a page===

Now assuming that Xdebug was configured to use a trigger for script profiling, open any php page on your server in your browser having added ''XDEBUG_PROFILE'' parameter to URL string:

<nowiki>http://servername/moodle2/index.php?XDEBUG_PROFILE</nowiki>

or if it already has some parameters, just add our trigger to the URL end:

<nowiki>http://servername/moodle2/mod/forum/view.php?id=5&XDEBUG_PROFILE</nowiki>

As a result, the new file should be generated in directory you specified with ''xdebug.profiler_output_dir'' directive:

cachegrind.out._moodle2_mod_forum_view_php.1289838411
cachegrind.out._moodle2_index_php.1289837892

Triggering profiling with POST requests or AJAX queries is also possible without code changes. [https://addons.mozilla.org/en-US/firefox/addon/58688/ Easy Xdebug] plugin for FireFox has profiling toggle button that inserts XDEBUG_PROFILE variable into cookie data, thus making profiling enabled for as long as you wish for all requests. Similar plugins exist for [https://chrome.google.com/extensions/detail/eadndfjplgieldjbigjakmdgkmoaaaoc Chrome] and [https://github.com/benmatselby/xdebug-toggler Safari] browsers.

===Analyzing Xdebug profiling files===

As it was pointed out earlier, profiling data is recoded in cachegrind format, so it can be analysed using one of external tools, such as [https://kcachegrind.github.io/html/Home.html KCachegrind], [http://sourceforge.net/projects/wincachegrind/ WinCacheGrind], [http://code.google.com/p/xdebugtoolkit/ xdebugtoolkit] or web-based analyser [https://github.com/jokkedk/webgrind#readme Webgrind]. Using these tools is pretty simple. I prefer KCachegrind which has a feature to show the matching code if web-server is run on the same box. Xdebug profiler documentation page has a [http://xdebug.org/docs/profiler#misc section] about KCachegrind that worth reading for everyone who starts using KCachegrind.

===Quick summary===

1. Install Xdebug 3 extension on your server.

2. Add something like this to your php.ini file:
zend_extension = /usr/local/lib/php/pecl/20190902/xdebug.so
xdebug.mode = profile
xdebug.start_with_request = trigger
xdebug.output_dir = /var/tmp
xdebug.profiler_output_name = cachegrind.out.%R.%t
or add something like this to your .htaccess file
php_flag xdebug.mode = profile
php_flag xdebug.start_with_request = trigger
php_value xdebug.output_dir /var/tmp
php_value xdebug.profiler_output_name cachegrind.out.%R.%t

3. When you want to profile a page, add <tt>&XDEBUG_PROFILE</tt> to the end of the URL.

4. Open the <tt>cachegrind.out....</tt> file that is generated with one of the tools mentioned above.

==See also==

* [http://moodle.org/mod/forum/discuss.php?d=162045 Forum thread about profiling Moodle 2.0]
XHProf articles:
* [https://docs.google.com/present/view?id=dcbkgbgf_45fbg3rnmk& Presentation about XHProf]
* [http://techportal.ibuildings.com/2009/12/01/profiling-with-xhprof/ Profiling with XHProf]
* [http://www.open.ac.uk/blogs/XHProf/?page_id=46 XHProf – Profiling and Reporting] from Moodle perspective written by [http://moodle.org/user/view.php?id=264538&course=5 James Brisland]
Xdebug articles:
* [http://devzone.zend.com/article/2803-Introducing-xdebug Introducing xdebug]
* [http://blog.teamlazerbeez.com/2010/05/04/xdebug-quickstart-profiling-in-php/ Xdebug Quickstart: Profiling in PHP]

Charts API

2021-05-11T03:24:21Z

Dongsheng:

{{Moodle 3.2}}

== Overview ==

The Charts API is a core set of methods intended to provide a simple and yet modern interface to generate dynamic charts.

== Chart types ==

The first step to create a new chart is to create an instance of the desired chart type.
At the moment bar, line and pie types are supported and using some display changing methods can change how the charts are displayed.

=== Bar ===
To create a new bar chart, just create a new instance of '''chart_bar''' class.
<code php>
$chart = new core\chart_bar();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:bar_chart.png]]

You might want change how your bar chart are displayed:

==== Stacked Bar ====
To display stacked bars, you can call '''set_stacked()''' method, setting the parameter to true.
<code php>
$chart = new core\chart_bar();
$chart->set_stacked(true);
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:stacked_bar_chart.png]]

==== Horizontal Bar ====
To display you bar chart in the horizontal position, you need to call '''set_horizontal()''':
<code php>
$chart = new core\chart_bar();
$chart->set_horizontal(true); // Calling set_horizontal() passing true as parameter will display horizontal bar charts.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:horizontal_bar_chart.png]]

=== Line ===
To create a new line chart, just create an instance of '''chart_line''' class. By default, lines are tensioned.
<code php>
$chart = new \core\chart_line();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:line_chart.png]]

==== Smooth lines ====
You might want to change you line chart to display '''smooth lines''':
<code php>
$chart = new \core\chart_line();
$chart->set_smooth(true); // Calling set_smooth() passing true as parameter, will display smooth lines.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:smooth_line_chart.png]]

=== Pie ===
To create a pie chart you just need to create a new instance of chart_pie class.
<code php>
$chart = new \core\chart_pie();
$chart->add_series($sales); // On pie charts we just need to set one series.
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:pie_chart.png]]

==== Doughnut ====
You might want to change you pie chart to be displayed as doughnut:
<code php>
$chart = new \core\chart_pie();
$chart->set_doughnut(true); // Calling set_doughnut(true) we display the chart as a doughnut.
$chart->add_series($sales);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:doughnut_pie_chart.png]]

== Series ==
Series can be defined as a set of values. To set series of a chart, you need to create an instance of '''chart_series''' object and pass the title and an array of values.
The title and the data are displayed when mouse over the specific point representing the serie on the chart.
'''The number of values of the series must be equal to the number of labels.'''
<code php>
$serie1 = new core\chart_series('My series title', [400, 460, 1120, 540]);
</code>

== Properties ==

=== Title ===
It is possible to set a chart title, by calling set_title and passing the title as string method.
<code php>
$chart->set_title('chart with a title');
</code>

=== Labels ===
Labels are the description in which the data will be grouped, in the example above labels are an array of years. The number of values on the series must match the number of labels.
<code php>
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>

== Axis ==
You can customise chart axis (X,Y) by setting title, position and change the step size.
Firstly, you need to select the axis by the side you want and providing the index and whether you want to create the axis if it does not exists.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis = $chart->get_yaxis(0, true) // Select the index 0 of Y axis and pass true to create the axis if not exists.
</code>
Once you get the axis you want change, you can change the attributes you need.
It is also possible to select multiple axis, for example if you want to have custom steps labels in one side and data in the other.
<code php>
$chart = new chart_line();
$xaxis1 = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis1 = $chart->get_yaxis(0, true) // Will display the default axis steps information.
$yaxis2 = $chart->get_yaxis(1, true) // Select the second Y axis.
$yaxis2->set_position(chart_axis::POS_RIGHT); // Now I can change positioning, labels and etc just on the right side.
</code>

=== Axis title ===
Axis titles can be displayed on the chart sides and can be used to provide additional information about the chart.
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true)->set_label("I'm the label for X");
$chart->get_yaxis(0, true)->set_label("I'm the label for Y");
</code>

=== Axis position ===
You can define the position of axis by calling '''set_position()''' and passing the axis position constant (POS_RIGHT, POS_LEFT, POS_BOTTOM, POS_TOP).
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$xaxis->set_label("I'm X, but at the top");
$xaxis->set_position(chart_axis::POS_TOP); // You can use POS_BOTTOM or POS_TOP for X axis, in this case let's change the position to the top.
$chart->set_xaxis($xaxis);

// Customise Y axis.
$yaxis = new chart_axis();
$yaxis->set_position(chart_axis::POS_RIGHT); // You can use POS_LEFT or POS_RIGHT for Y axis, in this case let's change the position to the right side.
$chart->set_yaxis($yaxis);
</code>

=== Step size ===
Step size can be described as the size between the axis labels. You can set the step size of the axis, by calling '''set_stepsize()''' and passing the step size number.
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$axis->set_stepsize(5); // Chart steps will be displayed as 5, 10, 15, 20...
</code>

=== Step labels ===
If you prefer to display custom labels instead of numbers, just get the axis and call set_labels passing an array of labels:
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true);
$chart->get_xaxis(1, true)->set_labels(['Poor', 'Average', 'Good', 'Perfect']);
</code>

=== Min and Max ===
You can customise the minimum and the maximum value of a axis step size.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(1, true);
$xaxis->set_min(1);
$xaxis->set_max(100);
</code>

== Mixed chart types ==
It is possible to combine two types of chart by setting the type of the series differently of the chart type. For example, to create a chart that combine a bar chart with a line chart:
<code php>
$chart = new \core\chart_bar(); // Create a bar chart instance.
$series1 = new \core\chart_series('Series 1 (Bar)', [1000, 1170, 660, 1030]);
$series2 = new \core\chart_series('Series 2 (Line)', [400, 460, 1120, 540]);
$series2->set_type(\core\chart_series::TYPE_LINE); // Set the series type to line chart.
$chart->add_series($series2);
$chart->add_series($series1);
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>
Please note the order you call add_series change the order series are displayed on the chart. In the example above, the first to be displayed will be line chart and in the background the bar chart.

Example:

[[File:bar_line_chart.png]]

== Rendering ==
All charts are rendered by render_chart() located on outputrenderers.php, once the chart object is ready, it must be passed to render() or render_chart().
<code php>
echo $OUTPUT->render($chart);
</code>

=== Show chart data table ===
By default, in order to make chart data accessible to users with special needs, a link at the bottom of the chart will display a table containing all of its data.
If that information is already displayed on the page in some other fashion, it may not be necessary to display the chart data table.
To remove it, a '''false''' value can be passed as the second parameter when calling the render_chart function:
<code php>
echo $OUTPUT->render_chart($chart, false);
</code>

=== Overriding chart colours ===
It is possible to override the default set of colours used on charts, you just need to the following setting to config.php and set an array of hex css colours:
<code php>
$CFG->chart_colorset = ['#001f3f', '#01ff70', '#F012BE', '#85144b', '#B10DC9'];
</code>

=== Legend options ===
It is possible to customize some aspects of the chart legend such position and visibility. In order to do that you need to call set_legend_options() method
and pass an array of options supported by ChartJS.
<code php>
$chart->set_legend_options(['display' => false]); // Hide chart legend.
$chart->set_legend_options(['position' => 'left']); // Change legend position to left side.
</code>

=== RTL ===
Please note numerical ranges should be displayed in LTR, and not in RTL for both RTL and LTR languages, you can wrap the chart with a container with '''dir''' tag.

<code PHP>
echo html_writer::tag('div', $OUTPUT->render($chart), ['dir' => 'ltr']);
</code>

Charts API

2021-05-11T03:23:40Z

Dongsheng:

{{Moodle 3.2}}

== Overview ==

The Charts API is a core set of methods intended to provide a simple and yet modern interface to generate dynamic charts.

== Chart types ==

The first step to create a new chart is to create an instance of the desired chart type.
At the moment bar, line and pie types are supported and using some display changing methods can change how the charts are displayed.

=== Bar ===
To create a new bar chart, just create a new instance of '''chart_bar''' class.
<code php>
$chart = new core\chart_bar();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:bar_chart.png]]

You might want change how your bar chart are displayed:

==== Stacked Bar ====
To display stacked bars, you can call '''set_stacked()''' method, setting the parameter to true.
<code php>
$chart = new core\chart_bar();
$chart->set_stacked(true);
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:stacked_bar_chart.png]]

==== Horizontal Bar ====
To display you bar chart in the horizontal position, you need to call '''set_horizontal()''':
<code php>
$chart = new core\chart_bar();
$chart->set_horizontal(true); // Calling set_horizontal() passing true as parameter will display horizontal bar charts.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:horizontal_bar_chart.png]]

=== Line ===
To create a new line chart, just create an instance of '''chart_line''' class. By default, lines are tensioned.
<code php>
$chart = new \core\chart_line();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:line_chart.png]]

==== Smooth lines ====
You might want to change you line chart to display '''smooth lines''':
<code php>
$chart = new \core\chart_line();
$chart->set_smooth(true); // Calling set_smooth() passing true as parameter, will display smooth lines.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:smooth_line_chart.png]]

=== Pie ===
To create a pie chart you just need to create a new instance of chart_pie class.
<code php>
$chart = new \core\chart_pie();
$chart->add_series($sales); // On pie charts we just need to set one series.
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:pie_chart.png]]

==== Doughnut ====
You might want to change you pie chart to be displayed as doughnut:
<code php>
$chart = new \core\chart_pie();
$chart->set_doughnut(true); // Calling set_doughnut(true) we display the chart as a doughnut.
$chart->add_series($sales);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:doughnut_pie_chart.png]]

== Series ==
Series can be defined as a set of values. To set series of a chart, you need to create an instance of '''chart_series''' object and pass the title and an array of values.
The title and the data are displayed when mouse over the specific point representing the serie on the chart.
'''The number of values of the series must be equal to the number of labels.'''
<code php>
$serie1 = new core\chart_series('My series title', [400, 460, 1120, 540]);
</code>

== Properties ==

=== Title ===
It is possible to set a chart title, by calling set_title and passing the title as string method.
<code php>
$chart->set_title('chart with a title');
</code>

=== Labels ===
Labels are the description in which the data will be grouped, in the example above labels are an array of years. The number of values on the series must match the number of labels.
<code php>
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>

== Axis ==
You can customise chart axis (X,Y) by setting title, position and change the step size.
Firstly, you need to select the axis by the side you want and providing the index and whether you want to create the axis if it does not exists.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis = $chart->get_yaxis(0, true) // Select the index 0 of Y axis and pass true to create the axis if not exists.
</code>
Once you get the axis you want change, you can change the attributes you need.
It is also possible to select multiple axis, for example if you want to have custom steps labels in one side and data in the other.
<code php>
$chart = new chart_line();
$xaxis1 = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis1 = $chart->get_yaxis(0, true) // Will display the default axis steps information.
$yaxis2 = $chart->get_yaxis(1, true) // Select the second Y axis.
$yaxis2->set_position(chart_axis::POS_RIGHT); // Now I can change positioning, labels and etc just on the right side.
</code>

=== Axis title ===
Axis titles can be displayed on the chart sides and can be used to provide additional information about the chart.
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true)->set_label("I'm the label for X");
$chart->get_yaxis(0, true)->set_label("I'm the label for Y");
</code>

=== Axis position ===
You can define the position of axis by calling '''set_position()''' and passing the axis position constant (POS_RIGHT, POS_LEFT, POS_BOTTOM, POS_TOP).
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$xaxis->set_label("I'm X, but at the top");
$xaxis->set_position(chart_axis::POS_TOP); // You can use POS_BOTTOM or POS_TOP for X axis, in this case let's change the position to the top.
$chart->set_xaxis($xaxis);

// Customise Y axis.
$yaxis = new chart_axis();
$yaxis->set_position(chart_axis::POS_RIGHT); // You can use POS_LEFT or POS_RIGHT for Y axis, in this case let's change the position to the right side.
$chart->set_yaxis($yaxis);
</code>

=== Step size ===
Step size can be described as the size between the axis labels. You can set the step size of the axis, by calling '''set_stepsize()''' and passing the step size number.
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$axis->set_stepsize(5); // Chart steps will be displayed as 5, 10, 15, 20...
</code>

=== Step labels ===
If you prefer to display custom labels instead of numbers, just get the axis and call set_labels passing an array of labels:
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true);
$chart->get_xaxis(1, true)->set_labels(['Poor', 'Average', 'Good', 'Perfect']);
</code>

=== Min and Max ===
You can customise the minimum and the maximum value of a axis step size.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(1, true);
$xaxis->set_min(1);
$xaxis->set_max(100);
</code>

== Mixed chart types ==
It is possible to combine two types of chart by setting the type of the series differently of the chart type. For example, to create a chart that combine a bar chart with a line chart:
<code php>
$chart = new \core\chart_bar(); // Create a bar chart instance.
$series1 = new \core\chart_series('Series 1 (Bar)', [1000, 1170, 660, 1030]);
$series2 = new \core\chart_series('Series 2 (Line)', [400, 460, 1120, 540]);
$series2->set_type(\core\chart_series::TYPE_LINE); // Set the series type to line chart.
$chart->add_series($series2);
$chart->add_series($series1);
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>
Please note the order you call add_series change the order series are displayed on the chart. In the example above, the first to be displayed will be line chart and in the background the bar chart.

Example:

[[File:bar_line_chart.png]]

== Rendering ==
All charts are rendered by render_chart() located on outputrenderers.php, once the chart object is ready, it must be passed to render() or render_chart().
<code php>
echo $OUTPUT->render($chart);
</code>

=== Show chart data table ===
By default, in order to make chart data accessible to users with special needs, a link at the bottom of the chart will display a table containing all of its data.
If that information is already displayed on the page in some other fashion, it may not be necessary to display the chart data table.
To remove it, a '''false''' value can be passed as the second parameter when calling the render_chart function:
<code php>
echo $OUTPUT->render_chart($chart, false);
</code>

=== Overriding chart colours ===
It is possible to override the default set of colours used on charts, you just need to the following setting to config.php and set an array of hex css colours:
<code php>
$CFG->chart_colorset = ['#001f3f', '#01ff70', '#F012BE', '#85144b', '#B10DC9'];
</code>

=== Legend options ===
It is possible to customize some aspects of the chart legend such position and visibility. In order to do that you need to call set_legend_options() method
and pass an array of options supported by ChartJS.
<code php>
$chart->set_legend_options(['display' => false]); // Hide chart legend.
$chart->set_legend_options(['position' => 'left']); // Change legend position to left side.
</code>

=== RTL ===
Please note numerical ranges should be displayed in LTR, and not in RTL for both RTL and LTR languages, you can wrap the chart with a container with <code>dir</code> tag.

<code PHP>
echo html_writer::tag('div', $OUTPUT->render($chart), ['dir' => 'ltr']);
</code>

Charts API

2021-05-11T03:21:24Z

Dongsheng:

{{Moodle 3.2}}

== Overview ==

The Charts API is a core set of methods intended to provide a simple and yet modern interface to generate dynamic charts.

== Chart types ==

The first step to create a new chart is to create an instance of the desired chart type.
At the moment bar, line and pie types are supported and using some display changing methods can change how the charts are displayed.

=== Bar ===
To create a new bar chart, just create a new instance of '''chart_bar''' class.
<code php>
$chart = new core\chart_bar();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:bar_chart.png]]

You might want change how your bar chart are displayed:

==== Stacked Bar ====
To display stacked bars, you can call '''set_stacked()''' method, setting the parameter to true.
<code php>
$chart = new core\chart_bar();
$chart->set_stacked(true);
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:stacked_bar_chart.png]]

==== Horizontal Bar ====
To display you bar chart in the horizontal position, you need to call '''set_horizontal()''':
<code php>
$chart = new core\chart_bar();
$chart->set_horizontal(true); // Calling set_horizontal() passing true as parameter will display horizontal bar charts.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:horizontal_bar_chart.png]]

=== Line ===
To create a new line chart, just create an instance of '''chart_line''' class. By default, lines are tensioned.
<code php>
$chart = new \core\chart_line();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:line_chart.png]]

==== Smooth lines ====
You might want to change you line chart to display '''smooth lines''':
<code php>
$chart = new \core\chart_line();
$chart->set_smooth(true); // Calling set_smooth() passing true as parameter, will display smooth lines.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:smooth_line_chart.png]]

=== Pie ===
To create a pie chart you just need to create a new instance of chart_pie class.
<code php>
$chart = new \core\chart_pie();
$chart->add_series($sales); // On pie charts we just need to set one series.
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:pie_chart.png]]

==== Doughnut ====
You might want to change you pie chart to be displayed as doughnut:
<code php>
$chart = new \core\chart_pie();
$chart->set_doughnut(true); // Calling set_doughnut(true) we display the chart as a doughnut.
$chart->add_series($sales);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:doughnut_pie_chart.png]]

== Series ==
Series can be defined as a set of values. To set series of a chart, you need to create an instance of '''chart_series''' object and pass the title and an array of values.
The title and the data are displayed when mouse over the specific point representing the serie on the chart.
'''The number of values of the series must be equal to the number of labels.'''
<code php>
$serie1 = new core\chart_series('My series title', [400, 460, 1120, 540]);
</code>

== Properties ==

=== Title ===
It is possible to set a chart title, by calling set_title and passing the title as string method.
<code php>
$chart->set_title('chart with a title');
</code>

=== Labels ===
Labels are the description in which the data will be grouped, in the example above labels are an array of years. The number of values on the series must match the number of labels.
<code php>
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>

== Axis ==
You can customise chart axis (X,Y) by setting title, position and change the step size.
Firstly, you need to select the axis by the side you want and providing the index and whether you want to create the axis if it does not exists.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis = $chart->get_yaxis(0, true) // Select the index 0 of Y axis and pass true to create the axis if not exists.
</code>
Once you get the axis you want change, you can change the attributes you need.
It is also possible to select multiple axis, for example if you want to have custom steps labels in one side and data in the other.
<code php>
$chart = new chart_line();
$xaxis1 = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis1 = $chart->get_yaxis(0, true) // Will display the default axis steps information.
$yaxis2 = $chart->get_yaxis(1, true) // Select the second Y axis.
$yaxis2->set_position(chart_axis::POS_RIGHT); // Now I can change positioning, labels and etc just on the right side.
</code>

=== Axis title ===
Axis titles can be displayed on the chart sides and can be used to provide additional information about the chart.
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true)->set_label("I'm the label for X");
$chart->get_yaxis(0, true)->set_label("I'm the label for Y");
</code>

=== Axis position ===
You can define the position of axis by calling '''set_position()''' and passing the axis position constant (POS_RIGHT, POS_LEFT, POS_BOTTOM, POS_TOP).
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$xaxis->set_label("I'm X, but at the top");
$xaxis->set_position(chart_axis::POS_TOP); // You can use POS_BOTTOM or POS_TOP for X axis, in this case let's change the position to the top.
$chart->set_xaxis($xaxis);

// Customise Y axis.
$yaxis = new chart_axis();
$yaxis->set_position(chart_axis::POS_RIGHT); // You can use POS_LEFT or POS_RIGHT for Y axis, in this case let's change the position to the right side.
$chart->set_yaxis($yaxis);
</code>

=== Step size ===
Step size can be described as the size between the axis labels. You can set the step size of the axis, by calling '''set_stepsize()''' and passing the step size number.
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$axis->set_stepsize(5); // Chart steps will be displayed as 5, 10, 15, 20...
</code>

=== Step labels ===
If you prefer to display custom labels instead of numbers, just get the axis and call set_labels passing an array of labels:
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true);
$chart->get_xaxis(1, true)->set_labels(['Poor', 'Average', 'Good', 'Perfect']);
</code>

=== Min and Max ===
You can customise the minimum and the maximum value of a axis step size.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(1, true);
$xaxis->set_min(1);
$xaxis->set_max(100);
</code>

== Mixed chart types ==
It is possible to combine two types of chart by setting the type of the series differently of the chart type. For example, to create a chart that combine a bar chart with a line chart:
<code php>
$chart = new \core\chart_bar(); // Create a bar chart instance.
$series1 = new \core\chart_series('Series 1 (Bar)', [1000, 1170, 660, 1030]);
$series2 = new \core\chart_series('Series 2 (Line)', [400, 460, 1120, 540]);
$series2->set_type(\core\chart_series::TYPE_LINE); // Set the series type to line chart.
$chart->add_series($series2);
$chart->add_series($series1);
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>
Please note the order you call add_series change the order series are displayed on the chart. In the example above, the first to be displayed will be line chart and in the background the bar chart.

Example:

[[File:bar_line_chart.png]]

== Rendering ==
All charts are rendered by render_chart() located on outputrenderers.php, once the chart object is ready, it must be passed to render() or render_chart().
<code php>
echo $OUTPUT->render($chart);
</code>

=== Show chart data table ===
By default, in order to make chart data accessible to users with special needs, a link at the bottom of the chart will display a table containing all of its data.
If that information is already displayed on the page in some other fashion, it may not be necessary to display the chart data table.
To remove it, a '''false''' value can be passed as the second parameter when calling the render_chart function:
<code php>
echo $OUTPUT->render_chart($chart, false);
</code>

=== Overriding chart colours ===
It is possible to override the default set of colours used on charts, you just need to the following setting to config.php and set an array of hex css colours:
<code php>
$CFG->chart_colorset = ['#001f3f', '#01ff70', '#F012BE', '#85144b', '#B10DC9'];
</code>

=== Legend options ===
It is possible to customize some aspects of the chart legend such position and visibility. In order to do that you need to call set_legend_options() method
and pass an array of options supported by ChartJS.
<code php>
$chart->set_legend_options(['display' => false]); // Hide chart legend.
$chart->set_legend_options(['position' => 'left']); // Change legend position to left side.
</code>

=== RTL ===
Please note numerical ranges should be displayed in LTR, and not in RTL for both RTL and LTR languages, you can wrap the chart with container with dir tag.

<code PHP>
echo html_writer::tag('div', $OUTPUT->render($chart), ['dir' => 'ltr']);
</code>

Charts API

2021-05-11T03:14:51Z

Dongsheng:

{{Moodle 3.2}}

== Overview ==

The Charts API is a core set of methods intended to provide a simple and yet modern interface to generate dynamic charts.

== Chart types ==

The first step to create a new chart is to create an instance of the desired chart type.
At the moment bar, line and pie types are supported and using some display changing methods can change how the charts are displayed.

=== Bar ===
To create a new bar chart, just create a new instance of '''chart_bar''' class.
<code php>
$chart = new core\chart_bar();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:bar_chart.png]]

You might want change how your bar chart are displayed:

==== Stacked Bar ====
To display stacked bars, you can call '''set_stacked()''' method, setting the parameter to true.
<code php>
$chart = new core\chart_bar();
$chart->set_stacked(true);
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:stacked_bar_chart.png]]

==== Horizontal Bar ====
To display you bar chart in the horizontal position, you need to call '''set_horizontal()''':
<code php>
$chart = new core\chart_bar();
$chart->set_horizontal(true); // Calling set_horizontal() passing true as parameter will display horizontal bar charts.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:horizontal_bar_chart.png]]

=== Line ===
To create a new line chart, just create an instance of '''chart_line''' class. By default, lines are tensioned.
<code php>
$chart = new \core\chart_line();
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:line_chart.png]]

==== Smooth lines ====
You might want to change you line chart to display '''smooth lines''':
<code php>
$chart = new \core\chart_line();
$chart->set_smooth(true); // Calling set_smooth() passing true as parameter, will display smooth lines.
$chart->add_series($sales);
$chart->add_series($expenses);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:smooth_line_chart.png]]

=== Pie ===
To create a pie chart you just need to create a new instance of chart_pie class.
<code php>
$chart = new \core\chart_pie();
$chart->add_series($sales); // On pie charts we just need to set one series.
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:pie_chart.png]]

==== Doughnut ====
You might want to change you pie chart to be displayed as doughnut:
<code php>
$chart = new \core\chart_pie();
$chart->set_doughnut(true); // Calling set_doughnut(true) we display the chart as a doughnut.
$chart->add_series($sales);
$chart->set_labels($labels);
echo $OUTPUT->render($chart);
</code>

Example:

[[File:doughnut_pie_chart.png]]

== Series ==
Series can be defined as a set of values. To set series of a chart, you need to create an instance of '''chart_series''' object and pass the title and an array of values.
The title and the data are displayed when mouse over the specific point representing the serie on the chart.
'''The number of values of the series must be equal to the number of labels.'''
<code php>
$serie1 = new core\chart_series('My series title', [400, 460, 1120, 540]);
</code>

== Properties ==

=== Title ===
It is possible to set a chart title, by calling set_title and passing the title as string method.
<code php>
$chart->set_title('chart with a title');
</code>

=== Labels ===
Labels are the description in which the data will be grouped, in the example above labels are an array of years. The number of values on the series must match the number of labels.
<code php>
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>

== Axis ==
You can customise chart axis (X,Y) by setting title, position and change the step size.
Firstly, you need to select the axis by the side you want and providing the index and whether you want to create the axis if it does not exists.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis = $chart->get_yaxis(0, true) // Select the index 0 of Y axis and pass true to create the axis if not exists.
</code>
Once you get the axis you want change, you can change the attributes you need.
It is also possible to select multiple axis, for example if you want to have custom steps labels in one side and data in the other.
<code php>
$chart = new chart_line();
$xaxis1 = $chart->get_xaxis(0, true) // Select the index 0 of X axis and pass true to create the axis if not exists.
$yaxis1 = $chart->get_yaxis(0, true) // Will display the default axis steps information.
$yaxis2 = $chart->get_yaxis(1, true) // Select the second Y axis.
$yaxis2->set_position(chart_axis::POS_RIGHT); // Now I can change positioning, labels and etc just on the right side.
</code>

=== Axis title ===
Axis titles can be displayed on the chart sides and can be used to provide additional information about the chart.
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true)->set_label("I'm the label for X");
$chart->get_yaxis(0, true)->set_label("I'm the label for Y");
</code>

=== Axis position ===
You can define the position of axis by calling '''set_position()''' and passing the axis position constant (POS_RIGHT, POS_LEFT, POS_BOTTOM, POS_TOP).
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$xaxis->set_label("I'm X, but at the top");
$xaxis->set_position(chart_axis::POS_TOP); // You can use POS_BOTTOM or POS_TOP for X axis, in this case let's change the position to the top.
$chart->set_xaxis($xaxis);

// Customise Y axis.
$yaxis = new chart_axis();
$yaxis->set_position(chart_axis::POS_RIGHT); // You can use POS_LEFT or POS_RIGHT for Y axis, in this case let's change the position to the right side.
$chart->set_yaxis($yaxis);
</code>

=== Step size ===
Step size can be described as the size between the axis labels. You can set the step size of the axis, by calling '''set_stepsize()''' and passing the step size number.
<code php>
$chart = new chart_line();

// Customise X axis.
$xaxis = new chart_axis();
$axis->set_stepsize(5); // Chart steps will be displayed as 5, 10, 15, 20...
</code>

=== Step labels ===
If you prefer to display custom labels instead of numbers, just get the axis and call set_labels passing an array of labels:
<code php>
$chart = new chart_line();
$chart->get_xaxis(0, true);
$chart->get_xaxis(1, true)->set_labels(['Poor', 'Average', 'Good', 'Perfect']);
</code>

=== Min and Max ===
You can customise the minimum and the maximum value of a axis step size.
<code php>
$chart = new chart_line();
$xaxis = $chart->get_xaxis(1, true);
$xaxis->set_min(1);
$xaxis->set_max(100);
</code>

== Mixed chart types ==
It is possible to combine two types of chart by setting the type of the series differently of the chart type. For example, to create a chart that combine a bar chart with a line chart:
<code php>
$chart = new \core\chart_bar(); // Create a bar chart instance.
$series1 = new \core\chart_series('Series 1 (Bar)', [1000, 1170, 660, 1030]);
$series2 = new \core\chart_series('Series 2 (Line)', [400, 460, 1120, 540]);
$series2->set_type(\core\chart_series::TYPE_LINE); // Set the series type to line chart.
$chart->add_series($series2);
$chart->add_series($series1);
$chart->set_labels(['2004', '2005', '2006', '2007']);
</code>
Please note the order you call add_series change the order series are displayed on the chart. In the example above, the first to be displayed will be line chart and in the background the bar chart.

Example:

[[File:bar_line_chart.png]]

== Rendering ==
All charts are rendered by render_chart() located on outputrenderers.php, once the chart object is ready, it must be passed to render() or render_chart().
<code php>
echo $OUTPUT->render($chart);
</code>

=== Show chart data table ===
By default, in order to make chart data accessible to users with special needs, a link at the bottom of the chart will display a table containing all of its data.
If that information is already displayed on the page in some other fashion, it may not be necessary to display the chart data table.
To remove it, a '''false''' value can be passed as the second parameter when calling the render_chart function:
<code php>
echo $OUTPUT->render_chart($chart, false);
</code>

=== Overriding chart colours ===
It is possible to override the default set of colours used on charts, you just need to the following setting to config.php and set an array of hex css colours:
<code php>
$CFG->chart_colorset = ['#001f3f', '#01ff70', '#F012BE', '#85144b', '#B10DC9'];
</code>

=== Legend options ===
It is possible to customize some aspects of the chart legend such position and visibility. In order to do that you need to call set_legend_options() method
and pass an array of options supported by ChartJS.
<code php>
$chart->set_legend_options(['display' => false]); // Hide chart legend.
$chart->set_legend_options(['position' => 'left']); // Change legend position to left side.
</code>

=== RTL ===
Please note numerical ranges should be displayed in LTR, and not in RTL for both RTL and LTR languages, you can wrap the chart with container with dir tag.

<code PHP>
echo html_writer::tag('div', $OUTPUT->render($chart), array_merge(['dir' => 'ltr']));
</code>

User:Dongsheng Cai

2015-03-23T02:56:13Z

Dongsheng:

I was a Moodle developer working at [http://moodle.com/hq/ Moodle HQ]

Moodle projects I got involved:

* [[Chat module]]
* [[dev:Repository_API|Moodle Repository API]]
* [[dev:Comments_2.0|Comments 2.0]]
* [[dev:Wiki_2.0|Wiki 2]] contributor
* [[dev:Mobile app|Moodle mobile app for iOS]]

About me:
* [http://dongsheng.org My website]
* [http://github.com/dcai Github]

Upgrading Airnotifier

2014-10-18T16:19:03Z

Dongsheng:

Steps to upgrade

* Stop the server, killing the process
<code>
ps aux | grep airnotifier
kill -9 PROCESS_ID
</code>

* Backup the MongoDB databases
*# Backup AirNotifier settings database <code>mongodump -h localhost --port 27017 -d airnotifier -o /var/airnotifier/</code>
*# Backup each application data <code>mongodump -h localhost --port 27017 -d commoodlemoodlemobile -o /var/airnotifier/;</code> <code>mongodump -h localhost --port 27017 -d commoodlemoodlemobiletest -o /var/airnotifier/</code>

* Pull latest version from git
<code>
cd /opt/airnotifier
git pull
</code>

* Check that your current version has a version number in the options collection (AirNotifier settings database) (Older AirNotifier versions doesn't have it), use the shell:
mongo
use airnotifier
db.options.find()

* If there is not a value option, create a new one:
db.options.insert({name: "version", value: 20140101})

* Run the upgrade.py script
cd /opt/airnotifier
python upgrade.py

* Start the server again
./startserver
or
sudo -u airnotifier python airnotifier.py >> /var/airnotifier/console_log 2>> /var/airnotifier/error_log &

[[Category: Mobile]]

Moodle Mobile Push Notifications

2014-06-30T00:11:19Z

Dongsheng:

==Goal==
Moodle Mobile should receive Moodle messages as push notifications on iOS and Android.

== Use case and web services ==
[[Image:Screenshot_2_05_13_5_16_PM.png|thumb]]
[[Image:Screenshot_2_05_13_5_14_PM.png|thumb]]
=== currently implemented use case ===
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. The mobile app PushPlugin plugin does it.
# The user enters its Moodle credentials. The app gets an [http://airnotifier.github.io AirNotifier] access key from the site. (note that this access key is currently the same for each site => meaningless). The key is retrieved by calling the webservice message_airnotifier_get_access_key(permissions). create_token is the only possible permission at the moment.
# The app sends its token to [http://airnotifier.github.io AirNotifier] server - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io AirNotifier] provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# The user receive the notification.

=== brainstorming use case ===
# During [http://airnotifier.github.io AirNotifier] provider installation, the site contacts Airnotifer server to request an access key for itself + an access key for the devices (to send their device token to [http://airnotifier.github.io AirNotifier] server).
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token.
# The user enters its Moodle credentials. The app gets an [http://airnotifier.github.io AirNotifier] access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [http://airnotifier.github.io AirNotifier] server + the site url (so airnotifier knows the device/site couple, and so [http://airnotifier.github.io AirNotifier] can allow site broadcast) - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# During this web service, the site confirms the device token to [http://airnotifier.github.io AirNotifier]. It's done by a HTTP POST request. [http://airnotifier.github.io AirNotifier] server deletes unconfirmed device tokens older than one minute every hours.
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io AirNotifier] provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# Time to time, the [http://airnotifier.github.io AirNotifier] server checks Apple feedback and delete the unactive device tokens.
# When the [http://airnotifier.github.io AirNotifier] provider tries to send a message to a deleted device, it receive an error from [http://airnotifier.github.io AirNotifier] server and stores it. Then it sends a unique alert to the device user. The device is disabled on the user messaging settings. After a month, the cron processes delete the device.

=== another brainstorming use case ===
# All the sites (public or privately) already registered in Moodle.net will have automatically keys for using [http://airnotifier.github.io AirNotifier]
# When registering a new site in Moodle.net there will be a new option for enabling PUSH Notificiations (true by default)
# If your site is not registered and your Mobile Services were enabled you will see a message in the Moodle Notifications page indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your site is not registered and you enable Mobile Services you will see a message near to the "Enable Mobile Services" indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your installation has a very big number of users, during the registration process and in the Notifications page you will see a message indicating "The public PUSH Notification server is limited to Moodle sites with less than N (5k, 20k...) users"
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. A user can disable at any time notifications for his device
# Every time the user open the app, the app gets an [http://airnotifier.github.io AirNotifier] access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [http://airnotifier.github.io AirNotifier] server + the site url (so airnotifier knows the device/site couple, and so [http://airnotifier.github.io AirNotifier] can allow site broadcast) - it's done by a HTTP POST request. This is done only if your device token has changed.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename, siteURL) This is done every time a user opens the app
# Since the app is sending its device token every time a user opens the app, a cron job in the Moodle Site can delete old device tokens (so we can avoid the [http://airnotifier.github.io AirNotifier] feedback service in the first stage)
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io AirNotifier] provider (Moodle site) sends a payload for a specific device token + specific site to airnotifier server - it's done by a HTTP POST
# If the user disable the PUSH Notification feature, a WS message_airnotifier_remove_user_device is called, also the [http://airnotifier.github.io AirNotifier] is requested to delete the user device

Benefits are:
-We can reuse the current registration process that is secure (as far as I know)
-The [http://airnotifier.github.io AirNotifier] API for register new sites will be exposed only to Moodle.net (not to all sites)
-It will be easier notify registered sites that are abusing (Moodle.net can communicate with sites (sending an automatic email to the user admin))
-Moodle will have statistics of sites using PUSH directly in Moodle.net (no need to ask [http://airnotifier.github.io AirNotifier])
-The registration process can be improved to retrieve the number of registered devices that are receiving notifications
-I think that it will "enforce" users to register their sites (for receive security notifications, etc..)
-Big companies with big Moodle installations usually doesn't register their Moodle sites, this will force these companies to use their own [http://airnotifier.github.io AirNotifier] server instance
-Since we have the number of users in a Moodle site, we can automatically disable [http://airnotifier.github.io AirNotifier] for big installations (more thant Nk users)
-I think that having all the information that the registration provides will help us to "monitor and prevent abuse" of the [http://airnotifier.github.io AirNotifier] server

== Payload content ==
The current payload content would need a bit of brainstorming. We need to identify what to send as it is extremely short. The app will behave following the content it receives.
=== Apple ===
The Apple payload is limited to 256bytes and it includes everything sent. Our current implementation is still quite of a wip:
<code php>
{type:'moodle_instantmessage',id:'8',urlparams:'{"user":"2","id":"8"}',userfrom:'Manager Moodle',date:'1367477362',alert:'This is a message text.',}
</code>
=== GCM ===
The Apple payload is about 4Kb. As it's more more permissive than Apple, it is not an issue. The content will be the same than Apple ones, just the format changes.

==Deliverables==
* push notification support on the mobile app - the app receives the push notification and behaves correctly.
* The [http://airnotifier.github.io AirNotifier] server - it sends notification to APNS / GCM.
* The [http://airnotifier.github.io AirNotifier] provider - it sends Moodle messages to [http://airnotifier.github.io AirNotifier] - MDL-36445

=== [http://airnotifier.github.io AirNotifier] messaging provider ===
In your messaging setting page you can select which of your devices can receive push notifications. MDL-36445
The provider has usual messaging provider settings.

=== [http://airnotifier.github.io AirNotifier] server ===
To install the server follow: https://github.com/airnotifier/airnotifier/wiki/Installation

Need to be done:
* feedback - MOBILE-191
* load stress
* IP banning - need to be tested MOBILE-192
* Site broadcast (site keys) MOBILE-395
* DDOS attacks
* iOS support - it works
* Android support MOBILE-396

=== App push notification support in the mobile app ===
MOBILE-168

==== The app is closed or is running in the background ====
It's a normal OS notification. The user can open the app from it. We need to define what happens when we open the app. See payload content section.

==== The app is opened ====
The app displays a notification popup. Or better, it's a small temporary notification message at the top that retarct by itself - less intrusive.

== Risks ==
These risks have been raised by Dan in MDL-36445:
* Have we load tested it with large numbers of users/notifications?
* What is stopping someone downloading Moodle getting the keys to DOS our push notification service (surely blacklisting us with apple)
* If someone gains access to device ids, could they then use our service to spam users?
* Are we checking the 'feedback service' and removing devices which no longer exist, apple ominously writes "Note: APNs monitors providers for their diligence in checking the feedback service and refraining from sending push notifications to nonexistent applications on devices."

== Improvements ==
See Apu's comment in https://tracker.moodle.org/browse/MDL-36445?focusedCommentId=200343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-200343.
Specially the mention about language.

Moodle Mobile Push Notifications

2014-06-29T16:48:11Z

Dongsheng:

==Goal==
Moodle Mobile should receive Moodle messages as push notifications on iOS and Android.

== Use case and web services ==
[[Image:Screenshot_2_05_13_5_16_PM.png|thumb]]
[[Image:Screenshot_2_05_13_5_14_PM.png|thumb]]
=== currently implemented use case ===
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. The mobile app PushPlugin plugin does it.
# The user enters its Moodle credentials. The app gets an [http://airnotifier.github.io/airnotifier/ AirNotifier] access key from the site. (note that this access key is currently the same for each site => meaningless). The key is retrieved by calling the webservice message_airnotifier_get_access_key(permissions). create_token is the only possible permission at the moment.
# The app sends its token to [http://airnotifier.github.io/airnotifier/ AirNotifier] server - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io/airnotifier/ AirNotifier] provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# The user receive the notification.

=== brainstorming use case ===
# During [http://airnotifier.github.io/airnotifier/ AirNotifier] provider installation, the site contacts Airnotifer server to request an access key for itself + an access key for the devices (to send their device token to [http://airnotifier.github.io/airnotifier/ AirNotifier] server).
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token.
# The user enters its Moodle credentials. The app gets an [http://airnotifier.github.io/airnotifier/ AirNotifier] access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [http://airnotifier.github.io/airnotifier/ AirNotifier] server + the site url (so airnotifier knows the device/site couple, and so [http://airnotifier.github.io/airnotifier/ AirNotifier] can allow site broadcast) - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# During this web service, the site confirms the device token to [http://airnotifier.github.io/airnotifier/ AirNotifier]. It's done by a HTTP POST request. [http://airnotifier.github.io/airnotifier/ AirNotifier] server deletes unconfirmed device tokens older than one minute every hours.
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io/airnotifier/ AirNotifier] provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# Time to time, the [http://airnotifier.github.io/airnotifier/ AirNotifier] server checks Apple feedback and delete the unactive device tokens.
# When the [http://airnotifier.github.io/airnotifier/ AirNotifier] provider tries to send a message to a deleted device, it receive an error from [http://airnotifier.github.io/airnotifier/ AirNotifier] server and stores it. Then it sends a unique alert to the device user. The device is disabled on the user messaging settings. After a month, the cron processes delete the device.

=== another brainstorming use case ===
# All the sites (public or privately) already registered in Moodle.net will have automatically keys for using [http://airnotifier.github.io/airnotifier/ AirNotifier]
# When registering a new site in Moodle.net there will be a new option for enabling PUSH Notificiations (true by default)
# If your site is not registered and your Mobile Services were enabled you will see a message in the Moodle Notifications page indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your site is not registered and you enable Mobile Services you will see a message near to the "Enable Mobile Services" indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your installation has a very big number of users, during the registration process and in the Notifications page you will see a message indicating "The public PUSH Notification server is limited to Moodle sites with less than N (5k, 20k...) users"
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. A user can disable at any time notifications for his device
# Every time the user open the app, the app gets an [http://airnotifier.github.io/airnotifier/ AirNotifier] access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [http://airnotifier.github.io/airnotifier/ AirNotifier] server + the site url (so airnotifier knows the device/site couple, and so [http://airnotifier.github.io/airnotifier/ AirNotifier] can allow site broadcast) - it's done by a HTTP POST request. This is done only if your device token has changed.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename, siteURL) This is done every time a user opens the app
# Since the app is sending its device token every time a user opens the app, a cron job in the Moodle Site can delete old device tokens (so we can avoid the [http://airnotifier.github.io/airnotifier/ AirNotifier] feedback service in the first stage)
# Someone/Something triggers a Moodle message. [http://airnotifier.github.io/airnotifier/ AirNotifier] provider (Moodle site) sends a payload for a specific device token + specific site to airnotifier server - it's done by a HTTP POST
# If the user disable the PUSH Notification feature, a WS message_airnotifier_remove_user_device is called, also the [http://airnotifier.github.io/airnotifier/ AirNotifier] is requested to delete the user device

Benefits are:
-We can reuse the current registration process that is secure (as far as I know)
-The [http://airnotifier.github.io/airnotifier/ AirNotifier] API for register new sites will be exposed only to Moodle.net (not to all sites)
-It will be easier notify registered sites that are abusing (Moodle.net can communicate with sites (sending an automatic email to the user admin))
-Moodle will have statistics of sites using PUSH directly in Moodle.net (no need to ask [http://airnotifier.github.io/airnotifier/ AirNotifier])
-The registration process can be improved to retrieve the number of registered devices that are receiving notifications
-I think that it will "enforce" users to register their sites (for receive security notifications, etc..)
-Big companies with big Moodle installations usually doesn't register their Moodle sites, this will force these companies to use their own [http://airnotifier.github.io/airnotifier/ AirNotifier] server instance
-Since we have the number of users in a Moodle site, we can automatically disable [http://airnotifier.github.io/airnotifier/ AirNotifier] for big installations (more thant Nk users)
-I think that having all the information that the registration provides will help us to "monitor and prevent abuse" of the [http://airnotifier.github.io/airnotifier/ AirNotifier] server

== Payload content ==
The current payload content would need a bit of brainstorming. We need to identify what to send as it is extremely short. The app will behave following the content it receives.
=== Apple ===
The Apple payload is limited to 256bytes and it includes everything sent. Our current implementation is still quite of a wip:
<code php>
{type:'moodle_instantmessage',id:'8',urlparams:'{"user":"2","id":"8"}',userfrom:'Manager Moodle',date:'1367477362',alert:'This is a message text.',}
</code>
=== GCM ===
The Apple payload is about 4Kb. As it's more more permissive than Apple, it is not an issue. The content will be the same than Apple ones, just the format changes.

==Deliverables==
* push notification support on the mobile app - the app receives the push notification and behaves correctly.
* The [http://airnotifier.github.io/airnotifier/ AirNotifier] server - it sends notification to APNS / GCM.
* The [http://airnotifier.github.io/airnotifier/ AirNotifier] provider - it sends Moodle messages to [http://airnotifier.github.io/airnotifier/ AirNotifier] - MDL-36445

=== [http://airnotifier.github.io/airnotifier/ AirNotifier] messaging provider ===
In your messaging setting page you can select which of your devices can receive push notifications. MDL-36445
The provider has usual messaging provider settings.

=== [http://airnotifier.github.io/airnotifier/ AirNotifier] server ===
To install the server follow: https://github.com/airnotifier/airnotifier/wiki/Installation

Need to be done:
* feedback - MOBILE-191
* load stress
* IP banning - need to be tested MOBILE-192
* Site broadcast (site keys) MOBILE-395
* DDOS attacks
* iOS support - it works
* Android support MOBILE-396

=== App push notification support in the mobile app ===
MOBILE-168

==== The app is closed or is running in the background ====
It's a normal OS notification. The user can open the app from it. We need to define what happens when we open the app. See payload content section.

==== The app is opened ====
The app displays a notification popup. Or better, it's a small temporary notification message at the top that retarct by itself - less intrusive.

== Risks ==
These risks have been raised by Dan in MDL-36445:
* Have we load tested it with large numbers of users/notifications?
* What is stopping someone downloading Moodle getting the keys to DOS our push notification service (surely blacklisting us with apple)
* If someone gains access to device ids, could they then use our service to spam users?
* Are we checking the 'feedback service' and removing devices which no longer exist, apple ominously writes "Note: APNs monitors providers for their diligence in checking the feedback service and refraining from sending push notifications to nonexistent applications on devices."

== Improvements ==
See Apu's comment in https://tracker.moodle.org/browse/MDL-36445?focusedCommentId=200343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-200343.
Specially the mention about language.

Moodle Mobile Push Notifications

2014-06-29T16:45:34Z

Dongsheng: Undo revision 45603 by Dongsheng (talk)

==Goal==
Moodle Mobile should receive Moodle messages as push notifications on iOS and Android.

== Use case and web services ==
[[Image:Screenshot_2_05_13_5_16_PM.png|thumb]]
[[Image:Screenshot_2_05_13_5_14_PM.png|thumb]]
=== currently implemented use case ===
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. The mobile app PushPlugin plugin does it.
# The user enters its Moodle credentials. The app gets an AirNotifier access key from the site. (note that this access key is currently the same for each site => meaningless). The key is retrieved by calling the webservice message_airnotifier_get_access_key(permissions). create_token is the only possible permission at the moment.
# The app sends its token to AirNotifier server - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# The user receive the notification.

=== brainstorming use case ===
# During AirNotifier provider installation, the site contacts Airnotifer server to request an access key for itself + an access key for the devices (to send their device token to AirNotifier server).
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token.
# The user enters its Moodle credentials. The app gets an AirNotifier access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to AirNotifier server + the site url (so airnotifier knows the device/site couple, and so AirNotifier can allow site broadcast) - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# During this web service, the site confirms the device token to AirNotifier. It's done by a HTTP POST request. AirNotifier server deletes unconfirmed device tokens older than one minute every hours.
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# Time to time, the AirNotifier server checks Apple feedback and delete the unactive device tokens.
# When the AirNotifier provider tries to send a message to a deleted device, it receive an error from AirNotifier server and stores it. Then it sends a unique alert to the device user. The device is disabled on the user messaging settings. After a month, the cron processes delete the device.

=== another brainstorming use case ===
# All the sites (public or privately) already registered in Moodle.net will have automatically keys for using AirNotifier
# When registering a new site in Moodle.net there will be a new option for enabling PUSH Notificiations (true by default)
# If your site is not registered and your Mobile Services were enabled you will see a message in the Moodle Notifications page indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your site is not registered and you enable Mobile Services you will see a message near to the "Enable Mobile Services" indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your installation has a very big number of users, during the registration process and in the Notifications page you will see a message indicating "The public PUSH Notification server is limited to Moodle sites with less than N (5k, 20k...) users"
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. A user can disable at any time notifications for his device
# Every time the user open the app, the app gets an AirNotifier access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to AirNotifier server + the site url (so airnotifier knows the device/site couple, and so AirNotifier can allow site broadcast) - it's done by a HTTP POST request. This is done only if your device token has changed.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename, siteURL) This is done every time a user opens the app
# Since the app is sending its device token every time a user opens the app, a cron job in the Moodle Site can delete old device tokens (so we can avoid the AirNotifier feedback service in the first stage)
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token + specific site to airnotifier server - it's done by a HTTP POST
# If the user disable the PUSH Notification feature, a WS message_airnotifier_remove_user_device is called, also the AirNotifier is requested to delete the user device

Benefits are:
-We can reuse the current registration process that is secure (as far as I know)
-The AirNotifier API for register new sites will be exposed only to Moodle.net (not to all sites)
-It will be easier notify registered sites that are abusing (Moodle.net can communicate with sites (sending an automatic email to the user admin))
-Moodle will have statistics of sites using PUSH directly in Moodle.net (no need to ask AirNotifier)
-The registration process can be improved to retrieve the number of registered devices that are receiving notifications
-I think that it will "enforce" users to register their sites (for receive security notifications, etc..)
-Big companies with big Moodle installations usually doesn't register their Moodle sites, this will force these companies to use their own AirNotifier server instance
-Since we have the number of users in a Moodle site, we can automatically disable AirNotifier for big installations (more thant Nk users)
-I think that having all the information that the registration provides will help us to "monitor and prevent abuse" of the AirNotifier server

== Payload content ==
The current payload content would need a bit of brainstorming. We need to identify what to send as it is extremely short. The app will behave following the content it receives.
=== Apple ===
The Apple payload is limited to 256bytes and it includes everything sent. Our current implementation is still quite of a wip:
<code php>
{type:'moodle_instantmessage',id:'8',urlparams:'{"user":"2","id":"8"}',userfrom:'Manager Moodle',date:'1367477362',alert:'This is a message text.',}
</code>
=== GCM ===
The Apple payload is about 4Kb. As it's more more permissive than Apple, it is not an issue. The content will be the same than Apple ones, just the format changes.

==Deliverables==
* push notification support on the mobile app - the app receives the push notification and behaves correctly.
* The AirNotifier server - it sends notification to APNS / GCM.
* The AirNotifier provider - it sends Moodle messages to AirNotifier - MDL-36445

=== AirNotifier messaging provider ===
In your messaging setting page you can select which of your devices can receive push notifications. MDL-36445
The provider has usual messaging provider settings.

=== AirNotifier server ===
To install the server follow: https://github.com/airnotifier/airnotifier/wiki/Installation

Need to be done:
* feedback - MOBILE-191
* load stress
* IP banning - need to be tested MOBILE-192
* Site broadcast (site keys) MOBILE-395
* DDOS attacks
* iOS support - it works
* Android support MOBILE-396

=== App push notification support in the mobile app ===
MOBILE-168

==== The app is closed or is running in the background ====
It's a normal OS notification. The user can open the app from it. We need to define what happens when we open the app. See payload content section.

==== The app is opened ====
The app displays a notification popup. Or better, it's a small temporary notification message at the top that retarct by itself - less intrusive.

== Risks ==
These risks have been raised by Dan in MDL-36445:
* Have we load tested it with large numbers of users/notifications?
* What is stopping someone downloading Moodle getting the keys to DOS our push notification service (surely blacklisting us with apple)
* If someone gains access to device ids, could they then use our service to spam users?
* Are we checking the 'feedback service' and removing devices which no longer exist, apple ominously writes "Note: APNs monitors providers for their diligence in checking the feedback service and refraining from sending push notifications to nonexistent applications on devices."

== Improvements ==
See Apu's comment in https://tracker.moodle.org/browse/MDL-36445?focusedCommentId=200343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-200343.
Specially the mention about language.

Moodle Mobile Push Notifications

2014-06-29T16:45:03Z

Dongsheng:

==Goal==
Moodle Mobile should receive Moodle messages as push notifications on iOS and Android.

== Use case and web services ==
[[Image:Screenshot_2_05_13_5_16_PM.png|thumb]]
[[Image:Screenshot_2_05_13_5_14_PM.png|thumb]]
=== currently implemented use case ===
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. The mobile app PushPlugin plugin does it.
# The user enters its Moodle credentials. The app gets an [AirNotifier](http://airnotifier.github.io/airnotifier/) access key from the site. (note that this access key is currently the same for each site => meaningless). The key is retrieved by calling the webservice message_airnotifier_get_access_key(permissions). create_token is the only possible permission at the moment.
# The app sends its token to [AirNotifier](http://airnotifier.github.io/airnotifier/) server - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# Someone/Something triggers a Moodle message. [AirNotifier](http://airnotifier.github.io/airnotifier/) provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# The user receive the notification.

=== brainstorming use case ===
# During [AirNotifier](http://airnotifier.github.io/airnotifier/) provider installation, the site contacts Airnotifer server to request an access key for itself + an access key for the devices (to send their device token to [AirNotifier](http://airnotifier.github.io/airnotifier/) server).
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token.
# The user enters its Moodle credentials. The app gets an [AirNotifier](http://airnotifier.github.io/airnotifier/) access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [AirNotifier](http://airnotifier.github.io/airnotifier/) server + the site url (so airnotifier knows the device/site couple, and so [AirNotifier](http://airnotifier.github.io/airnotifier/) can allow site broadcast) - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# During this web service, the site confirms the device token to [AirNotifier](http://airnotifier.github.io/airnotifier/). It's done by a HTTP POST request. [AirNotifier](http://airnotifier.github.io/airnotifier/) server deletes unconfirmed device tokens older than one minute every hours.
# Someone/Something triggers a Moodle message. [AirNotifier](http://airnotifier.github.io/airnotifier/) provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# Time to time, the [AirNotifier](http://airnotifier.github.io/airnotifier/) server checks Apple feedback and delete the unactive device tokens.
# When the [AirNotifier](http://airnotifier.github.io/airnotifier/) provider tries to send a message to a deleted device, it receive an error from [AirNotifier](http://airnotifier.github.io/airnotifier/) server and stores it. Then it sends a unique alert to the device user. The device is disabled on the user messaging settings. After a month, the cron processes delete the device.

=== another brainstorming use case ===
# All the sites (public or privately) already registered in Moodle.net will have automatically keys for using [AirNotifier](http://airnotifier.github.io/airnotifier/)
# When registering a new site in Moodle.net there will be a new option for enabling PUSH Notificiations (true by default)
# If your site is not registered and your Mobile Services were enabled you will see a message in the Moodle Notifications page indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your site is not registered and you enable Mobile Services you will see a message near to the "Enable Mobile Services" indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your installation has a very big number of users, during the registration process and in the Notifications page you will see a message indicating "The public PUSH Notification server is limited to Moodle sites with less than N (5k, 20k...) users"
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. A user can disable at any time notifications for his device
# Every time the user open the app, the app gets an [AirNotifier](http://airnotifier.github.io/airnotifier/) access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to [AirNotifier](http://airnotifier.github.io/airnotifier/) server + the site url (so airnotifier knows the device/site couple, and so [AirNotifier](http://airnotifier.github.io/airnotifier/) can allow site broadcast) - it's done by a HTTP POST request. This is done only if your device token has changed.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename, siteURL) This is done every time a user opens the app
# Since the app is sending its device token every time a user opens the app, a cron job in the Moodle Site can delete old device tokens (so we can avoid the [AirNotifier](http://airnotifier.github.io/airnotifier/) feedback service in the first stage)
# Someone/Something triggers a Moodle message. [AirNotifier](http://airnotifier.github.io/airnotifier/) provider (Moodle site) sends a payload for a specific device token + specific site to airnotifier server - it's done by a HTTP POST
# If the user disable the PUSH Notification feature, a WS message_airnotifier_remove_user_device is called, also the [AirNotifier](http://airnotifier.github.io/airnotifier/) is requested to delete the user device

Benefits are:
-We can reuse the current registration process that is secure (as far as I know)
-The [AirNotifier](http://airnotifier.github.io/airnotifier/) API for register new sites will be exposed only to Moodle.net (not to all sites)
-It will be easier notify registered sites that are abusing (Moodle.net can communicate with sites (sending an automatic email to the user admin))
-Moodle will have statistics of sites using PUSH directly in Moodle.net (no need to ask [AirNotifier](http://airnotifier.github.io/airnotifier/))
-The registration process can be improved to retrieve the number of registered devices that are receiving notifications
-I think that it will "enforce" users to register their sites (for receive security notifications, etc..)
-Big companies with big Moodle installations usually doesn't register their Moodle sites, this will force these companies to use their own [AirNotifier](http://airnotifier.github.io/airnotifier/) server instance
-Since we have the number of users in a Moodle site, we can automatically disable [AirNotifier](http://airnotifier.github.io/airnotifier/) for big installations (more thant Nk users)
-I think that having all the information that the registration provides will help us to "monitor and prevent abuse" of the [AirNotifier](http://airnotifier.github.io/airnotifier/) server

== Payload content ==
The current payload content would need a bit of brainstorming. We need to identify what to send as it is extremely short. The app will behave following the content it receives.
=== Apple ===
The Apple payload is limited to 256bytes and it includes everything sent. Our current implementation is still quite of a wip:
<code php>
{type:'moodle_instantmessage',id:'8',urlparams:'{"user":"2","id":"8"}',userfrom:'Manager Moodle',date:'1367477362',alert:'This is a message text.',}
</code>
=== GCM ===
The Apple payload is about 4Kb. As it's more more permissive than Apple, it is not an issue. The content will be the same than Apple ones, just the format changes.

==Deliverables==
* push notification support on the mobile app - the app receives the push notification and behaves correctly.
* The [AirNotifier](http://airnotifier.github.io/airnotifier/) server - it sends notification to APNS / GCM.
* The [AirNotifier](http://airnotifier.github.io/airnotifier/) provider - it sends Moodle messages to [AirNotifier](http://airnotifier.github.io/airnotifier/) - MDL-36445

=== [AirNotifier](http://airnotifier.github.io/airnotifier/) messaging provider ===
In your messaging setting page you can select which of your devices can receive push notifications. MDL-36445
The provider has usual messaging provider settings.

=== [AirNotifier](http://airnotifier.github.io/airnotifier/) server ===
To install the server follow: https://github.com/airnotifier/airnotifier/wiki/Installation

Need to be done:
* feedback - MOBILE-191
* load stress
* IP banning - need to be tested MOBILE-192
* Site broadcast (site keys) MOBILE-395
* DDOS attacks
* iOS support - it works
* Android support MOBILE-396

=== App push notification support in the mobile app ===
MOBILE-168

==== The app is closed or is running in the background ====
It's a normal OS notification. The user can open the app from it. We need to define what happens when we open the app. See payload content section.

==== The app is opened ====
The app displays a notification popup. Or better, it's a small temporary notification message at the top that retarct by itself - less intrusive.

== Risks ==
These risks have been raised by Dan in MDL-36445:
* Have we load tested it with large numbers of users/notifications?
* What is stopping someone downloading Moodle getting the keys to DOS our push notification service (surely blacklisting us with apple)
* If someone gains access to device ids, could they then use our service to spam users?
* Are we checking the 'feedback service' and removing devices which no longer exist, apple ominously writes "Note: APNs monitors providers for their diligence in checking the feedback service and refraining from sending push notifications to nonexistent applications on devices."

== Improvements ==
See Apu's comment in https://tracker.moodle.org/browse/MDL-36445?focusedCommentId=200343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-200343.
Specially the mention about language.

Moodle Mobile Push Notifications

2014-06-29T16:38:35Z

Dongsheng:

==Goal==
Moodle Mobile should receive Moodle messages as push notifications on iOS and Android.

== Use case and web services ==
[[Image:Screenshot_2_05_13_5_16_PM.png|thumb]]
[[Image:Screenshot_2_05_13_5_14_PM.png|thumb]]
=== currently implemented use case ===
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. The mobile app PushPlugin plugin does it.
# The user enters its Moodle credentials. The app gets an AirNotifier access key from the site. (note that this access key is currently the same for each site => meaningless). The key is retrieved by calling the webservice message_airnotifier_get_access_key(permissions). create_token is the only possible permission at the moment.
# The app sends its token to AirNotifier server - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# The user receive the notification.

=== brainstorming use case ===
# During AirNotifier provider installation, the site contacts Airnotifer server to request an access key for itself + an access key for the devices (to send their device token to AirNotifier server).
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token.
# The user enters its Moodle credentials. The app gets an AirNotifier access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to AirNotifier server + the site url (so airnotifier knows the device/site couple, and so AirNotifier can allow site broadcast) - it's done by a HTTP POST request.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename)
# During this web service, the site confirms the device token to AirNotifier. It's done by a HTTP POST request. AirNotifier server deletes unconfirmed device tokens older than one minute every hours.
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token to airnotifier server - it's done by a HTTP POST
# Time to time, the AirNotifier server checks Apple feedback and delete the unactive device tokens.
# When the AirNotifier provider tries to send a message to a deleted device, it receive an error from AirNotifier server and stores it. Then it sends a unique alert to the device user. The device is disabled on the user messaging settings. After a month, the cron processes delete the device.

=== another brainstorming use case ===
# All the sites (public or privately) already registered in Moodle.net will have automatically keys for using AirNotifier
# When registering a new site in Moodle.net there will be a new option for enabling PUSH Notificiations (true by default)
# If your site is not registered and your Mobile Services were enabled you will see a message in the Moodle Notifications page indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your site is not registered and you enable Mobile Services you will see a message near to the "Enable Mobile Services" indicating that "You must register your site in order to be able to send PUSH Notifications"
# If your installation has a very big number of users, during the registration process and in the Notifications page you will see a message indicating "The public PUSH Notification server is limited to Moodle sites with less than N (5k, 20k...) users"
# The user opens the app and accepts to receive push notifications. The app contacts Apple push notification server and gets a device token. A user can disable at any time notifications for his device
# Every time the user open the app, the app gets an AirNotifier access key from the site. It's done calling the webservice message_airnotifier_get_access_key(type). An admin could request the site access key, the others only get the device access key.
# The app sends its device token to AirNotifier server + the site url (so airnotifier knows the device/site couple, and so AirNotifier can allow site broadcast) - it's done by a HTTP POST request. This is done only if your device token has changed.
# The app registers its device to the site - it's done by a ws call message_airnotifier_add_user_device(appname, devicetoken, devicename, siteURL) This is done every time a user opens the app
# Since the app is sending its device token every time a user opens the app, a cron job in the Moodle Site can delete old device tokens (so we can avoid the AirNotifier feedback service in the first stage)
# Someone/Something triggers a Moodle message. AirNotifier provider (Moodle site) sends a payload for a specific device token + specific site to airnotifier server - it's done by a HTTP POST
# If the user disable the PUSH Notification feature, a WS message_airnotifier_remove_user_device is called, also the AirNotifier is requested to delete the user device

Benefits are:
-We can reuse the current registration process that is secure (as far as I know)
-The AirNotifier API for register new sites will be exposed only to Moodle.net (not to all sites)
-It will be easier notify registered sites that are abusing (Moodle.net can communicate with sites (sending an automatic email to the user admin))
-Moodle will have statistics of sites using PUSH directly in Moodle.net (no need to ask AirNotifier)
-The registration process can be improved to retrieve the number of registered devices that are receiving notifications
-I think that it will "enforce" users to register their sites (for receive security notifications, etc..)
-Big companies with big Moodle installations usually doesn't register their Moodle sites, this will force these companies to use their own AirNotifier server instance
-Since we have the number of users in a Moodle site, we can automatically disable AirNotifier for big installations (more thant Nk users)
-I think that having all the information that the registration provides will help us to "monitor and prevent abuse" of the AirNotifier server

== Payload content ==
The current payload content would need a bit of brainstorming. We need to identify what to send as it is extremely short. The app will behave following the content it receives.
=== Apple ===
The Apple payload is limited to 256bytes and it includes everything sent. Our current implementation is still quite of a wip:
<code php>
{type:'moodle_instantmessage',id:'8',urlparams:'{"user":"2","id":"8"}',userfrom:'Manager Moodle',date:'1367477362',alert:'This is a message text.',}
</code>
=== GCM ===
The Apple payload is about 4Kb. As it's more more permissive than Apple, it is not an issue. The content will be the same than Apple ones, just the format changes.

==Deliverables==
* push notification support on the mobile app - the app receives the push notification and behaves correctly.
* The AirNotifier server - it sends notification to APNS / GCM.
* The AirNotifier provider - it sends Moodle messages to AirNotifier - MDL-36445

=== AirNotifier messaging provider ===
In your messaging setting page you can select which of your devices can receive push notifications. MDL-36445
The provider has usual messaging provider settings.

=== AirNotifier server ===
To install the server follow: https://github.com/airnotifier/airnotifier/wiki/Installation

Need to be done:
* feedback - MOBILE-191
* load stress
* IP banning - need to be tested MOBILE-192
* Site broadcast (site keys) MOBILE-395
* DDOS attacks
* iOS support - it works
* Android support MOBILE-396

=== App push notification support in the mobile app ===
MOBILE-168

==== The app is closed or is running in the background ====
It's a normal OS notification. The user can open the app from it. We need to define what happens when we open the app. See payload content section.

==== The app is opened ====
The app displays a notification popup. Or better, it's a small temporary notification message at the top that retarct by itself - less intrusive.

== Risks ==
These risks have been raised by Dan in MDL-36445:
* Have we load tested it with large numbers of users/notifications?
* What is stopping someone downloading Moodle getting the keys to DOS our push notification service (surely blacklisting us with apple)
* If someone gains access to device ids, could they then use our service to spam users?
* Are we checking the 'feedback service' and removing devices which no longer exist, apple ominously writes "Note: APNs monitors providers for their diligence in checking the feedback service and refraining from sending push notifications to nonexistent applications on devices."

== Improvements ==
See Apu's comment in https://tracker.moodle.org/browse/MDL-36445?focusedCommentId=200343&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-200343.
Specially the mention about language.

Moodle 2.3 release notes

2012-05-24T08:01:24Z

Dongsheng: /* Core API changes */

[[Releases]] > {{FULLPAGENAME}}

Release date: 18th June 2012 (expected)

Here is [http://tracker.moodle.org/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=project+%3D+mdl+AND+resolution+%3D+fixed+AND+fixVersion+in+%28%222.3%22%29+ORDER+BY+priority+DESC&runQuery=true&clear=true the full list of fixed issues in 2.3].

Many thanks to [http://moodle.org/dev/contributions.php?version=2.3.x everyone that worked on the new features in this release], particularly:

* Everyone at [http://moodle.com/hq/team Moodle HQ]
* Tim Hunt and Sam Marshall from [http://www.open.ac.uk/ The Open University]
* Damyon and Raymond from [http://www.netspot.com.au/ NetSpot]
* Andrew and Ruslan from [http://www.luns.net.uk/ Lancaster University Network Services]
* Davo Smith from [http://www.synergy-learning.com/ Synergy Learning]
* Main web services contributors: Fabio Souto, Juan Leyva, Paul Charsley, Yang Yang
* ...

===Major new features===

====Files usability====
(insert screenshots with realistic example files)

* Nicer-looking File Picker with fewer clicks
* Images now display as true thumbnails in the file picker and file manager
* Other files have pretty icons for most file types
* Files view can be easily toggled between icons view, or a table view with sizes and dates, or a hierarchical list view.
* You can now drag and drop files directly from your desktop straight into file areas!
* File info (eg license information, sizes, dates) can be easily edited and viewed in a popup dialog
* Files can be created as "aliases/shortcuts" of other files. This allows you to, for example, use a single file in your private files area multiple times in all your courses. If you update the original file then all the aliases will automatically update!
* Aliases are easily identifiable in the file manager interface.
(MDL-31907)

====Repository improvements====

* MDL-28666 - If a repository supports it then it's possible to make an alias/shortcut to a file in an external repository. If the file is updated in the repository, then this change is reflected in Moodle. The fie remains under Moodle access control however, and the original URL is not usually revealed. In the 2.3 core release, this is supported by Equella, Filesystem, user private, coursefiles and box.net repositories.
* The repository plugin is now able to take over the whole right-hand pane of the file picker and provide it's own searching/browsing interface.
* MDL-31675 - Server files repository now available for database, forum and glossary activities.

====Improvements to editing course pages ====

* MDL-31263 - Blocks drag and drop returns
* MDL-31052 - All AJAX editing on the course pages has been modernised and cleaned up. It's on by default now too.
* MDL-32508 - Sections can now be displayed to all users as "one section per page" (via a new [https://docs.moodle.org/23/en/Course_settings course setting 'Course layout']) with full navigation. Currently only supported by Weeks and Topics formats. (AKA death to scroll-of-death)
* MDL-30617 - An optional new popup [https://docs.moodle.org/23/en/Course_homepage "Activity chooser"] has been added with full introduction, examples and links about each activity or resource module.
* MDL-22504 - You can now drag files straight into the course page and they will be added as resources.
* MDL-31215 - You can edit the name of any activity without having to enter the activity settings (works particularly well with drag and drop).
* MDL-31121 - You can now add/remove sections direct from the course page.

====Assignment module====

* MDL-26997 -Complete rewrite of the assignment module from scratch, by [http://www.netspot.com.au/ NetSpot] (Moodle Partner in Australia)
* Assignment subtypes no longer needed
* MDL-31731 - New 'marking guide' advanced grading method, where a teacher enters a comment per criterion and a mark up to a maximum

====Book module====

* MDL-32709 - The most popular third-party resource module ever, [https://docs.moodle.org/23/en/Book_module Book], by our very own Petr Skoda, finally joins core. Welcome!

====Quiz module====

* MDL-3030 - More robust handling of quiz attempts that are not submitted by the deadline.
* MDL-3054 & MDL-11047 - There is now an option for teacher to force students to answer the quiz questions strictly in order. As part of this, the quiz remembers which page the student was last on, and will take them back there when they resume an attempt.

====SCORM module====
* MDL-29745 - new graph report plugin

====Workshop module====

* MDL-26099 - Option to make the workshop switch to the assessment phase automatically after the submissions deadline (including automatic allocation of submissions for assessment)
* MDL-25660 - Workshop submission deadlines are shown in the calendar
* MDL-27508 - Improved support for pagination and filtering workshop submissions by group
* MDL-32638 - Workshop supports file browsing via Server files repository (including improved access control when serving submission files)

====Available update notifications====

MDL-20438 - Admins can check for any updates available for core code and for any contributed plugins installed on the site (from the [http://moodle.org/plugins plugins directory]) via buttons on the notifications and plugins overview pages.

===Other highlights===

* MDL-31121 - Option in [https://docs.moodle.org/23/en/File_module_settings file resource settings] to display file size and/or type on course page
* MDL-32009 - Admin option for uninstalling messaging outputs and report of messaging output statuses on plugins overview page
* MDL-29941 - Admin option to enable a CSS optimiser that analyses and refactors CSS before caching it.

===Security issues===

All security issues that were fixed in 2.2.x and 2.1.x were also fixed in 2.3.

===For developers: API changes===

====Core API changes====
* MDL-31902 All xxx_get_participants() functions are removed from core
* As a part of MDL-32471, the signature of send_stored_file() has been [http://git.moodle.org/gw?p=moodle.git;a=commitdiff;h=796495fed29f12e4a81bd406558d8eeffd0e64ac modified]. The last two parameters $filename and $dontdie were replaced with a single array containing additional options for the file serving. The pluginfile callbacks in plugins are supposed to transfer these options from the caller to send_stored_file() - see the note below.
* MDL-28666 Files API changes, added ability to create file reference using file_storage::create_file_from_reference() method, update file record attributes using stored_file class
* MDL-28666 Repository API changes, added new APIs repository::get_file_reference(), repository::get_file_by_reference(), repository::get_reference_details(), repository::send_file(), the new APIs make serving files from external repository possible

====Plugin API changes====

* As a part of MDL-32471, the API of the plugin function xyz_pluginfile() has been extended. There is a new array parameter passed to these callbacks containing additional options for the file serving. The array should be re-passed to send_stored_file(). The change is pretty trivial - see [http://git.moodle.org/gw?p=moodle.git;a=commitdiff;h=261cbbacc15ef1732a357d689908c91c15e0617a examples].

====Webservice changes====
Few changes could break existing web service clients in 2.3 - untill this version we tried not to break anything. However these changes will make the client's developer life easier, so we prefered to do them now than later. Please take in consideration these improvements and retest your clients:
* [https://docs.moodle.org/dev/Errors_handling_in_web_services Error codes and Warnings]
* All text fields have an additional format field as parameter and return value (MDL-32581)
* Thanks to the increasing number of contributions, we improved our [https://docs.moodle.org/dev/How_to_contribute_a_web_service_function_to_core contributor web service guide]
* From 2.3, all web service functions integrated in master will land (when possible) in supported minor versions (e.g. 2.3.1, 2.3.2...).
* Many [http://tracker.moodle.org/browse/MDL-31253 fixes] and new [http://tracker.moodle.org/browse/MDL-29934 API functions].

====Unit tests====

We have switched completely to using [[PHPUnit]] for all our unit tests now. All existing simpletests have been rewritten, and new tests have been added.

We intend to move towards a completely unit-test-driven development methodology (where the tests are written first!) for significant new code, and we also encourage all developers to implement unit tests covering at least the core features of their code.

Moodle HQ run these tests on an automated basis for all new code submitted for integration, as well as on each weekly release.

==== Community hub changes ====
Some bug fixes and improvements in [http://tracker.moodle.org/browse/MDL-30247 core] and in the [http://tracker.moodle.org/browse/CONTRIB-3348 plugin]. Hub administrators must update their hub to the most recent version regarding CONTRIB-3646.

<noinclude>==See also==

* [https://docs.moodle.org/23/en/Category:New_features User documentation of new features in Moodle 2.3]
* [https://docs.moodle.org/23/en/Upgrading_to_Moodle_2.3 Upgrading to Moodle 2.3] - information for admins who are upgrading from earlier versions
*[[Moodle 2.2 release notes]]

[[Category:Release notes]]
[[Category:Moodle 2.3]]

[[fr:Notes de mise à jour de Moodle 2.3]]
</noinclude>

Moodle 2.3 release notes

2012-05-21T03:46:47Z

Dongsheng: /* Repository improvements */

[[Releases]] > {{FULLPAGENAME}}

Release date: 18th June 2012 (expected)

Here is [http://tracker.moodle.org/secure/IssueNavigator!executeAdvanced.jspa?jqlQuery=project+%3D+mdl+AND+resolution+%3D+fixed+AND+fixVersion+in+%28%222.3%22%29+ORDER+BY+priority+DESC&runQuery=true&clear=true the full list of fixed issues in 2.3].

Many thanks to [http://moodle.org/dev/contributions.php?version=2.3.x everyone that worked on the new features in this release], particularly:

* Everyone at [http://moodle.com/hq/team Moodle HQ]
* Tim Hunt and Sam Marshall from OU
* Damyon and Raymond from [http://www.netspot.com.au/ NetSpot]
* Andrew and Ruslan from [http://www.luns.net.uk/ Lancaster University Network Services]
* Davo Smith from [http://www.synergy-learning.com/ Synergy Learning]
* ...

===Major new features===

====Files usability====
(insert screenshots with realistic example files)

* Nicer-looking File Picker with fewer clicks
* Images now display as true thumbnails throughout Moodle
* Other files have pretty icons for most file types
* Files view can be easily toggled between icons view, or a table view with sizes and dates, or a hierarchical list view.
* You can now drag and drop files directly from your desktop straight into file areas!
* File info (eg license information, sizes, dates) can be easily edited and viewed in a popup dialog
* Files can be created as "aliases/shortcuts" of other files. This allows you to, for example, use a single file in your private files area multiple times in all your courses. If you update the original file then all the aliases will automatically update!
* Aliases are easily identifiable in the file manager interface.
(MDL-31907)

====Repository improvements====
* If a repository supports it then it's possible to make an alias/shortcut to a file in an external repository. If the file is updated in the repository, then this change is reflected in Moodle. The fie remains under Moodle access control however, and the original URL is not usually revealed. In the 2.3 core release, this is supported by Equella, Filesystem, user private, coursefiles, box.net repositories.
* The repository plugin is now able to take over the whole right-hand pane of the file picker and provide it's own searching/browsing interface.
(MDL-28666)

====Improvements to editing course pages ====
* MDL-31263 - Blocks drag and drop returns
* MDL-31052 - All AJAX editing on the course pages has been modernised and cleaned up. It's on by default now too.
* MDL-32508 - Sections can now be displayed to all users as "one section per page" (via a course setting) with full navigation. Currently only supported by Weeks and Topics formats. (AKA death to scroll-of-death)
* MDL-30617 - An optional new popup "Activity chooser" has been added with full introduction, examples and links about each activity or resource module.
* MDL-22504 - You can now drag files, links or even plain text straight into the course page and they will be added as resources.
* MDL-31215 - You can edit the name of any activity without having to enter the activity settings (works particularly well with drag and drop).
* MDL-31121 - You can now add/remove sections direct from the course page.

====Book module====
* The most popular third-party resource module ever, "Book" by our very own Petr Skoda, finally joins core. Welcome!
(MDL-32709)

====Assignment module====

* MDL-26997 -Complete rewrite of the assignment module from scratch, by [http://www.netspot.com.au/ NetSpot] (Moodle Partner in Australia)
* Assignment subtypes no longer needed
* MDL-31731 - New 'marking guide' advanced grading method, where a teacher enters a comment per criterion and a mark up to a maximum

====Quiz module====

* MDL-3030 - More robust handling of quiz attempts that are not submitted by the deadline.
* MDL-3054 & MDL-11047 - There is now an option for teacher to force students to answer the quiz questions strictly in order. As part of this, the quiz remembers which page the student was last on, and will take them back there when they resume an attempt.

====Available update notifications====

MDL-20438 - Admins can check for any updates available for core code and for any contributed plugins installed on the site (from the [http://moodle.org/plugins plugins directory]) via buttons on the notifications and plugins overview pages.

===Other highlights===

* MDL-26099 - Option to make the workshop switch to the assessment phase automatically after the submissions deadline (including automatic allocation of submissions for assessment)
* MDL-25660 - Workshop submission deadlines are shown in the calendar
* MDL-31121 - Option to display file size and/or type on course page for file resource
* MDL-32009 - Admin option for uninstalling messaging outputs and report of messaging output statuses on plugins overview page

===Security issues===

All security issues that were fixed in 2.2.x and 2.1.x were also fixed in 2.3.

===For developers: API changes===

====Core API changes====
* MDL-31902 All xxx_get_participants() functions are removed from core

====Plugin API changes====

====Webservice changes====
Few changes could break existing web service clients in 2.3 - untill this version we tried not to break anything. However these changes will make the client's developer life easier, so we prefered to do them now than later. Please take in consideration these improvements and retest your clients:
* [https://docs.moodle.org/dev/Errors_handling_in_web_services Error codes and Warnings]
* All text fields have an additional format field as parameter and return value (MDL-32581)
* Thanks to the increasing number of contributions, we improved our [https://docs.moodle.org/dev/How_to_contribute_a_web_service_function_to_core contributor web service guide]
* From 2.3, all web service functions integrated in master will land (when possible) in supported minor versions (e.g. 2.3.1, 2.3.2...).
* Many [http://tracker.moodle.org/browse/MDL-31253 fixes] and new [http://tracker.moodle.org/browse/MDL-29934 API functions].

====Unit tests====

We have switched completely to using [[PHPUnit]] for all our unit tests now. All existing simpletests have been rewritten, and new tests have been added.

We intend to move towards a completely unit-test-driven development methodology (where the tests are written first!) for significant new code, and we also encourage all developers to implement unit tests covering at least the core features of their code.

Moodle HQ will shortly be running these tests on an automated basis for all new code submitted for integration, as well as on each weekly release.

==== Community hub changes ====
Some bug fixes and improvements in [http://tracker.moodle.org/browse/MDL-30247 core] and in the [http://tracker.moodle.org/browse/CONTRIB-3348 plugin]. Hub administrators must update their hub to the most recent version regarding CONTRIB-3646.

<noinclude>==See also==

* [https://docs.moodle.org/23/en/Category:New_features User documentation of new features in Moodle 2.3]
* [https://docs.moodle.org/23/en/Upgrading_to_Moodle_2.3 Upgrading to Moodle 2.3] - information for admins who are upgrading from earlier versions
*[[Moodle 2.2 release notes]]

[[Category:Release notes]]
[[Category:Moodle 2.3]]

[[fr:Notes de mise à jour de Moodle 2.3]]
</noinclude>

Improved support for external File content

2012-05-10T10:47:14Z

Dongsheng: /* Repository API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = Dongsheng Cai
}}
{{Moodle 2.3}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep files' reference field, we also need to the original file information when creating draft file copies, it will be used by filemanager element to look up references to original file, to make this happen, we have to create a temp file only known by filemanager: mdl_files_draft_info, it has following fields:
** id
** draftfileid
** name
** value
So when filemanager wants to decide whether or not the draft file's original file has references, so firstly, filemanager looks up this table by providing draftfileid and name="originalfileparams", the value field will be the stored_file params, the filemanager will be able to ask Files API to provide all references. In this way, we don't have to inject Files API to know more information, it's only the duty of filemanager to get the original file.

* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files
* stored_file class
** set_author()
** set_license()
** replace_content_with(stored_file $storedfile)
** rename($filepath, $filename): rename files
** delete_references(): delete all reference information
** get_reference_details($ref): Get human readable reference information

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* public function send_file($storedfile, $lifetime=86400 , $filter=0, $forcedownload=false, array $options = null): Serve repository files

* repository::get_file_reference($str): Create the file reference

* repository::get_file_by_reference($ref): Get an external file by providing reference, $ref is the file record in files_reference table, it has reference, lastsync and lifetime fields, repository plugin could decide what to do based on this information, the return value will be an object:
** $fileinfo->handle, returns a file handler
** $fileinfo->contenthash, this returns an existing moodle file
** $fileinfo->content, returns file content
** $fileinfo->filepath, returns a file path
Repository API will handle different types of return automatically.

* repository::sync_individual_file(stored_file $file): Decide whether or not the stored_file instance should be synced

* repository::get_reference_details($ref): Convert the reference info to human readable format

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-05-10T10:30:38Z

Dongsheng: /* File API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = Dongsheng Cai
}}
{{Moodle 2.3}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep files' reference field, we also need to the original file information when creating draft file copies, it will be used by filemanager element to look up references to original file, to make this happen, we have to create a temp file only known by filemanager: mdl_files_draft_info, it has following fields:
** id
** draftfileid
** name
** value
So when filemanager wants to decide whether or not the draft file's original file has references, so firstly, filemanager looks up this table by providing draftfileid and name="originalfileparams", the value field will be the stored_file params, the filemanager will be able to ask Files API to provide all references. In this way, we don't have to inject Files API to know more information, it's only the duty of filemanager to get the original file.

* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files
* stored_file class
** set_author()
** set_license()
** replace_content_with(stored_file $storedfile)
** rename($filepath, $filename): rename files
** delete_references(): delete all reference information
** get_reference_details($ref): Get human readable reference information

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file): Serve repository files

* repository::get_file_reference($str): Get a file reference

* repository::get_file_by_reference($ref): Get an external file by providing reference

* repository::sync_individual_file(stored_file $file): Decide whether or not the stored_file instance should be synced

* repository::get_reference_details($ref): Convert the reference info to human readable format

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-05-08T08:21:18Z

Dongsheng: /* File API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = Dongsheng Cai
}}
{{Moodle 2.3}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files
* stored_file class
** set_author()
** set_license()
** replace_content_with(stored_file $storedfile)
** rename($filepath, $filename): rename files
** delete_references(): delete all reference information
** get_reference_details($ref): Get human readable reference information

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file): Serve repository files

* repository::get_file_reference($str): Get a file reference

* repository::get_file_by_reference($ref): Get an external file by providing reference

* repository::sync_individual_file(stored_file $file): Decide whether or not the stored_file instance should be synced

* repository::get_reference_details($ref): Convert the reference info to human readable format

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-05-07T09:56:33Z

Dongsheng: /* Repository API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = Dongsheng Cai
}}
{{Moodle 2.3}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files
* stored_file class
** set_author()
** set_license()
** replace_content_with(stored_file $storedfile)
** rename($filepath, $filename): rename files
** delete_references(): delete all reference information

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file): Serve repository files

* repository::get_file_reference($str): Get a file reference

* repository::get_file_by_reference($ref): Get an external file by providing reference

* repository::sync_individual_file(stored_file $file): Decide whether or not the stored_file instance should be synced

* repository::get_reference_details($ref): Convert the reference info to human readable format

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-05-07T09:18:38Z

Dongsheng: /* File API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = Dongsheng Cai
}}
{{Moodle 2.3}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files
* stored_file class
** set_author()
** set_license()
** replace_content_with(stored_file $storedfile)
** rename($filepath, $filename): rename files
** delete_references(): delete all reference information

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Repository API

2012-04-26T08:20:16Z

Dongsheng: /* supported_returntypes() */

{{Moodle_2.0}}


The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions to ask, problems to report or major changes to suggest, please add them to the [[Development_talk:Repository_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavor to merge all such suggestions into the further development and fix all kinds of problems.


Note that parts of this document have been now split off into a separate [[File_API]]

==Objectives==

# Allow all Moodle users to easily bring content into Moodle from external repositories
# Provide a consistent interface to any external repository, for any Moodle module

==Use cases==

===Teacher adding an external file as a new resource===

# Teacher wants to add a new resource to a course
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# File is COPIED into Moodle and stored by the resource module
# File is marked as owned by that user
# Whenever someone wants to view that file, the resource module controls access (see [[File API]] )

===Teacher linking to an external file as a new resource (think video repository) ===

# Teacher wants to display a file in the repository
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# Link to the file is COPIED into Moodle and stored by the resource module
# Link is marked as owned by that user
# Whenever someone wants to follow that link, the resource module controls access (see [[File API]] )

===Student submitting an assignment===
# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories ([https://docs.moodle.org/en/Image:Filepicker_login.jpg file picker login], [https://docs.moodle.org/en/Image:Filepicker_browser.jpg file picker browser], [https://docs.moodle.org/en/Image:Filepicker_search.jpg file picker search])
# Student chooses MySpace from the list
# Student is prompted to enter MySpace username/password (if admin allows it, a checkbox could be there to "remember this for next time" but remember security)
# Student sees their files in MySpace and chooses one or more
# Files are copied from MySpace to Moodle
# Assignment module controls the permissions so that only the Student and assignment graders can see the file (other students would not have permission).

===Student attaching an image to a forum===
# Student needs to attach an image and presses the "Choose files" button in the posting screen
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses Mahara from the list
# Student is prompted to enter Mahara username/password
# Student sees their files in Mahara and chooses one image
# Image is copied to Moodle
# Image file is attached to forum post by Forum module (by reference)
# Forum module controls permissions so that anyone who can read that forum can see that file

===Student attaching the same image in another forum===

# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses "Local files" from the list and sees all the files they've permission to use
# A COPY of the image file is attached to forum post by Forum module
# Forum module controls access to this file.

Please add more use cases in this same format

==Mock screenshots==
When you first call up the file picker and choose a repository, you might be asked to log in (if saving of passwords is not allowed):

[[Image:Filepicker_login.jpg]]

Browsing files could look something like this:

[[Image:Filepicker_browser.jpg]]

And you can also search:

[[Image:Filepicker_search.jpg]]

==General architecture==

Each repository plugin (a standard Moodle plugin stored under /repository/xxx) will subclass the standard API and override methods specific to that repository.

As is usual in Moodle, there will be admin settings to disable/enable certain repository plugins as standard, as well as user settings so that users can add their own personal repositories to the standard list (eg [http://briefcase.yahoo.com Yahoo Briefcase] or [http://docs.google.com Google Docs]) and to select their default repository.

Once a repository has been used the file will usually be copied into Moodle there and then. However there will also be options to:
* only return the URL to the file if it's desired to keep it external (but this does present security and integrity risks), or
* refresh the local file copy regularly and automatically
* refresh the file manually if desired

Once in Moodle, it is subject to the [[File API]] for access control like any other file.

==Repository requirements==

From the Moodle point of view, each repository is just a hierarchy of nodes.

The repository MUST provide:
# A URI to download each node (eg file).
# A list of the nodes (eg files and directories) under a given node (eg directory). This allows Moodle to construct a standard browse interface (much like a standard OS file picker).

The repository can OPTIONALLY:
# Require some authentication credentials
# Provide more metadata about each node (mime type, size, dates, related files, dublin core stuff, etc)
# Describe a search facility (so that Moodle can construct a search form)
# Provide copyright and usage rules (or just information about the rules)

==Repository plugins==

Some plugins I'd like to see developed for the first version are:
* box - an interface to [http://box.net box.net]
* mahara - an interface to a Mahara installation
* Server Files - very similar to the current course-based file manager, except user-based
* Remote Moodle - an interface to another Moodle site, accessed over a secure mnet connection
* googledocs - an interface to [http://docs.google.com Google Docs]
* s3 - an interface to [http://www.amazon.com/gp/browse.html?node=16427261 Amazon S3]
* flickr - an interface to [http://flickr.com flickr]
* WebDAV - to access arbitrary external WebDAV servers
* merlot - an interface to the learning materials in [http://www.merlot.org/merlot/materials.htm Merlot.org]
* File System - a plugin to list files on local file system, of course, you can mount remote files to this local directory
* youtube - an interface to [http://youtube.com YouTube]
* jsr170 - an interface that can talk to anything that supports jsr170 (eg [http://www.alfresco.com/ Alfresco])
* oki - an OKI emulator allowing us to access things with OKI interfaces,like [http://www.fedora.info/ Fedora]
* briefcase - an interface to [http://briefcase.yahoo.com/ Yahoo Briefcase]
* myspace - an interface to MySpace files (perhaps via [http://www.programmableweb.com/api/myspace this MySpace API])
* skydrive - an interface to Microsoft's [http://skydrive.live.com/ SkyDrive] files
* Dropbox - an interface to Dropbox files [http://www.dropbox.com]
* facebook - an interface to Facebook files
* [http://www.dspace.org/ Dspace] - a repository from MIT
* DOOR - another popular open source repository
* SMB shares - An interface for windows shares e.g. personal folders on network drives. Would need to link with LDAP as usernames will often be wholly/partially the same as network folder names. This could be done using SAMBA, but would also need to work on windows machines natively. See [http://moodle.org/mod/data/view.php?d=13&rid=991 this block] for a linux implementation.

==Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

===File types===

The context at which someone is inserting a file may require certain file types (eg uploading a new user profile image is only looking for images).

To support this, the calling code needs to be able to specify the required mimetypes, and the listing code should be able to filter the results based on these mimetypes. Ideally the repository itself can do the filtering for ultimate speed (though not all repositories will support this).

We will have to develop special new mimetypes for Moodle files like backups (application/vnd.moodle.backup) and IMS learning design (application/vnd.moodle.imsld) etc

==Technical walkthrough==

(See also the functional spec for the [[Repository_File_Picker]] )

There are two main cases where the repository API will be used: as part of a Moodleform to add a file and as part of the HTML editor to add a media element into some HTML). We also have to cater for the using Moodleforms without Javascript.

In all of these cases the files will be uploaded to Moodle while using the file picker dialog and stored in a temporary file area owned by the currently active user. It is only AFTER the submission of the entire Moodleform that we will know the full context, itemids to store the file properly, so at this time the file will be copied into the correct filearea.

===Case 1: As part of a Moodleform with Javascript===

1. Moodle module code calls a "filepicker" moodleform item whenever a file is required, which includes the following information to pass to the File API:

eg $mform->addElement('filepicker', 'uniqueelementid', $fullname, $data)

2. When rendering the form, Moodle will display a read-only filename field with an '''"Add file"''' button next to it. There will also be a hidden field to store a file reference later (this is what actually gets used, the filename field is just for users to see something).

3. When the add file button is pressed, the form will be "replaced" in the page by a larger resizeable area containing an AJAX file picker. (After picking the display can be closed). (There could be a user option to make this a popup window instead, if required)

4. The AJAX file picker interface will list all the active repositories as a menu, and list files in one of several formats (like Windows/Mac/Linux): Details, Names, Icons.

5. For each plugin, the AJAX interface will prompt the user to login first (if required) asking the plugin to log in behind the scenes. It'll also ask the plugin to return listing data in response to clicks and searches.

6. Finally, when the user selects a file and clicks the "Select" button, the AJAX interface will trigger a method in the plugin that will fetch the file and call the File Storage API to '''store''' the file using the '''uniqueelementid''' and the current user info. While this is happening, the interface should show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

7. After a file has finally been selected we will have a file ID which we can pass back to the original Moodle form (to the hidden field named '''uniqueelementid_formid'''). The picker can then rename the read-only filename field before it hides itself.

8. Submitting the form will trigger the mform processing for this field, which will check fields, create things in the module etc. Once this has been finally successful the developer must call an mform function to "fix" the info for each file and "move" it into the module file area:

eg $mform->store_local_file('uniqueelementid', $context, $filearename, $itemid, $filepath);

9. Cron jobs in File Storage api should automatically delete any files in the user's tempfile area that are older than 7 days or move them into a trash can in the user's file area (perhaps).

===Case 2: As part of a Moodleform without Javascript===

Steps 1-2 are the same as for the case with Javascript.

3. The add file button is a submit button for the form with a different value. When the add file button is pressed,
* the whole form will be ''submitted'' to the original location (but with a different submit button value)
* moodleforms get_data() will detect this is a "repository save" and can save the full POST info in the current session tagged with the id of the openfile element, together with the URL to return to
* moodleforms get_data() then redirects the user to a new page showing the main picker interface

4. The file picker interface will have to be a completely new and separate interface from the AJAX one. It could be a long hierarchy listing, or reload a lot.

5. Finally, when the user selects a file and submits using the "Select" button to picker.php, it will trigger a method in the plugin that will fetch the file and call the [[File_API|File API]] to store the file using the filearea and context information we already had. While this is happening, the interface can show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

6. After this, picker.php will redirect/continue back to the original form page. The form can be constructed as usual, however, when the form is rendered using display() method moodleforms should now look for relevant saved content in the session and use that to override any content in the form (and then delete the saved info in the session).

Steps 8-9 are the same as for the case with Javascript.

===Case 3: As part of a HTML editor===

The key thing here is a move away from storing any absolute URLs to files in our HTML texts. Instead we'll store relative names.

1. The moodleform for a textarea (HTML editor) will require a path to the filearea associated with this HTML. eg '''wwwroot/pluginfile.php/13/content/0/'''. This would have to be the user_draft area if the filearea doesn't exist yet eg '''wwwroot/draftfile.php/userid/tempfile/uniquelementid'''

2. All textarea content will also need to have str_replace done on it to replace @@pluginfile@@/somefilenames.jpg in the content to use this path so that it comes up right in the editor. (Note this also needs to be done on every format_text command too when showing this text.)

3. The path parameter also needs to be added to the editor configuration in the current page.

4. Editor plugins can be modified to look for these variables in the editor configuration.

5. When adding an image or other media element, the same AJAX repository picker will show up as a popup div to allow people to pick from any repository and choose files to download. The repository picker is responsible for downloading the file in real-time, storing it as a user temporary file if the filearea doesn't already exist, prefixing the supplied path to the filename and returning a URL back to the dialog text input before closing.

6. On submission, and after the HTML is stored, we might now have a new permanent filearea, so we'll need to update any associated temporary files to make sure they have the proper file area information.

==Repository plugins==

===Required elements===

Each repository plugin is required to contain the following elements:

====class repository()====

This class implements the interface to a particular repository, for browsing, selecting and updating files. The base class (repository) is defined in /repository/lib.php, while each repository defines an inherited class (eg repository_alfresco) in /repository/repositoryname/repository.class.php

===Optional elements===

Repositories can redefine any of these methods as required (and in some instances, MUST redefine them):

====__construct($repositoryid, $contextid, $options=array(), $readonly)====

'''MUST''' redefine

Accept necessary parameters, and do initialization of repository.

====get_file($url, $file = '')====

Given a URL, download a file from there, save the file in a temporary directory.

====get_link($info)====
Get the url of external resource

====get_listing($path='/', $page='''''''')====

Given a path, and perhaps a search, get a listing of files. In the case of AJAX file picker, this function should return json format Javascript array.

====search($keyword)====
Search repository by given keyword, it will return an array of the same format of get_listing

====print_login()====

Show the login screen, if required. In the case of AJAX file picker, this function should return json format array which defined the login form.

====print_search====

Print the search form, it will return a json string

====get_meta()====
Return information for creating ajax request, it is private function, you don't need to rewrite it.

====create()====
Create an instance

====delete()====
Delete this instance from `repository` table

====hide()====
Hide a repository instance from file picker list

====set_option()====
set options in data1-data5 fields, can be overrided

====get_option()====
get option list or a specific option from database

====get_type_option_names()====
If this plugin needs admin settings, please refine this function to return option names.

====type_config_form()====
If get_type_option_names return non empty array, this function '''MUST''' redefine, it will help to build the setting form.

====type_form_validation()====
This function can be used for validating the data submitted by plugin setting form.

====get_instance_option_names()====
If plugin instance needs settings, this function will return instance option names.

====instance_config_form()====
If get_instance_option_names return non empty array, this function '''MUST''' redefine, it will help to build the instance setting form.

====instance_form_validation()====
This function can be used for validating the data submitted by instance setting form.

====supported_filetypes()====
What file types are supported by this repository plugin, it will return an array, the file type name is defined in a [http://freemind.sourceforge.net/wiki/index.php/Main_Page freemind] file in lib/file/file_types.mm

====supported_returntypes()====
The repository plugin could return external link, copy files to moodle or create file references to external repository. If the plugin supports file links only, developer should override this function to return FILE_EXTERNAL, if the plugin supports copying files only, it should return FILE_INTERNAL, if supports creating file references only, it returns FILE_REFERENCE. If repository supports more than one return types, use bit operation, for example <code php>return FILE_INTERNAL | FILE_REFERENCE;</code>

On the other hand, form elements, such as filemanager and filepicker can indicate what types of file can be accepted, for example, in assignment module, only copying file is accepted (to prevent cheating), so when adding filemanager element to moodle form, we use this option: <code php>$options['return_types'] = FILE_INTERNAL;</code>

Moodle 2.3 onward, filemanager's default return_types is FILE_INTERNAL | FILE_REFERENCE

====filter()====
Filter file listing to exclude specific file types

===icon.png===

A logo that represents the repository. Ideally square but we should handle all sizes.

==See also==

* [[Repository Administration Specification]]
* [[Repository Interface for Moodle/Course/User]]
* [[Repository plugins]]
* [[Repository File Picker]]
* [[File API]]
* [[Portfolio API]]
* MDL-13766 and MDL-16543 Repository API Meta issues

[[Category:Repositories]]
[[Category:Interfaces]]

Repository API

2012-04-26T08:17:25Z

Dongsheng: /* supported_returntypes() */

{{Moodle_2.0}}


The page is open for everyone so everyone can help correct mistakes and help with the evolution of this document. However, if you have questions to ask, problems to report or major changes to suggest, please add them to the [[Development_talk:Repository_API|page comments]], or start a discussion in the [http://moodle.org/mod/forum/view.php?id=1807 Repositories forum]. We'll endeavor to merge all such suggestions into the further development and fix all kinds of problems.


Note that parts of this document have been now split off into a separate [[File_API]]

==Objectives==

# Allow all Moodle users to easily bring content into Moodle from external repositories
# Provide a consistent interface to any external repository, for any Moodle module

==Use cases==

===Teacher adding an external file as a new resource===

# Teacher wants to add a new resource to a course
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# File is COPIED into Moodle and stored by the resource module
# File is marked as owned by that user
# Whenever someone wants to view that file, the resource module controls access (see [[File API]] )

===Teacher linking to an external file as a new resource (think video repository) ===

# Teacher wants to display a file in the repository
# Teacher clicks the "Choose a resource" button
# Teacher is presented with a simple file picker to choose a file (with a menu to switch between multiple configured repositories)
# Teacher chooses a file in an external repository
# Link to the file is COPIED into Moodle and stored by the resource module
# Link is marked as owned by that user
# Whenever someone wants to follow that link, the resource module controls access (see [[File API]] )

===Student submitting an assignment===
# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories ([https://docs.moodle.org/en/Image:Filepicker_login.jpg file picker login], [https://docs.moodle.org/en/Image:Filepicker_browser.jpg file picker browser], [https://docs.moodle.org/en/Image:Filepicker_search.jpg file picker search])
# Student chooses MySpace from the list
# Student is prompted to enter MySpace username/password (if admin allows it, a checkbox could be there to "remember this for next time" but remember security)
# Student sees their files in MySpace and chooses one or more
# Files are copied from MySpace to Moodle
# Assignment module controls the permissions so that only the Student and assignment graders can see the file (other students would not have permission).

===Student attaching an image to a forum===
# Student needs to attach an image and presses the "Choose files" button in the posting screen
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses Mahara from the list
# Student is prompted to enter Mahara username/password
# Student sees their files in Mahara and chooses one image
# Image is copied to Moodle
# Image file is attached to forum post by Forum module (by reference)
# Forum module controls permissions so that anyone who can read that forum can see that file

===Student attaching the same image in another forum===

# Student needs to submit an assignment and presses the "Choose files" button
# Student sees a "file picker" where they can see files listed on any of several configured repositories
# Student chooses "Local files" from the list and sees all the files they've permission to use
# A COPY of the image file is attached to forum post by Forum module
# Forum module controls access to this file.

Please add more use cases in this same format

==Mock screenshots==
When you first call up the file picker and choose a repository, you might be asked to log in (if saving of passwords is not allowed):

[[Image:Filepicker_login.jpg]]

Browsing files could look something like this:

[[Image:Filepicker_browser.jpg]]

And you can also search:

[[Image:Filepicker_search.jpg]]

==General architecture==

Each repository plugin (a standard Moodle plugin stored under /repository/xxx) will subclass the standard API and override methods specific to that repository.

As is usual in Moodle, there will be admin settings to disable/enable certain repository plugins as standard, as well as user settings so that users can add their own personal repositories to the standard list (eg [http://briefcase.yahoo.com Yahoo Briefcase] or [http://docs.google.com Google Docs]) and to select their default repository.

Once a repository has been used the file will usually be copied into Moodle there and then. However there will also be options to:
* only return the URL to the file if it's desired to keep it external (but this does present security and integrity risks), or
* refresh the local file copy regularly and automatically
* refresh the file manually if desired

Once in Moodle, it is subject to the [[File API]] for access control like any other file.

==Repository requirements==

From the Moodle point of view, each repository is just a hierarchy of nodes.

The repository MUST provide:
# A URI to download each node (eg file).
# A list of the nodes (eg files and directories) under a given node (eg directory). This allows Moodle to construct a standard browse interface (much like a standard OS file picker).

The repository can OPTIONALLY:
# Require some authentication credentials
# Provide more metadata about each node (mime type, size, dates, related files, dublin core stuff, etc)
# Describe a search facility (so that Moodle can construct a search form)
# Provide copyright and usage rules (or just information about the rules)

==Repository plugins==

Some plugins I'd like to see developed for the first version are:
* box - an interface to [http://box.net box.net]
* mahara - an interface to a Mahara installation
* Server Files - very similar to the current course-based file manager, except user-based
* Remote Moodle - an interface to another Moodle site, accessed over a secure mnet connection
* googledocs - an interface to [http://docs.google.com Google Docs]
* s3 - an interface to [http://www.amazon.com/gp/browse.html?node=16427261 Amazon S3]
* flickr - an interface to [http://flickr.com flickr]
* WebDAV - to access arbitrary external WebDAV servers
* merlot - an interface to the learning materials in [http://www.merlot.org/merlot/materials.htm Merlot.org]
* File System - a plugin to list files on local file system, of course, you can mount remote files to this local directory
* youtube - an interface to [http://youtube.com YouTube]
* jsr170 - an interface that can talk to anything that supports jsr170 (eg [http://www.alfresco.com/ Alfresco])
* oki - an OKI emulator allowing us to access things with OKI interfaces,like [http://www.fedora.info/ Fedora]
* briefcase - an interface to [http://briefcase.yahoo.com/ Yahoo Briefcase]
* myspace - an interface to MySpace files (perhaps via [http://www.programmableweb.com/api/myspace this MySpace API])
* skydrive - an interface to Microsoft's [http://skydrive.live.com/ SkyDrive] files
* Dropbox - an interface to Dropbox files [http://www.dropbox.com]
* facebook - an interface to Facebook files
* [http://www.dspace.org/ Dspace] - a repository from MIT
* DOOR - another popular open source repository
* SMB shares - An interface for windows shares e.g. personal folders on network drives. Would need to link with LDAP as usernames will often be wholly/partially the same as network folder names. This could be done using SAMBA, but would also need to work on windows machines natively. See [http://moodle.org/mod/data/view.php?d=13&rid=991 this block] for a linux implementation.

==Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

===File types===

The context at which someone is inserting a file may require certain file types (eg uploading a new user profile image is only looking for images).

To support this, the calling code needs to be able to specify the required mimetypes, and the listing code should be able to filter the results based on these mimetypes. Ideally the repository itself can do the filtering for ultimate speed (though not all repositories will support this).

We will have to develop special new mimetypes for Moodle files like backups (application/vnd.moodle.backup) and IMS learning design (application/vnd.moodle.imsld) etc

==Technical walkthrough==

(See also the functional spec for the [[Repository_File_Picker]] )

There are two main cases where the repository API will be used: as part of a Moodleform to add a file and as part of the HTML editor to add a media element into some HTML). We also have to cater for the using Moodleforms without Javascript.

In all of these cases the files will be uploaded to Moodle while using the file picker dialog and stored in a temporary file area owned by the currently active user. It is only AFTER the submission of the entire Moodleform that we will know the full context, itemids to store the file properly, so at this time the file will be copied into the correct filearea.

===Case 1: As part of a Moodleform with Javascript===

1. Moodle module code calls a "filepicker" moodleform item whenever a file is required, which includes the following information to pass to the File API:

eg $mform->addElement('filepicker', 'uniqueelementid', $fullname, $data)

2. When rendering the form, Moodle will display a read-only filename field with an '''"Add file"''' button next to it. There will also be a hidden field to store a file reference later (this is what actually gets used, the filename field is just for users to see something).

3. When the add file button is pressed, the form will be "replaced" in the page by a larger resizeable area containing an AJAX file picker. (After picking the display can be closed). (There could be a user option to make this a popup window instead, if required)

4. The AJAX file picker interface will list all the active repositories as a menu, and list files in one of several formats (like Windows/Mac/Linux): Details, Names, Icons.

5. For each plugin, the AJAX interface will prompt the user to login first (if required) asking the plugin to log in behind the scenes. It'll also ask the plugin to return listing data in response to clicks and searches.

6. Finally, when the user selects a file and clicks the "Select" button, the AJAX interface will trigger a method in the plugin that will fetch the file and call the File Storage API to '''store''' the file using the '''uniqueelementid''' and the current user info. While this is happening, the interface should show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

7. After a file has finally been selected we will have a file ID which we can pass back to the original Moodle form (to the hidden field named '''uniqueelementid_formid'''). The picker can then rename the read-only filename field before it hides itself.

8. Submitting the form will trigger the mform processing for this field, which will check fields, create things in the module etc. Once this has been finally successful the developer must call an mform function to "fix" the info for each file and "move" it into the module file area:

eg $mform->store_local_file('uniqueelementid', $context, $filearename, $itemid, $filepath);

9. Cron jobs in File Storage api should automatically delete any files in the user's tempfile area that are older than 7 days or move them into a trash can in the user's file area (perhaps).

===Case 2: As part of a Moodleform without Javascript===

Steps 1-2 are the same as for the case with Javascript.

3. The add file button is a submit button for the form with a different value. When the add file button is pressed,
* the whole form will be ''submitted'' to the original location (but with a different submit button value)
* moodleforms get_data() will detect this is a "repository save" and can save the full POST info in the current session tagged with the id of the openfile element, together with the URL to return to
* moodleforms get_data() then redirects the user to a new page showing the main picker interface

4. The file picker interface will have to be a completely new and separate interface from the AJAX one. It could be a long hierarchy listing, or reload a lot.

5. Finally, when the user selects a file and submits using the "Select" button to picker.php, it will trigger a method in the plugin that will fetch the file and call the [[File_API|File API]] to store the file using the filearea and context information we already had. While this is happening, the interface can show some sort of progress bar (ideally) or at least a "loading file" image/sign/message.

6. After this, picker.php will redirect/continue back to the original form page. The form can be constructed as usual, however, when the form is rendered using display() method moodleforms should now look for relevant saved content in the session and use that to override any content in the form (and then delete the saved info in the session).

Steps 8-9 are the same as for the case with Javascript.

===Case 3: As part of a HTML editor===

The key thing here is a move away from storing any absolute URLs to files in our HTML texts. Instead we'll store relative names.

1. The moodleform for a textarea (HTML editor) will require a path to the filearea associated with this HTML. eg '''wwwroot/pluginfile.php/13/content/0/'''. This would have to be the user_draft area if the filearea doesn't exist yet eg '''wwwroot/draftfile.php/userid/tempfile/uniquelementid'''

2. All textarea content will also need to have str_replace done on it to replace @@pluginfile@@/somefilenames.jpg in the content to use this path so that it comes up right in the editor. (Note this also needs to be done on every format_text command too when showing this text.)

3. The path parameter also needs to be added to the editor configuration in the current page.

4. Editor plugins can be modified to look for these variables in the editor configuration.

5. When adding an image or other media element, the same AJAX repository picker will show up as a popup div to allow people to pick from any repository and choose files to download. The repository picker is responsible for downloading the file in real-time, storing it as a user temporary file if the filearea doesn't already exist, prefixing the supplied path to the filename and returning a URL back to the dialog text input before closing.

6. On submission, and after the HTML is stored, we might now have a new permanent filearea, so we'll need to update any associated temporary files to make sure they have the proper file area information.

==Repository plugins==

===Required elements===

Each repository plugin is required to contain the following elements:

====class repository()====

This class implements the interface to a particular repository, for browsing, selecting and updating files. The base class (repository) is defined in /repository/lib.php, while each repository defines an inherited class (eg repository_alfresco) in /repository/repositoryname/repository.class.php

===Optional elements===

Repositories can redefine any of these methods as required (and in some instances, MUST redefine them):

====__construct($repositoryid, $contextid, $options=array(), $readonly)====

'''MUST''' redefine

Accept necessary parameters, and do initialization of repository.

====get_file($url, $file = '')====

Given a URL, download a file from there, save the file in a temporary directory.

====get_link($info)====
Get the url of external resource

====get_listing($path='/', $page='''''''')====

Given a path, and perhaps a search, get a listing of files. In the case of AJAX file picker, this function should return json format Javascript array.

====search($keyword)====
Search repository by given keyword, it will return an array of the same format of get_listing

====print_login()====

Show the login screen, if required. In the case of AJAX file picker, this function should return json format array which defined the login form.

====print_search====

Print the search form, it will return a json string

====get_meta()====
Return information for creating ajax request, it is private function, you don't need to rewrite it.

====create()====
Create an instance

====delete()====
Delete this instance from `repository` table

====hide()====
Hide a repository instance from file picker list

====set_option()====
set options in data1-data5 fields, can be overrided

====get_option()====
get option list or a specific option from database

====get_type_option_names()====
If this plugin needs admin settings, please refine this function to return option names.

====type_config_form()====
If get_type_option_names return non empty array, this function '''MUST''' redefine, it will help to build the setting form.

====type_form_validation()====
This function can be used for validating the data submitted by plugin setting form.

====get_instance_option_names()====
If plugin instance needs settings, this function will return instance option names.

====instance_config_form()====
If get_instance_option_names return non empty array, this function '''MUST''' redefine, it will help to build the instance setting form.

====instance_form_validation()====
This function can be used for validating the data submitted by instance setting form.

====supported_filetypes()====
What file types are supported by this repository plugin, it will return an array, the file type name is defined in a [http://freemind.sourceforge.net/wiki/index.php/Main_Page freemind] file in lib/file/file_types.mm

====supported_returntypes()====
The repository plugin could support external link, copying files to moodle, create a file reference to external repository. If the plugin support file link only, developer should override this function to return FILE_EXTERNAL, if plugin support copying file only, it should return FILE_INTERNAL, if support create file reference only, it returns FILE_REFERENCE. If repository support more than one return types, use bit operation, for example <code php>return FILE_INTERNAL | FILE_REFERENCE;</code>

On the other hand, form elements filemanager and filepicker can indicate what types of file are accepted, for example, in assignment module, only FILE_INTERNAL accepted (to prevent cheating), so when adding filemanager element: <code php>$options['return_types'] = FILE_INTERNAL;</code>, moodle 2.3 onward, filemanager's default return_types is FILE_INTERNAL | FILE_REFERENCE

====filter()====
Filter file listing to exclude specific file types

===icon.png===

A logo that represents the repository. Ideally square but we should handle all sizes.

==See also==

* [[Repository Administration Specification]]
* [[Repository Interface for Moodle/Course/User]]
* [[Repository plugins]]
* [[Repository File Picker]]
* [[File API]]
* [[Portfolio API]]
* MDL-13766 and MDL-16543 Repository API Meta issues

[[Category:Repositories]]
[[Category:Interfaces]]

Improved support for external File content

2012-03-06T09:41:24Z

Dongsheng: /* File picker walk-through */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will create a file reference, if selected file is already an alias, repository API will copy this alias, not create an alias to alias, repository plugin can cache external files in moodle local directory, but this is optional
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-03-06T09:28:24Z

Dongsheng: /* Solution summary */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Update local caches

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-03-06T09:27:37Z

Dongsheng: /* Repository API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-03-06T09:27:17Z

Dongsheng: /* File API changes */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-03-06T09:26:44Z

Dongsheng: /* Example use cases that will become possible */

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# If teacher picked a file which is an alias already, moodle will copy the alias, not create an alias to alias.

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* stored_file::delete() should detect repository files, and process it properly, and implement a method to update file, not delete then create
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Files usability 2.3

2012-02-20T08:50:07Z

Dongsheng: /* Solution C5: Simplify "Server Files" */

{{Infobox Project
|name = Files Usability 2.3
|state = Specifications, feedback sought
|tracker = TBA
|discussion =
|assignee = Moodle HQ DEV team
}}
{{Moodle 2.3}}

==Background==

The file handling of Moodle 1.x was a very simple system with basically one server directory of files per course and very basic access controls. This system had the benefits of being simple, which meant it was easy to understand and to perform hacks around, but it's naivety had a lot of disadvantages in security, consistency, disk use, activity portability, and in some cases even led to dataloss. It also encouraged a certain workflow of "dumping" huge amounts of content into the course in an unstructured way.

The Files system was re-designed in Moodle 2.0 to solve these problems by introducing a model where files are directly associated with texts and "file areas" within different plugins in Moodle, and access to them is finely controlled by those same plugins. For some examples, the assignment module is now able to control access to assignment submissions depending on due dates and so on. Backups now accurately contain all the files they need, and multiple copies of the same file take up no more disk space than a single copy. Files can be drawn as easily from external repositories as from your own computer. The files system in Moodle 2.0 is more capable and detailed than pretty much any other system around.

However, the interfaces to CONTROL all this had to increase significantly in complexity, and this, combined with limited development time and technical constraints in web browsers have led to a large drop in usability. Not only do Moodle users have to contend with a different mindset when adding their resources to Moodle, but they also had to contend with a long list of unfamiliar interface issues.

This project aims to make large improvements in the interface when dealing with files in Moodle and improve usability significantly in Moodle 2.3.

==Critical problems==

This short list of problems has been identified from user feedback via Moodle partners, tracker issues with votes, user surveys, usability studies and common sense. It's not exhaustive, but covers the most severe issues that we think need to be fixed. If you'd like to comment about this list, please use the discussion page for this project or comment directly on the linked tracker issues.

===Problem A: Uploading files take a lot of clicks ===

The most common case for most people is to upload files from their own desktop (not a repository). Since the file picker must be used for this, this can take a lot of clicks, and only one file can be uploaded at a time (unless you know how to use zip).

====Solution A1: Allow drag+drop files from desktop.====

We can add the ability to drag files from the desktop in multiple places. See other issue below about licensing.

# File manager form elements - this is already implemented in 2.3
# Into file picker (private files)
# Into file picker (upload)
# Into course sections (auto-creation of file resources)
# Into HTML editor window? (could add to filearea and insert link/image into text?)

====Solution A2: Add "quick upload" buttons that bring up dialog as quickly as possible====

We can add small buttons that take the user directly to a browse-and-upload button as fast as possible, in the following places:

# File manager form element
# HTML editor image dialog
# HTML editor media dialog

====Solution A3: Allow multiple files to be uploaded at the same time====

We can add the ability to browse/select multiple files from the desktop via one form, along with a way to update license information for all of them before saving.

This requires some sort of parameter from the calling form to be passed to the file picker, indicating how many new files are expected/allowed.

====Solution A4: More stickiness in file picker between invocations====

The file picker should remember more of your settings from the last time you used it.

# Last Icon/List view
# Last server files location

====Solution A5: Add infrastructure for default licensing, and license editing====

We need some infrastructure for licensing to set default licenses at site/course/user level, because there will be no opportunity to select that at upload time during a drag and drop. File manager form elements should have an "Info" item (in the context menu for each item) that gives a dialog to edit name/license.

===Problem B: Private files management===

The 'Private files" area is the main place that all users have to store files privately in Moodle. It is like a mini-repository that people can use to store files before sharing them in courses and other public places.

====Solution B1: Prevent leaving the Private Files page until changes are saved====

A big problem for people currently is that they can leave the filemanager page before saving their changes. A simple fix is to detect this situation just like Gmail and our Tracker do, and throw up a dialog to confirm they want to do this. The "Save" button can also be more a lot more obvious (should we also offer auto-save?)

====Solution B2: Allow multiple uploads directly via Filepicker repository page====

In the file picker, we should allow people to upload files directly into their Private files and then select it for use in whatever filearea they were editing, all in one step.

====Solution B3: Allow users to "link" to files in their private files====

Add the linking ability to private files, so that people can manage and update multiple copies of files.

===Problem C: File picker looks "ugly"===

Many people say that they just don't like the "MS Windows" look of the current filepicker. This look has been somewhat dictated by the choice of YUI2 to implement it, but it can be improved a lot.

====Solution C1: Improve default graphics styling/layout for the dialog====

It needs to look more like the rest of the Moodle interface.

It should use lightboxing to prevent access to the page, and should be larger by default.

Thumbnails and titles should flow more nicely with less cropping.

====Solution C2: Implement real thumbnail images where possible====

Currently we show images based on the mimetype of the file. For images, we should generate and cache small, efficient thumbnail images of the file for viewing in the file picker via lazy-loading.

Along with this, we should increase the length of pages as much as possible and reduce the number of pages. Scrolling works better than paging.

====Solution C3: Add better CSS/renderers so themes can style it better====

====Solution C4: Add better descriptions/help for options====

Options all need help icons with translated popup help. "linking files" needs to be better explained.

====Solution C5: Simplify "Server Files"====

# Server files can be improved by tweaking file tree structures to make them more logical
# Reducing empty folders
# Shall we list all files in modules? MDL-31675

===Problem D: File area management===

====Solution D1: A dialog in the HTML editor to manage embedded files directly====

We need a way to edit the filearea that is associated with a HTML text. We currently do have one for non-Javascript users but it's never seen. We need a proper one available in the HTML editor, probably accessed as a popup dialog from a button with a yellow folder on it. Warnings need to be shown to the effect that renaming or removing files can break the HTML text.

====Solution D2: Add the ability to "replace" files.====

When a file is added to the file area with the same name as one that exists, a dialog should be shown allowing the user to "replace the existing file" or "rename the new file". The dialog should also indicate how many "linked copies" will be affected, if that is the case.

"Replace" means that the old file record is updated with the new content (perhaps with a version increment?) - this means that any LINKS to that file are retained, and it becomes possible to update many copies at once. This needs to be a new function in the file API.

Files usability 2.3

2012-02-16T08:52:04Z

Dongsheng:

Improved support for external File content

2012-02-13T05:41:38Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* stored_file::delete() should detect repository files, and process it properly, and implement a method to update file, not delete then create
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache ====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-02-13T05:31:08Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* stored_file::delete() should detect repository files, and process it properly, and implement a method to update file, not delete then create
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-02-13T05:29:54Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s send the file contents, Repository API decide return the cached copy or fresh contents
* Preserve file reference information when call file_storage::create_file_from_storedfile
* file_storage::get_area_files should retrieve file reference information, also file_storage::get_file_by_id, file_storage::get_file_by_hash
* stored_file::delete() should detect repository files, and process it properly
* When call file_prepare_draft_area(), it should keep linking files' repository information
* file_save_draft_files should preserve file reference information
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Additions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

* repository::send_file(stored_file $stored_file)

* repository::get_file_reference($ref)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====

Returned stored_file instance or file path

* store($url, $string_to_be_hashed)
* get($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-02-13T03:31:20Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "reference" concept, with all files now having a "repositoryid" (the repository instance that the file came from) and a "reference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle don't have record in `files_reference` table, but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file) in reference and repostiroyid columns.
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem repository.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the file is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s callback to fetch file contents, Repository API decide return the cached copy or fresh contents
* When call file_prepare_draft_area(), it should keep linking files' repository information.
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====
* store_file($url, $string_to_be_hashed)
* get_file($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-02-03T03:37:10Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "source" concept, with all files now having a "sourcerepositoryid" (the repository instance that the file came from) and a "sourcereference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle have a "sourcerepositoryid" of "0", but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file).
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem browser.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the sourcerepository is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with file parameters and reference. File API stores the reference and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s callback to fetch file contents, Repository API decide return the cached copy or fresh contents
* When call file_prepare_draft_area(), it should keep linking files' repository information.
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====
* store_file($url, $string_to_be_hashed)
* get_file($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-02-02T08:32:56Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "source" concept, with all files now having a "sourcerepositoryid" (the repository instance that the file came from) and a "sourcereference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle have a "sourcerepositoryid" of "0", but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file).
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem browser.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the sourcerepository is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it in special area in file pool for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with content hash and UUID. File API stores the UUID and repository instance id in `mdl_files_reference` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_reference`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php and draftfile.php (it's actually send_stored_file()): when file isn't local file (`repositoryid` isn't zero), stored_file instance should ask repository plugin‘s callback to fetch file contents, Repository API decide return the cached copy or fresh contents
* When call file_prepare_draft_area(), it should keep linking files' repository information.
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE = 4 // 0100, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, extra file reference information is stored in `mdl_files_reference` table, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `reference` in `mdl_files_reference` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====
* store_file($url, $string_to_be_hashed)
* get_file($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Improved support for external File content

2012-01-31T06:56:08Z

Dongsheng:

{{Work in progress}}
{{Infobox Project
|name = File synching
|state = Planning
|tracker = MDL-28666
|discussion = TODO
|assignee = [[User:Martin Dougiamas|Martin Dougiamas]]
}}
{{Moodle 2.2}}

==Problems with Files in 2.0==

* It is not currently easy to use a file in multiple places throughout Moodle and update them all at once
* It is not currently easy to create a simple shared "course repository" for teachers to use

==Example use cases that will become possible==

A teacher wants to upload a file once and use it in multiple courses. When they update the file, it should be updated in all their courses automatically.

# The teacher uploads the file to their private file area (or other repository).
# In each place they want to add the file, they use the file picker to select the file from their private files area (or other repo) and select "link to latest version".
# Replacing the file will automatically mean the linked copies use it.
# Deleting the original file will force the linked copies to be static copies (the teacher will be informed of all the copies before proceeding with the delete).

Several teachers want to create a shared repository of files together
# One teacher adds a course repository block to the course
# Using the block, the teacher creates an instance of a "?filesystem? repository" inside the course (essentially a folder).
# The content of the repository can be edited via the "Course repositories" block.
# All of the teachers in that course can now see that appear in their filepickers and select files
# Files can be "linked" as above

A student wants to submit a linked file as an assignment, so they can continue updating it after the assignment due date.
# This will not be possible, because the assignment will not allow linking to files.
# The student is forced to upload a copy of a file and this is protected by the assignment module.

==Solution summary==

The basic idea is to allow the CONTENT of files to be stored outside of the filepool, while all the metadata and access is controlled by Moodle exactly the same as it is now.

# Extend Files API with a new "source" concept, with all files now having a "sourcerepositoryid" (the repository instance that the file came from) and a "sourcereference" (the address in that repository of the content of the file).
# All files '''copied''' into Moodle have a "sourcerepositoryid" of "0", but others can specify a UUID (usually a URL with file-specific tokens in it, created by the original user who placed the file).
# Improve the filesystem repository plugin to make it easier to create folder repositories on the fly, via a block.
# Add support to the filepicker UI to add "linking" to more repositories that support it, including the server files and filesystem browser.
# Virtual files are served via pluginfile.php URLs just like normal files:
## pluginfile.php uses module callback to determine access (as now), and if it passes then
## pluginfile.php calls a file logging subsystem to log the fact that this file is being served (useful for copyright reporting, for example)
## If the sourcerepository is not 'local', then pluginfile.php uses the relevant repository callback to get the content of the file and streams it to the user with appropriate mimetypes etc
## The repositories have a way to cache this content in the normal Moodle filepool, to avoid repeated downloads.
## If an external repository is down or not configured, then the repository plugin can choose to just serve the local cached version (useful for restored backups and disaster tolerance)
# When a file is linked to another another file in the local location, then it has a location of "filepool". We need to pay attention to these during:
## Garbage cleanups
## Deleting the original file should create real copies of that file where required (user will be informed).

Note that the original URL of files in external repositories are never revealed to users.

==Details==

=== File picker walk-through ===
[[File:File_picker_using_repository_reference_SD.png]]
# User clicks the "Insert image" button in TinyMCE, then launches our File Picker in the dialog
# User chooses a repository which supports UUID direct file references (eg Equella, Alfresco etc)
# A. File picker will ask repository plugin for customised repository UI if supported
# B. File Picker use repository UI and <object> type to display customised UI in file picker right pane
# User selects a file from the repository, if is customised UI, file picker Javascript API will be used to trigger file selection event (with all file related information), the repository plugin gives you two choices in the file picker interface http://tracker.moodle.org/secure/attachment/25823/2011-11-09+15.23.png http://tracker.moodle.org/secure/attachment/25825/File+picker+options.png
#* C. Use the current version: this will copy the file to Moodle (as now). No need to reference external content.
#* C. Use the latest version: this will use a file reference, so that the most recent version is always pulled from the repository
# D. If "Use the latest version" selected, Repository API will download the file as usual but store it in special area in file pool for caching, we could locate this cached file when we need it, cron will take responsibility of invalidating/updating it
# E. Repository API ask File API to create a file with content hash and UUID. File API stores the UUID and repository instance id in `mdl_files_source` table and returns the file URL which looks the same as any other Moodle file URL.
# File picker gives the file URL to TinyMCE
# When TinyMCE displays the resource, it will cause the browser to call the file URL, which contains pluginfile.php.
# Pluginfile.php uses File API to send file contents to browsers, if File API detects the requested file is not ordinary moodle files which is located at external repository, File API will ask Repository API for file content, Repository API will firstly look for the cached file, if file is too old or not found (removed by cron checking), Repository API will fetch the resource (and cache it), then return to File API. Alternatively, Repository API could disable caching, asking for fresh content all the time.

=== File request walk-through ===
[[File:File_Request_SD.png]]

# A. User request a file
# B. File API detects if the request file is regular moodle files or located at external repository, if it's external files, File API will ask repository API to grab the file
# C. File API collects the file reference information from database, it could be stored in php serialised or JSON format
# D. File API passes raw file reference information to repository plugin
# E. Repository plugin will firstly check if file available locally
# F. If not repository plugin will use file reference information to grab the file
# G. Repository API returns file content to File API to serve the file

=== Database changes ===

We can create another table for left joining, this requires File API query this table when locating files:
* New file table `mdl_files_source`
** `id` primary key
** `fileid` foreign key of `mdl_files`.`id`
** `repositoryid`
** `reference` - can be URL, UUID or other data format, repository plugin callbacks know the meaning of this field. File reference should be cached when adding to moodle, contenthash should be accurate.

* `mdl_files_log` table for files access log, File API should have a new function to insert records to this table
** `id` primary key
** `userid` (0 if it's guest),
** `timeaccess`,
** `fileid`

=== File API changes ===
* pluginfile.php: when file isn't local file (`sourcerepositoryid` isn't zero), pluginfile.php should ask repository plugin‘s callback to fetch file contents, Repository API decide return the cached copy or fresh contents
* draftfile.php should do the same as pluginfile.php
* When call file_prepare_draft_area(), it should keep linking files' repository information.
* Adding callback to files API delete function, if the actual file get deleted, callback needs to check all linked files, and convert them to actually files
* new method: file_storage::create_file_from_reference
<code php>
public function create_file_from_reference($file_record, $repoisitoryid, $reference, array $options = NULL)
</code>
* Cron: we probably need a Repository API function to cache/update external files

=== Repository API changes ===

* Addtions to FILE_INTERNAL and FILE_EXTERNAL, we need another type FILE_REFERENCE, repository plugin needs to declare what types of files are supported

* When users ask to create a reference (instead of copying) in file picker, Repository API try to cache the file in filepool(special file area for repository), after file downloaded, repository API should ask Files API to create a virtual file in `mdl_files` table, the existing fields stay the same, `sourcerepositoryid` will be repoistory instance id, `sourcereference` will be a reference, link or other format that repository plugins know. File API return the stored_file object, repository API generate the moodle url for users just like other moodle files without revealing the internal reference

* Making repository plugin upgrade and versioning possible, repository plugin may need to update `sourcereference` in `mdl_files` table if reference info changed. (we already have db/upgrade.php, make sure all new plugins have one)

* When deleting repository instance, all files imported by this instance will have to be converted to actual files, this has to be done by a File API function

* Instead of using get_file() to download file content, we need one new method repository::get_reference_info($file) to get file reference information

* Cron will use repository plugin callbacks to clean up cache files, repoistory::cron($repositoryid)

=== Repository plugins changes ===
* Server files: store file parameters in `reference` field
* Private files: same as server files plugin
* Alfresco: Store UUID in `reference` field, but alfresco will change UUID once tomcat restart, may need other information to locate files
* Flickr private: needs flickr secret and token and photo id to locate the files
* File system: store file path in `reference` field
* s3: store file path, s3 repository will use secret and token to fetch the file from s3 no matter files are public or not.
* EQUELLA
*# Admin installs EQUELLA and setup parameters for single sign on
*# Teacher clicks EQUELLA instance, EQUELLA will return an URL of repository UI (plugin code)
*# Teacher pick a file from EQUELLA, EQUELLA repository UI will revoke file picker JavaScript API to notify moodle download this resource (plugin code)
*# Repository API stores file UUID and SSO userid, creates file reference in moodle file pool
*# EQUELLA plugin implements method to download contents using stored UUID and SSO userid
*# EQUELLA plugin implements method to update/invalidate cached resources (by cron)

=== Content caching ===
* Create moodledata/repostory/cache directory
* Generate hash based on file URL and request parameters (provided by repository plugin), not content hash because we cannot send content hash to external repository for file information
* Cached files are stored using hash code as file name

==== class repository_cache =====
* store_file($url, $string_to_be_hashed)
* get_file($string_to_be_hashed)

=== Filepicker Javascript API for customizing===
* File picker should be able to dislable file references by taking an option
* Support <object> tag in filepicker container
* Provide Javascript API to allow plugin communicate with filepicker
** Notify file picker to download file
** Notify file picker to pop up authentication page

=== File manager to handle virtual files===
Not much trouble here, need to make sure draftfile.php can serve external resources, because all files managed by file manager is in draft area

[[Category:Project]]

Authentication plugins

2012-01-25T09:14:10Z

Dongsheng: /* Template */

==Introduction==

This page first gives an '''overview''' of the ''authentication process'' and then explains how ''authentication modules'' can be '''created''' using hooks to take over from the native authentication in Moodle.
=== Overview of Moodle authentication process ===
[[File:1.9.11_login_element.png|203px||thumb|right|The login UI element in Moodle 1.9]]The authentication use case in Moodle starts when a user clicks on the Login link in the UI. Then the following happens (skipping some minor details and rarer scenarios):

# The default login page (<tt>/login/index.php</tt>) is displayed. OR, if a system administrator has set the Alternate Login URL on the "Manage authentication" page, that URL will be displayed.
# A user enters their credentials and submits the form.
# The handler code in <tt>/login/index.php</tt> runs:
## Gets a list of enabled authentication plugins.
## Runs <tt>loginpage_hook()</tt> for each plugin, in case any of them needs to intercept the login request.
## Checks to make sure that the username meets Moodle's criteria (alphanumeric, with periods and hyphens allowed).
## Calls <tt>authenticate_user_login()</tt> in <tt>/lib/moodlelib.php</tt>, which returns a <code>$user</code> object. (Details of this code follow this main outline.)
## Determines whether authentication was successful (by checking whether <code>$user</code> is a valid object) and, if not, sends them back to the login page with an error message. Otherwise, it figures out where to send the user based on their original page request, whether their password is expired, etc., and redirects them there.

==History==
Authentication plugins exist from 1.9
==Example==
[http://moodle.org/plugins/view.php?plugin=auth_googleoauth2 Google / Facebook / Messenger Oauth2 Authentication plugin]

==Template==
Please see Moodle '''none''' authentication plugin (auth/none), it's the perfect plugin template to start with.

==Naming convention==
==File structure==
# Choose a name for your plugin. We'll use 'sentry' as an example below; change it to whatever name you have chosen.
# Under your Moodle installation root, create the directory <tt>/auth/sentry</tt>. It should be sibling to existing auth plugin directories: 'db', 'nologin', 'none', etc.
# Create the file <tt>/auth/sentry/auth.php</tt>. Within the file, create a class <tt>auth_plugin_sentry</tt> that extends <tt>auth_plugin_base</tt> from <tt>/lib/authlib.php</tt>. (You will need to <code>require_once</code> the authlib file.)
# Implement the <tt>user_login()</tt> function in your <tt>auth.php</tt> file, and create or override additional functions based on your plugin's requirements.
# Log in to your Moodle installation as a site administrator and find, in the site administrator block, the page "Users -> Authentication -> Manage authentication". You will see your plugin in the list, appearing as <nowiki>[[auth_sentrytitle]]</nowiki>. You can enable it and move it up and down in the order. '''At this point, with the plugin enabled, your plugin is registered and will be used by Moodle in its authentication process.'''
# If you don't like seeing <nowiki>[[auth_sentrytitle]]</nowiki> as the name of your plugin in the Moodle UI, you'll need to create language files for your plugin. Do this by creating the directory <tt>/auth/sentry/lang</tt>, and under it, a directory for each language that your installation needs to support. (Example: <tt>/auth/sentry/lang/en</tt>.) Within each of these language directories, create a file called <tt>auth_sentry.php</tt>. That file should set the desired value for <code>$string['auth_sentrytitle'] for that language (use $string['pluginname'] for Moodle2)</code>. You can also set the plugin description by setting <code>$string['auth_sentrydescription']</code>, and you can also assign other translatable strings that your plugin uses, in these files. '''Note:''' A folder named like '''en''' may not work for everyone. Please refer to the lang folder under your moodle installation directory to get an idea about the language/locale codes used in your moodle installation. For example, the lang folder in own users system contained folders named '''en''' and '''zh-cn'''. He emulated the same in the lang folder of the auth plugin and the plugin picked up the title and description strings immediately. ([[Places_to_search_for_lang_strings|More info on how Moodle handles language]]).
# If you want to configure your plugin through the Moodle UI, implement <tt>config_form()</tt> and <tt>process_config()</tt> in the plugin class. You might find it convenient to use the 'db' plugin as a model for this. The plugin's config settings can then be managed through the Manage authentication page by clicking on the Settings link for that plugin, and the values will be stored in the <tt>mdl_config_pluginstable</tt> in the database.

==Interfacing to API's==

=== <tt>authenticate_user_login()</tt>===
That's the main outline, but a lot of interesting stuff happens in <tt>authenticate_user_login()</tt>:

# It gets a list of enabled authentication plugins.
# It looks up the username in the mdl_user table to see if they are allowed to log in, and which authentication plugin handles their login requests. (This will be the plugin that handled their first-ever login request.)
# It creates a user object, which will contain the data from <tt>mdl_user</tt> if the username is known; if not, it will be an empty object.
# It does the following with the authentication plugin (note that for a username unknown to Moodle, it will do these steps for each authenticated plugin until one succeeds or it has tried them all):
## Calls the <tt>user_login()</tt> function provided by that plugin, which returns a boolean value based on whether the credentials authenticate or not. If the result is false (not authenticated), skips the rest of the steps below and continues to the next plugin.
## If the plugin authenticates against an external system (not Moodle's user database), its <tt>update_user_record()</tt> function is called to get the user's name, contact info, etc.
## Creates the Moodle user record if it doesn't already exist.
## Calls the plugin's <tt>sync_roles()</tt> function.
## Notifies each enabled authentication plugin that the user successfully authenticated, by calling each one's <tt>user_authenticated_hook()</tt> function.
# It returns the user object if everything was successful, or false if no plugin was able to successfully authenticate the credentials.
=== <tt>user_login($username, $password)</tt>===
This must be rewritten by plugin to return boolean value, returns true if the username and password work and false if they are wrong or don't exist.

=== <tt>can_change_password()</tt>===
Returns true if this authentication plugin can change users' password.

=== <tt>change_password_url()</tt>===
Returns the URL for changing the users' passwords, or empty if the default URL can be used.

=== <tt>can_edit_profile()</tt>===
Returns true if this authentication plugin can edit the users' profile.

=== <tt>edit_profile_url()</tt>===
Returns the URL for editing users' profile, or empty if the defaults URL can be used.

=== <tt>prevent_local_passwords()</tt>===
Indicates if password hashes should be stored in local moodle database.

=== <tt>user_update_password($user, $newpassword)</tt>===
Update the user's password.

=== <tt>user_update($olduser, $newuser)</tt>===
Called when the user record is updated. It will modify the user information in external database.

=== <tt>user_delete($olduser)</tt>===
User delete requested. Internal user record had been deleted.

=== <tt>can_reset_password()</tt>===
Returns true if plugin allows resetting of internal password.

=== <tt>user_signup($user, $notify=true)</tt>===
Sign up a new user ready for confirmation, password is passed in plaintext.

=== <tt>can_confirm()</tt>===
Returns true if plugin allows confirming of new users.

=== <tt>user_confirm($username, $confirmsecret)</tt>===
Confirm the new user as registered.

=== <tt>user_exists($username)</tt>===
Checks if user exists in external db.

=== <tt>password_expire($username)</tt>===
Returns number of days to user password expires.

=== <tt>sync_roles()</tt>===
Sync roles for this user - usually creator

=== <tt>get_userinfo($username)</tt>===
Read user information from external database and returns it as array.

=== <tt>config_form($config, $err, $user_fields)</tt>===
Prints a form for configuring this authentication plugin. It's called from admin/auth.php, and outputs a full page with a form for configuring this plugin.

=== <tt>validate_form($form, $err)</tt>===
Validate form data.

=== <tt>process_config($config)</tt>===
Processes and stores configuration data for this authentication plugin.

=== <tt>loginpage_hook()</tt>===
Hook for overriding behaviour of login page.

=== <tt>user_authenticated_hook($user, $username, $password)</tt>===
Post authentication hook. This method is called from authenticate_user_login() for all enabled auth plugins.

=== <tt>prelogout_hook()</tt>===
Pre logout hook.

=== <tt>logoutpage_hook()</tt>===
Hook for overriding behaviour of logout page.

==See also==

* [[Authentication API]]
* Using Moodle [http://moodle.org/mod/forum/discuss.php?d=102070 Overview of entire authentication code flow] forum discussion
* Using Moodle [http://moodle.org/mod/forum/view.php?id=42 User authentication forum]
* Moodle Docs [[:en:Manage authentication|Manage authentication]]
* [[:en:Authentication FAQ|Authentication FAQ]]

[[Category:Authentication]]
[[Category:Plugins]]

[[fr:Méthodes_d'authentification]]

Authentication plugins

2012-01-25T09:05:46Z

Dongsheng: /* File structure */

==Introduction==

This page first gives an '''overview''' of the ''authentication process'' and then explains how ''authentication modules'' can be '''created''' using hooks to take over from the native authentication in Moodle.
=== Overview of Moodle authentication process ===
[[File:1.9.11_login_element.png|203px||thumb|right|The login UI element in Moodle 1.9]]The authentication use case in Moodle starts when a user clicks on the Login link in the UI. Then the following happens (skipping some minor details and rarer scenarios):

# The default login page (<tt>/login/index.php</tt>) is displayed. OR, if a system administrator has set the Alternate Login URL on the "Manage authentication" page, that URL will be displayed.
# A user enters their credentials and submits the form.
# The handler code in <tt>/login/index.php</tt> runs:
## Gets a list of enabled authentication plugins.
## Runs <tt>loginpage_hook()</tt> for each plugin, in case any of them needs to intercept the login request.
## Checks to make sure that the username meets Moodle's criteria (alphanumeric, with periods and hyphens allowed).
## Calls <tt>authenticate_user_login()</tt> in <tt>/lib/moodlelib.php</tt>, which returns a <code>$user</code> object. (Details of this code follow this main outline.)
## Determines whether authentication was successful (by checking whether <code>$user</code> is a valid object) and, if not, sends them back to the login page with an error message. Otherwise, it figures out where to send the user based on their original page request, whether their password is expired, etc., and redirects them there.

==History==
Authentication plugins exist from 1.9
==Example==
[http://moodle.org/plugins/view.php?plugin=auth_googleoauth2 Google / Facebook / Messenger Oauth2 Authentication plugin]

==Template==
==Naming convention==
==File structure==
# Choose a name for your plugin. We'll use 'sentry' as an example below; change it to whatever name you have chosen.
# Under your Moodle installation root, create the directory <tt>/auth/sentry</tt>. It should be sibling to existing auth plugin directories: 'db', 'nologin', 'none', etc.
# Create the file <tt>/auth/sentry/auth.php</tt>. Within the file, create a class <tt>auth_plugin_sentry</tt> that extends <tt>auth_plugin_base</tt> from <tt>/lib/authlib.php</tt>. (You will need to <code>require_once</code> the authlib file.)
# Implement the <tt>user_login()</tt> function in your <tt>auth.php</tt> file, and create or override additional functions based on your plugin's requirements.
# Log in to your Moodle installation as a site administrator and find, in the site administrator block, the page "Users -> Authentication -> Manage authentication". You will see your plugin in the list, appearing as <nowiki>[[auth_sentrytitle]]</nowiki>. You can enable it and move it up and down in the order. '''At this point, with the plugin enabled, your plugin is registered and will be used by Moodle in its authentication process.'''
# If you don't like seeing <nowiki>[[auth_sentrytitle]]</nowiki> as the name of your plugin in the Moodle UI, you'll need to create language files for your plugin. Do this by creating the directory <tt>/auth/sentry/lang</tt>, and under it, a directory for each language that your installation needs to support. (Example: <tt>/auth/sentry/lang/en</tt>.) Within each of these language directories, create a file called <tt>auth_sentry.php</tt>. That file should set the desired value for <code>$string['auth_sentrytitle'] for that language (use $string['pluginname'] for Moodle2)</code>. You can also set the plugin description by setting <code>$string['auth_sentrydescription']</code>, and you can also assign other translatable strings that your plugin uses, in these files. '''Note:''' A folder named like '''en''' may not work for everyone. Please refer to the lang folder under your moodle installation directory to get an idea about the language/locale codes used in your moodle installation. For example, the lang folder in own users system contained folders named '''en''' and '''zh-cn'''. He emulated the same in the lang folder of the auth plugin and the plugin picked up the title and description strings immediately. ([[Places_to_search_for_lang_strings|More info on how Moodle handles language]]).
# If you want to configure your plugin through the Moodle UI, implement <tt>config_form()</tt> and <tt>process_config()</tt> in the plugin class. You might find it convenient to use the 'db' plugin as a model for this. The plugin's config settings can then be managed through the Manage authentication page by clicking on the Settings link for that plugin, and the values will be stored in the <tt>mdl_config_pluginstable</tt> in the database.

==Interfacing to API's==

=== <tt>authenticate_user_login()</tt>===
That's the main outline, but a lot of interesting stuff happens in <tt>authenticate_user_login()</tt>:

# It gets a list of enabled authentication plugins.
# It looks up the username in the mdl_user table to see if they are allowed to log in, and which authentication plugin handles their login requests. (This will be the plugin that handled their first-ever login request.)
# It creates a user object, which will contain the data from <tt>mdl_user</tt> if the username is known; if not, it will be an empty object.
# It does the following with the authentication plugin (note that for a username unknown to Moodle, it will do these steps for each authenticated plugin until one succeeds or it has tried them all):
## Calls the <tt>user_login()</tt> function provided by that plugin, which returns a boolean value based on whether the credentials authenticate or not. If the result is false (not authenticated), skips the rest of the steps below and continues to the next plugin.
## If the plugin authenticates against an external system (not Moodle's user database), its <tt>update_user_record()</tt> function is called to get the user's name, contact info, etc.
## Creates the Moodle user record if it doesn't already exist.
## Calls the plugin's <tt>sync_roles()</tt> function.
## Notifies each enabled authentication plugin that the user successfully authenticated, by calling each one's <tt>user_authenticated_hook()</tt> function.
# It returns the user object if everything was successful, or false if no plugin was able to successfully authenticate the credentials.
=== <tt>user_login($username, $password)</tt>===
This must be rewritten by plugin to return boolean value, returns true if the username and password work and false if they are wrong or don't exist.

=== <tt>can_change_password()</tt>===
Returns true if this authentication plugin can change users' password.

=== <tt>change_password_url()</tt>===
Returns the URL for changing the users' passwords, or empty if the default URL can be used.

=== <tt>can_edit_profile()</tt>===
Returns true if this authentication plugin can edit the users' profile.

=== <tt>edit_profile_url()</tt>===
Returns the URL for editing users' profile, or empty if the defaults URL can be used.

=== <tt>prevent_local_passwords()</tt>===
Indicates if password hashes should be stored in local moodle database.

=== <tt>user_update_password($user, $newpassword)</tt>===
Update the user's password.

=== <tt>user_update($olduser, $newuser)</tt>===
Called when the user record is updated. It will modify the user information in external database.

=== <tt>user_delete($olduser)</tt>===
User delete requested. Internal user record had been deleted.

=== <tt>can_reset_password()</tt>===
Returns true if plugin allows resetting of internal password.

=== <tt>user_signup($user, $notify=true)</tt>===
Sign up a new user ready for confirmation, password is passed in plaintext.

=== <tt>can_confirm()</tt>===
Returns true if plugin allows confirming of new users.

=== <tt>user_confirm($username, $confirmsecret)</tt>===
Confirm the new user as registered.

=== <tt>user_exists($username)</tt>===
Checks if user exists in external db.

=== <tt>password_expire($username)</tt>===
Returns number of days to user password expires.

=== <tt>sync_roles()</tt>===
Sync roles for this user - usually creator

=== <tt>get_userinfo($username)</tt>===
Read user information from external database and returns it as array.

=== <tt>config_form($config, $err, $user_fields)</tt>===
Prints a form for configuring this authentication plugin. It's called from admin/auth.php, and outputs a full page with a form for configuring this plugin.

=== <tt>validate_form($form, $err)</tt>===
Validate form data.

=== <tt>process_config($config)</tt>===
Processes and stores configuration data for this authentication plugin.

=== <tt>loginpage_hook()</tt>===
Hook for overriding behaviour of login page.

=== <tt>user_authenticated_hook($user, $username, $password)</tt>===
Post authentication hook. This method is called from authenticate_user_login() for all enabled auth plugins.

=== <tt>prelogout_hook()</tt>===
Pre logout hook.

=== <tt>logoutpage_hook()</tt>===
Hook for overriding behaviour of logout page.

==See also==

* [[Authentication API]]
* Using Moodle [http://moodle.org/mod/forum/discuss.php?d=102070 Overview of entire authentication code flow] forum discussion
* Using Moodle [http://moodle.org/mod/forum/view.php?id=42 User authentication forum]
* Moodle Docs [[:en:Manage authentication|Manage authentication]]
* [[:en:Authentication FAQ|Authentication FAQ]]

[[Category:Authentication]]
[[Category:Plugins]]

[[fr:Méthodes_d'authentification]]

Authentication plugins

2012-01-25T09:02:35Z

Dongsheng: /* Example */

==Introduction==

This page first gives an '''overview''' of the ''authentication process'' and then explains how ''authentication modules'' can be '''created''' using hooks to take over from the native authentication in Moodle.
=== Overview of Moodle authentication process ===
[[File:1.9.11_login_element.png|203px||thumb|right|The login UI element in Moodle 1.9]]The authentication use case in Moodle starts when a user clicks on the Login link in the UI. Then the following happens (skipping some minor details and rarer scenarios):

# The default login page (<tt>/login/index.php</tt>) is displayed. OR, if a system administrator has set the Alternate Login URL on the "Manage authentication" page, that URL will be displayed.
# A user enters their credentials and submits the form.
# The handler code in <tt>/login/index.php</tt> runs:
## Gets a list of enabled authentication plugins.
## Runs <tt>loginpage_hook()</tt> for each plugin, in case any of them needs to intercept the login request.
## Checks to make sure that the username meets Moodle's criteria (alphanumeric, with periods and hyphens allowed).
## Calls <tt>authenticate_user_login()</tt> in <tt>/lib/moodlelib.php</tt>, which returns a <code>$user</code> object. (Details of this code follow this main outline.)
## Determines whether authentication was successful (by checking whether <code>$user</code> is a valid object) and, if not, sends them back to the login page with an error message. Otherwise, it figures out where to send the user based on their original page request, whether their password is expired, etc., and redirects them there.

==History==
Authentication plugins exist from 1.9
==Example==
[http://moodle.org/plugins/view.php?plugin=auth_googleoauth2 Google / Facebook / Messenger Oauth2 Authentication plugin]

==Template==
==Naming convention==
==File structure==
# Choose a name for your plugin. We'll use 'sentry' as an example below; change it to whatever name you have chosen.

# Under your Moodle installation root, create the directory <tt>/auth/sentry</tt>. It should be sibling to existing auth plugin directories: 'db', 'nologin', 'none', etc.

# Create the file <tt>/auth/sentry/auth.php</tt>. Within the file, create a class <tt>auth_plugin_sentry</tt> that extends <tt>auth_plugin_base</tt> from <tt>/lib/authlib.php</tt>. (You will need to <code>require_once</code> the authlib file.)

# Implement the <tt>user_login()</tt> function in your <tt>auth.php</tt> file, and create or override additional functions based on your plugin's requirements.

# Log in to your Moodle installation as a site administrator and find, in the site administrator block, the page "Users -> Authentication -> Manage authentication". You will see your plugin in the list, appearing as <nowiki>[[auth_sentrytitle]]</nowiki>. You can enable it and move it up and down in the order. '''At this point, with the plugin enabled, your plugin is registered and will be used by Moodle in its authentication process.'''

# If you don't like seeing <nowiki>[[auth_sentrytitle]]</nowiki> as the name of your plugin in the Moodle UI, you'll need to create language files for your plugin. Do this by creating the directory <tt>/auth/sentry/lang</tt>, and under it, a directory for each language that your installation needs to support. (Example: <tt>/auth/sentry/lang/en</tt>.) Within each of these language directories, create a file called <tt>auth_sentry.php</tt>. That file should set the desired value for <code>$string['auth_sentrytitle'] for that language (use $string['pluginname'] for Moodle2)</code>. You can also set the plugin description by setting <code>$string['auth_sentrydescription']</code>, and you can also assign other translatable strings that your plugin uses, in these files.

'''Note:''' A folder named like '''en''' may not work for everyone. Please refer to the lang folder under your moodle installation directory to get an idea about the language/locale codes used in your moodle installation. For example, the lang folder in own users system contained folders named '''en''' and '''zh-cn'''. He emulated the same in the lang folder of the auth plugin and the plugin picked up the title and description strings immediately. ([[Places_to_search_for_lang_strings|More info on how Moodle handles language]]).

# If you want to configure your plugin through the Moodle UI, implement <tt>config_form()</tt> and <tt>process_config()</tt> in the plugin class. You might find it convenient to use the 'db' plugin as a model for this. The plugin's config settings can then be managed through the Manage authentication page by clicking on the Settings link for that plugin, and the values will be stored in the <tt>mdl_config_pluginstable</tt> in the database.

==Interfacing to API's==

=== <tt>authenticate_user_login()</tt>===
That's the main outline, but a lot of interesting stuff happens in <tt>authenticate_user_login()</tt>:

# It gets a list of enabled authentication plugins.
# It looks up the username in the mdl_user table to see if they are allowed to log in, and which authentication plugin handles their login requests. (This will be the plugin that handled their first-ever login request.)
# It creates a user object, which will contain the data from <tt>mdl_user</tt> if the username is known; if not, it will be an empty object.
# It does the following with the authentication plugin (note that for a username unknown to Moodle, it will do these steps for each authenticated plugin until one succeeds or it has tried them all):
## Calls the <tt>user_login()</tt> function provided by that plugin, which returns a boolean value based on whether the credentials authenticate or not. If the result is false (not authenticated), skips the rest of the steps below and continues to the next plugin.
## If the plugin authenticates against an external system (not Moodle's user database), its <tt>update_user_record()</tt> function is called to get the user's name, contact info, etc.
## Creates the Moodle user record if it doesn't already exist.
## Calls the plugin's <tt>sync_roles()</tt> function.
## Notifies each enabled authentication plugin that the user successfully authenticated, by calling each one's <tt>user_authenticated_hook()</tt> function.
# It returns the user object if everything was successful, or false if no plugin was able to successfully authenticate the credentials.
=== <tt>user_login($username, $password)</tt>===
This must be rewritten by plugin to return boolean value, returns true if the username and password work and false if they are wrong or don't exist.

=== <tt>can_change_password()</tt>===
Returns true if this authentication plugin can change users' password.

=== <tt>change_password_url()</tt>===
Returns the URL for changing the users' passwords, or empty if the default URL can be used.

=== <tt>can_edit_profile()</tt>===
Returns true if this authentication plugin can edit the users' profile.

=== <tt>edit_profile_url()</tt>===
Returns the URL for editing users' profile, or empty if the defaults URL can be used.

=== <tt>prevent_local_passwords()</tt>===
Indicates if password hashes should be stored in local moodle database.

=== <tt>user_update_password($user, $newpassword)</tt>===
Update the user's password.

=== <tt>user_update($olduser, $newuser)</tt>===
Called when the user record is updated. It will modify the user information in external database.

=== <tt>user_delete($olduser)</tt>===
User delete requested. Internal user record had been deleted.

=== <tt>can_reset_password()</tt>===
Returns true if plugin allows resetting of internal password.

=== <tt>user_signup($user, $notify=true)</tt>===
Sign up a new user ready for confirmation, password is passed in plaintext.

=== <tt>can_confirm()</tt>===
Returns true if plugin allows confirming of new users.

=== <tt>user_confirm($username, $confirmsecret)</tt>===
Confirm the new user as registered.

=== <tt>user_exists($username)</tt>===
Checks if user exists in external db.

=== <tt>password_expire($username)</tt>===
Returns number of days to user password expires.

=== <tt>sync_roles()</tt>===
Sync roles for this user - usually creator

=== <tt>get_userinfo($username)</tt>===
Read user information from external database and returns it as array.

=== <tt>config_form($config, $err, $user_fields)</tt>===
Prints a form for configuring this authentication plugin. It's called from admin/auth.php, and outputs a full page with a form for configuring this plugin.

=== <tt>validate_form($form, $err)</tt>===
Validate form data.

=== <tt>process_config($config)</tt>===
Processes and stores configuration data for this authentication plugin.

=== <tt>loginpage_hook()</tt>===
Hook for overriding behaviour of login page.

=== <tt>user_authenticated_hook($user, $username, $password)</tt>===
Post authentication hook. This method is called from authenticate_user_login() for all enabled auth plugins.

=== <tt>prelogout_hook()</tt>===
Pre logout hook.

=== <tt>logoutpage_hook()</tt>===
Hook for overriding behaviour of logout page.

==See also==

* [[Authentication API]]
* Using Moodle [http://moodle.org/mod/forum/discuss.php?d=102070 Overview of entire authentication code flow] forum discussion
* Using Moodle [http://moodle.org/mod/forum/view.php?id=42 User authentication forum]
* Moodle Docs [[:en:Manage authentication|Manage authentication]]
* [[:en:Authentication FAQ|Authentication FAQ]]

[[Category:Authentication]]
[[Category:Plugins]]

[[fr:Méthodes_d'authentification]]

Repository plugins

2012-01-25T08:48:42Z

Dongsheng:

== Introduction ==

Repository plugin allow Moodle to bring contents into Moodle from external repositories.

===Prerequisites===
Before starting coding, it is necessary to know how to use repository administration pages and how to use the file picker.

===Overview===

The 3 different parts to write
# Administration - You can customise the way administrators and users can configure their repositories.
# File picker integration - The core of your plugin, it will manage communication between Moodle and the repository service, and also the file picker display.
# I18n - Internationalization should be done at the same time as you're writing the other parts.

== History ==

Repository plugins exists from 2.0

== Example ==
*[[Box.net Repository Plugin|Box.net Repository Plugin]]
*[[Flickr Repository Plugin|Flickr Repository Plugin]]
*[[Moodle Repository Plugin|Remote Moodle Repository Plugin]]

== Template ==
There is a template available in [https://github.com/dongsheng/moodle-repository_demo Dongsheng Cai's github]

==File structure==
# Create a folder for your plugin in ''moodle/repository/'' e.g. ''moodle/repository/myplugin''
# Create the following files and add them to the plugin folder:
#* ''lib.php'', the main class will be named "repository_myplugin"
#* ''pix/icon.png'' (the icon displayed in the file picker)
#* ''[[version.php]]''
# Create the language file ''repository_myplugin.php'' and add it to the plugin folder, keeping the following folder structure:
#*''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php''
# Create access.php and upgrade scripts (optional)

==Administration APIs==

===Functions===
All of the following functions are optional. If they're not implemented, your plugin will not have manual settings and will have only one instance displayed in the File Picker (The repository API creates this unique instance when the administrator add the plugin).

==== get_instance_option_names====
''This function must be declared static'' 
Return an array of strings. These strings are setting names. These settings are specific to an instance.
If the function returns an empty array, the API will consider that the plugin displays only one repository in the file picker.
Parent function returns an empty array.

====instance_config_form(&$mform)====
This is for modifying the Moodle form displaying the settings specific to an instance.

For example, to add a required text box called email_address:
<code php>
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', $strrequired, 'required', null, 'client');
</code>

''Note: ''mform'' has by default a name text box (cannot be removed).''

Parent function does nothing.

====instance_form_validation($mform, $data, $errors)====
This allows us to validate what has been submitted in the instance configuration form. For instance:
<code php>
public static function instance_form_validation($mform, $data, $errors) {
if (empty($data['email_address'])) {
$errors['email_address'] = get_string('invalidemailsettingname', 'repository_flickr_public');
}
return $errors;
}
</code>

====get_type_option_names====
''This function must be declared static'' 
Return an array of string. These strings are setting names. These settings are shared by all instances.
Parent function return an empty array.

For example:
<code php>
public static function get_type_option_names() {
return array_merge(parent::get_type_option_names(), array('rootpath'));
}
</code>

====type_config_form(&$mform)====
This is for modifying the Moodle form displaying the plugin settings.

For example, to display the standard repository plugin settings along with the custom ones use:
<code php>
public function type_config_form($mform) {
parent::type_config_form($mform);

$rootpath = get_config('repository_someplugin', 'rootpath');
$mform->addElement('text', 'rootpath', get_string('rootpath', 'repository_someplugin'), array('size' => '40'));
$mform->setDefault('rootpath', $rootpath);
}
</code>

====type_form_validation($mform, $data, $errors)====
Use this function if you need to validate some variables submitted by plugin settings form. Push the items to $error array in the format ("fieldname" => "errormessage") to have them highlighted in the form.

With the example above, this function may look like:
<code php>
public static function type_form_validation($mform, $data, $errors) {
if (!is_dir($data['rootpath'])) {
$errors['rootpath'] = get_string('invalidrootpath', 'repository_someplugin');
}
return $errors;
}
</code>

====plugin_init()====
''This function must be declared static'' 
This function is called when the administrator adds the plugin. So unless the administrator deletes the plugin and re-adds it, it should be called only once.
Parent function does nothing.

As an example, let's create a Flickr plugin for accessing a public flickr account. The plugin will be called "Flickr Public".

Firstly the skeleton:

<code php>
<?php
/**
* repository_flickr_public class
* Moodle user can access public flickr account
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class repository_flickr_public extends repository {
}
?>
</code>

Then consider the question "What does my plugin do?"

In the Moodle file picker, we want to display some flickr public repositories directly linked to a flickr public account. For example ''My Public Flickr Pictures'', and also ''My Friend's Flickr Pictures''. When the user clicks on one of these repositories, the public pictures are displayed in the file picker.

In order to access to a flickr public account, the plugin needs to know the email address of the Flickr public account owner. So the administrator will need to set an email address for every repository. Let's add an "email address" setting to every repository.

<code php>
//We tell the API that the repositories have specific settings: "email address"
public static function get_instance_option_names() {
return array('email_address');
}

//We add an "email address" text box to the create/edit repository instance Moodle form
public function instance_config_form(&$mform) {
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', get_string('required'), 'required', null, 'client');
}
</code>

So at this moment all our Flickr Public Repositories will have a specific email address. However this is not enough. In order to communicate with Flickr, Moodle needs to know a Flickr API key (http://www.flickr.com/services/api/). This API key is the same for any repository. We could add it with the email address setting but the administrator would have to enter the same API key for every repository. Hopefully the administrator can add settings to the plugin level, impacting all repositories. The code is similar the repository instance settings:
<code php>
//We tell the API that the repositories have general settings: "api_key"
public static function get_type_option_names() {
return array('api_key');
}

//We add an "api key" text box to the create/edit repository plugin Moodle form (also called a Repository type Moodle form)
public function type_config_form(&$mform) {
//the following line is needed in order to retrieve the API key value from the database when Moodle displays the edit form
$api_key = get_config('flickr_public', 'api_key');
$mform->addElement('text', 'api_key', get_string('apikey', 'repository_flickr_public'),
array('value'=>$api_key,'size' => '40'));
$mform->addRule('api_key', get_string('required'), 'required', null, 'client');
}
</code>

Have we finished yet?

Yes! We have created everything necessary for the administration pages. But let's go further. It would be good if the user can enter any "Flickr public account email address" in the file picker. In fact we want to display in the file picker a Flickr Public repository that the Moodle administrator can never delete. Let's add:

<code php>
//this function is only called one time, when the Moodle administrator add the Flickr Public Plugin into the Moodle site.
public static function plugin_init() {
//here we create a default repository instance. The last parameter is 1 in order to set the instance as readonly.
repository_static_function('flickr_public','create', 'flickr_public', 0, get_system_context(),
array('name' => 'default instance','email_address' => null),1);
}
</code>

That's all - the administration part of our Flickr Public plugin is done. For your information, Box.net, Flickr, and Flickr Public all have similar administration APIs.

==Repository APIs==
=== Quick Start ===
First of all, the File Picker using intensively Ajax you will need a easy way to debug. Install FirePHP (MDL-16371) and make it works. It will save you a lot of time. (You might give the [http://moodle.org/mod/forum/discuss.php?d=119961 FirePHP plugin for Moodle] a try, it's still work in progress, though.)

* Your first question when you write your plugin specification is 'Does the user need to log-in'? If they do, in your plugin you have to detect user session in constructor() function, and use print_login() if required, see more details below.
* For most of plugins, you need to establish a connection with the remote repository. This connection can be done into the get_listing(), constructor() function, see more details below.
* You wanna retrieve the file that the user selected, rewrite get_file() if required, see more details below.
* Optional question that you should ask yourself is 'Does the user can execute a search', if they do, you will have to rewrite search() method, see more details below.

===Functions you *MUST* override===
====__construct====
You may initialize your plugin here, such as:
# Get options from database
# Get user name and password from HTTP POST

====get_listing($path="", $page="")====
This function will return a list of files, the list must be a array like this:
<code php>
$list = array(
//this will be used to build navigation bar
'path'=>array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder')),
'manage'=>'http://webmgr.moodle.com',
'list'=> array(
array('title'=>'filename1', 'date'=>'01/01/2009', 'size'=>'10MB', 'source'=>'http://www.moodle.com/dl.rar'),
array('title'=>'folder', 'date'=>'01/01/2009', 'size'=>'0', 'children'=>array())
)
);
</code>
'''The full specification of list element:'''
<code php>
array(
// Used to build navegation bar, so you need to include all parents folders
// array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder'))
'path' => (array) this will be used to build navigation bar
// dynload tells file picker to fetch list dynamically, when user click
// the folder, it will send a ajax request to server side.
// if you are using pagination, 'page' and 'pages' parameters should be used
// and note, you'd better don't use pagination and page at the same time
'page' => (int) which page is this list
'pages' => (pages) how many pages
'dynload' => (bool) use dynamic loading,
// will display a link in file picker
'manage' => (string) url of the file manager,
// set to true, the login link will be removed from file picker
'nologin' => (bool) requires login,
// set to true, the search link will be removed from file picker
'nosearch' => (bool) no search link,
// set this option will display a upload form in file picker
// only used in upload plugin currently
'upload' => array( // upload manager
'label' => (string) label of the form element,
'id' => (string) id of the form element
),
// file picker will build a file tree according this
// list
'list' => array(
array( // file
'title' => (string) file name,
'shorttitle' => (string) optional, if you prefer to display a short title
'date' => (string) file last modification time, usually userdate(...),
'size' => (int) file size,
'thumbnail' => (string) url to thumbnail for the file,
'thumbnail_width' => (int) the width of the thumbnail image,
'source' => plugin-dependent unique path to the file (id, url, path, etc.),
'url'=> the accessible url of file
),
array( // folder - same as file, but no 'source'.
'title' => (string) folder name,
'shorttitle' => (string) optional, if you prefer to display a short title
'path' => (string) path to this folder
'date' => (string) folder last modification time, usually userdate(...),
'size' => 0,
'thumbnail' => (string) url to thumbnail for the folder,
'children' => array( // an empty folder needs to have 'children' defined, but empty.
// content (files and folders)
)
),
)
)
</code>
Dynamically loading
Some repositories contain many files which cannot load in one time, in this case, we need dynamically loading to fetch them step by step, files in subfolder won't be listed until user click the folder in file picker treeview.

As a plug-in developer, if you set dynload flag as '''true''', you should return files and folders (set children as a null array) in current path instead of building the whole file tree.

Example of dynamically loading
See [http://cvs.moodle.org/moodle/repository/alfresco/lib.php?view=log Alfresco] plug-in

===Functions you can override===
====print_login====
This function will help to print a login form, for the Ajax file picker, this function will return a
PHP array to define this form.
<code php>
public function print_login(){
if ($this->options['ajax']) {
$user_field->label = get_string('username', 'repository_boxnet').': ';
$user_field->id = 'box_username';
$user_field->type = 'text';
$user_field->name = 'boxusername';
$user_field->value = $ret->username;

$passwd_field->label = get_string('password', 'repository_boxnet').': ';
$passwd_field->id = 'box_password';
$passwd_field->type = 'password';
$passwd_field->name = 'boxpassword';

$ret = array();
$ret['login'] = array($user_field, $passwd_field);
// if you are going to rename submit button label, use this option
//$ret['login_btn_label'] = get_string('submit_btn_label');
// if you are going to display a search form instead of login form
// set this option to true
//$ret['login_search_form'] = true;
return $ret;
}
}
</code>
This will help to generate a form by file picker which contains user name and password input elements.

If your plugin don't require logging in, you don't need to override it, it will call get_listing to list files automatically by default.
====check_login====
This function will return a boolean value to tell Moodle whether the user has logged in.
By default, this function will return true.
====logout====
When a user clicks the logout button in file picker, this function will be called. You may clean up the session or disconnect the connection with remote server here.
====print_search====
When a user clicks the search button on file picker, this function will be called to return a search form. By default, it will create a form with single search bar - you can override it to create a advanced search form.
====search====
This function will do the searching job. You can obtain the POST parameters from the from the form you created in print_search function
This function will return a file list exactly like the one from get_listing.
====get_file====
When a user clicks the "Get" button to transfer the file, this function will be called. Basically, it will download a file to Moodle - you can override it to modify the file then move it to a better location.
====get_name====
This function will return the name of the repository instance.

== I18n - Internationalization ==
These following strings are required in ''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php'':

<code php>
$string['pluginname'] = 'Flickr Public';
$string['configplugin'] = 'Flickr Public configuration';
$string['pluginname_help'] = 'A Flickr public repository';
</code>

== Database Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

==See also==

*[[Repository API| Repository API]]
*[[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
* MDL-16543 - A list of officially supported repository plugins
* MDL-16543 - Template plugin for developers
[[Category:Repositories]]
[[Category:Plugins]]

Authentication plugins

2012-01-25T08:48:12Z

Dongsheng:

==Introduction==

This page first gives an '''overview''' of the ''authentication process'' and then explains how ''authentication modules'' can be '''created''' using hooks to take over from the native authentication in Moodle.
=== Overview of Moodle authentication process ===
[[File:1.9.11_login_element.png|203px||thumb|right|The login UI element in Moodle 1.9]]The authentication use case in Moodle starts when a user clicks on the Login link in the UI. Then the following happens (skipping some minor details and rarer scenarios):

# The default login page (<tt>/login/index.php</tt>) is displayed. OR, if a system administrator has set the Alternate Login URL on the "Manage authentication" page, that URL will be displayed.
# A user enters their credentials and submits the form.
# The handler code in <tt>/login/index.php</tt> runs:
## Gets a list of enabled authentication plugins.
## Runs <tt>loginpage_hook()</tt> for each plugin, in case any of them needs to intercept the login request.
## Checks to make sure that the username meets Moodle's criteria (alphanumeric, with periods and hyphens allowed).
## Calls <tt>authenticate_user_login()</tt> in <tt>/lib/moodlelib.php</tt>, which returns a <code>$user</code> object. (Details of this code follow this main outline.)
## Determines whether authentication was successful (by checking whether <code>$user</code> is a valid object) and, if not, sends them back to the login page with an error message. Otherwise, it figures out where to send the user based on their original page request, whether their password is expired, etc., and redirects them there.

==History==
Authentication plugins exist from 1.9
==Example==
==Template==
==Naming convention==
==File structure==
# Choose a name for your plugin. We'll use 'sentry' as an example below; change it to whatever name you have chosen.

# Under your Moodle installation root, create the directory <tt>/auth/sentry</tt>. It should be sibling to existing auth plugin directories: 'db', 'nologin', 'none', etc.

# Create the file <tt>/auth/sentry/auth.php</tt>. Within the file, create a class <tt>auth_plugin_sentry</tt> that extends <tt>auth_plugin_base</tt> from <tt>/lib/authlib.php</tt>. (You will need to <code>require_once</code> the authlib file.)

# Implement the <tt>user_login()</tt> function in your <tt>auth.php</tt> file, and create or override additional functions based on your plugin's requirements.

# Log in to your Moodle installation as a site administrator and find, in the site administrator block, the page "Users -> Authentication -> Manage authentication". You will see your plugin in the list, appearing as <nowiki>[[auth_sentrytitle]]</nowiki>. You can enable it and move it up and down in the order. '''At this point, with the plugin enabled, your plugin is registered and will be used by Moodle in its authentication process.'''

# If you don't like seeing <nowiki>[[auth_sentrytitle]]</nowiki> as the name of your plugin in the Moodle UI, you'll need to create language files for your plugin. Do this by creating the directory <tt>/auth/sentry/lang</tt>, and under it, a directory for each language that your installation needs to support. (Example: <tt>/auth/sentry/lang/en</tt>.) Within each of these language directories, create a file called <tt>auth_sentry.php</tt>. That file should set the desired value for <code>$string['auth_sentrytitle'] for that language (use $string['pluginname'] for Moodle2)</code>. You can also set the plugin description by setting <code>$string['auth_sentrydescription']</code>, and you can also assign other translatable strings that your plugin uses, in these files.

'''Note:''' A folder named like '''en''' may not work for everyone. Please refer to the lang folder under your moodle installation directory to get an idea about the language/locale codes used in your moodle installation. For example, the lang folder in own users system contained folders named '''en''' and '''zh-cn'''. He emulated the same in the lang folder of the auth plugin and the plugin picked up the title and description strings immediately. ([[Places_to_search_for_lang_strings|More info on how Moodle handles language]]).

# If you want to configure your plugin through the Moodle UI, implement <tt>config_form()</tt> and <tt>process_config()</tt> in the plugin class. You might find it convenient to use the 'db' plugin as a model for this. The plugin's config settings can then be managed through the Manage authentication page by clicking on the Settings link for that plugin, and the values will be stored in the <tt>mdl_config_pluginstable</tt> in the database.

==Interfacing to API's==

=== <tt>authenticate_user_login()</tt>===
That's the main outline, but a lot of interesting stuff happens in <tt>authenticate_user_login()</tt>:

# It gets a list of enabled authentication plugins.
# It looks up the username in the mdl_user table to see if they are allowed to log in, and which authentication plugin handles their login requests. (This will be the plugin that handled their first-ever login request.)
# It creates a user object, which will contain the data from <tt>mdl_user</tt> if the username is known; if not, it will be an empty object.
# It does the following with the authentication plugin (note that for a username unknown to Moodle, it will do these steps for each authenticated plugin until one succeeds or it has tried them all):
## Calls the <tt>user_login()</tt> function provided by that plugin, which returns a boolean value based on whether the credentials authenticate or not. If the result is false (not authenticated), skips the rest of the steps below and continues to the next plugin.
## If the plugin authenticates against an external system (not Moodle's user database), its <tt>update_user_record()</tt> function is called to get the user's name, contact info, etc.
## Creates the Moodle user record if it doesn't already exist.
## Calls the plugin's <tt>sync_roles()</tt> function.
## Notifies each enabled authentication plugin that the user successfully authenticated, by calling each one's <tt>user_authenticated_hook()</tt> function.
# It returns the user object if everything was successful, or false if no plugin was able to successfully authenticate the credentials.
=== <tt>user_login($username, $password)</tt>===
This must be rewritten by plugin to return boolean value, returns true if the username and password work and false if they are wrong or don't exist.

=== <tt>can_change_password()</tt>===
Returns true if this authentication plugin can change users' password.

=== <tt>change_password_url()</tt>===
Returns the URL for changing the users' passwords, or empty if the default URL can be used.

=== <tt>can_edit_profile()</tt>===
Returns true if this authentication plugin can edit the users' profile.

=== <tt>edit_profile_url()</tt>===
Returns the URL for editing users' profile, or empty if the defaults URL can be used.

=== <tt>prevent_local_passwords()</tt>===
Indicates if password hashes should be stored in local moodle database.

=== <tt>user_update_password($user, $newpassword)</tt>===
Update the user's password.

=== <tt>user_update($olduser, $newuser)</tt>===
Called when the user record is updated. It will modify the user information in external database.

=== <tt>user_delete($olduser)</tt>===
User delete requested. Internal user record had been deleted.

=== <tt>can_reset_password()</tt>===
Returns true if plugin allows resetting of internal password.

=== <tt>user_signup($user, $notify=true)</tt>===
Sign up a new user ready for confirmation, password is passed in plaintext.

=== <tt>can_confirm()</tt>===
Returns true if plugin allows confirming of new users.

=== <tt>user_confirm($username, $confirmsecret)</tt>===
Confirm the new user as registered.

=== <tt>user_exists($username)</tt>===
Checks if user exists in external db.

=== <tt>password_expire($username)</tt>===
Returns number of days to user password expires.

=== <tt>sync_roles()</tt>===
Sync roles for this user - usually creator

=== <tt>get_userinfo($username)</tt>===
Read user information from external database and returns it as array.

=== <tt>config_form($config, $err, $user_fields)</tt>===
Prints a form for configuring this authentication plugin. It's called from admin/auth.php, and outputs a full page with a form for configuring this plugin.

=== <tt>validate_form($form, $err)</tt>===
Validate form data.

=== <tt>process_config($config)</tt>===
Processes and stores configuration data for this authentication plugin.

=== <tt>loginpage_hook()</tt>===
Hook for overriding behaviour of login page.

=== <tt>user_authenticated_hook($user, $username, $password)</tt>===
Post authentication hook. This method is called from authenticate_user_login() for all enabled auth plugins.

=== <tt>prelogout_hook()</tt>===
Pre logout hook.

=== <tt>logoutpage_hook()</tt>===
Hook for overriding behaviour of logout page.

==See also==

* [[Authentication API]]
* Using Moodle [http://moodle.org/mod/forum/discuss.php?d=102070 Overview of entire authentication code flow] forum discussion
* Using Moodle [http://moodle.org/mod/forum/view.php?id=42 User authentication forum]
* Moodle Docs [[:en:Manage authentication|Manage authentication]]
* [[:en:Authentication FAQ|Authentication FAQ]]

[[Category:Authentication]]
[[Category:Plugins]]

[[fr:Méthodes_d'authentification]]

Repository plugins

2012-01-25T06:14:22Z

Dongsheng:

== Introduction ==

Repository plugin allow Moodle to bring contents into Moodle from external repositories.

===Prerequisites===
Before starting coding, it is necessary to know how to use repository administration pages and how to use the file picker.

===Overview===

The 3 different parts to write
# Administration - You can customise the way administrators and users can configure their repositories.
# File picker integration - The core of your plugin, it will manage communication between Moodle and the repository service, and also the file picker display.
# I18n - Internationalization should be done at the same time as you're writing the other parts.

== History ==

Repository plugins exists from 2.0

== Example ==
*[[Box.net Repository Plugin|Box.net Repository Plugin]]
*[[Flickr Repository Plugin|Flickr Repository Plugin]]
*[[Moodle Repository Plugin|Remote Moodle Repository Plugin]]

== Template ==
There is a template available in [https://github.com/dongsheng/moodle-repository_demo Dongsheng Cai's github]

== Naming convention==

# Create a folder for your plugin in ''moodle/repository/'' e.g. ''moodle/repository/myplugin''
# Create the following files and add them to the plugin folder:
#* ''lib.php'', the main class will be named "repository_myplugin"
#* ''pix/icon.png'' (the icon displayed in the file picker)
#* ''[[version.php]]''
# Create the language file ''repository_myplugin.php'' and add it to the plugin folder, keeping the following folder structure:
#*''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php''
# Create access.php and upgrade scripts (optional)

==Administration APIs==

===Functions===
All of the following functions are optional. If they're not implemented, your plugin will not have manual settings and will have only one instance displayed in the File Picker (The repository API creates this unique instance when the administrator add the plugin).

==== get_instance_option_names====
''This function must be declared static'' 
Return an array of strings. These strings are setting names. These settings are specific to an instance.
If the function returns an empty array, the API will consider that the plugin displays only one repository in the file picker.
Parent function returns an empty array.

====instance_config_form(&$mform)====
This is for modifying the Moodle form displaying the settings specific to an instance.

For example, to add a required text box called email_address:
<code php>
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', $strrequired, 'required', null, 'client');
</code>

''Note: ''mform'' has by default a name text box (cannot be removed).''

Parent function does nothing.

====instance_form_validation($mform, $data, $errors)====
This allows us to validate what has been submitted in the instance configuration form. For instance:
<code php>
public static function instance_form_validation($mform, $data, $errors) {
if (empty($data['email_address'])) {
$errors['email_address'] = get_string('invalidemailsettingname', 'repository_flickr_public');
}
return $errors;
}
</code>

====get_type_option_names====
''This function must be declared static'' 
Return an array of string. These strings are setting names. These settings are shared by all instances.
Parent function return an empty array.

For example:
<code php>
public static function get_type_option_names() {
return array_merge(parent::get_type_option_names(), array('rootpath'));
}
</code>

====type_config_form(&$mform)====
This is for modifying the Moodle form displaying the plugin settings.

For example, to display the standard repository plugin settings along with the custom ones use:
<code php>
public function type_config_form($mform) {
parent::type_config_form($mform);

$rootpath = get_config('repository_someplugin', 'rootpath');
$mform->addElement('text', 'rootpath', get_string('rootpath', 'repository_someplugin'), array('size' => '40'));
$mform->setDefault('rootpath', $rootpath);
}
</code>

====type_form_validation($mform, $data, $errors)====
Use this function if you need to validate some variables submitted by plugin settings form. Push the items to $error array in the format ("fieldname" => "errormessage") to have them highlighted in the form.

With the example above, this function may look like:
<code php>
public static function type_form_validation($mform, $data, $errors) {
if (!is_dir($data['rootpath'])) {
$errors['rootpath'] = get_string('invalidrootpath', 'repository_someplugin');
}
return $errors;
}
</code>

====plugin_init()====
''This function must be declared static'' 
This function is called when the administrator adds the plugin. So unless the administrator deletes the plugin and re-adds it, it should be called only once.
Parent function does nothing.

As an example, let's create a Flickr plugin for accessing a public flickr account. The plugin will be called "Flickr Public".

Firstly the skeleton:

<code php>
<?php
/**
* repository_flickr_public class
* Moodle user can access public flickr account
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class repository_flickr_public extends repository {
}
?>
</code>

Then consider the question "What does my plugin do?"

In the Moodle file picker, we want to display some flickr public repositories directly linked to a flickr public account. For example ''My Public Flickr Pictures'', and also ''My Friend's Flickr Pictures''. When the user clicks on one of these repositories, the public pictures are displayed in the file picker.

In order to access to a flickr public account, the plugin needs to know the email address of the Flickr public account owner. So the administrator will need to set an email address for every repository. Let's add an "email address" setting to every repository.

<code php>
//We tell the API that the repositories have specific settings: "email address"
public static function get_instance_option_names() {
return array('email_address');
}

//We add an "email address" text box to the create/edit repository instance Moodle form
public function instance_config_form(&$mform) {
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', get_string('required'), 'required', null, 'client');
}
</code>

So at this moment all our Flickr Public Repositories will have a specific email address. However this is not enough. In order to communicate with Flickr, Moodle needs to know a Flickr API key (http://www.flickr.com/services/api/). This API key is the same for any repository. We could add it with the email address setting but the administrator would have to enter the same API key for every repository. Hopefully the administrator can add settings to the plugin level, impacting all repositories. The code is similar the repository instance settings:
<code php>
//We tell the API that the repositories have general settings: "api_key"
public static function get_type_option_names() {
return array('api_key');
}

//We add an "api key" text box to the create/edit repository plugin Moodle form (also called a Repository type Moodle form)
public function type_config_form(&$mform) {
//the following line is needed in order to retrieve the API key value from the database when Moodle displays the edit form
$api_key = get_config('flickr_public', 'api_key');
$mform->addElement('text', 'api_key', get_string('apikey', 'repository_flickr_public'),
array('value'=>$api_key,'size' => '40'));
$mform->addRule('api_key', get_string('required'), 'required', null, 'client');
}
</code>

Have we finished yet?

Yes! We have created everything necessary for the administration pages. But let's go further. It would be good if the user can enter any "Flickr public account email address" in the file picker. In fact we want to display in the file picker a Flickr Public repository that the Moodle administrator can never delete. Let's add:

<code php>
//this function is only called one time, when the Moodle administrator add the Flickr Public Plugin into the Moodle site.
public static function plugin_init() {
//here we create a default repository instance. The last parameter is 1 in order to set the instance as readonly.
repository_static_function('flickr_public','create', 'flickr_public', 0, get_system_context(),
array('name' => 'default instance','email_address' => null),1);
}
</code>

That's all - the administration part of our Flickr Public plugin is done. For your information, Box.net, Flickr, and Flickr Public all have similar administration APIs.

==Repository APIs==
=== Quick Start ===
First of all, the File Picker using intensively Ajax you will need a easy way to debug. Install FirePHP (MDL-16371) and make it works. It will save you a lot of time. (You might give the [http://moodle.org/mod/forum/discuss.php?d=119961 FirePHP plugin for Moodle] a try, it's still work in progress, though.)

* Your first question when you write your plugin specification is 'Does the user need to log-in'? If they do, in your plugin you have to detect user session in constructor() function, and use print_login() if required, see more details below.
* For most of plugins, you need to establish a connection with the remote repository. This connection can be done into the get_listing(), constructor() function, see more details below.
* You wanna retrieve the file that the user selected, rewrite get_file() if required, see more details below.
* Optional question that you should ask yourself is 'Does the user can execute a search', if they do, you will have to rewrite search() method, see more details below.

===Functions you *MUST* override===
====__construct====
You may initialize your plugin here, such as:
# Get options from database
# Get user name and password from HTTP POST

====get_listing($path="", $page="")====
This function will return a list of files, the list must be a array like this:
<code php>
$list = array(
//this will be used to build navigation bar
'path'=>array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder')),
'manage'=>'http://webmgr.moodle.com',
'list'=> array(
array('title'=>'filename1', 'date'=>'01/01/2009', 'size'=>'10MB', 'source'=>'http://www.moodle.com/dl.rar'),
array('title'=>'folder', 'date'=>'01/01/2009', 'size'=>'0', 'children'=>array())
)
);
</code>
'''The full specification of list element:'''
<code php>
array(
// Used to build navegation bar, so you need to include all parents folders
// array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder'))
'path' => (array) this will be used to build navigation bar
// dynload tells file picker to fetch list dynamically, when user click
// the folder, it will send a ajax request to server side.
// if you are using pagination, 'page' and 'pages' parameters should be used
// and note, you'd better don't use pagination and page at the same time
'page' => (int) which page is this list
'pages' => (pages) how many pages
'dynload' => (bool) use dynamic loading,
// will display a link in file picker
'manage' => (string) url of the file manager,
// set to true, the login link will be removed from file picker
'nologin' => (bool) requires login,
// set to true, the search link will be removed from file picker
'nosearch' => (bool) no search link,
// set this option will display a upload form in file picker
// only used in upload plugin currently
'upload' => array( // upload manager
'label' => (string) label of the form element,
'id' => (string) id of the form element
),
// file picker will build a file tree according this
// list
'list' => array(
array( // file
'title' => (string) file name,
'shorttitle' => (string) optional, if you prefer to display a short title
'date' => (string) file last modification time, usually userdate(...),
'size' => (int) file size,
'thumbnail' => (string) url to thumbnail for the file,
'thumbnail_width' => (int) the width of the thumbnail image,
'source' => plugin-dependent unique path to the file (id, url, path, etc.),
'url'=> the accessible url of file
),
array( // folder - same as file, but no 'source'.
'title' => (string) folder name,
'shorttitle' => (string) optional, if you prefer to display a short title
'path' => (string) path to this folder
'date' => (string) folder last modification time, usually userdate(...),
'size' => 0,
'thumbnail' => (string) url to thumbnail for the folder,
'children' => array( // an empty folder needs to have 'children' defined, but empty.
// content (files and folders)
)
),
)
)
</code>
Dynamically loading
Some repositories contain many files which cannot load in one time, in this case, we need dynamically loading to fetch them step by step, files in subfolder won't be listed until user click the folder in file picker treeview.

As a plug-in developer, if you set dynload flag as '''true''', you should return files and folders (set children as a null array) in current path instead of building the whole file tree.

Example of dynamically loading
See [http://cvs.moodle.org/moodle/repository/alfresco/lib.php?view=log Alfresco] plug-in

===Functions you can override===
====print_login====
This function will help to print a login form, for the Ajax file picker, this function will return a
PHP array to define this form.
<code php>
public function print_login(){
if ($this->options['ajax']) {
$user_field->label = get_string('username', 'repository_boxnet').': ';
$user_field->id = 'box_username';
$user_field->type = 'text';
$user_field->name = 'boxusername';
$user_field->value = $ret->username;

$passwd_field->label = get_string('password', 'repository_boxnet').': ';
$passwd_field->id = 'box_password';
$passwd_field->type = 'password';
$passwd_field->name = 'boxpassword';

$ret = array();
$ret['login'] = array($user_field, $passwd_field);
// if you are going to rename submit button label, use this option
//$ret['login_btn_label'] = get_string('submit_btn_label');
// if you are going to display a search form instead of login form
// set this option to true
//$ret['login_search_form'] = true;
return $ret;
}
}
</code>
This will help to generate a form by file picker which contains user name and password input elements.

If your plugin don't require logging in, you don't need to override it, it will call get_listing to list files automatically by default.
====check_login====
This function will return a boolean value to tell Moodle whether the user has logged in.
By default, this function will return true.
====logout====
When a user clicks the logout button in file picker, this function will be called. You may clean up the session or disconnect the connection with remote server here.
====print_search====
When a user clicks the search button on file picker, this function will be called to return a search form. By default, it will create a form with single search bar - you can override it to create a advanced search form.
====search====
This function will do the searching job. You can obtain the POST parameters from the from the form you created in print_search function
This function will return a file list exactly like the one from get_listing.
====get_file====
When a user clicks the "Get" button to transfer the file, this function will be called. Basically, it will download a file to Moodle - you can override it to modify the file then move it to a better location.
====get_name====
This function will return the name of the repository instance.

== I18n - Internationalization ==
These following strings are required in ''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php'':

<code php>
$string['pluginname'] = 'Flickr Public';
$string['configplugin'] = 'Flickr Public configuration';
$string['pluginname_help'] = 'A Flickr public repository';
</code>

== Database Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

==See also==

*[[Repository API| Repository API]]
*[[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
* MDL-16543 - A list of officially supported repository plugins
* MDL-16543 - Template plugin for developers
[[Category:Repositories]]
[[Category:Plugins]]

Repository plugins

2012-01-25T06:13:26Z

Dongsheng: /* I18n - Internationalization */

== Introduction ==

Repository plugin allow Moodle to bring contents into Moodle from external repositories.

===Prerequisites===
Before starting coding, it is necessary to know how to use repository administration pages and how to use the file picker.

===Overview===

The 3 different parts to write
# Administration - You can customise the way administrators and users can configure their repositories.
# File picker integration - The core of your plugin, it will manage communication between Moodle and the repository service, and also the file picker display.
# I18n - Internationalization should be done at the same time as you're writing the other parts.

== History ==

Repository plugins exists from 2.0

== Example ==
*[[Box.net Repository Plugin|Box.net Repository Plugin]]
*[[Flickr Repository Plugin|Flickr Repository Plugin]]
*[[Moodle Repository Plugin|Remote Moodle Repository Plugin]]

== Template ==
There is a template available in [https://github.com/dongsheng/moodle-repository_demo Dongsheng Cai's github]

== Naming convention==

# Create a folder for your plugin in ''moodle/repository/'' e.g. ''moodle/repository/myplugin''
# Create the following files and add them to the plugin folder:
#* ''lib.php'', the main class will be named "repository_myplugin"
#* ''pix/icon.png'' (the icon displayed in the file picker)
#* ''[[version.php]]''
# Create the language file ''repository_myplugin.php'' and add it to the plugin folder, keeping the following folder structure:
#*''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php''
# Create access.php and upgrade scripts (optional)

==Administration APIs==

===Functions===
All of the following functions are optional. If they're not implemented, your plugin will not have manual settings and will have only one instance displayed in the File Picker (The repository API creates this unique instance when the administrator add the plugin).

==== get_instance_option_names====
''This function must be declared static'' 
Return an array of strings. These strings are setting names. These settings are specific to an instance.
If the function returns an empty array, the API will consider that the plugin displays only one repository in the file picker.
Parent function returns an empty array.

====instance_config_form(&$mform)====
This is for modifying the Moodle form displaying the settings specific to an instance.

For example, to add a required text box called email_address:
<code php>
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', $strrequired, 'required', null, 'client');
</code>

''Note: ''mform'' has by default a name text box (cannot be removed).''

Parent function does nothing.

====instance_form_validation($mform, $data, $errors)====
This allows us to validate what has been submitted in the instance configuration form. For instance:
<code php>
public static function instance_form_validation($mform, $data, $errors) {
if (empty($data['email_address'])) {
$errors['email_address'] = get_string('invalidemailsettingname', 'repository_flickr_public');
}
return $errors;
}
</code>

====get_type_option_names====
''This function must be declared static'' 
Return an array of string. These strings are setting names. These settings are shared by all instances.
Parent function return an empty array.

For example:
<code php>
public static function get_type_option_names() {
return array_merge(parent::get_type_option_names(), array('rootpath'));
}
</code>

====type_config_form(&$mform)====
This is for modifying the Moodle form displaying the plugin settings.

For example, to display the standard repository plugin settings along with the custom ones use:
<code php>
public function type_config_form($mform) {
parent::type_config_form($mform);

$rootpath = get_config('repository_someplugin', 'rootpath');
$mform->addElement('text', 'rootpath', get_string('rootpath', 'repository_someplugin'), array('size' => '40'));
$mform->setDefault('rootpath', $rootpath);
}
</code>

====type_form_validation($mform, $data, $errors)====
Use this function if you need to validate some variables submitted by plugin settings form. Push the items to $error array in the format ("fieldname" => "errormessage") to have them highlighted in the form.

With the example above, this function may look like:
<code php>
public static function type_form_validation($mform, $data, $errors) {
if (!is_dir($data['rootpath'])) {
$errors['rootpath'] = get_string('invalidrootpath', 'repository_someplugin');
}
return $errors;
}
</code>

====plugin_init()====
''This function must be declared static'' 
This function is called when the administrator adds the plugin. So unless the administrator deletes the plugin and re-adds it, it should be called only once.
Parent function does nothing.

As an example, let's create a Flickr plugin for accessing a public flickr account. The plugin will be called "Flickr Public".

Firstly the skeleton:

<code php>
<?php
/**
* repository_flickr_public class
* Moodle user can access public flickr account
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class repository_flickr_public extends repository {
}
?>
</code>

Then consider the question "What does my plugin do?"

In the Moodle file picker, we want to display some flickr public repositories directly linked to a flickr public account. For example ''My Public Flickr Pictures'', and also ''My Friend's Flickr Pictures''. When the user clicks on one of these repositories, the public pictures are displayed in the file picker.

In order to access to a flickr public account, the plugin needs to know the email address of the Flickr public account owner. So the administrator will need to set an email address for every repository. Let's add an "email address" setting to every repository.

<code php>
//We tell the API that the repositories have specific settings: "email address"
public static function get_instance_option_names() {
return array('email_address');
}

//We add an "email address" text box to the create/edit repository instance Moodle form
public function instance_config_form(&$mform) {
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', get_string('required'), 'required', null, 'client');
}
</code>

So at this moment all our Flickr Public Repositories will have a specific email address. However this is not enough. In order to communicate with Flickr, Moodle needs to know a Flickr API key (http://www.flickr.com/services/api/). This API key is the same for any repository. We could add it with the email address setting but the administrator would have to enter the same API key for every repository. Hopefully the administrator can add settings to the plugin level, impacting all repositories. The code is similar the repository instance settings:
<code php>
//We tell the API that the repositories have general settings: "api_key"
public static function get_type_option_names() {
return array('api_key');
}

//We add an "api key" text box to the create/edit repository plugin Moodle form (also called a Repository type Moodle form)
public function type_config_form(&$mform) {
//the following line is needed in order to retrieve the API key value from the database when Moodle displays the edit form
$api_key = get_config('flickr_public', 'api_key');
$mform->addElement('text', 'api_key', get_string('apikey', 'repository_flickr_public'),
array('value'=>$api_key,'size' => '40'));
$mform->addRule('api_key', get_string('required'), 'required', null, 'client');
}
</code>

Have we finished yet?

Yes! We have created everything necessary for the administration pages. But let's go further. It would be good if the user can enter any "Flickr public account email address" in the file picker. In fact we want to display in the file picker a Flickr Public repository that the Moodle administrator can never delete. Let's add:

<code php>
//this function is only called one time, when the Moodle administrator add the Flickr Public Plugin into the Moodle site.
public static function plugin_init() {
//here we create a default repository instance. The last parameter is 1 in order to set the instance as readonly.
repository_static_function('flickr_public','create', 'flickr_public', 0, get_system_context(),
array('name' => 'default instance','email_address' => null),1);
}
</code>

That's all - the administration part of our Flickr Public plugin is done. For your information, Box.net, Flickr, and Flickr Public all have similar administration APIs.

==Repository APIs==
=== Quick Start ===
First of all, the File Picker using intensively Ajax you will need a easy way to debug. Install FirePHP (MDL-16371) and make it works. It will save you a lot of time. (You might give the [http://moodle.org/mod/forum/discuss.php?d=119961 FirePHP plugin for Moodle] a try, it's still work in progress, though.)

* Your first question when you write your plugin specification is 'Does the user need to log-in'? If they do, in your plugin you have to detect user session in constructor() function, and use print_login() if required, see more details below.
* For most of plugins, you need to establish a connection with the remote repository. This connection can be done into the get_listing(), constructor() function, see more details below.
* You wanna retrieve the file that the user selected, rewrite get_file() if required, see more details below.
* Optional question that you should ask yourself is 'Does the user can execute a search', if they do, you will have to rewrite search() method, see more details below.

===Functions you *MUST* override===
====__construct====
You may initialize your plugin here, such as:
# Get options from database
# Get user name and password from HTTP POST

====get_listing($path="", $page="")====
This function will return a list of files, the list must be a array like this:
<code php>
$list = array(
//this will be used to build navigation bar
'path'=>array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder')),
'manage'=>'http://webmgr.moodle.com',
'list'=> array(
array('title'=>'filename1', 'date'=>'01/01/2009', 'size'=>'10MB', 'source'=>'http://www.moodle.com/dl.rar'),
array('title'=>'folder', 'date'=>'01/01/2009', 'size'=>'0', 'children'=>array())
)
);
</code>
'''The full specification of list element:'''
<code php>
array(
// Used to build navegation bar, so you need to include all parents folders
// array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder'))
'path' => (array) this will be used to build navigation bar
// dynload tells file picker to fetch list dynamically, when user click
// the folder, it will send a ajax request to server side.
// if you are using pagination, 'page' and 'pages' parameters should be used
// and note, you'd better don't use pagination and page at the same time
'page' => (int) which page is this list
'pages' => (pages) how many pages
'dynload' => (bool) use dynamic loading,
// will display a link in file picker
'manage' => (string) url of the file manager,
// set to true, the login link will be removed from file picker
'nologin' => (bool) requires login,
// set to true, the search link will be removed from file picker
'nosearch' => (bool) no search link,
// set this option will display a upload form in file picker
// only used in upload plugin currently
'upload' => array( // upload manager
'label' => (string) label of the form element,
'id' => (string) id of the form element
),
// file picker will build a file tree according this
// list
'list' => array(
array( // file
'title' => (string) file name,
'shorttitle' => (string) optional, if you prefer to display a short title
'date' => (string) file last modification time, usually userdate(...),
'size' => (int) file size,
'thumbnail' => (string) url to thumbnail for the file,
'thumbnail_width' => (int) the width of the thumbnail image,
'source' => plugin-dependent unique path to the file (id, url, path, etc.),
'url'=> the accessible url of file
),
array( // folder - same as file, but no 'source'.
'title' => (string) folder name,
'shorttitle' => (string) optional, if you prefer to display a short title
'path' => (string) path to this folder
'date' => (string) folder last modification time, usually userdate(...),
'size' => 0,
'thumbnail' => (string) url to thumbnail for the folder,
'children' => array( // an empty folder needs to have 'children' defined, but empty.
// content (files and folders)
)
),
)
)
</code>
Dynamically loading
Some repositories contain many files which cannot load in one time, in this case, we need dynamically loading to fetch them step by step, files in subfolder won't be listed until user click the folder in file picker treeview.

As a plug-in developer, if you set dynload flag as '''true''', you should return files and folders (set children as a null array) in current path instead of building the whole file tree.

Example of dynamically loading
See [http://cvs.moodle.org/moodle/repository/alfresco/lib.php?view=log Alfresco] plug-in

===Functions you can override===
====print_login====
This function will help to print a login form, for the Ajax file picker, this function will return a
PHP array to define this form.
<code php>
public function print_login(){
if ($this->options['ajax']) {
$user_field->label = get_string('username', 'repository_boxnet').': ';
$user_field->id = 'box_username';
$user_field->type = 'text';
$user_field->name = 'boxusername';
$user_field->value = $ret->username;

$passwd_field->label = get_string('password', 'repository_boxnet').': ';
$passwd_field->id = 'box_password';
$passwd_field->type = 'password';
$passwd_field->name = 'boxpassword';

$ret = array();
$ret['login'] = array($user_field, $passwd_field);
// if you are going to rename submit button label, use this option
//$ret['login_btn_label'] = get_string('submit_btn_label');
// if you are going to display a search form instead of login form
// set this option to true
//$ret['login_search_form'] = true;
return $ret;
}
}
</code>
This will help to generate a form by file picker which contains user name and password input elements.

If your plugin don't require logging in, you don't need to override it, it will call get_listing to list files automatically by default.
====check_login====
This function will return a boolean value to tell Moodle whether the user has logged in.
By default, this function will return true.
====logout====
When a user clicks the logout button in file picker, this function will be called. You may clean up the session or disconnect the connection with remote server here.
====print_search====
When a user clicks the search button on file picker, this function will be called to return a search form. By default, it will create a form with single search bar - you can override it to create a advanced search form.
====search====
This function will do the searching job. You can obtain the POST parameters from the from the form you created in print_search function
This function will return a file list exactly like the one from get_listing.
====get_file====
When a user clicks the "Get" button to transfer the file, this function will be called. Basically, it will download a file to Moodle - you can override it to modify the file then move it to a better location.
====get_name====
This function will return the name of the repository instance.

== I18n - Internationalization ==
These following strings are required in ''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php'':

<code php>
$string['pluginname'] = 'Flickr Public';
$string['configplugin'] = 'Flickr Public configuration';
$string['pluginname_help'] = 'A Flickr public repository';
</code>

== Database Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

== Standard repository plugins ==
This is the functional specification list of the officially supported repository plugins.
For each plugins, the two mains part we are interested in are:
* How do I administer the plugin? See [[Repository_Administration_Specification| Repository Administration Specification - UC001-3]]
* How do I set up an account for this repository? See [[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]

==See also==

*[[Repository API| Repository API]]
*[[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
* MDL-16543 - A list of officially supported repository plugins
* MDL-16543 - Template plugin for developers
[[Category:Repositories]]
[[Category:Plugins]]

Repository plugins

2012-01-25T06:07:20Z

Dongsheng:

== Introduction ==

Repository plugin allow Moodle to bring contents into Moodle from external repositories.

===Prerequisites===
Before starting coding, it is necessary to know how to use repository administration pages and how to use the file picker.

===Overview===

The 3 different parts to write
# Administration - You can customise the way administrators and users can configure their repositories.
# File picker integration - The core of your plugin, it will manage communication between Moodle and the repository service, and also the file picker display.
# I18n - Internationalization should be done at the same time as you're writing the other parts.

== History ==

Repository plugins exists from 2.0

== Example ==
*[[Box.net Repository Plugin|Box.net Repository Plugin]]
*[[Flickr Repository Plugin|Flickr Repository Plugin]]
*[[Moodle Repository Plugin|Remote Moodle Repository Plugin]]

== Template ==
There is a template available in [https://github.com/dongsheng/moodle-repository_demo Dongsheng Cai's github]

== Naming convention==

# Create a folder for your plugin in ''moodle/repository/'' e.g. ''moodle/repository/myplugin''
# Create the following files and add them to the plugin folder:
#* ''lib.php'', the main class will be named "repository_myplugin"
#* ''pix/icon.png'' (the icon displayed in the file picker)
#* ''[[version.php]]''
# Create the language file ''repository_myplugin.php'' and add it to the plugin folder, keeping the following folder structure:
#*''moodle/repository/myplugin/lang/en/repository_myplugin.php'' or ''moodle/lang/en/repository_myplugin.php''
# Create access.php and upgrade scripts (optional)

==Administration APIs==

===Functions===
All of the following functions are optional. If they're not implemented, your plugin will not have manual settings and will have only one instance displayed in the File Picker (The repository API creates this unique instance when the administrator add the plugin).

==== get_instance_option_names====
''This function must be declared static'' 
Return an array of strings. These strings are setting names. These settings are specific to an instance.
If the function returns an empty array, the API will consider that the plugin displays only one repository in the file picker.
Parent function returns an empty array.

====instance_config_form(&$mform)====
This is for modifying the Moodle form displaying the settings specific to an instance.

For example, to add a required text box called email_address:
<code php>
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', $strrequired, 'required', null, 'client');
</code>

''Note: ''mform'' has by default a name text box (cannot be removed).''

Parent function does nothing.

====instance_form_validation($mform, $data, $errors)====
This allows us to validate what has been submitted in the instance configuration form. For instance:
<code php>
public static function instance_form_validation($mform, $data, $errors) {
if (empty($data['email_address'])) {
$errors['email_address'] = get_string('invalidemailsettingname', 'repository_flickr_public');
}
return $errors;
}
</code>

====get_type_option_names====
''This function must be declared static'' 
Return an array of string. These strings are setting names. These settings are shared by all instances.
Parent function return an empty array.

For example:
<code php>
public static function get_type_option_names() {
return array_merge(parent::get_type_option_names(), array('rootpath'));
}
</code>

====type_config_form(&$mform)====
This is for modifying the Moodle form displaying the plugin settings.

For example, to display the standard repository plugin settings along with the custom ones use:
<code php>
public function type_config_form($mform) {
parent::type_config_form($mform);

$rootpath = get_config('repository_someplugin', 'rootpath');
$mform->addElement('text', 'rootpath', get_string('rootpath', 'repository_someplugin'), array('size' => '40'));
$mform->setDefault('rootpath', $rootpath);
}
</code>

====type_form_validation($mform, $data, $errors)====
Use this function if you need to validate some variables submitted by plugin settings form. Push the items to $error array in the format ("fieldname" => "errormessage") to have them highlighted in the form.

With the example above, this function may look like:
<code php>
public static function type_form_validation($mform, $data, $errors) {
if (!is_dir($data['rootpath'])) {
$errors['rootpath'] = get_string('invalidrootpath', 'repository_someplugin');
}
return $errors;
}
</code>

====plugin_init()====
''This function must be declared static'' 
This function is called when the administrator adds the plugin. So unless the administrator deletes the plugin and re-adds it, it should be called only once.
Parent function does nothing.

As an example, let's create a Flickr plugin for accessing a public flickr account. The plugin will be called "Flickr Public".

Firstly the skeleton:

<code php>
<?php
/**
* repository_flickr_public class
* Moodle user can access public flickr account
*
* @license http://www.gnu.org/copyleft/gpl.html GNU Public License
*/
class repository_flickr_public extends repository {
}
?>
</code>

Then consider the question "What does my plugin do?"

In the Moodle file picker, we want to display some flickr public repositories directly linked to a flickr public account. For example ''My Public Flickr Pictures'', and also ''My Friend's Flickr Pictures''. When the user clicks on one of these repositories, the public pictures are displayed in the file picker.

In order to access to a flickr public account, the plugin needs to know the email address of the Flickr public account owner. So the administrator will need to set an email address for every repository. Let's add an "email address" setting to every repository.

<code php>
//We tell the API that the repositories have specific settings: "email address"
public static function get_instance_option_names() {
return array('email_address');
}

//We add an "email address" text box to the create/edit repository instance Moodle form
public function instance_config_form(&$mform) {
$mform->addElement('text', 'email_address', get_string('emailaddress', 'repository_flickr_public'));
$mform->addRule('email_address', get_string('required'), 'required', null, 'client');
}
</code>

So at this moment all our Flickr Public Repositories will have a specific email address. However this is not enough. In order to communicate with Flickr, Moodle needs to know a Flickr API key (http://www.flickr.com/services/api/). This API key is the same for any repository. We could add it with the email address setting but the administrator would have to enter the same API key for every repository. Hopefully the administrator can add settings to the plugin level, impacting all repositories. The code is similar the repository instance settings:
<code php>
//We tell the API that the repositories have general settings: "api_key"
public static function get_type_option_names() {
return array('api_key');
}

//We add an "api key" text box to the create/edit repository plugin Moodle form (also called a Repository type Moodle form)
public function type_config_form(&$mform) {
//the following line is needed in order to retrieve the API key value from the database when Moodle displays the edit form
$api_key = get_config('flickr_public', 'api_key');
$mform->addElement('text', 'api_key', get_string('apikey', 'repository_flickr_public'),
array('value'=>$api_key,'size' => '40'));
$mform->addRule('api_key', get_string('required'), 'required', null, 'client');
}
</code>

Have we finished yet?

Yes! We have created everything necessary for the administration pages. But let's go further. It would be good if the user can enter any "Flickr public account email address" in the file picker. In fact we want to display in the file picker a Flickr Public repository that the Moodle administrator can never delete. Let's add:

<code php>
//this function is only called one time, when the Moodle administrator add the Flickr Public Plugin into the Moodle site.
public static function plugin_init() {
//here we create a default repository instance. The last parameter is 1 in order to set the instance as readonly.
repository_static_function('flickr_public','create', 'flickr_public', 0, get_system_context(),
array('name' => 'default instance','email_address' => null),1);
}
</code>

That's all - the administration part of our Flickr Public plugin is done. For your information, Box.net, Flickr, and Flickr Public all have similar administration APIs.

==Repository APIs==
=== Quick Start ===
First of all, the File Picker using intensively Ajax you will need a easy way to debug. Install FirePHP (MDL-16371) and make it works. It will save you a lot of time. (You might give the [http://moodle.org/mod/forum/discuss.php?d=119961 FirePHP plugin for Moodle] a try, it's still work in progress, though.)

* Your first question when you write your plugin specification is 'Does the user need to log-in'? If they do, in your plugin you have to detect user session in constructor() function, and use print_login() if required, see more details below.
* For most of plugins, you need to establish a connection with the remote repository. This connection can be done into the get_listing(), constructor() function, see more details below.
* You wanna retrieve the file that the user selected, rewrite get_file() if required, see more details below.
* Optional question that you should ask yourself is 'Does the user can execute a search', if they do, you will have to rewrite search() method, see more details below.

===Functions you *MUST* override===
====__construct====
You may initialize your plugin here, such as:
# Get options from database
# Get user name and password from HTTP POST

====get_listing($path="", $page="")====
This function will return a list of files, the list must be a array like this:
<code php>
$list = array(
//this will be used to build navigation bar
'path'=>array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder')),
'manage'=>'http://webmgr.moodle.com',
'list'=> array(
array('title'=>'filename1', 'date'=>'01/01/2009', 'size'=>'10MB', 'source'=>'http://www.moodle.com/dl.rar'),
array('title'=>'folder', 'date'=>'01/01/2009', 'size'=>'0', 'children'=>array())
)
);
</code>
'''The full specification of list element:'''
<code php>
array(
// Used to build navegation bar, so you need to include all parents folders
// array(array('name'=>'root','path'=>'/'), array('name'=>'subfolder', 'path'=>'/subfolder'))
'path' => (array) this will be used to build navigation bar
// dynload tells file picker to fetch list dynamically, when user click
// the folder, it will send a ajax request to server side.
// if you are using pagination, 'page' and 'pages' parameters should be used
// and note, you'd better don't use pagination and page at the same time
'page' => (int) which page is this list
'pages' => (pages) how many pages
'dynload' => (bool) use dynamic loading,
// will display a link in file picker
'manage' => (string) url of the file manager,
// set to true, the login link will be removed from file picker
'nologin' => (bool) requires login,
// set to true, the search link will be removed from file picker
'nosearch' => (bool) no search link,
// set this option will display a upload form in file picker
// only used in upload plugin currently
'upload' => array( // upload manager
'label' => (string) label of the form element,
'id' => (string) id of the form element
),
// file picker will build a file tree according this
// list
'list' => array(
array( // file
'title' => (string) file name,
'shorttitle' => (string) optional, if you prefer to display a short title
'date' => (string) file last modification time, usually userdate(...),
'size' => (int) file size,
'thumbnail' => (string) url to thumbnail for the file,
'thumbnail_width' => (int) the width of the thumbnail image,
'source' => plugin-dependent unique path to the file (id, url, path, etc.),
'url'=> the accessible url of file
),
array( // folder - same as file, but no 'source'.
'title' => (string) folder name,
'shorttitle' => (string) optional, if you prefer to display a short title
'path' => (string) path to this folder
'date' => (string) folder last modification time, usually userdate(...),
'size' => 0,
'thumbnail' => (string) url to thumbnail for the folder,
'children' => array( // an empty folder needs to have 'children' defined, but empty.
// content (files and folders)
)
),
)
)
</code>
Dynamically loading
Some repositories contain many files which cannot load in one time, in this case, we need dynamically loading to fetch them step by step, files in subfolder won't be listed until user click the folder in file picker treeview.

As a plug-in developer, if you set dynload flag as '''true''', you should return files and folders (set children as a null array) in current path instead of building the whole file tree.

Example of dynamically loading
See [http://cvs.moodle.org/moodle/repository/alfresco/lib.php?view=log Alfresco] plug-in

===Functions you can override===
====print_login====
This function will help to print a login form, for the Ajax file picker, this function will return a
PHP array to define this form.
<code php>
public function print_login(){
if ($this->options['ajax']) {
$user_field->label = get_string('username', 'repository_boxnet').': ';
$user_field->id = 'box_username';
$user_field->type = 'text';
$user_field->name = 'boxusername';
$user_field->value = $ret->username;

$passwd_field->label = get_string('password', 'repository_boxnet').': ';
$passwd_field->id = 'box_password';
$passwd_field->type = 'password';
$passwd_field->name = 'boxpassword';

$ret = array();
$ret['login'] = array($user_field, $passwd_field);
// if you are going to rename submit button label, use this option
//$ret['login_btn_label'] = get_string('submit_btn_label');
// if you are going to display a search form instead of login form
// set this option to true
//$ret['login_search_form'] = true;
return $ret;
}
}
</code>
This will help to generate a form by file picker which contains user name and password input elements.

If your plugin don't require logging in, you don't need to override it, it will call get_listing to list files automatically by default.
====check_login====
This function will return a boolean value to tell Moodle whether the user has logged in.
By default, this function will return true.
====logout====
When a user clicks the logout button in file picker, this function will be called. You may clean up the session or disconnect the connection with remote server here.
====print_search====
When a user clicks the search button on file picker, this function will be called to return a search form. By default, it will create a form with single search bar - you can override it to create a advanced search form.
====search====
This function will do the searching job. You can obtain the POST parameters from the from the form you created in print_search function
This function will return a file list exactly like the one from get_listing.
====get_file====
When a user clicks the "Get" button to transfer the file, this function will be called. Basically, it will download a file to Moodle - you can override it to modify the file then move it to a better location.
====get_name====
This function will return the name of the repository instance.

== I18n - Internationalization ==
These following strings are required in ''moodle/repository/myplugin/lang/en_utf8/repository_myplugin.php'' or ''moodle/lang/en_utf8/repository_myplugin.php'':

<code php>
$string['configplugin'] = 'Flickr Public configuration';
$string['repositorydesc'] = 'A Flickr public repository';
$string['repositoryname'] = 'Flickr Public';
</code>

== Database Tables==

=== repository ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''type'''
|varchar(255)
|
|The type of the repository

|-
|'''visible'''
|tinyint(1)
|
|

|-
|sortorder
|int(10)
|
|
|}

=== repository_instances ===

This table contains one entry for every configured external repository instance.

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|name
|varchar 255
|
|A custom name for this repository (non-unique)

|-
|'''typeid'''
|int(10)
|
|The id of repository type

|-
|'''userid'''
|int(10)
|
|The person who created this repository instance

|-
|'''contextid'''
|int(10)
|
|The context that this repository is available to ( = system context for site-wide ones)

|-
|username
|varchar(255)
|
|username to log in with, if required (almost never!)

|-
|password
|varchar(255)
|
|password to log in with, if required (almost never!)

|-
|timecreated
|int(10)
|
|The time this repository was created

|-
|timemodified
|int(10)
|
|The last time the repository was modified
|}

=== repository_instance_config ===

{| border="1" cellpadding="2" cellspacing="0"
|'''Field'''
|'''Type'''
|'''Default'''
|'''Info'''

|-
|'''id'''
|int(10)
|
|autoincrementing

|-
|'''instanceid'''
|int(int)
|
|

|-
|'''name'''
|varchar(255)
|
|

|-
|value
|Text
|
|
|}

== Standard repository plugins ==
This is the functional specification list of the officially supported repository plugins.
For each plugins, the two mains part we are interested in are:
* How do I administer the plugin? See [[Repository_Administration_Specification| Repository Administration Specification - UC001-3]]
* How do I set up an account for this repository? See [[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]

==See also==

*[[Repository API| Repository API]]
*[[Repository_Interface_for_Moodle/Course/User| Repository Interface for Moodle/Course/User]]
*[[QA:Use Case Number Attribution| Use Case Number Attribution]]
* MDL-16543 - A list of officially supported repository plugins
* MDL-16543 - Template plugin for developers
[[Category:Repositories]]
[[Category:Plugins]]

File API

2012-01-16T05:48:08Z

Dongsheng: /* See also */

{{Moodle 2.0}}
==Overview==

The File API is for managing all the files stored by Moodle. If you are interested in how the file API works internally, see [[File API internals]]. The page is just about what you need to know to use the file API. Related is the [[Repository API]], which lets users get files into Moodle.

If you are looking for an explanation on how to manage moodle files in moodle forms, you most likely need to read [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]].

==File areas==

Files are conceptually stored in '''file areas'''. A file area is uniquely identified by:
* A context id.
* full component name (using [[Frankenstyle]]), for example 'course', 'mod_forum', 'mod_glossary', 'block_html'.
* A file area type, for example 'intro' or 'post'.
* A unique itemid. Normally, the itemid relates to something depending on the file area type. For example, for a 'course', 'intro' file area, the itemid is 0. For forum post, it is the post id.

File areas are not listed separately anywhere, they are stored implicitly in the files table. Please note that each subsystem is allowed to access only own file areas, for example only code in /mod/assignment/* may access files with component 'mod_assignment'.

===Naming file areas===

The names of the file areas are not strictly defined, but it is strongly recommended to use singulars and common names of areas if possible (intro, post, attachment, description, ...).

==Serving files to users==

You must refer to the file with a URL that includes a file-serving script, often pluginfile.php. For example

The general form of the URL is something like
<code php>
$url = $CFG->wwwroot/pluginfile.php/$contextid/$component/$filearea/arbitrary/extra/infomation.ext
</code>

A specific example might be
<code php>
$url = $CFG->wwwroot/pluginfile.php/$forumcontextid/mod_forum/post/$postid/image.jpg
</code>

The file serving script then looks at the context id, and component name, and the file area name, and based on that arranges for the file to be served, following appropriate security checks.

In most cases this is done by calling a callback function in the appropriate plugin, these functions are stored in lib.php files and named component_name_pluginfile() . The arbitrary/extra/infomation.ext is passed to the callback. For example, files in the mod_forum+post file area end up being served by the mod_forum_pluginfile function in mod/forum/lib.php.

You normally use an API function to generate these URL automatically, most often the <tt>file_rewrite_pluginfile_urls</tt> function.

==Getting files from the user==

* See [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]]

==Examples==
Please note that in reality developers outside of core will not deal with file api directly in majority of cases, instead use formslib elements which are doing all this automatically.

===Browsing files===
<code php>
$browser = get_file_browser();
$context = get_system_context();

$filearea = null;
$itemid = null;
$filename = null;
if ($fileinfo = $browser->get_file_info($context, $component, $filearea, $itemid, '/', $filename)) {
// build a Breadcrumb trail
$level = $fileinfo->get_parent();
while ($level) {
$path[] = array('name'=>$level->get_visible_name());
$level = $level->get_parent();
}
$path = array_reverse($path);
$children = $fileinfo->get_children();
foreach ($children as $child) {
if ($child->is_directory()) {
echo $child->get_visible_name();
// display contextid, itemid, component, filepath and filename
var_dump($child->get_params());
}
}
}

</code>

===Moving files around===

For example, if you have just built a file at the path
<code php>
$from_zip_file = $CFG->dataroot . '/temp/backup/' . $preferences->backup_unique_code .
'/' . $preferences->backup_name;
</code>
And you want to move it into the course_backup file area, do
<code php>
$context = get_context_instance(CONTEXT_COURSE, $preferences->backup_course);
$fs = get_file_storage();
$file_record = array('contextid'=>$context->id, 'component'=>'course', 'filearea'=>'backup',
'itemid'=>0, 'filepath'=>'/', 'filename'=>$preferences->backup_name,
'timecreated'=>time(), 'timemodified'=>time());
$fs->create_file_from_pathname($file_record, $from_zip_file);
</code>

=== List area files ===
<code php>
$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);
foreach ($files as $f) {
// $f is an instance of stored_file
echo $f->get_filename();
}
</code>

Or as links...

<code php>
$out = array();

$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);

foreach ($files as $file) {
$url = "{$CFG->wwwroot}/pluginfile.php/{$file->get_contextid()}/mod_assignment/submission}";
$filename = $file->get_filename();
$fileurl = $url.$file->get_filepath().$file->get_itemid().'/'.$filename;
$out[] = html_writer::link($fileurl, $filename);
}

$br = html_writer::empty_tag('br');

return implode($br, $out);
</code>

=== Create file ===

Here's how to create a file whose contents will be a text string. This is the equivalent of the PHP function <tt>file_put_contents</tt>.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'contextid' => $context->id, // ID of context
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Create file containing text 'hello world'
$fs->create_file_from_string($fileinfo, 'hello world');
</code>

If you want to create a file in the Moodle file area based on a 'real' file e.g. in a temporary folder, you can use <tt>create_file_from_pathname</tt> instead.

Unlike with ordinary files, this method will not automatically overwrite an existing file. If you wish to overwrite a file, you must first get the file and (if it exists) delete it, and only then create it again.

=== Read file ===

This is a way to read a file, equivalent to <tt>file_get_contents</tt>. '''Please note your are allowed to do this ONLY from mod/mymodule/* code, it is not acceptable to do this anywhere else.''' Other code has to use file_browser interface instead.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Read contents
if ($file) {
$contents = $file->get_content();
} else {
// file doesn't exist - do something
}
</code>

If you want to access the file directly on disk, this is not permitted. Instead, you need to make a copy of the file in a temporary area and use that. You can do this with <tt>$file->copy_content_to($pathname)</tt>.

=== Delete file ===

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule',
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Delete it if it exists
if ($file) {
$file->delete();
}
</code>

==See also==

* [[File API internals]] how the File API works internally.
* [[Using the File API in Moodle forms]]

[[Category:Files]]

Using the File API in Moodle forms

2012-01-16T05:31:25Z

Dongsheng:

{{Moodle 2.0}}

This document shows you exactly how to use Moodle forms to get files from users in a standard and secure way.

==Overview==

In Moodle 2.0 all files are stored in a central database accessible via the [[File API|File API]], and every file is associated with a component and a "file area" in Moodle, such as a particular module.

A common use case is to provide a form (using Moodle's [[lib/formslib.php|Forms API]]) which allows users to upload or import files as attachments or media embedded into HTML.

Normally this works like this:
# User starts creation or re-edits an existing item in Moodle (eg forum post, resource, glossary entry etc)
# User presses some sort of button to browse for new files to attach or embed
# User sees our "Choose file..." dialog, which contains one or more repository instances.
# User chooses a file, the [[Repository API|Repository API]] takes care of copying the file into a "draft file area" within Moodle
# File appears in the text or as an attachment in the form.
# When the user hits save, the [[File API|File API]] is invoked to move the file from the draft file area into a permanent file area associated with that data

This document shows you exactly how to use Moodle forms to interact with users in a standard and secure way.

If you just want to write code to manipulate Moodle files internally (without user input) then see [[File API]].

==Form elements==

In Moodle 2.0 there are three file-related form elements for interacting with users:

# filemanager - the way to attach one or more files as a set
# editor - the way to specify a textarea with a HTML editor, and all the handling of images and movies within that HTML
# filepicker - a way to specify one file for the case when you want to process the file and throw it away

In Moodle 1.9 there were two other types which are now '''deprecated''' (they work, but please do not use these anymore)
# file - used to just allow a normal file upload from the desktop only.
# htmleditor - this old method of embedding a HTML editor in a textarea is not able to support repositories etc.

===filepicker===

File picker (''filepicker'') is a direct replacement of the older ''file'' formslib element.

It is intended for situations when you want the user to upload '''one''' file so you can process it and delete it, such as when you are importing data from a CSV file.

==== Using the filepicker element ====

<code php>
$mform->addElement('filepicker', 'userfile', get_string('file'), null, array('maxbytes' => $maxbytes, 'accepted_types' => '*'));
</code>

==== Obtain the chosen file ====

The API for getting file contents is exactly the same as for ''file'' element.

<code php>
$content = $mform->get_file_content('userfile');
</code>

=== filemanager ===

The File Manager element improves on file picker by allowing you to manage more than one file. It is expected that the files will be stored permanently for future use (such as forum and glossary attachments).

==== Add file manager element ====

Example:
<code php>
$mform->addElement('filemanager', 'attachments', get_string('attachment', 'moodle'), null,
array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50, 'accepted_types' => array('document') ));
</code>

Here are the fields for filemanager:

;'filemanager':This is a filemanager element :)
;elementname:The unique name of the element in the form
;elementlabel:The label string that users see
;attributes:(leave it as null)
;options: an array of further options for the filepicker (see below)

The options array can contain:

;subdirs:(Default 1) Are subdirectories allowed? (true or false)
;maxbytes:(Default 0) Restricts the total size of all the files.
;maxfiles:(Default -1) Restricts the total number of files.
;accepted_types:(Default *) You can specify what file types are accepted by filemanager. All current file types are listed in this file: [http://cvs.moodle.org/moodle/lib/filestorage/file_types.mm moodle/lib/filestorage/file_types.mm]. This is a [http://freemind.sourceforge.net/wiki/index.php/Main_Page freemind] file: if it is edited the changes will be immediately reflected in Moodle. Example usage: '''array('audio', 'video', 'documents')''', you can include file extensions as well, for example: '''array('*.txt', '*.jpg', 'audio')'''.

==== Load existing files into draft area ====

<code php>
if (empty($entry->id)) {
$entry = new stdClass;
$entry->id = null;
}

$draftitemid = file_get_submitted_draft_itemid('attachments');
file_prepare_draft_area($draftitemid, $context->id, 'mod_glossary', 'attachment', $entry->id, array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50));
$entry->attachments = $draftitemid;

$mform->set_data($entry);

</code>

==== Store updated set of files ====

<code php>
if ($data = $mform->get_data()) {
// ... store or update $entry
file_save_draft_area_files($data->attachments, $context->id, 'mod_glossary', 'attachment', $entry->id, array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50));
}
</code>

===editor===
There are two way for using of editor element in code, the first one is easier but expects some standardized fields. The second method is more low level.

====Simple use====
# name database fields: ''textfield'', ''textfieldformat'' (and ''textfieldtrust'' if required)
# create options array <code php>$textfieldoptions = array('trusttext'=>true, 'subdirs'=>true, 'maxfiles'=>$maxfiles, 'maxbytes'=>$maxbytes);</code>
# add editor ''textfield_editor'' to moodle form, pass options through custom data in form constructor, set $data->id to null if data not exist yet <code php>$mform->addElement('editor', 'textfield_editor', get_string('fieldname', 'somemodule'), null, $textfieldoptions);</code>
# prepare data <code php>$data = file_prepare_standard_editor($data, 'textfield', $textfieldoptions, $context, 'mod_somemodule', 'somearea', $data->id);</code>
# get submitted data and after inserting/updating of data <code php>$data = file_postupdate_standard_editor($data, 'textfield', $textfieldoptions, $context, 'mod_somemodule', 'somearea', $data->id);</code>

Real world examples are in mod/glossary/edit.php and mod/glossary/comment.php

====Low level use====

When using editor element you need to preprocess and postprocess the data:
# detect if form was already submitted (usually means draft is area already exists) - ''file_get_submitted_draft_itemid()''
# prepare draft file area, temporary storage of all files attached to the text - ''file_prepare_draft_area()''
# convert encoded relative links to absolute links - ''file_prepare_draft_area()''
# create form and set current data
# after submission the changed files must be merged back into original area - ''file_save_draft_area_files()''
# absolute links have to be replaced by relative links - ''file_save_draft_area_files()''

=====Replace old htmleditor with editor=====

The file picker has been integrated with with TinyMCE to make the editor element. This new element should support all types on editors and should be able to switch them on-the-fly. Instances of the old htmleditor element in your forms should be replaced by the new editor element, this may need adding of new format and trusttext columns. For example:
<code php>
$mform->addElement('editor', 'entry', get_string('definition', 'glossary'), null,
array('maxfiles' => EDITOR_UNLIMITED_FILES));
</code>
The editor element can take following options: maxfiles, maxbytes, subdirs and changeformat. Please note that the embedded files is optional feature and is not expected be used everywhere.

'''Note''': the editor element now includes text format option. You should no longer use the separate format element type.

=====Prepare current data - text and files=====

<code php>
if (empty($entry->id)) {
$entry = new object();
$entry->id = null;
$entry->definition = '';
$entry->format = FORMAT_HTML;
}

$draftid_editor = file_get_submitted_draft_itemid('entry');
$currenttext = file_prepare_draft_area($draftid_editor, $context->id, 'mod_glossary', 'entry', $entry->id, array('subdirs'=>true), $entry->definition);
$entry->entry = array('text'=>$currenttext, 'format'=>$entry->format, 'itemid'=>$draftid_editor);

$mform->set_data($entry);
</code>

If there are multiple files, they will share the same itemid.

=====Obtain text, format and save draft files=====

To retrieve editor content, you need to use following code:

<code php>
if ($fromform = $mform->get_data()) {
// content of editor
$messagetext = $fromform->entry['text'];
// format of content
$messageformat = $fromform->entry['format'];
}
</code>

When a user selects a file using the file picker, the file is initially stored in a draft file area, and a URL is inserted into the HTML in the editor that lets the person editing the content (but no one else) see the file.

When the user submits the form, we then need to save the draft files to the correct place in permanent storage. (Just like you have to call $DB->update_record('tablename', $data); to have the other parts of the form submission stored correctly.)

The save_files_from_draft_area function and replace absolute links with internal relative links do:
<code php>
$messagetext = file_save_draft_area_files($draftid_editor, $context->id, 'mod_glossary', 'entry', $entry->id, array('subdirs'=>true), $messagetext);
</code>
; $context->id, 'component', 'proper_file_area' and $entry->id : correspond to the contextid, filearea and itemid columns in the [[File_API#Table:_files|files table]].
; $messagetext : this is the message text. As the files are saved to the real file area, the URLs in this content are rewritten.

All URLs in content that point to files managed to the File API are converted to a form that starts '@@PLUGINFILE@@/' before the content is stored in the database. That is what we mean by rewriting.

== File serving==

=== Convert internal relative links to absolute links ===

Before text content is displayed to the user, any URLs in the '@@PLUGINFILE@@/' form in the content need to be rewritten to the real URL where the user can access the files.
<code php>
$messagetext = file_rewrite_pluginfile_urls($messagetext, 'pluginfile.php',
$context->id, 'mod_mymodule', 'proper_file_area', $itemid);
</code>
; $messagetext : is the content containing the @@PLUGINFILE@@ URLs from the database.
; 'pluginfile.php' : there are a number of different scripts that can serve files with different permissions checks. You need to specify which one to use.
; $context->id, 'mod_mymodule', 'proper_file_area', $itemid : uniquely identifies the file area, as before.

=== Implement file serving access control ===

Attachments and embedded images should have the same access control like the text itself, in majority of cases these files are served using pluginfile.php. Access control is defined in ''module/lib.php'' file in function ''module_pluginfile()''.

== File browsing support ==
Only owner of each file area is allowed to use low level File API function to access files, other parts of Moodle should use file browsing API.

Activities may specify browsing support in own module/lib.php file by implementing functions module_get_file_areas() and module_get_file_info().

== Upgrading your code ==
Here I will attempt to describe some simple steps you can take to upgrade your file-handling form elements from pre-2.0 code to 2.0. We will use the example of glossary, since it has been used above.

=== Preparing your options ===
Unless you are happy with the defaults, you will need to define an array of options for each file-handling form element. You could define it at different places, but it's best to put it in one place and make the array(s) available to other files if they need it. In the majority of cases, this will be in a file like edit.php

Previous code in mod/glossary/edit.php:
<code php>
$mform =& new mod_glossary_entry_form(null, compact('cm', 'glossary', 'hook', 'mode', 'e', 'context'));
</code>
New code:
<code php>
$maxbytes = $course->maxbytes; // Could also use $CFG->maxbytes if you are not coding within a course context
$definitionoptions = array('subdirs'=>false, 'maxfiles'=>99, 'maxbytes'=>$maxbytes, 'trusttext'=>true, 'context'=>$context);
$attachmentoptions = array('subdirs'=>false, 'maxfiles'=>99, 'maxbytes'=>$maxbytes);
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
Note that the data being passed to the form constructor have changed also, but this is not part of the file API changes, I just include them to avoid confusion.

These options are for the htmleditor (definition field) and the filemanager (attachment field). They are used by a file called edit_form.php.

=== Element preparation ===
Before we look at this, however, we need to "prepare" the elements so that they can correctly display existing embedded images and attached files when you are editing a record instead of just creating one. So, let's take the code we've got so far in edit.php and add to it:

Currently upgraded code in edit.php:
<code php>
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
New code with element preparation:
<code php>
$entry = file_prepare_standard_editor($entry, 'definition', $definitionoptions, $context, 'mod_glossary', 'entry', $entry->id);
$entry = file_prepare_standard_filemanager($entry, 'attachment', $attachmentoptions, $context, 'mod_glossary', 'attachment', $entry->id);
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
Things to note:
* $entry in this case is simply a stdClass object which may either represent a new glossary entry or an existing one.
* $entry->id must be the unique identifier for the current object. If we are creating a new entry, it will be null, but in all cases it must be defined.
* These two functions (file_prepare_standard_editor and file_prepare_standard_filemanager) are shortcuts functions that take care of some of the tedious setting up for you, but they make a couple of assumptions:
*# You '''must''' name the form element as {element}_editor or {element}_filemanager (see next section)
*# You '''must''' have at least the following fields in the database: {element} and {element}summary, as described earlier in this documentation

We can now look at the upgrades needed in the form definition file.

=== Form definition ===
Previous code in mod/glossary/edit_form.php:
<code php>
$mform->addElement('htmleditor', 'definition', get_string('definition', 'glossary'), array('rows'=>20));
$mform->setType('definition', PARAM_RAW);
$mform->addRule('definition', null, 'required', null, 'client');
$mform->setHelpButton('definition', array('writing', 'richtext'), false, 'editorhelpbutton');
$mform->addElement('format');
// a bit further...
$this->set_upload_manager(new upload_manager('attachment', true, false, $COURSE, false, 0, true, true, false));
$mform->addElement('file', 'attachment', get_string('attachment', 'forum'));
$mform->setHelpButton('attachment', array('attachment', get_string('attachment', 'glossary'), 'glossary'));
</code>
New code:
<code php>
$definitionoptions = $this->_customdata['definitionoptions'];
$attachmentoptions = $this->_customdata['attachmentoptions'];
// a bit further...
$mform->addElement('editor', 'definition_editor', get_string('definition', 'glossary'), null, $definitionoptions);
$mform->setType('definition_editor', PARAM_RAW);
$mform->addRule('definition_editor', get_string('required'), 'required', null, 'client');
// a bit further...
$mform->addElement('filemanager', 'attachment_filemanager', get_string('attachment', 'glossary'), null, $attachmentoptions);
$mform->setHelpButton('attachment_filemanager', array('attachment2', get_string('attachment', 'glossary'), 'glossary'));
</code>

Note the following:
* The format element and the help button are no longer required for the HTML editor element
* The name of the form element needs to be changed by adding '_editor' or '_manager' to the original name. This is a naming convention that is used by a couple of functions we will look at shortly
* Make sure $definitionoptions has context parameter, else system context is used and editor will not respect filter settings.

=== Handling submitted data ===
The final step is to handle the submitted data properly, i.e. retrieve the files and save them to disk, associating them with the record we have just created (a glossary entry in our example). This happens in edit.php:

Previous code in edit.php:
<code php>
// Section that updates an entry:
$todb->id = $e;
$dir = glossary_file_area_name($todb);
if ($mform->save_files($dir) and $newfilename = $mform->get_new_filename()) {
$todb->attachment = $newfilename;
}

// Section that adds an entry:
if ($todb->id = insert_record("glossary_entries", $todb)) {
$e = $todb->id;
$dir = glossary_file_area_name($todb);
if ($mform->save_files($dir) and $newfilename = $mform->get_new_filename()) {
set_field("glossary_entries", "attachment", $newfilename, "id", $todb->id);
}
}
</code>
New code:
<code php>
// $todb was renamed to $entry, and the code was refactored
// so that the file-handling code is only used once for either an add or an update action.
// If an entry is being added, $DB->insert() has already been called, so we have a valid $entry->id
$entry = file_postupdate_standard_editor($entry, 'definition', $definitionoptions, $context, 'mod_glossary', 'entry', $entry->id);
$entry = file_postupdate_standard_filemanager($entry, 'attachment', $attachmentoptions, $context, 'mod_glossary', 'attachment', $entry->id);
// store the updated value values
$DB->update_record('glossary_entries', $entry);
</code>
Things to note:
* If you are adding a new record, you will still need to call update_record after calling the file_postupdate* functions

=== Gotchas ===
A few things to keep in mind:
* Make sure that you instantiate the moodle form before any call to $OUTPUT->header()

== See also ==

* [[File API]]
* [[Using the file API]]
* [[Repository API]]
* [[Portfolio API]]
* MDL-14589 - File API Meta issue

[[Category:Files]]
[[Category:Repositories]]

File API

2012-01-16T04:00:57Z

Dongsheng:

{{Moodle 2.0}}
==Overview==

The File API is for managing all the files stored by Moodle. If you are interested in how the file API works internally, see [[File API internals]]. The page is just about what you need to know to use the file API. Related is the [[Repository API]], which lets users get files into Moodle.

If you are looking for an explanation on how to manage moodle files in moodle forms, you most likely need to read [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]].

==File areas==

Files are conceptually stored in '''file areas'''. A file area is uniquely identified by:
* A context id.
* full component name (using [[Frankenstyle]]), for example 'course', 'mod_forum', 'mod_glossary', 'block_html'.
* A file area type, for example 'intro' or 'post'.
* A unique itemid. Normally, the itemid relates to something depending on the file area type. For example, for a 'course', 'intro' file area, the itemid is 0. For forum post, it is the post id.

File areas are not listed separately anywhere, they are stored implicitly in the files table. Please note that each subsystem is allowed to access only own file areas, for example only code in /mod/assignment/* may access files with component 'mod_assignment'.

===Naming file areas===

The names of the file areas are not strictly defined, but it is strongly recommended to use singulars and common names of areas if possible (intro, post, attachment, description, ...).

==Serving files to users==

You must refer to the file with a URL that includes a file-serving script, often pluginfile.php. For example

The general form of the URL is something like
<code php>
$url = $CFG->wwwroot/pluginfile.php/$contextid/$component/$filearea/arbitrary/extra/infomation.ext
</code>

A specific example might be
<code php>
$url = $CFG->wwwroot/pluginfile.php/$forumcontextid/mod_forum/post/$postid/image.jpg
</code>

The file serving script then looks at the context id, and component name, and the file area name, and based on that arranges for the file to be served, following appropriate security checks.

In most cases this is done by calling a callback function in the appropriate plugin, these functions are stored in lib.php files and named component_name_pluginfile() . The arbitrary/extra/infomation.ext is passed to the callback. For example, files in the mod_forum+post file area end up being served by the mod_forum_pluginfile function in mod/forum/lib.php.

You normally use an API function to generate these URL automatically, most often the <tt>file_rewrite_pluginfile_urls</tt> function.

==Getting files from the user==

* See [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]]

==Examples==
Please note that in reality developers outside of core will not deal with file api directly in majority of cases, instead use formslib elements which are doing all this automatically.

===Browsing files===
<code php>
$browser = get_file_browser();
$context = get_system_context();

$filearea = null;
$itemid = null;
$filename = null;
if ($fileinfo = $browser->get_file_info($context, $component, $filearea, $itemid, '/', $filename)) {
// build a Breadcrumb trail
$level = $fileinfo->get_parent();
while ($level) {
$path[] = array('name'=>$level->get_visible_name());
$level = $level->get_parent();
}
$path = array_reverse($path);
$children = $fileinfo->get_children();
foreach ($children as $child) {
if ($child->is_directory()) {
echo $child->get_visible_name();
// display contextid, itemid, component, filepath and filename
var_dump($child->get_params());
}
}
}

</code>

===Moving files around===

For example, if you have just built a file at the path
<code php>
$from_zip_file = $CFG->dataroot . '/temp/backup/' . $preferences->backup_unique_code .
'/' . $preferences->backup_name;
</code>
And you want to move it into the course_backup file area, do
<code php>
$context = get_context_instance(CONTEXT_COURSE, $preferences->backup_course);
$fs = get_file_storage();
$file_record = array('contextid'=>$context->id, 'component'=>'course', 'filearea'=>'backup',
'itemid'=>0, 'filepath'=>'/', 'filename'=>$preferences->backup_name,
'timecreated'=>time(), 'timemodified'=>time());
$fs->create_file_from_pathname($file_record, $from_zip_file);
</code>

=== List area files ===
<code php>
$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);
foreach ($files as $f) {
// $f is an instance of stored_file
echo $f->get_filename();
}
</code>

Or as links...

<code php>
$out = array();

$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);

foreach ($files as $file) {
$url = "{$CFG->wwwroot}/pluginfile.php/{$file->get_contextid()}/mod_assignment/submission}";
$filename = $file->get_filename();
$fileurl = $url.$file->get_filepath().$file->get_itemid().'/'.$filename;
$out[] = html_writer::link($fileurl, $filename);
}

$br = html_writer::empty_tag('br');

return implode($br, $out);
</code>

=== Create file ===

Here's how to create a file whose contents will be a text string. This is the equivalent of the PHP function <tt>file_put_contents</tt>.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'contextid' => $context->id, // ID of context
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Create file containing text 'hello world'
$fs->create_file_from_string($fileinfo, 'hello world');
</code>

If you want to create a file in the Moodle file area based on a 'real' file e.g. in a temporary folder, you can use <tt>create_file_from_pathname</tt> instead.

Unlike with ordinary files, this method will not automatically overwrite an existing file. If you wish to overwrite a file, you must first get the file and (if it exists) delete it, and only then create it again.

=== Read file ===

This is a way to read a file, equivalent to <tt>file_get_contents</tt>. '''Please note your are allowed to do this ONLY from mod/mymodule/* code, it is not acceptable to do this anywhere else.''' Other code has to use file_browser interface instead.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Read contents
if ($file) {
$contents = $file->get_content();
} else {
// file doesn't exist - do something
}
</code>

If you want to access the file directly on disk, this is not permitted. Instead, you need to make a copy of the file in a temporary area and use that. You can do this with <tt>$file->copy_content_to($pathname)</tt>.

=== Delete file ===

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule',
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Delete it if it exists
if ($file) {
$file->delete();
}
</code>

==See also==

* [[File API internals]] how the File API works internally.
* [[Using_the_File_API_in_Moodle_forms]]

[[Category:Files]]

Using the File API in Moodle forms

2012-01-16T03:20:10Z

Dongsheng:

{{Moodle 2.0}}

This document shows you exactly how to use Moodle forms to get files from users in a standard and secure way.

==Overview==

In Moodle 2.0 all files are stored in a central database accessible via the [[File API|File API]], and every file is associated with a component and a "file area" in Moodle, such as a particular module.

A common use case is to provide a form (using Moodle's [[lib/formslib.php|Forms API]]) which allows users to upload or import files as attachments or media embedded into HTML.

Normally this works like this:
# User starts creation or re-edits an existing item in Moodle (eg forum post, resource, glossary entry etc)
# User presses some sort of button to browse for new files to attach or embed
# User sees our "Choose file..." dialog, which contains one or more repository instances.
# User chooses a file, the [[Repository API|Repository API]] takes care of copying the file into a "draft file area" within Moodle
# File appears in the text or as an attachment in the form.
# When the user hits save, the [[File API|File API]] is invoked to move the file from the draft file area into a permanent file area associated with that data

This document shows you exactly how to use Moodle forms to interact with users in a standard and secure way.

If you just want to write code to manipulate Moodle files internally (without user input) then see [[Using_the_File_API]].

==Form elements==

In Moodle 2.0 there are three file-related form elements for interacting with users:

# filemanager - the way to attach one or more files as a set
# editor - the way to specify a textarea with a HTML editor, and all the handling of images and movies within that HTML
# filepicker - a way to specify one file for the case when you want to process the file and throw it away

In Moodle 1.9 there were two other types which are now '''deprecated''' (they work, but please do not use these anymore)
# file - used to just allow a normal file upload from the desktop only.
# htmleditor - this old method of embedding a HTML editor in a textarea is not able to support repositories etc.

===filepicker===

File picker (''filepicker'') is a direct replacement of the older ''file'' formslib element.

It is intended for situations when you want the user to upload '''one''' file so you can process it and delete it, such as when you are importing data from a CSV file.

==== Using the filepicker element ====

<code php>
$mform->addElement('filepicker', 'userfile', get_string('file'), null, array('maxbytes' => $maxbytes, 'accepted_types' => '*'));
</code>

==== Obtain the chosen file ====

The API for getting file contents is exactly the same as for ''file'' element.

<code php>
$content = $mform->get_file_content('userfile');
</code>

=== filemanager ===

The File Manager element improves on file picker by allowing you to manage more than one file. It is expected that the files will be stored permanently for future use (such as forum and glossary attachments).

==== Add file manager element ====

Example:
<code php>
$mform->addElement('filemanager', 'attachments', get_string('attachment', 'moodle'), null,
array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50, 'accepted_types' => array('document') ));
</code>

Here are the fields for filemanager:

;'filemanager':This is a filemanager element :)
;elementname:The unique name of the element in the form
;elementlabel:The label string that users see
;attributes:(leave it as null)
;options: an array of further options for the filepicker (see below)

The options array can contain:

;subdirs:(Default 1) Are subdirectories allowed? (true or false)
;maxbytes:(Default 0) Restricts the total size of all the files.
;maxfiles:(Default -1) Restricts the total number of files.
;accepted_types:(Default *) You can specify what file types are accepted by filemanager. All current file types are listed in this file: [http://cvs.moodle.org/moodle/lib/filestorage/file_types.mm moodle/lib/filestorage/file_types.mm]. This is a [http://freemind.sourceforge.net/wiki/index.php/Main_Page freemind] file: if it is edited the changes will be immediately reflected in Moodle. Example usage: '''array('audio', 'video', 'documents')''', you can include file extensions as well, for example: '''array('*.txt', '*.jpg', 'audio')'''.

==== Load existing files into draft area ====

<code php>
if (empty($entry->id)) {
$entry = new stdClass;
$entry->id = null;
}

$draftitemid = file_get_submitted_draft_itemid('attachments');
file_prepare_draft_area($draftitemid, $context->id, 'mod_glossary', 'attachment', $entry->id, array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50));
$entry->attachments = $draftitemid;

$mform->set_data($entry);

</code>

==== Store updated set of files ====

<code php>
if ($data = $mform->get_data()) {
// ... store or update $entry
file_save_draft_area_files($data->attachments, $context->id, 'mod_glossary', 'attachment', $entry->id, array('subdirs' => 0, 'maxbytes' => $maxbytes, 'maxfiles' => 50));
}
</code>

===editor===
There are two way for using of editor element in code, the first one is easier but expects some standardized fields. The second method is more low level.

====Simple use====
# name database fields: ''textfield'', ''textfieldformat'' (and ''textfieldtrust'' if required)
# create options array <code php>$textfieldoptions = array('trusttext'=>true, 'subdirs'=>true, 'maxfiles'=>$maxfiles, 'maxbytes'=>$maxbytes);</code>
# add editor ''textfield_editor'' to moodle form, pass options through custom data in form constructor, set $data->id to null if data not exist yet <code php>$mform->addElement('editor', 'textfield_editor', get_string('fieldname', 'somemodule'), null, $textfieldoptions);</code>
# prepare data <code php>$data = file_prepare_standard_editor($data, 'textfield', $textfieldoptions, $context, 'mod_somemodule', 'somearea', $data->id);</code>
# get submitted data and after inserting/updating of data <code php>$data = file_postupdate_standard_editor($data, 'textfield', $textfieldoptions, $context, 'mod_somemodule', 'somearea', $data->id);</code>

Real world examples are in mod/glossary/edit.php and mod/glossary/comment.php

====Low level use====

When using editor element you need to preprocess and postprocess the data:
# detect if form was already submitted (usually means draft is area already exists) - ''file_get_submitted_draft_itemid()''
# prepare draft file area, temporary storage of all files attached to the text - ''file_prepare_draft_area()''
# convert encoded relative links to absolute links - ''file_prepare_draft_area()''
# create form and set current data
# after submission the changed files must be merged back into original area - ''file_save_draft_area_files()''
# absolute links have to be replaced by relative links - ''file_save_draft_area_files()''

=====Replace old htmleditor with editor=====

The file picker has been integrated with with TinyMCE to make the editor element. This new element should support all types on editors and should be able to switch them on-the-fly. Instances of the old htmleditor element in your forms should be replaced by the new editor element, this may need adding of new format and trusttext columns. For example:
<code php>
$mform->addElement('editor', 'entry', get_string('definition', 'glossary'), null,
array('maxfiles' => EDITOR_UNLIMITED_FILES));
</code>
The editor element can take following options: maxfiles, maxbytes, subdirs and changeformat. Please note that the embedded files is optional feature and is not expected be used everywhere.

'''Note''': the editor element now includes text format option. You should no longer use the separate format element type.

=====Prepare current data - text and files=====

<code php>
if (empty($entry->id)) {
$entry = new object();
$entry->id = null;
$entry->definition = '';
$entry->format = FORMAT_HTML;
}

$draftid_editor = file_get_submitted_draft_itemid('entry');
$currenttext = file_prepare_draft_area($draftid_editor, $context->id, 'mod_glossary', 'entry', $entry->id, array('subdirs'=>true), $entry->definition);
$entry->entry = array('text'=>$currenttext, 'format'=>$entry->format, 'itemid'=>$draftid_editor);

$mform->set_data($entry);
</code>

If there are multiple files, they will share the same itemid.

=====Obtain text, format and save draft files=====

To retrieve editor content, you need to use following code:

<code php>
if ($fromform = $mform->get_data()) {
// content of editor
$messagetext = $fromform->entry['text'];
// format of content
$messageformat = $fromform->entry['format'];
}
</code>

When a user selects a file using the file picker, the file is initially stored in a draft file area, and a URL is inserted into the HTML in the editor that lets the person editing the content (but no one else) see the file.

When the user submits the form, we then need to save the draft files to the correct place in permanent storage. (Just like you have to call $DB->update_record('tablename', $data); to have the other parts of the form submission stored correctly.)

The save_files_from_draft_area function and replace absolute links with internal relative links do:
<code php>
$messagetext = file_save_draft_area_files($draftid_editor, $context->id, 'mod_glossary', 'entry', $entry->id, array('subdirs'=>true), $messagetext);
</code>
; $context->id, 'component', 'proper_file_area' and $entry->id : correspond to the contextid, filearea and itemid columns in the [[File_API#Table:_files|files table]].
; $messagetext : this is the message text. As the files are saved to the real file area, the URLs in this content are rewritten.

All URLs in content that point to files managed to the File API are converted to a form that starts '@@PLUGINFILE@@/' before the content is stored in the database. That is what we mean by rewriting.

== File serving==

=== Convert internal relative links to absolute links ===

Before text content is displayed to the user, any URLs in the '@@PLUGINFILE@@/' form in the content need to be rewritten to the real URL where the user can access the files.
<code php>
$messagetext = file_rewrite_pluginfile_urls($messagetext, 'pluginfile.php',
$context->id, 'mod_mymodule', 'proper_file_area', $itemid);
</code>
; $messagetext : is the content containing the @@PLUGINFILE@@ URLs from the database.
; 'pluginfile.php' : there are a number of different scripts that can serve files with different permissions checks. You need to specify which one to use.
; $context->id, 'mod_mymodule', 'proper_file_area', $itemid : uniquely identifies the file area, as before.

=== Implement file serving access control ===

Attachments and embedded images should have the same access control like the text itself, in majority of cases these files are served using pluginfile.php. Access control is defined in ''module/lib.php'' file in function ''module_pluginfile()''.

== File browsing support ==
Only owner of each file area is allowed to use low level File API function to access files, other parts of Moodle should use file browsing API.

Activities may specify browsing support in own module/lib.php file by implementing functions module_get_file_areas() and module_get_file_info().

== Upgrading your code ==
Here I will attempt to describe some simple steps you can take to upgrade your file-handling form elements from pre-2.0 code to 2.0. We will use the example of glossary, since it has been used above.

=== Preparing your options ===
Unless you are happy with the defaults, you will need to define an array of options for each file-handling form element. You could define it at different places, but it's best to put it in one place and make the array(s) available to other files if they need it. In the majority of cases, this will be in a file like edit.php

Previous code in mod/glossary/edit.php:
<code php>
$mform =& new mod_glossary_entry_form(null, compact('cm', 'glossary', 'hook', 'mode', 'e', 'context'));
</code>
New code:
<code php>
$maxbytes = $course->maxbytes; // Could also use $CFG->maxbytes if you are not coding within a course context
$definitionoptions = array('subdirs'=>false, 'maxfiles'=>99, 'maxbytes'=>$maxbytes, 'trusttext'=>true, 'context'=>$context);
$attachmentoptions = array('subdirs'=>false, 'maxfiles'=>99, 'maxbytes'=>$maxbytes);
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
Note that the data being passed to the form constructor have changed also, but this is not part of the file API changes, I just include them to avoid confusion.

These options are for the htmleditor (definition field) and the filemanager (attachment field). They are used by a file called edit_form.php.

=== Element preparation ===
Before we look at this, however, we need to "prepare" the elements so that they can correctly display existing embedded images and attached files when you are editing a record instead of just creating one. So, let's take the code we've got so far in edit.php and add to it:

Currently upgraded code in edit.php:
<code php>
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
New code with element preparation:
<code php>
$entry = file_prepare_standard_editor($entry, 'definition', $definitionoptions, $context, 'mod_glossary', 'entry', $entry->id);
$entry = file_prepare_standard_filemanager($entry, 'attachment', $attachmentoptions, $context, 'mod_glossary', 'attachment', $entry->id);
$mform = new mod_glossary_entry_form(null, array(
'current'=>$entry,
'cm'=>$cm,
'glossary'=>$glossary,
'definitionoptions'=>$definitionoptions,
'attachmentoptions'=>$attachmentoptions));
</code>
Things to note:
* $entry in this case is simply a stdClass object which may either represent a new glossary entry or an existing one.
* $entry->id must be the unique identifier for the current object. If we are creating a new entry, it will be null, but in all cases it must be defined.
* These two functions (file_prepare_standard_editor and file_prepare_standard_filemanager) are shortcuts functions that take care of some of the tedious setting up for you, but they make a couple of assumptions:
*# You '''must''' name the form element as {element}_editor or {element}_filemanager (see next section)
*# You '''must''' have at least the following fields in the database: {element} and {element}summary, as described earlier in this documentation

We can now look at the upgrades needed in the form definition file.

=== Form definition ===
Previous code in mod/glossary/edit_form.php:
<code php>
$mform->addElement('htmleditor', 'definition', get_string('definition', 'glossary'), array('rows'=>20));
$mform->setType('definition', PARAM_RAW);
$mform->addRule('definition', null, 'required', null, 'client');
$mform->setHelpButton('definition', array('writing', 'richtext'), false, 'editorhelpbutton');
$mform->addElement('format');
// a bit further...
$this->set_upload_manager(new upload_manager('attachment', true, false, $COURSE, false, 0, true, true, false));
$mform->addElement('file', 'attachment', get_string('attachment', 'forum'));
$mform->setHelpButton('attachment', array('attachment', get_string('attachment', 'glossary'), 'glossary'));
</code>
New code:
<code php>
$definitionoptions = $this->_customdata['definitionoptions'];
$attachmentoptions = $this->_customdata['attachmentoptions'];
// a bit further...
$mform->addElement('editor', 'definition_editor', get_string('definition', 'glossary'), null, $definitionoptions);
$mform->setType('definition_editor', PARAM_RAW);
$mform->addRule('definition_editor', get_string('required'), 'required', null, 'client');
// a bit further...
$mform->addElement('filemanager', 'attachment_filemanager', get_string('attachment', 'glossary'), null, $attachmentoptions);
$mform->setHelpButton('attachment_filemanager', array('attachment2', get_string('attachment', 'glossary'), 'glossary'));
</code>

Note the following:
* The format element and the help button are no longer required for the HTML editor element
* The name of the form element needs to be changed by adding '_editor' or '_manager' to the original name. This is a naming convention that is used by a couple of functions we will look at shortly
* Make sure $definitionoptions has context parameter, else system context is used and editor will not respect filter settings.

=== Handling submitted data ===
The final step is to handle the submitted data properly, i.e. retrieve the files and save them to disk, associating them with the record we have just created (a glossary entry in our example). This happens in edit.php:

Previous code in edit.php:
<code php>
// Section that updates an entry:
$todb->id = $e;
$dir = glossary_file_area_name($todb);
if ($mform->save_files($dir) and $newfilename = $mform->get_new_filename()) {
$todb->attachment = $newfilename;
}

// Section that adds an entry:
if ($todb->id = insert_record("glossary_entries", $todb)) {
$e = $todb->id;
$dir = glossary_file_area_name($todb);
if ($mform->save_files($dir) and $newfilename = $mform->get_new_filename()) {
set_field("glossary_entries", "attachment", $newfilename, "id", $todb->id);
}
}
</code>
New code:
<code php>
// $todb was renamed to $entry, and the code was refactored
// so that the file-handling code is only used once for either an add or an update action.
// If an entry is being added, $DB->insert() has already been called, so we have a valid $entry->id
$entry = file_postupdate_standard_editor($entry, 'definition', $definitionoptions, $context, 'mod_glossary', 'entry', $entry->id);
$entry = file_postupdate_standard_filemanager($entry, 'attachment', $attachmentoptions, $context, 'mod_glossary', 'attachment', $entry->id);
// store the updated value values
$DB->update_record('glossary_entries', $entry);
</code>
Things to note:
* If you are adding a new record, you will still need to call update_record after calling the file_postupdate* functions

=== Gotchas ===
A few things to keep in mind:
* Make sure that you instantiate the moodle form before any call to $OUTPUT->header()

== See also ==

* [[File API]]
* [[Using the file API]]
* [[Repository API]]
* [[Portfolio API]]
* MDL-14589 - File API Meta issue

[[Category:Files]]
[[Category:Repositories]]

File API

2012-01-16T03:16:23Z

Dongsheng:

{{Moodle 2.0}}
==Overview==

The File API is for managing all the files stored by Moodle. If you are interested in how the file API works internally, see [[File API]]. The page is just about what you need to know to use the file API. Related is the [[Repository API]], which lets users get files into Moodle.

If you are looking for an explanation on how to manage moodle files in moodle forms, you most likely need to read [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]].

==File areas==

Files are conceptually stored in '''file areas'''. A file area is uniquely identified by:
* A context id.
* full component name, for example 'course', 'mod_forum', 'mod_glossary', 'block_html'.
* A file area type, for example 'intro' or 'post'.
* A unique itemid. Normally, the itemid relates to something depending on the file area type. For example, for a 'course', 'intro' file area, the itemid is 0. For forum post, it is the post id.

File areas are not listed separately anywhere, they are stored implicitly in the files table. Please note that each subsystem is allowed to access only own file areas, for example only code in /mod/assignment/* may access files with component 'mod_assignment'.

===Naming file areas===

The names of the file areas are not strictly defined, but it is strongly recommended to use singulars and common names of areas if possible (intro, post, attachment, description, ...).

==Serving files to users==

You must refer to the file with a URL that includes a file-serving script, often pluginfile.php. For example

The general form of the URL is something like
<code php>
$url = $CFG->wwwroot/pluginfile.php/$contextid/$component/$filearea/arbitrary/extra/infomation.ext
</code>

A specific example might be
<code php>
$url = $CFG->wwwroot/pluginfile.php/$forumcontextid/mod_forum/post/$postid/image.jpg
</code>

The file serving script then looks at the context id, and component name, and the file area name, and based on that arranges for the file to be served, following appropriate security checks.

In most cases this is done by calling a callback function in the appropriate plugin, these functions are stored in lib.php files and named component_name_pluginfile() . The arbitrary/extra/infomation.ext is passed to the callback. For example, files in the mod_forum+post file area end up being served by the mod_forum_pluginfile function in mod/forum/lib.php.

You normally use an API function to generate these URL automatically, most often the file_rewrite_pluginfile_urls function.

==Getting files from the user==

* See [[Using_the_File_API_in_Moodle_forms|Using the File API in Moodle forms]]
==Examples==
Please note that in reality developers outside of core will not deal with file api directly in majority of cases, instead use formslib elements which are doing all this automatically.

===Browsing files===
<code php>
$browser = get_file_browser();
$context = get_system_context();

$filearea = null;
$itemid = null;
$filename = null;
if ($fileinfo = $browser->get_file_info($context, $component, $filearea, $itemid, '/', $filename)) {
// build a Breadcrumb trail
$level = $fileinfo->get_parent();
while ($level) {
$params = base64_encode(serialize($level->get_params()));
$path[] = array('name'=>$level->get_visible_name(), 'path'=>$params);
$level = $level->get_parent();
}
$path = array_reverse($path);
$children = $fileinfo->get_children();
foreach ($children as $child) {
if ($child->is_directory()) {
echo $child->get_visible_name();
// display contextid, itemid, component, filepath and filename
var_dump($child->get_params());
}
}
}

</code>

===Moving files around===

For example, if you have just built a file at the path
<code php>
$from_zip_file = $CFG->dataroot . '/temp/backup/' . $preferences->backup_unique_code .
'/' . $preferences->backup_name;
</code>
And you want to move it into the course_backup file area, do
<code php>
$context = get_context_instance(CONTEXT_COURSE, $preferences->backup_course);
$fs = get_file_storage();
$file_record = array('contextid'=>$context->id, 'component'=>'course', 'filearea'=>'backup',
'itemid'=>0, 'filepath'=>'/', 'filename'=>$preferences->backup_name,
'timecreated'=>time(), 'timemodified'=>time());
$fs->create_file_from_pathname($file_record, $from_zip_file);
</code>

===Create a copy of stored file===
TODO: this example is bogus, nobody should ever write code like this (skodak)
If you need to create a copy of stored file (actually, it add a new record in database):
<code php>
$context = get_context_instance_by_id($contextid);
$file_info = $browser->get_file_info($context, $component, $filearea, $fileitemid, $filepath, $filename);
// copy this file to draft area
$file_info->copy_to_storage($user_context->id, 'user', 'draft', $newitemid, '/', $title);
</code>

The above code is intended for user-interface behaviour which respects the current user's permissions. If you need to write code which copies files at the back-end in areas which are not directly editable by users (for example, when copying a glossary entry to another glossary, and you want to copy its attachments), you do not need to use the user-level $browser at all. All operations are carried out with the file_storage object (usually $fs). The method to use is create_file_from_storedfile.

For example, the glossary stores per-entry files in two areas ('mod_glossary', 'attachment' which stores attachments and 'mod_glossary', 'entry' which stores files directly related to the HTML definition, such as embedded images). The following code copies all the files relating to a specific glossary entry from one such area to a different glossary item in a different glossary. Note that it changes the context ID and item ID of each file to represent the new context (for the different glossary module) and the new glossary entry (a new row was added to glossary_entries).

<code php>
$fs = get_file_storage();
if ($files = $fs->get_area_files($oldcontextid, 'mod_glossary', 'attachment', $oldentryid)) {
foreach ($files as $file) {
$fs->create_file_from_storedfile(array(
'contextid' => $newcontextid,
'itemid' => $newentryid), $file);
}
}
</code>

=== List area files ===
<code php>
$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);
foreach ($files as $f) {
// $f is an instance of stored_file
echo $f->get_filename();
}
</code>

Or as links...

<code php>
$out = array();

$fs = get_file_storage();
$files = $fs->get_area_files($contextid, 'mod_assignment', 'submission', $submission->id);

foreach ($files as $file) {
$url = "{$CFG->wwwroot}/pluginfile.php/{$file->get_contextid()}/mod_assignment/submission}";
$filename = $file->get_filename();
$fileurl = $url.$file->get_filepath().$file->get_itemid().'/'.$filename;
$out[] = html_writer::link($fileurl, $filename);
}

$br = html_writer::empty_tag('br');

return implode($br, $out);
</code>

=== Create file ===

Here's how to create a file whose contents will be a text string. This is the equivalent of the PHP function <tt>file_put_contents</tt>.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'contextid' => $context->id, // ID of context
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Create file containing text 'hello world'
$fs->create_file_from_string($fileinfo, 'hello world');
</code>

If you want to create a file in the Moodle file area based on a 'real' file e.g. in a temporary folder, you can use <tt>create_file_from_pathname</tt> instead.

Unlike with ordinary files, this method will not automatically overwrite an existing file. If you wish to overwrite a file, you must first get the file and (if it exists) delete it, and only then create it again.

=== Read file ===

This is a way to read a file, equivalent to <tt>file_get_contents</tt>. '''Please note your are allowed to do this ONLY from mod/mymodule/* code, it is not acceptable to do this anywhere else.''' Other code has to use file_browser interface instead.

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule', // usually = table name
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Read contents
if ($file) {
$contents = $file->get_content();
} else {
// file doesn't exist - do something
}
</code>

If you want to access the file directly on disk, this is not permitted. Instead, you need to make a copy of the file in a temporary area and use that. You can do this with <tt>$file->copy_content_to($pathname)</tt>.

=== Delete file ===

<code php>
$fs = get_file_storage();

// Prepare file record object
$fileinfo = array(
'component' => 'mod_mymodule',
'filearea' => 'myarea', // usually = table name
'itemid' => 0, // usually = ID of row in table
'contextid' => $context->id, // ID of context
'filepath' => '/', // any path beginning and ending in /
'filename' => 'myfile.txt'); // any filename

// Get file
$file = $fs->get_file($fileinfo->contextid, $fileinfo->component, $fileinfo->filearea,
$fileinfo->itemid, $fileinfo->filepath, $fileinfo->filename);

// Delete it if it exists
if ($file) {
$file->delete();
}
</code>

==See also==

* [[File API]] how the File API works internally.
* [[Roadmap|Moodle 2.0 roadmap]]

[[Category:Files]]

File API internals

2012-01-16T03:11:22Z

Dongsheng:

{{Infobox Project
|name = File API
|state = Implemented
|tracker = MDL-14589
|discussion = n/a
|assignee = [[User:Petr Škoda (škoďák)|Petr Škoda (škoďák)]]
}}
{{Moodle 2.0}}

==Objectives==

The goals of the new File API are:

* allow files to be stored within Moodle, as part of the content (as we do now).
* use a consistent and flexible approach for all file handling throughout Moodle.
* give modules control over which users can access a file, using capabilities and other local rules.
* make it easy to determine which parts of Moodle use which files, to simplify operations like backup and restore.
* track where files originally came from.
* avoid redundant storage, when the same file is used twice.
* fully support Unicode file names, irrespective of the capabilities of the underlying file system.

==Overview==

The File API is a set of core interfaces to allow the rest of Moodle to store, serve and manage files. It applies only to files that are part of the Moodle site's content. It is not used for internal files, such as those in the following subdirectories of dataroot: temp, lang, cache, environment, filter, search, sessions, upgradelogs, ...

To learn how to use File API, please visit [[Using the File API]].

The API can be subdivided into the following parts:
; File storage
: Low level file storage without access control information. Stores the content of files on disc, with metadata in associated database tables.
; File serving
: Lets users accessing a Moodle site get the files (file.php, draftfile.php, pluginfile.php, userfile.php, etc.)
:* Serve the files on request
:* with appropriate security checks
; File related user interfaces
: Provides the interface for (lib/form/file.php, filemanager.php, filepicker.php and files/index.php, draftfiles.php)
:* Form elements allowing users to select a file using the file picker, and have it stored within Moodle.
:* UI for users to manage their files, replacing the old course files UI
; File browsing API
: Allows code to browse and optionally manipulate the file areas
:* find information about available files in each area.
:* print links to files.
:* optionally move/rename/copy/delete/etc.

== File API internals ==

=== File storage on disk ===

Files are stored in $CFG->dataroot (also known as moodledata) in the filedir subfolder.

Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.

Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce.

This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

=== Files table ===

This table contains one entry for each usage of a file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary. It is necessary because some databases have hard limit on index size.

If, for example, the same image is used in a user's profile, and a forum post, then there will be two rows in this table, one for each use of the file, and Moodle will treat the two as separate files, even though the file is only stored once on disc.

{| class="nicetable"
! Field
! Type
! Default
! Info
|-
| '''id'''
| int(10)
| auto-incrementing
| The unique ID for this file.
|-
| '''contenthash'''
| varchar(40)
|
| The sha1 hash of content.
|-
| '''pathnamehash'''
| varchar(40)
|
| The sha1 hash of "/contextid/component/filearea/itemid/filepath/filename.ext" - prevents file duplicates and allows fast lookup. It is necessary because some databases have hard limit on index size.
|-
| '''contextid'''
| int(10)
|
| The context id defined in context table - identifies the instance of plugin owning the file.
|-
| '''component'''
| varchar(50)
|
| Like "mod_forum", "course", "mod_assignment", "backup"
|-
| '''filearea'''
| varchar(50)
|
| Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.
|-
| '''itemid'''
| int(10)
|
| Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)
|-
| filepath
| text
|
| relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this
|-
| filename
| varchar(255)
|
| The full Unicode name of this file (case sensitive)
|-
| '''userid'''
| int(10)
| NULL
| Optional - general user id field - meaning depending on plugin
|-
| filesize
| int(10)
|
| size of file - bytes
|-
| mimetype
| varchar(100)
| NULL
| type of file
|-
| status
| int(10)
|
| general file status flag - will be used for lost or infected files
|-
| source
| text
|
| file source - usually url
|-
| author
| varchar(255)
|
| original author of file, used when importing from other systems
|-
| license
| varchar(255)
|
| license type, empty means site default
|-
| timecreated
| int(10)
|
| The time this file was created
|-
| timemodified
| int(10)
|
| The last time the file was last modified
|}

Indexes:
* non-unique index on (contextid, component, filearea, itemid)
* non-unique index on (contenthash)
* unique index on (pathnamehash).

The plugin type does not need to be specified because it can be derived from the context. Items like blog that do not have their own context will use their own file area inside a suitable context. In this case, the user context.

Entries with filename = '.' represent directories. Directory entries like this are created automatically when a file is added within them.

Note: 'files' plural is used even thought that goes against the [[Database|coding guidelines]] because 'file' is a reserved word in some SQL dialects.

===Implementation of basic operations===

'''Each plugin may directly access only files in own context and areas!'''

Low level access API is defined in ''file_storage'' class which is obtained from <code>get_file_storage()</code>.

====Storing a file====

# Calculate the SHA1 hash of the file contents.
# Check if a file with this SHA1 hash already exists on disc in file directory or file trash. If not, store the file there.
# Add the record for this file to the files table using the low level address

====Reading a file====

# Fetch the record (which includes the SHA1 hash) for the file you want from the files table. You can fetch either all area files or quickly get one file with a specific contenthash.
# Retrieve the contents using the SHA1 hash from the file directory.

====Deleting a file====

# Delete the record from the files table.
# Verify if some other file is still needing the content, if not move the content file into file trash
# Later, admin/cron.php deletes content files from trash directory

== File serving ==

Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server properly (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

=== legacy file.php ===

Serves legacy course files, the file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files are stored in <code>array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>

The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.

=== pluginfile.php ===
All plugins should use this script to serve all files.
* plugins decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* links to these files are constructed on the fly from the relative links stored in database, this means that plugin may link only own files

Absolute file links need to be rewritten if html editing allowed in plugin. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/intro/image.jpg, it is converted back to internal links before saving.

Script parameters are virtual file names, in most cases the parameters match the low level file storage, but they do not have to:

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

==== Assignment example ====

/pluginfile.php/assignmentcontextid/mod_assignment/intro/someimage.jpg
/pluginfile.php/assignmentcontextid/mod_assignment/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/mod_assignment/allsubmissions/groupid/allsubmissionfiles.zip

The last line example of virtual file that should created on the fly, it is not implemented yet.

====scorm example====

/pluginfile.php/scormcontextid/mod_scorm/intro/someimage.jpg
/pluginfile.php/scormcontextid/mod_scorm/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems.

====quiz example====

pluginfile.php/quizcontextid/mod_quiz/intro/niceimage.jpg

====questions example====

This section was out of date. See [[File_storage_conversion_Quiz_and_Questions]] for the latest thinking.

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>

/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>

=== Temporary files ===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* processing by executable files (latex, mimetex)

These files should never use utf-8 file names.

=== Legacy file storage and serving ===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

== File browsing API ==

This is what other parts of Moodle use to access files that they do not own.

=== Class: file_browser ===

=== Class: file_info and subclasses ===

== File related user interfaces ==

All files are obtained through from the file repositories.

=== Formslib fields ===
* file picker
* file manager
* file upload (obsolete, do not use)

=== Integration with the HTML editor ===

Each instance of the HTML editor can be told to store related files in a particular file area.

During editing, files are stored in a draft files area. Then when the form is submitted they are moved into the real file area.

Files are selected using the repository file picker.

=== Legacy file manager ===

Available only for legacy reasons. It is not supposed to be used.

All the contexts, file areas and files now form a single huge tree structure, although each user only has access to certain parts of that tree. The file manager (files/index.php) allow users to browse this tree, and manage files within it, according to the level of permissions they have.

Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

== Backwards compatibility ==

=== Content backwards compatibility ===

This should be preserved as much as possible. This will involve rewriting links in content during the upgrade to 2.0.

Some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Code backwards compatibility===

Other Moodle code (for example plugins) will have to be converted to the new APIs. See [[Using_the_file_API]] for guidance.

It is not possible to provide backwards-compatibility here. For example, the old $CFG->dataroot/$courseid/ will no longer exist, and there is no way to emulate that, so we won't try.

== Upgrade and migration ==

When a site is upgraded to Moodle 2.0, all the files in moodledata will have to be migrated. This is going to be a pain, like DML/DDL was :-(

The upgrade process should be interruptible (like the Unicode upgrade was) so it can be stopped/restarted any time.

=== Migration of content ===

* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new area, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be many outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

=== Moving files to files table and file pool ===

The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.

Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some people use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

== Other issues ==

=== Unicode support in zip format ===

Zip format is an old standard for compressing files. It was created long before Unicode existed, and Unicode support was only recently added. There are several ways used for encoding of non-ASCII characters in path names, but unfortunately it is not very standardised. Most Windows packers use DOS encoding.

Client software:
* Windows built-in compression - bundled with Windows, non-standard DOS encoding only
* WinZip - shareware, Unicode option (since v11.2)
* TotalCommander - shareware, single byte(DOS) encoding only
* 7-Zip - free, Unicode or DOS encoding depending on characters used in file name (since v4.58beta)
* Info-ZIP - free, uses some weird character set conversions

PHP extraction:
* Info-ZIP binary execution - no Unicode support at all, mangles character sets in file names (depends on OS, see docs), files must be copied to temp directory before compression and after extraction
* PclZip PHP library - reads single byte encoded names only, problems with random problems and higher memory usage.
* Zip PHP extension - kind of works in latest PHP versions

Large file support:
PHP running under 32bit operating systems does not support files >2GB (do not expect fix before PHP 6). This might be a potential problem for larger backups.

Tar Alternative:
* tar with gzip compression - easy to implement in PHP + zlib extension (PclTar, Tar from PEAR or custom code)
* no problem with unicode in *nix, Windows again expects DOS encoding :-(
* seems suitable for backup/restore - yay!

Summary:
# added zip processing class that fully hides the underlying library
# using single byte encoding "garbage in/garbage out" approach for encoding of files in zip archives; add new 'zipencoding' string into lang packs (ex: cp852 DOS charset for Czech locale) and use it during extraction (we might support true unicode later when PHP Zip extension does that)

== Not implemented yet ==
* antivirus scanning - this needs a different api because the upload of files is now handled via repository plugins

== See also ==

* [[Using the File API]]
* [[Repository API]]
* [[Portfolio API]]
* [[Resource module file API migration]]
* MDL-14589 - File API Meta issue

[[Category:Files]]
[[Category:Interfaces]]

File API internals

2012-01-16T03:04:57Z

Dongsheng: /* See also */

{{Work in progress}}
{{Infobox Project
|name = File API
|state = Implemented
|tracker = MDL-14589
|discussion = n/a
|assignee = [[User:Petr Škoda (škoďák)|Petr Škoda (škoďák)]]
}}
{{Moodle 2.0}}

==Objectives==

The goals of the new File API are:

* allow files to be stored within Moodle, as part of the content (as we do now).
* use a consistent and flexible approach for all file handling throughout Moodle.
* give modules control over which users can access a file, using capabilities and other local rules.
* make it easy to determine which parts of Moodle use which files, to simplify operations like backup and restore.
* track where files originally came from.
* avoid redundant storage, when the same file is used twice.
* fully support Unicode file names, irrespective of the capabilities of the underlying file system.

==Overview==

The File API is a set of core interfaces to allow the rest of Moodle to store, serve and manage files. It applies only to files that are part of the Moodle site's content. It is not used for internal files, such as those in the following subdirectories of dataroot: temp, lang, cache, environment, filter, search, sessions, upgradelogs, ...

The API can be subdivided into the following parts:
; File storage
: Low level file storage without access control information. Stores the content of files on disc, with metadata in associated database tables.
; File serving
: Lets users accessing a Moodle site get the files (file.php, draftfile.php, pluginfile.php, userfile.php, etc.)
:* Serve the files on request
:* with appropriate security checks
; File related user interfaces
: Provides the interface for (lib/form/file.php, filemanager.php, filepicker.php and files/index.php, draftfiles.php)
:* Form elements allowing users to select a file using the file picker, and have it stored within Moodle.
:* UI for users to manage their files, replacing the old course files UI
; File browsing API
: Allows code to browse and optionally manipulate the file areas
:* find information about available files in each area.
:* print links to files.
:* optionally move/rename/copy/delete/etc.

== File API internals ==

=== File storage on disk ===

Files are stored in $CFG->dataroot (also known as moodledata) in the filedir subfolder.

Files are stored according to the SHA1 hash of their content. This means each file with particular contents is stored once, irrespective of how many times it is included in different places, even if it is referred to by different names. (This idea comes from the git version control system.) To relate a file on disc to a user-comprehensible path or filename, you need to use the ''files'' database table. See the next section.

Suppose a file has SHA1 hash 081371cb102fa559e81993fddc230c79205232ce. Then it will be stored in on disc as moodledata/filedir/08/13/081371cb102fa559e81993fddc230c79205232ce.

This means Moodle can not store two files with the same SHA1 hash, luckily it is extremely unlikely that this would ever happen. Technically it is also possible to implement reliable collision tests (with some performance cost), for now we just test file lengths in addition to SHA1 hash.

=== Files table ===

This table contains one entry for each usage of a file. Enough information is kept here so that the file can be fully identified and retrieved again if necessary. It is necessary because some databases have hard limit on index size.

If, for example, the same image is used in a user's profile, and a forum post, then there will be two rows in this table, one for each use of the file, and Moodle will treat the two as separate files, even though the file is only stored once on disc.

{| class="nicetable"
! Field
! Type
! Default
! Info
|-
| '''id'''
| int(10)
| auto-incrementing
| The unique ID for this file.
|-
| '''contenthash'''
| varchar(40)
|
| The sha1 hash of content.
|-
| '''pathnamehash'''
| varchar(40)
|
| The sha1 hash of "/contextid/component/filearea/itemid/filepath/filename.ext" - prevents file duplicates and allows fast lookup. It is necessary because some databases have hard limit on index size.
|-
| '''contextid'''
| int(10)
|
| The context id defined in context table - identifies the instance of plugin owning the file.
|-
| '''component'''
| varchar(50)
|
| Like "mod_forum", "course", "mod_assignment", "backup"
|-
| '''filearea'''
| varchar(50)
|
| Like "submissions", "intro" and "content" (images and swf linked from summaries), etc.; "blogs" and "userfiles" are special case that live at the system context.
|-
| '''itemid'''
| int(10)
|
| Some plugin specific item id (eg. forum post, blog entry or assignment submission or user id for user files)
|-
| filepath
| text
|
| relative path to file from module content root, useful in Scorm and Resource mod - most of the mods do not need this
|-
| filename
| varchar(255)
|
| The full Unicode name of this file (case sensitive)
|-
| '''userid'''
| int(10)
| NULL
| Optional - general user id field - meaning depending on plugin
|-
| filesize
| int(10)
|
| size of file - bytes
|-
| mimetype
| varchar(100)
| NULL
| type of file
|-
| status
| int(10)
|
| general file status flag - will be used for lost or infected files
|-
| source
| text
|
| file source - usually url
|-
| author
| varchar(255)
|
| original author of file, used when importing from other systems
|-
| license
| varchar(255)
|
| license type, empty means site default
|-
| timecreated
| int(10)
|
| The time this file was created
|-
| timemodified
| int(10)
|
| The last time the file was last modified
|}

Indexes:
* non-unique index on (contextid, component, filearea, itemid)
* non-unique index on (contenthash)
* unique index on (pathnamehash).

The plugin type does not need to be specified because it can be derived from the context. Items like blog that do not have their own context will use their own file area inside a suitable context. In this case, the user context.

Entries with filename = '.' represent directories. Directory entries like this are created automatically when a file is added within them.

Note: 'files' plural is used even thought that goes against the [[Database|coding guidelines]] because 'file' is a reserved word in some SQL dialects.

===Implementation of basic operations===

'''Each plugin may directly access only files in own context and areas!'''

Low level access API is defined in ''file_storage'' class which is obtained from <code>get_file_storage()</code>.

====Storing a file====

# Calculate the SHA1 hash of the file contents.
# Check if a file with this SHA1 hash already exists on disc in file directory or file trash. If not, store the file there.
# Add the record for this file to the files table using the low level address

====Reading a file====

# Fetch the record (which includes the SHA1 hash) for the file you want from the files table. You can fetch either all area files or quickly get one file with a specific contenthash.
# Retrieve the contents using the SHA1 hash from the file directory.

====Deleting a file====

# Delete the record from the files table.
# Verify if some other file is still needing the content, if not move the content file into file trash
# Later, admin/cron.php deletes content files from trash directory

== File serving ==

Deals with serving of files - browser requests file, Moodle sends it back. We have three main files. It is important to setup slasharguments on server properly (file.php/some/thing/xxx.jpg), any content that relies on relative links can not work without it (scorm, uploaded html pages, etc.).

=== legacy file.php ===

Serves legacy course files, the file name and parameter structure is critical for backwards compatibility of existing course content.

/file.php/courseid/dir/dir/filename.ext

Internally the files are stored in <code>array('contextid'=>$coursecontextid, 'component;=>'course', 'filearea'=>'legacy', 'itemid'=>0)</code>

The legacy course files are completely disabled in all new courses created in 2.0. The major problem here is to how to educate our users that they can not make huge piles of files in each course any more.

=== pluginfile.php ===
All plugins should use this script to serve all files.
* plugins decide about access control
* optional XSS protection - student submitted files must not be served with normal headers, we have to force download instead; ideally there should be second wwwroot for serving of untrusted files
* links to these files are constructed on the fly from the relative links stored in database, this means that plugin may link only own files

Absolute file links need to be rewritten if html editing allowed in plugin. The links are stored internally as relative links. Before editing or display the internal link representation is converted to absolute links using simple str_replace() @@thipluginlink/summary@@/image.jpg --> /pluginfile.php/assignmentcontextid/intro/image.jpg, it is converted back to internal links before saving.

Script parameters are virtual file names, in most cases the parameters match the low level file storage, but they do not have to:

/pluginfile.php/contextid/areaname/arbitrary/params/or/dirs/filename.ext

pluginfile.php detects the type of plugin from context table, fetches basic info (like $course or $cm if appropriate) and calls plugin function (or later method) which does the access control and finally sends the file to user. ''areaname'' separates files by type and divides the context into several subtrees - for example ''summary'' files (images used in module intros), post attachments, etc.

==== Assignment example ====

/pluginfile.php/assignmentcontextid/mod_assignment/intro/someimage.jpg
/pluginfile.php/assignmentcontextid/mod_assignment/submission/submissionid/attachmentname.ext
/pluginfile.php/assignmentcontextid/mod_assignment/allsubmissions/groupid/allsubmissionfiles.zip

The last line example of virtual file that should created on the fly, it is not implemented yet.

====scorm example====

/pluginfile.php/scormcontextid/mod_scorm/intro/someimage.jpg
/pluginfile.php/scormcontextid/mod_scorm/content/revisionnumber/dir/somescormfile.js

The revision counter is incremented when any file changes in order to prevent caching problems.

====quiz example====

pluginfile.php/quizcontextid/mod_quiz/intro/niceimage.jpg

====questions example====

This section was out of date. See [[File_storage_conversion_Quiz_and_Questions]] for the latest thinking.

====blog example====
Blog entries or notes in general do not have context id (because they live in system context, SYSCONTEXTID below is the id of system context).
The note attachments are always served with XSS protection on, ideally we should use separate wwwroot for this. Access control can be hardcoded.

/pluginfile.php/SYSCONTEXTID/blog/attachment/blogentryid/attachmentname.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'attachment', 'itemid'=>$blogentryid)</code>

/pluginfile.php/SYSCONTEXTID/blog/post/blogentryid/embeddedimage.ext

Internally stored in <code>array('contextid'=>SYSCONTEXTID, 'component'=>'blog', 'filearea'=>'post', 'itemid'=>$blogentryid)</code>

=== Temporary files ===
Temporary files are usually used during the lifetime of one script only.
uses:
* exports
* imports
* processing by executable files (latex, mimetex)

These files should never use utf-8 file names.

=== Legacy file storage and serving ===
Going to use good-old separate directories in $CFG->dataroot.

file serving and storage:
# user avatars - user/pix.php
# group avatars - user/pixgroup.php
# tex, algebra - filter/tex/* and filter/algebra/*
# rss cache (?full rss rewrite soon?) - backwards compatibility only rss/file.php

only storage:
#sessions

== File browsing API ==

This is what other parts of Moodle use to access files that they do not own.

=== Class: file_browser ===

=== Class: file_info and subclasses ===

== File related user interfaces ==

All files are obtained through from the file repositories.

=== Formslib fields ===
* file picker
* file manager
* file upload (obsolete, do not use)

=== Integration with the HTML editor ===

Each instance of the HTML editor can be told to store related files in a particular file area.

During editing, files are stored in a draft files area. Then when the form is submitted they are moved into the real file area.

Files are selected using the repository file picker.

=== Legacy file manager ===

Available only for legacy reasons. It is not supposed to be used.

All the contexts, file areas and files now form a single huge tree structure, although each user only has access to certain parts of that tree. The file manager (files/index.php) allow users to browse this tree, and manage files within it, according to the level of permissions they have.

Single pane file manager is hard to implement without drag & drop which is notoriously problematic in web based applications. I propose to implement a two pane commander-style file manager. Two pane manager allows you to easily copy/move files between two different contexts (ex: courses).

File manager must not interact directly with filesystem API, instead each module should return traversable tree of files and directories with both real and localised names (localised names are needed for dirs like backupdata).

== Backwards compatibility ==

=== Content backwards compatibility ===

This should be preserved as much as possible. This will involve rewriting links in content during the upgrade to 2.0.

Some new features (like resource sharing - if implemented) may not work with existing data that still uses files from course files area.

There might be a breakage of links due to special characters stripping in uploaded files which will not match the links in uploaded html files any more. This should not be very common I hope.

===Code backwards compatibility===

Other Moodle code (for example plugins) will have to be converted to the new APIs. See [[Using_the_file_API]] for guidance.

It is not possible to provide backwards-compatibility here. For example, the old $CFG->dataroot/$courseid/ will no longer exist, and there is no way to emulate that, so we won't try.

== Upgrade and migration ==

When a site is upgraded to Moodle 2.0, all the files in moodledata will have to be migrated. This is going to be a pain, like DML/DDL was :-(

The upgrade process should be interruptible (like the Unicode upgrade was) so it can be stopped/restarted any time.

=== Migration of content ===

* resources - move files to new resource content file area; can be done automatically for pdf, image resources; definitely not accurate for uploaded web pages
* questions - image file moved to new area, image tag appended to questions
* moddata files - the easiest part, just move to new storage
* coursefiles - there might be many outdated files :-( :-(
* rss feeds links in readers - will be broken, the new security related code would break it anyway

=== Moving files to files table and file pool ===

The migration process must be interruptable because it might take a very long time. The files would be moved from old location, the restarting would be straightforward.

Proposed stages:
#migration of all course files except moddata - finish marked by some $CFG->files_migrated=true; - this step breaks the old file manager and html editor integration
#migration of blog attachments
#migration of question files
#migration of moddata files - each module is responsible to copy data from converted coursefiles or directly from moddata which is not converted automatically

Some people use symbolic links in coursefiles - we must make sure that those will be copied to new storage in both places, though they can not be linked any more - anybody wanting to have content synced will need to move the files to some repository and set up the sync again.

::Talked about a double task here, when migrating course files to module areas:
::# Parse html files to detect all the dependencies and move them together.
::# Fallback in pluginfile.php so, if something isn't found in module filearea, search for it in course filearea, copying it and finally, serving it.

:: Also we talked about the possibility of add a new setting to resource in order to define if it should work against old coursefiles or new autocontained file areas. Migrated resources will point to old coursefiles while new ones will enforce autocontained file areas.

:: it seems that only resource files will be really complex (because allow arbitrary HTML inclusion). The rest (labels, intros... doesn't) and should be easier to parse.

::[[User:Eloy Lafuente (stronk7)|Eloy Lafuente (stronk7)]] 19:00, 29 June 2008 (CDT)

== Other issues ==

=== Unicode support in zip format ===

Zip format is an old standard for compressing files. It was created long before Unicode existed, and Unicode support was only recently added. There are several ways used for encoding of non-ASCII characters in path names, but unfortunately it is not very standardised. Most Windows packers use DOS encoding.

Client software:
* Windows built-in compression - bundled with Windows, non-standard DOS encoding only
* WinZip - shareware, Unicode option (since v11.2)
* TotalCommander - shareware, single byte(DOS) encoding only
* 7-Zip - free, Unicode or DOS encoding depending on characters used in file name (since v4.58beta)
* Info-ZIP - free, uses some weird character set conversions

PHP extraction:
* Info-ZIP binary execution - no Unicode support at all, mangles character sets in file names (depends on OS, see docs), files must be copied to temp directory before compression and after extraction
* PclZip PHP library - reads single byte encoded names only, problems with random problems and higher memory usage.
* Zip PHP extension - kind of works in latest PHP versions

Large file support:
PHP running under 32bit operating systems does not support files >2GB (do not expect fix before PHP 6). This might be a potential problem for larger backups.

Tar Alternative:
* tar with gzip compression - easy to implement in PHP + zlib extension (PclTar, Tar from PEAR or custom code)
* no problem with unicode in *nix, Windows again expects DOS encoding :-(
* seems suitable for backup/restore - yay!

Summary:
# added zip processing class that fully hides the underlying library
# using single byte encoding "garbage in/garbage out" approach for encoding of files in zip archives; add new 'zipencoding' string into lang packs (ex: cp852 DOS charset for Czech locale) and use it during extraction (we might support true unicode later when PHP Zip extension does that)

== Not implemented yet ==
* antivirus scanning - this needs a different api because the upload of files is now handled via repository plugins

== See also ==

* [[Using the File API]]
* [[Repository API]]
* [[Portfolio API]]
* [[Resource module file API migration]]
* MDL-14589 - File API Meta issue

[[Category:Files]]
[[Category:Interfaces]]