Data formats: Difference between revisions
David Mudrak (talk | contribs) m (Mudrd8mz moved page Dataformats to Data formats) |
David Mudrak (talk | contribs) m (Reformat, change Dataformat to human readable data format) |
||
| Line 1: | Line 1: | ||
Data formats are plugins that define how a table of data can be exported for download. Moodle comes with plugins for CSV, Excel, ODS, JSON, and HTML. | |||
= Components = | == Components == | ||
; dataformat/FORMATNAME/lang/en/dataformat_FORMATNAME.php : Contains the English language strings used in the format. You can also define strings for other languages. | |||
; dataformat/FORMATNAME/classes/writer.php : This file is the real meat of the form, it contains class which extends <tt>\core\dataformat\base</tt> defining how the data stream should be encoded. | |||
; dataformat/FORMATNAME/version.php : Version definitions, see [[version.php]]. It is highly recommended always to have it and it is required if you have any files in db folder | |||
Version definitions, see [[version.php]]. It is highly recommended always to have it and it is required if you have any files in db folder | |||
Generally no other files are needed, but all the other files available to other plugins could also be used here such as db files. | Generally no other files are needed, but all the other files available to other plugins could also be used here such as db files. | ||
= Creating a new | == Creating a new data format == | ||
The | The data format interface is quite simple, to create a new format the easiest approach would be to clone and rename one of the existing text based formats such as /dataformat/json or /dataformat/html. | ||
= Performance = | == Performance == | ||
The design intent of the | The design intent of the data formats is to have low and ideally fixed memory overhead, and to stream data to the browser record by record. This enables very large datasets to be exported easily and safely without running into a whole range of issues including php memory thresholds, php execution timeouts, and other timeouts in every layer of the stack at the apache / nginx layer, a caching layer such as varnish, or load balancers, any intermediate proxies, and finally any browser time-outs. | ||
So please avoid building up a complete data structure in memory, or ideally not even on disk either. Instead just encode the data record by record as the data stream is processed and echo it to the browser. The HTML, JSON, and CSV | So please avoid building up a complete data structure in memory, or ideally not even on disk either. Instead just encode the data record by record as the data stream is processed and echo it to the browser. The HTML, JSON, and CSV data formats are good examples of this. The Excel and ODS formats are far more performant than in earlier versions of moodle, but still suffer from network inactivity timeouts at very large scale because they buffer the file to disc before finally sending when complete. | ||
Revision as of 09:39, 20 May 2016
Data formats are plugins that define how a table of data can be exported for download. Moodle comes with plugins for CSV, Excel, ODS, JSON, and HTML.
Components
- dataformat/FORMATNAME/lang/en/dataformat_FORMATNAME.php
- Contains the English language strings used in the format. You can also define strings for other languages.
- dataformat/FORMATNAME/classes/writer.php
- This file is the real meat of the form, it contains class which extends \core\dataformat\base defining how the data stream should be encoded.
- dataformat/FORMATNAME/version.php
- Version definitions, see version.php. It is highly recommended always to have it and it is required if you have any files in db folder
Generally no other files are needed, but all the other files available to other plugins could also be used here such as db files.
Creating a new data format
The data format interface is quite simple, to create a new format the easiest approach would be to clone and rename one of the existing text based formats such as /dataformat/json or /dataformat/html.
Performance
The design intent of the data formats is to have low and ideally fixed memory overhead, and to stream data to the browser record by record. This enables very large datasets to be exported easily and safely without running into a whole range of issues including php memory thresholds, php execution timeouts, and other timeouts in every layer of the stack at the apache / nginx layer, a caching layer such as varnish, or load balancers, any intermediate proxies, and finally any browser time-outs.
So please avoid building up a complete data structure in memory, or ideally not even on disk either. Instead just encode the data record by record as the data stream is processed and echo it to the browser. The HTML, JSON, and CSV data formats are good examples of this. The Excel and ODS formats are far more performant than in earlier versions of moodle, but still suffer from network inactivity timeouts at very large scale because they buffer the file to disc before finally sending when complete.
