Paint tool integration

PaintWeb is the paint tool which is going to be integrated into Moodle. For more information about PaintWeb please read the project specification.

Overview

PaintWeb is currently a strictly client-side project, a Web application. It provides an API for developers who wish to extend its functionality, via new drawing tools, new extensions and new commands.

The plan is to integrate PaintWeb in Moodle in the appropriate places. Meaning, in some cases, users would like to be able to edit images, in place, in the HTML editor (TinyMCE mostly). Another place of integration would be the file manager - there users should be allowed to pick images files and edit them. Another use-case for PaintWeb is separate loading, stand-alone usage - for example a page which only allows the user to draw on the Canvas and save the result.

PaintWeb already includes a TinyMCE plugin which allows users to start editing any image in the HTML document. The plugin allows users to start editing an image by clicking an Edit button which shows overlayed on top of images when they are selected. Additionally, users can right-click an image to pick the Edit in PaintWeb. Lastly, when the advanced theme is used, the paintwebEdit button is also available.

Things to take into consideration:

• PaintWeb can only edit images from the same domain, due to security restrictions. This means that any image from a different domain is either denied from editing, or we can use a server-side script which downloads the image locally on the server. The latter option is more cumbersome, and I believe it implies some security risks: the user can point to arbitrary malicious files which the server downloads and sends back to the browser. The TinyMCE plugin I have does not allow the users to edit of any image from a different domain.

• The TinyMCE plugin needs to be loaded dynamically. The PaintWeb image editor should not show up in all cases where TinyMCE is used. Thus, Moodle needs an API which other developers can use to show the HTML editor and configure it: "hey, I want PaintWeb as well!" or not.

• There's an important difference between the use of PaintWeb inside TinyMCE and standalone use. Inside TinyMCE the user already has a dialog for inserting images - thus in PaintWeb we do not need an option to load another image. In PaintWeb-TinyMCE we only need the options to save and cancel the changes. In a standalone instance of PaintWeb we might want to allow the user to load and save images at will. The file manager use-case is the same as in TinyMCE: the file manager handles the files itself, PaintWeb should be strictly the image editor (with save/cancel).

Given the above, here is what I consider needs to be done:

• Moodle 2 has an API for inserting the HTML editor (TinyMCE) anywhere desired. This API needs to be extended such that the developer can tell which plugins need to be loaded - in this case PaintWeb.

• Moodle 2 has a File API which needs to be used by PaintWeb when the file save/load operations are performed. On the client-side of things this can be done by writing an extension for PaintWeb which handles events like imageSave. PaintWeb in Moodle needs some "common backend" which handles image save/load in all use-cases.

• It should be noted that having a PaintWeb extensions specific for Moodle would replace the need of having custom patches applied to PaintWeb - like Moodle does for TinyMCE.

• The PaintWeb script needs to communicate with the server to perform file load/save. So, on the server-side these operations can be handled by a piece of common code, e.g. /lib/paintweb/ext/moodle/lib.php. Suggestions for a different file name / place are welcome. This PHP script needs to use the File API to perform the desired operations. An API needs to be exposed by this file, an API others can use to integrate PaintWeb in a standalone page. For example paintweb_setup() which sets-up the required JS file in the page, for the current page. Then paintweb_insert(options) to insert the PaintWeb initialization code, with user-defined options.

• The upcoming file manager needs an API for adding "file handlers". For example, PaintWeb can be a file handler for images. Thus, a button, or some context menu item for images could be displayed Edit this image in PaintWeb.

An additional vector of PaintWeb integration is Moodleforms, something like $mform->addElement('image', 'targetInputElement', options).

The Moodle 2.0 editors API

Currently, developers who want to have a textarea element in their page with TinyMCE can do the following:

// setup the page requirements (JS files needed to be loaded)
editors_head_setup();

// get an instance of the editor
$editor = get_preferred_texteditor(FORMAT_HTML);

// use the editor: textarea element ID, and options.
$editor->use_editor('elementId', array());

echo '<textarea rows="20" cols="10"
id="elementId">some HTML</textarea>';

According to Petr Škoda, the developer of the API, the API still needs lots of work. Personally I consider the API only needs ... further work. Here's how I'd like to do things:

$editor->use_editor('elementId',
  array('imageEditing' => true,
        'mathEditing' => true
  )
);

Basically, I don't think there's something inherently "bad" about the current API. I only want to tell the editor instance "hey, I'd like image editing, math editing, etc, if possible".

I'd say try to keep things abstracted. So, if I say in the options array I want image editing, then TinyMCE can load the PaintWeb plugin. If I want math editing, then load dragmath and anything related.

The way each editor implements these features should be left up to the editor itself.

Obviously, based on such options, even the editor GUI can be altered: say if the PaintWeb plugin is desired, then the paintwebEdit button can be included in the theme toolbar.

Additionally, we should have options for configuring the GUI - but not something tied to the TinyMCE config. I don't think the API should go down the road of allowing the direct input of TinyMCE configuration options. If that's desired, then do not have multiple editors. Switch to a single editor: TinyMCE and have all the API tailored to this editor.

The Moodle 2.0 file manager

As explained above, the Moodle file manager should provide an API for registering file handlers, file processing tools, etc.

In the file manager the user might want to edit HTML files with TinyMCE, edit images with PaintWeb, or select multiple files to apply some batch process - like image resize.

It should be taken into consideration that some file operations can be performed on the server or client-side. Meaning, image resize can be performed server-side, but image editing needs to be performed client-side. When registering file handlers, one should be able to define the kind of operation: does the action need to load and invoke a client-side JavaScript method from a library? or is the action supposed to be performed by a server-side script from some Moodle library?

The PaintWeb extension

The whole PaintWeb package should live in /lib/paintweb.

In the overview I pointed out a PaintWeb extension needs to be developed to properly integrate into Moodle. Sample code:

pwlib.extensions.moodle = function (app) {
  this.extensionRegister = function () {
    // add an event listener for the imageSave event.
    app.events.add('imageSave', this.imageSave);

    // add a hidden form in the document
    // used for submitting the image to the server
    // ...
  };

  this.imageSave = function (ev) {
    ev.preventDefault();

    // a hidden form we can submit to the server
    form.imageFile.value = ev.dataURI;
    form.submit();
  };
  // ...
};

When the user decides to save, the browser outputs the image data as a data URI, base64 encoded. We can submit this to the server.

The PaintWeb lib code

The /lib/paintweb/ext/moodle/lib.php file.

On the server side, all the requests for file save need to be handled using the new File API:

// reference to the temporary imageFile uploaded by php
$imgfile = $_FILES['imageFile']['tmp_name'];
$filename = $_FILES['imageFile']['name'];
$fs = get_file_storage();
$file_record = array('contextid' => $context->id,
    'filearea' => $filearea,
    'itemid'=> 0,
    'filepath'=> '/',
    'filename' => $filename,
    'timecreated' => time(),
    'timemodified' => time());
$fs->create_file_from_pathname($file_record, $imgfile);

Questions:

• What context should be used? A different context depending where PaintWeb is used? Or a single context for all the files saved by PaintWeb? If different contexts need to be used, how should we pass the context to PaintWeb? Maybe the script which integrates PaintWeb also needs to pass the context?

• How about the file area? Same question as above.

The PaintWeb library code must also provide API for using PaintWeb standalone in any page desired by the developer. Here's an idea:

function paintweb_setup () {
  global $PAGE;
  $PAGE->requires->js('/lib/paintweb/build/paintweb.js');
  $PAGE->requires->js('/lib/paintweb/ext/moodle/lib.js');
};

function paintweb_insert ($elementId, $config = array()) {
  global $PAGE;

  $PAGE->requires->js_function_call('paintweb_moodle_init',
      array($elementId, $config));
};

The ext/moodle/lib.js can initialize PaintWeb:

function paintweb_moodle_init (elemId, config) {
  var pw = new PaintWeb();
  pw.config.guiPlaceholder = document.getElementById(elemId);

  // The Moodle config file.
  pw.config.configFile = 'config-moodle.json';

  // Use the configuration provided by the Moodle
  // server-side script.
  if (config) {
    for (var prop in config) {
      pw.config[prop] = config[prop];
    }
  }

  pw.init();
};

Localization

The TinyMCE integration has a script which generates Moodle language files from the original files provided by TinyMCE.

I have two scripts which convert the PaintWeb JSON language files into Moodle PHP languages files, and back.

I presume that Moodle translators will want to work with the Moodle PHP language files. This means that they (or someone else) can update the PaintWeb JSON language files using the convertor script.

Should PaintWeb load static JSON files or should it load a script which automatically generates the JSON using the strings stored in the moodle/lang/*/paintweb.php files?

Image load/save in PaintWeb

The data URI scheme allows Web developers to include raw binary content inside HTML, CSS and JavaScript documents. This scheme holds the binary data using base64 encoding which makes it ideal most of the time for small binary files. PaintWeb builds use image data URLs inside CSS to avoid loading tens of separate PNG icons. Almost anywhere you can provide an URL, say inside an <img src> tag, you can use a data URL instead.

Loading an image inside PaintWeb is a matter of only having a reference to a DOM image element in the page. The browser can immediately render the image element inside the Canvas, such that PaintWeb can edit the pixels. It's only simple code like canvas2dcontext.drawImage(imageElement, x, y).

Saving an image from PaintWeb is similarly easy: canvasElement.toDataURL(). This tells the browser to return the image in PNG format, base64-encoded. The value can be POSTed via a simple HTML form or via an XMLHttpRequest. Note: this value is NOT a file! This is a typical JavaScript string, and the server-side scripts recognize it like any other string. It is up to the server-side script to base64-decode the value and save it as a file. From PHP doing this is trivial:

file_put_contents('image.png', base64_decode($base64_value));

Images from different servers than those of the page running the scripts can be rendered inside Canvas, but when they are rendered, they mark the Canvas element as "dirty" due to security concerns. Once this happens, browsers no longer allow methods like canvas.toDataURL() - or any other method which provides access to read the pixels.

Firefox 3.0 (Gecko 1.9.0) and Safari/Chrome (Webkit) consider as "external resource" any image element which uses a data URL as a value of the src attribute. This pretty much means that images with data URLs cannot be directly edited and saved by PaintWeb, unless workarounds are used for these browsers. The work around consists of taking the image data URL, POSTing it using an XMLHttpRequest to a server-side script which saves the image file as a binary (base64-decoded), and returns the URL pointing to the file on the server. Only after that PaintWeb can use the image element. Luckily Firefox 3.5 (Gecko 1.9.1) fixes this issue.

Results: Moodle 1.9 integration

I have published a Git repository which holds my work with Moodle 1.9, the mdl19-paintweb and mdl19-tinymce3 branches over at repo.or.cz. The mdl19-tinymce3 branch builds upon the TinyMCE 3 patches published by Martin Langhoff in october 2008. I took his work and updated it to the latest Moodle 1.9 stable branch, updated TinyMCE 3 to the latest version and made some additional fixes needed for PaintWeb. The commit change logs provide detailed information on what I did.

The PaintWeb integration code from the mdl19-paintweb branch starts from the mdl19-tinymce3 branch.

What I did:

• included all the PaintWeb code base inside lib/paintweb.

• added a new lang/en_utf8/paintweb.php file which holds all the PaintWeb language strings, in Moodle format (PHP).

• added my TinyMCE plugin at lib/editor/tinymce/jscripts/tiny_mce/plugins/paintweb.

• updated the lang/en_utf8/tinymce.php file to include the new language strings from my plugin for TinyMCE.

• patched the lib/editor/tinymce.js.php file. This file initializes TinyMCE in all pages. My patch adds my PaintWeb plugin for TinyMCE to the list of loaded plugins. PaintWeb is only loaded when $CFG->tinymcePaintWeb is set to true.

• patched the theme/standard/styles_layout.css file to not affect the rendering of the PaintWeb user interface (minor fix).

More details below.

Localization

For localization integration I have three scripts in lib/paintweb/ext/moodle:

• gen_moodlelang.php generates the Moodle language files (PHP) from PaintWeb language files (JSON). This updates lang/*/paintweb.php - depending on the JSON language files included in the PaintWeb packages.

• gen_paintweblang.php generates the PaintWeb language files (JSON) from Moodle language files (PHP). This can be used when a translator updates some paintweb.php language file to regenerate the PaintWeb JSON file.

• lang.json.php is used by PaintWeb inside Moodle when it loads. This PHP script is similar to that of TinyMCE. It outputs a JSON language file, dynamically generated using the Moodle API, with the configured language, from the Moodle PHP language files.

All in all, the localization integration seems fine for me. I am only bothered that each time lang.json.php loads, it generates the JSON dynamically, which uses server-side resources for something quite static (language files seldom change). It does have HTTP caching headers and that certainly helps.

PaintWeb inside Moodle

PaintWeb in Moodle uses its own configuration file: lib/paintweb/ext/moodle/config.json. This tells the paint tool to load a Moodle extension script I wrote. The Moodle extension for PaintWeb hides the textarea icons added by Moodle (those two icons below the textarea), when PaintWeb is shown, and it implements the image saving operation.

Image load and save

The image save operation is implemented in two ways:

• images are saved using data URLs. This means that instead of using files on the server, the whole image is saved inside the HTML generated by TinyMCE.

• images are saved using files inside the Moodle data directory, without using any data URLs (only transiently).

To change between the two file save methods, you only need to edit lib/paintweb/ext/moodle/config.json (search for moodleSaveMethod). Each save method has its own advantages and disadvantages presented below.

Martin Langhoff, my mentor, has posted a relevant thread about storing images as attachements to text fields. It's worth reading!

Data URLs

Saving the images using data URLs has the important advantage of being very easy to implement: just install the PaintWeb plugin inside TinyMCE. This requires no additional file security measures, storage, etc.

The disadvantages are:

• Data URLs do not work in IE 6 / 7 and thus require ugly work-arounds.

• Loading PaintWeb to edit an image with a data URL always requires that the image data URL is sent to the server, saved on disk temporarily, and served back as a binary. As explained above, this is required only for Firefox 3.0 (Gecko 1.9.0) and Safari/Chrome (Webkit) due to security constrains. The script used for saving the image on disk is lib/paintweb/ext/moodle/imagesave.php. The file saved inside the Moodle data dir. The script which serves the images from the Moodle data dir is lib/paintweb/ext/moodle/imageview.php.

• Vanilla TinyMCE 3 builds mangle/corrupt data URLs, so they need to be patched. The patch is trivial and we will most likely get it into official builds really soon.

• Users with images disabled still load the data URLs.

• Page size increases from a few KB to almost MB or several megabytes. The small image I have in my TinyMCE demo uses 800kb for the data URL. I imagine usage scenarios from kids who have digital cameras which save much bigger images (like over 1280x1024), and they want to edit these images in Moodle.

• Images do not cache at all, affecting the perceived performance of Moodle, and it especially affects users of slow connections (OLPC XO wireless is typically slow).

• I presume images with data URLs use more memory.

• Moodle server-side scripts and the database structure does not allow textarea content to hold so much raw data. The default image included in my TinyMCE does not fit in a typical Moodle textarea (only about quarter of it). For proper support of images with data URLs Moodle needs to be patched to allow much bigger textarea content (at least 5-10 MB).

• Images with data URLs require about 37% more storage (due to the use of base64 encoding).

• TinyMCE does some amount of raw HTML processing (with regex at times) which is slowed down quite much when images with data URLs are used. On my Athlon XP 1800+ I usually get the "a script is running slow, do you want to stop script execution" message from Firefox. Similarly, I think server-side HTML filters in Moodle are slowed down when image data URLs are used.

File storage

When PaintWeb is configured to save images using the Moodle file storage, any save operation consists of retrieving the image from the browser, in the form of a data URL: canvas.toDataURL(). The value is POSTed using an XMLHttpRequest to the server-side script lib/paintweb/ext/moodle/imagesave.php which saves the image inside the Moodle data dir configured by the value of $CFG->paintwebImagesFolder.

If the image being edited by the user comes from a course then the image file is directly updated and no new file is created. If the security checks fail, then a new image is created.

When new images are created their file names represent the unique sha1 of their content.

The client-side scripts typically expect that the imagesave.php script returns the new image URL, when an image is saved. Since the images are saved in the Moodle data dir, a new server-side script was required which is able to serve the desired image. This script is imageview.php.

Advantages: the current file storage does not require a rewrite of any database structure. Textarea content always points to URL on the Moodle server, no data URLs are used. Image saving and loading is quick, and the image files are cached by the browser - the user experience is very good.

Disadvantages: there's a concern related to security. Who edits what? Who views what? Currently only course images are edited in-place, otherwise all images are considered "new".

It should be noted that the use of hashes for the image file names is for security purposes. Nobody can overwrite any of the existing images, since their file name is a unique hash of the content. Additionally, nobody can really guess the address of an image of another user - unless, obviously, you are given the address (and the hash) to it. Thus you are also given the right to view the image.

Another important concern is that the Moodle data directory ends-up having far too many image files which need to be deleted, because they are unused. The matter of tracking which image is used/unused is not trivial. We are currently trying to determine the best approach to solving these issues. Any ideas are welcome.

Then there's the problem of backing-up and restoring courses...

Conclusions

Personally I prefer that PaintWeb uses files to save the images, since it's faster and cleaner. User interaction is also much better.

I would like Moodle developers to check the imagesave.php script and provide their feedback and suggestions for improvements.

Further integration work can be done for Moodleforms and I could also create a script with some API to allow developers to include the paint tool stand-alone in any page they want. This depends on what Martin Langhoff, my mentor, decides.

Results: Moodle 2.0 integration

I have published a Git repository which holds my work with Moodle 2.0 (CVS HEAD), the mdl20-paintweb branch over at repo.or.cz.

The PaintWeb integration code from the mdl20-paintweb branch starts from the Moodle 2.0 CVS HEAD branch.

What I did:

• included all the PaintWeb code base inside lib/paintweb.

• added a new lang/en_utf8/paintweb.php file which holds all the PaintWeb language strings, in Moodle format (PHP).

• added my TinyMCE plugin at lib/editor/tinymce/plugins/paintweb.

• updated the lang/en_utf8/editor_tinymce.php file to include the new language strings from my plugin for TinyMCE.

• patched the lib/editor/tinymce/lib.php file. This file initializes TinyMCE in all pages. My patch adds my PaintWeb plugin for TinyMCE to the list of loaded plugins. PaintWeb is only loaded when $options['maxfiles'] is not 0. Additionally, developers can disable PaintWeb by setting $options['plugin/paintweb'] to false.

• patched the theme/standard/styles_layout.css file to not affect the rendering of the PaintWeb user interface (minor fix).

More details below.

Localization

For localization integration I have the same scripts, as for the Moodle 1.9 integration. See the details about the Moodle 1.9 localization integration.

PaintWeb inside Moodle

PaintWeb in Moodle uses its own configuration file: lib/paintweb/ext/moodle/config.json. This tells the paint tool to load a Moodle extension script I wrote. The Moodle extension for PaintWeb implements the image saving operation.

You can read more about how PaintWeb glues into Moodle.

Image load and save

The image save operation is implemented using the new File API. The idea is that textareas have draft file areas which holds files attached to them. The lib/paintweb/ext/moodle/imagesave20.php script saves the image in the draft file area of the textarea where the user edits the article. The client-side PaintWeb scripts determine the context ID and the draft item ID associated to the textarea. When the image save action is invoked, the script receives them, and uses them to save the image. The URL of the draft file is generated by the Moodle File API, and returned to the client-side code, which updates the image URL.

Moodle should handle the garbage collection (automatic deletion of obsolete files). Once the user submits the form containing the textarea, all the image URLs are updated to point to a permanent file area - they are moved out of the temporary/draft file area. Again, this does no longer depends on PaintWeb.

Conclusions

I am very happy to see the numerous improvements made for Moodle 2, especially the new File API. Now PaintWeb saves images properly inside Moodle. These new files are backed-up, restored, cached and nicely usable in general.

A further integration point could be Moodleforms, but I am not sure. Heard of plans for rewrites.