Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

JS Framework Specification: Difference between revisions

From MoodleDocs
m (→‎JQuery (Recommended): Grammar edits to match style of list in previous section (No Frameworks))
(35 intermediate revisions by 4 users not shown)
Line 1: Line 1:
{{Infobox Project
{{Infobox Project
|name = JS Framework Specification
|name = JS Framework Specification
|state = Planning
|state = Implemented
|tracker = MDL-48392
|tracker = MDL-49045
|discussion = https://moodle.org/mod/forum/discuss.php?d=268190
|discussion = https://moodle.org/mod/forum/discuss.php?d=268190
|assignee = [[User:Damyon Wiese|Damyon Wiese]]
|assignee = [[User:Damyon Wiese|Damyon Wiese]]
Line 28: Line 28:
=== Summary of options for JS library ===
=== Summary of options for JS library ===
==== No framework ====
==== No framework ====
pros:
Pros:
* less overhead on each page (maybe)
* Less overhead on each page (maybe)
* Won't find ourself in the same positino in future.
* Won't find ourselves in the same position in future.


cons:
Cons:
* we need to re-implement the cross browser support (drag and drop, ajax ...),
* We need to re-implement the cross browser support (drag and drop, ajax ...),
* we are supposed to be building an LMS and tools for teachers, not JS libraries
* We are supposed to be building an LMS and tools for teachers, not JS libraries


==== JQuery ====
==== JQuery (Recommended) ====
pros:
Pros:
* Most popular choice,
* Most popular choice
* lots of developers with experience,
* Lots of developers with experience
* lots of other libraries rely on jquery,
* Lots of other libraries rely on jquery
* well supported, long life time, "60.4% of web sites use jQuery",
* Well supported, long life time, "60.4% of web sites use jQuery"
* fast,
* Fast
* many corporate supporters,
* Many corporate supporters
* clearly the most voted for option in the discussion,
* Clearly the most voted for option in the discussion
* backing from deque (accessibility),
* Backing from deque (accessibility)
* not too different to YUI
* Not too different to YUI
* [https://github.com/search?q=%24PAGE-%3Erequires-%3Ejquery&ref=searchresults&type=Code already used by add-ons ]
* [https://github.com/search?q=%24PAGE-%3Erequires-%3Ejquery&ref=searchresults&type=Code already used by add-ons ]
* already bundled with moodle
* Already bundled with moodle


cons:
Cons:
* Is it too much?,
* Is it too much?
* quality of plugins,
* Quality of plugins


Note: This does not include JQuery UI. We may include that eventually, but probably not AS-IS. Moodle has very strict requirements for accessibility that go beyond what JQuery UI provides. It is possible to use JQuery UI in an accessible way, but this would need a moodle wrapper to make sure we do it consistently (e.g. http://www.deque.com/blog/accessible-jquery-ui-datepicker/ ). Also we want to get to a point where the JS UI look consistent to the php generated UI elements. This could be by making themers use http://api.jqueryui.com/theming/css-framework/ to mirror their styles, or by modifying the widgets to pull their HTML from templates, or by building new widgets from the ground up that do what we want.
Note: This does not include JQuery UI. We may include that eventually, but probably not AS-IS. Moodle has very strict requirements for accessibility that go beyond what JQuery UI provides. It is possible to use JQuery UI in an accessible way, but this would need a moodle wrapper to make sure we do it consistently (e.g. http://www.deque.com/blog/accessible-jquery-ui-datepicker/ ). Also we want to get to a point where the JS UI look consistent to the php generated UI elements. This could be by making themers use http://api.jqueryui.com/theming/css-framework/ to mirror their styles, or by modifying the widgets to pull their HTML from templates, or by building new widgets from the ground up that do what we want.
Line 81: Line 81:


Note: Just because we have it now in YUI does not mean we need to have it from day 1 of the new framework - YUI will exist for a while and we  choose the best extra library available for non-standard things (like 2d drawing, accessible widgets).
Note: Just because we have it now in YUI does not mean we need to have it from day 1 of the new framework - YUI will exist for a while and we  choose the best extra library available for non-standard things (like 2d drawing, accessible widgets).
=== Summary of options for Accessible Javascript Widgets ===
==== Roll our own ====
* Yuk!
* Aria recommends against this: http://www.w3.org/TR/2009/WD-wai-aria-practices-20090224/#reuse_comp_lib
==== AccDC http://whatsock.com/ ====
Awesome project by one developer with a deep understanding of accessibility to create a widget library that implements accessible widgets first and expects you to make them look pretty later.
Pros: Accessibility definitely comes first with this library
Cons: Hurt by lack of maturity - no unit tests, no information on who is using this library, lacking some developer focused docs, lacking a real community
==== WET http://wet-boew.github.io/  ====
More of a complete framework than a single accessible widget library - and as such it is hard to recommend this. There would be no way to provide backwards compatibility and we would be locked into the selected combination of libraries etc.
==== Assets http://assets.cms.gov  ====
More of a complete framework than a single accessible widget library - and as such it is hard to recommend this. There would be no way to provide backwards compatibility and we would be locked into the selected combination of libraries etc. Also docs note various known and un-fixed accessibility problems with various widgets.
==== JQTest ====
Name does not inspire confidence. It's a set of enhancements to JQuery UI to make those components accessibles. Seems to be abandonware.
==== JQuery UI (Recommended) ====
Pros: Huge community, stable releases, good UI for sighted users
Cons: Accessibility reviews have been done on older versions and found numerous bugs. I couldn't find an up-to date review - but from reviewing the demos on their site there are accessibility issues remaining (e.g. some controls are not keyboard accessible). We would have to wrap the JQuery UI components in order to ensure they are used in a way that guarantees accessibility.


=== Summary of options for loading modules ===
=== Summary of options for loading modules ===
Line 93: Line 117:
note: it has ways to load ES6 modules (after transpiling them http://yuilibrary.com/yui/docs/yui/es6-modules.html)
note: it has ways to load ES6 modules (after transpiling them http://yuilibrary.com/yui/docs/yui/es6-modules.html)


==== RequireJS ====
==== RequireJS (Recommended) ====
implements: modules, dependency resolution, build tools (minification, linting) (requires node.js)
implements: modules (AMD), dependency resolution, build tools (minification, linting) (requires node.js)
cons: ES6 is a newer standard combining RequireJS and CommonJS syntax
 
pros: Huge user base, most libraries available now ship as AMD compatible modules. Lightweight, simple support for JQuery, means we are doing things like the rest of the world.
 
cons: We may want to switch to ES6 in a few years


==== ECMAScript 6 modules ====
==== ECMAScript 6 modules ====
Line 102: Line 129:
pros: best forward looking option (standards based)
pros: best forward looking option (standards based)


cons: new standard - no browser support yet - but there is a polyfill (https://github.com/google/traceur-compiler)
cons: new standard - no browser support yet - but there is a polyfill (https://github.com/google/traceur-compiler). After testing, the polyfill is slow and not 100% compatible with the draft spec. No existing libraries use this format - most of them use a pattern that allows them to co-exist as AMD, CommonJS or standalone libraries (Mustache, JQuery UI etc).


=== Summary of options for build tools ===
=== Summary of options for build tools ===
Line 112: Line 139:
cons: doesn't support ES6 modules except to convert them to YUI modules
cons: doesn't support ES6 modules except to convert them to YUI modules


==== Use a PHP library (not minify - maybe jshrink) ====
==== Use a PHP library (not minify - maybe jshrink) (Recommended) ====
pros: Php library - we can automate the minification and tie it to purge caches, don't have to check JS has been compiled, don't have to include build products in git
pros: Php library - we can automate the minification and tie it to purge caches, don't have to check JS has been compiled, don't have to include build products in git


cons: minify is old and deprecated need to replace it with something better, linting would need a separate tool (maybe part of code checker or cibot)
cons: minify is old and deprecated need to replace it with something better, linting would need a separate tool (maybe part of code checker or cibot or just run jshint manually)


==== Grunt ====
==== Grunt ====
pros: used by a lot of developers, can do more (unit tests), compile less, could run shifter too - so would be a combined process covering old/new
pros: used by a lot of developers, can do more (unit tests), compile less, could run shifter too - so would be a combined process covering old/new, Forced linting step exposes the developer to a lot of jshint warnings while they are developing. Uglifier 2 is the "standard" js minifier (ie we are using the best tools).


cons: another build tool for new developers to support (barriers)
cons: another build tool for new developers to support (barriers), Requires a set of files in the moodle root (Gruntfile.js, package.json) - and each moodle instance will need to install node packages with npm into a node_modules folder in their moodle root.


=== Prototypes ===
=== Prototypes ===
Line 128: Line 155:
With JQuery - http://requirejs.org/docs/jquery.html
With JQuery - http://requirejs.org/docs/jquery.html


Git branch: https://github.com/damyon/moodle/tree/REQUIREJS
Git branch: https://github.com/damyon/moodle/tree/MUSTACHE


This branch uses vanilla RequireJS with no build step. Modules are minified with our current JS minification library on the fly for production only.  
This branch uses vanilla RequireJS with no build step. Modules are minified with our current JS minification library on the fly for production only and all modules are concatenated to a single JS file (one request). When modules are concatenated the module name is injected by the server (this is what r.js would do).  


The module format is AMD and config for JQuery is built in.
The module format is AMD and config for JQuery is built in.
Line 158: Line 185:


How to use the above module:
How to use the above module:
<code javascript>
require(['mod_assign/test'], function(test) {
require(['mod_assign/test'], function(test) {
     console.log(test.get());
     console.log(test.get());
});
});
See the jquery dependency was automatically pulled in - and the component name mapped to the modules area for the component.
</code>
 
The jquery dependency was automatically pulled in - and the component name mapped to the modules area for the component.
Plugin authors need to use r.js manually to build bundles using this approach (A bundle is a collection of modules that is meant to be loaded together).
 
Need to test creating a bundle (either with a build system, or a dynamic one).


==== ES6 Modules (with polyfill) ====
==== ES6 Modules (with polyfill) ====
Line 174: Line 199:
After testing this - it seems a bit of a dodgy solution because of the way the polyfill works. It either requires "transpiles" (a build step) the ES6 module to ES5, or requires the entire traceur library to do the transpiliing in the browser (slow).
After testing this - it seems a bit of a dodgy solution because of the way the polyfill works. It either requires "transpiles" (a build step) the ES6 module to ES5, or requires the entire traceur library to do the transpiliing in the browser (slow).


Since traceur can transpile ES6 to AMD modules, maybe we should test an automated version of this so we write in ES6, but transpile to AMD and load with RequireJS. This gives us a future proof format that works now with a stable loader.
Since traceur can transpile ES6 to AMD modules, maybe we should test an automated version of this so we write in ES6, but transpile to AMD and load with RequireJS. This gives us a future proof format that works now with a stable loader. Also - try 6in5.
 
Note - in researching this - I did not find any current libraries using the ES6 format with a transpile step in their build. This would introduce a difference between our JS and the JS we find in the real world, which is not a good thing. I'm not recommending this approach until it becomes more widespread.


==== Grunt ====
==== Grunt ====
There is some code on https://tracker.moodle.org/browse/MDL-40156 as a starting point.
Dave has been working on grunt support and has a branch where you can run grunt in any directory, and it will either run shifter with the correct options, or manually run the required uglifier steps the minify AMD javascript.
 
There is a demo of GruntJS's capabilities which can be found here: https://github.com/gurgus/gruntdemo
 
This shows off the capability of a developer only having to run one command (`grunt`) from within a YUI or AMD directory and have the necessary build steps applied to their code.
 
==== Automated minification ====
We have a prototype of this approach here:
https://github.com/damyon/moodle/tree/MUSTACHE
 
(It also has mustache support because I have been working on both).  
 
How this branch works:
 
When you request a module from requirejs like this:
<code javascript>
require(['core/alert'], function(Alert) {
    var a = new Alert('I have a message', 'to say');
});
</code>
 
The requirejs config tells requirejs that the "lib/requirejs.php" script is the base of all urls.
 
The "lib/requirejs.php" script will return the requested module, correctly minified and with the correct module name inserted - but also all other AMD modules from all plugins and subsystem dirs in Moodle. This is similar to what the "r.js" optimiser tool does, but we
need it to be done at request time because you can add/remove plugins from Moodle.
 
Question: So - you get one mega minified JS file with all known modules as your response - won't that be huge?
Answer: No - not too huge - and the performance benefit of reducing JS requests is worth it. And it will be cached. The sum of all of our minified JS in all of our custom written yui modules is 800k. YUI itself is 2.8MB which is why it doesn't survive without a combo loader. Some google engineers in this presentation: https://www.youtube.com/watch?v=mGENRKrdoGY suggest that the point you want to start splitting your JS up is aroung 1MB - we have a lot of JS to write in the new format before we get to that point. Finally - our current JS uses handlebar templates defined in the Javascript - we have a new way of loading templates via AJAX so our new code will be a bit smaller.
 
Pros: No build steps


==== Automated minification with JShrink ====
Cons: No lint step (could be done in codechecker, or by manually running jshint).
Need to test failures linting, speed
(Note - if we decide we need other build steps like transpiling ES6 to AMD - we will need to use grunt anyway, so no need to explore this).

Revision as of 08:43, 21 December 2016

JS Framework Specification
Project state Implemented
Tracker issue MDL-49045
Discussion https://moodle.org/mod/forum/discuss.php?d=268190
Assignee Damyon Wiese


Background

Moodle has used YUI as it's javascript framework (since 2006). We have built in support for using their combo loaders, javascript build chain, module support, debugging and many other nice features. We use YUI to provide a standard API between different browser features and for the many nice things that are built in like dialogs, ajax requests, events, classes and inheritance.

Earlier this year (2014) Yahoo announced:

we have made the difficult decision to immediately stop all new development on YUI in order to focus our efforts on this new technology landscape. This means that, going forward, new YUI releases will likely be few and far between, and will only contain targeted fixes that are absolutely critical to Yahoo properties.

This started a (very constructive) forum discussion on the future of JS in Moodle (https://moodle.org/mod/forum/discuss.php?d=268190).

And following that discussion we made a policy decision (https://tracker.moodle.org/browse/MDL-47036) to create a "first prototype/example as described by Dan above with JQuery, RequireJS and grunt".

This document is meant to clearly document the features / characteristics of this prototype/example and compare it to our existing YUI integration. It should discuss how the 2 solutions can co-exist during a transition period.

Summary of forum discussion

The forum discussion has had many contributors and is full of valuable comments. Please read it: https://moodle.org/mod/forum/discuss.php?d=268190.

Here is a much shorter summary of most of the things that were discussed:

Summary of options for JS library

No framework

Pros:

  • Less overhead on each page (maybe)
  • Won't find ourselves in the same position in future.

Cons:

  • We need to re-implement the cross browser support (drag and drop, ajax ...),
  • We are supposed to be building an LMS and tools for teachers, not JS libraries

JQuery (Recommended)

Pros:

  • Most popular choice
  • Lots of developers with experience
  • Lots of other libraries rely on jquery
  • Well supported, long life time, "60.4% of web sites use jQuery"
  • Fast
  • Many corporate supporters
  • Clearly the most voted for option in the discussion
  • Backing from deque (accessibility)
  • Not too different to YUI
  • already used by add-ons
  • Already bundled with moodle

Cons:

  • Is it too much?
  • Quality of plugins

Note: This does not include JQuery UI. We may include that eventually, but probably not AS-IS. Moodle has very strict requirements for accessibility that go beyond what JQuery UI provides. It is possible to use JQuery UI in an accessible way, but this would need a moodle wrapper to make sure we do it consistently (e.g. http://www.deque.com/blog/accessible-jquery-ui-datepicker/ ). Also we want to get to a point where the JS UI look consistent to the php generated UI elements. This could be by making themers use http://api.jqueryui.com/theming/css-framework/ to mirror their styles, or by modifying the widgets to pull their HTML from templates, or by building new widgets from the ground up that do what we want.

Use a set of polyfills and use ES6 syntax for everything

pros:

  • nice "elegant" solution

cons:

  • testing requirements go up greatly,
  • we will probably end up rolling our own JS frame work piece by piece,
  • plugins devs would all include jquery anyway (bootstrap).

Angular

pros:

  • trendy,
  • clean MVC design

cons:

  • requires backend rewrite

Ember

cons:

  • requires backend rewrite

Attributes we want in a JS framework

long term support cycles, standardize things that are not standard across browsers (dom manipulation, positioning, ajax, events, drag and drop?, 2d drawing?, transitions, accessible widgets?), sandboxing, dependency loading, able to (easily) load third party modules, performance

Note: Just because we have it now in YUI does not mean we need to have it from day 1 of the new framework - YUI will exist for a while and we choose the best extra library available for non-standard things (like 2d drawing, accessible widgets).

Summary of options for Accessible Javascript Widgets

Roll our own

AccDC http://whatsock.com/

Awesome project by one developer with a deep understanding of accessibility to create a widget library that implements accessible widgets first and expects you to make them look pretty later.

Pros: Accessibility definitely comes first with this library Cons: Hurt by lack of maturity - no unit tests, no information on who is using this library, lacking some developer focused docs, lacking a real community

WET http://wet-boew.github.io/

More of a complete framework than a single accessible widget library - and as such it is hard to recommend this. There would be no way to provide backwards compatibility and we would be locked into the selected combination of libraries etc.

Assets http://assets.cms.gov

More of a complete framework than a single accessible widget library - and as such it is hard to recommend this. There would be no way to provide backwards compatibility and we would be locked into the selected combination of libraries etc. Also docs note various known and un-fixed accessibility problems with various widgets.

JQTest

Name does not inspire confidence. It's a set of enhancements to JQuery UI to make those components accessibles. Seems to be abandonware.

JQuery UI (Recommended)

Pros: Huge community, stable releases, good UI for sighted users Cons: Accessibility reviews have been done on older versions and found numerous bugs. I couldn't find an up-to date review - but from reviewing the demos on their site there are accessibility issues remaining (e.g. some controls are not keyboard accessible). We would have to wrap the JQuery UI components in order to ensure they are used in a way that guarantees accessibility.

Summary of options for loading modules

YUI loader

implements modules, sandboxing namespaces, dependency resolution, combo loading, build tools (minification, linting), classes, inheritance, small set of supported widgets, events

pros: we already include the library

cons: does not look to external devs like we have shifted from yui, requires YUI syntax to load modules (e.g. after the page has loaded)

note: it has ways to load ES6 modules (after transpiling them http://yuilibrary.com/yui/docs/yui/es6-modules.html)

RequireJS (Recommended)

implements: modules (AMD), dependency resolution, build tools (minification, linting) (requires node.js)

pros: Huge user base, most libraries available now ship as AMD compatible modules. Lightweight, simple support for JQuery, means we are doing things like the rest of the world.

cons: We may want to switch to ES6 in a few years

ECMAScript 6 modules

implements: modules, sandboxing, dependency resolution

pros: best forward looking option (standards based)

cons: new standard - no browser support yet - but there is a polyfill (https://github.com/google/traceur-compiler). After testing, the polyfill is slow and not 100% compatible with the draft spec. No existing libraries use this format - most of them use a pattern that allows them to co-exist as AMD, CommonJS or standalone libraries (Mustache, JQuery UI etc).

Summary of options for build tools

What is a build tool - think minification, linting, module packaging, generate documentation.

Shifter

pros: We use it now

cons: doesn't support ES6 modules except to convert them to YUI modules

Use a PHP library (not minify - maybe jshrink) (Recommended)

pros: Php library - we can automate the minification and tie it to purge caches, don't have to check JS has been compiled, don't have to include build products in git

cons: minify is old and deprecated need to replace it with something better, linting would need a separate tool (maybe part of code checker or cibot or just run jshint manually)

Grunt

pros: used by a lot of developers, can do more (unit tests), compile less, could run shifter too - so would be a combined process covering old/new, Forced linting step exposes the developer to a lot of jshint warnings while they are developing. Uglifier 2 is the "standard" js minifier (ie we are using the best tools).

cons: another build tool for new developers to support (barriers), Requires a set of files in the moodle root (Gruntfile.js, package.json) - and each moodle instance will need to install node packages with npm into a node_modules folder in their moodle root.

Prototypes

Based on the above summaries (and the lengthy discussion) - there are some worthwhile candidates we should test out:

RequireJS integration

With JQuery - http://requirejs.org/docs/jquery.html

Git branch: https://github.com/damyon/moodle/tree/MUSTACHE

This branch uses vanilla RequireJS with no build step. Modules are minified with our current JS minification library on the fly for production only and all modules are concatenated to a single JS file (one request). When modules are concatenated the module name is injected by the server (this is what r.js would do).

The module format is AMD and config for JQuery is built in.

Example of loading (sandboxed) jquery and calling it in a module.js file:

require(['jquery'], function( $ ) {

   $('h1').hide('slow');

});

Example of creating a module in a plugin:

Create file mod/assign/amd/test.js: // This module depends on jquery define(['jquery'], function($) {

   return {
       // This module has one get function which returns the text from the heading node in the page.
       get: function() {
           return $('h1').text();
       }
   }

});

How to use the above module: require(['mod_assign/test'], function(test) {

   console.log(test.get());

}); The jquery dependency was automatically pulled in - and the component name mapped to the modules area for the component.

ES6 Modules (with polyfill)

With JQuery loaded as ES6 module (maybe transpiled)

Need to test loading speed - concurrency etc

After testing this - it seems a bit of a dodgy solution because of the way the polyfill works. It either requires "transpiles" (a build step) the ES6 module to ES5, or requires the entire traceur library to do the transpiliing in the browser (slow).

Since traceur can transpile ES6 to AMD modules, maybe we should test an automated version of this so we write in ES6, but transpile to AMD and load with RequireJS. This gives us a future proof format that works now with a stable loader. Also - try 6in5.

Note - in researching this - I did not find any current libraries using the ES6 format with a transpile step in their build. This would introduce a difference between our JS and the JS we find in the real world, which is not a good thing. I'm not recommending this approach until it becomes more widespread.

Grunt

Dave has been working on grunt support and has a branch where you can run grunt in any directory, and it will either run shifter with the correct options, or manually run the required uglifier steps the minify AMD javascript.

There is a demo of GruntJS's capabilities which can be found here: https://github.com/gurgus/gruntdemo

This shows off the capability of a developer only having to run one command (`grunt`) from within a YUI or AMD directory and have the necessary build steps applied to their code.

Automated minification

We have a prototype of this approach here: https://github.com/damyon/moodle/tree/MUSTACHE

(It also has mustache support because I have been working on both).

How this branch works:

When you request a module from requirejs like this: require(['core/alert'], function(Alert) {

   var a = new Alert('I have a message', 'to say');

});

The requirejs config tells requirejs that the "lib/requirejs.php" script is the base of all urls.

The "lib/requirejs.php" script will return the requested module, correctly minified and with the correct module name inserted - but also all other AMD modules from all plugins and subsystem dirs in Moodle. This is similar to what the "r.js" optimiser tool does, but we need it to be done at request time because you can add/remove plugins from Moodle.

Question: So - you get one mega minified JS file with all known modules as your response - won't that be huge? Answer: No - not too huge - and the performance benefit of reducing JS requests is worth it. And it will be cached. The sum of all of our minified JS in all of our custom written yui modules is 800k. YUI itself is 2.8MB which is why it doesn't survive without a combo loader. Some google engineers in this presentation: https://www.youtube.com/watch?v=mGENRKrdoGY suggest that the point you want to start splitting your JS up is aroung 1MB - we have a lot of JS to write in the new format before we get to that point. Finally - our current JS uses handlebar templates defined in the Javascript - we have a new way of loading templates via AJAX so our new code will be a bit smaller.

Pros: No build steps

Cons: No lint step (could be done in codechecker, or by manually running jshint).