Render library specification

Jump to: navigation, search
Renderer consistency
Project state Specification
Tracker issue https://tracker.moodle.org/browse/MDL-45770
Discussion https://moodle.org/mod/forum/discuss.php?d=261202
Assignee Damyon, Sam

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Project Goals

  • Add an element library to Moodle (A page in Moodle listing all of the renderables and how they look in the current theme)
  • Add a complete set of core renderables that should be capable of completely rendering every new user interface
  • Update the Output API documentation
  • Create a output guide documentation including best practices.

Benefits

  • Make it easier to create user interfaces.
  • Make it easier to style the user interface.
  • Make it easier for themers to customise the user interface.
  • Better facilitate frontend frameworks in Themes.
  • Better facilitate usable and accessible design.
  • Give Moodle a more consistent look and feel.
  • Improve site performance by reducing the CSS footprint.

Understanding renderers in Moodle

Renderers have been around since Moodle 2.0 and are now the requirement when writing any code that produces output. Renderers come in many shapes and forms depending upon the developers take on how output should be produced.

When renderables were introduced - the API was designed to make it easy for existing code to be updated to make use of renderers and renderables. A renderable is an interface with no methods, so any existing class could be made to implement renderable. This means that the renderable class can be a mixture of data and methods that perform logic associated with the module. This also means that the renderer can call those methods on the renderable class - effectively using the API of the module. The problem with this - is that it complicates the renderer and makes it harder to override in a theme - the themer requires knowledge about the API for the module. Some renderables in core are built as separate classes - only used for the purpose of supplying data to the renderer - here there is a clear separation of the logic - and the properties of the renderable are made up of only simple types that can be checked for in the renderer - e.g. access checks are done in advance and the result is put into a boolean in the renderable. The current renderers / renderables in Moodle are a mixture of these styles.

Because there is no mandated organisation, nor has there ever been, there is immense variation in what constitutes content to a renderer.

One of the upsides to our current render system is that you are not limited to having a single renderer for a component, you can make use of language capabilities and the output framework and create layers of renderers to cut back code duplication and to aid in preparing a base for consistent structuring and output.

Into the equation we add targets and subtypes. Targets allow us to create both standard renderers and renderers specific to the needs of the output method. Standard being HTML, without output methods including CLI, AJAX, HTML email and plain text email.

People have never being constrained as to how they implement a renderer. We've being happy enough to simply guide them towards using renderers because of the benefits they provide and while there are numerous benefits there are two fundamental benefits:

A change in thinking 
It requires developers to think about their code organisation. Renderers were introduced in a time before Object Orientation was used in Moodle, and logic and output were tightly intermingled in most code. Renderers were about separating logic and design and while what was created was not always perfect it was always an improvement.
An extensible approach to output 
The render system allows for the overriding of renderers to occur. Theme designers can overwrite any renderer in their theme. The more that renderers that get implement the more of Moodle you can customise.

Understanding themes

Themes are without a doubt the most widely customised plugin provided by Moodle. But we are currently struggling with several problems with renderers and themes as they are being implemented currently.

1. There is no strict relationship between the HTML generated by a renderable, and the accompanying CSS that is required to make it display properly. The HTML is coded in the renderer - and then the CSS is added, either directly to the styles.css for the plugin in an unorganised way - or directly in the CSS in the theme - again in a loosely organised way. There is no systematic way to determine if any line of CSS is still in use - and there is no simple way to find the CSS that requires updating when the HTML is updated.

2. The CSS framework (bootstrap 2, bootstrap 3, zurb, pureio) specific attributes for renderables are currently being hardcoded in the HTML generated by the renderables. This prevents themers from using a different framework without creating an impossible amount of work to overwrite every renderer in Moodle.

So what is the problem

There are several. Lets be blunt about this. This specification is not a single task. It is about improving the state of output in Moodle and there are going to be several steps in achieving this. On a positive note these steps can in many cases be broken down into granular tasks that can be worked on asynchronously. First lets understand some of the major problems:

  • We have no established best practices, nor even documentation on what constitutes a good renderer, or considerations that should be paid mind in planning and designing of code.
  • We have no established style guide, nor pattern library, nor element library, nor anything that could be referenced when someone starts building a new user interface - or creating a new theme.
  • Inconsistency - because of a lack of best practice documentation and examples, there is huge variation in the design of our different modules. Different modules exhibit different problems such as a mixing of logic and output and code duplication.
  • Developers are encouraged to push functionality boundaries in order to take it places observed in other projects or to challenge oneself. While consistency is often talked about it is rarely observed when designing output. We end up with numerous "similar" looking widgets all functioning slightly differently and consequently there is a feeling that this is the way to achieve a new fresh look. (E.g. we have multiple implementations of a "toolbar" - one in Atto and one in EditPDF).
  • Existing renderer implementations vary widely and in many cases contain logic they should not. Access checks, database queries, object manipulations. All of which should not be present as it introduces muddling of concepts and logic that must be duplicated and then maintained for anyone overriding renderers.
  • Themers have a mammoth task at present in applying a consistent style to Moodle, or in choosing to implement a new frontend framework (bootstrap, foundation, pureio etc). Our lack of organisation in the output leaves a lot to be desired. Having the theme responsible for choosing a frontend framework is the right way to handle this, however backend Moodle doesn't support themeing as well as it should be/could be.
  • On top of the above it introduces a need to override several renderers should you want to create a consistent style and because of inconsistent renderer implementations this can easily lead to a maintenance nightmare. E.g. the user_picture renderable is not used in mod_forum - so there is no way to override it there.
  • JavaScript introduces problems of its own. We are doing more and more with JavaScript these days. However there is a heavy dependence on the output that gets produced and the JavaScript enhancing it. In a lot of cases the relationship is ill defined and due to the complexity and structure of the JavaScript overriding the output structure of the rendered component is completely impossible for all but the most experienced developers.
  • Inline with the above we lack good examples and documentation on how to deliver output templates for JavaScript User Interfaces as well as the creation, and manipulation of HTML structures that are enhanced by JavaScript.

Identified issues

There is a general trend in the problems noted, when looking at output we lack organisation, documentation, and consistency across the board. However the actual issues we are focusing on in this spec can be identified as below:

  • Lack of a style guide / element library / pattern library.
    • Leading to inconsistent output and style.
    • Increased workload due to usability and accessibility thinking and testing needing to occur repetitively.
    • Increased workload due to multilang (rtl) support usually coming after the designs of new output.
  • Inconsistent renderer implementations.
    • Lack of best practices.
    • Lacking plus outdated documentation.
    • Logic-full renderers greatly hinder theming.
    • An unnecessarily increased requirement to understand both PHP and Moodle code as each renderer differs and are usually poorly documented.
  • No abstraction from CSS framework.
    • Without organised structure classes are applied in a chaotic fashion leading to the requirement to duplicate core styles.
    • As there is no core styles newly created output widgets need to be styled in all base themes leaving chance of design regressions occurring in predominantly major upgrades but occasionally minor releases as well.
    • Inconsistent output structure ensures CSS quickly balloons as styles need to be both created and copied to several areas.
  • Poor linking between UI components and the JavaScript.
    • Ill-defined link between output and the JavaScript enhancing it.
    • No common technique to template UIs generated by javascript

Manageable Tasks

The following are the major tasks we have identified so far. These tasks can be though of as Epic's, or tasks for which we will create subtasks (if this output specification itself becomes an Epic). Within this section will include notes and implementation ideas on the major tasks we've identified.

Define the elements that make up Moodle

There are several tasks that will go into this, but the first and most important will be deciding upon an organisational methodology. Some way to compartmentalise the Moodle output. At present a reducing granularity system is looking the most appealing. Go and read Atomic Design written by Brad Frost - it describes very well the type of compartmentalisation we are proposing.

Once a system has being chosen the following tasks can be tackled:

  • Identify the layers we require.
  • Define the boundaries and responsibilities of those layers.
  • Identify the initial set of elements, likely the finest granularity first as I imagine starting at the bottom and working up will be the best way to proceed.

Further elements (those relating to Moodle will be defined as part of the element library work).

Element library and style guide

Draft docs for element library tool: Element_Library

Add an admin tool that can show a categorised list of renderable objects for each plugin/and core - filled with test data. The goal of this tool is to:

  • Provide a page for themers to check that their new theme correctly displays all the renderables
  • Provide a page for developers to see all the reusable renderables (in a well defined structure) they can use when building their own pages

The tasks here will be:

  • Spec the element library tool with regard to our chosen element methodology.
  • Create the element library tool.
  • Establish a definition method of elements, and compositions of elements as renderables.
  • Define the core elements from the task above in code somehow and verify the element library tool reveals them.
  • Finally define a comprehensive list of low level widgets we can add to core as renderables. This is a library of components that should be used by other renderers (by compositing them). E.g. list, table, warning, grid. These renderables should be structured into layers, starting from renderables that cannot be broken into smaller parts, up to widgets made of smaller renderables, up to layouts (and possibly up to entire pages). Sources for this list should come from:
    • Existing CSS Frameworks (bootstrap, pureio) (but must be framework agnostic)
    • Existing renderables in Moodle that we see as “perfect” already (maybe there are some)
    • Existing patterns in Moodle that we do over and over but don’t have a renderable for yet

On top of this there is functionality that would be great to add to it:

  • Validate that the renderer/renderable follows best practice (no DB queries, complex types, complex logic)
  • Provide responsive testing immediately within the tool. Adjusting the viewing area on queue to allow both developers and designers to experience how the individual renderables respond in varying presentation medias.
  • Provide language direction testing immediately within the tool. Like the above allow designers and developers to quickly experience a renderable in a different language directions.
  • Renderable linear reporting. So that you can see which other renderables make use of this renderable and which renderables get used by this renderable. This will be hugely helpful in judging impact when themeing.


Thoughts on naming

We will need to define useful names for the different categories of renderables so that we can effectively organise them and provide guidelines for writing each type - the names are a small part of this solution but will possibly be a point for contention. Different frameworks have different names for these things and people will prefer something that sounds like something they are already using.

(draft!) Categories for renderables - taken straight from Atomic design

http://bradfrostweb.com/blog/post/atomic-web-design/

  • Atoms == Tiny reusable elements that cannot be broken down into smaller elements
  • Molecules == Relatively simple combinations of atoms built for reuse
  • Organisms == Organisms are groups of molecules joined together to form a relatively complex, distinct section of an interface.
  • Templates == Layout files in a theme
  • Pages == The actual pages in Moodle

Refs for categorizing patterns

Style guide

This has two real advantages.

  1. First assuming this will be developed initially while defining the elements that will make up Moodle output and will aid us in verifying what we are doing. If it can be easily documented then we are on the right track. If we start encountering to many conditions and or too much variation then we are drifting off track.
  2. When defining new elements we are able to use our style guide to do so and it'll provide us measure of how Moodle output evolves.

Define and document renderer best practices

This will be tricky and no doubt how we decide to define a best practice will have to have regard for the output methodology we choose.

Update the JavaScript documentation

Discuss and agree on a best practice for writing JS that depends on HTML structures in the DOM (e.g. progressive enhancement - or just ids for action listeners) (perhaps a specific renderable for a JS region)

Update renderer documentation

This will of course need to be done after the element library and renderer best practices but should be pretty simple at that point.

  • Read and rate each of the existing docs pages (see related links below)
  • Write new complete docs for everyone (at Output API)
  • Cross link old docs to the new ones

Update theme documentation and theme tutorials

There are likely the most well documented plugin type within Moodle. There are good specification docs, good 2.0 conversion docs and good tutorials on many aspects of theming. Many of these pages will need to be updated to reflect best practices and such after we start ticking off some of the other tasks.

Progressively convert renderers

This is the very last task really - but we will start work on it in parallel with creating the new set of core renderables (to prove the design is usable). There is no deadline for this to be complete - and it will take a long time to convert all the existing renderers (and code not using renderers).

Notes on renderers / renderables

Proposed improvements to the docs on renderers / renderables. First - the docs need an overhaul to more clearly explain how renderers / renderables work for core devs, plugin devs and themers. There needs to be clearer guidelines for the things that can be done in a renderable, and in a renderer.

Goals for renderers - should be devoid of logic for these reasons:

  • Theme overridden renderers do not need to be updated when logic is changed
  • Themers can override renderers without understanding all the logic
  • Renderables can be filled with mock data for display in the element library

A "good renderer" is:

  • Receives some “data” through a renderable
  • Produces some “output”
  • Where “output” is some HTML and any javascript init calls
  • Can use basic PHP functions/loops, like for(), foreach(), count()

A "good renderer" is not:

  • nasty php logic (or even non-nasty php logic)
  • access checks
  • non-trivial function calls
  • using any Moodle functions not related to output
  • calling the database

A “renderable” is:

  • All the data required to generate the output.
  • Properties should be simple data types only, or renderables, or array of those

Evaluations of some existing renderers / renderables

Existing renderer styles - because there are few rules/guidelines for how renderers should be written - there are differing examples of renderers in core. Looking at the 3 of the better maintained modules to see how they are implemented will be useful in evaluating the benefits of any stricter rules for renderers.

Assign

Contains a list of renderables (in renderables.php) and a single renderer. The logic is mostly out of the renderer methods (pre-calculated and added as data to the renderer). There are some cases when renderables could be broken down further - e.g. the expand/collapse on the submission history (mod/assign/yui/src/history/js/history.js). Also the assignment sub-plugins do not use renderers (and they should).

Quiz

Uses a lot of renderers - but not renderables. The renderer methods directly call methods of the business logic to determine what to display. e.g. "attemptobj->get_access_manager(time())->attempt_must_be_in_popup()". This is not terrible for a themer trying to override a renderer method because the calls are mostly simple - but it is a grey area for when this becomes business logic in the renderer (and would e.g. need updating to fix bugs etc). If we want to add these renderer methods to the element library - it would require a way to pass mock business logic classes to the renderer methods (and a way for the generator to return a renderer and a method - instead of only a renderable).

Workshop

Very nice renderers and renderables in workshop. Would only need updating to re-use core renderables instead of using html_writer directly - and adding the existing renderables to the element library.

Badges

Mix of renderer methods and renderables. This is similar to quiz where the renderer is calling lots of methods of the business model in the renderer methods and has the same effect (harder to override, maintain etc). To add these renderer methods to the element library would require the same sort of changes as mentioned in quiz.

Example of forum post renderer using composition

This is not a real example (yet), it's just a demonstration of the type of renderers we should be aiming for (using composition etc). This structure would be created in the render_form_post method of the forum renderer. This method would receive a forum_post renderable containing only data - and build up this structure of smaller renderable components, then return the result of calling render(x) on each of the smaller renderables.

Example forum renderer

Initial thoughts on renderables and what we should provide (Draft)

This list is not intended to be the final list - it is more of a scoping exercise. One of the tasks from this project is to decide on the full set of required renderables. There are several renderables within Moodle that have being included in this table and we will need to look closely at how they fit within our chosen model and whether they are still required.

Name Description Source (links)
action_menu UI component for a drop down edit menu Existing renderable (should be renamed "menu")
action_menu_link UI component for a menu item in an action menu Existing renderable
action_menu_filler UI component for a filler menu item in an action menu Existing renderable
action_menu_link_primary UI component for a primary menu item in an action menu Existing renderable
action_menu_link_secondary UI component for a secondary menu item in an action menu Existing renderable
action_link Link with alt text, and an icon Existing renderable
single_button A form with a single button Existing renderable
confirm A form with a message and cancel/confirm buttons Existing renderable
single_select A form with a single drop down list that submits on change Existing renderable
url_select A navigation element consisting of a single drop down list of urls that navigates on change Existing renderable
doc_link A link to the Moodle docs Existing renderer method
pix_icon A small icon Existing renderable
emoticon_icon A small emoticon Existing renderable
heading_with_help A page heading with a link to help docs Existing renderer method
help_icon A help icon that opens a help popup when clicked Existing renderable
help_icon_scale A help icon that opens a help popup when clicked Existing renderable
user_picture A user profile picture which links to their profile Existing renderable
container A block level element used to surround something. Can have a class to allow specific targeting with CSS. Existing renderable
error_text An error to show to the user. Existing renderable
notification A message for the user Existing renderable
continue_button A message and a button to continue to the next page Existing renderable
paging_bar A list of next previous and specific page links Existing renderable
skip_link_to A link to a section on the page Existing renderable
skip_link_target A target for a matching skip_link_to call Existing renderable
heading A page heading Existing renderable
box A page section with a border Existing renderable
rarrow A right arrow Existing renderable
larrow A left arrow Existing renderable
tabtree A list of tabs Existing renderable
tabobject A single tab panel Existing renderable
toolbar *new A container for toolbar button groups Bootstrap equiv btn-toolbar (needs aria support)
toolbar-group *new A group of toolbar buttons Bootstrap equiv btn-group (needs aria support)
toolbar-button *new A toolbar button Bootstrap equiv btn (needs aria support)
toolbar-menu *new A toolbar button that opens a drop down menu Bootstrap equiv nested btn-groups (needs aria support)
navbar *new The navigation bar that shows at the top of the page Bootstrap equiv navbar - There is an existing renderer method that needs converting into renderables
navbar_item *new Items in the navigation bar that shows at the top of the page Bootstrap equiv list item in a navbar - There is an existing renderer method that needs converting into renderables
navbar_item *new Items in the navigation bar that shows at the top of the page Bootstrap equiv list item in a navbar - There is an existing renderer method that needs converting into renderables
progress_bar *new A bar showing progress of something Bootstrap equiv progress_bar - There is an existing moodle class that needs converting to use renderables
progress_bar *new A bar showing progress of something Bootstrap equiv progress_bar - There is an existing moodle class that needs converting to use renderables
panel *new A container with a heading and borders Bootstrap equiv panel, panel-heading, panel-body, panel-footer
vertical-list *new A vertical list of items Bootstrap equiv - list-group
horizontal-list *new A horizontal list of items Bootstrap equiv - list-inline
Two column list *new Short cut for a list of pairs with the left column fixed-width and right aligned Similar to our current moodle forms
table (and cells headings etc) * new We need to convert tablelib to use renderers to output all table elements
forms (and labels, inputs etc) * new We need to convert formslib to use renderers to output all form elements
grids *new We need to support renderables for a simple responsive grid system Everything has grids...
floats *new We need to support floating elements left and right Bootstrap - pull-left, pull-right
clearfix *new If we have this simple thing as a renderable we can avoid repeating the CSS Bootstrap - clearfix
expandable *new Initially shows only a summary with an expand button to show more details mod_assign
dialog *new Tie into M.core.dialogue so it can be adjusted in a renderer aria
timer *new There is an aria role for this - so we should make sure it is used correctly if required. aria
tree *new Reusable version of the nav tree aria tree, tree-item, tree-grid etc
indent *new (eg threaded forum post)
media block *new large media element side by side with some text http://demo.patternlab.io/
hero *new content section with borders/oversized text etc designed to be a landing point Bootstrap jumbotron

Prototyping

Because there are several ways to implement a renderer/renderable - we have spent some time exploring an example renderer/renderable implementation. What we intended to achieve with these dev branches is an implementation of a "menu" and a "list" renderable in several themes, using several CSS frameworks. The frameworks chosen are: Bootstrap 2 (clean), Bootstrap 3, Zurb Foundation and YUI Pure IO. The prototypes both produce exactly the same output (and only a little effort was made to get nice output) - it's just the implementations we are interested in at this stage. Each prototype branch adds a local plugin with a test page that can be viewed at "/local/output/index.php" and viewed in several themes (clean, output_bs3, output_pureio, output_zurb).

Some different approaches were explored here:

Approach A

This design takes a strong objected oriented view of renderables and introduces some concepts of the HTML DOM to the renderables. There is an abstract class core_ui_element which implements renderable, that all the new renderables inherit from. Each core_ui_element has additional properties including a type, a chain of parent types, a list of attributes to be mixed into the html and a generic list of name/value properties. The type variable can influence which renderer method renders this core_ui_element (this requires a core change). The renderer method can adjust the output based on the chain of parent types of the current node, and the properties of the immediate parent.

The concepts applied in this approach:

Everything is an element 
Everything that forms a part of output will be represented by a class and can be interacted with in a consistent fashion.
All elements are renderables 
All elements and components inherit from a common core_ui_element class that implements the renderable interface.
All elements are also core_ui_elements 
The core_ui_element class allows for common functionality across every element and component. It facilitates ownership of one element to another (building complex elements aka components) as well as adding attributes and properties.
Attributes 
Element objects have attributes, these are of direct relation to HTML attributes, they are managed as a public array and can easily added to and worked with. There is also a helper method on the renderer that turns them into an HTML attribute string that can be put immediately within a tag for ease of use.
Properties 
These are used for non-renderable attributes of the element. For instance when producing a menu with items that are disabled, and where one item is active. Properties get assigned by the parent element allowing for properties to appear with specific relation to the context the element is appearing in. For example a link can be a menu item, when it is a menu item it has "active" as a property. When it is used elsewhere it does not have that property.
Chained render methods 
When render is called on a link that is by itself render_link gets called. When render is called on a link within a menu render_menu_link gets called, and if it doesn't exist render_link gets called. When render is called on a link that is within a menu that is within a dropdown render_dropdown_menu_link gets called, if that doesn't exist then render_menu_link gets called and so on and so forth. Through this chaining you can override the individual elements within a complex component using predictable means.


Pros

  • All elements take on a predictable, consistent structure through the core_ui_element.
  • The magic gets rolled into the core_ui_element class so that elements need only publish the public properties that they consist of, any any helper methods used in the construction of the object or in setting its properties. There is no need to add any more logic than that.
  • Absolutely all logic not associated with producing output to the needs of the theme gets rolled into the element objects. When rendering an element in most cases you should only need to render the structure for the element and then call render on its properties in the order you want.
  • It will be very easy to deem good interfaces from bad as anything that is not consisting of core_ui_elements will not be an ideal interface.
  • It will be very easy to see in code when writing a renderer what elements are available as each will be a class.
  • Its a clean approach - we will be able to do things as we believe is correct without concern of getting in the way of existing practises.

Cons

  • There is going to be a lot of classes. I don't mean hundreds, but we may end up with over a hundred easily enough.
  • Complexity of knowing PHP. In order to override a renderer you will need to know enough about PHP to understand what an object is.
  • This is a completely new approach, while it provides clean separation from current practices it is yet another learning curve.
  • There is a potential limitation if you have multiple elements of the same time occurring within a component each that needs to be rendered differently. I've not actually thought of a use case for this yet, but am mentioning it as something to be investigated.


Branch details

To view this branch at github:

   https://github.com/samhemelryk/moodle/tree/output_prototype

To get a copy of this code into your own local repository run the following commands:

   git checkout -b output_prototype
   git fetch http://github.com/samhemelryk/moodle.git output_prototype
   git reset --hard FETCH_HEAD

Viewing it in your browser

  1. First get a copy of the code into your local repository.
  2. Browse to your site and turn on allowthemechangeonurl (Settings > Appearance > Themes > Theme settings)
  3. To view in clean: browse to /local/output/index.php?theme=clean
  4. To view with Bootstrap 3: browse to /local/output/index.php?theme=output_bs3
  5. To view with YUI Pureio: browse to /local/output/index.php?theme=output_pureio
  6. To view with Zurb foundation: browse to /local/output/index.php?theme=output_zurb

Approach B

This design requires as little change to the existing code/implementations as possible. The idea, is simply to add a new list of renderables / renderer methods that implement the new things. There is no abstract class for renderables with no additional properties etc. Additionally, there are no restrictions on what can be put in a list/menu etc - it really is a just a list of renderables/strings. There is an edge case here, which is "How can a renderer method for a container modify the HTML for the elements it contains?". An example of this is that the Bootstrap 3 theme wants to put the role="menuitem" on the first link in each of the elements in it's "items" list. There are several possible approaches here - and this is one of the points of difference of this branch. One approach (too nasty) is to perform a str_replace on the first "<a" in the rendered html from the item - the downside of this approach is that it doesn't allow for differences in syntax for links, or merging with existing attributes on a link - it is doomed to fail. The approach chosen for this branch, is to parse the rendered html from the item into a DOMDocument - and use Dom manipulation to amend to the link before rendering it back to a string. There is a util method that takes this complexity out of the renderer method (and we can add more util methods as needed).

$attributes = array('role'=>'menuitem', 'tabindex'=>'-1');
$newitemhtml = dom_utils::add_attributes_to_first_node_of_type('a', $attributes, $itemhtml);

Branch details: git pull http://github.com/damyon/moodle.git MENU_RENDERABLE

Documents created as part of this spec

See also