Behat integration

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Introduction

This page describes the internals of Behat and the integration with Moodle. For the functional description see https://docs.moodle.org/dev/Acceptance_testing.

Behat is a framework for behaviour driven development (BDD) allows us to specify Moodle functionalities (aka features) as a human-readable list of steps and parse this steps to execute actions to simulate user interaction, it executes the actions against a headless browsers (without javascript support, only curl-kind petitions) or user simulation tools like Selenium, which interacts with browsers and allows Javascript events simulation.

(This is not in standard Moodle yet: to play a bit with features files and Moodle clone you can install the proof of concept we worked on from https://github.com/moodlehq/moodle-behat-features and follow the installation and usage instructions, you may need to hack a bit to pass all the tests, it depends on you Moodle site contents. Is an example of tool usage but not an example of good practices to create tests and specify fixtures; the current work in progress is being done in http://tracker.moodle.org/browse/MDL-35611 and subtasks, you can checkout https://github.com/dmonllao/moodle/tree/MDL-36269_master this branch is always up-to-date according to this documentation page)

Objective

The aim of this integration is to allow Moodle components to have its own set of features and steps definitions, this allows BDD in Moodle and allows us to execute periodically the whole set of tests to detect regressions and test the Moodle features in different environments (browsers, DBs engines, web servers...). The Moodle QA tests will be progressively rewritten according to this format to run automatically.

How Behat works

This section aims to explain the basics about BDD and Behat and a quick view of how Behat internally works from the CLI command execution to the results output.

Some terms used:

• Features: Human-readable list of scenarios that describes a feature

 @auth
 Feature: Login
   In order to login
   As a moodle user
   I need to be able to validate the username and password against moodle
   
   Scenario: Login as an existing user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"
   
   Scenario: Login as an unexisting user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"

• Scenario: Human-readable list of steps to describe an expected behaviour

 Scenario: Login as an existing user
   Given I am on "login/index.php"
   When I fill in "username" with "admin"
   And I fill in "password" with "moodle"
   And I press "loginbtn"
   Then I should see "Moodle 101: Course Name"

• Steps: Human-readable sentences that describes an action. There are 3 types of steps, "Given" describing the initial context, "When" the event that provokes a change and "Then" where the outcomes should be asserted.

 I click on the "Add user" button

• Steps definitions: PHP methods referenced by steps when matching it's regular expression. The @Given, @When and @Then tags are descriptive and they are not taken into account when matching steps with steps definitions. The regular expressions placeholders are returned to the PHP method as arguments so methods can use them to tell the browser which button (for example) they want to click.

 /**
  * @When /^I click on the "(.*)" button$/
  */
 public function i_click_on_the_button($button) {
   // Simulates the user interaction (see Mink description below for more info)
   $this->getSession()->getPage()->pressButton($button);
 }
 

• Behat: PHP framework and CLI application that wraps the whole process of features files loading + features files parsing + execution of actions in the browser + results output (http://behat.org/)
• Gherkin: Human-readable language used to define features that can be parsed and translated into PHP methods. For more info, it's the same language used by Cucumber, the BDD Ruby framework (https://github.com/cucumber/cucumber/wiki/Gherkin)
• Context: In Behat scope a context is a PHP class that groups steps definitions (as methods)
• Mink: Is the component which interacts with browsers, simulating a real user interaction. It allows us to write PHP code (or use the available PHP methods) to send petitions to the different browsers APIs through a common interface or extend it to allow browser-specific actions. The supported browsers includes Selenium, Selenium2, Sahi... http://mink.behat.org/
• Selenium 2: Web browser automation tool, applications like Mink can communicate with it through a RESTful API (http://code.google.com/p/selenium/wiki/JsonWireProtocol) to execute actions simulating user interaction.

All this components are written in PHP, open sourced and packaged in a single and extensible framework.

Quick view of the whole process

1. Behat CLI execution
   • Behat application initialization and loading of arguments (features files to execute, output format...)
   • Reads the Behat config file (browser servers are specified here)
   • Extensions overrides management
   • Gherkin initialization
2. Features files selection
   • According to the arguments Gherkin looks for .features files
     • It can use different features loaders (single file, a directory, the default directory...)
     • The framework can be extended to allow multiple folders loading
3. Features parsing (Gherkin)
   • Loops through the loaded features files looking for scenarios
   • Gets the list of steps of each scenario
   • There are hooks at different levels (http://docs.behat.org/guides/3.hooks.html)
4. Steps parsing (Gherkin)
   • Gherkin looks in the available steps definitions for a regular expression that matches the step text
5. Step definition execution
   • The step definition code is executed
   • Steps definitions most of the time uses the Mink component to communicate with the browser API sending petitions like "click on that button" or "go to XXX page"
6. Scenario outcomes
   • The scenario counts as failed if an exception is thrown when executing a step definition (for example trying to click a non-existing button)
   • The scenario counts as passed if no exception is thrown during it's steps execution
7. Finishing CLI execution
   • A summary with all the scenario results is displayed
   • It accepts different output formats (like JUnitXML) to it's execution in continuous integration systems (http://docs.behat.org/guides/6.cli.html#format-options)

Moodle integration

It follows the approach chosen with PHPUnit:

• It comes disabled by default, Behat is not included within Moodle and it has to be installed separately with the composer installer
• Moodle components (subsystems and plugins) can have a tests/behat/ folder
• The scenarios are executed in a test environment, the production database and dataroot are not affected by the tests modifications
• The scenarios specifies their own fixtures and it's execution is isolated from other scenarios and features, resetting the test database and the test dataroot before each scenario
• Moodle lists the features files and steps definitions of it's components in a behat.yml file, similar to the phpunit.xml manifest
• A basic behat.yml.dist config file has been included (Behat will use behat.yml rather than behat.yml.dist if both are available)

Alternative environment

Acceptance testing implies interaction with the browser like real users does, so it requires the site to be accessible via URL. The Moodle integration uses the PHP 5.4 built-in web server to run the tests in a sandbox without affecting the production environment, switching the regular $CFG->wwwroot, $CFG->dataroot and $CFG->prefix to alternatives only accessible from localhost, info about how to run the tests in https://docs.moodle.org/dev/Acceptance_testing#Running_tests.

This default configuration is useful when developing in a local host, but to run the tests automatically with Jenkins, travis, other CI systems or with saucelabs we provide a few extra settings; more info in https://docs.moodle.org/dev/Acceptance_testing#Advanced_usage

Javascript

There are two types of tests depending on if they scenario needs a real browser capable of execute Javascript or if they can run in a headless browser.

• Tests with Javascript requires interaction with a browser through a user simulation tool like Selenium or ZombieJS to be executed; see http://mink.behat.org/#different-browsers-drivers for all available drivers
• Test that does not requires Javascript are faster to run but can not test rich applications like Moodle

In most of the cases a Javascript test would be more appropriate because most of the users uses Javascript-capable browsers, non-Javascript tests can be useful to ensure that Moodle maintains it's functionality without Javascript enabled and to ensure there are no big issues, regressions or exceptions in general.

Admin tool "Acceptance testing"

There is an admin tool to run and ease the creation of acceptance tests.

• Web interface: The web interface allows you to list and filter the available steps definitions, a non-technical user can use this interface to write new features (admin/tool/behat/index.php)
• CLI: Command to enable and disable the test environment and to update the behat.yml file with the system tests and steps definitions (admin/tool/behat/cli/util.php)

Behat extension

A new Behat extension (https://github.com/moodlehq/moodle-behat-extension) has been created to maintain Behat and it's dependencies as they comes from upstream.

The aim of this extension is:

• Load features from multiple folders (Moodle subsystems and plugins)
• Load steps definitions from multiple folders and add them as subcontexts (Moodle subsystems and plugins)
• Return the available steps definitions in a more human-readable format without regexps

All the other particularities of this integration can managed playing with different Behat config parameters.