Behat integration

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Introduction

This page describes the internals of Behat and the integration with Moodle. For the functional description see https://docs.moodle.org/dev/Acceptance_testing.

Behat is a framework for behaviour driven development (BDD) allows us to specify Moodle functionalities (aka features) as a human-readable list of steps and parse this steps to execute actions to simulate user interaction, it executes the actions against a headless browsers (without javascript support, only curl-kind petitions) or automation tools like Selenium, which interacts with browsers and allows Javascript events simulation.

(This is not in standard Moodle yet: to play a bit with it now clone https://github.com/moodlehq/moodle-behat-features and follow the installation and usage instructions, you may need to hack a bit to pass all the tests, it depends on you Moodle site contens. Is an example of tool usage but not an example of good practices to create tests and specify fixtures)

Objective

The aim of this integration is to allow Moodle components to have its own set of features and steps definitions, this allows BDD in Moodle and allows us to execute periodically the whole set of tests to detect regressions and test the Moodle features in different environments (browsers, DBs engines, web servers...).

How Behat works

This section aims to explain the basics about BDD and Behat and a quick view of how Behat internally works from the CLI command execution to the results output.

Some terms used:

Features: Human-readable list of scenarios that describes a feature

 @auth
 Feature: Login
   In order to login
   As a moodle user
   I need to be able to validate the username and password against moodle
   
   Scenario: Login as an existing user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"
   
   Scenario: Login as an unexisting user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"

Scenario: Human-readable list of steps to describe an expected behaviour

 Scenario: Login as an existing user
   Given I am on "login/index.php"
   When I fill in "username" with "admin"
   And I fill in "password" with "moodle"
   And I press "loginbtn"
   Then I should see "Moodle 101: Course Name"

Steps: Human-readable sentences that describes an action. There are 3 types of steps, "Given" describing the initial context, "When" the event that provokes a change and "Then" where the outcomes should be asserted.

 I click on the "Add user" button

Steps definitions: PHP methods referenced by steps when matching it's regular expression. The @Given, @When and @Then tags are descriptive and they are not taken into account when matching steps with steps definitions. The regular expressions placeholders are returned to the PHP method as arguments so methods can use them to tell the browser which button (for example) they want to click.

 /**
  * @When /^I click on the "(.*)" button$/
  */
 public function i_click_on_the_button($button) {
   // Simulates the user interaction (see Mink description below for more info)
   $this->getSession()->getPage()->pressButton($button);
 }

Behat: PHP framework and CLI application that wraps the whole process of features files loading + features files parsing + execution of actions in the browser + results output (http://behat.org/)
Gherkin: Human-readable language used to define features that can be parsed and translated into PHP methods. For more info, it's the same language used by Cucumber, the BDD Ruby framework (https://github.com/cucumber/cucumber/wiki/Gherkin)
Context: In Behat scope a context is a PHP class that groups steps definitions (as methods)
Mink: Is the component which interacts with browsers, simulating a real user interaction. It allows us to write PHP code (or use the available PHP methods) to send petitions to the different browsers APIs through a common interface or extend it to allow browser-specific actions. The supported browsers includes Selenium, Selenium2, Sahi... http://mink.behat.org/
Selenium 2: Web browser automation tool, applications like Mink can communicate with it through a RESTful API (http://code.google.com/p/selenium/wiki/JsonWireProtocol) to execute actions simulating user interaction.

All this components are written in PHP, open sourced and packaged in a single and extensible framework.

Quick view of the whole process

Behat CLI execution
- Behat application initialization and loading of arguments (features files to execute, output format...)
- Reads the Behat config file (browser servers are specified here)
- Extensions overrides management
- Gherkin initialization
Features files selection
- According to the arguments Gherkin looks for .features files
  - It can use different features loaders (single file, a directory, the default directory...)
  - The framework can be extended to allow multiple folders loading
Features parsing (Gherkin)
- Loops through the loaded features files looking for scenarios
- Gets the list of steps of each scenario
- There are hooks at different levels (http://docs.behat.org/guides/3.hooks.html)
Steps parsing (Gherkin)
- Gherkin looks in the available steps definitions for a regular expression that matches the step text
Step definition execution
- The step definition code is executed
- Steps definitions most of the time uses the Mink component to communicate with the browser API sending petitions like "click on that button" or "go to XXX page"
Scenario outcomes
- The scenario counts as failed if an exception is thrown when executing a step definition (for example trying to click a non-existing button)
- The scenario counts as passed if no exception is thrown during it's steps execution
Finishing CLI execution
- A summary with all the scenario results is displayed
- It accepts different output formats (like JUnitXML) to it's execution in continuous integration systems (http://docs.behat.org/guides/6.cli.html#format-options)

Moodle integration

It follows the approach chosen with PHPUnit:

Moodle components (subsystems and plugins) can have a tests/behat/ folder
The behat project is not included in Moodle codebase and has to be installed separately (https://github.com/dmonllao/behat-moodle/blob/master/README.md)
The scenarios are executed in a test environment (the same test environment used to execute PHPUnit tests)
The scenarios specifies their own fixtures and it's execution is isolated from other scenarios and features
Moodle lists the features files and steps definitions of it's components in a config.yml file, similar to the phpunit.xml manifest, that is read by Behat

This tests can not be executed in production sites; before executing the tests the whole site switches to "test environment" and all the logged users are kicked out. This is required because Behat runs black box tests simulating user interaction, it accesses via URL to the site like an user does.

Admin tool "Acceptance testing"

There is an admin tool in admin/tool/behat/index.php (https://github.com/dmonllao/moodle/tree/MDL-35611_master/admin/tool/behat) to run and ease the creation of acceptance tests. All the features listed below can be executed through CLI and web although it's better to run the tests with CLI. You can:

List the available steps (with an option to filter by keyword)
Run the tests (with an option to filter by tag or to add extra behat command options) it also updates the config.yml file

Also you can switch the real and test environments (uses $CFG->phpunit_prefix and $CFG->phpunit_dataroot instead of the regular $CFG->prefix and $CFG->dataroot) if you need to check something

Behat extension

The Behat framework is extended to load features and steps definitions from multiple folders (subsystems and plugins).

The basic Behat framework only allows one folder to look for features, we can set $CFG->dirroot as a single features project, but Behat will have a huge amount of work to do every time is executed to find where the features classes are. The basic framework has been extended to read the Moodle config.yml file, and with it's data, to load the features and steps definitions files spread along different Moodle components. (https://github.com/dmonllao/moodle-behat-extension)

Documentation