Behat

Moodle 2.5

Nota: Pendiente de ACTUALIZAR esta traducción respecto a la página original en inglés (ver enlace hacia English en el cuadro abajo a la derecha). (otras páginas pendientes)

Introducción

Ésta página describe el funcionamiento interno de Behat y la integración con Moodle. Para la descripción funcional, vea Prueba_de_aceptación.

Behat es un armazón para el desarrollo impulsado por el comportamiento (behaviour driven development (BDD)) , que nos permite especificar funcionalidades en Moodle (conocidas como características (features)) como una lista legible-por-humanos de pasos y analizarr éstos pasos para ejecutar accionespara estimular la interacción con el usuario, ejecuta las acciones en contra de navegadores descabezados (headless) (sin soporte javascript , sólamente peticiones del tipo curl) o herramientas para simulación de usuarios como Selenium, que interactuan con navegadores y permiten la simulación de eventos JavaScript.

Objetivo

The aim of this integration is to allow Moodle components to have its own set of features and steps definitions, this allows BDD in Moodle and allows us to execute periodically the whole set of tests to detect regressions and test the Moodle features in different environments (browsers, DBs engines, web servers...). The Moodle QA tests will be progressively rewritten according to this format to run automatically.

Cómo funciona Behat

This section aims to explain the basics about BDD and Behat and a quick view of how Behat internally works from the CLI command execution to the results output.

Some terms used:

Features: Human-readable list of scenarios that describes a feature

 @auth
 Feature: Login
   In order to login
   As a moodle user
   I need to be able to validate the username and password against moodle
   
   Scenario: Login as an existing user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"
   
   Scenario: Login as an unexisting user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"

Scenario: Human-readable list of steps to describe an expected behaviour

 Scenario: Login as an existing user
   Given I am on "login/index.php"
   When I fill in "username" with "admin"
   And I fill in "password" with "moodle"
   And I press "loginbtn"
   Then I should see "Moodle 101: Course Name"

Steps: Human-readable sentences that describes an action. There are 3 types of steps, "Given" describing the initial context, "When" the event that provokes a change and "Then" where the outcomes should be asserted.

 I click on the "Add user" button

Steps definitions: PHP methods referenced by steps when matching it's regular expression. The @Given, @When and @Then tags are descriptive and they are not taken into account when matching steps with steps definitions. The regular expressions placeholders are returned to the PHP method as arguments so methods can use them to tell the browser which button (for example) they want to click.

 /**
  * @When /^I click on the "(.*)" button$/
  */
 public function i_click_on_the_button($button) {
   // Simulates the user interaction (see Mink description below for more info)
   $this->getSession()->getPage()->pressButton($button);
 }

Behat: PHP framework and CLI application that wraps the whole process of features files loading + features files parsing + execution of actions in the browser + results output (http://behat.org/)
Gherkin: Human-readable language used to define features that can be parsed and translated into PHP methods. For more info, it's the same language used by Cucumber, the BDD Ruby framework (https://github.com/cucumber/cucumber/wiki/Gherkin)
Context: In Behat scope a context is a PHP class that groups steps definitions (as methods)
Mink: Is the component which interacts with browsers, simulating a real user interaction. It allows us to write PHP code (or use the available PHP methods) to send petitions to the different browsers APIs through a common interface or extend it to allow browser-specific actions. The supported browsers includes Selenium, Selenium2, Sahi... http://mink.behat.org/
Selenium 2: Web browser automation tool, applications like Mink can communicate with it through a RESTful API (http://code.google.com/p/selenium/wiki/JsonWireProtocol) to execute actions simulating user interaction.
Selector type: Related with locator, is a way to select a node inside the page DOM, more info in https://docs.moodle.org/dev/Acceptance_testing#Providing_values_to_steps
Locator: Is what we are looking for inside the page DOM, it completely depends on the associated selector type, a few examples of it:
- Selector type = "link", Locator = "Link text"
- Selector type = "field", Locator = "Field legend text"
- Selector type = "css", Locator = ".css-class #id"
- Selector type = "xpath", Locator = "//input[@id='id-value']"

All this components are written in PHP, open sourced and packaged in a single and extensible framework.

Vista rápida del proceso completo

Behat CLI execution
- Behat application initialization and loading of arguments (features files to execute, output format...)
- Reads the Behat config file (browser servers are specified here)
- Extensions overrides management
- Gherkin initialization
Features files selection
- According to the arguments Gherkin looks for .features files
  - It can use different features loaders (single file, a directory, the default directory...)
  - The framework can be extended to allow multiple folders loading
Features parsing (Gherkin)
- Loops through the loaded features files looking for scenarios
- Gets the list of steps of each scenario
- There are hooks at different levels (http://docs.behat.org/guides/3.hooks.html)
Steps parsing (Gherkin)
- Gherkin looks in the available steps definitions for a regular expression that matches the step text
Step definition execution
- The step definition code is executed
- Steps definitions most of the time uses the Mink component to communicate with the browser API sending petitions like "click on that button" or "go to XXX page"
Scenario outcomes
- The scenario counts as failed if an exception is thrown when executing a step definition (for example trying to click a non-existing button)
- The scenario counts as passed if no exception is thrown during it's steps execution
Finishing CLI execution
- A summary with all the scenario results is displayed
- It accepts different output formats (like JUnitXML) to it's execution in continuous integration systems (http://docs.behat.org/guides/6.cli.html#format-options)

Integración con Moodle

It follows the approach chosen with PHPUnit:

It comes disabled by default, Behat is not included within Moodle and it has to be installed separately with the composer installer
Moodle components (subsystems and plugins) can have a tests/behat/ folder
The scenarios are executed in a test environment, the production database and dataroot are not affected by the tests modifications
The scenarios specifies their own fixtures and it's execution is isolated from other scenarios and features, resetting the test database and the test dataroot before each scenario
Moodle lists the features files and steps definitions of it's components in a behat.yml file, similar to the phpunit.xml manifest
A basic behat.yml.dist config file has been included

Alternative environment

Acceptance testing implies interaction with the browser like real users does, so it requires the site to be accessible via URL. The Moodle integration uses the PHP 5.4 built-in web server to run the tests in a sandbox without affecting the production environment, switching the regular $CFG->wwwroot, $CFG->dataroot and $CFG->prefix to alternatives only accessible from localhost or internal networks. Info about how to run the tests in https://docs.moodle.org/dev/Acceptance_testing#Running_tests.

This default configuration is useful when developing in a local host, but to run the tests automatically with Jenkins, travis, other CI systems or with saucelabs we provide a few extra settings. More info in https://docs.moodle.org/dev/Acceptance_testing#Advanced_usage

All the behat CLI utilities we provide with Moodle codebase (admin/tool/behat/cli/*) are using $CFG->behat_prefix and $CFG->behat_dataroot instead of $CFG->prefix and $CFG->dataroot, this scripts are self-contained, but as we are accessing through a browser, we also need to switch the whole Moodle instance to test mode. For this there are two requirements:

Test mode is enabled if
- php admin/tool/behat/cli/init.php or php admin/tool/behat/cli/util.php --enable has been executed
Test mode is requested if
- The vendor/bin/behat command is running, we know it because we hook the Behat process before the tests begins to run and we require moodle config.php after it
- We are accessing Moodle through the PHP 5.4 built-in web server, php_sapi_name() == 'cli-server'
- We are using $CFG->behat_switchcompletely (https://docs.moodle.org/dev/Acceptance_testing#Advanced_usage for more info)

With this approach we also allow the execution of the regular Moodle environment using the PHP 5.4 built-in web server for testing/dev purposes and we restrict the possibilities of a human mistake that can prevent production site's users to access their site.

Javascript

There are two types of tests depending on if their scenario needs a real browser capable of execute Javascript or if they can run in a headless browser.

Tests with Javascript requires interaction with a browser through a user simulation tool like Selenium or ZombieJS to be executed; see http://mink.behat.org/#different-browsers-drivers for all available drivers
Test that does not requires Javascript are faster to run but can not test rich applications like Moodle

In most of the cases a Javascript test would be more appropriate because most of the users uses Javascript-capable browsers, non-Javascript tests can be useful to ensure that Moodle maintains it's functionality without Javascript enabled and to ensure there are no big issues, regressions or exceptions in general.

Herramienta administrativa ""Prueba de Aceptación" (Acceptance testing)

There is an admin tool to run and ease the creation of acceptance tests.

Web interface: The web interface allows you to list and filter the available steps definitions, a non-technical user can use this interface to write new features (admin/tool/behat/index.php)
CLI: Command to enable and disable the test environment and to update the behat.yml file with the system tests and steps definitions (admin/tool/behat/cli/util.php and admin/tool/behat/cli/init.php for a quick start)

Pasos disponibles para crear pruebas

There are libraries with tons of steps definitions to run all sort of processes and interactions with the browser, some of them overlaps Moodle-specific libraries and tests writers can be confused not only by this also by the amount of steps and vague or too technical stepsdescriptions. Moodle provides a set of steps definitions written in a common format to make tests writers life easier. New steps definitions must follow this guidelines: https://docs.moodle.org/dev/Acceptance_testing#Adding_steps_definitions

Extensión de Behat

A new Behat extension (https://github.com/moodlehq/moodle-behat-extension) has been created to maintain Behat and its dependencies as they comes from upstream.

The aim of this extension is:

Load features from multiple folders (Moodle subsystems and plugins)
Load steps definitions from multiple folders and add them as subcontexts (Moodle subsystems and plugins)
Return the available steps definitions in a more human-readable format without regexps
Look for exceptions, debugging() calls, PHP error messages and other backtraces in Moodle's output
Extend the Selenium2 behat driver to allow extra Selenium capabilities

All the other particularities of this integration can managed playing with different Behat config parameters.

Documentation