Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Acceptance testing

From MoodleDocs
Revision as of 14:31, 23 March 2017 by Tim Hunt (talk | contribs)

See also Running acceptance test

Introduction

This page describes how we describe Moodle's functionalities and automatically test them.

Behat is a behavioural driven development (BDD) tool written in PHP, it can parse a human-readable list of sentences (called steps) and execute actions in a browser using Selenium or other tools to simulate user interactions.

For technical info: Behat integration

How it works

Behat parses and executes features files which describe Moodle's features (for example Post in a forum). Each feature file is composed of many scenarios (for example Add a post to a discussion or Create a new discussion), and finally each scenario is composed of steps (for example  I press "Post to forum" or I should see "My post title"). When the feature file is executed, every step internally is translated into a PHP method and is executed.

These features are executed nightly on the HQ servers using all the supported databases (MySQL, PostgreSQL, MSSQL and Oracle) and with different browsers (Firefox, Internet Explorer, Safari and Chrome) to avoid regressions and to test new functionalities.

Examples

Note that these snippets are only examples and may not work.

  • There is a closed list of steps to use in the features, a feature written with the basic (or low-level) steps looks like this:
 @auth
 Feature: Login
   In order to login
   As a moodle user
   I need to be able to validate the username and password against moodle
   
   Scenario: Login as an existing user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"
   
   Scenario: Login as an unexisting user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"

Note that The 3 sentences below Feature: Login are only information about what we want to test.

These are simple scenarios, but most of Moodle's functionalities would require a huge list of this steps to test a scenario, imagine a Add a post to a discussion scenario; you need to login, create a course, create a user and enrol it in the course... Most of this steps is not what we intend to test in a Post in a forum feature, Moodle provides extra steps to quickly set up the context required to test a Moodle feature, for example:

 @mod @mod_forum
 Feature: Add forum activities and discussions
   In order to discuss topics with other users
   As a moodle teacher
   I need to add forum activities to moodle courses
   
   Scenario: Add a forum and a discussion
     Given the following "users" exist:
       | username | firstname | lastname | email |
       | teacher1 | Teacher | 1 | teacher1@asd.com |
     And the following "courses" exist:
       | fullname | shortname | category |
       | Course 1 | C1 | 0 |
     And the following "course enrolments" exist:
       | user | course | role |
       | teacher1 | C1 | editingteacher |
     And I log in as "teacher1"
     And I follow "Course 1"
     And I turn editing mode on
     And I add a "Forum" to section "1" and I fill the form with:
       | Forum name | Test forum name |
       | Forum type | Standard forum for general use |
       | Description | Test forum description |
     When I add a new discussion to "Test forum name" forum with:
       | Subject | Forum post subject |
       | Message | This is the body |
     Then I should see "Test forum name"

Note that:

  • Each scenario is executed in an isolated testing environment, so the first step begins with an empty moodle site and what you set up in an scenario (like the Test forum name forum in the example above) is cleaned up after the scenario execution
  • The prefixes "Given", "When" and "Then" are only informative and they are used to define the context (Given), specify the action (When) and check the results (Then), using them properly helps to understand what the scenario is testing.
  • It is good practice for each scenario to only test one thing. (That is, set up a aprticular situation, perform one action, and verify the result). Therefore, Given, When and Then should only be used once per scenario.

Quick start

This is a quick introduction to write a functional test (acceptance tests) using steps in a development/testing site, please DON'T USE THIS IN A PRODUCTION SITE.

To let you experience the pleasure of watching a feature file doing "your work" automatically in a real browser, this guide includes 2 optional steps to download Selenium and run it in another CLI.

  1. Open a command line interface (you might need to run as administrator to get curl to work - one windows, right click the command prompt and 'choose run as administrator')
  2. cd /to/your/moodle/dirroot
  3. Edit config.php adding the following lines before the lib/setup.php include
    $CFG->behat_prefix = 'b_';

$CFG->behat_dataroot = '/path/to/your/behat/dataroot/directory'; $CFG->behat_wwwroot = 'http://127.0.0.1'; // must be different from wwwroot

  1. curl http://getcomposer.org/installer | php (If you get a curl error saying 'curl is not recognized as an internal or external command' run the command line as administrator. In case you still have problems read https://docs.moodle.org/dev/Acceptance_testing#Installation)
  2. php admin/tool/behat/cli/init.php (If you get a curl error saying 'curl is not recognized as an internal or external command' run the command line as administrator.)
  3. Download selenium-server-standalone-2.NN.N.jar from http://seleniumhq.org/download/, under "Selenium server (formerly the Selenium RC Server)"
  4. Open another command line interface and run java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar
  5. vendor/bin/behat --config /path/to/your/behat/dataroot/directory/behatrun/behat/behat.yml
  6. You just ran the current Moodle tests, now let's add your own test, add a blog entry for example
  7. Browse to your $CFG->behat_wwwroot, this is an empty test site and it is reset before each test (called scenario)
  8. From this point follow the steps you would follow to add manually a blog entry (login credentials are admin/admin)
  9. When you are done go to 'Site administration' -> 'Development' -> 'Acceptance testing', you will find the list of "actions" that can be run automatically, you can filter them to find what do you need to do (more steps can be added if you need, more info in https://docs.moodle.org/dev/Acceptance_testing#Adding_steps_definitions)
  10. To 'add a blog entry' we need to:
    1. Log in the system as a valid user
    2. Expand 'My profile' node of the navigation block
    3. Expand the 'Blogs' node of the navigation block
    4. Follow he 'Add a new entry' link
    5. Fill the moodle form with values for 'Entry title' and 'Blog entry body'
    6. Press the 'Save changes' button
    7. Verify you see the values you entered in the form and verify you are not in the form page
  11. This translated to steps is:
    Given I log in as "admin"

And I expand "My profile" node And I expand "Blogs" node And I follow "Add a new entry" And I set the following fields to these values:

 | Entry title | I'm the name |
 | Blog entry body | I'm the description |

When I press "Save changes" Then I should see "User Blog" And I should see "I'm the description" And I should not see "Required"

  1. We need to wrap this steps following the behavior driven development guidelines (more info in https://docs.moodle.org/dev/Acceptance_testing#Writing_features)

@core @core_blog Feature: Add a blog entry

 In order to let the world know about me
 As a user
 I need to write blog entries
 @javascript
 Scenario: Add a blog entry with valid data
   Given I log in as "admin"
   And I expand "My profile" node
   And I expand "Blogs" node
   And I follow "Add a new entry"
   And I fill the moodle form with:
     | Entry title | I'm the name |
     | Blog entry body | I'm the description |
   When I press "Save changes"
   Then I should see "View all of my entries"
   And I should see "I'm a description"
   And I should not see "Required"

  1. And save it into a file, in this case blog/tests/behat/add_entry.feature
  2. php admin/tool/behat/cli/util.php --enable (This will update the available tests and steps definitions)
  3. vendor/bin/behat --config /path/to/your/behat/dataroot/directory/behatrun/behat/behat.yml --tags @core_blog
  4. Selenium will open a browser (firefox by default) and you will see how the steps you have been writting are executed

You can also try to expand non existing nodes or change the 'Then' assertions to get a beautiful failure.

For detailed steps and/or troubleshooting:

Requirements

Installation

  • Edit config.php
    • Use $CFG->behat_dataroot to set the directory where behat test environment dataroot will be stored, something like $CFG->behat_dataroot = '/your/directory/path';. Ensure the directory can be created or have write permissions
    • Use $CFG->behat_prefix to set the database prefix of the behat test environment database tables, something like $CFG->behat_prefix = 'behat_';
    • Use $CFG->behat_wwwroot to set address to be used to access behat instance. It has to be different from $CFG->wwwroot, you can use for example localhost, 127.0.0.1 or any custom local host name specified in you /etc/hosts. If you use the built-in PHP server use "http://localhost:8000" or the value you set when you started it.
  • Download composer
  • Install behat dependencies and enable the test environment
    • cd /your/moodle/dirroot
    • php admin/tool/behat/cli/init.php
  • (Optional) If you want to run tests that involves Javascript (most of them) you will also need Selenium. What version do I need?

Verify your installation

  • Test your installation by browsing your /admin/tool/behat/index.php page.

If you are using MAMP/WAMP/XAMPP and don't get the report working... then you need to ensure that your web servers knows where the PHP executable (commad line) is. To do that, edit the "envvars" file (usually under library/bin) and add to $PATH the path to your php executable. Then restart the apache server and try again. For example, for MAMP you probably need to add:

  1. Adding MAMP binaries communicate where CLI is

PATH="/Applications/MAMP/bin/php/php5.x.y/bin:$PATH"; export PATH

at the end of MAMP/Library/bin/envvars.

Running tests

For running acceptance test you need to do following:

  1. Initialise acceptance test environment
  2. Running acceptance test environment

Run behat using different browsers

Following these instructions you can run behat in Firefox browser through Selenium, but you can run behat using different browsers through Selenium and even use phantomjs (Webkit). More info

Advanced usage

There are a few settings for advanced use of Behat and execution in continuous integration systems, by default all this options are disabled, use this settings only if you know what you are doing.

  • Different test server URL. If for example your are interested in allowing accesses from your local network because your Jenkins server is there you can set $CFG->behat_wwwroot to http://my.computer.local.ip:8000
  • Behat configuration, Moodle writes a behat.yml config file with info about the available tests and steps definitions along with other Behat parameters, you can override the Behat parameters we set and add your new parameters, your parameters will be merged with the Moodle ones giving priority to your values in case of conflict. This is useful for an advanced use of Behat, with multiple profiles, output formats, integration with continuous servers...
  • Save screenshots of failures. You can use $CFG->behat_faildump_path to specify a directory where behat will generate a screenshot with the browser state each time a scenario fails. This is useful to detect where the problem was and work on a solution.
  • Running with a browser other than Firefox, by adding the following code to your config.php you can change the selected browser that is run when behat is invoked. In this case Chrome is selected, but internet explorer, firefox, iphone, android, chrome, htmlunit should be valid options. You will need to run php admin/tool/behat/cli/init.php for changes to take effect.

$CFG->behat_profiles = array(

  'chrome' => array(
      'browser' => 'chrome',
      'tags' => '@javascript',
      'wd_host' => 'http://127.0.0.1:4444/wd/hub',
      'capabilities' => array(
          'platform' => 'Linux'
      )
  )

);

Note that for Chrome, you will need the Selenium Chrome Driver (https://code.google.com/p/selenium/wiki/ChromeDriver), and it will need to be installed in the command search path.
  • Switch completely to test environment option was removed in 2.7, it is recommended to always set $CFG->behat_wwwroot even in older versions instead.
  • Note that when using cloud-based systems that can make use of non-standard capabilities like Saucelabs, you might want to provide configuration attributes containing the '-' character, which is automatically converted to '_' by the Symfony configuration manager that Behat is making use of (@see Symfony\Component\Config\Definition\Processor::normalizeKeys()) a way to avoid this restriction is to, adding to the vars you set like 'max-duration' add the same var replacing dashes for underscores, this way the configuration manager will maintain the attribute containing dashes.
  • Extra allowed settings, moodle allows users to define many settings in config.php (see config-dist.php) when running the behat test site those settings are skipped to avoid interaction with the production environment, in case you are interested in allowing some of those extra settings to run the tests using a configuration similar to the one you are using in your production environment you can whitelist them adding them to $CFG->behat_extraallowedsettings (see config-dist.php for examples).

You can find more info and examples of how to use this settings in the config-dist.php file included in the Moodle codebase.

  • Setting different parameters for parallel behat run, by adding the following code to your config.php you can specify parameters for each parallel run. $CFG->behat_parallel_run is an array with first set of array params applied to first behat run and so on.

$CFG->behat_parallel_run = array (

      array (
          'dbtype' => 'mysqli',
          'dblibrary' => 'native',
          'dbhost' => 'localhost',
          'dbname' => 'moodletest',
          'dbuser' => 'moodle',
          'dbpass' => 'moodle',
          'behat_prefix' => 'mdl_',
          'wd_host' => 'http://127.0.0.1:4444/wd/hub',
          'behat_wwwroot' => 'http://127.0.0.1/moodle',
          'behat_dataroot' => '/home/example/bht_moodledata'
      ),
  );

Note: If you are running 2 runs and copy/paste above code in config.php then 2nd behat run will use default behat values like $CFG->dbtype, $CFG->behat_prefix etc.

Contributing

You can contribute the effort to automatically test all of Moodle's functionalities, as described in the guide to contributing automated tests. Here you can find information about how to write new features and how to write new step definitions if your changes requires a base change in the Moodle behat extension you can find here how to do it following the integration workflow: Acceptance_testing/Contributing_to_Moodle_behat_extension.

Writing features

All Moodle components and plugins (including 3rd party plugins) can specify their tests in .feature files using all the available steps.

Once you decided which functionality you want to specify as a feature you should:

  1. Select the most appropriate Moodle component to include your test and create a COMPONENTNAME/tests/behat/FEATURENAME.feature file
  2. Add a tag with the component name in Frankenstyle format (https://docs.moodle.org/dev/Frankenstyle) on the first line along with the plugin type or @core if it's a core subsystem
  3. Begin writing the user story of the feature, including in the 'As a ...' statement the main beneficiary of the feature:
    @plugintype @plugintype_pluginname

Feature: FEATURENAME

 In order to ...    // Why this feature is useful
 As ...    // It can be 'an admin', 'a teacher', 'a student', 'a guest', 'a user', 'a tests writer' and 'a developer'
 I need to ...      // The feature we want
  1. From the beneficiary point of view, think of different scenarios to ensure the feature works as expected
  2. For each scenario you thought:
    1. Think of the initial context you need, for example 1 course with 2 students on it and an assignment, and which steps do you need to follow (interacting with the browser) to verify the scenario works as expected
    2. What you are testing requires Javascript? Think only on the feature you are testing (for example if you want to test that you can view your profile you don't need Javascript to click on a link and assert against plain HTML, but if you want to test something related with the course's gradebook you might want to test it with Javascript)
    3. Check the steps list (more info in https://docs.moodle.org/dev/Acceptance_testing#Available_steps) and set the initial context data (see https://docs.moodle.org/dev/Acceptance_testing#Fixtures for more info) and the steps to follow to verify all works as it should work.
    4. The prefixes Given, When and Then separates the scenario in 3 parts, the initial context setup (Given), the action that provokes a change in the system (When) and the validation of that change outcomes (Then) So with a quick view at the scenario you can see what it is testing as the prefixes will be something like Given -> And -> And -> When -> And -> And -> Then -> And -> And -> And. Once we begin with the first Then we can consider that we are checking the outcomes so all the steps from there should be prefixed with Then
    5. Copy the list of steps to the .feature file with the Scenario header:
      Scenario: Short description of the scenario
 Given step 1
 And step 2
 And step 3
 When step 4
 And step 5
 Then step 6
    1. If the steps you are using requires Javascript add the @javascript tag above the "Scenario:" headline
      @javascript

Scenario: Short description of the scenario

 ...
 ...
  1. Run the tests, when creating your new features/scenarios you can specify a '@wip' (work in progress) tag in both the line above the Scenario description and the tests runner (vendor/bin/behat) to execute only the new scenario instead of running the whole set of tests.
  2. Add extra tags to the scenario or the feature if required according to https://docs.moodle.org/dev/Acceptance_testing#Tests_filters

Available steps

Moodle provides a interface to list and filter the steps you can use when writing features. You can access it through the Administration block, following Site Administration -> Development -> Acceptance testing. It allows filtering by keyword, by the Moodle component or by the type of step:

  • Processes to set up the environment
  • Actions that provokes an event
  • Checkings to ensure the outcomes are the expected ones

Acceptance testing UI 2.5.png

  • This interface only works on sites where Behat is installed. If you are using the local PHP server, for example, you can access it on that site at http://localhost:8000 (log in as admin/admin).

Tips

  • You can use a Background section before the Scenario sections, this steps will be executed before the steps of each scenario (http://docs.behat.org/guides/1.gherkin.html#backgrounds)
  • You can use Scenario outlines if your scenarios are nearly the same and depends on a few vars; check out the link for an explicative example (http://docs.behat.org/guides/1.gherkin.html#scenario-outlines)
    • If your scenario outline consists of many steps it may be useful to add a comment with the number of steps. If the the test fails behat will tell you in which step overall and you will have to divide by the number of steps of the scenario to know in which example.
  • Is better to test the outcomes against the given data than against language strings, which are depending on the selected language.
  • In case you need to interact with popup windows you need to switch to the window you want to interact with after opening it using the I switch to "popupwindowname" window, close it when you finish interacting with it and return to the main window using I switch to main window
  • The format of the .feature files is YAML which finds out the data hierarchy from the indentation of it's elements, so be sure that the elements are correctly nested and the indentation is correct using spaces when necessary

Providing values to steps

Most of the steps requires values, there are five methods to provide values to steps, the method depends on the step specification, you can know when a steps requires a value because you will see a drop down menu with a closed list of options that the step accepts as argument or an upper case string between double quotes, something like I press "BUTTON_STRING" or it ends with a : . The five methods are:

  • A string/text; is the most common case, the texts are wrapped between double quotes (" character) you have to replace the info about the expected value for your value; for example something like I press "BUTTON_STRING" should become I press "Save and return to course". If you want to add a string which contains a " character, you can escape it with \", for example I fill the "Name" field with "Alan alias \"the legend\"". You can identify this steps because they ends with _STRING
  • A number; some steps requires numbers as values, to be more specific an undetermined number of digits from 0 to 9 (Natural numbers + 0) you can identify them because the expected value info string ends with _NUMBER
  • A table; is a relation between values, the most common use of it is to fill forms. The steps which requires tables are easily identifiable because they finish with : The steps description gives info about what the table columns must contain, for example Fills a moodle form with field/value data. Here you don't need to escape the double quotes if you want to include them as part of the value.
  • A PyString; is a multiline string, most commonly used to fill out forms when a newline is required. Like steps with tables, steps which require PyStrings will end with ":"
  • A field value; There are many different field types, if an argument requires a field value the expected value will depend on the field type:
    • Text-based fields: It expects the text. This includes textareas, input type text, input type password...
    • Checkbox: It expects 1 to check and for checked and "" to uncheck or for unchecked
    • Select: It expects the option text or the option value. In case you interact with a multi-select you should specify the options separating them with commas. For example: option1, option2, option3
    • Radio: The text of the radio option
  • A selector; there are steps that can be used with different kinds of elements, for example I click on "User Name" "link" or I click on "User Name" "button" this is a closed list of elements, they always works together with another argument, where you specify the locator (eg. the link text in a link) In the 'Acceptance testing' interface you can see a drop-down menu to select one of these options:
    • field - for searching a field by its id, name, value or label
    • link - for searching a link by its href, id, title, img alt or value
    • button - for searching a button by its name, id, value, img alt or title
    • link_or_button - for searching for both, links and buttons
    • select - for searching a select field by its id, name or label
    • checkbox - for searching a checkbox by its id, name, or label
    • radio - for searching a radio button by its id, name, or label
    • file - for searching a file input by its id, name, or label
    • optgroup - for searching optgroup by its label
    • option - for searching an option by its content
    • dialogue - for searching a dialogue with the specified header text
    • filemanager - for searching a filemanager by it's id or label
    • block - for searching a Moodle block by it's English name or it's frankenstyle name
    • section - for searching for a section on a course page by it's title or its written out date (e.g. "1 January - 7 January"). Use "frontpage" "section" for the frontpage section if it has no title (default)
    • activity - for searching for an activity module in a course list by it's title
    • region - for searching a Moodle page region with that id, in fact it works with all the page's ids
    • table_row - for searching a table row which contains the specified text
    • table - for searching a table by its id or caption
    • fieldset - for searching a fieldset by it's id or legend
    • css_element - for searching an element by its CSS selector
    • xpath_element - for searching an element by its XPath
  • A text selector; similar to a selector but those are the elements that returns an area of the DOM, they are useful in steps following the format ... in the "Community finder" "block" where you are clicking or looking for some text inside a specific area. In the 'Acceptance testing' interface you can see a drop-down menu to select one of these options:
    • dialogue - for searching a dialogue with the specified header text
    • block - for searching a Moodle block by it's English name or it's frankenstyle name
    • section - for searching for a section on a course page by it's title or its written out date (e.g. "1 January - 7 January"). Use "frontpage" "section" for the frontpage section if it has no title (default)
    • activity - for searching for an activity module in a course list by it's title
    • region - for searching a Moodle page region with that id, in fact it works with all the page's ids
    • table_row - for searching a table row which contains the specified text
    • table - for searching a table by its id or caption
    • fieldset - for searching a fieldset by it's id or legend
    • css_element - for searching an element by its CSS selector
    • xpath_element - for searching an element by its XPath

Uploading files

Note than some tests requires files to be uploaded, in this case

  • The I upload "FILEPATH_STRING" file to "FILEPICKER_FIELD_STRING" filepicker step can be used when located in the form page
  • The file to upload should be included along with the Moodle codebase in COMPONENTNAME/tests/fixtures/*
  • The file to upload is specified by it's path, which should be relative to the codebase root (lib/tests/fixtures/users.csv for example)
  • / should be used as directory separator and the file names can not include this / character as all of them would be converted to the OS-dependant directory separator to maintain the compatibility with Windows systems.
  • The scenarios that includes files uploading should be tagged using the @_file_upload tag

Checking table values

You can check if specific value exists or not in a table row/column by using:

  • Then "STRING_IN_ROW" row "COLUMN_HEADER" column of "TABLE_ID" table should contain "VALUE_TO_CHECK"
  • Then the following should exist in the "TABLE_ID" table:
   | COLUMN_HEADER1 | COLUMN_HEADER2 |
   | VALUE_IN_ROW_1 | VALUE_IN_ROW_1 |
   | VALUE_IN_ROW_2 | VALUE_IN_ROW_2 |

Fixtures

As seen in [examples] Moodle provides a way to quickly set up the contextual data (courses, users, enrolments...) that you need to properly test scenarios, this can be done creating entities in the background section (common for all the steps) or in the "Given" part of your scenario. Note that this steps can only be used to set up the contextual data required to test the feature but they don't test what they are doing; for example, the "Given the following "users" exists" is not testing that Moodle is able to create a user, but to test that a user can add a blog entry you might want to use this step. For further info, acceptance tests are supposed to be black-boxed tests (the tester don't know about the internals of the application) and this steps are using internal Moodle data generators instead of running all the steps required to create a user or to create a course, which speeds up the test execution. There are other features to test that all this elements can be properly created.

Available elements

Most of the available elements can only be created in relation to other elements, to hide the complexity of the Moodle internals (references by contexts, ids...) the references can be done using more human-friendly mappings.

The examples below shows how to add elements referencing other elements, there are required fields to reference the elements, other attributes will be filled with random data if they are not specified.

  • Course categories
    • The required field is idnumber
    • References between parent/children by their idnumber, using the "category" field
 Given the following "categories" exist:
   | name       | category | idnumber |
   | Category 1 | 0        | CAT1     |
   | Category 2 | CAT1     | CAT2     |
  • Courses
    • The required field is shortname
    • Uses the category idnumber as category reference
 Given the following "courses" exist:
   | fullname | shortname | category | format | 
   | Course 1 | COURSE1   | CAT1     | topics |
   | Course 2 | COURSE2   | CAT2     |        |
  • Activities *(note that this step does not work with all kind of activities, only the ones that have data generators)*
    • The required fields are activity, course and idnumber
    • Uses activity to specify the activity type
    • Uses the course shortname as course reference
    • Other activity-dependant fields can be specified by it's field name
 Given the following "activities" exist:
   | activity | course | idnumber | name                 | intro                       |
   | assign   | C1     | assign1  | Test assignment name | Test assignment description |
   | data     | C1     | data1    | Test database name   | Test database description   |
  • Groups
    • The required fields are course and idnumber
    • Uses the course shortname as course reference
 Given the following "groups" exist:
   | name    | description | course  | idnumber |
   | Group 1 | Anything    | COURSE1 | GROUP1   |
  • Groupings
    • The required fields are course and idnumber
    • Uses the course shortname as course reference
 Given the following "groupings" exist:
   | name       | course  | idnumber  |
   | Grouping 1 | COURSE1 | GROUPING1 |
   | Grouping 2 | COURSE1 | GROUPING2 |
  • Users
    • The required field is username (if password is not set username value will be used as password too)
 Given the following "users" exist:
   | username | email       | firstname | lastname |
   | testuser | asd@asd.com | Test      | User     |
  • Course enrolments
    • The required fields are user, course and role
    • Uses the course shortname as course reference
    • Uses the user username as user reference
    • Uses the role shortname as role reference
    • Uses the enrolment name as enrol reference
 Given the following "course enrolments" exist:
   | user     | course  | role           | enrol  |
   | testuser | COURSE1 | editingteacher | manual |
  • Roles
    • The required field is shortname
    • Uses a one of the following roles as archetype: manager, coursecreator, editingteacher, teacher, student, guest, user and frontpage
 Given the following "roles" exist:
   | shortname | name          | description | archetype      |
   | custom1   | Custom Role 1 |             | editingteacher |
  • Role assigns
    • The required fields are user, role, contextlevel and reference
    • Uses the user username as user reference
    • Uses the role shortname as role reference
    • Uses contextlevel + reference to specify the context. See #Referencing_contexts for more info.
 Given the following "role assigns" exist:
   | user  | role           | contextlevel | reference |
   | user1 | manager        | System       |           |
   | user2 | editingteacher | Category     | CATEGORY1 |
   | user3 | editingteacher | Course       | COURSE1   |
  • System role assigns (deprecated in favour of role assigns, see item right above this one)
    • The required fields are user and role
    • Uses the user username as user reference
    • Uses the role shortname as role reference
 Given the following "system role assigns" exist:
   | user     | role    |
   | testuser | manager |
  • Permission overrides
    • The required fields are capability, permissions, role, and the contextlevel + it's reference
    • Uses contextlevel + reference to specify the context. See #Referencing_contexts for more info.
 Given the following "permission overrides" exist:
   | capability            | permission | role           | contextlevel | reference |
   | mod/forum:editanypost | Allow      | student        | Course       | C1        |
   | mod/forum:replynews   | Prevent    | editingteacher | Course       | C1        |
   | mod/paquiro:sings     | Prohibit   | student        | System       |           |
  • Group members
    • The required fields are user and group
    • Uses the group idnumber as group reference
    • Uses the user username as user reference
 Given the following "group members" exist:
   | user     | group  |
   | testuser | GROUP1 |
  • Grouping groups
    • The required fields are grouping and group
    • Uses the group idnumber as group reference
    • Uses the grouping idnumber as grouping reference
 Given the following "grouping groups" exist:
   | grouping  | group  |
   | GROUPING1 | GROUP1 |
  • Cohorts
    • The required field is idnumber
 Given the following "cohorts" exist:
   | name     | idnumber |
   | Cohort 1 | COHORT1  |

Referencing contexts

Moodle has different context levels, internally they have an identifier, but to reference them from steps we can use a more human way, using the level of the context (as specified below) and the reference, which will depend on the contextlevel we are using:

  • contextlevel: System, User, Category, Course or Activity module
  • reference:
    • System: Nothing, just leave the the cell empty
    • User: The user username
    • Category: The category idnumber
    • Course: The course shortname
    • Activity module: The activity idnumber

The data generators which makes use of these format are pointing to here.

Features check list

Adding steps definitions

Each Moodle component and plugin (including 3rd party plugins) can add new steps definitions. If you are writing tests and you notice that you are repeating the same group of steps you might want to create a new step definition that allows you to substitute the group of steps for one single step, something like I add a forum post with "blablabla" as description for example; also you can create whole new steps using the APIs provided by Behat and Mink if what you need to do is not covered by any of the available steps.

As commented in https://docs.moodle.org/dev/Acceptance_testing#Fixtures, this are black box tests, so we are not supposed to know about Moodle internals; translated to developer language it means don't use Moodle internals API calls, for example you should not try to cheat using a set_config() call, you should follow Moodle's user interface to reach the setting page and change it's value.

Example

You can use this example below or any of the existing steps definitions as a template.

  • auth/tests/behat/behat_auth.php
 class behat_auth extends behat_base {
     /**
      * Logs in the user. There should exist a user with the same value as username and password.
      *
      * @Given /^I log in as "(?P<username_string>(?:[^"]|\\")*)"$/
      */
     public function i_log_in_as($username) {
         // Visit login page.
         $this->getSession()->visit($this->locate_path('login/index.php'));
         // Enter username and password.
         $this->execute('behat_forms::i_set_the_field_to', array('Username', $this->escape($username)));
         $this->execute('behat_forms::i_set_the_field_to', array('Password', $this->escape($username)));
         // Press log in button, no need to check for exceptions as it will checked after this step execution.
         $this->execute('behat_forms::press_button', get_string('login'));
     }
 }

Tips

If you are creating a completely new step definition there are also a few things to consider:

  • Steps definitions should be compatible with both Javascript and non-Javascript tests, you can use $this->running_javascript() to deal with both
  • The definition code will be executed by Behat, not by Moodle, you have to keep this in mind for example when throwing exceptions, Behat exceptions will give more info to the user about where is the problem
    • You can find these exceptions in vendor/behat/mink/src/Behat/Mink/Exception/*
  • Selenium is fast, sometimes it tries to interact with DOM elements or tries to execute actions that requires JS that are not loaded or ready to used; this is why, sometimes and randomly, you can see an "element not found" failure
    • The quickest way to solve this problem is using behat_base::find*() methods (where the * corresponds to '', _all, or to a named selector preceded by _, http://mink.behat.org/#named-selectors) which only requires the locator as argument. This methods will wait for the requested element to be ready or return an exception if the element is not found after the timeout value expires, you can also force the timeout value, which defaults to 6 seconds. An example of a named selector use is $button = $this->find_button("Save changes"); if you are not sure about the element being available you always can wrap the find*() call in a try & catch.
    • For advanced usages, the spin method is defined in lib/behat/behat_base::spin, consider that all the contents of the closures passed to spin() can be executed more than once, so don't use irreversible actions that can invalidate the tests results (for example use find() methods but don't use click() methods)
  • If a test should be skipped under certain circumstances you can trigger a \Moodle\BehatExtension\Exception\SkippedException exception, it will get caught and the scenario using that step will be reported as skipped.
  • Chained steps are deprecated as of Moodle 3.0 - your behat step definition should not return an array of Given objects - e.g.
    • Bad
   function do_not_write_steps_like_this() {
       $steps = [
           new Given('I click on "' . get_string('testbutton') . '"')
       ];
       return $steps;
   }
    • Good
   function do_write_steps_like_this() {
       $this->execute('behat_general::i_click_on', [get_string('testbutton'), 'link']);
   }

If you create new steps definitions or tests you must run php admin/tool/behat/cli/util.php --enable to update the Behat config file before running vendor/bin/behat

Check list

New steps should be/have:

  • Implemented as public methods of a PHP class whose name must begin with 'behat_COMPONENTNAME_' prefix and with '.php extension (Enurse class name is uniquely identified)
  • Using the class name as filename (adding the '.php' extension) and extending MOODLEDIRROOT/lib/behat/behat_base.php (or MOODLEDIRROOT/lib/behat/behat_files.php if it's a repository or is files-related)
  • With a descriptive class name, for example the component name (it will be used when filtering steps definitions)
  • Stored in COMPONENTNAME/tests/behat/ directory or lib/tests/behat/ if is not part of any other component
  • Describe it's purpose in a single line inside the method doc comment, the size of the comment is not a problem
  • Describe the regular expression with the most appropriate tag inside the method doc comment:
    • @Given - A step to set up the initial context (for example the following "courses" exists)
    • @When - An action that provokes an event (for example I press the button "buttonname")
    • @Then - Checkings to ensure the outcomes are the expected (for example I should see "whatever")
  • Depending on the inputs your definition expects you must use a different regular expression:
    • If you expect a number: "(?P<info_about_what_you_expect_number>\d+)" (note that the regular expression is quoted between ")
    • If you expect a string or a text: "(?P<info_about_what_you_expect_string>(?:[^"]|\\")*)" Don't use text_selector_string and selector_string as info strings, they are reserved to selector types (note that the regular expression is quoted between ")
    • If you expect a table with key/value pairs (for example to fill a form): Finish your regular expression with : and provide info in the description about the contents of the table
    • If you expect a selector type: "(?P<selector_string>[^"]*)" or "(?P<text_selector_string>[^"]*)" depending on whether you want to use any selector or you want a text-based selector (more info about selectors in https://docs.moodle.org/dev/Acceptance_testing#Providing_values_to_steps)
    • If you expect a value from a closed list: (?P<info>firstchoice|secondchoice|asmanyasyouwant)
  • To make test writer's life better is good to include explicative info in the subexpressions of the regular expression about what the test writer is supposed to put in there (for example I expand "(?P<nodetext>(?:[^"]|\\")*)" node)
  • Is recommended to use the static part of the regular expression as the name of the method, using underscores instead of spaces (see current steps definitions)

Troubleshooting

How can you tell if Selenium is running?

Try going to http://localhost:4444/selenium-server/. If Selenium is not running, nothing will happen. You will get a time-out. If selenium is running, you will get a 404 error page saying powered by Jetty at the bottom.

How can I stop Selenium?

Go to the URL http://localhost:4444/selenium-server/driver/?cmd=shutDownSeleniumServer. I think when it works it outputs "OKOK".

How to pause behat to check browser html

You can use following step to pause behat execution. To resume execution, press Enter/Return key in console.

And I pause scenario execution

Behat tells me that lots of steps are undefined

  • You probably forgot the --config option in the behat command.
  • With opcache enabled, definitions are not loaded properly. Try a run with opcache disabled.

Behat is showing no output

With opcache enabled, definitions are not loaded properly. Enure opcache.save_comments is set to 1 or disable opcache.

Behat is running slow

Independent of machine speed, Behat can be slow because of internet speed, as MathJax filter is enabled by default and points to cdn. To use a local installation of MathJax, first download the full MathJax library from http://www.mathjax.org/. Then install it on a web server. Finally add following to config.php $CFG->forced_plugin_settings = array('filter_mathjaxloader' =>

   array('httpurl' => 'http://localhost/MathJax/MathJax.js',
         'httpsurl' => 'https://localhost/MathJax/MathJax.js'),

); $CFG->behat_extraallowedsettings = array('forced_plugin_settings');

Uncheck checkbox without javascript

Uncheck a checkbox doesn't work in non-javascript scenario. This is a known issue with MinkGoutteDriver, please use tag @javascript for the scenarios where you need to uncheck a checkbox.

Browser specific fixes

If you are interested in running behat in other browsers you might be interested on https://docs.moodle.org/dev/Acceptance_testing/Browsers

When running acceptance tests in conjunction with Selenium there is a fix for the navigation bar that gets applied in order to avoid errors arising from a bug in the webdrivers for those browsers.

The issue: When an acceptance test goes to interact with an element on the page it first ensures that the element in the view-port and if not scrolls the browser to get the element into the view-port. The browser scrolls the element only just inside the view-port. It doesn't however allow for any fixed position elements such as the navigation bar. What happens: In some situations these browsers scroll up to reach a button, however not enough as the button ends up behind the navigation bar and cannot be interacted with.

Our solution: We can not change browser driver behaviour so we have integrated a work around. When running acceptance tests we change the position attribute of the navigation bar from fixed to absolute. This is not ideal as it is not how the user experiences the site, however it allows us to run the full acceptance test suite against all browsers so we allowed it. A notice will be displayed when you start an acceptance test run if the browser specific fixes have been applied. See MDL-54589 / MDL-51881 / MDL-47734 / MDL-45231 for more details.


Using Docker to start selenium server

What is Docker

Docker is a app container, it's a kind of virtual machine, but only for one app, service, so you can download a docker image and run a selenium server without worry in how to configure selenium in your machine, one for chrome, others for firefox, you either don't need to install the browsers in your machine. To install docker follow this link; https://docs.docker.com/engine/installation/

Selenium docker images

There is many docker images available, for many browser, the complete list is in https://hub.docker.com/u/selenium/ for moodle you can use standalone version. You can download specific selenium version too, for example, for firefox, moodle recommend selenium 2.53.1, see: What version do I need?

so the command will be: docker run -d -p 4444:4444 selenium/standalone-firefox:2.53.1-beryllium to see all available version click in tags. For firefox you can find at: https://hub.docker.com/r/selenium/standalone-firefox/tags/

Change config.php file

In config.php file you must change the $CFG->behat_wwwroot= to your network card (NIC) ip address, you can't use localhost , 127.0.0.1, ... or selenium docker server will fail

Running @_file_upload

You need to map your moodle code at same place in selenium docker instance, else all the @_file_upload scenarios will fail.

See also