Acceptance testing
Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.
Introduction
This page describes how we describe Moodle functionalities and how we automatically test all of this Moodle's functionalities.
Behat is a behavioural driven development (BDD) tool written in PHP, it can parse a human-readable list of sentences (called steps) and execute actions in a browser using Selenium or other tools to simulate users interactions.
(This is not in standard Moodle yet: to play a bit with it now clone https://github.com/moodlehq/moodle-behat-features and follow the installation and usage instructions, you may need to hack a bit to pass all the tests, it depends on you Moodle site contens. Is an example of tool usage but not an example of good practices to create tests and specify fixtures)
How it works
Behat parses and executes features files which describes Moodle features (for example Post in a forum), each feature file is composed by many scenarios (for example Add a post to a discussion or Create a new discussion), and finally each scenario is composed by steps (for example I press "Post to Forum" or I should see "My post title"). When the feature file is executed, every step internally is translated into an PHP method and is executed.
This features are executed nightly in the HQ servers with all the supported databases (MySQL, PostgreSQL, MSSQL and Oracle) and with different browsers (Firefox, Internet Explorer, Safari and Chrome) to avoid regressions and test new functionalities.
Examples
- There is a closed list of steps to use in the features, a feature written with the basic (or low-level) steps looks like this:
@auth
Feature: Login
In order to login
As a moodle user
I need to be able to validate the username and password against moodle
Scenario: Login as an existing user
Given I am on "login/index.php"
When I fill in "username" with "admin"
And I fill in "password" with "moodle"
And I press "loginbtn"
Then I should see "Moodle 101: Course Name"
Scenario: Login as an unexisting user
Given I am on "login/index.php"
When I fill in "username" with "admin"
And I fill in "password" with "moodle"
And I press "loginbtn"
Then I should see "Moodle 101: Course Name"
Note that The 3 sentences below Feature: Login are only information about what we want to test.
These are simple scenarios, but most of Moodle functionalities would require a huge list of this steps to test a scenario, imagine a Add a post to a discussion scenario; you need to login, create a course, create a user and enrol it in the course... Most of this steps is not what we intend to test in a Post in a forum feature, Moodle provides extra steps to quickly set up the context required to test a Moodle feature, for example:
@mod_forum
Feature: Post in a forum
In order to add contents to a forum
As a moodle user
I need to be able to enter the contents and verify they are correctly shown
Background:
Given a "basic" moodle site
And the following "courses" exists
| fullname | shortname |
| Course 1 | C1 |
And the following "users" exists
| username | firstname | lastname | email |
| student1 | First | Student | stu1@asd.com |
And the following "enrolments" exists
| username | courseshortname | enrolmenttype |
| student1 | C1 | manual |
Scenario: Create a forum
Given I am logged as an "admin"
And I go to "Course 1" course
And I add a "forum" to section "1"
And I fill the form with
| field_name | value |
| Forum name | ForumTest1 |
| Forum description | Test description |
| Forum type | Standard forum for general use |
| Any other field | Value |
When I press "Save and display"
Then I should see "Test1"
And I should see "Add a new discussion topic"
Scenario: Add a post to a discussion
Given I run "Create a forum" scenario
And I go to "Course 1" course
And I follow "ForumTest1"
When I press "Add a new discussion topic"
And I fill the form with
| field_name | value |
| Subject | test subject |
| Message | message body |
And I press "Post to forum"
Then I should see "test subject"
Note that:
- Background is executed before each scenario execution
- Each scenario is executed in an isolated testing environment, so what you set up in an scenario (like the ForumTest1 forum in the example above) is cleaned up after the scenario execution
- The prefixes "Given", "When" and "Then" are only informative and they as used to define the context (Given), specify the action (When) and check the results (Then).
- In this example the Background section is used to set up the environment to distinct it from the test steps, but it can also be set in the Scenaio sections
Requirements
- Selenium server (formerly the Selenium RC Server) You can download it from http://seleniumhq.org/download/
- Other dependencies are managed by the behat-moodle composer installer
Installation
- Set up your Moodle installation and initialize your PHPUnit environment https://docs.moodle.org/23/en/Installing_Moodle and https://docs.moodle.org/dev/PHPUnit
- Clone the behat-moodle project https://github.com/dmonllao/behat-moodle and follow the README https://github.com/dmonllao/behat-moodle/blob/master/README.md
- In Moodle, set the 'behatpath' setting pointing to your behat-moodle/ root (setting under Site Administration -> Server -> System paths)
Usage
- Run your Selenium server with "java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar"
- Open another CLI and "cd /to/your/moodle/dirroot/" and initialize your PHPUnit environment
- php admin/tool/behat/cli/util.php --runtests # To run the whole list of tests, you can execute only a set of tests with --tags="mod_forum" option or add extra behat commands with --extra="--format junit --out /path/to/junit/reports"
Writting features
- Select the most appropriate Moodle component to include your test and create a COMPONENTNAME/tests/behat/an_example.feature
- Write your scenarios checking the list of available steps to include (Site Administration -> Development -> Acceptance testing) and maintaining a correct indentation
- The line above each scenario must specify the Moodle component in a frankensyle format to allow execution of specific sets of tests https://docs.moodle.org/dev/Frankenstyle
- Run the tests, you can specify a tag in both the .feature file and the test runner to limit the executed tests.
Fixtures
As seen in [examples] Moodle provides a way to quickly set up the contextual data (courses, users, enrolments...) that you need to properly test scenarios, this can be done using one of the site templates (TODO) or creating entities in the background section (common for all the steps) or in the "Given" part of your scenario. Note that this steps can only be used to set up the contextual data required to test the feature but they don't test what they are doing; for example, the "Given the following user exists" is not testing that Moodle is able to create a user, but to test that a user can add a blog entry you might want to use this step. For further info, acceptance tests are supposed to be black-boxed tests (the tester don't know about the internals of the application) and this steps are using internal Moodle data generators instead of running all the steps required to create a user or create a course, which speeds up the test execution. There are other features to test that all this elements can be properly created.
Available elements
Most of the available elements can only be created in relation to other elements, to hide the complexity of the Moodle internals (references by contexts, ids...) the references can also be done using more human-friendly mappings.
The examples below shows how to add elements referencing other elements, there are required fields to reference the elements, other attributes will be filled with random data if they are not specified.
- Course categories
- The required field is idnumber
- References between parent/children by it's idnumber with the "parent" field
Given the following "categories" exists: | name | parent | idnumber | | Category 1 | 0 | CAT1 | | Category 2 | CAT1 | CAT2 |
- Courses
- The required field is shortname
- Uses the category idnumber as category reference
Given the following "courses" exists: | fullname | shortname | category | format | | Course 1 | COURSE1 | CAT1 | topics | | Course 2 | COURSE2 | CAT2 | |
- Groups
- The required fields are course and idnumber
- Uses the course shortname as course reference
Given the following "groups" exists: | name | description | course | idnumber | | Group 1 | Anything | COURSE1 | GROUP1 |
- Groupings
- The required fields are course and idnumber
- Uses the course shortname as course reference
Given the following "groupings" exists: | name | course | idnumber | | Grouping 1 | COURSE1 | GROUPING1 | | Grouping 2 | COURSE1 | GROUPING2 |
- Users
- The required field is username (if password is not set username value will be used as password too)
Given the following "users" exists: | username | password | email | firstname | lastname | | testuser | testuser | asd@asd.com | Test | User |
- Course enrolments
- The required fields are user, course and role
- Uses the course shortname as course reference
- Uses the user username as user reference
- Uses the role shortname as role reference
Given the following "course enrolments" exists: | user | course | role | | testuser | COURSE1 | student |
- Group members
- The required fields are user and group
- Uses the group idnumber as group reference
- Uses the user username as user reference
Given the following "group members" exists: | user | group | | testuser | GROUP1 |
- Grouping groups
- The required fields are grouping and group
- Uses the group idnumber as group reference
- Uses the grouping idnumber as grouping reference
Given the following "grouping groups" exists: | grouping | group | | GROUPING1 | GROUP1 |
Links
Technical info: https://docs.moodle.org/dev/Behat_integration
