Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Acceptance testing: Difference between revisions

From MoodleDocs
Line 116: Line 116:
# Select the most appropriate Moodle component to include your test and create a COMPONENTNAME/tests/behat/FEATURENAME.feature file
# Select the most appropriate Moodle component to include your test and create a COMPONENTNAME/tests/behat/FEATURENAME.feature file
# Add a tag with the component name in Frankenstyle format (https://docs.moodle.org/dev/Frankenstyle) on the first line
# Add a tag with the component name in Frankenstyle format (https://docs.moodle.org/dev/Frankenstyle) on the first line
# Begin writting the user story of the feature:<!--
# Begin writting the user story of the feature:
   Feature: FEATURENAME
   Feature: FEATURENAME
     In order to ...
     In order to ...
     As a ...
     As a ...
     I need to ...-->
     I need to ...
# Think of different scenarios to ensure the feature works as expected
# Think of different scenarios to ensure the feature works as expected
# For each scenario you have thought of:
# For each scenario you have thought of:
## Think of the initial context you need, for example ''1 course with 2 students on it and an assignment'', and which steps do you need to follow (interacting with the browser) to verify the scenario works as expected
## Think of the initial context you need, for example ''1 course with 2 students on it and an assignment'', and which steps do you need to follow (interacting with the browser) to verify the scenario works as expected
## Check the available steps (Site Administration -> Development -> Acceptance testing) and set the initial context data (see https://docs.moodle.org/dev/Acceptance_testing#Fixtures for more info) and the steps to follow to verify all works as it should work. Remember to use ''Given'', ''When'' and ''Then'' in a way that reflects what the scenario is testing
## Check the available steps (Site Administration -> Development -> Acceptance testing) and set the initial context data (see https://docs.moodle.org/dev/Acceptance_testing#Fixtures for more info) and the steps to follow to verify all works as it should work. Remember to use ''Given'', ''When'' and ''Then'' in a way that reflects what the scenario is testing
## Copy the list of steps to the .feature file with the Scenario header:<!--
## Copy the list of steps to the .feature file with the Scenario header:
     Scenario: Short description of the scenario
     Scenario: Short description of the scenario
       Given step 1
       Given step 1
Line 132: Line 132:
       When step 4
       When step 4
       And step 5
       And step 5
       Then step 6-->
       Then step 6
# Run the tests, when running your new features/scenarios you can specify a 'wip' (work in progress) tag in both the line above the Scenario description and the test runner to execute only the new scenario instead of running the whole set of tests.
# Run the tests, when running your new features/scenarios you can specify a 'wip' (work in progress) tag in both the line above the Scenario description and the test runner to execute only the new scenario instead of running the whole set of tests.



Revision as of 07:13, 26 November 2012

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.


Introduction

This page describes how we describe Moodle's functionalities and how we automatically test all of them.

Behat is a behavioural driven development (BDD) tool written in PHP, it can parse a human-readable list of sentences (called steps) and execute actions in a browser using Selenium or other tools to simulate users interactions.

(This is not in standard Moodle yet: to play a bit with features files and Moodle clone you can install the proof of concept we worked on from https://github.com/moodlehq/moodle-behat-features and follow the installation and usage instructions, you may need to hack a bit to pass all the tests, it depends on you Moodle site contents. Is an example of tool usage but not an example of good practices to create tests and specify fixtures)

How it works

Behat parses and executes features files which describes Moodle's features (for example Post in a forum), each feature file is composed by many scenarios (for example Add a post to a discussion or Create a new discussion), and finally each scenario is composed by steps (for example  I press "Post to Forum" or I should see "My post title"). When the feature file is executed, every step internally is translated into an PHP method and is executed.

This features are executed nightly in the HQ servers with all the supported databases (MySQL, PostgreSQL, MSSQL and Oracle) and with different browsers (Firefox, Internet Explorer, Safari and Chrome) to avoid regressions and test new functionalities.

Examples

  • There is a closed list of steps to use in the features, a feature written with the basic (or low-level) steps looks like this:
 @auth
 Feature: Login
   In order to login
   As a moodle user
   I need to be able to validate the username and password against moodle
   
   Scenario: Login as an existing user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"
   
   Scenario: Login as an unexisting user
     Given I am on "login/index.php"
     When I fill in "username" with "admin"
     And I fill in "password" with "moodle"
     And I press "loginbtn"
     Then I should see "Moodle 101: Course Name"

Note that The 3 sentences below Feature: Login are only information about what we want to test.

These are simple scenarios, but most of Moodle's functionalities would require a huge list of this steps to test a scenario, imagine a Add a post to a discussion scenario; you need to login, create a course, create a user and enrol it in the course... Most of this steps is not what we intend to test in a Post in a forum feature, Moodle provides extra steps to quickly set up the context required to test a Moodle feature, for example:

 @mod_forum
 Feature: Post in a forum
   In order to add contents to a forum
   As a moodle user
   I need to be able to enter the contents and verify they are correctly shown
   
   Background:
     Given a "basic" moodle site
     And the following "courses" exists
       | fullname | shortname |
       | Course 1 | C1        |
     And the following "users" exists
       | username | firstname | lastname | email        |
       | student1 | First     | Student  | stu1@asd.com |
     And the following "enrolments" exists
       | username | courseshortname | enrolmenttype |
       | student1 | C1              | manual        |
   
   Scenario: Create a forum
     Given I am logged as an "admin"
     And I go to "Course 1" course
     And I add a "forum" to section "1"
     And I fill the form with
       | field_name        | value                          |
       | Forum name        | ForumTest1                     |
       | Forum description | Test description               |
       | Forum type        | Standard forum for general use |
       | Any other field   | Value                          |
     When I press "Save and display"
     Then I should see "Test1"
     And I should see "Add a new discussion topic"
   
   Scenario: Add a post to a discussion
     Given I run "Create a forum" scenario
     And I go to "Course 1" course
     And I follow "ForumTest1"
     When I press "Add a new discussion topic"
     And I fill the form with
       | field_name | value        |
       | Subject    | test subject |
       | Message    | message body |
     And I press "Post to forum"
     Then I should see "test subject"

Note that:

  • Background is executed before each scenario execution
  • Each scenario is executed in an isolated testing environment, so what you set up in an scenario (like the ForumTest1 forum in the example above) is cleaned up after the scenario execution
  • The prefixes "Given", "When" and "Then" are only informative and they as used to define the context (Given), specify the action (When) and check the results (Then).
  • In this example the Background section is used to set up the environment to distinct it from the test steps, but it can also be set in the Scenaio sections

Requirements

  • PHP 5.4
  • PHPUnit 3.6.0
  • Other dependencies are managed by the behat-moodle composer installer
  • Optional: If you want to run tests that involves Javascript you will also need Selenium, you can download it from http://seleniumhq.org/download/, named "Selenium server (formerly the Selenium RC Server)"

Installation

Usage

  • "cd /to/your/moodle/dirroot/"
  • If you also wants to test features that involves Javascript, start your Selenium server with "java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar"
  • php admin/tool/behat/cli/util.php --runtests # To run the whole list of tests (javascript tests excluded by default, see --with-javascript option)
    • You can include tests that involves Javascript tests with --with-javascript option (php admin/tool/behat/cli/util.php --runtests --with-javascript)
    • You can execute only a set of tests with --tags option (php admin/tool/behat/cli/util.php --runtests --tags="mod_forum")
    • You can add extra behat commands with the --extra option (php admin/tool/behat/cli/util.php --runtests --extra="--format failed") More info in http://docs.behat.org/guides/6.cli.html#command-line-tool-behat

Writting features

All Moodle components and plugins (including 3rd party plugins) can specify their tests in .feature files using all the available steps.

Once you decided which functionality you want to specify as a feature you should:

  1. Select the most appropriate Moodle component to include your test and create a COMPONENTNAME/tests/behat/FEATURENAME.feature file
  2. Add a tag with the component name in Frankenstyle format (https://docs.moodle.org/dev/Frankenstyle) on the first line
  3. Begin writting the user story of the feature:
 Feature: FEATURENAME
   In order to ...
   As a ...
   I need to ...
  1. Think of different scenarios to ensure the feature works as expected
  2. For each scenario you have thought of:
    1. Think of the initial context you need, for example 1 course with 2 students on it and an assignment, and which steps do you need to follow (interacting with the browser) to verify the scenario works as expected
    2. Check the available steps (Site Administration -> Development -> Acceptance testing) and set the initial context data (see https://docs.moodle.org/dev/Acceptance_testing#Fixtures for more info) and the steps to follow to verify all works as it should work. Remember to use Given, When and Then in a way that reflects what the scenario is testing
    3. Copy the list of steps to the .feature file with the Scenario header:
   Scenario: Short description of the scenario
     Given step 1
     And step 2
     And step 3
     When step 4
     And step 5
     Then step 6
  1. Run the tests, when running your new features/scenarios you can specify a 'wip' (work in progress) tag in both the line above the Scenario description and the test runner to execute only the new scenario instead of running the whole set of tests.

Tips

  • When is possible is better to check the test outcomes against the given data than against language strings, which are depending on the selected language.
  • The format of the .feature files is YAML which finds out the data hierarchy from the indentation of it's elements, so be sure that the elements are correctly nested and indentated using spaces

Fixtures

As seen in [examples] Moodle provides a way to quickly set up the contextual data (courses, users, enrolments...) that you need to properly test scenarios, this can be done using one of the site templates (TODO) or creating entities in the background section (common for all the steps) or in the "Given" part of your scenario. Note that this steps can only be used to set up the contextual data required to test the feature but they don't test what they are doing; for example, the "Given the following user exists" is not testing that Moodle is able to create a user, but to test that a user can add a blog entry you might want to use this step. For further info, acceptance tests are supposed to be black-boxed tests (the tester don't know about the internals of the application) and this steps are using internal Moodle data generators instead of running all the steps required to create a user or create a course, which speeds up the test execution. There are other features to test that all this elements can be properly created.

Available elements

Most of the available elements can only be created in relation to other elements, to hide the complexity of the Moodle internals (references by contexts, ids...) the references can also be done using more human-friendly mappings.

The examples below shows how to add elements referencing other elements, there are required fields to reference the elements, other attributes will be filled with random data if they are not specified.

  • Course categories
    • The required field is idnumber
    • References between parent/children by it's idnumber with the "parent" field
 Given the following "categories" exists:
   | name       | parent   | idnumber |
   | Category 1 | 0        | CAT1     |
   | Category 2 | CAT1     | CAT2     |
  • Courses
    • The required field is shortname
    • Uses the category idnumber as category reference
 Given the following "courses" exists:
   | fullname | shortname | category | format | 
   | Course 1 | COURSE1   | CAT1     | topics |
   | Course 2 | COURSE2   | CAT2     |        |
  • Groups
    • The required fields are course and idnumber
    • Uses the course shortname as course reference
 Given the following "groups" exists:
   | name    | description | course  | idnumber |
   | Group 1 | Anything    | COURSE1 | GROUP1   |
  • Groupings
    • The required fields are course and idnumber
    • Uses the course shortname as course reference
 Given the following "groupings" exists:
   | name       | course  | idnumber  |
   | Grouping 1 | COURSE1 | GROUPING1 |
   | Grouping 2 | COURSE1 | GROUPING2 |
  • Users
    • The required field is username (if password is not set username value will be used as password too)
 Given the following "users" exists:
   | username | password | email       | firstname | lastname |
   | testuser | testuser | asd@asd.com | Test      | User     |
  • Course enrolments
    • The required fields are user, course and role
    • Uses the course shortname as course reference
    • Uses the user username as user reference
    • Uses the role shortname as role reference
 Given the following "course enrolments" exists:
   | user     | course  | role    |
   | testuser | COURSE1 | student |
  • Group members
    • The required fields are user and group
    • Uses the group idnumber as group reference
    • Uses the user username as user reference
 Given the following "group members" exists:
   | user     | group  |
   | testuser | GROUP1 |
  • Grouping groups
    • The required fields are grouping and group
    • Uses the group idnumber as group reference
    • Uses the grouping idnumber as grouping reference
 Given the following "grouping groups" exists:
   | grouping  | group  |
   | GROUPING1 | GROUP1 |

Adding steps definitions

Each Moodle component and plugin (including 3rd party plugins) can add new steps definitions. If you are writing tests and you notice that you are repeating the same group of steps you might want to create a new step definition that allows you to substitute the group of steps for one single step, something like I add a forum post with "blablabla" as description for example; also you can create whole new steps using the APIs provided by Behat and Mink if what you need to do is not covered by any of the available steps.

New steps should be/have:

  • Implemented as public methods of a PHP class whose name must begin with 'behat_' prefix and with '.php extension
  • Using the class name as filename (adding the '.php' extension) and extending MOODLEDIRROOT/lib/behat/behat_base.php
  • With a descriptive class name, for example the component name (it will be used when filtering steps definitions)
  • Stored in COMPONENTNAME/tests/behat/ directory
  • Describe it's purpose in a single line inside the method doc comment
  • Describe the regular expression with the most appropriate tag inside the method doc comment:
    • @Given - A step to set up the initial context (for example the following "courses" exists)
    • @When - An action that provokes an event (for example I press the button "buttonname")
    • @Then - Checkings to ensure the outcomes are the expected (for example I should see "whatever")
  • Include info in the subexpressions of the regular expression about what the features writer is supposed to put in there (for example I expand "(?P<nodetext>[^"]*)" node)

You can use this example below or any of the existing steps definitions as a template.

  • auth/tests/behat/behat_auth.php
 class behat_auth extends behat_base {
     /**
      * Logs in the user. There should exist a user with the same value as username and password
      *
      * @Given /^I log in as "(?P<username>[^"]*)"$/
      */
     public function i_log_in_as($username) {
         return array(new Given('I am on homepage'),
             new Given('I follow "Login"'),
             new Given('I fill in "Username" with "'.$username.'"'),
             new Given('I fill in "Password" with "'.$username.'"'),
             new Given('I press "Login"'),
             new Given('I should see "You are logged in as"'));
     }
 }

Every time you list the available steps using the web interface or you run tests through CLI the steps definitions list and the feature files list are updated so the new steps are ready to use in the new feature files you create.

Links

Technical info: https://docs.moodle.org/dev/Behat_integration