User talk:Rajesh Taneja

Jump to: navigation, search

Acceptance testing

Introduction

This page describes how we describe Moodle's functionalities and automatically test them.

Behat is a behavioural driven development (BDD) tool written in PHP, it can parse a human-readable list of sentences (called steps) and execute actions in a browser using Selenium or other tools to simulate user interactions.

For technical info: Behat integration

Quick start

This is a quick introduction to run a functional test (acceptance tests) using steps in a development/testing site, please DON'T USE THIS IN A PRODUCTION SITE.

Requirements

Moodle requirements

We put following requirement on behat run, to ensure acceptance site don't change data of existing moodle site.

1. Different wwwroot
A Moodle instance running on a webserver which you can connect to, just like you use for development and manual testing. To set different wwwroot to your 'normal' moodle install, you will need to configure apache to listen on two urls for the same moodle checkout. e.g.
$CFG->wwwroot = 'http://test.moodle.local/'
$CFG->behat_wwwroot = 'http://127.0.0.1/'
2. Separate database
To use different database for acceptance testing set following.
$CFG->behat_prefix = 'behat_'; // Different then $CFG->prefix;
3. Separate dataroot
To use different dataroot for acceptance testing set following.
$CFG->behat_dataroot = '/PATH/FOR/BEHAT_DATAROOT'; // Different then $CFG->dataroot;

Third party requirements

Following are required to execute Behat

1. Selenium server running in background
Download Latest version of selenium-server-standalone-x.xx.x.jar and use following command to run it. (where x.xx.x is the version number)
::: java -jar selenium-server-standalone-x.xx.x.jar
2. Behat environment initialise
Behat should be installed from composer and moodel site should be initilaised. Admin tool will attempt to do this.
::: php admin/tool/behat/cli/init.php
If above fails then you can install latest composer and execute following from terminal.
::: php composer.phar install
::: php admin/tool/behat/cli/init.php

Executing

Run the behat command which "php admin/tool/behat/cli/init.php" tells you to do

vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml

Advanced usage

  1. [Running behat using parallel run]
  2. [Run options]
  3. Test filters
  4. Output formats
  5. Run behat using different browsers

Troubleshooting

  1. Selenium server not running :
    1. Ensure you have selenium server running in another window
    2. Enter http://localhost:4444/wd/hub in browser and ensure page gets loaded
    3. Check config.php for wd_host and make sure it's set to proper port
  2. Not a behat site
    1. From your browser go to $CFG->behat_wwwroot value and ensure site is accessible.

Contributing

You can contribute the effort to automatically test all of Moodle's functionalities, as described in the guide to contributing automated tests. Here you can find information about how to write new features and how to write new step definitions if your changes requires a base change in the Moodle behat extension you can find here how to do it following the integration workflow: Acceptance_testing/Contributing_to_Moodle_behat_extension.

Browser specific fixes

If you are interested in running behat in other browsers you might be interested on https://docs.moodle.org/dev/Acceptance_testing/Browsers

When running acceptance tests in conjunction with Selenium there is a fix for the navigation bar that gets applied in order to avoid errors arising from a bug in the webdrivers for those browsers.

The issue: When an acceptance test goes to interact with an element on the page it first ensures that the element in the view-port and if not scrolls the browser to get the element into the view-port. The browser scrolls the element only just inside the view-port. It doesn't however allow for any fixed position elements such as the navigation bar. What happens: In some situations these browsers scroll up to reach a button, however not enough as the button ends up behind the navigation bar and cannot be interacted with.

Our solution: We can not change browser driver behaviour so we have integrated a work around. When running acceptance tests we change the position attribute of the navigation bar from fixed to absolute. This is not ideal as it is not how the user experiences the site, however it allows us to run the full acceptance test suite against all browsers so we allowed it. A notice will be displayed when you start an acceptance test run if the browser specific fixes have been applied. See MDL-51881 / MDL-47734 / MDL-45231 for more details.


References

  1. Writing feature files
  2. Adding step definition

Rajesh Taneja (talk) 12:13, 3 March 2016 (AWST)

Migrating from Behat 2.5 to 3.x in Moodle

Behat 3 brings a lot of extensibility and modularity for the price of backward compatibility break. This guide describes several issues you might face when updating your plugin acceptance test to the latest Moodle 3.1@dev version.

Running test

1. $CFG->behat_config
structure of $CFG->behat_config has been modified and you should now use
$CFG->behat_config = array(
   'phantomjs' => array(
      'suites' => array (
          'default' => array(
              'filters' => array(
                 'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs'
              ),
           ),
       ),
       'extensions' => array(
            'Behat\MinkExtension' => array(
                  'selenium2' => array(
                      'browser' => 'phantomjs',
                      'wd_host' => "http://127.0.0.1:4443/wd/hub"
                  )
              )
          )
     ),
  );
2. $CFG->behat_profiles
This config variable has been introduced to avoid any breakage in future. This is simple to use and will get preference over $CFG->behat_config value to set profile values. All the values are optional, similar to $CFG->behat_config.
$CFG->behat_profiles = array(
   'phantomjs' => array(
       'browser' => 'phantomjs',
       'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs',
       'wd_host' => 'http://127.0.0.1:4443/wd/hub',
       'capabilities' => array(
           'platform' => 'Linux',
           'version' => 2.1
       )
   )
);
3. Formats and Outputs
html and failed outputs are not maintained by behat. We are planning to include BehatHTMLFormatter plugin maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are
1. Single option per formatter: Multiple formats should be sent as separate options with respective output option
     vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR
    
2. Terminal output is std: For output to terminal use std
      vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std
    
3. Formatter can be used once: One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress
      vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt
      
4. Custom formatters: Following custom formatter are supported
1. moodle_progress: Prints Moodle branch information and dots for each step.
2. moodle_list: List all scenarios.
3. moodle_stepcount: List all features with total steps in each feature file. Used for parallel run.
4. moodle_screenshot: Take screenshot and core dump of each step. With following options you can dump either or both.
* --format-settings '{"formats": "image"}'**: will dump image only
* --format-settings '{"formats": "html"}'**: will dump html only.
* --format-settings '{"formats": "html,image"}'**: will dump both.
* --format-settings '{"formats": "html", "dir_permissions": "0777"}'**
4. Rerun option doesn't accept filepath
Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun.
:::vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml
:::exitcode=${PIPESTATUS[0]}
:::if [ "${exitcode}" -ne 0 ]; then
::::vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --rerun
:::fi
'Exit code for parallel run has improved': Exit code now contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures.
If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config:
$CFG->behat_config= array(
    'default' => array(
        'testers' => array(
            'rerun_cache' => '/home/rajesh/Desktop/rerun'
        ),
    ),
);
5. Failed output is different
Failed output display step and line in file where it failed and not the scenario line number.

Changes required in context file

1. Custom chained steps support
Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example following chained steps steps
$steps = array(
           new Given('I click on "' . get_string('login') . '" "link" in the ".logininfo" "css_element"'),		
           new Given('I set the field "' . get_string('username') . '" to "' . $this->escape($username) . '"'),		
           new Given('I set the field "' . get_string('password') . '" to "'. $this->escape($username) . '"'),		
           new Given('I press "' . get_string('login') . '"')		
);
can be replaced with
$behatforms = behat_context_helper::get('behat_forms');
$behatforms->i_set_the_field_to('Username', $this->escape($username)); $behatforms->i_set_the_field_to('Password', $this->escape($username));
2. TableNode
Table Node class has changed and requires following modifications:
  1. TableNode constructor has changed, and it requires least empty array().
  2. addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode()
$rows = $options->getRows();

$newrows = array(); foreach ($rows as $k => $data) {

   $newrows[$k] = strtr($data, "äåö", "aao");

}

$modifiedtable = = new TableNode($newrows);
3. Named selectors are depreciated
named selector is deprecated, you should either use named_exact or named_partial

$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode); <code>