Running acceptance test

(Redirected from Acceptance testing)
Jump to: navigation, search

Prerequisite

Before initializing acceptance test environment for running behat, you should ensure:

  1. Acceptance_testing#Requirements Meet min. system requirements for running tests
  2. Acceptance_testing#Installation Have set min. config variable in config.php for behat
  3. Acceptance_testing#Installation Downloaded composer dependencies

Overview

Acceptance tests (also known as behat), use Selenium server and can be run as:

  1. Single run: In single run, only one behat run is executed. So all features are executed in this single run.
  2. Parallel runs: (Since Moodle 3.0) Parallel runs allow dev's to execute multiple behat runs together. This was introduced to get acceptance tests results faster. To achieve this:
    • Features are divided between multiple behat runs
    • Symlinks behatrun{x} (x being the run process number), are created pointing to moodle directory, so site for run 1 is accessible via https://localhost/moodle/behatrun1
    • Process number is included as suffix to $CFG->behat_prefix.
    • Process number is suffixed to $CFG->behat_dataroot.

Step 1: Initialise acceptance test environment

Before running acceptance tests, environment needs to be initialised for acceptance testing.

Single run

For initialising acceptance tests for single run, above command is sufficient.

php admin/tool/behat/cli/init.php

Parallel runs

For initialising acceptance tests for parallel runs, you can use one of the following options

  1. -j or --parallel (required) Number of parallel behat run to initialise
  2. -m or --maxruns (optional) Max parallel site which should be initialised at one time. If your system is slow, then you can initialise sites in chucks.
  3. --fromrun (optional) Initialise site to run specified run from. Used for running acceptance tests on different vms
  4. --torun (optional) Initialise site to run specified run till. Used for running acceptance tests on different vms
  5. -o or --optimize-runs (optional) This option will split features with specified tags in all parallel runs, so they are executed first when parallel run gets executed.
  6. -a or --add-core-features-to-theme (optional) Since Moodle 3.2. Use this option to add all core features to specified theme's (comma separated list of themes)
// Below command will initialise moodle to run 2 parallel tests.
php admin/tool/behat/cli/init.php --parallel=2

Step 2: Running acceptance test environment

Single run

Run either of the following commands. For more options vendor/bin/behat --help or http://docs.behat.org/guides/6.cli.html

vendor/bin/behat --config /path/to/your/CFG_behat_dataroot/behatrun/behat/behat.yml
php admin/tool/behat/cli/run.php

Parallel runs

For running parallel runs, use following command

php admin/tool/behat/cli/run.php

Following optional options are available for custom run:

  1. --feature Only execute specified feature file (Absolute path of feature file).
  2. --suite Features for specified theme will be executed.
  3. --replace Replace args string with run process number, useful for output and reruns.
  4. --fromrun Execute run starting from (Used for parallel runs on different vms)
  5. --torun Execute run till (Used for parallel runs on different vms)
  6. -a or --add-core-features-to-theme (optional) Since Moodle 3.2. Use this option to add all core features to specified theme's (comma separated list)
  7. Behat options can be passed for filtering features/scenarios:
    • In case you don't want to run Javascript tests, use the Behat tags option to skip them, --tags="~@javascript"
    • In case you want to run specific scenario, use the Behat name option to run it, --name="Filter user accounts by role and cohort"
    • In case you want to run specific feature file, use the Behat feature option to run it, --feature="/PATH/TO/MOODLE/admin/tests/behat/filter_users.feature"

Common options for running tests

Tests filters

With the --tags or the -name Behat options you can filter which tests are going to run or which ones are going to be skipped. There are a few tags that you might be interested in:

  • @javascript: All the tests that runs in a browser using Javascript; they require Selenium to be running, otherwise an exception will be thrown.
  • @_file_upload: All the tests that involves file uploading or any OS feature that is not 100% part of the browser. They should only be executed when Selenium is running in the same machine where the tests are running.
  • @_alert: All the tests that involves Javascript dialogs (alerts, confirms...) are using a feature that is OS-dependant and out of the browser scope, so they should be tag appropriately as not all browsers manage them properly.
  • @_switch_window: All the tests that are using the I switch to "NAME" window step should be tagged as not all browsers manage them properly.
  • @_switch_iframe: All the tests that are using the I switch to "NAME" window steps should be tagged as it is an advanced feature and some browsers may have problems dealing with them
  • @_cross_browser: All the tests that should run against multiple combinations of browsers + OS in a regular basis. The features that are sensitive to different combinations of OS and browsers should be tagges as @_cross_browser.
  • @componentname: Moodle features uses the Frankenstyle component name to tag the features according to the Moodle subsystem they belong to.

Output formats

Since Moodle 3.1 option for output is:

--format=pretty --out=/path/to/pretty.txt --format=moodle_progress --out=std

Before Moodle 3.1 option for output was:

--format='moodle_progress,pretty' --out=',/path/to/pretty.txt'

Following output formats are supported:

  1. progress: Prints one character per step.
  2. pretty: Prints the feature as is.
  3. junit: Outputs the failures in JUnit compatible files.
  4. moodle_progress: Prints Moodle branch information and dots for each step.
  5. moodle_list: List all scenarios.
  6. moodle_stepcount: List all features with total steps in each feature file. Used for parallel run.
  7. moodle_screenshot: (since Moodle 3.1) Take screenshot and core dump of each step. With following options you can dump either or both.
    1. --format-settings '{"formats": "image"}'**: will dump image only
    2. --format-settings '{"formats": "html"}'**: will dump html only.
    3. --format-settings '{"formats": "html,image"}'**: will dump both.
    4. --format-settings '{"formats": "html", "dir_permissions": "0777"}'**

If you want to see the failures immediately (rather than waiting ~3 hours for all the tests to finish) then either use the -v option to output a bit more information, or change the output format using --format. Format 'pretty' (-f pretty) is sufficient for most cases, as it outputs each step outcomes in the command line making easier to see the progress.

Advance usage

Rerun failed scenarios

With slow systems or parallel run you might see some random failures, to rerun only failed scenarios (to eliminate random failures), use --rerun option

  1. Single run: --run="absolute_path_to_empty_file" (Behat will record failed scenarios in this file, and when run again only failed scenarios will be run)
  2. Parallel run: --rerun="absolute_path_to_empty_file_{runprocess}.txt --replace="{runprocess}" ({runprocess} will be replaced with the process number for recording fails in the specific run process).

Since Moodle 3.1 --rerun option don't accept any value, as it is handled internally by behat

Running behat with specified theme (Since Moodle 3.2)

You can run behat with any theme installed. To execute behat with specified theme use --suite={THEME_NAME} option, while running behat. By default the features in theme behat folder will be executed for the suite. But if you want to run all core features with the specific theme then initialise acceptance test with -a option. For example, -a {THEME_NAME}

Override behat core context for theme suite

To override behat step definitions so as to run behat with specified theme, you should create a contexts within /theme/{MYTHEME}/tests/behat/ with prefix behat_theme_{MYTHEME}_ and suffixed with the context being overridden. For example, if you want to override behat_mod_forum context, then you should create a class /theme/{MYTHEME}/tests/behat/mod_forum/behat_theme_{MYTHEME}_behat_mod_forum.php

Blacklist behat context or features to run in theme suite

To blacklist contexts/ features to be executed by theme suite you should create a /theme/{MYTHEME}/tests/behat/blacklist.json file with following format. Following will not use step_definitions from behat_grade and behat_navigation while running theme suite. Also, scenarios in auth/tests/behat/login.feature and grade/tests/behat/grade_hidden_items.feature won't be executed with theme suite.

{
  "contexts": [
    "behat_grade",
    "behat_navigation",
  ],
  "features": [
    "auth/tests/behat/login.feature",
    "grade/tests/behat/grade_hidden_items.feature",
   ]
}

Override core behat selectors

To override behat selectors in specific theme, you should create a class behat_theme_{MYTHEME}_behat_selectors in /theme/{MYTHEME}/tests/behat/behat_theme_{MYTHEME}_behat_selectors.php extending behat_selectors.

Use php built in web server

You can use php built-in-web server for executing behat runs. To do so:

  1. Open a command line interface and cd /to/your/moodle/dirroot
  2. php -S localhost:8000 (This is the test site URL that moodle uses by default, if you want to use another one you can override it in config.php with $CFG->behat_wwwroot attribute; more info in https://docs.moodle.org/dev/Acceptance_testing#Advanced_usage or config-dist.php)
  3. Update $CFG->behat_wwwroot = localhost:8000; in config.php

Define custom options for parallel runs

You can set following custom config options for parallel runs via $CFG->behat_parallel_run. It's an array of options where 1st array is for 1st run and so on.

       array (
           'dbtype' => 'mysqli',
           'dblibrary' => 'native',
           'dbhost' => 'localhost',
           'dbname' => 'moodletest',
           'dbuser' => 'moodle',
           'dbpass' => 'moodle',
           'behat_prefix' => 'mdl_',
           'wd_host' => 'http://127.0.0.1:4444/wd/hub',
           'behat_wwwroot' => 'http://127.0.0.1/moodle',
           'behat_dataroot' => '/home/example/bht_moodledata'
       )

To set different selenium servers for parallel runs, you can use following. NOTE: Running parallel (headless) runs on different selenium servers avoid random focus failures.

    $CFG->behat_parallel_run = array (
        array ('wd_host' => 'http://127.0.0.1:4444/wd/hub'),
        array ('wd_host' => 'http://127.0.0.1:4445/wd/hub'),
        array ('wd_host' => 'http://127.0.0.1:4446/wd/hub'),
    );

Running acceptance tests with different browser

By default behat will run with Firefox browser through Selenium. By adding the following code to your config.php you can change the selected browser that is run when behat is invoked. You will need to run php admin/tool/behat/cli/init.php for changes to take effect. Then use --profile='chrome'.

$CFG->behat_profiles = array(
   'chrome' => array(
       'browser' => 'chrome',
       'tags' => '@javascript',
   )
);

More info about alternative browsers

Start multiple selenium servers

From command line Start selenium servers at different ports (say 4444, 4445, 4446 for 3 parallel runs)

java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4444 &
java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4445 &
java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4446

Using Docker to start selenium server

What is Docker

Docker is a app container, it's a kind of virtual machine, but only for one app, service, so you can download a docker image and run a selenium server without worry in how to configure selenium in your machine, one for chrome, others for firefox, you either don't need to install the browsers in your machine To install docker follow this link; https://docs.docker.com/engine/installation/

Selenium docker images

There is many docker images available, for many browser, the complete list is in https://hub.docker.com/u/selenium/ for moodle you can use standalone version. You can download specific selenium version too, for example, for firefox, moodle recommend selenium 2.53.1, see: What version do I need?

so the command will be:

docker run -d -p4444:4444 selenium/standalone-firefox:2.53.1-beryllium

to see all available version click in tags. For firefox you can find at: https://hub.docker.com/r/selenium/standalone-firefox/tags/

Change config.php file

In config.php file you must change the $CFG->behat_wwwroot= to your network card (NIC) ip address, you can't use localhost , 127.0.0.1, ... or selenium docker server will fail

NOTE

  1. Start the Selenium server (in case you want to run tests that involves Javascript)

Trouble shooting

New step definitions or features are not executed

If you are adding new tests or steps definitions update the tests list

php admin/tool/behat/cli/util.php --enable

For parallel runs, all options for initialising parallel runs are valid

Tests are failing

If you followed all the steps and you receive an unknown weird error probably your system's Firefox version is not compatible with the Selenium version you are running. Please refer Working combinations to ensure you have correct Acceptance_testing/Browsers#Working_combinations_of_OS.2BBrowser.2Bselenium of them to run acceptance test.

Disable acceptance test environment

if you want to prevent access to test environment

php admin/tool/behat/cli/util.php --disable

Note that if you have the HTTP_PROXY environment variable set, which you may have had to do to run composer, then you also need to set NO_PROXY=localhost.

External links