Running acceptance test: Difference between revisions
m (Add a brief introduction on how easy is to use moodle-docker locally.) |
(Major rewrite to remove cruft and out-of-date information) |
||
Line 1: | Line 1: | ||
== Really short version == | |||
If you are using the MDK tooling, you can simply run <tt>mdk behat</tt> to configure behat for the first time. | |||
== Short version == | == Short version == | ||
... | This is a quick walk through to get Behat running for the first time. | ||
=== Requirements === | |||
# Any recent OS with a valid Moodle installation of a [[Releases|supported version of Moodle]] | |||
# A recent Java Runtime Environment | |||
# Selenium | |||
# A recent browser (We support Firefox, Chrome as standard, but other browsers are possible) | |||
# The WebDriver implementation for your browser | |||
=== Setting up === | |||
There are a number of ways of configuring Behat on Moodle. This is one of the simplest. | |||
We recommend use of the [https://github.com/andrewnicols/moodle-browser-config moodle-browser-config] tool. This provides a number of standard browser configurations for you to use. This includes working configurations for Firefox, and Chrome in both standard, and headless variants. It also supports advanced configuration including custom parameters such as debugging, use of Microsoft Edge, and browsers using BrowserStack. | |||
==== Install the moodle-browser-config utility ==== | |||
These instructions assume that you keep your git repositories in <tt>/Users/nicols/git</tt>. You will probably want to use a different location. | |||
# Change directory to your git root, for example: | |||
cd /Users/nicols/git | |||
# Clone the moodle-browser-config repository | |||
git clone https://github.com/andrewnicols/moodle-browser-config | |||
# Open your Moodle installations <tt>config.php</tt> in your preferred editor, and require the tool's init.php: | |||
require_once('/Users/nicols/git/moodle-browser-config/init.php'); | |||
# | ==== Setting up Selenium ==== | ||
Generally we recommend use of Selenium, though this is not a fixed requirement. You can use the browser's driver implementation directly but this is harder to setup for the first time. | |||
Selenium is written in Java, and requires a recent version of the JRE to run. Please ensure that you have this installed prior to starting. | |||
Since Moodle 3.9.5 / 3.10.2 / 3.11.0 Moodle will work with any modern version of Selenium. At time of writing that is 3.141.59, and 4.0.0-beta-3. | |||
# Download the Selenium Server (Grid) from [https://www.selenium.dev/downloads/ https://www.selenium.dev/downloads/]. This is a single JAR file, put it anywhere handy. | |||
# Start Selenium: | |||
## Version 3.141.59: <tt>java -jar selenium-server-standalone-3.141.59.jar</tt> | |||
## Version 4.0.0 and later: <tt>java -jar selenium-server-4.0.0-beta-3.jar standalone</tt> | |||
==== Setting up your browsers ==== | |||
Selenium is just an intelligient wrapper to start, and manage your browser sessions. It doesn't actually include any web browsers itself. | |||
Moodle runs all behat tests against both Firefox and Chrome multiple times per day. Other combinations, including Microsoft Edge, are also supported. | |||
To use Behat, you will need a recent version of your preferred browser, as well as a ''driver'' for that browser. The driver is responsible for communication between Selenium (or Moodle directly) and the browser. | |||
Both the browser, and its driver, must be placed inside your <tt>$PATH</tt> - this may be somewhere like <tt>/usr/bin</tt>, <tt>/usr/local/bin</tt>, or perhaps a user bin directory like <tt>~/bin</tt> which is present in your <tt>$PATH</tt> | |||
===== Chrome ===== | |||
You can download Google Chrome from https://www.google.com.au/chrome | |||
You will need the [https://chromedriver.chromium.org/downloads correct version of the chromedriver] as per the documentation. Alternatively you can make use of my [https://github.com/andrewnicols/chromedriver-wrapper chromedriver-wrapper utility] which will inspect Chrome and then download and run the correct version of chromedriver for your version of Chrome. | |||
Either the <tt>chromedriver</tt> binary must be in a directory in your <tt>$PATH</tt>, or the <tt>chromedriver-wrapper/bin</tt> folder must be in your <tt>$PATH</tt>. | |||
===== Firefox ===== | |||
You can download Mozilla Firefox from https://www.mozilla.org/en-US/firefox/new/ | |||
You will need the [https://github.com/mozilla/geckodriver/releases correct version of geckodriver as per the [https://firefox-source-docs.mozilla.org/testing/geckodriver/Support.html documentation]. | |||
The <tt>geckodriver</tt> binary must be in a directory in your <tt>$PATH</tt>. | |||
==== 3. Set up Moodle ==== | ==== 3. Set up Moodle ==== | ||
# Create a new 'dataroot' area for files especially for behat | # Create a new 'dataroot' area for files especially for behat | ||
# | # Set the following in your Moodle <tt>config.php</tt>: | ||
$CFG->behat_dataroot = '/path/to/the/dataroot/you/created'; | |||
$CFG->behat_wwwroot = 'http://127.0.0.1/path/to/your/site'; | |||
$CFG->behat_prefix = 'beh_'; | |||
# We recommend that you also include the <tt>behat-browser-config</tt> if you have not done so already. | |||
require_once('/path/to/moodle-browser-config/init.php'); | |||
===== Notes about the behat_wwwroot ===== | |||
You will need to set the <tt>behat_wwwroot</tt> to your Moodle site, but it ''must'' use a different value to your <tt>$CFG->wwwroot</tt>. | |||
One common way to do this is to use <tt>127.0.0.1</tt> for behat, but <tt>localhost</tt> for standard use. Alternatively you can add an additional hostname in your <tt>/etc/hosts</tt> file and use this instead. | |||
If you use Docker, then you may be able to use <tt>host.docker.internal</tt> where your site is hosted on the docker host | |||
==== Configure Behat for Moodle ==== | |||
After setting your configuration, you can simply initialise behat: | |||
php admin/tool/behat/cli/init.php | |||
This will install all required Composer dependencies, install a new Moodle site, and put configuration in place | |||
When it finishes it will give advice on how to run Behat. | |||
==== | ==== Run Behat tests ==== | ||
Before running behat, ensure that your Selenium server is running. | |||
The easiest way to run behat, and test that everything works is by simply running it using the command provided when you | |||
initialised Behat. If you didn't make a note of it, you can just run the initialisation again. | |||
php admin/tool/behat/cli/init.php | |||
This will give you a command which you can then run. This command will run every behat scenario, which will take a considerable amount of time. This command will look a bit like this: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml | |||
To make this more useful you an combine it with flags, for example to only run certain <tt>tags</tt> or for a specific Behat Feature file, or Scenario. | |||
To run all features/scenarios which are tagged with <tt>mod_forum</tt>: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --tags=@mod_forum | |||
To run one specific feature file: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml `pwd`/mod/forum/tests/behat/private_replies.feature | |||
To run one specific scenario within a feature file: | |||
# To run the Scenario on line 38 of the file: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml `pwd`/mod/forum/tests/behat/private_replies.feature:38 | |||
To run one specific scenario by name: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --name="As a teacher I can see my own response" | |||
See the upstream documentation on Behat, and Gherkin filters for more information. | |||
===== Running using a different browser ===== | |||
The default browser in Behat is <tt>Firefox</tt>. To specify a different browser profile, you can add the <tt>--profile</tt> argument. For example, to use Chrome in Headless mode: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=headlesschrome | |||
== Prerequisite == | == Prerequisite == | ||
Before initializing acceptance test environment for running behat, you should ensure: | Before initializing acceptance test environment for running behat, you should ensure: | ||
# [[Acceptance_testing#Requirements Meet min. system requirements for running tests]] | # [[Acceptance_testing#Requirements|Meet min. system requirements for running tests]] | ||
# [[Acceptance_testing#Installation Have set min. config variable in config.php for behat]] | # [[Acceptance_testing#Installation|Have set min. config variable in config.php for behat]] | ||
# [[Acceptance_testing#Installation | # [[Acceptance_testing#Installation|Installed the Behat system]] | ||
== Overview == | == Overview == | ||
Line 202: | Line 293: | ||
To override behat step definitions so as to run behat with specified theme, you should create a contexts within '''/theme/{MYTHEME}/tests/behat/''' with prefix behat_theme_{MYTHEME}_ and suffixed with the context being overridden. For example, if you want to override behat_mod_forum context, then you should create a class /theme/{MYTHEME}/tests/behat/mod_forum/behat_theme_{MYTHEME}_behat_mod_forum.php | To override behat step definitions so as to run behat with specified theme, you should create a contexts within '''/theme/{MYTHEME}/tests/behat/''' with prefix behat_theme_{MYTHEME}_ and suffixed with the context being overridden. For example, if you want to override behat_mod_forum context, then you should create a class /theme/{MYTHEME}/tests/behat/mod_forum/behat_theme_{MYTHEME}_behat_mod_forum.php | ||
==== | ==== Disable behat context or features to run in theme suite ==== | ||
To | To disable specific contexts and features from being executed by a specific theme/suite you can create a <tt>/theme/{MYTHEME}/tests/behat/blacklist.json</tt> file with following format. | ||
<code> | <code> | ||
{ | { | ||
Line 216: | Line 308: | ||
} | } | ||
</code> | </code> | ||
The above will: | |||
# disable the use of step_definitions from <tt>behat_grade</tt> and <tt>behat_navigation</tt> while running theme suite; and | |||
# disable running of scenarios in <tt>auth/tests/behat/login.feature</tt> and <tt>grade/tests/behat/grade_hidden_items.feature</tt>. | |||
==== Override core behat selectors ==== | ==== Override core behat selectors ==== | ||
To override behat selectors in specific theme, you should create a class behat_theme_{MYTHEME}_behat_selectors in /theme/{MYTHEME}/tests/behat/behat_theme_{MYTHEME}_behat_selectors.php extending behat_selectors. | To override behat selectors in specific theme, you should create a class behat_theme_{MYTHEME}_behat_selectors in /theme/{MYTHEME}/tests/behat/behat_theme_{MYTHEME}_behat_selectors.php extending behat_selectors. | ||
=== Use php built in web server === | === Use php built in web server === | ||
Note: 'This is not a recommended approach because it is single-threaded and slow.' | |||
You can use php built-in-web server for executing behat runs. To do so: | You can use php built-in-web server for executing behat runs. To do so: | ||
# Open a command line interface and '''cd /to/your/moodle/dirroot''' | # Open a command line interface and '''cd /to/your/moodle/dirroot''' | ||
# | # Run the PHP Server: | ||
php -S localhost:8000 | |||
# Update your config.php to set: | |||
$CFG->behat_wwwroot = 'http://localhost:8000'; | |||
=== Define custom options for parallel runs === | === Define custom options for parallel runs === | ||
Line 253: | Line 358: | ||
=== Write new tests and behat methods === | === Write new tests and behat methods === | ||
If you want to write tests for your own integration, you can do so by creating new tests with format .feature. Follow instructions in [[Writing_acceptance_tests|this page]] to write new tests. | If you want to write tests for your own integration, you can do so by creating new tests with format .feature. Follow instructions in [[Writing_acceptance_tests|this page]] to write new tests. | ||
It is also possible to add new steps the moodle behat integration. In order to do so, you will have to create a new .php class with the prefix '''behat_'''. Copy the format from '''lib\behat\behat_base.php''', but set your class to extend the behat_base class instead of the MinkExtension. You can define new behat steps by declaring functions with the appropriate heading. | It is also possible to add new steps the moodle behat integration. In order to do so, you will have to create a new .php class with the prefix '''behat_'''. Copy the format from '''lib\behat\behat_base.php''', but set your class to extend the behat_base class instead of the MinkExtension. You can define new behat steps by declaring functions with the appropriate heading. | ||
You will not be able to use these steps and features right away. Check [[Running_acceptance_test#New_step_definitions_or_features_are_not_executed|this section]] for instructions on how to update the behat integration. | You will not be able to use these steps and features right away. Check [[Running_acceptance_test#New_step_definitions_or_features_are_not_executed|this section]] for instructions on how to update the behat integration. | ||
Line 265: | Line 370: | ||
If you follow the steps above, Behat will run with Chrome. | If you follow the steps above, Behat will run with Chrome. | ||
You can get it to run with other browsers. The basic idea is to expand the $CFG->behat_profiles array in config.php to list more browsers. | |||
You can use the <tt>moodle-browser-config</tt> tool to do so. This is the recommended way, and is detailed above. | |||
==== Manually configuring other browsers ===== | |||
You can get it to run with other browsers. The basic idea is to expand the $CFG->behat_profiles array in config.php to list more browsers. | You can get it to run with other browsers. The basic idea is to expand the $CFG->behat_profiles array in config.php to list more browsers. | ||
Line 293: | Line 404: | ||
=== Run tests directly in Chrome, with no Selenium === | === Run tests directly in Chrome, with no Selenium === | ||
Historically, | Historically, Behat required Selenium server, however browsers now make use of their own automation layer. For example, | ||
Firefox uses <tt>Geckodriver</tt> and Chrome uses <tt>Chromedriver</tt>. As a result the use of Selenium itself is now | |||
optional. | |||
The moodle-browser-config tool includes standard profiles to use these drivers directly and without the use of Selenium. | |||
The standard profiles are: | |||
# <tt>firefox</tt> Use Firefox via Selenium | |||
# <tt>headlessfirefox</tt> Use Firefox via Selenium, without displaying the GUI | |||
# <tt>geckodriver</tt> Use Firefox with Geckodriver directly | |||
# <tt>headlessgeckodriver</tt> Use Firefox with Geckodriver directly, without displaying the GUI | |||
# <tt>chrome</tt> Use Chrome via Selenium | |||
# <tt>headlesschrome</tt> Use Chrome via Selenium, without displaying the GUI | |||
# <tt>chromedriver</tt> Use Chrome with Chromedriver directly | |||
# <tt>headlesschromedriver</tt> Use Chrome with Chromedriver directly, without displaying the GUI | |||
To use the drivers directly, you must run the driver itself, for example to run Chromedriver: | |||
$ chromedriver | |||
To run geckodriver: | |||
$ geckodriver | |||
Note: geckodriver runs on port 4444 by default. You cannot geckodriver at the same time as selenium. | |||
After starting your preferred browser, you can then run behat and specify an alternative profile, for example: | |||
= | vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=geckodriver | ||
=== Headless browsers === | |||
If you have Moodle running on a remote (headless) server, but don't want to install a window manager, you can make use of a headless browser profile. | |||
The following are provided in the moodle-browser-config tool as standard: | |||
< | # <tt>headlessfirefox</tt> Use Firefox via Selenium, without displaying the GUI | ||
# <tt>headlessgeckodriver</tt> Use Firefox with Geckodriver directly, without displaying the GUI | |||
# <tt>headlesschrome</tt> Use Chrome via Selenium, without displaying the GUI | |||
# <tt>headlesschromedriver</tt> Use Chrome with Chromedriver directly, without displaying the GUI | |||
You can use a command such as the following to run it: | |||
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=headlesschrome | |||
=== Using Docker to start selenium server === | === Using Docker to start selenium server === | ||
==== Selenium docker images ==== | ==== Selenium docker images ==== | ||
There is many docker images available, for many browser combinations. The complete list is at https://hub.docker.com/u/selenium/ | |||
docker | Moodle uses the ''standalone'' version and any recent version with version 3.141.59 or higher is supported. | ||
An example usage is: | |||
docker run -d -p 4444:4444 selenium/standalone-firefox:latest | |||
==== Change config.php file ==== | ==== Change config.php file ==== | ||
In config.php file you must change the $CFG->behat_wwwroot | |||
localhost | In config.php file you must change the <tt>$CFG->behat_wwwroot</tt> to an address that can be reached from within the docker image. | ||
You cannot use <tt>localhost</tt> or <tt>127.0.0.1</tt>. | |||
On some more recent versions of Docker you can use <tt>http://host.docker.internal/</tt>, for example if my site is located at <tt>/sm</tt> on my development machine, I can set the following: | |||
$CFG->behat_wwwroot = 'http://host.docker.internal/sm'; | |||
=== Increasing timeouts === | === Increasing timeouts === | ||
Line 387: | Line 476: | ||
<code> | <code> | ||
Javascript code and/or AJAX requests are not ready after 10 seconds. | Javascript code and/or AJAX requests are not ready after 10 seconds. | ||
There is a Javascript error or the code is extremely slow. | There is a Javascript error or the code is extremely slow. | ||
</code> | </code> | ||
Sometimes this indicates a genuine problem with the code, but if you are using a slow computer, it | Sometimes this indicates a genuine problem with the code, but if you are using a slow computer, it may just mean that your browser did not finish generating parts of the UI before behat tried was finished. | ||
If you find that this happens regularly on different scenarios then you may want to increase the timeout: | |||
$CFG->behat_increasetimeout = 2; | |||
$CFG->behat_increasetimeout = 2; | |||
This will increase all the timeouts by a factor of 2; if that isn't sufficient, you could use 3. | This will increase all the timeouts by a factor of 2; if that isn't sufficient, you could use 3. | ||
Increasing timeouts will make tests run a bit slower (because there are points where Behat waits up to a timeout to make sure something doesn't happen) so don't do this unless you need to. | Increasing timeouts will make tests run a bit slower (because there are points where Behat waits up to a timeout to make sure something doesn't happen) so don't do this unless you need to. | ||
Note: This is usually an indicator that your development machine is not well tuned. A better option would be to find out where the bottleneck is. This is usually the database configuration. | |||
=== | == Troubleshooting == | ||
=== New step definitions or features are not executed === | |||
=== New step definitions or features are not executed === | |||
If you are adding new tests or steps definitions update the tests list | If you are adding new tests or steps definitions update the tests list | ||
<code> | <code> | ||
Line 446: | Line 523: | ||
If you are using chrome, you need to ensure that the driver matches the version of the installed chrome browser – which may change on OS updates/upgrades. Moodle or Selenium will not give the appropriate message – see [https://tracker.moodle.org/browse/MDL-67659/ MDL-67659]. One solution is the one suggested in the issue and use Andrew Nicols’ [https://github.com/andrewnicols/chromedriver-wrapper/ Chromedriver Wrapper] which will ensure you have the appropriate driver before running the tests. | If you are using chrome, you need to ensure that the driver matches the version of the installed chrome browser – which may change on OS updates/upgrades. Moodle or Selenium will not give the appropriate message – see [https://tracker.moodle.org/browse/MDL-67659/ MDL-67659]. One solution is the one suggested in the issue and use Andrew Nicols’ [https://github.com/andrewnicols/chromedriver-wrapper/ Chromedriver Wrapper] which will ensure you have the appropriate driver before running the tests. | ||
== External links == | == External links == |
Revision as of 08:04, 18 May 2021
Really short version
If you are using the MDK tooling, you can simply run mdk behat to configure behat for the first time.
Short version
This is a quick walk through to get Behat running for the first time.
Requirements
- Any recent OS with a valid Moodle installation of a supported version of Moodle
- A recent Java Runtime Environment
- Selenium
- A recent browser (We support Firefox, Chrome as standard, but other browsers are possible)
- The WebDriver implementation for your browser
Setting up
There are a number of ways of configuring Behat on Moodle. This is one of the simplest.
We recommend use of the moodle-browser-config tool. This provides a number of standard browser configurations for you to use. This includes working configurations for Firefox, and Chrome in both standard, and headless variants. It also supports advanced configuration including custom parameters such as debugging, use of Microsoft Edge, and browsers using BrowserStack.
Install the moodle-browser-config utility
These instructions assume that you keep your git repositories in /Users/nicols/git. You will probably want to use a different location.
- Change directory to your git root, for example:
cd /Users/nicols/git
- Clone the moodle-browser-config repository
git clone https://github.com/andrewnicols/moodle-browser-config
- Open your Moodle installations config.php in your preferred editor, and require the tool's init.php:
require_once('/Users/nicols/git/moodle-browser-config/init.php');
Setting up Selenium
Generally we recommend use of Selenium, though this is not a fixed requirement. You can use the browser's driver implementation directly but this is harder to setup for the first time.
Selenium is written in Java, and requires a recent version of the JRE to run. Please ensure that you have this installed prior to starting.
Since Moodle 3.9.5 / 3.10.2 / 3.11.0 Moodle will work with any modern version of Selenium. At time of writing that is 3.141.59, and 4.0.0-beta-3.
- Download the Selenium Server (Grid) from https://www.selenium.dev/downloads/. This is a single JAR file, put it anywhere handy.
- Start Selenium:
- Version 3.141.59: java -jar selenium-server-standalone-3.141.59.jar
- Version 4.0.0 and later: java -jar selenium-server-4.0.0-beta-3.jar standalone
Setting up your browsers
Selenium is just an intelligient wrapper to start, and manage your browser sessions. It doesn't actually include any web browsers itself.
Moodle runs all behat tests against both Firefox and Chrome multiple times per day. Other combinations, including Microsoft Edge, are also supported.
To use Behat, you will need a recent version of your preferred browser, as well as a driver for that browser. The driver is responsible for communication between Selenium (or Moodle directly) and the browser.
Both the browser, and its driver, must be placed inside your $PATH - this may be somewhere like /usr/bin, /usr/local/bin, or perhaps a user bin directory like ~/bin which is present in your $PATH
Chrome
You can download Google Chrome from https://www.google.com.au/chrome
You will need the correct version of the chromedriver as per the documentation. Alternatively you can make use of my chromedriver-wrapper utility which will inspect Chrome and then download and run the correct version of chromedriver for your version of Chrome.
Either the chromedriver binary must be in a directory in your $PATH, or the chromedriver-wrapper/bin folder must be in your $PATH.
Firefox
You can download Mozilla Firefox from https://www.mozilla.org/en-US/firefox/new/
You will need the correct version of geckodriver as per the [https://firefox-source-docs.mozilla.org/testing/geckodriver/Support.html documentation.
The geckodriver binary must be in a directory in your $PATH.
3. Set up Moodle
- Create a new 'dataroot' area for files especially for behat
- Set the following in your Moodle config.php:
$CFG->behat_dataroot = '/path/to/the/dataroot/you/created'; $CFG->behat_wwwroot = 'http://127.0.0.1/path/to/your/site'; $CFG->behat_prefix = 'beh_';
- We recommend that you also include the behat-browser-config if you have not done so already.
require_once('/path/to/moodle-browser-config/init.php');
Notes about the behat_wwwroot
You will need to set the behat_wwwroot to your Moodle site, but it must use a different value to your $CFG->wwwroot.
One common way to do this is to use 127.0.0.1 for behat, but localhost for standard use. Alternatively you can add an additional hostname in your /etc/hosts file and use this instead.
If you use Docker, then you may be able to use host.docker.internal where your site is hosted on the docker host
Configure Behat for Moodle
After setting your configuration, you can simply initialise behat:
php admin/tool/behat/cli/init.php
This will install all required Composer dependencies, install a new Moodle site, and put configuration in place
When it finishes it will give advice on how to run Behat.
Run Behat tests
Before running behat, ensure that your Selenium server is running.
The easiest way to run behat, and test that everything works is by simply running it using the command provided when you initialised Behat. If you didn't make a note of it, you can just run the initialisation again.
php admin/tool/behat/cli/init.php
This will give you a command which you can then run. This command will run every behat scenario, which will take a considerable amount of time. This command will look a bit like this:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml
To make this more useful you an combine it with flags, for example to only run certain tags or for a specific Behat Feature file, or Scenario.
To run all features/scenarios which are tagged with mod_forum:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --tags=@mod_forum
To run one specific feature file:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml `pwd`/mod/forum/tests/behat/private_replies.feature
To run one specific scenario within a feature file:
# To run the Scenario on line 38 of the file: vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml `pwd`/mod/forum/tests/behat/private_replies.feature:38
To run one specific scenario by name:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --name="As a teacher I can see my own response"
See the upstream documentation on Behat, and Gherkin filters for more information.
Running using a different browser
The default browser in Behat is Firefox. To specify a different browser profile, you can add the --profile argument. For example, to use Chrome in Headless mode:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=headlesschrome
Prerequisite
Before initializing acceptance test environment for running behat, you should ensure:
- Meet min. system requirements for running tests
- Have set min. config variable in config.php for behat
- Installed the Behat system
Overview
Acceptance tests (also known as behat), use Selenium server and can be run as:
- Single run: In single run, only one behat run is executed. So all features are executed in this single run.
- Parallel runs: (Since Moodle 3.0) Parallel runs allow dev's to execute multiple behat runs together. This was introduced to get acceptance tests results faster. To achieve this:
- Features are divided between multiple behat runs
- Symlinks behatrun{x} (x being the run process number), are created pointing to moodle directory, so site for run 1 is accessible via https://localhost/moodle/behatrun1
- Process number is included as suffix to $CFG->behat_prefix.
- Process number is suffixed to $CFG->behat_dataroot.
Step 1: Initialise acceptance test environment
Before running acceptance tests, environment needs to be initialised for acceptance testing.
Single run
For initialising acceptance tests for single run, above command is sufficient.
php admin/tool/behat/cli/init.php
Parallel runs
For initialising acceptance tests for parallel runs, you can use one of the following options
- -j=<number> or --parallel=<number> (required) Number of parallel behat run to initialise
- -m=<number> or --maxruns=<number> (optional) Max parallel site which should be initialised at one time. If your system is slow, then you can initialise sites in chucks.
- --fromrun=<number> (optional) Initialise site to run specified run from. Used for running acceptance tests on different vms
- --torun=<number> (optional) Initialise site to run specified run till. Used for running acceptance tests on different vms
- -o or --optimize-runs (optional) This option will split features with specified tags in all parallel runs, so they are executed first when parallel run gets executed.
- -a=<name> or --add-core-features-to-theme=<name> (optional) Since Moodle 3.2. Use this option to add all core features to specified themes (comma separated list of themes)
// Below command will initialise moodle to run 2 parallel tests.
php admin/tool/behat/cli/init.php --parallel=2
Step 2: Running acceptance test environment
Single run
Run either of the following commands. For more options vendor/bin/behat --help or http://docs.behat.org/guides/6.cli.html
vendor/bin/behat --config /path/to/your/CFG_behat_dataroot/behatrun/behat/behat.yml
You almost always want to limit the number of tests that are run. To run all the tests in one plugin:
vendor/bin/behat --config /path/to/your/CFG_behat_dataroot/behatrun/behat/behat.yml --tags mod_myplugin
To run all the tests in one .feature file:
vendor/bin/behat --config /path/to/your/CFG_behat_dataroot/behatrun/behat/behat.yml /path/to/moodle/mod/myplugin/tests/behat/testsomething.feature
To run a single scenario:
vendor/bin/behat --config /path/to/your/CFG_behat_dataroot/behatrun/behat/behat.yml /path/to/moodle/mod/myplugin/tests/behat/testsomething.feature:40
Here, '40' is the line-number of the feature file where the Scenario starts.
Parallel runs
For running parallel runs, use following command
php admin/tool/behat/cli/run.php
Following optional options are available for custom run:
- --feature Only execute specified feature file (Absolute path of feature file).
- --suite Features for specified theme will be executed.
- --replace Replace args string with run process number, useful for output and reruns.
- --fromrun Execute run starting from (Used for parallel runs on different vms)
- --torun Execute run till (Used for parallel runs on different vms)
- Behat options can be passed for filtering features/scenarios:
- In case you don't want to run Javascript tests, use the Behat tags option to skip them, --tags="~@javascript"
- In case you want to run specific scenario, use the Behat name option to run it, --name="Filter user accounts by role and cohort"
- In case you want to run specific feature file, use the Behat feature option to run it, --feature="/PATH/TO/MOODLE/admin/tests/behat/filter_users.feature"
Example: Initialise and run Behat tests for a custom plugin under a custom theme:
php admin/tool/behat/cli/init.php --parallel=3 --add-core-features-to-theme="mytheme" php admin/tool/behat/cli/run.php --tags="@tool_myplugin" --suite="mytheme"
Common options for running tests
Tests filters
With the --tags or the -name Behat options you can filter which tests are going to run or which ones are going to be skipped. There are a few tags that you might be interested in:
- @javascript: All the tests that runs in a browser using Javascript; they require Selenium to be running, otherwise an exception will be thrown.
- @_file_upload: All the tests that involves file uploading or any OS feature that is not 100% part of the browser. They should only be executed when Selenium is running in the same machine where the tests are running.
- @_alert: All the tests that involves Javascript dialogs (alerts, confirms...) are using a feature that is OS-dependant and out of the browser scope, so they should be tag appropriately as not all browsers manage them properly.
- @_switch_window: All the tests that are using the I switch to "NAME" window step should be tagged as not all browsers manage them properly.
- @_switch_iframe: All the tests that are using the I switch to "NAME" iframe steps should be tagged as it is an advanced feature and some browsers may have problems dealing with them
- @_cross_browser: All the tests that should run against multiple combinations of browsers + OS in a regular basis. The features that are sensitive to different combinations of OS and browsers should be tagges as @_cross_browser.
- @componentname: Moodle features uses the Frankenstyle component name to tag the features according to the Moodle subsystem they belong to.
Output formats
Since Moodle 3.1 option for output is:
--format=pretty --out=/path/to/pretty.txt --format=moodle_progress --out=std
Before Moodle 3.1 option for output was:
--format='moodle_progress,pretty' --out=',/path/to/pretty.txt'
Following output formats are supported:
- progress: Prints one character per step.
- pretty: Prints the feature as is.
- junit: Outputs the failures in JUnit compatible files.
- moodle_progress: Prints Moodle branch information and dots for each step.
- moodle_list: List all scenarios.
- moodle_stepcount: List all features with total steps in each feature file. Used for parallel run.
- moodle_screenshot: (since Moodle 3.1) Take screenshot and core dump of each step. With following options you can dump either or both.
- --format-settings '{"formats": "image"}'**: will dump image only
- --format-settings '{"formats": "html"}'**: will dump html only.
- --format-settings '{"formats": "html,image"}'**: will dump both.
- --format-settings '{"formats": "html", "dir_permissions": "0777"}'**
If you want to see the failures immediately (rather than waiting ~3 hours for all the tests to finish) then either use the -v option to output a bit more information, or change the output format using --format. Format 'pretty' (-f pretty) is sufficient for most cases, as it outputs each step outcomes in the command line making easier to see the progress.
Advance usage
Rerun failed scenarios
With slow systems or parallel run you might see some random failures, to rerun only failed scenarios (to eliminate random failures), use --rerun option
- Single run: --run="absolute_path_to_empty_file" (Behat will record failed scenarios in this file, and when run again only failed scenarios will be run)
- Parallel run: --rerun="absolute_path_to_empty_file_{runprocess}.txt --replace="{runprocess}" ({runprocess} will be replaced with the process number for recording fails in the specific run process).
Since Moodle 3.1 --rerun option don't accept any value, as it is handled internally by behat
Running behat with specified theme (Since Moodle 3.2)
You can run behat with any theme installed. To execute behat with specified theme use --suite=[classic | customtheme ] option, while running behat. By default the features in theme behat folder will be executed for the suite. But if you want to run all core features with the specific theme then initialise acceptance test with --add-core-features-to-theme={THEME_NAME}, e.g.
php admin/tool/behat/cli/init.php --add-core-features-to-theme=clean
The first time init runs can be time consuming as it creates a paralell set of tables to core, i.e. bh_user,bh_assign etc. After init the tests can be run with the alternative theme as follows.
vendor/bin/behat --suite=classic --tags="@enrol_foobar"
If there is no --suite= parameter it will fall back to the default boost theme. No = or quotes needed around the theme name.
Make sure that $CFG->theme is not set in your config.php.
Override behat core context for theme suite
To override behat step definitions so as to run behat with specified theme, you should create a contexts within /theme/{MYTHEME}/tests/behat/ with prefix behat_theme_{MYTHEME}_ and suffixed with the context being overridden. For example, if you want to override behat_mod_forum context, then you should create a class /theme/{MYTHEME}/tests/behat/mod_forum/behat_theme_{MYTHEME}_behat_mod_forum.php
Disable behat context or features to run in theme suite
To disable specific contexts and features from being executed by a specific theme/suite you can create a /theme/{MYTHEME}/tests/behat/blacklist.json file with following format.
{
"contexts": [
"behat_grade",
"behat_navigation",
],
"features": [
"auth/tests/behat/login.feature",
"grade/tests/behat/grade_hidden_items.feature",
]
}
The above will:
- disable the use of step_definitions from behat_grade and behat_navigation while running theme suite; and
- disable running of scenarios in auth/tests/behat/login.feature and grade/tests/behat/grade_hidden_items.feature.
Override core behat selectors
To override behat selectors in specific theme, you should create a class behat_theme_{MYTHEME}_behat_selectors in /theme/{MYTHEME}/tests/behat/behat_theme_{MYTHEME}_behat_selectors.php extending behat_selectors.
Use php built in web server
Note: 'This is not a recommended approach because it is single-threaded and slow.'
You can use php built-in-web server for executing behat runs. To do so:
- Open a command line interface and cd /to/your/moodle/dirroot
- Run the PHP Server:
php -S localhost:8000
- Update your config.php to set:
$CFG->behat_wwwroot = 'http://localhost:8000';
Define custom options for parallel runs
You can set following custom config options for parallel runs via $CFG->behat_parallel_run. It's an array of options where 1st array is for 1st run and so on.
array (
'dbtype' => 'mysqli',
'dblibrary' => 'native',
'dbhost' => 'localhost',
'dbname' => 'moodletest',
'dbuser' => 'moodle',
'dbpass' => 'moodle',
'behat_prefix' => 'mdl_',
'wd_host' => 'http://127.0.0.1:4444/wd/hub',
'behat_wwwroot' => 'http://127.0.0.1/moodle',
'behat_dataroot' => '/home/example/bht_moodledata'
)
To set different selenium servers for parallel runs, you can use following. NOTE: Running parallel (headless) runs on different selenium servers avoid random focus failures.
$CFG->behat_parallel_run = array (
array ('wd_host' => 'http://127.0.0.1:4444/wd/hub'),
array ('wd_host' => 'http://127.0.0.1:4445/wd/hub'),
array ('wd_host' => 'http://127.0.0.1:4446/wd/hub'),
);
Write new tests and behat methods
If you want to write tests for your own integration, you can do so by creating new tests with format .feature. Follow instructions in this page to write new tests.
It is also possible to add new steps the moodle behat integration. In order to do so, you will have to create a new .php class with the prefix behat_. Copy the format from lib\behat\behat_base.php, but set your class to extend the behat_base class instead of the MinkExtension. You can define new behat steps by declaring functions with the appropriate heading.
You will not be able to use these steps and features right away. Check this section for instructions on how to update the behat integration.
For further information on how to create new steps definitions, check Acceptance testing/Custom acceptance steps.
Running acceptance tests with different browser
If you follow the steps above, Behat will run with Chrome.
You can get it to run with other browsers. The basic idea is to expand the $CFG->behat_profiles array in config.php to list more browsers.
You can use the moodle-browser-config tool to do so. This is the recommended way, and is detailed above.
Manually configuring other browsers =
You can get it to run with other browsers. The basic idea is to expand the $CFG->behat_profiles array in config.php to list more browsers.
$CFG->behat_profiles = array(
'chrome' => array(
'browser' => 'chrome',
'tags' => '@javascript',
)
);
More info about alternative browsers
Start multiple selenium servers
From command line Start selenium servers at different ports (say 4444, 4445, 4446 for 3 parallel runs)
java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4444 &
java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4445 &
java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port 4446
Alternative way of running three Selenium servers in parallel:
$ printf %d\\n {4444..4446} | xargs -n 1 -P 3 java -jar /path/to/your/selenium/server/selenium-server-standalone-2.NN.N.jar -port
Run tests directly in Chrome, with no Selenium
Historically, Behat required Selenium server, however browsers now make use of their own automation layer. For example, Firefox uses Geckodriver and Chrome uses Chromedriver. As a result the use of Selenium itself is now optional.
The moodle-browser-config tool includes standard profiles to use these drivers directly and without the use of Selenium. The standard profiles are:
- firefox Use Firefox via Selenium
- headlessfirefox Use Firefox via Selenium, without displaying the GUI
- geckodriver Use Firefox with Geckodriver directly
- headlessgeckodriver Use Firefox with Geckodriver directly, without displaying the GUI
- chrome Use Chrome via Selenium
- headlesschrome Use Chrome via Selenium, without displaying the GUI
- chromedriver Use Chrome with Chromedriver directly
- headlesschromedriver Use Chrome with Chromedriver directly, without displaying the GUI
To use the drivers directly, you must run the driver itself, for example to run Chromedriver:
$ chromedriver
To run geckodriver:
$ geckodriver
Note: geckodriver runs on port 4444 by default. You cannot geckodriver at the same time as selenium.
After starting your preferred browser, you can then run behat and specify an alternative profile, for example:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=geckodriver
Headless browsers
If you have Moodle running on a remote (headless) server, but don't want to install a window manager, you can make use of a headless browser profile.
The following are provided in the moodle-browser-config tool as standard:
- headlessfirefox Use Firefox via Selenium, without displaying the GUI
- headlessgeckodriver Use Firefox with Geckodriver directly, without displaying the GUI
- headlesschrome Use Chrome via Selenium, without displaying the GUI
- headlesschromedriver Use Chrome with Chromedriver directly, without displaying the GUI
You can use a command such as the following to run it:
vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml --profile=headlesschrome
Using Docker to start selenium server
Selenium docker images
There is many docker images available, for many browser combinations. The complete list is at https://hub.docker.com/u/selenium/
Moodle uses the standalone version and any recent version with version 3.141.59 or higher is supported.
An example usage is:
docker run -d -p 4444:4444 selenium/standalone-firefox:latest
Change config.php file
In config.php file you must change the $CFG->behat_wwwroot to an address that can be reached from within the docker image.
You cannot use localhost or 127.0.0.1.
On some more recent versions of Docker you can use http://host.docker.internal/, for example if my site is located at /sm on my development machine, I can set the following:
$CFG->behat_wwwroot = 'http://host.docker.internal/sm';
Increasing timeouts
You may see errors such as:
Javascript code and/or AJAX requests are not ready after 10 seconds.
There is a Javascript error or the code is extremely slow.
Sometimes this indicates a genuine problem with the code, but if you are using a slow computer, it may just mean that your browser did not finish generating parts of the UI before behat tried was finished.
If you find that this happens regularly on different scenarios then you may want to increase the timeout:
$CFG->behat_increasetimeout = 2;
This will increase all the timeouts by a factor of 2; if that isn't sufficient, you could use 3.
Increasing timeouts will make tests run a bit slower (because there are points where Behat waits up to a timeout to make sure something doesn't happen) so don't do this unless you need to.
Note: This is usually an indicator that your development machine is not well tuned. A better option would be to find out where the bottleneck is. This is usually the database configuration.
Troubleshooting
New step definitions or features are not executed
If you are adding new tests or steps definitions update the tests list
php admin/tool/behat/cli/util.php --enable
For parallel runs, all options for initialising parallel runs are valid
Tests are failing
If you followed all the steps and you receive an unknown weird error probably your browser version is not compatible with the Selenium version you are running. Please refer Working combinations to ensure you have correct Acceptance_testing/Browsers#Working_combinations_of_OS.2BBrowser.2Bselenium of them to run acceptance test.
The tests are failing, and the error message is completely useless
For example, it just says "Error writing to database" with no stack trace.
Add -vv command-line option to get very verbose output.
Errors during setup (before test are launched)
Typical errors are:
- Behat requirement not satisfied: http://127.0.0.1/m/stable_master is not available, ensure you specified correct url and that the server is set up and started.
- Behat is configured but not enabled on this test site.
In order to fix those errors please check that: the behat_dataroot has correct write permissions and that the $CFG->behat* variables are placed before the lib/setup.php include:
require_once(__DIR__ . '/lib/setup.php');
Selenium server is not running
Chrome specific
If you are using chrome, you need to ensure that the driver matches the version of the installed chrome browser – which may change on OS updates/upgrades. Moodle or Selenium will not give the appropriate message – see [https://tracker.moodle.org/browse/MDL-67659 MDL-67659]. One solution is the one suggested in the issue and use Andrew Nicols’ Chromedriver Wrapper which will ensure you have the appropriate driver before running the tests.
External links
- Vagrant profile with Moodle and Behat preconfigured: https://github.com/mackensen/moodle-hat
- Docker containers for Moodle Developers and Behat: https://github.com/moodlehq/moodle-docker
- Docker environment with Behat preconfigured : https://github.com/tobiga/docker_moodle_environment
Quick setup and testing using moodle-docker
This is a quick guide to help locally pass tests for your developments, before submitting them:
- Set up a default Moodle install using moodle-docker, with the database and Moodle version of your choice. See its README for more details. This will start some docker containers.
- Initialize behat to start testing with this command, from the webserver container:
php admin/tool/behat/cli/init.php
- Run the behat test of your choice, from the webserver container. For instance:
vendor/bin/behat --config /var/www/behatdata/behatrun/behat/behat.yml --tags tool_task
And you'll see something like:
Moodle 4.0dev (Build: 20210507), 0b47ea0a44a092f9000729ca7b15fff23111538b
Php: 7.3.26, mysqli: 5.7.21, OS: Linux 5.4.0-66-generic x86_64
Run optional tests:
Accessibility: No
Server OS "Linux", Browser: "firefox"
Started at 09-05-2021, 06:00
...................................................................... 70
...............................
12 scenarios (12 passed)
101 steps (101 passed)
2m53.27s (47.61Mb)