Acceptance testing/Migrating from Behat 2.5 to 3.x in Moodle: Difference between revisions
(Reword to clarify) |
m (Note about intent to not migrate this page to moodledev.io) |
||
(15 intermediate revisions by 6 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:WillNotMigrate}} | |||
== Migrating from Behat 2.5 to 3.x in Moodle == | == Migrating from Behat 2.5 to 3.x in Moodle == | ||
Line 4: | Line 6: | ||
=== Config variables === | === Config variables === | ||
====behat_profiles==== | |||
Most of the configuration for a behat profile can be set in $CFG->behat_profiles. This configuration parameter is available since Moodle 2.9.6, Moodle 3.0.4, and Moodle 3.1. | |||
Here is a quick example that changes the browser from the default firefox to chrome on Linux systems (note that you have to have the [http://www.seleniumhq.org/download/ Chrome web driver installed] in your system path) | |||
<syntaxhighlight lang="php"> | |||
$CFG->behat_profiles = [ | |||
'default' => [ | |||
'browser' => 'chrome', | |||
'extensions' => [ | |||
'Behat\MinkExtension' => [ | |||
'selenium2' => [ | |||
'browser' => 'chrome', | |||
] | |||
] | |||
] | |||
] | |||
]; | |||
</syntaxhighlight> | |||
The settings available here include the browser name, tags, webdriver configuration, and capabilities: | |||
<syntaxhighlight lang="php"> | |||
$CFG->behat_profiles = array( | |||
'phantomjs' => array( | |||
'browser' => 'phantomjs', | |||
'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs', | |||
'wd_host' => 'http://127.0.0.1:4443/wd/hub', | |||
'capabilities' => array( | |||
'platform' => 'Linux', | |||
'version' => 2.1 | |||
) | |||
) | |||
); | |||
</syntaxhighlight> | |||
In certain situations more advanced configuration is required - for example specification of a proxy server to use. This configuration must be specified in $CFG->behat_config. | |||
Note: If a value is specified in both behat_config, and behat_profiles, the version in behat_profiles will take precedence | |||
==== behat_config ==== | ==== behat_config ==== | ||
Structure of $CFG->behat_config has been modified and you should now use | Structure of $CFG->behat_config has been modified and you should now use | ||
<syntaxhighlight lang="php"> | |||
$CFG->behat_config = array( | $CFG->behat_config = array( | ||
'phantomjs' => array( | 'phantomjs' => array( | ||
Line 26: | Line 65: | ||
), | ), | ||
); | ); | ||
</ | </syntaxhighlight> | ||
=== Formats and outputs === | === Formats and outputs === | ||
html and failed outputs are not maintained by behat. We are planning to include [https://github.com/dutchiexl/BehatHtmlFormatterPlugin BehatHTMLFormatter plugin] maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are | html and failed outputs are not maintained by behat. We are planning to include [https://github.com/dutchiexl/BehatHtmlFormatterPlugin BehatHTMLFormatter plugin] maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are | ||
====Single option per formatter==== | ====Single option per formatter==== | ||
Multiple formats should be sent as separate options with respective output option | Multiple formats should be sent as separate options with respective output option | ||
<syntaxhighlight lang="php"> | |||
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR | vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR | ||
</syntaxhighlight> | |||
====Terminal output is std==== | ====Terminal output is std==== | ||
For output to terminal use std | For output to terminal use std | ||
<syntaxhighlight lang="php"> | |||
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std | vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=std | ||
</syntaxhighlight> | |||
====Formatter can be used once==== | ====Formatter can be used once==== | ||
One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress | One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress | ||
<syntaxhighlight lang="php"> | |||
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt | vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt | ||
</syntaxhighlight> | |||
====Custom formatters==== | ====Custom formatters==== | ||
Following custom formatter are supported | Following custom formatter are supported | ||
Line 79: | Line 97: | ||
====Rerun option doesn't accept filepath==== | ====Rerun option doesn't accept filepath==== | ||
Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun. | Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun. | ||
< | <syntaxhighlight lang="php"> | ||
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml | |||
exitcode=${PIPESTATUS[0]} | |||
if [ "${exitcode}" -ne 0 ]; then | |||
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --rerun | |||
fi | |||
</ | </syntaxhighlight> | ||
====Rerun for parallel run==== | ====Rerun for parallel run==== | ||
Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this: | Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this: | ||
Line 95: | Line 113: | ||
Exit code is now '''bitmasked''' contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures. | Exit code is now '''bitmasked''' contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures. | ||
:: If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config: | :: If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config: | ||
<syntaxhighlight lang="php"> | |||
$CFG->behat_config= array( | $CFG->behat_config = array( | ||
'default' => array( | 'default' => array( | ||
'testers' => array( | 'testers' => array( | ||
Line 103: | Line 121: | ||
), | ), | ||
); | ); | ||
</ | </syntaxhighlight> | ||
=== Failed output === | === Failed output === | ||
Line 110: | Line 128: | ||
=== Changes required in context file === | === Changes required in context file === | ||
====Custom chained steps support==== | ====Custom chained steps support==== | ||
Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example following chained | Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example the following chained steps | ||
<syntaxhighlight lang="php"> | |||
$steps = array( | $steps = array( | ||
new Given('I click on "' . get_string('login') . '" "link" in the ".logininfo" "css_element"'), | new Given('I click on "' . get_string('login') . '" "link" in the ".logininfo" "css_element"'), | ||
Line 118: | Line 136: | ||
new Given('I press "' . get_string('login') . '"') | new Given('I press "' . get_string('login') . '"') | ||
); | ); | ||
</ | </syntaxhighlight> | ||
can be replaced with | |||
<syntaxhighlight lang="php"> | |||
$ | // Click on login link. | ||
$ | $this->execute('behat_general::i_click_on_in_the', | ||
$ | array(get_string('login'), 'link', '.logininfo', 'css_element')); | ||
</ | // Enter username and password. | ||
$this->execute('behat_forms::i_set_the_field_to', array('Username', $this->escape($username))); | |||
$this->execute('behat_forms::i_set_the_field_to', array('Password', $this->escape($username))); | |||
// Press log in button. | |||
$this->execute('behat_forms::press_button', get_string('login')); | |||
</syntaxhighlight> | |||
====TableNode==== | ====TableNode==== | ||
Line 130: | Line 153: | ||
# TableNode constructor has changed, and it requires least empty array(). | # TableNode constructor has changed, and it requires least empty array(). | ||
# addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode() | # addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode() | ||
<syntaxhighlight lang="php"> | |||
$rows = $options->getRows(); | $rows = $options->getRows(); | ||
$newrows = array(); | $newrows = array(); | ||
Line 137: | Line 160: | ||
} | } | ||
$modifiedtable = = new TableNode($newrows); | $modifiedtable = = new TableNode($newrows); | ||
</ | </syntaxhighlight> | ||
====Named selectors are depreciated==== | ====Named selectors are depreciated==== | ||
Named selector is deprecated, you should either use '''named_exact''' or '''named_partial''' | Named selector is deprecated, you should either use '''named_exact''' or '''named_partial''' | ||
<syntaxhighlight lang="php"> | |||
$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode); | $this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode); | ||
</ | </syntaxhighlight> | ||
[[Category:Behat]] |
Latest revision as of 15:59, 30 April 2024
Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable. |
Migrating from Behat 2.5 to 3.x in Moodle
Behat 3 brings a lot of extensibility and modularity for the price of backward compatibility break. This guide describes several issues you might face when updating your plugin acceptance test to the latest Moodle 3.1@dev version.
Config variables
behat_profiles
Most of the configuration for a behat profile can be set in $CFG->behat_profiles. This configuration parameter is available since Moodle 2.9.6, Moodle 3.0.4, and Moodle 3.1.
Here is a quick example that changes the browser from the default firefox to chrome on Linux systems (note that you have to have the Chrome web driver installed in your system path)
$CFG->behat_profiles = [
'default' => [
'browser' => 'chrome',
'extensions' => [
'Behat\MinkExtension' => [
'selenium2' => [
'browser' => 'chrome',
]
]
]
]
];
The settings available here include the browser name, tags, webdriver configuration, and capabilities:
$CFG->behat_profiles = array(
'phantomjs' => array(
'browser' => 'phantomjs',
'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs',
'wd_host' => 'http://127.0.0.1:4443/wd/hub',
'capabilities' => array(
'platform' => 'Linux',
'version' => 2.1
)
)
);
In certain situations more advanced configuration is required - for example specification of a proxy server to use. This configuration must be specified in $CFG->behat_config. Note: If a value is specified in both behat_config, and behat_profiles, the version in behat_profiles will take precedence
behat_config
Structure of $CFG->behat_config has been modified and you should now use
$CFG->behat_config = array(
'phantomjs' => array(
'suites' => array (
'default' => array(
'filters' => array(
'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs'
),
),
),
'extensions' => array(
'Behat\MinkExtension' => array(
'selenium2' => array(
'browser' => 'phantomjs',
'wd_host' => "http://127.0.0.1:4443/wd/hub"
)
)
)
),
);
Formats and outputs
html and failed outputs are not maintained by behat. We are planning to include BehatHTMLFormatter plugin maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are
Single option per formatter
Multiple formats should be sent as separate options with respective output option
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR
Terminal output is std
For output to terminal use std
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=std
Formatter can be used once
One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt
Custom formatters
Following custom formatter are supported
- moodle_progress: Prints Moodle branch information and dots for each step.
- moodle_list: List all scenarios.
- moodle_stepcount: List all features with total steps in each feature file. Used for parallel run.
- moodle_screenshot: Take screenshot and core dump of each step. With following options you can dump either or both.
- --format-settings '{"formats": "image"}'**: will dump image only
- --format-settings '{"formats": "html"}'**: will dump html only.
- --format-settings '{"formats": "html,image"}'**: will dump both.
- --format-settings '{"formats": "html", "dir_permissions": "0777"}'**
Running test
Rerun option doesn't accept filepath
Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun.
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml
exitcode=${PIPESTATUS[0]}
if [ "${exitcode}" -ne 0 ]; then
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behatrun/behat/behat.yml --rerun
fi
Rerun for parallel run
Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this:
- Add 3 profiles in your config for each run say: firefox_1, firefox_2, firefox_3 for 3 parallel runs
- use --profile="firefox_{runprocess} --replace="{runprocess}" options for run.php
- NOTE: Behat issue have been waiting for this to be resolved by including basepath to generate the unique key.
Exit code for parallel run is bitmasked
Exit code is now bitmasked contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures.
- If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config:
$CFG->behat_config = array(
'default' => array(
'testers' => array(
'rerun_cache' => '/home/rajesh/Desktop/rerun'
),
),
);
Failed output
Failed output is different: Failed output display step and line in file where it failed and not the scenario line number.
Changes required in context file
Custom chained steps support
Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example the following chained steps
$steps = array(
new Given('I click on "' . get_string('login') . '" "link" in the ".logininfo" "css_element"'),
new Given('I set the field "' . get_string('username') . '" to "' . $this->escape($username) . '"'),
new Given('I set the field "' . get_string('password') . '" to "'. $this->escape($username) . '"'),
new Given('I press "' . get_string('login') . '"')
);
can be replaced with
// Click on login link.
$this->execute('behat_general::i_click_on_in_the',
array(get_string('login'), 'link', '.logininfo', 'css_element'));
// Enter username and password.
$this->execute('behat_forms::i_set_the_field_to', array('Username', $this->escape($username)));
$this->execute('behat_forms::i_set_the_field_to', array('Password', $this->escape($username)));
// Press log in button.
$this->execute('behat_forms::press_button', get_string('login'));
TableNode
Table Node class has changed and requires following modifications:
- TableNode constructor has changed, and it requires least empty array().
- addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode()
$rows = $options->getRows();
$newrows = array();
foreach ($rows as $k => $data) {
$newrows[$k] = strtr($data, "äåö", "aao");
}
$modifiedtable = = new TableNode($newrows);
Named selectors are depreciated
Named selector is deprecated, you should either use named_exact or named_partial
$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode);