Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Acceptance testing/Migrating from Behat 2.5 to 3.x in Moodle: Difference between revisions

From MoodleDocs
(Rerun cache file name.)
(Changes.)
Line 3: Line 3:
[http://docs.behat.org/en/v3.0/ Behat 3] brings a lot of extensibility and modularity  for the price of backward compatibility break. This guide describes several issues you might face when updating your plugin acceptance test to the latest Moodle 3.1@dev version.
[http://docs.behat.org/en/v3.0/ Behat 3] brings a lot of extensibility and modularity  for the price of backward compatibility break. This guide describes several issues you might face when updating your plugin acceptance test to the latest Moodle 3.1@dev version.


=== Running test ===
=== Config variables ===
; 1. $CFG->behat_config: structure of $CFG->behat_config has been modified and you should now use
# ''$CFG->behat_config'': structure of $CFG->behat_config has been modified and you should now use
::<code php>
::<code php>
$CFG->behat_config = array(
$CFG->behat_config = array(
Line 27: Line 27:
</code>
</code>


; 2. $CFG->behat_profiles: This config variable has been introduced to avoid any breakage in future. This is simple to use and will get preference over $CFG->behat_config value to set profile values. All the values are optional, similar to $CFG->behat_config.
# ''$CFG->behat_profiles:'' This config variable has been introduced to avoid any breakage in future. This is simple to use and will get preference over $CFG->behat_config value to set profile values. All the values are optional, similar to $CFG->behat_config.
:: <code php>
:: <code php>
$CFG->behat_profiles = array(
$CFG->behat_profiles = array(
Line 41: Line 41:
);
);
</code>
</code>
; 3. Formats and Outputs: html and failed outputs are not maintained by behat. We are planning to include [https://github.com/dutchiexl/BehatHtmlFormatterPlugin BehatHTMLFormatter plugin] maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are
 
:1.  '''Single option per formatter''': Multiple formats should be sent as separate options with respective output option
=== Formats and outputs ===
html and failed outputs are not maintained by behat. We are planning to include [https://github.com/dutchiexl/BehatHtmlFormatterPlugin BehatHTMLFormatter plugin] maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are
# '''Single option per formatter''': Multiple formats should be sent as separate options with respective output option
::<code>
::<code>
     vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR
     vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR
     </code>
     </code>
:2. '''Terminal output is std''': For output to terminal use std
# '''Terminal output is std''': For output to terminal use std
::<code>
::<code>
       vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std
       vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std
     </code>
     </code>
:3. '''Formatter can be used once''': One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress
# '''Formatter can be used once''': One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress
::<code>
::<code>
       vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt
       vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt
       </code>
       </code>
:4. '''Custom formatters''': Following custom formatter are supported
# '''Custom formatters''': Following custom formatter are supported
::1. '''moodle_progress''': Prints Moodle branch information and dots for each step.
## '''moodle_progress''': Prints Moodle branch information and dots for each step.
::2. '''moodle_list''': List all scenarios.
## '''moodle_list''': List all scenarios.
::3. '''moodle_stepcount''': List all features with total steps in each feature file. Used for parallel run.
## '''moodle_stepcount''': List all features with total steps in each feature file. Used for parallel run.
::4. '''moodle_screenshot''': Take screenshot and core dump of each step. With following options you can dump either or both.
## '''moodle_screenshot''': Take screenshot and core dump of each step. With following options you can dump either or both.
::: * --format-settings '{"formats": "image"}'**: will dump image only
### --format-settings '{"formats": "image"}'**: will dump image only
::: * --format-settings '{"formats": "html"}'**: will dump html only.
### --format-settings '{"formats": "html"}'**: will dump html only.
::: * --format-settings '{"formats": "html,image"}'**: will dump both.
### --format-settings '{"formats": "html,image"}'**: will dump both.
::: * --format-settings '{"formats": "html", "dir_permissions": "0777"}'**
### --format-settings '{"formats": "html", "dir_permissions": "0777"}'**


;4. Rerun option doesn't accept filepath: Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun.
=== Running test ===
# Rerun option doesn't accept filepath: Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun.
<code>
<code>
:::vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml
:::vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml
Line 72: Line 75:
:::fi
:::fi
</code>
</code>
:: ''''Rerun for parallel run'''': Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this:
# ''''Rerun for parallel run'''': Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this:
::: 1. Add 3 profiles in your config for each run say: firefox_1, firefox_2, firefox_3 for 3 parallel runs
## Add 3 profiles in your config for each run say: firefox_1, firefox_2, firefox_3 for 3 parallel runs
::: 2. use --profile="firefox_{runprocess} --replace="{runprocess}" options for run.php
## use --profile="firefox_{runprocess} --replace="{runprocess}" options for run.php
::: NOTE: [https://github.com/Behat/Behat/issues/859 Behat issue] have been waiting for this to be resolved by including basepath to generate the unique key.
::: NOTE: [https://github.com/Behat/Behat/issues/859 Behat issue] have been waiting for this to be resolved by including basepath to generate the unique key.


:: ''''Exit code for parallel run has improved'''': Exit code now contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures.
# ''''Exit code for parallel run has improved'''': Exit code is now '''bitmasked''' contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures.
:: If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config:
:: If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config:
:::<code php>
:::<code php>
Line 89: Line 92:
</code>
</code>


;5. '''Failed output is different''': Failed output display step and line in file where it failed and not the scenario line number.
=== Failed output ===
'''Failed output is different''': Failed output display step and line in file where it failed and not the scenario line number.


=== Changes required in context file ===
=== Changes required in context file ===
; 1. Custom chained steps support: Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example following chained steps steps
# '''Custom chained steps support:''' Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example following chained steps steps
::<code php>
::<code php>
$steps = array(
$steps = array(
Line 108: Line 112:
</code>
</code>


; 2. TableNode: Table Node class has changed and requires following modifications:
# '''TableNode:''' Table Node class has changed and requires following modifications:
:# TableNode constructor has changed, and it requires least empty array().  
## TableNode constructor has changed, and it requires least empty array().  
:# addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode()
## addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode()
::<code php>
::<code php>
$rows = $options->getRows();
$rows = $options->getRows();
Line 120: Line 124:
</code>
</code>


; 3. Named selectors are depreciated: named selector is deprecated, you should either use '''named_exact''' or '''named_partial'''
# '''Named selectors are depreciated:''' named selector is deprecated, you should either use '''named_exact''' or '''named_partial'''
:;<code php>
:;<code php>
$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode);
$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode);
</code>
</code>

Revision as of 03:19, 22 March 2016

Migrating from Behat 2.5 to 3.x in Moodle

Behat 3 brings a lot of extensibility and modularity for the price of backward compatibility break. This guide describes several issues you might face when updating your plugin acceptance test to the latest Moodle 3.1@dev version.

Config variables

  1. $CFG->behat_config: structure of $CFG->behat_config has been modified and you should now use

$CFG->behat_config = array(

  'phantomjs' => array(
     'suites' => array (
         'default' => array(
             'filters' => array(
                'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs'
             ),
          ),
      ),
      'extensions' => array(
           'Behat\MinkExtension' => array(
                 'selenium2' => array(
                     'browser' => 'phantomjs',
                     'wd_host' => "http://127.0.0.1:4443/wd/hub"
                 )
             )
         )
    ),

);

  1. $CFG->behat_profiles: This config variable has been introduced to avoid any breakage in future. This is simple to use and will get preference over $CFG->behat_config value to set profile values. All the values are optional, similar to $CFG->behat_config.

$CFG->behat_profiles = array(

  'phantomjs' => array(
      'browser' => 'phantomjs',
      'tags' => '~@_file_upload&&~@_alert&&~@_bug_phantomjs',
      'wd_host' => 'http://127.0.0.1:4443/wd/hub',
      'capabilities' => array(
          'platform' => 'Linux',
          'version' => 2.1
      )
  )

);

Formats and outputs

html and failed outputs are not maintained by behat. We are planning to include BehatHTMLFormatter plugin maintained by Neal Vanmeert and write our custom failed formatter. Changes required to use formatter and outputs are

  1. Single option per formatter: Multiple formats should be sent as separate options with respective output option
    vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=/PATH/TO/SAVE/progress.txt --format=moodle_screenshot --out=/PATH/TO/SAVE/SCREENSHOT_DIR
   
  1. Terminal output is std: For output to terminal use std
     vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std
   
  1. Formatter can be used once: One formatter is supported for each run. Means if you want to output progress to file and terminal then it is not supported. You need to use 2 different formatter like moodle_progress and progress
     vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --format=moodle_progress --out=std --format=progress --out=/PATH/TO/SAVE/progress.txt
     
  1. Custom formatters: Following custom formatter are supported
    1. moodle_progress: Prints Moodle branch information and dots for each step.
    2. moodle_list: List all scenarios.
    3. moodle_stepcount: List all features with total steps in each feature file. Used for parallel run.
    4. moodle_screenshot: Take screenshot and core dump of each step. With following options you can dump either or both.
      1. --format-settings '{"formats": "image"}'**: will dump image only
      2. --format-settings '{"formats": "html"}'**: will dump html only.
      3. --format-settings '{"formats": "html,image"}'**: will dump both.
      4. --format-settings '{"formats": "html", "dir_permissions": "0777"}'**

Running test

  1. Rerun option doesn't accept filepath: Rerun should be used without the filepath, as the failed scenarios are now cached by behat and rerun.

vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml
exitcode=${PIPESTATUS[0]}
if [ "${exitcode}" -ne 0 ]; then
vendor/bin/behat --config /PATH/FOR/BEHAT_DATAROOT/behat/behat.yml --rerun
fi

  1. 'Rerun for parallel run': Rerun file is cached with a unique key which is generated using input arguments. As in parallel run they all are same, the rerun file gets overwritten by the last failing run. To avoid this:
    1. Add 3 profiles in your config for each run say: firefox_1, firefox_2, firefox_3 for 3 parallel runs
    2. use --profile="firefox_{runprocess} --replace="{runprocess}" options for run.php
NOTE: Behat issue have been waiting for this to be resolved by including basepath to generate the unique key.
  1. 'Exit code for parallel run has improved': Exit code is now bitmasked contains pass/fail information of each run in every bit (starting from 0th bit). Example: If you have 3 parallel runs, out of which Run 1 and 3 failed. Then exit code will be 5 (0b0101). This will help to identify which runs need to be executed again for avoiding random failures.
If you want to use specific path where failed scenarios should be saved for rerun then you can set the following config:

$CFG->behat_config= array(

   'default' => array(
       'testers' => array(
           'rerun_cache' => '/home/rajesh/Desktop/rerun'
       ),
   ),

);

Failed output

Failed output is different: Failed output display step and line in file where it failed and not the scenario line number.

Changes required in context file

  1. Custom chained steps support: Chained steps are not supported by Behat 3. We have added support for custom chained step via moodle-behat-extension, but planning to depreciate this in future. To make your context backward compatible you should replace chained steps with the api calls in your context. Example following chained steps steps

$steps = array(

           new Given('I click on "' . get_string('login') . '" "link" in the ".logininfo" "css_element"'),		
           new Given('I set the field "' . get_string('username') . '" to "' . $this->escape($username) . '"'),		
           new Given('I set the field "' . get_string('password') . '" to "'. $this->escape($username) . '"'),		
           new Given('I press "' . get_string('login') . '"')		
       );

can be replaced with

$behatforms = behat_context_helper::get('behat_forms'); $behatforms->i_set_the_field_to('Username', $this->escape($username)); $behatforms->i_set_the_field_to('Password', $this->escape($username));

  1. TableNode: Table Node class has changed and requires following modifications:
    1. TableNode constructor has changed, and it requires least empty array().
    2. addRow() api is deprecated, if you are modifying rows then save that in an array and create a new TableNode()

$rows = $options->getRows(); $newrows = array(); foreach ($rows as $k => $data) {

   $newrows[$k] = strtr($data, "äåö", "aao");

} $modifiedtable = = new TableNode($newrows);

  1. Named selectors are depreciated: named selector is deprecated, you should either use named_exact or named_partial

$this->find('named_partial', array('link', get_string('hide')), $nohideexception, $activitynode);