Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Acceptance testing for the Moodle App: Difference between revisions

From MoodleDocs
(Update migration status and path)
 
(30 intermediate revisions by 9 users not shown)
Line 1: Line 1:
From Moodle 3.7 it is possible to write Behat tests for mobile app features.
{{Template:Migrated|newDocId=/general/app/development/testing/acceptance-testing}}
{{Moodle App (Ionic 5)}}
In order to run tests that carry out automated functionality testing for the Moodle App, you can write [[Acceptance testing|Acceptance tests]]. This can be useful if you want to test plugins that are compatible with the app, or you're contributing to the app core. Behat tests for the app work the same way as tests for Moodle core, but they are not run as part of a normal Behat execution and there are some differences that we'll go through in this page.


== Summary ==
A key point is that these tests are run using the Moodle Behat infrastructure, and don't depend only on the app codebase. Therefore, you will need a Moodle development setup as described in [[Setting up development environment]].
 
It is now possible to create Behat tests that carry out automated functionality testing on the mobile app, for example so that you can test plugins that you may have written for the app.
 
By default, these do not run as part of a normal Behat run. This page tells you how to run the tests, and how to write them.
 
A key point is that these tests for some parts of the mobile app are '''included within the Moodle codebase''', not within the app codebase, because they are run using the Moodle Behat infrastructure. This is definitely appropriate for Moodle plugins that add app support. It may also be acceptable for tests of the app itself, but this is not yet agreed.


The main advantages of this approach are:
The main advantages of this approach are:
* It is easy for third-party plugin authors to create tests for app features in exactly the same way that they create tests for website features.
* It is easy for third-party plugin authors to create tests for app features in exactly the same way that they create tests for website features.
* Where institutions run tests automatically, it should be relatively easy to include some app tests within the existing approach.
* Where institutions run tests automatically, it should be relatively easy to include some app tests within the existing approach.
* This system does not require any mobile device hardware and should work on all common platforms.
* This system does not require any mobile device hardware and should work on all common platforms.


This system has been tested on Windows 7 and Ubuntu 18.04 LTS.
== Running Behat tests for the mobile app ==
=== Set up a mobile app development environment ===
First you will need to set up a mobile app development environment. There are two ways to do this: you can either set up your own environment manually (which will be useful if you intend to submit changes or bugfixes to the core app), or you can use Docker to set up a virtual environment.
However you set up the environment, if you update the app, you must re-run Behat init on the corresponding Moodle installation so that it knows about the newer app version.
==== Setting up the environment yourself ====
Follow the first part of the instructions on this page:
* [[Setting up your development environment for Moodle Mobile 2]]
You need to get as far as the part in section 5 where you open the app in the browser; this is what Behat will do. You don't need to complete the later steps.
* You will need to update this environment periodically, for example when a new version of the mobile app is released. Behat does not do this for you.
==== Setting up the environment using Docker ====
(For this to work, you must have a Docker installation and know roughly how to use it.)
You can run the app using a Docker image provided by Moodle HQ, with commands like these:
* Specific version 3.6.0
    docker run --rm -p 8100:8100 moodlehq/moodlemobile2:3.6.0
* Latest stable version:
    docker run --rm -p 8100:8100 moodlehq/moodlemobile2:latest
* Nightly build of the next release
    docker run --rm -p 8100:8100 moodlehq/moodlemobile2:next


=== Add the mobile app Behat configuration ===
== Installation ==
In order to run tests for the app, you will need to run both a Moodle site and the Moodle App.


You need to add one or two lines to your config.php to enable app testing. There are several ways to run the Ionic app and you will want to use the configuration for the way that you prefer.
The Moodle site should be version 3.9.7+, 3.10.4+ or newer (3.11, 4.0, etc.). You also need to install the [https://github.com/moodlehq/moodle-local_moodlemobileapp/ local_moodlemobileapp] plugin, using the version that corresponds with the version of the Moodle App that you're testing on. If you have tests for an older version, you can read the [[#Upgrading_tests_from_an_older_version|Upgrading tests from an older version]] section.


Broadly speaking the options are:
We recommend that you use [https://github.com/moodlehq/moodle-docker#use-containers-for-running-behat-tests-for-the-mobile-app moodle-docker], because it's configured to run mobile tests and you can skip reading this entire section. You won't even need to clone the app repository.
# Manage the Ionic app yourself
# Have Behat control the lifecycle (start/stop) of the ionic app


The recommended option is to launch the App yourself using Docker.
Nevertheless, if you still have to run the projects in your local machine, you can read the following instructions.
=== Configuring the Moodle site ===
You can learn how to run a Moodle site locally in [[Setting up development environment]].


==== Manually launch the app environment yourself ====
Remember to install the [https://github.com/moodlehq/moodle-local_moodlemobileapp/ local_moodlemobileapp] plugin with the same version that you're using for the mobile app.
=== Configuring the Moodle App ===
If you are going to modify the application code, you can learn how to run it locally in [[Setting up your development environment for the Moodle App]]. You only need to run the application in a browser, so you can skip the instructions for Android/iOS. Make sure to launch the application on the testing environment, running <code>npm run serve:test</code>.


When manually launching the app yourself, you will be responsible for starting the App before Behat starts, and stopping it when you finish your testing.
If you only intend to run the app with the goal of executing Behat tests, you can use [[Moodle App Docker Images|the Docker images for the Mobile App]]. Again, make sure that you're running them on the testing environment using the <code>-test</code> suffix.


The advantage of this approach is that you are in charge of bringing up and taking down the Ionic server, so you can do this efficiently, share a copy between parallel runs, etc. The disadvantage is that you do have to remember to do it; if the server isn't running, tests which use the app will fail.
However you set up the environment, if you change the version of the app you'll need to re-run the Behat init command so that your Moodle site knows about it.
 
=== Configuring Behat ===
To use this method then you need to add the following line to config.php:
In order to enable app testing, you need to add the following configuration to your site's <code>config.php</code> file:
 
<syntaxhighlight lang="php">
<code php>
$CFG->behat_ionic_wwwroot = 'http://localhost:8100';
$CFG->behat_ionic_wwwroot = 'http://localhost:8100';
</code>
</syntaxhighlight>
 
The url you use here must be reachable by your Moodle site, and the application needs to be served at this url when running tests and also when you initialise the Behat environment.
There are two ways to run the app yourself:
# Using Docker
# Launching ionic yourself
 
===== Using Docker (Recommended) =====
 
If you are using the Docker image, you just need to start the Docker container:
 
<code>
docker run -p 8100:8100 moodlehq/moodlemobile2
</code>
 
===== Using a local installation ======
 
If you have installed the development environment locally, you can launch it using <tt>ionic serve -b</tt>. After launching it you will see output like:
 
<code>
[OK] Development server running!
    Local: http://localhost:8100
    External: http://137.108.5.43:8100, http://192.168.56.1:8100
</code>
 
==== Let Behat launch the app environment ====
 
If you want Behat to launch the app environment for you, then you need to add the following line to config.php. This only works if you installed the app development environment locally, not if you are using Docker.
 
<code php>
$CFG->behat_ionic_dirroot = '/path/to/app/workspace/moodlemobile2';
</code>
 
This may be sufficient, but you need to be aware of a couple of facts about the Ionic server used for app testing:
 
* Depending on your computer, it may take about 3 minutes to start up.
* The server uses about 1GB RAM.
 
When you do this, the Ionic server will be started automatically when Behat runs a test that has the @app tag. It will be automatically terminated when the Behat test run finishes. If the test run includes multiple scenarios that use the app, they will all reuse the one server; it won't restart each time.
 
This is simple and convenient, but it is probably not a good approach for developers who frequently re-run a short Behat run (as you have to wait for it to start Ionic every time) or for complex systems that run Behat in parallel (as you may end up with multiple copies of Ionic eating up your RAM).
 
== Browser profiles ==
 
Mobile tests only run in Chrome, so you need to make sure you have a Chrome profile set up in your config.php Behat settings.
 
* See [[Running acceptance test]] for more information on profiles.
 
Behat will automatically run app tests (those with @app tag) only in a Chrome browser profile. So, if you run multiple browser tests, it won't waste time trying to run the app tests in each one.
 
== Behat init ==
 
After you have set up the config.php, you will need to re-run Behat init:
 
<code php>
php admin/tool/behat/cli/init.php
</code>
 
This is necessary because by default, Behat won't run app tests (those with @app tag) at all, since you didn't have it configured.
 
When you run Behat init, the system needs to know which version of the app you are running. This is used in order to select tests that only work on certain versions of the app (see below).
 
* If you specify behat_ionic_dirroot, files in this location will be used to determine the app version. (If you also specify behat_ionic_wwwroot, this will not be used to determine the app version, but you should ensure that the version of the running app is the same as the version of the code in behat_ionic_dirroot.)
* If you only specify behat_ionic_wwwroot, the version number will be taken from the running app, so you must ensure the app is running when you run the Behat init command, not only when you start tests. You will get an error if it isn't. (This version detection only works with app version 3.6.1 and above; for older versions you must specify behat_ionic_dirroot.)
 
The detected version number will be displayed in the output from the init command:


$ php admin/tool/behat/cli/init.php
The Moodle App [[Using the Moodle App in a browser|only works in Chromium-based browsers]], so mobile tests will be ignored if you are using any other browser. You can learn how to configure the browser used in your tests in the [[Running acceptance test]] page.
You are already using composer version 1.8.4 (stable channel).
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Nothing to install or update
Generating autoload files
Behat test environment already installed
'''Configured app tests for version 3.6.1'''
2.5 behat profile detected, automatically converted to current 3.x format
Acceptance tests environment enabled on http://localhost/core-moodle-github, to run the tests use:
vendor/bin/behat --config C:/mylocation/behat/behat.yml


If everything is configured properly, you should see "Configured app tests for version X.X.X" after running <code>admin/tool/behat/cli/init.php</code>.
== Running Behat ==
== Running Behat ==
To run mobile tests in Behat, simply launch Behat in the usual way. The app tests all have the <code>@app</code> tag, so if you want to run all the mobile tests you can use <code>--tags=@app</code>.


To run mobile tests in Behat, simply launch Behat in the usual way, but make sure you are using a Chrome profile. (Depending on your setup, this might mean using <tt>--profile=chrome</tt>.)
It is OK to combine mobile and web tests in the same run.
 
You can specify the scenarios to run as normal. The app tests all have the @app tag, so if you want to run all the mobile tests you can specify --tags=app, but you can also run any other set of scenarios. It is OK to combine app and normal tests in the same run.
 
== Writing tests ==
== Writing tests ==
This page assumes you already know all about [[Writing acceptance tests]] in general.
This page assumes you already know all about [[Writing acceptance tests]] in general.
=== Test structure ===
=== Test structure ===
Mobile app test scenarios should be marked <code>@app</code> and <code>@javascript</code> in addition to any other tags.


* Mobile app test scenarios should be marked <tt>@app</tt> and <tt>@javascript</tt> in addition to any other tags that may be required.
You are writing a normal Behat test and this is likely to require background steps, such as creating courses, users, groups, etc.
* If creating a feature file specifically for app tests, call it <tt>app_whatever.feature</tt> (i.e. use the <tt>app_</tt> prefix). This is not technically required, it's just for consistency.
 
You are writing a normal Behat test and this is likely to require background steps similar to any other Moodle Behat test, for example <tt>the following "courses" exist</tt>, and so on.
 
=== Start the app ===
=== Start the app ===
 
When you want to get started testing the application, you can use the following step to launch the application:
Once all necessary Moodle configuration steps (creating courses, users, groups, etc.) are done, use this Behat step to start the app:
<syntaxhighlight lang="gherkin">
 
Given I enter the app
<tt>Given '''I enter the app'''</tt>
</syntaxhighlight>
 
This will:
This will:
* Set up all the Moodle server settings to allow the Moodle App to connect.
* Restart the browser, this is needed to ensure that it doesn't use data from previous runs.
* Set the browser to a suitable phone size (you can change it with other steps if you want a tablet or a different size).
* Open the app in the test browser.
* Inject the necessary JavaScript code to support Behat testing.
* Skip the onboarding and enter the site url in the initial screens of the app, if necessary.


* Set up all the Moodle server settings to allow the mobile app to connect.
* Launch Ionic if necessary
* Restart the browser. This is needed to ensure it doesn't contain any stored data from previous app testing.
* Set the browser to a suitable phone size (you can change it later if you want a tablet or other size).
* Open the Ionic server address in the test Chrome browser.
* Install necessary JavaScript code in the page that supports Behat testing.
* Automatically enter the server URL into the app if necessary.


After this step completes, if it is the first time you ran the app inside this scenario, you will be left at the login screen. If you already logged in earlier, then you will be at the start page.
After this step completes, if it is the first time you ran the app inside this scenario, you will be at the login screen. If you logged in earlier, you will be at the start page.


You can use this step even when you are already in the app; this will restart it.
You can also use this step if you are already using the app and it will restart it.
=== Log in to the app ===
To log in, you can use the following step:
<syntaxhighlight lang="gherkin">
When I log in as "student"
</syntaxhighlight>
This is the same step that's used to log into standard Moodle, and it works in the app as well. You should have created the user in background steps, and it will log in using the text as both username and password.


=== Log in to the app ===
You will then be left at the start page.
=== Standard steps ===
Technically, you can use any standard Behat action in the app. However, most of them will probably not work as you expect because the app runs on a different environment. It is still a website, but it's built using [https://ionicframework.com/docs/ Ionic Framework].


To log in:
One important problem is that the app has a complex DOM, and previous pages that are "back" from your current page may still be present in the DOM. Which means that any steps that just look for the first matching element in the DOM are likely to look for elements on a page you're not even on.


<tt>When '''I log in as "student1"'''</tt>
Another issue is that Ionic relies heavily on [https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM the Shadow DOM], and most steps in standard Moodle are not prepared to handle that.


This is the same step as used to log into standard Moodle, but if you are on the app login page it will automatically work to log into the app instead. It will log in with the given username, using the same password as username. You will then be left at the start page.
For these and other reasons, there are some steps that have been implemented specifically for the app. You can distinguish them from others because most of them end with "in the app".


Having said that, here's a list of steps that work and you can use reliably.
* Any step you normally need to set up information in Moodle — For example, creating courses, users, etc.
* <code>I change viewport size to "{width}x{height}"</code> — You can use this step to simulate switching between portrait and landscape formats.
* <code>I pause</code> — This step works and it is very useful to debug your scenario.
=== Actions ===
=== Actions ===
<syntaxhighlight lang="gherkin">
When I press "Course 1" in the app
</syntaxhighlight>
This will click an element found using accessibility rules, so it could be visible text, content inside <code>aria-label</code>, content described by <code>ara-labelledby</code>, etc. It should work for links, buttons and other clickable elements.
<syntaxhighlight lang="gherkin">
When I press "Course 1" near "Unique text" in the app
</syntaxhighlight>
You can use this step to narrow matches if the text you're providing is duplicated throughout the page.


All the other app-specific Behat steps end with the words 'in the app' to distinguish them from the normal steps.  
The second value, "Unique text" in this example, should be unique on the page. Otherwise, you may have issues finding the element that you seek. The system will press the element matching your text that is nearest to the one found using the unique text.


<tt>And '''I press "Course 1" in the app'''</tt>
Nearest is defined in terms of the DOM rather than pixel position; it is based on the number of steps you would have to take up the tree from the candidate element before you get to a shared ancestor with the unique text.
 
<syntaxhighlight lang="gherkin">
This finds an element which contains either the visible text, or Aria label, 'Course 1' and clicks it. It should work for links, buttons and similar.
When I select "Item 1" in the app
 
When I select "Item 1" near "Unique text" in the app
You should be able to use this for almost any actions that would be carried out by pressing something - pressing a button, following a link, changing a checkbox, switching a switch, opening a dropdown, selecting something from the popup, etc.
When I unselect "Item 1" in the app
 
When I unselect "Item 1" near "Unique text" in the app
For buttons that are icons with no text, you can usually find them using the Chrome inspector - look for the 'aria-label' attribute.
</syntaxhighlight>
 
You can use these steps to select or unselect radio buttons, check boxes, and such.
You can press the main (bottom) menu buttons using this step. For example, the home button icon has the label 'home'.
 
* Exact matches (an element which contains only the specified text, or where the Aria label is exactly the specified text) will be preferred. If there are no exact matches, then partial ones (anything containing that text) will be considered.
* If there are multiple matches, or none, the step will fail. You can avoid this by specifying an exact match (provided there is only one exact match, this will not fail even if there are other partial matches) or by clicking on an icon instead of text.
* If the item you try to press is a label for some other form field (using <tt>ion-label</tt> and the <tt>aria-labelled-by</tt> attribute) then it will actually press the field; this is useful in the settings menus.
 
<tt>And '''I press "Course 1" near "Unique text" in the app'''</tt>
 
This is a variant of the above step which is useful when there are multiple elements with the same text on the page. The second value ('Unique text' in this example) should be some text that is unique on the page. The system will press the instance of 'Course 1' that is nearest to the supplied unique text.
 
(This is intended as a simpler alternative to the standard Behat steps that use the word 'in', such as <tt>I click on "X" "thing" in "Y" "css_element"</tt>. Those steps are complex and can be difficult to use. This one is not as generic but hopefully will handle most circumstances.)
 
* Nearest is defined in terms of the DOM rather than pixel position; it is based on the number of steps you would have to take up the tree from the candidate item before you get to a shared ancestor with the unique text.
 
<tt>And '''I set the field "field name" to "text value" in the app'''</tt>
 
This sets a text field to the given value. For the field name, you can use the placeholder text (exact match will be found first, otherwise partial match if any).
 
This works with single-line text fields, multi-line text fields with rich text editor switched off (textarea) and rich-text-editor fields.
 
* The normal version of this step supports various form fields, but in the app this only supports text fields at the moment. Use the press step (above) for other types of field.
* When used with a rich-text editor, you can include HTML tags in the value if necessary.
 
<tt>And '''I press the back button in the app'''</tt>
 
<tt>And '''I press the main menu button in the app'''</tt>
 
<tt>And '''I press the page menu button in the app'''</tt>


You could use the previous <code>I press</code> step as well, but in some cases you will notice that it doesn't work as you expect. This is due to some internal quirks of how Ionic renders these components, so prefer using this specific steps where possible.
<syntaxhighlight lang="gherkin">
When I set the field "field name" to "text value" in the app
When I set the field "field name" near "Unique text" to "text value" in the app
</syntaxhighlight>
This sets a text field with the given value. The same rules will apply to find the input element as for clicking, so using the input name will not suffice. This is in order to encourage accessibility best practices. The only difference with the previous step is that this only matches fillable elements such as <code><input></code>, <code><textarea></code> and elements with <code>contenteditable="true"</code>.
<syntaxhighlight lang="gherkin">
When I press the back button in the app
When I press the more menu button in the app
When I press the page menu button in the app
When I press the user menu button in the app
</syntaxhighlight>
These steps will press, respectively:
These steps will press, respectively:
* The '''back button''' — This is the left pointing arrow at top left of the app.
* The '''more menu''' button — This is the icon with at bottom right of the app.
* The '''page menu''' button, if present — This is the icon with the three dots at top right of the app.
* The '''user menu''' button, if present — This is the avatar button at top right of the app present on navigation level 1.


* The back button (the left pointing arrow at top left of the app).
* The main menu button (the '...' icon at bottom right of the app).
* The page menu button, if present (the '...' icon at top right of the app).
Note that both the main menu and page menu use a 'more' icon so they are annoying to activate with the generic press command.
<tt>And I switch to the browser tab opened by the app</tt>
<tt>And I close the browser tab opened by the app</tt>
These two steps are necessary if you want to test the transition between the app and browser (e.g. test 'Open in browser' links). For example, after pressing 'Open in browser' you can use the first step above, and then you will be able to use normal Moodle Behat steps to check the browser tab. Then when finished, use the second step above.
=== Tests ===
<tt>Then '''the header should be "Course 1" in the app'''</tt>


This checks the text of the current page header (orange stripe at top of page) in the app. It must be an exact match for the specified text.
<syntaxhighlight lang="gherkin">
When I switch to the browser tab opened by the app
When I close the browser tab opened by the app
</syntaxhighlight>
These two steps are necessary if you want to test the transition between the app and a browser.


For this scenario, <tt>I should see</tt> would also work, but this allows you to specifically check the header as opposed to the text appearing elsewhere on the page.
For example, after pressing "Open in browser" you can use the first step above, and you will be able to use normal Moodle Behat steps to work in the browser tab. Once you're finished, you can use the second step to go back to the app.
=== Assertions ===
Like actions, there are some Behat assertions that are specific to the app.
<syntaxhighlight lang="gherkin">
Then I should find "Course 1" in the app
Then I should find "Course 1" near "Unique text" in the app
Then I should not find "Course 1" in the app
Then I should not find "Course 1" near "Unique text" in the app
</syntaxhighlight>
These steps can be used to assert that the specified text exists somewhere in the app.


=== Standard test steps ===
Notice that the standard <code>I should see</code> step may not work in the app because of the Shadow DOM. This step will also search using accessibility rules, so text within <code>aria-label</code> or described with <code>aria-labelledby</code> will work as well.
 
<syntaxhighlight lang="gherkin">
You can use all the normal Moodle Behat test steps while carrying out app testing, but some of them don't work very well. The app has a complex DOM and previous pages that are 'back' from your current page may still be present in the DOM, which means that any steps that just look for the first matching element in the DOM are likely to look for elements on a page you're not even on.
Then the header should be "Course 1" in the app
 
</syntaxhighlight>
==== Before the app starts ====
This checks the text of the current page header. It must be an exact match for the specified text.
 
Before starting the app, you normally need to set up information in Moodle (e.g. creating a course and user). For this part of your test you can obviously use all the normal Moodle steps.
 
==== Useful, working steps ====
 
* <tt>I should see</tt> and <tt>I should not see</tt> are very useful for checking results.
* <tt>I change viewport size to "640x360"</tt> is a useful step if you need to simulate switching between portrait and landscape formats.
* <tt>I pause</tt> works and is very useful to debug your scenario.
 
==== Problematic steps ====
 
* The <tt>I reload the page</tt> step does not work correctly in the app and may leave your test in a mess. Use <tt>I enter the app</tt> if you want to reload the app.


You could have used the <code>I should find</code> step described previously, but this allows you to specifically check the header as opposed to anything in the page.
<syntaxhighlight lang="gherkin">
Then "Item 1" should be selected in the app
Then "Item 1" near "Unique text" should be selected in the app
Then "Item 1" should not be selected in the app
Then "Item 1" near "Unique text" should not be selected in the app
</syntaxhighlight>
You can use these steps to assert whether radio buttons, check boxes, and such are selected or not.
=== Leaving the app ===
=== Leaving the app ===
 
If you want to leave the app and go back to Moodle within a scenario, simply use a Moodle step that goes to a page. For example, use <code>I am on site homepage</code> or <code>I am on "Course 1" course homepage</code>.
If you want to leave the app and go back to Moodle within a scenario, simply use a Moodle step that goes to a page, such as <tt>I am on site homepage</tt> or <tt>I am on "Course 1" course homepage</tt>.


You only need to do this if you want to carry out actions within Moodle after using the app, within the scenario. At the end of your scenario, there is no need to explicitly leave the app; Moodle will automatically start the next scenario on the Moodle start page as usual.
You only need to do this if you want to carry out actions within Moodle after using the app, within the scenario. At the end of your scenario, there is no need to explicitly leave the app; Moodle will automatically start the next scenario on the Moodle start page as usual.
=== A complete example ===
=== A complete example ===
 
This example is a complete feature file that loads the app, opens a course, and asserts that the app is showing the course page:
This example is a complete feature file that loads the app, clicks on a course, and checks the app has now gone to the course page.
<syntaxhighlight lang="gherkin">
 
<code>
@app @javascript
@app @javascript
Feature: Test app (demo)
Feature: Test app (demo)
Line 300: Line 177:
     And the following "users" exist:
     And the following "users" exist:
       | username |
       | username |
       | student1 |
       | student  |
     And the following "course enrolments" exist:
     And the following "course enrolments" exist:
       | user    | course | role    |
       | user    | course | role    |
       | student1 | C1    | student |
       | student  | C1    | student |


   Scenario: Try going into the course
   Scenario: Try going into the course
     When I enter the app
     When I enter the app
     And I log in as "student1"
     And I log in as "student"
     And I press "Course 1" near "Course overview" in the app
     And I press "Course 1" near "Course overview" in the app
     Then the header should be "Course 1" in the app
     Then the header should be "Course 1" in the app
</code>
</syntaxhighlight>
 
You can find more complex examples looking at the [https://github.com/moodlehq/moodle-local_moodlemobileapp tests for the app core].
== Limitations ==
== Limitations ==
 
Using this approach, there are some limitations you should be aware of:
I have split the limitations of this approach into three categories, below.
* Lack of native functionality — Fundamentally, it is not possible to test behaviour specific to native devices because tests are run in a browser.
 
* Missing functionality — There are some known limitations and unsupported features, for example there is currently no obvious way to attach files. Some of these are possible, but they haven't been implemented yet. If something is missing for your use-case, you can submit feature requests in [https://tracker.moodle.org/browse/MOBILE the tracker] using the <code>Behat</code> component.
=== Fundamental limitations ===
 
* It is not possible to test behaviour specific to iOS, because tests run in Chrome. (Most iOS-specific failures are caused by problems with the Safari browser used on iOS, which isn't very good.)
* Device features such as the camera cannot be tested, because tests run in a browser and not on a device.
 
It's my firm belief that even given those restrictions on test coverage, being able to run Behat tests is massively beneficial compared to having to test everything by hand for every app release.
 
=== Extra steps that might be needed ===
 
* There is no obvious way to attach files.
 
Probably there are also other extra steps that would be useful - these would be discovered by trying to write tests. At the OU we plan to write Behat tests for our plugins once this feature is approved, so we will identify and add some extra steps at that point.
 
=== Testing the app itself ===
 
This system can absolutely be used to test features built into the app as well a plugins.
 
However, there is a limitation, which is that currently the Behat feature files must live in the Moodle project. This may be less appropriate for app features.
 
A future extension of this feature could use the behat_ionic_dirroot variable and automatically include feature files from a directory within the app's source tree. I think this would probably be quite simple but I have not yet tried it.
 
== Advanced ==
== Advanced ==
=== Versioning ===
=== Versioning ===
Behat tests can relate to particular versions of the mobile app. For these situations, there are two types of tags you can add to your scenario or feature:
* <code>@app_from{version}</code> — These will be included in every app matching the specified version and newer.
* <code>@app_upto{version}</code> — These will be included in every app matching the specified version and older.


The Behat tests are stored in the Moodle codebase, so they always relate to a particular Moodle version, but sometimes it might be necessary to have different tests for different versions of the mobile app. For example, you may be writing a test in Moodle 3.6, where the behaviour in the Moodle 3.6 app is different from behaviour in the Moodle 3.7 app (but both apps can connect to the server).
For these situations:


* In addition to the @app tag, add a version-specific tag to your scenario or feature.
You can use two-digit or three-digit version numbers. For example, you could use <code>@app_from4.0</code> or <code>@app_upto3.9.5</code>.
* There are two types of tag: '''@app_from3.7''' (include for every app version from 3.7 and newer) or '''@app_upto3.6.3''' (include for every app version up to 3.6.3, but not after that).
* You can use a two-digit or three-digit version number (3.6 or 3.6.1).
 
After adding a test with one of these tags (or changing the app version used for testing), make sure you re-run Behat init; it is the init step that decides which tests to include.


After changing the app version used for testing, make sure you re-run Behat init. It is the initialisation process that stores which version of the app you're using.
=== Testing against multiple app versions ===
=== Testing against multiple app versions ===
If you need to run tests against multiple versions of the app, you can do it in two ways:
# Update the code in the app workspace by checking out a different version.
# Maintain multiple copies of the mobile app workspace and switch between them by changing <code>config.php</code>.


If you need to run tests against multiple versions of the mobile app, you can do this in two ways:


# Update the code in the mobile app workspace (check out a different version). Then re-run Behat init and run the Behat tests again.
In both cases, you'll need to re-run the Behat init command and run the tests again.
# Maintain multiple copies of the mobile app workspace and switch between them by changing config.php. Then re-run Behat init and run the Behat tests again.
 
In both cases, because you need to run Behat init and change the Behat configuration, you cannot do this in parallel; if you want to run these tests in parallel you will also need separate Moodle installations with their own config.php, wwwroot, and dataroot.


Unfortunately, the only way to run this in parallel is to have separate Moodle installations with their own configurations.
=== Debugging tests ===
=== Debugging tests ===
If you insert a pause into your test and open the developer tools, you can debug the application like you would during development. You can learn how to do that in [[Using the Moodle App in a browser]].


If you insert a pause into your test and open the developer tools (F12), you can see log information in the console about which actions were carried out so far, and whether Behat is waiting for anything. Here is an example:
Additionally, you can see log information in the console about which Behat steps have been carried out so far, and whether Behat is waiting for anything. Here is an example:
 
<syntaxhighlight lang="bash">
<code>
VM649:391 BEHAT: 17:45:15.477 Action - Set field Username to: student2
VM649:391 BEHAT: 17:45:15.477 Action - Set field Username to: student2
VM649:391 BEHAT: 17:45:15.480 PENDING+: DELAY,dom-mutation
VM649:391 BEHAT: 17:45:15.480 PENDING+: DELAY,dom-mutation
Line 373: Line 225:
VM649:391 BEHAT: 17:45:16.607 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.607 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.653 PENDING-:  
VM649:391 BEHAT: 17:45:16.653 PENDING-:  
</code>
</syntaxhighlight>
While the test is paused, you can also carry out some of the app Behat steps manually by typing commands into the console, which is convenient if you're not quite sure what command would work. The commands are available in a global object called <code>behat</code>. Here are some examples:
<syntaxhighlight lang="javascript">
behat.setField('Password', 'student2');
behat.press('Log in', 'Forgotten');
behat.pressStandard('back');
</syntaxhighlight>
There are more functions in the object; try using the browser's autocomplete to see the options, or look at the source in [https://github.com/moodlehq/moodle-local_moodlemobileapp/blob/master/tests/behat/app_behat_runtime.js app_behat_runtime.js].
 
If you're using <code>moodle-docker</code>, remember that you can interact with the browser [https://github.com/moodlehq/moodle-docker#using-vnc-to-view-behat-tests using VNC]. With a VNC client you can view in real-time what behat is doing in the browser.
=== Writing custom steps ===
If you find something missing to test your code, you can always implement custom steps.
 
If you're writing a plugin, you can include a new class under <code>tests/behat/behat_{youpluginname}.php</code>. If you're working on application code, you can always update [https://github.com/moodlehq/moodle-local_moodlemobileapp/blob/master/tests/behat/behat_app.php behat_app.php] as well.
 
You can learn more about writing custom steps in the [[Writing new acceptance test step definitions]] page, and if you want to see how the steps that are specific to the app work, you should look into [https://github.com/moodlehq/moodle-local_moodlemobileapp/blob/master/tests/behat/behat_app.php behat_app.php] and [https://github.com/moodlehq/moodle-local_moodlemobileapp/blob/master/tests/behat/app_behat_runtime.js app_behat_runtime.js].
== Upgrading tests from an older version ==
If you wrote tests before the app started using Ionic 5 (in version 3.9.5), it is possible that your tests are not working any longer. This guide has been rewritten with the latest information, so make sure to read the rest of the document to see what changed.
 
In general though, upgrading your current tests should be mostly straightforward. Here are some overall things to keep in mind:
* Ionic 5 starts using [https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM the Shadow DOM], and that's the source of many issues for the previous version of the Behat tests. The built-in steps have been rewritten to keep that in mind, and mostly should work the same way. One important difference is that they look for content using accessibility rules, so some of your previous assumptions may not be working.
* Before this update, it was safe to use the standard <code>I should see</code> step. But given the problems mentioned, it is likely that it breaks down in many situations. Instead, you should use the new <code>I should find ... in the app</code> steps.
* Similar to the last point, some radio inputs and checkboxes that worked before are broken. You should be able to use the new <code>I select ... in the app</code> steps instead.
* If you were relying in xpath or css selectors before, they will probably not work anymore. Even if you try to patch them, these selectors don't pierce through the Shadow DOM. In any case, it's always better to use accessible locators in your test like a real user would, so you can use this opportunity to improve accessibility in your plugin.
* Pay special attention to any assertions such as <code>should not exist</code> that you have in your tests. These assertions will not fail, because the elements won't be found. But if you get an eventual bug when something is shown that shouldn't, those steps will probably not pick it up because the locators may have changed.
* If you were running your tests in CI, there is a new dependency even if you're only testing a plugin and not running the application tests. The test definitions have been moved from core to the [https://github.com/moodlehq/moodle-local_moodlemobileapp/ local_moodlemobileapp] plugin, and you should have it installed in your Moodle site running the tests. This was done in order to decouple application code from the core.
 
 
If you also need to upgrade your plugin, and not just the tests, check out the [[Adapt your Mobile plugins to Ionic 5]] page.
== Troubleshooting ==
=== General advice ===
If you are stuck with an error and you can't find a way to continue, here's a list of things you can do:
* Make sure you added <code>$CFG->behat_ionic_wwwroot = "http://localhost:8100";</code> (or equivalent) to your <code>config.php</code> file, and that url is reachable from the host where your Moodle site is running.
* Remember when you need to re-run <code>admin/tool/behat/cli/init.php</code>, and make sure that you see "Configured app tests for version X.X.X". When in doubt, just run it again; it may fix your problem.
* It is possible that your tests break if you're using an unstable version of the app. Try to use stable versions using the <code>master</code> branch if you're working with the source code or tagged releases if you're using Docker.
* Mobile Behat tests don't work well with XDebug, so if you're using it, turn it off in <code>php.ini</code> while running the tests. Also, remember to restart Apache if necessary.
=== Unable to load app version from http://moodleapp:8100/config.json ===
This message appears when the Moodle site is not able to reach the app. Make sure that the url is available from the host you're running the Behat commands from. Also make sure that the app is actually running at the specified url.
 
It's ok if the actual <code>/config.json</code> url doesn't work, that's actually a remnant from legacy code. The url that Moodle is actually looking for is <code>/assets/env.json</code>.
=== The plugins required by this course could not be loaded correctly... ===
This means either some activity on the course is not adapted to support the moodle app or there is a timeout in the request to your behat site.
 
To clear the timeout message, open the app in your [[Using_the_Moodle_App_in_a_browser|development browser]], open the Inspector, open the Application tab, select Clear storage, press Clear site data, close Inspector, close the tab with mobile site, re-open mobile site in new tab and log in. Then in a separate tab, log in to your behat site (you can find the url in <code>$CFG->behat_wwwroot</code> within your <code>config.php</code> file) and make sure you can get into the course without seeing the error.
=== Fatal error: Maximum execution time of 30 seconds exceeded in... ===
This means that your local site has not been updated/visited since an upgrade. Just go to your local behat site (you can find the url in <code>$CFG->behat_wwwroot</code> within your <code>config.php</code> file), log in as admin and run notifications, then visit a course. Do this step often to avoid timeouts!
=== Test fails because of the browser language ===
If your operating system is in a different language than English, the tests may fail.


While the test is paused you can also carry out some of the app Behat steps manually by typing commands into the console, which is convenient if you're not quite sure what command would work. Here are examples of the most useful commands:
Chrome does not have an easy way to force the browser language to English, so the best way to solve the issue is forcing the app default language to English.


* <tt>behat.setField('Password', 'student2')</tt>
To do so, just set the <code>forcedefaultlanguage</code> attribute to <code>"en"</code> in your <code>moodle.config.json</code> file in the app.
* <tt>behat.press('Log in', 'Forgotten')</tt>
=== Application build gets killed without any error information ===
* <tt>behat.pressStandard('back')</tt>
In some situations, it is possible that you see <code>Killed</code> in the console and a command suddenly stops without any further information. In these situations, make sure to check the [[#General_advice|General advice]] section, but it is possible that your computer is running out of memory.


There are a few other functions in the 'behat' object; try using the browser's autocomplete to see the options.
If you are running the scripts inside of a Docker container, make sure that Docker is allocated enough memory. If you are using Docker desktop (for example, on a Mac), you can inspect these settings under Preferences > Resources > Advanced > Memory.
=== [Mac] running moodle-docker commands show grep usage options and do nothing else ===
This is [https://github.com/moodlehq/moodle-docker/issues/188 a known issue] in moodle-docker for Mac. The workaround for now is just to explicitly initialize the <code>MOODLE_DOCKER_APP_RUNTIME</code> variable in your local environment.
[[Category: Quality Assurance]]
[[Category: Behat]]
[[Category: Mobile]]
[[Category: Moodle App Ionic 5]]

Latest revision as of 13:01, 14 July 2022

Important:

This content of this page has been updated and migrated to the new Moodle Developer Resources. The information contained on the page should no longer be seen up-to-date.

Why not view this page on the new site and help us to migrate more content to the new site!


In order to run tests that carry out automated functionality testing for the Moodle App, you can write Acceptance tests. This can be useful if you want to test plugins that are compatible with the app, or you're contributing to the app core. Behat tests for the app work the same way as tests for Moodle core, but they are not run as part of a normal Behat execution and there are some differences that we'll go through in this page.

A key point is that these tests are run using the Moodle Behat infrastructure, and don't depend only on the app codebase. Therefore, you will need a Moodle development setup as described in Setting up development environment.

The main advantages of this approach are:

  • It is easy for third-party plugin authors to create tests for app features in exactly the same way that they create tests for website features.
  • Where institutions run tests automatically, it should be relatively easy to include some app tests within the existing approach.
  • This system does not require any mobile device hardware and should work on all common platforms.


Installation

In order to run tests for the app, you will need to run both a Moodle site and the Moodle App.

The Moodle site should be version 3.9.7+, 3.10.4+ or newer (3.11, 4.0, etc.). You also need to install the local_moodlemobileapp plugin, using the version that corresponds with the version of the Moodle App that you're testing on. If you have tests for an older version, you can read the Upgrading tests from an older version section.

We recommend that you use moodle-docker, because it's configured to run mobile tests and you can skip reading this entire section. You won't even need to clone the app repository.

Nevertheless, if you still have to run the projects in your local machine, you can read the following instructions.

Configuring the Moodle site

You can learn how to run a Moodle site locally in Setting up development environment.

Remember to install the local_moodlemobileapp plugin with the same version that you're using for the mobile app.

Configuring the Moodle App

If you are going to modify the application code, you can learn how to run it locally in Setting up your development environment for the Moodle App. You only need to run the application in a browser, so you can skip the instructions for Android/iOS. Make sure to launch the application on the testing environment, running npm run serve:test.

If you only intend to run the app with the goal of executing Behat tests, you can use the Docker images for the Mobile App. Again, make sure that you're running them on the testing environment using the -test suffix.

However you set up the environment, if you change the version of the app you'll need to re-run the Behat init command so that your Moodle site knows about it.

Configuring Behat

In order to enable app testing, you need to add the following configuration to your site's config.php file:

$CFG->behat_ionic_wwwroot = 'http://localhost:8100';

The url you use here must be reachable by your Moodle site, and the application needs to be served at this url when running tests and also when you initialise the Behat environment.

The Moodle App only works in Chromium-based browsers, so mobile tests will be ignored if you are using any other browser. You can learn how to configure the browser used in your tests in the Running acceptance test page.

If everything is configured properly, you should see "Configured app tests for version X.X.X" after running admin/tool/behat/cli/init.php.

Running Behat

To run mobile tests in Behat, simply launch Behat in the usual way. The app tests all have the @app tag, so if you want to run all the mobile tests you can use --tags=@app.

It is OK to combine mobile and web tests in the same run.

Writing tests

This page assumes you already know all about Writing acceptance tests in general.

Test structure

Mobile app test scenarios should be marked @app and @javascript in addition to any other tags.

You are writing a normal Behat test and this is likely to require background steps, such as creating courses, users, groups, etc.

Start the app

When you want to get started testing the application, you can use the following step to launch the application:

Given I enter the app

This will:

  • Set up all the Moodle server settings to allow the Moodle App to connect.
  • Restart the browser, this is needed to ensure that it doesn't use data from previous runs.
  • Set the browser to a suitable phone size (you can change it with other steps if you want a tablet or a different size).
  • Open the app in the test browser.
  • Inject the necessary JavaScript code to support Behat testing.
  • Skip the onboarding and enter the site url in the initial screens of the app, if necessary.


After this step completes, if it is the first time you ran the app inside this scenario, you will be at the login screen. If you logged in earlier, you will be at the start page.

You can also use this step if you are already using the app and it will restart it.

Log in to the app

To log in, you can use the following step:

When I log in as "student"

This is the same step that's used to log into standard Moodle, and it works in the app as well. You should have created the user in background steps, and it will log in using the text as both username and password.

You will then be left at the start page.

Standard steps

Technically, you can use any standard Behat action in the app. However, most of them will probably not work as you expect because the app runs on a different environment. It is still a website, but it's built using Ionic Framework.

One important problem is that the app has a complex DOM, and previous pages that are "back" from your current page may still be present in the DOM. Which means that any steps that just look for the first matching element in the DOM are likely to look for elements on a page you're not even on.

Another issue is that Ionic relies heavily on the Shadow DOM, and most steps in standard Moodle are not prepared to handle that.

For these and other reasons, there are some steps that have been implemented specifically for the app. You can distinguish them from others because most of them end with "in the app".

Having said that, here's a list of steps that work and you can use reliably.

  • Any step you normally need to set up information in Moodle — For example, creating courses, users, etc.
  • I change viewport size to "{width}x{height}" — You can use this step to simulate switching between portrait and landscape formats.
  • I pause — This step works and it is very useful to debug your scenario.

Actions

When I press "Course 1" in the app

This will click an element found using accessibility rules, so it could be visible text, content inside aria-label, content described by ara-labelledby, etc. It should work for links, buttons and other clickable elements.

When I press "Course 1" near "Unique text" in the app

You can use this step to narrow matches if the text you're providing is duplicated throughout the page.

The second value, "Unique text" in this example, should be unique on the page. Otherwise, you may have issues finding the element that you seek. The system will press the element matching your text that is nearest to the one found using the unique text.

Nearest is defined in terms of the DOM rather than pixel position; it is based on the number of steps you would have to take up the tree from the candidate element before you get to a shared ancestor with the unique text.

When I select "Item 1" in the app
When I select "Item 1" near "Unique text" in the app
When I unselect "Item 1" in the app
When I unselect "Item 1" near "Unique text" in the app

You can use these steps to select or unselect radio buttons, check boxes, and such.

You could use the previous I press step as well, but in some cases you will notice that it doesn't work as you expect. This is due to some internal quirks of how Ionic renders these components, so prefer using this specific steps where possible.

When I set the field "field name" to "text value" in the app
When I set the field "field name" near "Unique text" to "text value" in the app

This sets a text field with the given value. The same rules will apply to find the input element as for clicking, so using the input name will not suffice. This is in order to encourage accessibility best practices. The only difference with the previous step is that this only matches fillable elements such as <input>, <textarea> and elements with contenteditable="true".

When I press the back button in the app
When I press the more menu button in the app
When I press the page menu button in the app
When I press the user menu button in the app

These steps will press, respectively:

  • The back button — This is the left pointing arrow at top left of the app.
  • The more menu button — This is the icon with at bottom right of the app.
  • The page menu button, if present — This is the icon with the three dots at top right of the app.
  • The user menu button, if present — This is the avatar button at top right of the app present on navigation level 1.


When I switch to the browser tab opened by the app
When I close the browser tab opened by the app

These two steps are necessary if you want to test the transition between the app and a browser.

For example, after pressing "Open in browser" you can use the first step above, and you will be able to use normal Moodle Behat steps to work in the browser tab. Once you're finished, you can use the second step to go back to the app.

Assertions

Like actions, there are some Behat assertions that are specific to the app.

Then I should find "Course 1" in the app
Then I should find "Course 1" near "Unique text" in the app
Then I should not find "Course 1" in the app
Then I should not find "Course 1" near "Unique text" in the app

These steps can be used to assert that the specified text exists somewhere in the app.

Notice that the standard I should see step may not work in the app because of the Shadow DOM. This step will also search using accessibility rules, so text within aria-label or described with aria-labelledby will work as well.

Then the header should be "Course 1" in the app

This checks the text of the current page header. It must be an exact match for the specified text.

You could have used the I should find step described previously, but this allows you to specifically check the header as opposed to anything in the page.

Then "Item 1" should be selected in the app
Then "Item 1" near "Unique text" should be selected in the app
Then "Item 1" should not be selected in the app
Then "Item 1" near "Unique text" should not be selected in the app

You can use these steps to assert whether radio buttons, check boxes, and such are selected or not.

Leaving the app

If you want to leave the app and go back to Moodle within a scenario, simply use a Moodle step that goes to a page. For example, use I am on site homepage or I am on "Course 1" course homepage.

You only need to do this if you want to carry out actions within Moodle after using the app, within the scenario. At the end of your scenario, there is no need to explicitly leave the app; Moodle will automatically start the next scenario on the Moodle start page as usual.

A complete example

This example is a complete feature file that loads the app, opens a course, and asserts that the app is showing the course page:

@app @javascript
Feature: Test app (demo)
  In order to test something in the app
  As a developer
  I need for this test script to run the app

  Background:
    Given the following "courses" exist:
      | fullname | shortname |
      | Course 1 | C1        |
    And the following "users" exist:
      | username |
      | student  |
    And the following "course enrolments" exist:
      | user     | course | role    |
      | student  | C1     | student |

  Scenario: Try going into the course
    When I enter the app
    And I log in as "student"
    And I press "Course 1" near "Course overview" in the app
    Then the header should be "Course 1" in the app

You can find more complex examples looking at the tests for the app core.

Limitations

Using this approach, there are some limitations you should be aware of:

  • Lack of native functionality — Fundamentally, it is not possible to test behaviour specific to native devices because tests are run in a browser.
  • Missing functionality — There are some known limitations and unsupported features, for example there is currently no obvious way to attach files. Some of these are possible, but they haven't been implemented yet. If something is missing for your use-case, you can submit feature requests in the tracker using the Behat component.

Advanced

Versioning

Behat tests can relate to particular versions of the mobile app. For these situations, there are two types of tags you can add to your scenario or feature:

  • @app_from{version} — These will be included in every app matching the specified version and newer.
  • @app_upto{version} — These will be included in every app matching the specified version and older.


You can use two-digit or three-digit version numbers. For example, you could use @app_from4.0 or @app_upto3.9.5.

After changing the app version used for testing, make sure you re-run Behat init. It is the initialisation process that stores which version of the app you're using.

Testing against multiple app versions

If you need to run tests against multiple versions of the app, you can do it in two ways:

  1. Update the code in the app workspace by checking out a different version.
  2. Maintain multiple copies of the mobile app workspace and switch between them by changing config.php.


In both cases, you'll need to re-run the Behat init command and run the tests again.

Unfortunately, the only way to run this in parallel is to have separate Moodle installations with their own configurations.

Debugging tests

If you insert a pause into your test and open the developer tools, you can debug the application like you would during development. You can learn how to do that in Using the Moodle App in a browser.

Additionally, you can see log information in the console about which Behat steps have been carried out so far, and whether Behat is waiting for anything. Here is an example:

VM649:391 BEHAT: 17:45:15.477 Action - Set field Username to: student2
VM649:391 BEHAT: 17:45:15.480 PENDING+: DELAY,dom-mutation
VM649:391 BEHAT: 17:45:15.982 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.28 PENDING-: 
VM649:391 BEHAT: 17:45:16.98 Action - Set field Password to: student2
VM649:391 BEHAT: 17:45:16.106 PENDING+: DELAY,dom-mutation
VM649:391 BEHAT: 17:45:16.607 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.653 PENDING-:

While the test is paused, you can also carry out some of the app Behat steps manually by typing commands into the console, which is convenient if you're not quite sure what command would work. The commands are available in a global object called behat. Here are some examples:

behat.setField('Password', 'student2');
behat.press('Log in', 'Forgotten');
behat.pressStandard('back');

There are more functions in the object; try using the browser's autocomplete to see the options, or look at the source in app_behat_runtime.js.

If you're using moodle-docker, remember that you can interact with the browser using VNC. With a VNC client you can view in real-time what behat is doing in the browser.

Writing custom steps

If you find something missing to test your code, you can always implement custom steps.

If you're writing a plugin, you can include a new class under tests/behat/behat_{youpluginname}.php. If you're working on application code, you can always update behat_app.php as well.

You can learn more about writing custom steps in the Writing new acceptance test step definitions page, and if you want to see how the steps that are specific to the app work, you should look into behat_app.php and app_behat_runtime.js.

Upgrading tests from an older version

If you wrote tests before the app started using Ionic 5 (in version 3.9.5), it is possible that your tests are not working any longer. This guide has been rewritten with the latest information, so make sure to read the rest of the document to see what changed.

In general though, upgrading your current tests should be mostly straightforward. Here are some overall things to keep in mind:

  • Ionic 5 starts using the Shadow DOM, and that's the source of many issues for the previous version of the Behat tests. The built-in steps have been rewritten to keep that in mind, and mostly should work the same way. One important difference is that they look for content using accessibility rules, so some of your previous assumptions may not be working.
  • Before this update, it was safe to use the standard I should see step. But given the problems mentioned, it is likely that it breaks down in many situations. Instead, you should use the new I should find ... in the app steps.
  • Similar to the last point, some radio inputs and checkboxes that worked before are broken. You should be able to use the new I select ... in the app steps instead.
  • If you were relying in xpath or css selectors before, they will probably not work anymore. Even if you try to patch them, these selectors don't pierce through the Shadow DOM. In any case, it's always better to use accessible locators in your test like a real user would, so you can use this opportunity to improve accessibility in your plugin.
  • Pay special attention to any assertions such as should not exist that you have in your tests. These assertions will not fail, because the elements won't be found. But if you get an eventual bug when something is shown that shouldn't, those steps will probably not pick it up because the locators may have changed.
  • If you were running your tests in CI, there is a new dependency even if you're only testing a plugin and not running the application tests. The test definitions have been moved from core to the local_moodlemobileapp plugin, and you should have it installed in your Moodle site running the tests. This was done in order to decouple application code from the core.


If you also need to upgrade your plugin, and not just the tests, check out the Adapt your Mobile plugins to Ionic 5 page.

Troubleshooting

General advice

If you are stuck with an error and you can't find a way to continue, here's a list of things you can do:

  • Make sure you added $CFG->behat_ionic_wwwroot = "http://localhost:8100"; (or equivalent) to your config.php file, and that url is reachable from the host where your Moodle site is running.
  • Remember when you need to re-run admin/tool/behat/cli/init.php, and make sure that you see "Configured app tests for version X.X.X". When in doubt, just run it again; it may fix your problem.
  • It is possible that your tests break if you're using an unstable version of the app. Try to use stable versions using the master branch if you're working with the source code or tagged releases if you're using Docker.
  • Mobile Behat tests don't work well with XDebug, so if you're using it, turn it off in php.ini while running the tests. Also, remember to restart Apache if necessary.

Unable to load app version from http://moodleapp:8100/config.json

This message appears when the Moodle site is not able to reach the app. Make sure that the url is available from the host you're running the Behat commands from. Also make sure that the app is actually running at the specified url.

It's ok if the actual /config.json url doesn't work, that's actually a remnant from legacy code. The url that Moodle is actually looking for is /assets/env.json.

The plugins required by this course could not be loaded correctly...

This means either some activity on the course is not adapted to support the moodle app or there is a timeout in the request to your behat site.

To clear the timeout message, open the app in your development browser, open the Inspector, open the Application tab, select Clear storage, press Clear site data, close Inspector, close the tab with mobile site, re-open mobile site in new tab and log in. Then in a separate tab, log in to your behat site (you can find the url in $CFG->behat_wwwroot within your config.php file) and make sure you can get into the course without seeing the error.

Fatal error: Maximum execution time of 30 seconds exceeded in...

This means that your local site has not been updated/visited since an upgrade. Just go to your local behat site (you can find the url in $CFG->behat_wwwroot within your config.php file), log in as admin and run notifications, then visit a course. Do this step often to avoid timeouts!

Test fails because of the browser language

If your operating system is in a different language than English, the tests may fail.

Chrome does not have an easy way to force the browser language to English, so the best way to solve the issue is forcing the app default language to English.

To do so, just set the forcedefaultlanguage attribute to "en" in your moodle.config.json file in the app.

Application build gets killed without any error information

In some situations, it is possible that you see Killed in the console and a command suddenly stops without any further information. In these situations, make sure to check the General advice section, but it is possible that your computer is running out of memory.

If you are running the scripts inside of a Docker container, make sure that Docker is allocated enough memory. If you are using Docker desktop (for example, on a Mac), you can inspect these settings under Preferences > Resources > Advanced > Memory.

[Mac] running moodle-docker commands show grep usage options and do nothing else

This is a known issue in moodle-docker for Mac. The workaround for now is just to explicitly initialize the MOODLE_DOCKER_APP_RUNTIME variable in your local environment.