Acceptance testing for the mobile app

Revision as of 17:50, 15 January 2019 by sam marshall (talk | contribs) (Added some help about using commands in the developer console.)

Jump to: navigation, search

This is only a proposal, MDL-63977, and is not yet available in Moodle. Please contact sam marshall if you have any queries and do not use this information for anything yet!

From Moodle 3.7 if this change is accepted it will be possible to write Behat tests for mobile app features.

Summary

It is now possible to create Behat tests that carry out automated functionality testing on the mobile app, for example so that you can test plugins that you may have written for the app.

By default, these do not run as part of a normal Behat run. This page tells you how to run the tests, and how to write them.

A key point is that these tests for some parts of the mobile app are included within the Moodle codebase, not within the app codebase, because they are run using the Moodle Behat infrastructure. This is definitely appropriate for Moodle plugins that add app support. It may also be acceptable for tests of the app itself, but this is not yet agreed.

The main advantages of this approach are:

  • It is easy for third-party plugin authors to create tests for app features in exactly the same way that they create tests for website features.
  • Where institutions run tests automatically, it should be relatively easy to include some app tests within the existing approach.
  • This system does not require any mobile device hardware and should work on all common platforms.

This system has been tested on Windows 7 and Ubuntu 18.04 LTS.

Running Behat tests for the mobile app

Set up a mobile app development environment

First you will need to set up a mobile app development environment.

Follow the first part of the instructions on this page:

You need to get as far as the part in section 5 where you open the app in the browser; this is what Behat will do. You don't need to complete the later steps.

  • You will need to update this environment periodically, for example when a new version of the mobile app is released. Behat does not do this for you.
  • When you update the mobile app, you must re-run Behat init on the corresponding Moodle installation.

Add the mobile app Behat configuration

You need to add one or two lines to your config.php to enable app testing. First, put this line in:

$CFG->behat_approot = '/path/to/app/workspace/moodlemobile2';

This may be sufficient, but you need to be aware of a couple of facts about the Ionic server used for app testing:

  • Depending on your computer, it may take about 3 minutes to start up.
  • The server uses about 1GB RAM.

There are two options, covered next.

Let Behat launch the app environment

If you want Behat to launch the app environment for you, then you don't need to add anything else to config.php.

When you do this, the Ionic server will be started automatically when Behat runs a test that uses the 'I enter the mobile app' step. It will be automatically terminated when the Behat test run finishes. If the test run includes multiple scenarios that use the app, they will all reuse the one server; it won't restart each time.

This is simple and convenient, but it is probably not a good approach for developers who frequently re-run a short Behat run (as you have to wait for it to start Ionic every time) or for complex systems that run Behat in parallel (as you may end up with multiple copies of Ionic eating up your RAM).

Manually launch the app environment yourself

The other option is to launch the Ionic server yourself (using ionic serve -b) before running any Behat tests, and also kill it yourself once you are finished with it. In this case, after launching it you will see output like:

[OK] Development server running!
     Local: http://localhost:8100
     External: http://137.108.5.43:8100, http://192.168.56.1:8100

To use it in Behat, add this extra line to the Moodle config.php:

$CFG->behat_ionicaddress = 'http://localhost:8100';

The advantage of this approach is that you are in charge of bringing up and taking down the Ionic server, so you can do this efficiently, share a copy between parallel runs, etc. The disadvantage is that you do have to remember to do it; if the server isn't running, tests which use the app will fail.

The running Ionic server should correspond to the app root setting, above, because the version number from the mobile app installation is used to determine which tests to run.

Browser profiles

Mobile tests only run in Chrome, so you need to make sure you have a Chrome profile set up in your config.php Behat settings.

Behat will automatically run app tests (those with @app tag) only in a Chrome browser profile. So, if you run multiple browser tests, it won't waste time trying to run the app tests in each one.

Behat init

After you have set up the config.php, you will need to re-run Behat init:

php admin/tool/behat/cli/init.php

This is necessary because by default, Behat won't run app tests (those with @app tag) at all, since you didn't have it configured.

Running Behat

To run mobile tests in Behat, simply launch Behat in the usual way, but make sure you are using a Chrome profile. (Depending on your setup, this might mean using --profile=chrome.)

You can specify the scenarios to run as normal. The app tests all have the @app tag, so if you want to run all the mobile tests you can specify --tags=app, but you can also run any other set of scenarios. It is OK to combine app and normal tests in the same run.

Writing tests

This page assumes you already know all about Writing acceptance tests in general.

Test structure

  • Mobile app test scenarios should be marked @app and @javascript in addition to any other tags that may be required.
  • If creating a feature file specifically for app tests, call it app_whatever.feature (i.e. use the app_ prefix). This is not technically required, it's just for consistency.

You are writing a normal Behat test and this is likely to require background steps similar to any other Moodle Behat test, for example the following "courses" exist, and so on.

Start the app

Once all necessary Moodle configuration steps (creating courses, users, groups, etc.) are done, use this Behat step to start the app:

Given I enter the app

This will:

  • Set up all the Moodle server settings to allow the mobile app to connect.
  • Launch Ionic if necessary
  • Restart the browser. This is needed to ensure it doesn't contain any stored data from previous app testing.
  • Set the browser to a suitable phone size (you can change it later if you want a tablet or other size).
  • Open the Ionic server address in the test Chrome browser.
  • Install necessary JavaScript code in the page that supports Behat testing.
  • Automatically enter the server URL into the app if necessary.

After this step completes, if it is the first time you ran the app inside this scenario, you will be left at the login screen. If you already logged in earlier, then you will be at the start page.

You can use this step even when you are already in the app; this will restart it.

Log in to the app

To log in:

When I log in as "student1"

This is the same step as used to log into standard Moodle, but if you are on the app login page it will automatically work to log into the app instead. It will log in with the given username, using the same password as username. You will then be left at the start page.

Actions

All the other app-specific Behat steps end with the words 'in the app' to distinguish them from the normal steps.

And I press "Course 1" in the app

This finds an element which contains either the visible text, or Aria label, 'Course 1' and clicks it. It should work for links, buttons and similar.

You should be able to use this for almost any actions that would be carried out by pressing something - pressing a button, following a link, changing a checkbox, switching a switch, opening a dropdown, selecting something from the popup, etc.

For buttons that are icons with no text, you can usually find them using the Chrome inspector - look for the 'aria-label' attribute.

You can press the main (bottom) menu buttons using this step. For example, the home button icon has the label 'home'.

  • Exact matches (an element which contains only the specified text, or where the Aria label is exactly the specified text) will be preferred. If there are no exact matches, then partial ones (anything containing that text) will be considered.
  • If there are multiple matches, or none, the step will fail. You can avoid this by specifying an exact match (provided there is only one exact match, this will not fail even if there are other partial matches) or by clicking on an icon instead of text.
  • If the item you try to press is a label for some other form field (using ion-label and the aria-labelled-by attribute) then it will actually press the field; this is useful in the settings menus.

And I press "Course 1" near "Unique text" in the app

This is a variant of the above step which is useful when there are multiple elements with the same text on the page. The second value ('Unique text' in this example) should be some text that is unique on the page. The system will press the instance of 'Course 1' that is nearest to the supplied unique text.

(This is intended as a simpler alternative to the standard Behat steps that use the word 'in', such as I click on "X" "thing" in "Y" "css_element". Those steps are complex and can be difficult to use. This one is not as generic but hopefully will handle most circumstances.)

  • Nearest is defined in terms of the DOM rather than pixel position; it is based on the number of steps you would have to take up the tree from the candidate item before you get to a shared ancestor with the unique text.

And I set the field "field name" to "text value" in the app

This sets a text field to the given value. For the field name, you can use the placeholder text (exact match will be found first, otherwise partial match if any).

This works with single-line text fields, multi-line text fields with rich text editor switched off (textarea) and rich-text-editor fields.

  • The normal version of this step supports various form fields, but in the app this only supports text fields at the moment. Use the press step (above) for other types of field.
  • When used with a rich-text editor, you can include HTML tags in the value if necessary.

And I press the back button in the app

And I press the main menu button in the app

And I press the page menu button in the app

These steps will press, respectively:

  • The back button (the left pointing arrow at top left of the app).
  • The main menu button (the '...' icon at bottom right of the app).
  • The page menu button, if present (the '...' icon at top right of the app).

Note that both the main menu and page menu use a 'more' icon so they are annoying to activate with the generic press command.

And I switch to the browser tab opened by the app

And I close the browser tab opened by the app

These two steps are necessary if you want to test the transition between the app and browser (e.g. test 'Open in browser' links). For example, after pressing 'Open in browser' you can use the first step above, and then you will be able to use normal Moodle Behat steps to check the browser tab. Then when finished, use the second step above.

Tests

Then the header should be "Course 1" in the app

This checks the text of the current page header (orange stripe at top of page) in the app. It must be an exact match for the specified text.

For this scenario, I should see would also work, but this allows you to specifically check the header as opposed to the text appearing elsewhere on the page.

Standard test steps

You can use all the normal Moodle Behat test steps while carrying out app testing, but some of them don't work very well. The app has a complex DOM and previous pages that are 'back' from your current page may still be present in the DOM, which means that any steps that just look for the first matching element in the DOM are likely to look for elements on a page you're not even on.

Before the app starts

Before starting the app, you normally need to set up information in Moodle (e.g. creating a course and user). For this part of your test you can obviously use all the normal Moodle steps.

Useful, working steps

  • I should see and I should not see are very useful for checking results.
  • I change viewport size to "640x360" is a useful step if you need to simulate switching between portrait and landscape formats.
  • I pause works and is very useful to debug your scenario.

Problematic steps

  • The I reload the page step does not work correctly in the app and may leave your test in a mess. Use I enter the app if you want to reload the app.

Leaving the app

If you want to leave the app and go back to Moodle within a scenario, simply use a Moodle step that goes to a page, such as I am on site homepage or I am on "Course 1" course homepage.

You only need to do this if you want to carry out actions within Moodle after using the app, within the scenario. At the end of your scenario, there is no need to explicitly leave the app; Moodle will automatically start the next scenario on the Moodle start page as usual.

Limitations

I have split the limitations of this approach into three categories, below.

Fundamental limitations

  • It is not possible to test behaviour specific to iOS, because tests run in Chrome. (Most iOS-specific failures are caused by problems with the Safari browser used on iOS, which isn't very good.)
  • Device features such as the camera cannot be tested, because tests run in a browser and not on a device.

It's my firm belief that even given those restrictions on test coverage, being able to run Behat tests is massively beneficial compared to having to test everything by hand for every app release.

Extra steps that might be needed

  • There is no obvious way to attach files.

Probably there are also other extra steps that would be useful - these would be discovered by trying to write tests. At the OU we plan to write Behat tests for our plugins once this feature is approved, so we will identify and add some extra steps at that point.

Testing the app itself

This system can absolutely be used to test features built into the app as well a plugins.

However, there is a limitation, which is that currently the Behat feature files must live in the Moodle project. This may be less appropriate for app features.

A future extension of this feature could use the behat_approot variable and automatically include feature files from a directory within the app's source tree. I think this would probably be quite simple but I have not yet tried it.

Advanced

Versioning

The Behat tests are stored in the Moodle codebase, so they always relate to a particular Moodle version, but sometimes it might be necessary to have different tests for different versions of the mobile app. For example, you may be writing a test in Moodle 3.6, where the behaviour in the Moodle 3.6 app is different from behaviour in the Moodle 3.7 app (but both apps can connect to the server).

For these situations:

  • In addition to the @app tag, add a version-specific tag to your scenario or feature.
  • There are two types of tag: @app_from3.7 (include for every app version from 3.7 and newer) or @app_upto3.6.3 (include for every app version up to 3.6.3, but not after that).
  • You can use a two-digit or three-digit version number (3.6 or 3.6.1).

Testing against multiple app versions

If you need to run tests against multiple versions of the mobile app, you can do this in two ways:

  1. Update the code in the mobile app workspace (check out a different version). Then re-run Behat init and run the Behat tests again.
  2. Maintain multiple copies of the mobile app workspace and switch between them by changing config.php. Then re-run Behat init and run the Behat tests again.

In both cases, because you need to run Behat init and change the Behat configuration, you cannot do this in parallel; if you want to run these tests in parallel you will also need separate Moodle installations with their own config.php, wwwroot, and dataroot.

Debugging tests

If you insert a pause into your test and open the developer tools (F12), you can see log information in the console about which actions were carried out so far, and whether Behat is waiting for anything. Here is an example:

VM649:391 BEHAT: 17:45:15.477 Action - Set field Username to: student2
VM649:391 BEHAT: 17:45:15.480 PENDING+: DELAY,dom-mutation
VM649:391 BEHAT: 17:45:15.982 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.28 PENDING-: 
VM649:391 BEHAT: 17:45:16.98 Action - Set field Password to: student2
VM649:391 BEHAT: 17:45:16.106 PENDING+: DELAY,dom-mutation
VM649:391 BEHAT: 17:45:16.607 PENDING-: DELAY
VM649:391 BEHAT: 17:45:16.653 PENDING-: 

While the test is paused you can also carry out some of the app Behat steps manually by typing commands into the console, which is convenient if you're not quite sure what command would work. Here are examples of the most useful commands:

  • behatSetField('Password', 'student2')
  • behatPress('Log in', 'Forgotten')
  • behatPressStandard('back')

There are a few others which all begin with 'behat'; try using the browser's autocomplete to see the options.