Moodle Mobile 2 (Ionic 1) End To End Testing

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

IMPORTANT: This documentation will be obsolete in a few weeks, please read this announcement.

Moodle Mobile End-to-End testing is done using Protractor, Appium, Selenium, SauceLabs.

Moodle Mobile can be tested in three different testing environments:

• Testing on SauceLabs browsers via Travis CI (important)
• Testing on SauceLabs via local machine (emulators, simulators)
• Testing on local machine (browsers, emulator)

Before proceeding

• Make sure Moodle Mobile development environment has been set up in your local machine
• Sign Up for SauceLabs OpenSauce(Free services for Open Source Projects) account

Testing on SauceLabs via Travis CI

Setting-up testing environment

You need to sign up for a free Travis account and connect your Moodle Mobile fork with Travis. Enter the SauceLabs Username and AccessKey in the environment variables as SAUCE_USERNAME and SAUCE_ACCESS_KEY in Moodle Mobile repo.

Protractor config file

Config files for SauceLabs Browser are present in the repo at e2e/sauce/browser/

There are multiple files which test different TestSuites. This is done to avoid Travis job time out of 50 min. Actual Travis Build + Test time takes nearly 4 hours. With the current job matrix, it is completed in ~40 minutes.

You can modify the Browser settings(OS version, Device, Resolution, Appium Version, etc) using the SauceLabs Platform Configurator. We highly recommend you to use Chrome browser for testing.

Running the suite

Scripts to run tests are present in the .travis.yaml file. With every push to the GitHub repo, Travis automatically starts the build and Tests.

You can log in to SauceLabs dashboard to see the test logs

Testing on SauceLabs via local machine

Setting-up testing environment

You need to have protractor v4.0.9 installed. If you have already setup the Moodle Mobile Development Environment then you would be having it installed. Verify with this command:

 protractor --version

If it isn’t already installed, run the following command:

 npm install protractor@4.0.9

Links for Moodle Mobile v3.3.1 .apk and .ipa files are provided for testing. However, the tests are updated regularly with development with respect to integration branch of the moodleapp repo.

Protractor config file

Config files for SauceLabs Android Emulators, Saucelabs iOS simulators are already present in the repo at e2e/sauce/

You need to add SauceLabs Username and AccessKey in multiCapabilities and seleniumAddress sections.

Now, you can modify the Emulator and Simulator settings(OS version, Device, Resolution, Appium Version, etc) using the SauceLabs Platform Configurator.

Here, we recommend you to test not more than 2 spec files at once. The Emulators, Simulators fail frequently under bulk requests.

Running the suite

Run the respective Protractor files for Android Emulator and iOS Simulator

• Run Protractor
  • ./node_modules/protractor/bin/protractor e2e/sauce/protractor-sauce-android.js
  • ./node_modules/protractor/bin/protractor e2e/suace/protractor-sauce-ios.js

You can log in to SauceLabs dashboard to see the test logs

Testing on Local Machine

Setting-up testing environment

This currently only covers testing on Android 4.4 onwards and Chrome. iOS testing is a work in progress.

• Install the Chrome web driver

./node_modules/protractor/bin/webdriver-manager update

• Make sure your checkout is up to date, all the Cordova plugins are installed and the android platform is set-up. The following should be enough.

gulp
ionic state restore
ionic build android

Once this is done, you will need to generate a protractor configuration file for the platform you intend to test the app on.

Protractor config file

Protractor config file contains all the information for running the tests, as well as to provide the tests with information about the site being tested.

• For Android
  • gulp e2e-build --target android --device abcdefg --version 4.4
• For Chrome (browser)
  • gulp e2e-build --target browser

By default the configuration file is saved at e2e/build/protractor.conf.js, should you wish to save the configuration file to re-use it later, you can use the argument --output <filename>.

Other arguments are available, you can find more about them using:

   gulp e2e-build -h
   gulp e2e-build --target android -h
   gulp e2e-build --target browser -h

Running the suite

• Run the ionic first
  • ionic serve -b -a
• Start the web driver
  • For Android: ./node_modules/appium/bin/appium.js
  • For Chrome : ./node_modules/protractor/bin/webdriver-manager start
• Wait for the driver to be fully started
• Run Protractor
  • ./node_modules/protractor/bin/protractor e2e/build/protractor.conf.js

Writing tests

Test files must:

• Be located in an e2e/ folder in the right component/addon
• End with .spec.js
• Use Jasmine as syntax

All helper functions are present at e2e/lib.js

For example, if you want write tests for mod_book:

• www/addons/mod_book/e2e/suite1.spec.js
• www/addons/mod_book/e2e/suite2.spec.js

Protractor mostly does everything for us to ensure that Angular is running, however, I noticed that in order for this to work we need to extensively use promises. Clicking on an element, or interacting with the app should return a promise which the tester will wait for before executing the next operation. For example:

it('Does something', function(done) {
    return MM.loginAsStudent().then(function() {
        return MM.clickOnInSideMenu('Messages');
    }).then(function() {
        return MM.clickOn('Contacts');
    }).then(function() {
        expect(MM.getView().getText()).toMatch('Something');
        done();
    });
});

MM comes with some helper functions present at e2e/lib.js to make it easier to test various things. Helper methods can be added when they are useful however they should not be specific to add-ons. Should helper methods be required for add-ons we can improve the building process to include them. In general, the steps taken by helper methods should be at least tested once without the helper method. For instance, if you implement a helper to log in, one of the tests should manually follow the steps to log in to confirm that it works as expected.

See the file e2e/lib.js for a list of helpers. The most helpful helpers might be the ones allowing you to follow a link in a certain region. Protractor often finds the first DOM element matching our selector, even if it's not visible on the screen. Often leading to failures the test tries to click of that element. Here is an example of a click in the visible view:

MM.clickOn($('.my-button'), MM.getView()).then(function() {
    // It worked!
})

All tests have access to global variables. These are used to conditionally decide what tests to run, or how tests should behave. We generally want to write tests which are compatible with any site, but at times we will need to alter them or to write other ones for specific scenarios. For example, a site running Moodle 2.7 with or without local mobile will generally behave differently and the tests should be written to handle this.

Example: 

MM.clickOn($('.my-button'), MM.getView()).then(function() {
    if (SITEHASLM) {
        // Assert something.
    } else {
        // Assert something else.
    }
})

These global variables are defined at the time when the protractor file is generated, hence the need for a different configuration file per site to be tested.

List of global variables:

• ISANDROID
• ISBROWSER
• ISIOS
• ISTABLET
• DEVICEURL
• DEVICEVERSION
• SITEURL
• SITEVERSION
• SITEHASLM (has local mobile)
• USERS.<role>.LOGIN
• USERS.<role>.PASSWORD

Common Errors

Most of these errors occur when an element fails to load in a specific amount of time. You can easily avoid these by re-running the tests or changing the locator/selector used for a specific element.

• Wait timed out after xxxxx ms
• No element found using locator: <locator>
• Timeout - Async callback was not invoked within the timeout specified by jasmine.DEFAULT_TIMEOUT_INTERVAL.
• Element not visible
• Element A not clickable at point (x, y). Instead, other element B receives the click.