Testing strategy/Testing Tools and Their Use

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Testing Tools and Their Use

All of the tools specified below are free with most being open source and extendible. The advantages are:

• There is no cost to Moodle for licences.
• Applications are web based with a tiny footprint and will run on low spec hardware/VMs.
• Applications with databases all run on MySQL,which is free.
• Moodle can extend the tools/create plugins, in-house, if customisations are required.

Continuous Integration

Continuous integration is the process of running tests frequently when code is checked into a repository. The aim of this is to keep code clean and allow the early capture of broken/incomplete code, early warning of conflicting changes (regressions) and a constantly available current “build” for demo and testing. Moodle's continuous integration process uses a second "Test" integration server where code must be submitted to prior to being reviewed by integrators.

Jenkins

Jenkins is a fully extendible application that monitors executions of repeated jobs. Results are displayed via a web application. Jenkins is an information hub to provide Moodle with continuous feedback about the quality of Moodle software. Jenkins can monitor any/multiple GIT repository for changes without adding hooks to the code and be used to run a set of tasks such as:

• Pull git
• Run unit tests
• Create a moodle site
• Add some information to it
• Run the selenium tests
• Report errors/warnings/failures

Performance Testing Matrix

Measuring system performance during this testing will assist with preventing degradation of performance in existing code caused by regressions. To assist with early capture of performance degradation due to regression; a performance matrix records performance against automated user interactions generated using JMeter.

PHPUnit

Unit testing tool for PHP; implementation is already planned.

SimpleTest

Moodle's legacy unit testing tool.

Other Automated Testing

Methods of testing other than unit testing is automated. Selenium is used for functional testing in conjunction with the Behat behavioural driven development tool.

Selenium

Selenium is an extremely powerful, free and fully extendible browser automation tool. Selenium Webdriver – Selenium Server is a language independent server application that runs Selenium test cases. Selenium WebDriver is a collection of language specific bindings to drive a browser. There are unofficial language bindings for PHP and WebDriver. Selenium test cases can be written in an object oriented manner and re-usable methods can be created. Selenium is highly resistant to CSS changes and supports using XPath to locate page elements. Furthermore Selenium server can be run in a grid configuration so that tests can run in parallel across multiple test environments controlled by one hub: Fig.2 Selenium Grid When a test is required to run on a specific platform Selenium Grid can be made aware of the environments you wish to use. The Selenium Hub will then ensure that a test runs only on the Selenium slaves providing the requested environment: Fig.3 Requesting Grid Environments

Behat

Tests that will run in Selenium are written in Behat. Behat is a PHP port of Cucumber, the popular Behavioural Driven Development (BDD) tool. Gherkin is a ubiquitous, domain specific language (DSL) that is used to document system behaviours. This DSL can be understood in the context of the business domain and acts as a layer of abstraction to automated tests. A great way of documenting requirements.

PHPUnit Selenium Extension

Selenium/Behat test results are output in PHPUnit xml format.

Hosting Test Environments

Hosting instances of Jenkins and test environments will be performed using server virtualization.

XEN

Xen is a robust, secure, high performance type 1 baremetal hypervisor, or a virtual machine monitor (VMM). Xen can securely execute multiple virtual machines, each running its own OS, on a single physical system with close-to-native performance. Xen is open source, and it's used as the core virtualization engine for many vendors products.

Version Control

GIT

Moodle use GIT for version control.

Test Management and Defect Tracking

TestLink

TestLink is a free test management tool written in PHP. The test management tool is used to specify and manage tests and test suites, then report on test execution. A test management tool is useful for managing and co-ordinating organisational testing effort and documenting testing. Currently Moodle use Jira as the test management tool. Jira is software development management tool and not a test management tool.

In TestLink we can create testing projects, test plans and document test cases, manage test execution runs then log and report results for manual and automated tests.

TestLink Jenkins Plugin

This plugin integrates TestLink with Jenkins and generates reports on automated test execution.

Jira

Moodle already use Jira for development and project planning and tracking issues.