Testing strategy/Testing Tools and Their Use

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Testing Tools and Their Use

All of the tools specified below are free with most being open source and extendible. The advantages are:

• There is no cost to Moodle for licences.
• Applications are web based with a tiny footprint and will run on low spec hardware/VMs.
• Applications with databases all run on MySQL,which is free.
• Moodle can extend the tools/create plugins, in-house, if customisations are required.

Continuous Integration

Continuous integration is the process of running tests frequently when code is checked into a repository. The aim of this is to keep code clean and allow the early capture of broken/incomplete code, early warning of conflicting changes (regressions) and a constantly available current “build” for demo and testing. Moodle's continuous integration process uses a second "Test" integration server where code must be submitted to prior to being reviewed by integrators.

Jenkins

Jenkins is a fully extendible application that monitors executions of repeated jobs. Results are displayed via a web application. Jenkins will be used as an information hub to provide Moodle with continuous feedback about the quality of Moodle software. Jenkins can monitor any/multiple GIT repository for changes without adding hooks to the code and be used to run a set of tasks such as:

• Pull git
• Run unit tests
• Create a moodle site
• Add some information to it
• Run the selenium tests
• Report errors/warnings/failures

Performance Testing Matrix

Measuring system performance during this testing will assist with preventing degradation of performance in existing code caused by regressions. To assist with early capture of performance degradation due to regression; a performance matrix will record performance against automated user interactions generated by the functional test harness.

Consideration will be given to modifying the functional test harness, selecting relevant test cases that will be re-run a given number of times to generate aggregated results as part of a performance and load suite. i.e. running the same test several times as part of a suite of tests in one “test run” will provide more accurate performance data, providing mean values, than running a test once.

Clover PHP Plugin

To capture code coverage reports from PHPUnit.

PHPUnit

Unit testing tool for PHP; implementation is already planned.

SimpleTest

Moodle's legacy unit testing tool.

Other Automated Testing

Methods of testing other than unit testing will be automated. This includes, but is not limited to functional testing.

Selenium

Selenium is an extremely powerful, free and fully extendible browser automation tool that contains the following features:

• Selenium IDE – Firefox plugin app for recording browser interactions with any web application. This is ideal for creating quick bug recreation test scripts and automation aided exploratory testing.
• Selenium Webdriver – Selenium Server is a server application that runs Selenium test cases. Selenium webdriver is a collection of language specific bindings to drive a browser.

Selenium provides extensions to existing unit test tools such as JUnit. There are unofficial language bindings for PHPUnit and webdriver and an official backwards compatibility to version one of Selenium through PHPUnit. Selenium test cases can be written in an object oriented manner and re-usable methods can be created. Selenium is highly resistant to CSS changes and supports using XPath to locate page elements. Furthermore Selenium server can be run in a grid configuration so that tests can run in parallel across multiple test environments controlled by one hub: Fig.2 Selenium Grid When a test is required to run on a specific platform Selenium Grid can be made aware of the environments you wish to use. The Selenium Hub will then ensure that a test runs only on the Selenium slaves providing the requested environment: Fig.3 Requesting Grid Environments

Firebug

Firebug is Firefox plugin that gives a user the ability to easily inspect page source.

Firepath

Firepath is a Firefox plugin that can generate XPath expressions.

Eclipse IDE

Eclipse IDE will be used to design Selenium test cases.

JUnit/PHPUnit Selenium Extension

SeleniumHQ officially support JUnit extensions for Selenium. The JUnit support is more mature than PHPUnit and is fully updated to work with the latest version of Selenium Server.

Hosting Test Environments

Hosting instances of Jenkins and test environments will be performed using server virtualization.

XEN

Xen is a robust, secure, high performance type 1 baremetal hypervisor, or a virtual machine monitor (VMM). Xen can securely execute multiple virtual machines, each running its own OS, on a single physical system with close-to-native performance. Xen is open source, and it's used as the core virtualization engine for many vendors products.

Version Control

GIT

Moodle use GIT for version control.

Test Management and Defect Tracking

TestLink

TestLink is a free test management tool written in PHP. The test management tool is used to specify and manage tests and test suites, then report on test execution. A test management tool is useful for managing and co-ordinating organisational testing effort and documenting testing. Currently Moodle use Jira as the test management tool. Jira is software development management tool and not a test management tool.

In TestLink we can create testing projects, test plans and document test cases, manage test execution runs then log and report results for manual and automated tests.

TestLink Jenkins Plugin

This plugin integrates TestLink with Jenkins and generates reports on automated test execution.

Jira

Moodle already use Jira for development and project planning and tracking issues.