Testing strategy
THIS DOCUMENT IS CURRENTLY UNDER DEVELOPMENT AND IS THEREFORE INCOMPLETE. SOME CONTENT HAS YET TO BE UPLOADED OR INCLUDED AND CHANGES TO THE EXISTING CONTENT ARE PLANNED.
Introduction
Purpose
The purpose of this document is to define a high level QA strategy at Moodle and communicate that process to the relevant stakeholders. This document is loosely based upon the IEEE829 standard for software testing documentation; adaptations have been made to this standard to make the document more lightweight, suitable for use in an agile process and relevant to the organisation.
This document is test centric, only explaining the development process from a testing point of view. The scope of this document does not cover the details of the entire development process at Moodle but how testing is applied to it. This document also discusses potential benefits to the organisation of the processes discussed.
Scope
This document encompasses all testing activities at Moodle and how they will be performed. This includes:
Hardware requirements
Testing tools and how they will be used
Test Environments
Roles and responsibilities
Training requirements
Hardware Requirements
A cloud based server will be used to virtualize all test and build platforms. The server is already in the process of being provisioned.
Server specifications are:
Make/Model: Dell T710
CPU: 2 x Intel Xeon E5645 2.4GHz, 12MB Cache 6 core CPU
Network: 2 x Broadcom 5709C dual-port gigabit ethernet module
RAM: 48GB (12x4GB) 1333MHz LV RDIMMS (triple channel)
Storage: 4 x 600GB 3.5" 15,000 RPM 6GBPS SAS hot plug hard disks with Dell H700 1GB Raid card configured for RAID 10.
Testing Tools and Their Use
All of the tools specified below are free with most being open source and extendible. The advantages are:
There is no cost to Moodle for licences.
Applications are web based with a tiny footprint and will run on low spec hardware/VMs.
Applications with databases all run on MySQL, which is free.
Moodle can extend the tools/create plugins, in-house, if customisations are required.
Continuous Integration
Continuous integration is the process of running tests frequently when code is checked into a repository. The aim of this is to keep code clean and allow the early capture of broken/incomplete code, early warning of conflicting changes (regressions) and a constantly available current “build” for demo and testing.
Jenkins
Jenkins is a fully extendible application that monitors executions of repeated jobs. Results are displayed via a web application.
Jenkins will be used as an information hub to provide Moodle with continuous feedback about the quality of Moodle software.
Jenkins can monitor any/multiple GIT repository for changes without adding hooks to the code and be used to run a set of tasks such as:
Pull git
Run unit tests
Create a moodle site
Add some information to it
Run the selenium tests
Report errors/warnings/failures
Phing
“PHing Is Not GNU make” is a build tool for PHP based upon Apache Ant. Phing commands are specified in XML just like in Ant. Phing can run PHPUnit and SimpleTest unit tests, execute shell commands, perform actions with GIT and much, much more.
As part of continuous integration Phing will be the engine behind Jenkins, performing tasks like getting the latest code for a test environment for the GIT repository and running unit tests.
Phing Jenkins Plugin
This plugin integrates Phing with Jenkins.
Clover PHP Plugin
To capture code coverage reports from PHPUnit.
PHPUnit
Unit testing tool for PHP; implementation is already planned.
SimpleTest
Moodle's legacy unit testing tool.
Other Automated Testing
Methods of testing other than unit testing will be automated. This includes, but is not limited to functional testing.
Selenium
Selenium is an extremely powerful, free and fully extendible browser automation tool that contains the following features:
Selenium IDE – Firefox plugin app for recording browser interactions with any web application. This is ideal for creating quick bug recreation test scripts and automation aided exploratory testing.
Selenium Webdriver – Selenium Server is a server application that runs Selenium test cases. Selenium webdriver is a collection of language specific bindings to drive a browser.
Selenium provides extensions to existing unit test tools such as JUnit. There are unofficial language bindings for PHPUnit and webdriver and an official backwards compatibility to version one of Selenium through PHPUnit.
Selenium test cases can be written in an object oriented manner and re-usable methods can be created.
Selenium is highly resistant to CSS changes and supports using XPath to locate page elements.
Furthermore Selenium server can be run in a grid configuration so that tests can run in parallel across multiple test environments controlled by one hub:
Fig.2 Selenium Grid
When a test is required to run on a specific platform Selenium Grid can be made aware of the environments you wish to use. The Selenium Hub will then ensure that a test runs only on the Selenium slaves providing the requested environment:
Fig.3 Requesting Grid Environments
Selenium Jenkins Plugin
A Jenkins plugin that gives Jenkins the ability to turn a Jenkins cluster into a Selenium Cluster. The plugin installs Selenium Grid on all of the slaves automatically and sets up the grid on it's own. Great for deploying the latest code to multiple Selenium test environments automatically.
Firebug
Firebug is Firefox plugin that gives a user the ability to easily inspect page source.
Firepath
Firepath is a Firefox plugin that can generate XPath expressions.
Eclipse IDE
Eclipse IDE will be used to design Selenium test cases.
JUnit/PHPUnit Selenium Extension
SeleniumHQ officially support JUnit extensions for Selenium. The JUnit support is more mature than PHPUnit and is fully updated to work with the latest version of Selenium Server.
Hosting Test Environments
Hosting instances of Jenkins and test environments will be performed using server virtualization.
XEN
Xen is a robust, secure, high performance type 1 baremetal hypervisor, or a virtual machine monitor (VMM). Xen can securely execute multiple virtual machines, each running its own OS, on a single physical system with close-to-native performance. Xen is open source, and it's used as the core virtualization engine for many vendors products.
Version Control
GIT
Moodle use GIT for version control.
Test Management and Defect Tracking
TestLink
TestLink is a free test management tool written in PHP. The test management tool is used to specify and manage tests and test suites, then report on test execution. A test management tool is useful for managing and co-ordinating organisational testing effort and documenting testing. Currently Moodle use Jira as the test management tool. Jira is software development management tool and not a test management tool.
In TestLink we can create testing projects, test plans and document test cases, manage test execution runs then log and report results for manual and automated tests.
TestLink Jenkins Plugin
This plugin integrates TestLink with Jenkins and generates reports on automated test execution.
Jira
Moodle already use Jira for development and project planning and tracking issues.
Test Environments
Test environments for automation will be hosted on the VM server. These will include a CI server (the build machine), database servers, web servers and client PCs.
Requirements Gathering
As a task while provisioning the VM Server, two surveys have been run to determine popular Test Platforms. Survey 1 gathered metrics from the Moodle Community and Survey 2 gathered metrics from Moodle Partners. Limiting factors of these surveys are that both samples are relatively small. The results of the surveys are as follows:
Survey 1 The Moodle Community:
Fig. 4 Separate Database and Web Server Yes/No:
Fig. 5 Operating System of Separate Web Server:
Fig. 6 Operating System of Separate Database Server:
Fig. 7 What Databases are Used on a Split System:
Fig. 8 What Databases are Used on an all in one Database Server and Webserver:
Fig. 9 What Web Server is Used on a Split System:
Fig. 10 What Web Server is Used on an all in one Database Server and Webserver:
As Debian and Ubuntu are similar and Red Hat and Centos are similar that,based upon our sample data, our three main operating used by the Moodle community Debian/Ubuntu, Red Hat/Centos and MS Server 2008 with a weighting of almost 75% towards the two Linux operating systems.
Of the five supported databases, the majority of community users prefer MySQL with PostgreSQL and MSSQL a fairly equal but distant second and third.
The Majority of community users also use Apache with IIS a very distant second.
7.1.1Survey 2 The Moodle Partners:
Fig. 11 Separate Database and Web Server Yes/No:
Fig. 12 Partner OS Database Server (combining answers for fig 11):
Fig. 13 Partner OS Web Server (combining answers for fig 8):
Fig. 14 What Databases are Used on a Split System:
Fig. 15 What Databases are Used on a Combined System:
Fig. 16 What Web Servers are used on a Split System:
Fig. 17 What Web Servers are used on a Combined System:
This suggests that the majority of partners use Debian/Ubuntu or Centos/Rad Hat Linux, Apache and either a MySQL or PostgreSQL database.
VM Requirements
In true agile fashion and to promote flexibility, VM's for test environments will be modular i.e. they will be separate database servers and web servers. This will allow the mixing and matching of databases and their host OS to the web servers and their host OS as requirements change. At any time, any Web Server can connect to any Database Server. It should also reduce potential numbers of running VMs required for a particular platform.
A master hub VM is required for Jenkins and the hub for the Selenium grid.
Implementation
Agile testing guru and automation expert Brian Marick coined the term “Hump of Pain” to describe the initial phase of implementing test automation. The impact of this hump can be minimised by good planning, good coaching and a strong managerial support for the new processes. As time progresses, particularly as infrastructure becomes implemented, the test cases in the automated suite are completed and the introduced practices become second nature; the hump disappears.
Agile Implementation of Testing Practices
Modifications to the process should be made in an incremental manner based up scrum methodology. A series of 2 weekly sprints will be used to time-box test planning and implementation tasks.
Phase 1: Infrastructure and the Integration Process
The infrastructure to maintain automated testing effort must be implemented. This process will start with the installation of cloud based hardware at Moodle HQ and the setup of VM's to support the required test environments.
The integration process affects quality for the whole organisation as all code must go through the integration process to become 'live'. The processes described above will will be initially implemented at integration. Any changes to the existing processes will take place side-by-side with existing processes to manage risk. i.e. existing processes impacted by changes will be maintained in their current state until implementation of changes becomes stable.
The setup of test environments will initially start with a restricted number of high risk test environments so that the automation process, particularly test cases, can be developed incrementally.
Roles and Responsibilities
Below is a list of stakeholders and their roles relevant to this document
|
Name |
Role |
Responsibilities |
|
Tim Barker |
Test Manager |
Co-ordinate testing effort Define organisational test strategy and requirements Specify test plans and cases |
|
Jordan Tomkinson |
System Administrator |
Provision of Hardware Commission test platforms and supporting infrastructure |
|
Martin Dougiamas |
Moodle Founder and MD |
Approval |
|
Michael de Raadt |
Development Manager |
Approval |
|
Eloy Lafuente |
Knight in Shining Armour |
Technical guidance and advice |
|
Helen Foster |
Community Manager |
Community liaison with regard to requirements gathering |
|
Moodle Developers |
Development |
Input into VM requirements Design and write unit tests Test Execution Performance Test Matrix (Sam Hemelryk) |
Training Requirements
Useful Resources
[http://jenkins-php.org/ Template for Jenkins Jobs for PHP Projects]
[https://wiki.jenkins-ci.org/display/JENKINS/Phing+Plugin" Phing Jenkins Plugin]
[https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin Testlink Jenkins Plugin]
[https://wiki.jenkins-ci.org/display/JENKINS/Clover+PHP+Plugin Clover Jenkins Plugin]
[https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin">Selenium Jenkins Plugin]
