Testing strategy

THIS DOCUMENT AND IT'S SUB PAGES ARE CURRENTLY UNDER DEVELOPMENT AND IS THEREFORE INCOMPLETE. SOME CONTENT HAS YET TO BE UPLOADED OR INCLUDED AND CHANGES TO THE EXISTING CONTENT ARE PLANNED.

Introduction

Purpose

The purpose of this document is to define a high level QA strategy at Moodle and communicate that process to the relevant stakeholders. This document is loosely based upon the IEEE829 standard for software testing documentation; adaptations have been made to this standard to make the document more lightweight, suitable for use in an agile process and relevant to the organisation.

This document is test centric, only explaining the development process from a testing point of view. The scope of this document does not cover the details of the entire development process at Moodle but how testing is applied to it. This document also discusses potential benefits to the organisation of the processes discussed.

Scope

This document encompasses all testing activities at Moodle and how they will be performed. This includes:

• Testing methodology at Moodle

• Managing legacy items

• Hardware requirements

• Testing Tools and Their Use

• Test Environments

• Roles and responsibilities

• Training requirements

Test Environments

Test environments for automation will be hosted on the VM server. These will include a CI server (the build machine), database servers, web servers and client PCs.

Requirements Gathering

As a task while provisioning the VM Server, two surveys have been run to determine popular Test Platforms. Survey 1 gathered metrics from the Moodle Community and Survey 2 gathered metrics from Moodle Partners. Limiting factors of these surveys are that both samples are relatively small. The results of the surveys are as follows:

Survey 1 The Moodle Community:

Fig. 4 Separate Database and Web Server Yes/No:

Fig. 5 Operating System of Separate Web Server:

Fig. 6 Operating System of Separate Database Server:

Fig. 7 What Databases are Used on a Split System:

Fig. 8 What Databases are Used on an all in one Database Server and Webserver:

Fig. 9 What Web Server is Used on a Split System:

Fig. 10 What Web Server is Used on an all in one Database Server and Webserver:

As Debian and Ubuntu are similar and Red Hat and Centos are similar that,based upon our sample data, our three main operating used by the Moodle community Debian/Ubuntu, Red Hat/Centos and MS Server 2008 with a weighting of almost 75% towards the two Linux operating systems.

Of the five supported databases, the majority of community users prefer MySQL with PostgreSQL and MSSQL a fairly equal but distant second and third.

The Majority of community users also use Apache with IIS a very distant second.

7.1.1Survey 2 The Moodle Partners:

Fig. 11 Separate Database and Web Server Yes/No:

Fig. 12 Partner OS Database Server (combining answers for fig 11):

Fig. 13 Partner OS Web Server (combining answers for fig 8):

Fig. 14 What Databases are Used on a Split System:

Fig. 15 What Databases are Used on a Combined System:

Fig. 16 What Web Servers are used on a Split System:

Fig. 17 What Web Servers are used on a Combined System:

This suggests that the majority of partners use Debian/Ubuntu or Centos/Rad Hat Linux, Apache and either a MySQL or PostgreSQL database.

VM Requirements

In true agile fashion and to promote flexibility, VM's for test environments will be modular i.e. they will be separate database servers and web servers. This will allow the mixing and matching of databases and their host OS to the web servers and their host OS as requirements change. At any time, any Web Server can connect to any Database Server. It should also reduce potential numbers of running VMs required for a particular platform.

A master hub VM is required for Jenkins and the hub for the Selenium grid.

Implementation

Agile testing guru and automation expert Brian Marick coined the term “Hump of Pain” to describe the initial phase of implementing test automation. The impact of this hump can be minimised by good planning, good coaching and a strong managerial support for the new processes. As time progresses, particularly as infrastructure becomes implemented, the test cases in the automated suite are completed and the introduced practices become second nature; the hump disappears.

Agile Implementation of Testing Practices

Modifications to the process should be made in an incremental manner based up scrum methodology. A series of 2 weekly sprints will be used to time-box test planning and implementation tasks.

Phase 1: Infrastructure and the Integration Process

The infrastructure to maintain automated testing effort must be implemented. This process will start with the installation of cloud based hardware at Moodle HQ and the setup of VM's to support the required test environments.

The integration process affects quality for the whole organisation as all code must go through the integration process to become 'live'. The processes described above will will be initially implemented at integration. Any changes to the existing processes will take place side-by-side with existing processes to manage risk. i.e. existing processes impacted by changes will be maintained in their current state until implementation of changes becomes stable.

The setup of test environments will initially start with a restricted number of high risk test environments so that the automation process, particularly test cases, can be developed incrementally.

Roles and Responsibilities

Below is a list of stakeholders and their roles relevant to this document

Name	Role	Responsibilities
Tim Barker	Test Manager	Co-ordinate testing effort Define organisational test strategy and requirements Specify test plans and cases
Jordan Tomkinson	System Administrator	Provision of Hardware Commission test platforms and supporting infrastructure
Martin Dougiamas	Moodle Founder and MD	Approval
Michael de Raadt	Development Manager	Approval
Eloy Lafuente	Knight in Shining Armour	Technical guidance and advice
Helen Foster	Community Manager	Community liaison with regard to requirements gathering
Moodle Developers	Development	Input into VM requirements Design and write unit tests Test Execution Performance Test Matrix (Sam Hemelryk)

Useful Resources

[http://jenkins-php.org/ Template for Jenkins Jobs for PHP Projects]

Phing

[https://wiki.jenkins-ci.org/display/JENKINS/Phing+Plugin" Phing Jenkins Plugin]

TestLink

[https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin Testlink Jenkins Plugin]

[https://wiki.jenkins-ci.org/display/JENKINS/Clover+PHP+Plugin Clover Jenkins Plugin]

" Selenium

[https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin">Selenium Jenkins Plugin]