Testing strategy/Test Environments

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Test Automation Environments

Test environments for automation are hosted on the Xen VM server. These will include a CI server (the build machine), The nightly build machine database servers and web servers.

Requirements Gathering

As a task while provisioning the VM Server, two surveys have been run to determine popular Test Platforms. Survey 1 gathered metrics from the Moodle Community and Survey 2 gathered metrics from Moodle Partners. Limiting factors of these surveys are that both samples are relatively small. The results of the surveys are as follows:

Survey 1 The Moodle Community:

Fig. 1 Separate Database and Web Server Yes/No:

Fig. 2 Operating System of Separate Web Server:

Fig. 3 Operating System of Separate Database Server:

Fig. 4 What Databases are Used on a Split System:

Fig. 5 What Databases are Used on an all in one Database Server and Webserver:

Fig. 6 What Web Server is Used on a Split System:

Fig. 7 What Web Server is Used on an all in one Database Server and Webserver:

As Debian and Ubuntu are similar and Red Hat and Centos are similar that,based upon our sample data, our three main operating used by the Moodle community Debian/Ubuntu, Red Hat/Centos and MS Server 2008 with a weighting of almost 75% towards the two Linux operating systems.

Of the five supported databases, the majority of community users prefer MySQL with PostgreSQL and MSSQL a fairly equal but distant second and third.

The Majority of community users also use Apache with IIS a very distant second.

Survey 2 The Moodle Partners:

Fig. 8 Separate Database and Web Server Yes/No:

Fig. 9 Partner OS Database Server:

Fig. 10 Partner OS Web Server (combining answers for fig 8):

Fig. 11 What Databases are Used on a Split System:

Fig. 12 What Databases are Used on a Combined System:

Fig. 13 What Web Servers are used on a Split System:

Fig. 14 What Web Servers are used on a Combined System:

This suggests that the majority of partners use Debian/Ubuntu or Centos/Rad Hat Linux, Apache and either a MySQL or PostgreSQL database.

VM Requirements

VM's for test environments are modular i.e. they are separate database servers and web servers. This allows the mixing and matching of databases and their host OS to the web servers and their host OS as requirements change. At any time, any Web Server can connect to any Database Server. It should also reduce potential numbers of running VMs required for a particular platform. A master hub VM is required for Jenkins and the hub for the Selenium grid.

CI Server

There are two CI Servers.

Integration CI

This is used to run integration jobs, checks and tests when code is checked in. The unit tests run on this machine on a MySQL database.

Nightly Integration Build

This is used to run PHPUnit tests on Postgres, performance tests against the Moodle supported databases and Browser driven Selenium tests.

Fig. 15 Selenium Test VM Architecture: