Note:

This site is no longer used and is in read-only mode. Instead please go to our new Moodle Developer Resource site.

Testing strategy: Difference between revisions

From MoodleDocs
← Older edit
VisualWikitext
Tim Barker (talk | contribs)
361 edits
No edit summary
No edit summary
 
(23 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<p class="note">'''THIS DOCUMENT IS CURRENTLY UNDER DEVELOPMENT AND IS THEREFORE INCOMPLETE. SOME CONTENT HAS YET TO BE UPLOADED OR INCLUDED AND CHANGES TO THE EXISTING CONTENT ARE PLANNED.'''</p>
{{Work in progress}}
<H1 CLASS="western">Introduction</H1>
 
<H2 CLASS="western">Purpose</H2>
=Introduction=
<P>The purpose of this document is to define a high level QA strategy
 
==Purpose==
 
The purpose of this document is to define a high level QA strategy
at Moodle and communicate that process to the relevant stakeholders.
at Moodle and communicate that process to the relevant stakeholders.
This document is loosely based upon the IEEE829 standard for software
 
testing documentation; adaptations have been made to this standard to
This document is test centric, only explaining the development
make the document more lightweight, suitable for use in an agile
process and relevant to the organisation.
</P>
<P>This document is test centric, only explaining the development
process from a testing point of view. The scope of this document does
process from a testing point of view. The scope of this document does
not cover the details of the entire development process at Moodle but
not cover the details of the entire development process at Moodle but
how testing is applied to it. This document also discusses potential
how testing is applied to it. This document also discusses potential
benefits to the organisation of the processes discussed.</P>
benefits to the organization of the processes discussed.
<H2 CLASS="western">Scope</H2>
Currently the processes that are documented here are under development. Progress can be viewed via the [[Testing Roadmap]]
<P STYLE="margin-bottom: 0cm">This document encompasses all testing
 
activities at Moodle and how they will be performed. This includes:</P>
==Scope==
<UL>
 
<LI><P STYLE="margin-bottom: 0cm">[https://docs.moodle.org/dev/Testing_strategy/Testing_Methodology_At_Moodle Testing methodology at Moodle]</P>
This document encompasses all testing activities at Moodle and how they will be performed. This includes:
<LI><P STYLE="margin-bottom: 0cm">Managing legacy items</P>
 
<LI><P STYLE="margin-bottom: 0cm">Hardware requirements</P>
* [[Testing strategy/Testing Methodology At Moodle|Testing Methodology at Moodle]]
<LI><P STYLE="margin-bottom: 0cm">Testing tools and how they will be
* [[Testing strategy/The Moodle Testing Process|The Moodle Testing Process]]
used</P>
* [[Testing strategy/Hardware Requirements|Hardware requirements]]
<LI><P STYLE="margin-bottom: 0cm">Test Environments</P>
* [[Testing strategy/Testing Tools and Their Use|Testing Tools and Their Use]]
<LI><P STYLE="margin-bottom: 0cm">Roles and responsibilities</P>
* [[Testing strategy/Test Automation Process|Test Automation Process]]
<LI><P STYLE="margin-bottom: 0cm">Training requirements</P>
* [[Testing strategy/Test Environments|Test Environments]]
</UL>
* [[Testing strategy/Useful Resources|Useful Resources]]
<H1 CLASS="western">Managing Legacy Items</H1>
'''TODO'''
<H1 CLASS="western">Hardware Requirements</H1>
<P>A cloud based server will be used to virtualize all test and build
platforms. The server is already in the process of being provisioned.</P>
<P>Server specifications are:</P>
<P>Make/Model: Dell T710</P>
<P>CPU: 2 x Intel Xeon E5645 2.4GHz, 12MB Cache 6 core CPU</P>
<P>Network: 2 x Broadcom 5709C dual-port gigabit ethernet module</P>
<P>RAM: 48GB (12x4GB) 1333MHz LV RDIMMS (triple channel)</P>
<P>Storage: 4 x 600GB 3.5&quot; 15,000 RPM 6GBPS SAS hot plug hard
disks with Dell H700 1GB Raid card configured for RAID 10.</P>
<H1 CLASS="western">Testing Tools and Their Use</H1>
<P>All of the tools specified below are free with most
being open source and extendible. The advantages are:</P>
<UL>
<LI><P>There is no cost to Moodle for licences.</P>
<LI><P>Applications are web based with a tiny footprint
and will run on low spec hardware/VMs.</P>
<LI><P>Applications with databases all run on MySQL,
which is free.</P>
<LI><P>Moodle can extend the tools/create plugins,
in-house, if customisations are required.</P>
</UL>
<H2 CLASS="western">Continuous Integration</H2>
<P>Continuous integration is the process of running
tests frequently when code is checked into a repository. The aim of
this is to keep code clean and allow the early capture of
broken/incomplete code, early warning of conflicting changes
(regressions) and a constantly available current “build” for demo
and testing.</P>
<H3 CLASS="western">Jenkins</H3>
<P>Jenkins is a fully extendible application that
monitors executions of repeated jobs. Results are displayed via a web
application.</P>
<P>Jenkins will be used as an information hub to provide
Moodle with continuous feedback about the quality of Moodle software.</P>
<P>Jenkins can monitor any/multiple GIT repository for
changes without adding hooks to the code and be used to run a set of
tasks such as:</P>
<UL>
<LI><P>Pull git</P>
<LI><P>Run unit tests</P>
<LI><P>Create a moodle site</P>
<LI><P>Add some information to it</P>
<LI><P>Run the selenium tests</P>
<LI><P>Report errors/warnings/failures</P>
</UL>
<H3 CLASS="western">Phing
</H3>
<P>“PHing Is Not GNU make” is a build tool for PHP based upon Apache Ant. Phing commands
are specified in XML just like in Ant. Phing can run PHPUnit and
SimpleTest unit tests, execute shell commands, perform actions with
GIT and much, much more.</P>
<P STYLE="font-weight: normal">As part of continuous
integration Phing will be the engine behind Jenkins, performing tasks
like getting the latest code for a test environment for the GIT
repository and running unit tests.</P>
<H3 CLASS="western">Phing Jenkins Plugin</H3>
<P>This plugin integrates Phing with Jenkins.</P>
<H3 CLASS="western">Clover PHP Plugin</H3>
<P>To capture code coverage reports from PHPUnit.</P>
<H3 CLASS="western">PHPUnit</H3>
<P>Unit testing tool for PHP; implementation is already
planned.</P>
<H3 CLASS="western">SimpleTest</H3>
<P>Moodle's legacy unit testing tool.</P>
<H2 CLASS="western">Other Automated Testing</H2>
<P>Methods of testing other than unit testing will be
automated. This includes, but is not limited to functional testing.</P>
<H3 CLASS="western">Selenium</H3>
<P>Selenium is an extremely powerful, free and fully
extendible browser automation tool that contains the following
features:</P>
<UL>
<LI><P>Selenium IDE – Firefox plugin app for
recording browser interactions with any web application. This is
ideal for creating quick bug recreation test scripts and automation
aided exploratory testing.</P>
<LI><P>Selenium Webdriver – Selenium Server is a
server application that runs Selenium test cases. Selenium webdriver
is a collection of language specific bindings to drive a browser.
</P>
</UL>
<P>Selenium provides extensions to existing unit test
tools such as JUnit. There are unofficial language bindings for
PHPUnit and webdriver and an official backwards compatibility to
version one of Selenium through PHPUnit.</P>
<P>Selenium test cases can be written in an object
oriented manner and re-usable methods can be created.</P>
<P>Selenium is highly
resistant to CSS changes and supports using XPath to locate page
elements.</P>
<P>Furthermore
Selenium server can be run in a grid configuration so that tests can
run in parallel across multiple test environments controlled by one
hub:</P>
<P>Fig.2 Selenium Grid</P>
<P>[[File:grid1.png]]</P>
<P>When a test is
required to run on a specific platform Selenium Grid can be
made aware of the environments you wish to use. The Selenium Hub will
then ensure that a test runs only on the Selenium slaves providing
the requested environment:</P>
<P>Fig.3 Requesting Grid Environments</P>
<P>[[File:grid2.png]]
</P>
<H3 CLASS="western">Selenium Jenkins Plugin</H3>
<P>A Jenkins plugin that gives Jenkins the ability to
turn a Jenkins cluster into a Selenium Cluster. The plugin installs
Selenium Grid on all of the slaves automatically and sets up the grid
on it's own. Great for deploying the latest code to multiple Selenium
test environments automatically.</P>
<H3 CLASS="western">Firebug</H3>
<P>Firebug is Firefox plugin that gives a user the
ability to easily inspect page source.</P>
<H3 CLASS="western">Firepath</H3>
<P>Firepath is a Firefox plugin that can generate XPath
expressions.</P>
<H3 CLASS="western">Eclipse IDE</H3>
<P>Eclipse IDE will be used to design Selenium test
cases.</P>
<H3 CLASS="western">JUnit/PHPUnit Selenium
Extension</H3>
<P>SeleniumHQ officially support JUnit extensions for
Selenium. The JUnit support is more mature than PHPUnit and is fully
updated to work with the latest version of Selenium Server.</P>
<H2 CLASS="western">Hosting Test Environments</H2>
<P>Hosting instances of Jenkins and test environments
will be performed using server virtualization.</P>
<H3 CLASS="western">XEN</H3>
<P>Xen is a robust, secure, high performance type 1 baremetal
hypervisor, or a virtual machine monitor (VMM). Xen can securely
execute multiple virtual machines, each running its own OS, on a
single physical system with close-to-native performance. Xen is open
source, and it's used as the core virtualization engine for many
vendors products.</P>
<H2 CLASS="western">Version Control</H2>
<H3 CLASS="western">GIT</H3>
<P>Moodle use GIT for version control.</P>
<H2 CLASS="western">Test Management and Defect
Tracking</H2>
<H3 CLASS="western">TestLink</H3>
<P>TestLink is a free test management tool written in
PHP. The test management tool is used to specify and manage tests and
test suites, then report on test execution. A test management tool is
useful for managing and co-ordinating organisational testing effort
and documenting testing. Currently Moodle use Jira as the test
management tool. Jira is software development management tool and not
a test management tool.</P>
<P>In TestLink we can create testing projects, test
plans and document test cases, manage test execution runs then log
and report results for manual and automated tests.</P>
<H3 CLASS="western">TestLink Jenkins Plugin</H3>
<P>This plugin integrates TestLink with Jenkins and
generates reports on automated test execution.</P>
<H3 CLASS="western">Jira</H3>
<P>Moodle already use Jira for development and project
planning and tracking issues.</P>
<H1 CLASS="western">Test Environments</H1>
<P>Test environments for automation will be hosted on
the VM server. These will include a CI server (the build machine),
database servers, web servers and client PCs.
</P>
<H2 CLASS="western">Requirements Gathering</H2>
<P>As a task while provisioning the VM Server, two
surveys have been run to determine popular Test Platforms. Survey 1
gathered metrics from the Moodle Community and Survey 2 gathered
metrics from Moodle Partners. Limiting factors of these surveys are
that both samples are relatively small. The results of the surveys
are as follows:</P>
<H3 CLASS="western">Survey
1 The Moodle Community:</H3>
<P>Fig. 4 Separate Database and Web Server Yes/No:</P>
<P>[[File:Fig._2_user_split_system.png]]</P>
<P>Fig. 5 Operating System of Separate Web Server:</P>
<P>[[File:fig.3_user_os_separatewebserver.gif]]
</P>
<P>Fig. 6 Operating
System of Separate Database Server:</P>
<P>[[File:fig.6_user_OS_SeparateDB.gif]]</P>
<P>Fig. 7 What Databases are Used on a Split System:</P>
<P>[[File:Fig7_user_dbs_used_split.png]]</P>
<P>Fig. 8 What Databases are Used on an all in one
Database Server and Webserver:</P>
<P>[[File:fig8_user_database_combined.png]]</P>
<P>Fig. 9 What Web
Server is Used on a Split System:</P>
<P>[[File:user_webserver_split.png]]</P>
<P>Fig. 10 What Web Server is Used on an all in one
Database Server and Webserver:</P>
<P>[[File:user_webserver_combined.png]]</P>
<P>As Debian and Ubuntu are similar and Red Hat and
Centos are similar that,based upon our sample data, our three main
operating used by the Moodle community Debian/Ubuntu, Red Hat/Centos
and MS Server 2008 with a weighting of almost 75% towards the two
Linux operating systems.</P>
<P>Of the five supported databases, the majority of
community users prefer MySQL with PostgreSQL and MSSQL a fairly equal
but distant second and third.</P>
<P>The Majority of community users also use Apache with
IIS a very distant second.</P>
<H3 CLASS="western">7.1.1Survey
2 The Moodle Partners:</H3>
<P>Fig. 11 Separate Database and Web Server Yes/No:</P>
<P>[[File:partner_split_combine.png]]
</P>
<P>Fig. 12 Partner OS Database Server (combining answers
for fig 11):</P>
<P>[[File:partner_OS_DB.gif]]
</P>
<P>Fig. 13 Partner OS
Web Server (combining answers for fig 8):</P>
<P>[[File:partner_OS_Web.gif]]
</P>
<P>Fig. 14  What Databases are Used on a Split System:</P>
<P>[[File:partner_db_split.png]]
</P>
<P>Fig. 15 What Databases are Used on a Combined System:</P>
<P>[[File:partner_db_combine.png]]
</P>
<P>Fig. 16 What Web
Servers are used on a Split System:</P>
<P>[[File:partner_webserver_split.png]]</P>
<P>Fig. 17 What Web Servers are used on a Combined
System:</P>
<P>[[File:partner_webserver_combined.png]]
</P>
<P>This suggests that the majority of partners use
Debian/Ubuntu or Centos/Rad Hat Linux, Apache and either a MySQL or
PostgreSQL database.
</P>
<H2 CLASS="western">VM Requirements</H2>
<P>In true agile fashion and to promote flexibility,
VM's for test environments will be modular i.e. they will be separate
database servers and web servers. This will allow the mixing and
matching of databases and their host OS to the web servers and their
host OS as requirements change. At any time, any Web Server can
connect to any Database Server. It should also reduce potential
numbers of running VMs required for a particular platform.</P>
<P>A master hub VM is required for Jenkins and the hub for the Selenium
grid.</P>
<H1 CLASS="western" STYLE="page-break-before: always">Implementation</H1>
<P>Agile testing guru and automation expert Brian Marick
coined the term “Hump of Pain” to describe the initial phase of
implementing test automation. The impact of this hump can be
minimised by good planning, good coaching and a strong managerial
support for the new processes. As time progresses, particularly as
infrastructure becomes implemented, the test cases in the automated
suite are completed and the introduced practices become second
nature; the hump disappears.</P>
<H2 CLASS="western">Agile Implementation of Testing
Practices</H2>
<P>Modifications to the process should be made in an incremental
manner based up scrum methodology. A series of 2 weekly sprints will
be used to time-box test planning and implementation tasks.</P>
<H2 CLASS="western">Phase 1: Infrastructure and the
Integration Process</H2>
<P>The infrastructure to maintain automated testing
effort must be implemented. This process will start with the
installation of cloud based hardware at Moodle HQ and the setup of
VM's to support the required test environments.</P>
<P>The integration process affects quality for the whole
organisation as all code must go through the integration process to
become 'live'. The processes described above will will be initially
implemented at integration. Any changes to the existing processes
will take place side-by-side with existing processes to manage risk.
i.e. existing processes impacted by changes will be maintained in
their current state until implementation of changes becomes stable.</P>
<P>The setup of test environments will initially start
with a restricted number of high risk test environments so that the
automation process, particularly test cases, can be developed
incrementally.</P>
<H1 CLASS="western">Roles and Responsibilities</H1>
<P>Below is a list of stakeholders and their roles
relevant to this document</P>
<TABLE WIDTH=100% BORDER=1 BORDERCOLOR="#000000" CELLPADDING=4 CELLSPACING=0>
<TR VALIGN=TOP>
<TD WIDTH=33% BGCOLOR="#ffff99">
<P ALIGN=CENTER><B>Name</B></P>
</TD>
<TD WIDTH=33% BGCOLOR="#ffff99">
<P ALIGN=CENTER><B>Role</B></P>
</TD>
<TD WIDTH=33% BGCOLOR="#ffff99">
<P ALIGN=CENTER><B>Responsibilities</B></P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Tim Barker</P>
</TD>
<TD WIDTH=33%>
<P>Test Manager</P>
</TD>
<TD WIDTH=33%>
<P>Co-ordinate testing effort</P>
<P>Define organisational test strategy and requirements</P>
<P>Specify test plans and cases</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Jordan Tomkinson</P>
</TD>
<TD WIDTH=33%>
<P>System Administrator</P>
</TD>
<TD WIDTH=33%>
<P>Provision of Hardware</P>
<P>Commission test platforms and supporting infrastructure</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Martin Dougiamas</P>
</TD>
<TD WIDTH=33%>
<P>Moodle Founder and MD</P>
</TD>
<TD WIDTH=33%>
<P>Approval</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Michael de Raadt</P>
</TD>
<TD WIDTH=33%>
<P>Development Manager</P>
</TD>
<TD WIDTH=33%>
<P>Approval</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Eloy Lafuente</P>
</TD>
<TD WIDTH=33%>
<P>Knight in Shining Armour</P>
</TD>
<TD WIDTH=33%>
<P>Technical guidance and advice</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Helen Foster</P>
</TD>
<TD WIDTH=33%>
<P>Community Manager</P>
</TD>
<TD WIDTH=33%>
<P>Community liaison with regard to requirements gathering</P>
</TD>
</TR>
<TR VALIGN=TOP>
<TD WIDTH=33%>
<P>Moodle Developers</P>
</TD>
<TD WIDTH=33%>
<P>Development</P>
</TD>
<TD WIDTH=33%>
<P>Input into VM requirements</P>
<P>Design and write unit tests</P>
<P>Test Execution</P>
<P>Performance Test Matrix (Sam Hemelryk)</P>
</TD>
</TR>
</TABLE>
<H1 CLASS="western">Training Requirements</H1>
<H1 CLASS="western">Useful Resources</H1>
<P>[http://jenkins-php.org/ Template for
Jenkins Jobs for PHP Projects]</P>
<P>[http://www.phing.info/trac/ Phing]</P>
<P>[https://wiki.jenkins-ci.org/display/JENKINS/Phing+Plugin" Phing
Jenkins Plugin]</P>
<P>[http://www.teamst.org/ TestLink]</P>
<P>[https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin Testlink
Jenkins Plugin]</P>
<P>[https://wiki.jenkins-ci.org/display/JENKINS/Clover+PHP+Plugin Clover
Jenkins Plugin]</P>
<P>[http://seleniumhq.org/" Selenium]</P>
<P>[https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin">Selenium
Jenkins Plugin]</P>

Latest revision as of 16:56, 25 September 2012

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.


Introduction

Purpose

The purpose of this document is to define a high level QA strategy at Moodle and communicate that process to the relevant stakeholders.

This document is test centric, only explaining the development process from a testing point of view. The scope of this document does not cover the details of the entire development process at Moodle but how testing is applied to it. This document also discusses potential benefits to the organization of the processes discussed. Currently the processes that are documented here are under development. Progress can be viewed via the Testing Roadmap

Scope

This document encompasses all testing activities at Moodle and how they will be performed. This includes:

Retrieved from "https://docs.moodle.org/dev/index.php?title=Testing_strategy&oldid=35670"
Powered by MediaWiki