Testing strategy: Difference between revisions
Tim Barker (talk | contribs) No edit summary |
Tim Barker (talk | contribs) No edit summary |
||
Line 1: | Line 1: | ||
<H1 CLASS="western"> | <H1 CLASS="western">Introduction</H1> | ||
<H2 CLASS="western"> | <H2 CLASS="western">Purpose</H2> | ||
<P>The purpose of this document is to define a high level QA strategy | <P>The purpose of this document is to define a high level QA strategy | ||
at Moodle and communicate that process to the relevant stakeholders. | at Moodle and communicate that process to the relevant stakeholders. | ||
Line 13: | Line 13: | ||
how testing is applied to it. This document also discusses potential | how testing is applied to it. This document also discusses potential | ||
benefits to the organisation of the processes discussed.</P> | benefits to the organisation of the processes discussed.</P> | ||
<H2 CLASS="western"> | <H2 CLASS="western">Scope</H2> | ||
<P STYLE="margin-bottom: 0cm">This document encompasses all testing | <P STYLE="margin-bottom: 0cm">This document encompasses all testing | ||
activities at Moodle and how they will be performed. This includes:</P> | activities at Moodle and how they will be performed. This includes:</P> | ||
Line 28: | Line 28: | ||
<LI><P STYLE="margin-bottom: 0cm">Training requirements</P> | <LI><P STYLE="margin-bottom: 0cm">Training requirements</P> | ||
</UL> | </UL> | ||
<H1 CLASS="western"> | <H1 CLASS="western">Testing Methodology at Moodle</H1> | ||
<P>The full <A HREF="https://docs.moodle.org/dev/Process">development | <P>The full <A HREF="https://docs.moodle.org/dev/Process">development | ||
life-cycle</A> at Moodle is documented in detail <A HREF="https://docs.moodle.org/dev/Process">elsewhere</A>. | life-cycle</A> at Moodle is documented in detail <A HREF="https://docs.moodle.org/dev/Process">elsewhere</A>. | ||
Line 40: | Line 40: | ||
<LI><P>3 point releases between every major release</P> | <LI><P>3 point releases between every major release</P> | ||
</UL> | </UL> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Testing Methodology</FONT></H2> | ||
<P>It is desirable on an agile project that a process of test driven | <P>It is desirable on an agile project that a process of test driven | ||
development and continuous integration (CI) is performed. The aim of | development and continuous integration (CI) is performed. The aim of | ||
Line 79: | Line 79: | ||
of required testing on Moodle projects. | of required testing on Moodle projects. | ||
</P> | </P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Quadrant Terms Explained</FONT></H3> | ||
<UL> | <UL> | ||
<LI><P><FONT SIZE=3>Tests that support the team – tests that | <LI><P><FONT SIZE=3>Tests that support the team – tests that | ||
Line 95: | Line 95: | ||
communicate user requirements.</FONT></P> | communicate user requirements.</FONT></P> | ||
</UL> | </UL> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Testing Disciplines Explained</FONT></H3> | ||
<UL> | <UL> | ||
<LI><P><FONT SIZE=3>Functional Testing – Interacting positively | <LI><P><FONT SIZE=3>Functional Testing – Interacting positively | ||
Line 139: | Line 139: | ||
<H1 CLASS="western"><BR><BR> | <H1 CLASS="western"><BR><BR> | ||
</H1> | </H1> | ||
<H1 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=4> | <H1 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=4>Methodologies | ||
in the Moodle Development Process</FONT></H1> | in the Moodle Development Process</FONT></H1> | ||
<P><FONT SIZE=3>As previously discussed, Moodle has an established | <P><FONT SIZE=3>As previously discussed, Moodle has an established | ||
Line 151: | Line 151: | ||
acceptance test cases. The tools and other aspects of automation will | acceptance test cases. The tools and other aspects of automation will | ||
be discussed later in the document.</FONT></P> | be discussed later in the document.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Weekly (Continuous) Integration | ||
Cycle</FONT></H2> | Cycle</FONT></H2> | ||
<P><FONT SIZE=3>Current testing of integration issues occurs towards | <P><FONT SIZE=3>Current testing of integration issues occurs towards | ||
Line 166: | Line 166: | ||
from the Wednesday testing phase to earlier in the SDLC whilst also | from the Wednesday testing phase to earlier in the SDLC whilst also | ||
increasing test coverage.</FONT></P> | increasing test coverage.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Continuous Integration (CI)</SPAN></FONT></H3> | ||
<P><FONT SIZE=3><SPAN STYLE="background: #ffff00">Continuous | <P><FONT SIZE=3><SPAN STYLE="background: #ffff00">Continuous | ||
integration provides early feedback of problematic code to the team. | integration provides early feedback of problematic code to the team. | ||
Line 174: | Line 174: | ||
currently exists for the integration process. This server will be | currently exists for the integration process. This server will be | ||
moved to a VM on the VM Server.</SPAN></FONT></P> | moved to a VM on the VM Server.</SPAN></FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Functional Test Automation</FONT></H3> | ||
<P><FONT SIZE=3>Frequently running tests similar to those currently | <P><FONT SIZE=3>Frequently running tests similar to those currently | ||
executed during the 6 monthly major release more regularly, earlier | executed during the 6 monthly major release more regularly, earlier | ||
Line 193: | Line 193: | ||
The set up of test VMs will be performed by the </FONT> | The set up of test VMs will be performed by the </FONT> | ||
</P> | </P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Performance Testing Matrix</FONT></H3> | ||
<P><FONT SIZE=3>Measuring system performance during this testing will | <P><FONT SIZE=3>Measuring system performance during this testing will | ||
assist with preventing degradation of performance in existing code | assist with preventing degradation of performance in existing code | ||
Line 208: | Line 208: | ||
more accurate performance data, providing mean values, than running a | more accurate performance data, providing mean values, than running a | ||
test once.</FONT></P> | test once.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>New Unit Tests</FONT></H3> | ||
<P><FONT SIZE=3>Creating unit tests prior to creating the code starts | <P><FONT SIZE=3>Creating unit tests prior to creating the code starts | ||
the whole SDLC off with a focus on quality. If the code isn't | the whole SDLC off with a focus on quality. If the code isn't | ||
Line 228: | Line 228: | ||
as regression tests using a CI server. The tests will run whenever | as regression tests using a CI server. The tests will run whenever | ||
any new code is added to any monitored GIT repository.</FONT></P> | any new code is added to any monitored GIT repository.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Legacy Unit Tests</FONT></H3> | ||
<P><FONT SIZE=3>Legacy unit tests are not run automatically. These | <P><FONT SIZE=3>Legacy unit tests are not run automatically. These | ||
tests can be run automatically on a regular basis via CI. Running | tests can be run automatically on a regular basis via CI. Running | ||
Line 237: | Line 237: | ||
tool. These tests are included with Moodle and can be run as an Admin | tool. These tests are included with Moodle and can be run as an Admin | ||
user. These legacy tests must run automatically using a CI server.</FONT></P> | user. These legacy tests must run automatically using a CI server.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Causal Analysis of Issues</FONT></H3> | ||
<P><FONT SIZE=3>As part of the weekly integration cycle, issues will | <P><FONT SIZE=3>As part of the weekly integration cycle, issues will | ||
be analysed and documented to determine the likely cause of the | be analysed and documented to determine the likely cause of the | ||
Line 252: | Line 252: | ||
<P><FONT SIZE=3>Fewer route causes of issues will reduce the number | <P><FONT SIZE=3>Fewer route causes of issues will reduce the number | ||
of issues and increase ROI and overall quality.</FONT></P> | of issues and increase ROI and overall quality.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Restricted Scope Manual Testing</FONT></H3> | ||
<P><FONT SIZE=3>The scope of testing issue fixes must be limited to | <P><FONT SIZE=3>The scope of testing issue fixes must be limited to | ||
retesting the issue itself. Clear instructions to specifically | retesting the issue itself. Clear instructions to specifically | ||
Line 263: | Line 263: | ||
will only be possible if those raising issues provide precise | will only be possible if those raising issues provide precise | ||
instructions for recreating the problem.</FONT></P> | instructions for recreating the problem.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Stable Sprints</FONT></H2> | ||
<P><FONT SIZE=3>Development work during stable sprints is performed | <P><FONT SIZE=3>Development work during stable sprints is performed | ||
in-house by Moodle HQ developers and will follow a process of Test | in-house by Moodle HQ developers and will follow a process of Test | ||
Line 276: | Line 276: | ||
matrix that will prove their fixes to code works. Then they will | matrix that will prove their fixes to code works. Then they will | ||
write their fixes</FONT></P> | write their fixes</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Continuous Integration</FONT></H3> | ||
<P><FONT SIZE=3>To allow early feedback on fixes provided by the | <P><FONT SIZE=3>To allow early feedback on fixes provided by the | ||
stable team they will require their own branch in GIT and their own | stable team they will require their own branch in GIT and their own | ||
Line 282: | Line 282: | ||
branch at the end of each day and unit tests will run at check in and | branch at the end of each day and unit tests will run at check in and | ||
again overnight.</FONT></P> | again overnight.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Functional Test Automation</FONT></H3> | ||
<P><FONT SIZE=3>Automated functional testing will take place as part | <P><FONT SIZE=3>Automated functional testing will take place as part | ||
of the regular integration cycle.</FONT></P> | of the regular integration cycle.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Major Releases and New | ||
Development</SPAN></FONT></H2> | Development</SPAN></FONT></H2> | ||
<P><BR><BR> | <P><BR><BR> | ||
Line 292: | Line 291: | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Point Releases</FONT></H2> | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
Line 300: | Line 299: | ||
</P> | </P> | ||
<H1 CLASS="western"></H1> | <H1 CLASS="western"></H1> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Managing Legacy Items</FONT></H1> | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
Line 307: | Line 306: | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Hardware Requirements</FONT></H1> | ||
<P>A cloud based server will be used to virtualize all test and build | <P>A cloud based server will be used to virtualize all test and build | ||
platforms. The server is already in the process of being provisioned.</P> | platforms. The server is already in the process of being provisioned.</P> | ||
Line 317: | Line 316: | ||
<P>Storage: 4 x 600GB 3.5" 15,000 RPM 6GBPS SAS hot plug hard | <P>Storage: 4 x 600GB 3.5" 15,000 RPM 6GBPS SAS hot plug hard | ||
disks with Dell H700 1GB Raid card configured for RAID 10.</P> | disks with Dell H700 1GB Raid card configured for RAID 10.</P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Testing Tools and Their Use</FONT></H1> | ||
<P><FONT SIZE=3>All of the tools specified below are free with most | <P><FONT SIZE=3>All of the tools specified below are free with most | ||
being open source and extendible. The advantages are:</FONT></P> | being open source and extendible. The advantages are:</FONT></P> | ||
Line 329: | Line 328: | ||
in-house, if customisations are required.</FONT></P> | in-house, if customisations are required.</FONT></P> | ||
</UL> | </UL> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Continuous Integration</FONT></H2> | ||
<P><FONT SIZE=3>Continuous integration is the process of running | <P><FONT SIZE=3>Continuous integration is the process of running | ||
tests frequently when code is checked into a repository. The aim of | tests frequently when code is checked into a repository. The aim of | ||
Line 336: | Line 335: | ||
(regressions) and a constantly available current “build” for demo | (regressions) and a constantly available current “build” for demo | ||
and testing.</FONT></P> | and testing.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Jenkins</FONT></H3> | ||
<P><FONT SIZE=3>Jenkins is a fully extendible application that | <P><FONT SIZE=3>Jenkins is a fully extendible application that | ||
monitors executions of repeated jobs. Results are displayed via a web | monitors executions of repeated jobs. Results are displayed via a web | ||
Line 353: | Line 352: | ||
<LI><P><FONT SIZE=3>Report errors/warnings/failures</FONT></P> | <LI><P><FONT SIZE=3>Report errors/warnings/failures</FONT></P> | ||
</UL> | </UL> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Phing </FONT> | ||
</H3> | </H3> | ||
<P>“<FONT SIZE=3><B>PH</B></FONT><FONT SIZE=3>ing </FONT><FONT SIZE=3><B>I</B></FONT><FONT SIZE=3>s | <P>“<FONT SIZE=3><B>PH</B></FONT><FONT SIZE=3>ing </FONT><FONT SIZE=3><B>I</B></FONT><FONT SIZE=3>s | ||
Line 366: | Line 365: | ||
like getting the latest code for a test environment for the GIT | like getting the latest code for a test environment for the GIT | ||
repository and running unit tests.</FONT></P> | repository and running unit tests.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Phing Jenkins Plugin</FONT></H3> | ||
<P><FONT SIZE=3>This plugin integrates Phing with Jenkins.</FONT></P> | <P><FONT SIZE=3>This plugin integrates Phing with Jenkins.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Clover PHP Plugin</FONT></H3> | ||
<P><FONT SIZE=3>To capture code coverage reports from PHPUnit.</FONT></P> | <P><FONT SIZE=3>To capture code coverage reports from PHPUnit.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>PHPUnit</FONT></H3> | ||
<P><FONT SIZE=3>Unit testing tool for PHP; implementation is already | <P><FONT SIZE=3>Unit testing tool for PHP; implementation is already | ||
planned.</FONT></P> | planned.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>SimpleTest</FONT></H3> | ||
<P><FONT SIZE=3>Moodle's legacy unit testing tool.</FONT></P> | <P><FONT SIZE=3>Moodle's legacy unit testing tool.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Other Automated Testing</FONT></H2> | ||
<P><FONT SIZE=3>Methods of testing other than unit testing will be | <P><FONT SIZE=3>Methods of testing other than unit testing will be | ||
automated. This includes, but is not limited to functional testing.</FONT></P> | automated. This includes, but is not limited to functional testing.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Selenium</FONT></H3> | ||
<P><FONT SIZE=3>Selenium is an extremely powerful, free and fully | <P><FONT SIZE=3>Selenium is an extremely powerful, free and fully | ||
extendible browser automation tool that contains the following | extendible browser automation tool that contains the following | ||
Line 418: | Line 417: | ||
<P><IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_635b180f.png" NAME="graphics3" ALIGN=LEFT WIDTH=643 HEIGHT=454 BORDER=0><BR><BR> | <P><IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_635b180f.png" NAME="graphics3" ALIGN=LEFT WIDTH=643 HEIGHT=454 BORDER=0><BR><BR> | ||
</P> | </P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Selenium Jenkins Plugin</FONT></H3> | ||
<P><FONT SIZE=3>A Jenkins plugin that gives Jenkins the ability to | <P><FONT SIZE=3>A Jenkins plugin that gives Jenkins the ability to | ||
turn a Jenkins cluster into a Selenium Cluster. The plugin installs | turn a Jenkins cluster into a Selenium Cluster. The plugin installs | ||
Line 424: | Line 423: | ||
on it's own. Great for deploying the latest code to multiple Selenium | on it's own. Great for deploying the latest code to multiple Selenium | ||
test environments automatically.</FONT></P> | test environments automatically.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Firebug</FONT></H3> | ||
<P><FONT SIZE=3>Firebug is Firefox plugin that gives a user the | <P><FONT SIZE=3>Firebug is Firefox plugin that gives a user the | ||
ability to easily inspect page source.</FONT></P> | ability to easily inspect page source.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Firepath</FONT></H3> | ||
<P><FONT SIZE=3>Firepath is a Firefox plugin that can generate XPath | <P><FONT SIZE=3>Firepath is a Firefox plugin that can generate XPath | ||
expressions.</FONT></P> | expressions.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Eclipse IDE</FONT></H3> | ||
<P><FONT SIZE=3>Eclipse IDE will be used to design Selenium test | <P><FONT SIZE=3>Eclipse IDE will be used to design Selenium test | ||
cases.</FONT></P> | cases.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>JUnit/PHPUnit Selenium | ||
Extension</FONT></H3> | Extension</FONT></H3> | ||
<P><FONT SIZE=3>SeleniumHQ officially support JUnit extensions for | <P><FONT SIZE=3>SeleniumHQ officially support JUnit extensions for | ||
Selenium. The JUnit support is more mature than PHPUnit and is fully | Selenium. The JUnit support is more mature than PHPUnit and is fully | ||
updated to work with the latest version of Selenium Server.</FONT></P> | updated to work with the latest version of Selenium Server.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Hosting Test Environments</FONT></H2> | ||
<P><FONT SIZE=3>Hosting instances of Jenkins and test environments | <P><FONT SIZE=3>Hosting instances of Jenkins and test environments | ||
will be performed using server virtualization.</FONT></P> | will be performed using server virtualization.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>XEN</FONT></H3> | ||
<P>Xen is a robust, secure, high performance type 1 baremetal | <P>Xen is a robust, secure, high performance type 1 baremetal | ||
hypervisor, or a virtual machine monitor (VMM). Xen can securely | hypervisor, or a virtual machine monitor (VMM). Xen can securely | ||
Line 448: | Line 447: | ||
source, and it's used as the core virtualization engine for many | source, and it's used as the core virtualization engine for many | ||
vendors products.</P> | vendors products.</P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Version Control</FONT></H2> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>GIT</FONT></H3> | ||
<P><FONT SIZE=3>Moodle use GIT for version control.</FONT></P> | <P><FONT SIZE=3>Moodle use GIT for version control.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Test Management and Defect | ||
Tracking</FONT></H2> | Tracking</FONT></H2> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>TestLink</FONT></H3> | ||
<P><FONT SIZE=3>TestLink is a free test management tool written in | <P><FONT SIZE=3>TestLink is a free test management tool written in | ||
PHP. The test management tool is used to specify and manage tests and | PHP. The test management tool is used to specify and manage tests and | ||
Line 464: | Line 463: | ||
plans and document test cases, manage test execution runs then log | plans and document test cases, manage test execution runs then log | ||
and report results for manual and automated tests.</FONT></P> | and report results for manual and automated tests.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>TestLink Jenkins Plugin</FONT></H3> | ||
<P><FONT SIZE=3>This plugin integrates TestLink with Jenkins and | <P><FONT SIZE=3>This plugin integrates TestLink with Jenkins and | ||
generates reports on automated test execution.</FONT></P> | generates reports on automated test execution.</FONT></P> | ||
<H3 CLASS="western"><FONT SIZE=3> | <H3 CLASS="western"><FONT SIZE=3>Jira</FONT></H3> | ||
<P><FONT SIZE=3>Moodle already use Jira for development and project | <P><FONT SIZE=3>Moodle already use Jira for development and project | ||
planning and tracking issues.</FONT></P> | planning and tracking issues.</FONT></P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Test Environments</FONT></H1> | ||
<P><FONT SIZE=3>Test environments for automation will be hosted on | <P><FONT SIZE=3>Test environments for automation will be hosted on | ||
the VM server. These will include a CI server (the build machine), | the VM server. These will include a CI server (the build machine), | ||
database servers, web servers and client PCs. </FONT> | database servers, web servers and client PCs. </FONT> | ||
</P> | </P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Requirements Gathering</FONT></H2> | ||
<P><FONT SIZE=3>As a task while provisioning the VM Server, two | <P><FONT SIZE=3>As a task while provisioning the VM Server, two | ||
surveys have been run to determine popular Test Platforms. Survey 1 | surveys have been run to determine popular Test Platforms. Survey 1 | ||
Line 484: | Line 483: | ||
<H3 CLASS="western"><BR><BR> | <H3 CLASS="western"><BR><BR> | ||
</H3> | </H3> | ||
<H3 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=3> | <H3 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=3>Survey | ||
1 The Moodle Community:</FONT></H3> | 1 The Moodle Community:</FONT></H3> | ||
<P><BR><BR> | <P><BR><BR> | ||
Line 790: | Line 789: | ||
PostgreSQL database. </FONT> | PostgreSQL database. </FONT> | ||
</P> | </P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>VM Requirements</FONT></H2> | ||
<P><FONT SIZE=3>In true agile fashion and to promote flexibility, | <P><FONT SIZE=3>In true agile fashion and to promote flexibility, | ||
VM's for test environments will be modular i.e. they will be separate | VM's for test environments will be modular i.e. they will be separate | ||
Line 801: | Line 800: | ||
master hub VM is required for Jenkins and the hub for the Selenium | master hub VM is required for Jenkins and the hub for the Selenium | ||
grid.</FONT></P> | grid.</FONT></P> | ||
<H1 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=4> | <H1 CLASS="western" STYLE="page-break-before: always"><FONT SIZE=4>Implementation</FONT></H1> | ||
<P><FONT SIZE=3>Agile testing guru and automation expert Brian Marick | <P><FONT SIZE=3>Agile testing guru and automation expert Brian Marick | ||
coined the term “Hump of Pain” to describe the initial phase of | coined the term “Hump of Pain” to describe the initial phase of | ||
Line 810: | Line 809: | ||
suite are completed and the introduced practices become second | suite are completed and the introduced practices become second | ||
nature; the hump disappears.</FONT></P> | nature; the hump disappears.</FONT></P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Agile Implementation of Testing | ||
Practices</FONT></H2> | Practices</FONT></H2> | ||
<P>Modifications to the process should be made in an incremental | <P>Modifications to the process should be made in an incremental | ||
manner based up scrum methodology. A series of 2 weekly sprints will | manner based up scrum methodology. A series of 2 weekly sprints will | ||
be used to time-box test planning and implementation tasks.</P> | be used to time-box test planning and implementation tasks.</P> | ||
<H2 CLASS="western"><FONT SIZE=3> | <H2 CLASS="western"><FONT SIZE=3>Phase 1: Infrastructure and the | ||
Integration Process</FONT></H2> | Integration Process</FONT></H2> | ||
<P><FONT SIZE=3>The infrastructure to maintain automated testing | <P><FONT SIZE=3>The infrastructure to maintain automated testing | ||
Line 832: | Line 831: | ||
automation process, particularly test cases, can be developed | automation process, particularly test cases, can be developed | ||
incrementally.</FONT></P> | incrementally.</FONT></P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Roles and Responsibilities</FONT></H1> | ||
<P><FONT SIZE=3>Below is a list of stakeholders and their roles | <P><FONT SIZE=3>Below is a list of stakeholders and their roles | ||
relevant to this document</FONT></P> | relevant to this document</FONT></P> | ||
Line 938: | Line 937: | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Training Requirements</FONT></H1> | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> | ||
<H1 CLASS="western"><FONT SIZE=4> | <H1 CLASS="western"><FONT SIZE=4>Useful Resources</FONT></H1> | ||
<P><BR><BR> | <P><BR><BR> | ||
</P> | </P> |
Revision as of 06:31, 13 February 2012
Introduction
Purpose
The purpose of this document is to define a high level QA strategy at Moodle and communicate that process to the relevant stakeholders. This document is loosely based upon the IEEE829 standard for software testing documentation; adaptations have been made to this standard to make the document more lightweight, suitable for use in an agile process and relevant to the organisation.
This document is test centric, only explaining the development process from a testing point of view. The scope of this document does not cover the details of the entire development process at Moodle but how testing is applied to it. This document also discusses potential benefits to the organisation of the processes discussed.
Scope
This document encompasses all testing activities at Moodle and how they will be performed. This includes:
Testing methodology at Moodle
How testing applies to the Moodle development life cycle
Managing legacy items
Hardware requirements
Testing tools and how they will be used
Test Environments
Roles and responsibilities
Training requirements
Testing Methodology at Moodle
The full <A HREF="https://docs.moodle.org/dev/Process">development life-cycle</A> at Moodle is documented in detail <A HREF="https://docs.moodle.org/dev/Process">elsewhere</A>. Moodle is currently developed using agile methodologies including scrum. The following stages of development take place within this life-cycle:
Weekly integration cycle
3 weekly stable sprints
6 monthly major releases
3 point releases between every major release
Testing Methodology
It is desirable on an agile project that a process of test driven development and continuous integration (CI) is performed. The aim of this is to increase business value and return on investment (ROI) by allowing software defects and issues - such as conflicting or problematic requirements - to be detected early in the software development life cycle (SDLC) when they can be resolved more easily. Although some manual testing is still required; to provide frequent and swift feedback, it is desirable that test automation is used where possible.
Currently there are no definition of distinct testing methodologies at Moodle. To give context to what types of tests should run and when, Moodle will adopt a process of categorising distinct types of test and their purpose.
Leading agile testing exponent, Brian Marick, introduced the concept of testing quadrants to define different categories of tests that accomplish different purposes:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_65cb6144.png" NAME="graphics1" ALIGN=LEFT WIDTH=643 HEIGHT=511 BORDER=0>Lisa Crispin and Janet Gregory explain these quadrants in detail in their book: “Agile Testing: A Practical Guide or Testers and Agile Teams”. To summarise, this is a list of traditional testing disciplines categorised relative to their context in an Agile process such as scrum.
Moodle is open source, community based software and this poses a number of unique challenges to a test driven, agile, process. Because Moodle is community based, with features and bug fixes developed externally and submitted to integration, some forms of testing defined here, such as paper prototyping, may not be relevant to the organisation. This section is therefore intended as reference material only, with the methods of testing described in this section being potential weapons in Moodle's testing arsenal; actual testing process will be defined later in the document.
Moodle will adopt the agile testing quadrant to define testing categories relevant to the organisation and facilitate communication of required testing on Moodle projects.
Quadrant Terms Explained
Tests that support the team – tests that assist with development and requirements gathering.
Tests that critique the product – constructive appraisal of the software under test to drive future development.
Technology facing – tests that verify scientific and technological aspects of the software under test.
Business facing – tests that verify the business logic, processes and communicate user requirements.
Testing Disciplines Explained
Functional Testing – Interacting positively and negatively with the software under test the way a user or process would, with specific expectations of inputs and outputs from the system under test.
Examples – Examples of user scenarios.
Story Tests – Defining a development story using a test.
Prototypes – e.g. paper prototyping of the system under development to gather and communicate customer requirements for the user interface.
Simulations – e.g. Wizard of Oz testing; similar to paper prototyping but involving inputs to and outputs from a human representing the system. Used for gathering functional requirements from users.
Unit testing – testing the behaviour of a particular piece of code.
Component testing – testing the interaction of interrelated pieces of code.
Exploratory testing – freestyle testing of the system under test. The scope of exploratory testing can be limited by defined constraints.
Scenarios – Specific business scenarios.
Usability testing – Using roles to exercise the system in a way the user might (something Moodle already does).
UAT – Real users executing tests to evaluate it's suitability relevant to business requirements.
Alpha/Beta – Suitable for software distributed to a large customer base where formal UAT may not be practical. Both have a similar goal to UAT. Alpha is an early distribution of the software, that may not be fully complete, to selected users for early feedback. Beta is distribution of almost finished software to users.
Performance and Load Testing – Testing the responsiveness of the software in normal circumstances and under high usage conditions.
Security Testing – Testing application security features.
'ility' Testing – Security, Maintainability, Interoperability, Compatibility, Scalability etc. etc.
Methodologies in the Moodle Development Process
As previously discussed, Moodle has an established process for developing and integrating software. This section applies relevant testing methodology to the existing process.
Generally a re-usable automated suite of functional regression tests, a test automation harness, it's supporting infrastructure and phased development of regular automated functional regression tests will be developed in an iterative manner using scrum. Initially these test cases will be based upon the current user acceptance test cases. The tools and other aspects of automation will be discussed later in the document.
Weekly (Continuous) Integration Cycle
Current testing of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. It will result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevent the issues from being dealt with in a timely manner when integrators are able to resolve those issues relatively easily.
The processes described here will move the bulk of testing and the majority of responsibility for capturing regressions from the Wednesday testing phase to earlier in the SDLC whilst also increasing test coverage.
Continuous Integration (CI)
Continuous integration provides early feedback of problematic code to the team. We will know early in the SDLC whether a working version of the code is available and what to do to make it work.
A CI server currently exists for the integration process. This server will be moved to a VM on the VM Server.
Functional Test Automation
Frequently running tests similar to those currently executed during the 6 monthly major release more regularly, earlier in the SDLC will assist with trapping regression issues at the point where integration into the existing code base occurs. This will help integrators either:
Protect existing code by allowing integrators to stop the new code being added.
Allow the immediate rectification of the regression issues at integration rather than finding them later in the SDLC.
The weekly integration cycle has a phase of manual bug retesting throughout the working day on Wednesdays (AWST). The automated functional regression suite described above will run outside of working hours (AWST) and can be run as often as required. The set up of test VMs will be performed by the
Performance Testing Matrix
Measuring system performance during this testing will assist with preventing degradation of performance in existing code caused by regressions.
To assist with early capture of performance degradation due to regression; a performance matrix will record performance against automated user interactions generated by the functional test harness.
Consideration will be given to modifying the functional test harness, selecting relevant test cases that will be re-run a given number of times to generate aggregated results as part of a performance and load suite. i.e. running the same test several times as part of a suite of tests in one “test run” will provide more accurate performance data, providing mean values, than running a test once.
New Unit Tests
Creating unit tests prior to creating the code starts the whole SDLC off with a focus on quality. If the code isn't considered complete until the test passes then the code itself will be of a high quality by the time it is implemented. It also creates a repository of unit regression tests in an iterative manner whilst not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing on new code.
The various XUnit code driven unit testing frameworks are universally popular, incredibly powerful and the industry standard for unit testing tools. PHPUnit will provide developers with a more powerful suite of tools for unit testing.
Everything submitted to integration must include unit tests in PHPUnit. It is to be encouraged that anyone developing software for Moodle should work in a test driven manner, where unit tests are written before writing the code that makes the test pass. The unit tests will remain with the system under test and can be run as regression tests using a CI server. The tests will run whenever any new code is added to any monitored GIT repository.
Legacy Unit Tests
Legacy unit tests are not run automatically. These tests can be run automatically on a regular basis via CI. Running these using CI, even using a workaround by executing them automatically via the Moodle user interface, will provide early regression testing on legacy code.
Unit tests are currently written using the SimpleTest tool. These tests are included with Moodle and can be run as an Admin user. These legacy tests must run automatically using a CI server.
Causal Analysis of Issues
As part of the weekly integration cycle, issues will be analysed and documented to determine the likely cause of the issue. The analysis will provide metrics for determining and removing areas of risk where defects are injected during development.
The metrics gathered during this exercise can be used to identify problem areas of the system.
Problematic areas of the system can be tested more thoroughly than less problematic areas.
The cause of the issues can be identified and eliminated.
Fewer route causes of issues will reduce the number of issues and increase ROI and overall quality.
Restricted Scope Manual Testing
The scope of testing issue fixes must be limited to retesting the issue itself. Clear instructions to specifically recreate the issue must be included in the testing instructions of the issue.
Restricting the scope of manual testing during the Wednesday (AWST) testing phase will reduce the effort required to retest the issues and yet there will actually be more testing performed than there is at the moment and earlier in the SDLC. This will only be possible if those raising issues provide precise instructions for recreating the problem.
Stable Sprints
Development work during stable sprints is performed in-house by Moodle HQ developers and will follow a process of Test Driven Development (TDD). The requirements for Stable sprints are gathered with testing steps from tracker issues. To allow this process to work effectively, so that requirements are not ambiguous, any new tracker issues must be raised with clear and unambiguous testing instructions.
Programmers working on the stable sprint will write unit tests and component tests from Q1 of the agile testing quadrants matrix that will prove their fixes to code works. Then they will write their fixes
Continuous Integration
To allow early feedback on fixes provided by the stable team they will require their own branch in GIT and their own instance of Jenkins. Developers will check their code into this branch at the end of each day and unit tests will run at check in and again overnight.
Functional Test Automation
Automated functional testing will take place as part of the regular integration cycle.
Major Releases and New Development
Point Releases
Managing Legacy Items
Hardware Requirements
A cloud based server will be used to virtualize all test and build platforms. The server is already in the process of being provisioned.
Server specifications are:
Make/Model: Dell T710
CPU: 2 x Intel Xeon E5645 2.4GHz, 12MB Cache 6 core CPU
Network: 2 x Broadcom 5709C dual-port gigabit ethernet module
RAM: 48GB (12x4GB) 1333MHz LV RDIMMS (triple channel)
Storage: 4 x 600GB 3.5" 15,000 RPM 6GBPS SAS hot plug hard disks with Dell H700 1GB Raid card configured for RAID 10.
Testing Tools and Their Use
All of the tools specified below are free with most being open source and extendible. The advantages are:
There is no cost to Moodle for licences.
Applications are web based with a tiny footprint and will run on low spec hardware/VMs.
Applications with databases all run on MySQL, which is free.
Moodle can extend the tools/create plugins, in-house, if customisations are required.
Continuous Integration
Continuous integration is the process of running tests frequently when code is checked into a repository. The aim of this is to keep code clean and allow the early capture of broken/incomplete code, early warning of conflicting changes (regressions) and a constantly available current “build” for demo and testing.
Jenkins
Jenkins is a fully extendible application that monitors executions of repeated jobs. Results are displayed via a web application.
Jenkins will be used as an information hub to provide Moodle with continuous feedback about the quality of Moodle software.
Jenkins can monitor any/multiple GIT repository for changes without adding hooks to the code and be used to run a set of tasks such as:
Pull git
Run unit tests
Create a moodle site
Add some information to it
Run the selenium tests
Report errors/warnings/failures
Phing
“PHing Is Not GNU make” is a build tool for PHP based upon Apache Ant. Phing commands are specified in XML just like in Ant. Phing can run PHPUnit and SimpleTest unit tests, execute shell commands, perform actions with GIT and much, much more.
As part of continuous integration Phing will be the engine behind Jenkins, performing tasks like getting the latest code for a test environment for the GIT repository and running unit tests.
Phing Jenkins Plugin
This plugin integrates Phing with Jenkins.
Clover PHP Plugin
To capture code coverage reports from PHPUnit.
PHPUnit
Unit testing tool for PHP; implementation is already planned.
SimpleTest
Moodle's legacy unit testing tool.
Other Automated Testing
Methods of testing other than unit testing will be automated. This includes, but is not limited to functional testing.
Selenium
Selenium is an extremely powerful, free and fully extendible browser automation tool that contains the following features:
Selenium IDE – Firefox plugin app for recording browser interactions with any web application. This is ideal for creating quick bug recreation test scripts and automation aided exploratory testing.
Selenium Webdriver – Selenium Server is a server application that runs Selenium test cases. Selenium webdriver is a collection of language specific bindings to drive a browser.
Selenium provides extensions to existing unit test tools such as JUnit. There are unofficial language bindings for PHPUnit and webdriver and an official backwards compatibility to version one of Selenium through PHPUnit.
Selenium test cases can be written in an object oriented manner and re-usable methods can be created.
Selenium is highly resistant to CSS changes and supports using XPath to locate page elements.
Furthermore Selenium server can be run in a grid configuration so that tests can run in parallel across multiple test environments controlled by one hub:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_1eb45cb5.png" NAME="graphics2" ALIGN=LEFT WIDTH=643 HEIGHT=454 BORDER=0>
When a test is required to run on a specific platform Selenium Grid can be made aware of the environments you wish to use. The Selenium Hub will then ensure that a test runs only on the Selenium slaves providing the requested environment:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_635b180f.png" NAME="graphics3" ALIGN=LEFT WIDTH=643 HEIGHT=454 BORDER=0>
Selenium Jenkins Plugin
A Jenkins plugin that gives Jenkins the ability to turn a Jenkins cluster into a Selenium Cluster. The plugin installs Selenium Grid on all of the slaves automatically and sets up the grid on it's own. Great for deploying the latest code to multiple Selenium test environments automatically.
Firebug
Firebug is Firefox plugin that gives a user the ability to easily inspect page source.
Firepath
Firepath is a Firefox plugin that can generate XPath expressions.
Eclipse IDE
Eclipse IDE will be used to design Selenium test cases.
JUnit/PHPUnit Selenium Extension
SeleniumHQ officially support JUnit extensions for Selenium. The JUnit support is more mature than PHPUnit and is fully updated to work with the latest version of Selenium Server.
Hosting Test Environments
Hosting instances of Jenkins and test environments will be performed using server virtualization.
XEN
Xen is a robust, secure, high performance type 1 baremetal hypervisor, or a virtual machine monitor (VMM). Xen can securely execute multiple virtual machines, each running its own OS, on a single physical system with close-to-native performance. Xen is open source, and it's used as the core virtualization engine for many vendors products.
Version Control
GIT
Moodle use GIT for version control.
Test Management and Defect Tracking
TestLink
TestLink is a free test management tool written in PHP. The test management tool is used to specify and manage tests and test suites, then report on test execution. A test management tool is useful for managing and co-ordinating organisational testing effort and documenting testing. Currently Moodle use Jira as the test management tool. Jira is software development management tool and not a test management tool.
In TestLink we can create testing projects, test plans and document test cases, manage test execution runs then log and report results for manual and automated tests.
TestLink Jenkins Plugin
This plugin integrates TestLink with Jenkins and generates reports on automated test execution.
Jira
Moodle already use Jira for development and project planning and tracking issues.
Test Environments
Test environments for automation will be hosted on the VM server. These will include a CI server (the build machine), database servers, web servers and client PCs.
Requirements Gathering
As a task while provisioning the VM Server, two surveys have been run to determine popular Test Platforms. Survey 1 gathered metrics from the Moodle Community and Survey 2 gathered metrics from Moodle Partners. Limiting factors of these surveys are that both samples are relatively small. The results of the surveys are as follows:
Survey 1 The Moodle Community:
Fig. 1 Separate Database and Web Server Yes/No:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m118a7170.png" NAME="graphics4" ALIGN=LEFT WIDTH=507 HEIGHT=312 BORDER=0>
Fig. 2 Operating System of Separate Web Server:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_1f770811.gif" NAME="Object1" ALIGN=LEFT WIDTH=333 HEIGHT=283>
Fig. 3 Operating System of Separate Database Server:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m7a7f2e77.gif" NAME="Object2" ALIGN=LEFT WIDTH=327 HEIGHT=291>
Fig. 4 What Databases are Used on a Split System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m6006a390.png" NAME="graphics5" ALIGN=LEFT WIDTH=394 HEIGHT=241 BORDER=0>
Fig. 5 What Databases are Used on an all in one Database Server and Webserver:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_171d6c07.png" NAME="graphics6" ALIGN=LEFT WIDTH=395 HEIGHT=248 BORDER=0>
Fig. 6 What Web Server is Used on a Split System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_42e1d441.png" NAME="graphics8" ALIGN=LEFT WIDTH=384 HEIGHT=293 BORDER=0>
Fig. 7 What Web Server is Used on an all in one Database Server and Webserver:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m5762a09b.png" NAME="graphics7" ALIGN=LEFT WIDTH=397 HEIGHT=300 BORDER=0>
As Debian and Ubuntu are similar and Red Hat and Centos are similar that,based upon our sample data, our three main operating used by the Moodle community Debian/Ubuntu, Red Hat/Centos and MS Server 2008 with a weighting of almost 75% towards the two Linux operating systems.
Of the five supported databases, the majority of community users prefer MySQL with PostgreSQL and MSSQL a fairly equal but distant second and third.
The Majority of community users also use Apache with IIS a very distant second.
7.1.1Survey 2 The Moodle Partners:
Fig. 8 Separate Database and Web Server Yes/No:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_331421f3.png" NAME="graphics9" ALIGN=LEFT WIDTH=381 HEIGHT=232 BORDER=0>
Fig. 9 Partner OS Database Server (combining answers for fig 8):
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_2141134d.gif" NAME="Object4" ALIGN=LEFT WIDTH=302 HEIGHT=265>
Fig. 10 Partner OS Web Server (combining answers for fig 8):
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m3bf269d.gif" NAME="Object3" ALIGN=LEFT WIDTH=302 HEIGHT=265>
Fig. 11 What Databases are Used on a Split System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m4d80602f.png" NAME="graphics10" ALIGN=LEFT WIDTH=389 HEIGHT=240 BORDER=0>
Fig. 12 What Databases are Used on a Combined System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m9c16939.png" NAME="graphics11" ALIGN=LEFT WIDTH=394 HEIGHT=250 BORDER=0>
Fig. 12 What Web Servers are used on a Split System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_17446f97.png" NAME="graphics12" ALIGN=LEFT WIDTH=398 HEIGHT=296 BORDER=0>
Fig. 13 What Web Servers are used on a Combined System:
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_m6e98ab34.png" NAME="graphics13" ALIGN=LEFT WIDTH=382 HEIGHT=298 BORDER=0>
This suggests that the majority of partners use Debian/Ubuntu or Centos/Rad Hat Linux, Apache and either a MySQL or PostgreSQL database.
VM Requirements
In true agile fashion and to promote flexibility, VM's for test environments will be modular i.e. they will be separate database servers and web servers. This will allow the mixing and matching of databases and their host OS to the web servers and their host OS as requirements change. At any time, any Web Server can connect to any Database Server. It should also reduce potential numbers of running VMs required for a particular platform.
<IMG SRC="moodle%20test%20process%20v0.1%20(draft)_html_6185ac88.gif" ALIGN=LEFT>A master hub VM is required for Jenkins and the hub for the Selenium grid.
Implementation
Agile testing guru and automation expert Brian Marick coined the term “Hump of Pain” to describe the initial phase of implementing test automation. The impact of this hump can be minimised by good planning, good coaching and a strong managerial support for the new processes. As time progresses, particularly as infrastructure becomes implemented, the test cases in the automated suite are completed and the introduced practices become second nature; the hump disappears.
Agile Implementation of Testing Practices
Modifications to the process should be made in an incremental manner based up scrum methodology. A series of 2 weekly sprints will be used to time-box test planning and implementation tasks.
Phase 1: Infrastructure and the Integration Process
The infrastructure to maintain automated testing effort must be implemented. This process will start with the installation of cloud based hardware at Moodle HQ and the setup of VM's to support the required test environments.
The integration process affects quality for the whole organisation as all code must go through the integration process to become 'live'. The processes described above will will be initially implemented at integration. Any changes to the existing processes will take place side-by-side with existing processes to manage risk. i.e. existing processes impacted by changes will be maintained in their current state until implementation of changes becomes stable.
The setup of test environments will initially start with a restricted number of high risk test environments so that the automation process, particularly test cases, can be developed incrementally.
Roles and Responsibilities
Below is a list of stakeholders and their roles relevant to this document
<COL WIDTH=85*> <COL WIDTH=85*> <COL WIDTH=85*>
Name |
Role |
Responsibilities |
Tim Barker |
Test Manager |
Co-ordinate testing effort Define organisational test strategy and requirements Specify test plans and cases |
Jordan Tomkinson |
System Administrator |
Provision of Hardware Commission test platforms and supporting infrastructure |
Martin Dougiamas |
Moodle Founder and MD |
Approval |
Michael de Raadt |
Development Manager |
Approval |
Eloy Lafuente |
Knight in Shining Armour |
Technical guidance and advice |
Helen Foster |
Community Manager |
Community liaison with regard to requirements gathering |
Moodle Developers |
Development |
Input into VM requirements Design and write unit tests Test Execution Performance Test Matrix (Sam Hemelryk) |
Training Requirements
Useful Resources
<A HREF="http://jenkins-php.org/">Template for Jenkins Jobs for PHP Projects</A>
<A HREF="http://www.phing.info/trac/">Phing</A>
<A HREF="https://wiki.jenkins-ci.org/display/JENKINS/Phing+Plugin">Phing Jenkins Plugin</A>
<A HREF="http://www.teamst.org/">TestLink</A>
<A HREF="https://wiki.jenkins-ci.org/display/JENKINS/TestLink+Plugin">Testlink Jenkins Plugin</A>
<A HREF="https://wiki.jenkins-ci.org/display/JENKINS/Clover+PHP+Plugin">Clover Jenkins Plugin</A>
<A HREF="http://seleniumhq.org/">Selenium</A>
<A HREF="https://wiki.jenkins-ci.org/display/JENKINS/Selenium+Plugin">Selenium Jenkins Plugin</A>
</BODY> </HTML>