Testing strategy/The Moodle Testing Process

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Methodologies in the Moodle Development Process

Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process. The full development life-cycle at Moodle is documented in detail elsewhere. Moodle is currently developed using Agile methodologies including Scrum. The following stages of development take place within this life-cycle:

• Weekly integration cycle

• 3-weekly stable sprints

• 6-monthly major releases

• 3 minor releases between every major release

Weekly (Continuous) Integration Cycle

A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevents issues from being dealt with in a timely manner when integrators, or even those writing the code, are able to resolve those issues relatively easily.

The processes described here ensure the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.

New Unit Tests

Creating unit tests prior to coding starts the whole SDLC off with a focus on quality. If code isn't considered complete until unit tests pass, then the code itself will be of a high quality by the time it is implemented. This also creates a repository of unit regression tests in an iterative manner while not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing of new code.

Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.

Integration Steps

Step 1: GIT Pull to integration CI server

When developers are ready to submit their code into the integration process, they set the status of the tracker issue to 'Ready for integration'. This in turn triggers the Jenkins CI job "Pre-check remote branch". This job performs the following tasks:

• Pull the changes to a local branch on the Integration CI Server.
  • git reset --hard origin => so it is in the same status than the official git repository
  • git pull => get the changes proposed by a developer
• Run checks on the code.
• Generate a results artefact that is sent to the developer.

Please note; each job begins by resetting the local git repository to the original state so the pulls are never permanent. This mitigates the risk posed by "collisions" as the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.

Step 2: Peer Review

Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.

Step 3: GIT Pull to integration

Code passing peer review can be pulled from the developers repository to integration where, among other things, the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.

Step 4: GIT Pull to Nightly Build Machine, performance and browser driven automation suite

A set of regression tests are run at 2AM AWST Daily. These tests are scheduled to run automatically from the Moodle test automation framework and run on all supported platforms e.g. all databases MySQL, Oracle, MSSQL, PostgreSQL. The following tests are performed in this order:

1. Moodle unit tests, including those submitted by developers.
2. PerfComp performance comparison tests.
3. Selenium acceptance tests, automated browser tests based upon the existing Moodle QA test suite.

Step 5: Issue retest

On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue test steps must be clear and concise, allowing the issue to be recreated easily. Testing steps must go in the testing instructions field to prove that the bug is resolved. Areas of Moodle to investigate with extra exploratory testing should be entered here also.

The format for writing test steps is: Pre-requisites: Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS. Test Steps: A clear and concise list of test steps required to test the issue with the expected results of each step. Actual Result: The actual result of what happened when the test is complete. Exploratory focus: Areas of Moodle that may need to be investigated further to identify regressions etc. An issue passes when the actual result of test no longer occurs and no regressions have been created.

Stable Sprints

TODO

Major Releases and New Development

TODO

Point Releases

TODO