Testing strategy/The Moodle Testing Process

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Methodologies in the Moodle Development Process

Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process.

The full development life-cycle at Moodle is documented in detail elsewhere. Moodle is currently developed using Agile methodologies including Scrum. The following stages of development take place within this life-cycle:

• Weekly integration cycle

• 3-weekly stable sprints

• 6-monthly major releases

• 3 minor releases between every major release

Weekly (Continuous) Integration Cycle

A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevents issues from being dealt with in a timely manner when integrators, or even those writing the code, are able to resolve those issues relatively easily.

The processes described here ensure the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.

New Unit Tests

Creating unit tests prior to coding starts the whole SDLC off with a focus on quality. If code isn't considered complete until unit tests pass, then the code itself will be of a high quality by the time it is implemented. This also creates a repository of unit regression tests in an iterative manner while not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing of new code.

Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.

Integration Steps

Step 1: GIT Push to pre-integration test server

Developers push changes to the pre-integration test repository. The changes are picked up by the Jenkins CI server, which runs a set of jobs duplicating those run at integration. The changes must not break the "build" before they pass through to the next step of the process. Therefore, the Jenkins jobs must all pass.

NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.

If changes cause a breakage then they must be backed out by the developer and a healthy "build" will be restored.

Step 2: Peer Review

Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.

Step 3: GIT Pull to integration

Code passing peer review can be pulled from the developers repository to integration where, among other things, the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.

Step 4: Functional automation suite

On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.

Step 5: Issue retest

On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue test steps must be clear and concise, allowing the issue to be recreated easily. Test recreation steps must go in the testing instructions field.

The format for writing test steps is:

Pre-requisites:

Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.

Test Steps:

A clear and concise list of test steps required to test the issue with the expected results of each step.

Actual Result:

The actual result of what happened when the test is complete.

Exploratory focus:

Areas of Moodle that may need to be investigated further to identify regressions etc.

An issue passes when the actual result of test no longer occurs and no regressions have been created.

Stable Sprints

TODO

Major Releases and New Development

TODO

Point Releases

TODO