Testing strategy/The Moodle Testing Process

Methodologies in the Moodle Development Process

Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process.

The full development life-cycle at Moodle is documented in detail elsewhere. Moodle is currently developed using agile methodologies including scrum. The following stages of development take place within this life-cycle:

• Weekly integration cycle

• 3 weekly stable sprints

• 6 monthly major releases

• 3 point releases between every major release

Weekly (Continuous) Integration Cycle

A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevent the issues from being dealt with in a timely manner when integrators or even those writing the code are able to resolve those issues relatively easily.

The processes described here ensures the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.

New Unit Tests

Creating unit tests prior to creating the code starts the whole SDLC off with a focus on quality. If the code isn't considered complete until the test passes then the code itself will be of a high quality by the time it is implemented. It also creates a repository of unit regression tests in an iterative manner whilst not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing on new code.

Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.

Integration Steps

Step 1: GIT Push to integration test server

Developers push changes to the integration test repository. The changes are picked up by the Jenkins CI server, which runs a set of duplicate set of jobs to those for the integration. The changes must not break the "build" before they pass through to the next step of the process. I.E. the Jenkins jobs must all pass.

NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.

If changes cause a break then then they must be backed out by the developer and a healthy "build" will be restored.

Step 2: Peer Review

Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.

Step 3: GIT Pull to integration

Code passing peer review can be pulled from integration test to integration where, amongst other things the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.

Step 4: Functional automation suite

On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.

Step 5: Issue retest

On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue recreation test steps must be clear and concise allowing the issue to be recreated easily. Test recreation steps must go in the testing instructions field.

The format for writing Issue recreation steps is:

Pre-requisites:

Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.

Test Steps:

A clear and concise list of test steps required to recreate the issue with the expected results of each step. The expected results must include include what was supposed to happen at the failure step.

Actual Result:

The actual result of what happened when the test failed i.e. how the issue manifests itself.

Exploratory focus:

Areas of Moodle that may need to be investigated further to identify regressions etc.

An issue passes when the actual result of test no longer occurs. Therefore an issue is passed before any additional exploratory testing has taken place.

Step 6: Guided exploratory testing

Towards the end of the working day at Moodle HQ, testers perform guided exploratory testing in an effort to identify regressions and new issues. The focus for this testing for each tester is derived from the "Exploratory focus" instructions in all of the issues that have been assigned to them.

When a tester is assigned issues to retest, the test immediately collates and makes a note of the "Exploratory focus" in each of the issues. This is done prior to commencing step 5 because the issues pass or fail before any exploratory testing takes place.

Guided exploratory testing is then performed after issue retests are done and the issues pass or fail. Any issues found during guided exploratory testing are dealt with at the discretion of the Development Manager.

Stable Sprints

TODO

Major Releases and New Development

TODO

Point Releases

TODO