Testing strategy/The Moodle Testing Process

Methodologies in the Moodle Development Process

Moodle has an established process for developing and integrating software. This section applies relevant testing methodology to the existing process.

Weekly (Continuous) Integration Cycle

A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevent the issues from being dealt with in a timely manner when integrators or even those writing the code are able to resolve those issues relatively easily.

The processes described here ensures the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Regression testing is performed much earlier in the SDLC and as an added bonus also increases test coverage.

Integration Steps

Functional Test Automation

Frequently running tests similar to those currently executed during the 6 monthly major release more regularly, earlier in the SDLC will assist with trapping regression issues at the point where integration into the existing code base occurs. This will help integrators either:

1. Protect existing code by allowing integrators to stop the new code being added.

2. Allow the immediate rectification of the regression issues at integration rather than finding them later in the SDLC.

The weekly integration cycle has a phase of manual bug retesting throughout the working day on Wednesdays (AWST). The automated functional regression suite described above will run outside of working hours (AWST) and can be run as often as required. The set up of test VMs will be performed by the

Performance Testing Matrix

Measuring system performance during this testing will assist with preventing degradation of performance in existing code caused by regressions.

To assist with early capture of performance degradation due to regression; a performance matrix will record performance against automated user interactions generated by the functional test harness.

Consideration will be given to modifying the functional test harness, selecting relevant test cases that will be re-run a given number of times to generate aggregated results as part of a performance and load suite. i.e. running the same test several times as part of a suite of tests in one “test run” will provide more accurate performance data, providing mean values, than running a test once.

New Unit Tests

Creating unit tests prior to creating the code starts the whole SDLC off with a focus on quality. If the code isn't considered complete until the test passes then the code itself will be of a high quality by the time it is implemented. It also creates a repository of unit regression tests in an iterative manner whilst not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing on new code.

The various XUnit code driven unit testing frameworks are universally popular, incredibly powerful and the industry standard for unit testing tools. PHPUnit will provide developers with a more powerful suite of tools for unit testing.

Everything submitted to integration must include unit tests in PHPUnit. It is to be encouraged that anyone developing software for Moodle should work in a test driven manner, where unit tests are written before writing the code that makes the test pass. The unit tests will remain with the system under test and can be run as regression tests using a CI server. The tests will run whenever any new code is added to any monitored GIT repository.

Legacy Unit Tests

Legacy unit tests are not run automatically. These tests can be run automatically on a regular basis via CI. Running these using CI, even using a workaround by executing them automatically via the Moodle user interface, will provide early regression testing on legacy code.

Unit tests are currently written using the SimpleTest tool. These tests are included with Moodle and can be run as an Admin user. These legacy tests must run automatically using a CI server.

Causal Analysis of Issues

As part of the weekly integration cycle, issues will be analysed and documented to determine the likely cause of the issue. The analysis will provide metrics for determining and removing areas of risk where defects are injected during development.

The metrics gathered during this exercise can be used to identify problem areas of the system.

1. Problematic areas of the system can be tested more thoroughly than less problematic areas.

2. The cause of the issues can be identified and eliminated.

Fewer route causes of issues will reduce the number of issues and increase ROI and overall quality.

Restricted Scope Manual Testing

The scope of testing issue fixes must be limited to retesting the issue itself. Clear instructions to specifically recreate the issue must be included in the testing instructions of the issue.

Restricting the scope of manual testing during the Wednesday (AWST) testing phase will reduce the effort required to retest the issues and yet there will actually be more testing performed than there is at the moment and earlier in the SDLC. This will only be possible if those raising issues provide precise instructions for recreating the problem.

Stable Sprints

Development work during stable sprints is performed in-house by Moodle HQ developers and will follow a process of Test Driven Development (TDD). The requirements for Stable sprints are gathered with testing steps from tracker issues. To allow this process to work effectively, so that requirements are not ambiguous, any new tracker issues must be raised with clear and unambiguous testing instructions.

Programmers working on the stable sprint will write unit tests and component tests from Q1 of the agile testing quadrants matrix that will prove their fixes to code works. Then they will write their fixes

Continuous Integration

To allow early feedback on fixes provided by the stable team they will require their own branch in GIT and their own instance of Jenkins. Developers will check their code into this branch at the end of each day and unit tests will run at check in and again overnight.

Functional Test Automation

Automated functional testing will take place as part of the regular integration cycle.

Major Releases and New Development

TODO

Point Releases

TODO