Note:

This site is no longer used and is in read-only mode. Instead please go to our new Moodle Developer Resource site.

Testing strategy/The Moodle Testing Process: Difference between revisions

From MoodleDocs
← Older editNewer edit →
VisualWikitext
Tim Barker (talk | contribs)
361 edits
No edit summary
Tim Barker (talk | contribs)
361 edits
No edit summary
Line 14: Line 14:
<P>The processes described here ensures the bulk of
<P>The processes described here ensures the bulk of
testing and the majority of responsibility for capturing regressions
testing and the majority of responsibility for capturing regressions
does not lie with the Wednesday testing phase. Regression testing is performed much earlier in the SDLC and as an added bonus also
does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.</P>
increases test coverage.</P>
<H2 CLASS="western">New Unit Tests</H2>
<P>Creating unit tests prior to creating the code starts
the whole SDLC off with a focus on quality. If the code isn't
considered complete until the test passes then the code itself will
be of a high quality by the time it is implemented. It also creates a
repository of unit regression tests in an iterative manner whilst not
requiring of a great deal of extra effort to write those tests. As
these tests are added, they will provide early regression testing on
new code.</P>
<P>Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.</P>
 
<H2 CLASS="western">Integration Steps</H2>
<H2 CLASS="western">Integration Steps</H2>
<H3 CLASS="western">Step 1: GIT Push to integration test server</H3>
<H3 CLASS="western">Step 1: GIT Push to integration test server</H3>
Line 44: Line 54:
<P>Guided exploratory testing is then performed after issue retests are done and the issues pass or fail. Any issues found during guided exploratory testing are dealt with at the discretion of the Development Manager.</P>
<P>Guided exploratory testing is then performed after issue retests are done and the issues pass or fail. Any issues found during guided exploratory testing are dealt with at the discretion of the Development Manager.</P>


<H3 CLASS="western">Functional Test Automation</H3>
<P>Frequently running tests similar to those currently
executed during the 6 monthly major release  more regularly, earlier
in the SDLC will assist with trapping regression issues at the point
where integration into the existing code base occurs. This will help
integrators either:</P>
<OL>
<LI><P>Protect existing code by allowing integrators to
stop the new code being added.</P>
<LI><P>Allow the immediate rectification of the
regression issues at integration rather than finding them later in
the SDLC.</P>
</OL>
<P>The weekly integration cycle has a phase of manual
bug retesting throughout the working day on Wednesdays (AWST).  The
automated functional regression suite described above will run
outside of working hours (AWST) and can be run as often as required.
The set up of test VMs will be performed by the
</P>
<H3 CLASS="western">Performance Testing Matrix</H3>
<P>Measuring system performance during this testing will
assist with preventing degradation of performance in existing code
caused by regressions.</P>
<P>To assist with early capture of performance
degradation due to regression; a performance matrix will record
performance against automated user interactions generated by the
functional test harness.</P>
<P>Consideration will be given to modifying the
functional test harness, selecting relevant test cases that will be
re-run a given number of times to generate aggregated results as part
of a performance and load suite. i.e. running the same test several
times as part of a suite of tests in one “test run” will provide
more accurate performance data, providing mean values, than running a
test once.</P>
<H3 CLASS="western">New Unit Tests</H3>
<P>Creating unit tests prior to creating the code starts
the whole SDLC off with a focus on quality. If the code isn't
considered complete until the test passes then the code itself will
be of a high quality by the time it is implemented. It also creates a
repository of unit regression tests in an iterative manner whilst not
requiring of a great deal of extra effort to write those tests. As
these tests are added, they will provide early regression testing on
new code.</P>
<P>The various XUnit code driven unit testing frameworks
are universally popular, incredibly powerful and the industry
standard for unit testing tools. PHPUnit will provide developers with
a more powerful suite of tools for unit testing.</P>
<P>Everything submitted to integration must include unit
tests in PHPUnit. It is to be encouraged that anyone developing
software for Moodle should work in a test driven manner, where unit
tests are written before writing the code that makes the test pass.
The unit tests will remain with the system under test and can be run
as regression tests using a CI server. The tests will run whenever
any new code is added to any monitored GIT repository.</P>
<H3 CLASS="western">Legacy Unit Tests</H3>
<P>Legacy unit tests are not run automatically. These
tests can be run automatically on a regular basis via CI. Running
these using CI, even using a workaround by executing them
automatically via the Moodle user interface, will provide early
regression testing on legacy code.</P>
<P>Unit tests are currently written using the SimpleTest
tool. These tests are included with Moodle and can be run as an Admin
user. These legacy tests must run automatically using a CI server.</P>
<H3 CLASS="western">Causal Analysis of Issues</H3>
<P>As part of the weekly integration cycle, issues will
be analysed and documented to determine the likely cause of the
issue. The analysis will provide metrics for determining and removing
areas of risk where defects are injected during development.</P>
<P>The metrics gathered during this exercise can be used
to identify problem areas of the system.</P>
<OL>
<LI><P>Problematic areas of the system can be tested
more thoroughly than less problematic areas.</P>
<LI><P>The cause of the issues can be identified and
eliminated.</P>
</OL>
<P>Fewer route causes of issues will reduce the number
of issues and increase ROI and overall quality.</P>
<H3 CLASS="western">Restricted Scope Manual Testing</H3>
<P>The scope of testing issue fixes must be limited to
retesting the issue itself. Clear instructions to specifically
recreate the issue must be included in the testing instructions of
the issue.</P>
<P>Restricting the scope of manual testing during the
Wednesday (AWST) testing phase will reduce the effort required to
retest the issues and yet there will actually be more testing
performed than there is at the moment and earlier in the SDLC. This
will only be possible if those raising issues provide precise
instructions for recreating the problem.</P>
<H2 CLASS="western">Stable Sprints</H2>
<H2 CLASS="western">Stable Sprints</H2>
<P>Development work during stable sprints is performed
'''TODO'''
in-house by Moodle HQ developers and will follow a process of Test
Driven Development (TDD). The requirements for Stable sprints are
gathered with testing steps from tracker issues. To allow this
process to work effectively, so that requirements are not ambiguous,
any new tracker issues must be raised with clear and unambiguous
testing instructions.
</P>
<P>Programmers working on the stable sprint will write
unit tests and component tests from Q1 of the agile testing quadrants
matrix that will prove their fixes to code works. Then they will
write their fixes</P>
<H3 CLASS="western">Continuous Integration</H3>
<P>To allow early feedback on fixes provided by the
stable team they will require their own branch in GIT and their own
instance of Jenkins. Developers will check their code into this
branch at the end of each day and unit tests will run at check in and
again overnight.</P>
<H3 CLASS="western">Functional Test Automation</H3>
<P>Automated functional testing will take place as part
of the regular integration cycle.</P>
<H2 CLASS="western">Major Releases and New
<H2 CLASS="western">Major Releases and New
Development</H2>
Development</H2>

Revision as of 04:00, 15 March 2012

Methodologies in the Moodle Development Process

Moodle has an established process for developing and integrating software. This section applies relevant testing methodology to the existing process.

Weekly (Continuous) Integration Cycle

A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevent the issues from being dealt with in a timely manner when integrators or even those writing the code are able to resolve those issues relatively easily.

The processes described here ensures the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.

New Unit Tests

Creating unit tests prior to creating the code starts the whole SDLC off with a focus on quality. If the code isn't considered complete until the test passes then the code itself will be of a high quality by the time it is implemented. It also creates a repository of unit regression tests in an iterative manner whilst not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing on new code.

Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.

Integration Steps

Step 1: GIT Push to integration test server

Developers push changes to the integration test repository. The changes are picked up by the Jenkins CI server, which runs a set of duplicate set of jobs to those for the integration. The changes must not break the "build" before they pass through to the next step of the process. I.E. the Jenkins jobs must all pass.

NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.

If changes cause a break then then they must be backed out by the developer before they can go to peer review.

Step 2: Peer Review

Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.

Step 3: GIT Pull to integration

Code passing peer review can be pulled from integration test to integration where, amongst other things the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.

Step 4: Functional automation suite

On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.

Step 5: Issue retest

On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue recreation test steps must be clear and concise allowing the issue to be recreated easily. Test recreation steps must go in the testing instructions field.

The format for writing Issue recreation steps is:

Pre-requisites:

Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.

Test Steps:

A clear and concise list of test steps required to recreate the issue with the expected results of each step. The expected results must include include what was supposed to happen at the failure step.

Actual Result:

The actual result of what happened when the test failed i.e. how the issue manifests itself.

Exploratory focus:

Areas of Moodle that may need to be investigated further to identify regressions etc.

An issue passes when the actual result of test no longer occurs. Therefore an issue is passed before any additional exploratory testing has taken place.

Step 6: Guided exploratory testing

Towards the end of the working day at Moodle HQ, testers perform guided exploratory testing in an effort to identify regressions and new issues. The focus for this testing for each tester is derived from the "Exploratory focus" instructions in all of the issues that have been assigned to them.

When a tester is assigned issues to retest, the test immediately collates and makes a note of the "Exploratory focus" in each of the issues. This is done prior to commencing step 5 because the issues pass or fail before any exploratory testing takes place.

Guided exploratory testing is then performed after issue retests are done and the issues pass or fail. Any issues found during guided exploratory testing are dealt with at the discretion of the Development Manager.

Stable Sprints

TODO

Major Releases and New Development

TODO

Point Releases

TODO

Retrieved from "https://docs.moodle.org/dev/index.php?title=Testing_strategy/The_Moodle_Testing_Process&oldid=32831"
Powered by MediaWiki