Testing strategy/The Moodle Testing Process: Difference between revisions
Tim Barker (talk | contribs) No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
{{Work in progress}} | |||
<H1 CLASS="western">Methodologies in the Moodle Development Process</H1> | <H1 CLASS="western">Methodologies in the Moodle Development Process</H1> | ||
<P>Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process.</P> | <P>Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process.</P> | ||
<P>The full [https://docs.moodle.org/dev/Process development] | <P>The full [https://docs.moodle.org/dev/Process development] | ||
life-cycle at Moodle is documented in detail [https://docs.moodle.org/dev/ | life-cycle at Moodle is documented in detail [https://docs.moodle.org/dev/Releases#General_release_calendar elsewhere]. | ||
Moodle is currently developed using | Moodle is currently developed using Agile methodologies including | ||
Scrum. The following stages of development take place within this | |||
life-cycle:</P> | life-cycle:</P> | ||
<UL> | <UL> | ||
<LI><P>Weekly integration cycle</P> | <LI><P>Weekly integration cycle</P> | ||
<LI><P>3 weekly stable sprints</P> | <LI><P>3-weekly stable sprints</P> | ||
<LI><P>6 monthly major releases</P> | <LI><P>6-monthly major releases</P> | ||
<LI><P>3 | <LI><P>3 minor releases between every major release</P> | ||
</UL> | </UL> | ||
<H2 CLASS="western">Weekly (Continuous) Integration | <H2 CLASS="western">Weekly (Continuous) Integration | ||
| Line 20: | Line 21: | ||
during unguided exploratory testing and worse still the introduction | during unguided exploratory testing and worse still the introduction | ||
of regressions into Master. Discovery of issues late in the SDLC | of regressions into Master. Discovery of issues late in the SDLC | ||
prevents issues from being dealt with in a timely manner when | |||
integrators or even those writing the code are able to resolve those issues relatively easily. | integrators, or even those writing the code, are able to resolve those issues relatively easily. | ||
</P> | </P> | ||
<P>The processes described here | <P>The processes described here ensure the bulk of | ||
testing and the majority of responsibility for capturing regressions | testing and the majority of responsibility for capturing regressions | ||
does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.</P> | does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.</P> | ||
<H2 CLASS="western">New Unit Tests</H2> | <H2 CLASS="western">New Unit Tests</H2> | ||
<P>Creating unit tests prior to | <P>Creating unit tests prior to coding starts | ||
the whole SDLC off with a focus on quality. If | the whole SDLC off with a focus on quality. If code isn't | ||
considered complete until | considered complete until unit tests pass, then the code itself will | ||
be of a high quality by the time it is implemented. | be of a high quality by the time it is implemented. This also creates a | ||
repository of unit regression tests in an iterative manner | repository of unit regression tests in an iterative manner while not | ||
requiring of a great deal of extra effort to write those tests. As | requiring of a great deal of extra effort to write those tests. As | ||
these tests are added, they will provide early regression testing | these tests are added, they will provide early regression testing of | ||
new code.</P> | new code.</P> | ||
<P>Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.</P> | <P>Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.</P> | ||
| Line 39: | Line 40: | ||
<H2 CLASS="western">Integration Steps</H2> | <H2 CLASS="western">Integration Steps</H2> | ||
<H3 CLASS="western">Step 1: GIT Push to integration test server</H3> | <H3 CLASS="western">Step 1: GIT Push to integration test server</H3> | ||
<P>Developers push changes to the integration test repository. The changes are picked up by the Jenkins CI server, which runs a | <P>Developers push changes to the integration test repository. The changes are picked up by the Jenkins CI server, which runs a set of jobs duplicating those run at integration. The changes must not break the "build" before they pass through to the next step of the process. Therefore, the Jenkins jobs must all pass.</P> | ||
<P>NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.</P> | <P>NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.</P> | ||
<P>If changes cause a | <P>If changes cause a breakage then they must be backed out by the developer and a healthy "build" will be restored.</P> | ||
<H3 CLASS="western">Step 2: Peer Review</H3> | <H3 CLASS="western">Step 2: Peer Review</H3> | ||
<P>Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.</P> | <P>Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.</P> | ||
<H3 CLASS="western">Step 3: GIT Pull to integration</H3> | <H3 CLASS="western">Step 3: GIT Pull to integration</H3> | ||
<P>Code passing peer review can be pulled from | <P>Code passing peer review can be pulled from the developers repository to integration where, among other things, the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.</P> | ||
<H3 CLASS="western">Step 4: Functional automation suite</H3> | <H3 CLASS="western">Step 4: Functional automation suite</H3> | ||
<P>On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.</P> | <P>On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.</P> | ||
<H3 CLASS="western">Step 5: Issue retest</H3> | <H3 CLASS="western">Step 5: Issue retest</H3> | ||
<P>On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue | <P>On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue test steps must be clear and concise, allowing the issue to be recreated easily. Test recreation steps must go in the testing instructions field.</P> | ||
<P>The format for writing | <P>The format for writing test steps is:</P> | ||
<P>'''Pre-requisites:'''</P> | <P>'''Pre-requisites:'''</P> | ||
<P>Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.</P> | <P>Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.</P> | ||
<P>'''Test Steps:'''</P> | <P>'''Test Steps:'''</P> | ||
<P>A clear and concise list of test steps required to | <P>A clear and concise list of test steps required to test the issue with the expected results of each step.</P> | ||
<P>'''Actual Result:'''</P> | <P>'''Actual Result:'''</P> | ||
<P>The actual result of what happened when the test | <P>The actual result of what happened when the test is complete.</P> | ||
<P>'''Exploratory focus:'''</P> | <P>'''Exploratory focus:'''</P> | ||
<P>Areas of Moodle that may need to be investigated further to identify regressions etc.</P> | <P>Areas of Moodle that may need to be investigated further to identify regressions etc.</P> | ||
<P>An issue passes when the actual result of test no longer occurs. Therefore an issue is passed before any additional exploratory testing has taken place.</P> | <P>An issue passes when the actual result of test no longer occurs. Therefore an issue is passed before any additional exploratory testing has taken place.</P> | ||
<H3 CLASS="western">Step 6: Guided exploratory testing</H3> | <H3 CLASS="western">Step 6: Guided exploratory testing</H3> | ||
<P> | <P>Following explicit testing following test steps, testers perform guided exploratory testing in an effort to identify regressions and new issues. The focus for this testing is derived from the "Exploratory focus" instructions.</P> | ||
<H2 CLASS="western">Stable Sprints</H2> | <H2 CLASS="western">Stable Sprints</H2> | ||
'''TODO''' | '''TODO''' | ||
Revision as of 05:54, 15 March 2012
Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.
Methodologies in the Moodle Development Process
Moodle has an established process for developing and integrating software. This section describes relevant testing methodology within the existing process.
The full development life-cycle at Moodle is documented in detail elsewhere. Moodle is currently developed using Agile methodologies including Scrum. The following stages of development take place within this life-cycle:
Weekly integration cycle
3-weekly stable sprints
6-monthly major releases
3 minor releases between every major release
Weekly (Continuous) Integration Cycle
A manual testing phase of integration issues occurs towards the end of the SDLC at Moodle during the Wednesday (AWST) testing phase. This is the last possible chance Moodle gets to capture regressions. Relying soley on this phase to capture regressions could result in the late discovery of regressions during unguided exploratory testing and worse still the introduction of regressions into Master. Discovery of issues late in the SDLC prevents issues from being dealt with in a timely manner when integrators, or even those writing the code, are able to resolve those issues relatively easily.
The processes described here ensure the bulk of testing and the majority of responsibility for capturing regressions does not lie with the Wednesday testing phase. Testing is performed as early in the SDLC as possible.
New Unit Tests
Creating unit tests prior to coding starts the whole SDLC off with a focus on quality. If code isn't considered complete until unit tests pass, then the code itself will be of a high quality by the time it is implemented. This also creates a repository of unit regression tests in an iterative manner while not requiring of a great deal of extra effort to write those tests. As these tests are added, they will provide early regression testing of new code.
Unit tests in the PHPUnit framework are a requirement when submitting code to Moodle.
Integration Steps
Step 1: GIT Push to integration test server
Developers push changes to the integration test repository. The changes are picked up by the Jenkins CI server, which runs a set of jobs duplicating those run at integration. The changes must not break the "build" before they pass through to the next step of the process. Therefore, the Jenkins jobs must all pass.
NOTE: the purpose of this step is not to test for "collisions" between conflicting patches but to verify that code meets the required standard to proceed to peer review.
If changes cause a breakage then they must be backed out by the developer and a healthy "build" will be restored.
Step 2: Peer Review
Any code sent back to the developer at Peer Review, after any changes have been made, must be must pass step 1 again before re-review.
Step 3: GIT Pull to integration
Code passing peer review can be pulled from the developers repository to integration where, among other things, the main Jenkins integration jobs run. At this stage integrators also check for "collisions" and conflicts between submitted patches.
Step 4: Functional automation suite
On Tuesday (AWST) a set of regression tests are run overnight (Tues-Weds). These tests are scheduled to run automatically from the Moodle test automation framework and are based upon Moodle's User Acceptance Test suite.
Step 5: Issue retest
On Wednesday (AWST) the manual issue retesting is performed at Moodle HQ. Issue test steps must be clear and concise, allowing the issue to be recreated easily. Test recreation steps must go in the testing instructions field.
The format for writing test steps is:
Pre-requisites:
Everything required to allow the test to start. This can include, but is not limited to: issue specific data, test platform information: e.g. OS.
Test Steps:
A clear and concise list of test steps required to test the issue with the expected results of each step.
Actual Result:
The actual result of what happened when the test is complete.
Exploratory focus:
Areas of Moodle that may need to be investigated further to identify regressions etc.
An issue passes when the actual result of test no longer occurs. Therefore an issue is passed before any additional exploratory testing has taken place.
Step 6: Guided exploratory testing
Following explicit testing following test steps, testers perform guided exploratory testing in an effort to identify regressions and new issues. The focus for this testing is derived from the "Exploratory focus" instructions.
Stable Sprints
TODO
Major Releases and New Development
TODO
Point Releases
TODO
