Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

QA testing: Difference between revisions

From MoodleDocs
(QA testing latest update - starting 2.8 testing)
(→‎Running tests: when it is OK to assign more than one issue)
(42 intermediate revisions by 5 users not shown)
Line 3: Line 3:
Real users systematically try each feature in Moodle and test that it works in the current version of the Moodle code. These tests are repeated in series of cycles, usually just before major releases.
Real users systematically try each feature in Moodle and test that it works in the current version of the Moodle code. These tests are repeated in series of cycles, usually just before major releases.


<p class="note">'''QA testing latest''': ''We're currently involved in [https://tracker.moodle.org/browse/MDLQA-7170 Moodle 2.8 QA testing]. More testers are always welcome (see below).''</p>
<p class="note">'''QA testing latest''': ''We're currently involved in [https://tracker.moodle.org/browse/MDLQA-10999 Moodle 3.4 QA testing]. More testers are always welcome (see below).''</p>


==Getting involved==
==Getting involved==


Would you like to help with QA testing? If so, please make sure you have created an account in the Moodle tracker, then contact [https://moodle.org/user/view.php?id=24152&course=1 Helen]. Also, make sure you're subscribed to the [https://moodle.org/mod/forum/view.php?id=56 Testing and QA forum] in order to receive QA testing news updates.
Would you like to help with QA testing? If so, please make sure you have created an account in the [[Tracker_introduction|Moodle tracker]], then contact [https://moodle.org/user/profile.php?id=24152&course=1 Helen]. Also, make sure you're subscribed to the [https://moodle.org/mod/forum/view.php?id=56 Testing and QA forum] in order to receive QA testing news updates.


All testers are members of the testers group in the Moodle Tracker, enabling them to pass or fail QA tests and add comments.
All testers are members of the testers group in the Moodle Tracker, enabling them to pass or fail QA tests and add comments.
Line 13: Line 13:
==Running tests==
==Running tests==


# Select a test from the list in MDLQA-7170.If you wish, you can click the 'Assign to me' button, so that nobody else chooses the same test to run. (If you then find you are unable to run the test, you can assign the issue to 'Nobody'.)
# Select a test from MDLQA-10999. When viewing a test, if you wish, you can click the 'Assign to me' link on the right, so that nobody else chooses the same test to run. (If you then find you are unable to run the test, you can assign the issue back to 'Unassigned'.) Please assign issues to yourself which no one else is testing (Assignee = Unassigned). Only assign one issue at a time unless you plan to test a number of related issues within the next 24 hours. In other words, don't assign several issues to yourself then do nothing for several days. ;-)
# Using either the [http://qa.moodle.net/ Moodle QA Testing Site] or your own test site running the latest Moodle 2.8 beta (available from Git on the integration/master branch ''<nowiki>git://git.moodle.org/integration.git</nowiki>'') with [[:en:Debugging|debugging]] set to developer, perform each of the steps listed in the test
#* The label ''test_server_required'' indicates issues that can't be tested on the QA testing site.
#* Note that the '''test_server_required''' label has been added to the MDLQA issues that can not be tested using only the Moodle QA Testing Site and requires an additional or external server.
#* The label ''credentials_required'' indicates that credentials such as an OAuth 2 service client ID and secret are required.
# Using either the [http://qa.moodle.net/ Moodle QA Testing Site] or your own test site running the latest Moodle 3.4 beta (available from Git on the integration/master branch ''<nowiki>git://git.moodle.org/integration.git</nowiki>'') with [[:en:Debugging|debugging]] set to developer, perform each of the steps listed in the test. ''Please test using both the Clean and Boost themes.''
# Choose an appropriate workflow action:
# Choose an appropriate workflow action:
#* ''Pass'' - Test runs perfectly :)
#* ''Pass'' - Test runs perfectly. Add comment such as feedback about a new feature, browsers used for testing (if applicable), or simply "This test passes - yippee!"
#* ''Fail'' - Something doesn't work, or you obtain debugging messages. See below for actions to take. If in doubt whether to pass a test, give it a fail and add a comment describing your doubts.
#* ''Fail'' - Something doesn't work, or you obtain debugging messages. Add comment describing the step that doesn't work. If in doubt whether to pass a test, give it a fail and add a comment describing your doubts.
#* ''Obsolete'' - Test is no longer relevant in the current Moodle version.
#* ''Obsolete'' - Test is no longer relevant in the current Moodle version. Add comment explaining why.
# Add a comment (e.g. "This test passes - yippee!") to confirm that the test wasn't accidentally passed without being run.


Please note: If you are behind a firewall you do not control or content filtering system, it may interfere with the use of the Moodle QA site. It is advisable that testing is completed where such complications are not in place.
If you notice that the test description is out-of-date, add a comment suggesting how it can be updated and type @Helen Foster then select Helen from the menu below the input box. Alternatively, if you'd like to help with updating the test yourself, see below.
 
If you notice that the test description is out-of-date, add Helen as a watcher then add a comment suggesting how it can be updated.
 
Note: Always use the latest version of Moodle 2.8 beta for testing or the Moodle QA Testing Site.


===Moodle QA Testing Site===
===Moodle QA Testing Site===
Line 34: Line 30:
To prevent the site being used for sending spam, no emails are sent from it. Thus, tests involving email cannot be run using the Moodle QA Testing Site. (If such tests are attempted, an email debug message is displayed. This is not a bug but rather expected behaviour.)
To prevent the site being used for sending spam, no emails are sent from it. Thus, tests involving email cannot be run using the Moodle QA Testing Site. (If such tests are attempted, an email debug message is displayed. This is not a bug but rather expected behaviour.)


Manager, teacher and student accounts are provided. Please contact [https://moodle.org/user/view.php?id=24152&course=1 Helen] if you would like admin access to the Moodle QA Testing Site for running certain tests.
Manager, teacher and student accounts are provided. Please contact Helen if you would like admin access to the Moodle QA Testing Site for running certain tests.


==<div id="failedTests">Failed tests</div>==
==<div id="failedTests">Failed tests</div>==
Line 56: Line 52:
After integrating a fix,  
After integrating a fix,  


1. Reset the MDLQA test, adding the following comment (or similar):
# Reset the MDLQA test, adding a comment.
# Remove the 'mdlqa' label from the MDL issue.
# If the issue doesn't have testing instructions, pass it with message "Will be tested by MDLQA-XXXX".
 
The tester will then receive email notification that the bug is fixed and will hopefully decide to run the test again soon.


Suggested comment:
  Thanks for running this QA test. The bug causing it to fail has now been fixed, so the test can be reset and run again.
  Thanks for running this QA test. The bug causing it to fail has now been fixed, so the test can be reset and run again.
  Please note that if you are using http://qa.moodle.net for testing, you will need to wait for the site to be updated at 13:00 UTC in order to test the bug fix.
  Please note that if you are using http://qa.moodle.net for testing, you will need to wait for the site to be updated at 13:00 UTC in order to test the bug fix.


2. Remove the 'mdlqa' label from the MDL issue.
==Updating tests==
 
3. If the issue has not own testing instructions, pass it with message "Will be tested by MDLQA-XXXX"
 
The tester will then receive email notification that the bug is fixed and will hopefully decide to run the test again soon.
 
==The role of core developers in creating tests==


When a developer adds functionality to Moodle, they should also do the following.
QA tests often become out-of-date due to new developments. If you would like to help with updating tests, you'll need to be a member of the test writers group in the Tracker. Please contact Helen about being added.


# Write user documentation in the dev docs about the new feature (this should happen as part of the normal development process). Ask someone to proof read this documentation.
To update a QA test:
# Add a feature to the release notes. This feature should include a link to the appropriate Tracker issue, which will contain testing instructions. The feature should also include a link to the user documentation for the feature.


The QA tests will be written by a test writer based on the documentation and release notes written by the developer.
# Search for the master copy of the test - with affects version as 'Master copy' and MDLQA-1 as parent.
# Edit the test description.
# Optional: Add a comment describing your edit.


==Writing new tests==
==Writing new tests==


Would you like to help with writing new QA tests? If so, you'll need to be a member of the test writers group in the Tracker. Please contact [http://moodle.org/user/view.php?id=24152&course=1 Helen] about being added.
Would you like to help with writing new QA tests? If so, as for updating tests, you'll need to be a member of the test writers group in the Tracker. Please contact Helen about being added.


To create a new QA test:
To create a new QA test:
Line 86: Line 82:
# Select Master copy as affected version
# Select Master copy as affected version
# Select appropriate components including at least one of the following: User, Student, Teacher, Administrator
# Select appropriate components including at least one of the following: User, Student, Teacher, Administrator
# Leave the assignee as automatic (nobody)
# Set assignee as unassigned
# Write the test (usually between 3 and 10 steps)
# Write the test (usually between 3 and 10 steps). It's a good idea to try doing the steps yourself as you write the test.
# Add a comment to the MDL issue mentioning the new QA test and remove the qa_test_required label.
# Go to the MDL issue and create a ‘has a QA test’ link to the new QA test, adding a comment “Thanks for this improvement which is now covered by the QA test MDLQA....”, and if necessary remove the qa_test_required label. ''Make sure that all QA tests are linked back to a corresponding MDL issue.''


New tests will be included in the next QA cycle.
New tests will be included in the next QA cycle.
Line 94: Line 90:
==Feedback==
==Feedback==


Feedback on all aspects of our QA testing process is welcome. If you have any questions or comments, please post in the [http://moodle.org/mod/forum/view.php?id=56 Testing and QA forum] in Using Moodle.
Feedback on all aspects of our QA testing process is welcome. If you have any questions or comments, please post in the [http://moodle.org/mod/forum/view.php?id=56 Testing and QA forum].


==See also==
==See also==
Line 113: Line 109:
* [https://tracker.moodle.org/browse/MDLQA-5740 Moodle 2.6 QA]
* [https://tracker.moodle.org/browse/MDLQA-5740 Moodle 2.6 QA]
* [https://tracker.moodle.org/browse/MDLQA-6693 Moodle 2.7 QA]
* [https://tracker.moodle.org/browse/MDLQA-6693 Moodle 2.7 QA]
* [https://tracker.moodle.org/browse/MDLQA-7170 Moodle 2.8 QA]
* [https://tracker.moodle.org/browse/MDLQA-7660 Moodle 2.9 QA]
* [https://tracker.moodle.org/browse/MDLQA-8205 Moodle 3.0 QA]
* [https://tracker.moodle.org/browse/MDLQA-9267 Moodle 3.1 QA]
* [https://tracker.moodle.org/browse/MDLQA-9827 Moodle 3.2 QA]
* [https://tracker.moodle.org/browse/MDLQA-10403 Moodle 3.3 QA]


[[Category:Quality Assurance]]
[[Category:Quality Assurance]]

Revision as of 16:58, 18 October 2017

Quality Assurance tests look at the functionality of Moodle from a user's point of view.

Real users systematically try each feature in Moodle and test that it works in the current version of the Moodle code. These tests are repeated in series of cycles, usually just before major releases.

QA testing latest: We're currently involved in Moodle 3.4 QA testing. More testers are always welcome (see below).

Getting involved

Would you like to help with QA testing? If so, please make sure you have created an account in the Moodle tracker, then contact Helen. Also, make sure you're subscribed to the Testing and QA forum in order to receive QA testing news updates.

All testers are members of the testers group in the Moodle Tracker, enabling them to pass or fail QA tests and add comments.

Running tests

  1. Select a test from MDLQA-10999. When viewing a test, if you wish, you can click the 'Assign to me' link on the right, so that nobody else chooses the same test to run. (If you then find you are unable to run the test, you can assign the issue back to 'Unassigned'.) Please assign issues to yourself which no one else is testing (Assignee = Unassigned). Only assign one issue at a time unless you plan to test a number of related issues within the next 24 hours. In other words, don't assign several issues to yourself then do nothing for several days. ;-)
    • The label test_server_required indicates issues that can't be tested on the QA testing site.
    • The label credentials_required indicates that credentials such as an OAuth 2 service client ID and secret are required.
  2. Using either the Moodle QA Testing Site or your own test site running the latest Moodle 3.4 beta (available from Git on the integration/master branch git://git.moodle.org/integration.git) with debugging set to developer, perform each of the steps listed in the test. Please test using both the Clean and Boost themes.
  3. Choose an appropriate workflow action:
    • Pass - Test runs perfectly. Add comment such as feedback about a new feature, browsers used for testing (if applicable), or simply "This test passes - yippee!"
    • Fail - Something doesn't work, or you obtain debugging messages. Add comment describing the step that doesn't work. If in doubt whether to pass a test, give it a fail and add a comment describing your doubts.
    • Obsolete - Test is no longer relevant in the current Moodle version. Add comment explaining why.

If you notice that the test description is out-of-date, add a comment suggesting how it can be updated and type @Helen Foster then select Helen from the menu below the input box. Alternatively, if you'd like to help with updating the test yourself, see below.

Moodle QA Testing Site

The Moodle QA Testing Site is updated daily at 13:00 UTC with the latest bug fixes to enable you to re-run QA tests.

To prevent the site being used for sending spam, no emails are sent from it. Thus, tests involving email cannot be run using the Moodle QA Testing Site. (If such tests are attempted, an email debug message is displayed. This is not a bug but rather expected behaviour.)

Manager, teacher and student accounts are provided. Please contact Helen if you would like admin access to the Moodle QA Testing Site for running certain tests.

Failed tests

So you ran a test and it failed? Congratulations on finding a bug! Please do the following.

  1. Click the Fail button at the top of the page.
  2. Add a comment to the QA test stating that there was a problem and that you will report it as a Moodle bug.
  3. Note the MDLQA number; it will be something like MDLQA-448.
  4. Try searching for whether the bug has been reported previously, and if not create a new issue for it (as described in Tracker introduction).
  5. In the new Moodle (MDL) issue select 'Link' from the 'More actions' dropdown menu.
    Linking to the QA issue in the tracker
  6. Link to the QA test by selecting 'blocks' as the link type, entering the MDLQA number that you noted earlier, and optionally adding a comment.
    Adding details for a link to the QA issue
  7. Give the issue the label 'mdlqa'.
  8. (Optional) Add yourself as a watcher to the MDL issue so that you receive email notification when the issue is fixed.
  9. When the MDL issue is fixed, hopefully within a day or two, the QA test can be reset and can then be run again.

Resetting tests

Note for integrators:

After integrating a fix,

  1. Reset the MDLQA test, adding a comment.
  2. Remove the 'mdlqa' label from the MDL issue.
  3. If the issue doesn't have testing instructions, pass it with message "Will be tested by MDLQA-XXXX".

The tester will then receive email notification that the bug is fixed and will hopefully decide to run the test again soon.

Suggested comment:

Thanks for running this QA test. The bug causing it to fail has now been fixed, so the test can be reset and run again.
Please note that if you are using http://qa.moodle.net for testing, you will need to wait for the site to be updated at 13:00 UTC in order to test the bug fix.

Updating tests

QA tests often become out-of-date due to new developments. If you would like to help with updating tests, you'll need to be a member of the test writers group in the Tracker. Please contact Helen about being added.

To update a QA test:

  1. Search for the master copy of the test - with affects version as 'Master copy' and MDLQA-1 as parent.
  2. Edit the test description.
  3. Optional: Add a comment describing your edit.

Writing new tests

Would you like to help with writing new QA tests? If so, as for updating tests, you'll need to be a member of the test writers group in the Tracker. Please contact Helen about being added.

To create a new QA test:

  1. Choose an issue from the closed qa_test_required-labelled issues
  2. Create a sub-task of MDLQA-1
  3. Select Master copy as affected version
  4. Select appropriate components including at least one of the following: User, Student, Teacher, Administrator
  5. Set assignee as unassigned
  6. Write the test (usually between 3 and 10 steps). It's a good idea to try doing the steps yourself as you write the test.
  7. Go to the MDL issue and create a ‘has a QA test’ link to the new QA test, adding a comment “Thanks for this improvement which is now covered by the QA test MDLQA....”, and if necessary remove the qa_test_required label. Make sure that all QA tests are linked back to a corresponding MDL issue.

New tests will be included in the next QA cycle.

Feedback

Feedback on all aspects of our QA testing process is welcome. If you have any questions or comments, please post in the Testing and QA forum.

See also

Comments on tests from previous QA cycles: