Quality assurance

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

What is Quality assurance (QA)?

QA is generally one of the last stages of the software development. However QA can start even before the developers write any code. From the functional specification and use case, a QA test plans writer writes test cases, usually one test case per use case. Once the test cases are written, QA testers will test the software following these test cases. When they find a bug, they will report the issue in the tracker. Once all test cases have been completed, the QA manager can review the results to determine how ready the software is for release. A test case management system may be used for managing test cases and reviewing results.

QA objective

The goals of QA testing are to:

1. Bring more clarity and thoroughness to the process of testing a new Moodle release.
2. Ensure that all significant functionality has been tested before a new version is released.
3. Give people a new way to easily contribute to the Moodle project.

A major version is ready for release when there are:

• < 1% open blocker issues
• < 5% open critical issues
• < 20% open major issues

Also, all test cases should have been written/updated for new features and all test cases marked as 'Passed'.

A successful QA cycle assures all major Moodle functionality has been tested.

The QA cycle

Setting a QA cycle

Test cases should be created in the tracker then updated for each new QA cycle. Ideally, poeple writing/updating the functional specs would write/update test cases at the same time. Alternatively a 'Cannot be run test' case status could be used when a tester cannot run a test case because the feature changed. This would alert the QA manager that the test case needs to be updated.

Running a QA cycle

Testers will choose a feature to test (with status 'Not run' or 'To retest').

• If the tester finds a blocker/critical/major issue, the test case is set to 'Failed'. The tester then writes a bug issue and links the test case to the bug issue. Once the bug is fixed, the bug fixer changes the test case status from 'Failed' to 'To retest'.
• If the bug is minor and the feature is working, the tester still writes a bug issue and still links the test case to the bug issue. However, the test case status may be set to 'Passed'.

Validating a QA cycle

The test case management system needs to display:

• The number of failed, passed, not run, to retest test case issues for a particular version or component
• The number of critical/major issues for a specific version or component

The QA manager can review these results to determine how ready the software is for release.

For further consideration

Need to be identify the time cost of writing test cases, preparing a QA cycle (updating test cases, getting a new QA system ready for testing) and running all test cases.

Additional questions:

• When should a QA cycle be run?
• What functional specs/use cases do we have?
• Who can write test cases?
• Who can run test cases?
• Do we need test data?

Jerome's ideas/notes:

• it would be good to identify the different kind of Moodledocs: User Manuals, Function Specifications, Technical Specifications, Requirements, ... due to the wiki collaborative aspect, it is understandable that sometimes documents are a mix of User Manual, Functional and Technical specs. Maybe could we create these specific category: User Manuals, Function Specifications, Technical Specifications, Requirements
• There is a lack of detailed functional specifications and Use Cases => at this moment only developers and highly experimented users would be able to write test cases with missing use cases.
• Moodle doesn't have deadline. It is released when it will be ready. However we have to take care about no-end QA phase.
• there are many experimented users who can test Moodle.
• some people/companies using Moodle probably have already written test plans, test cases, maybe even use cases.
• need documentation on how to write a test case and how to run test cases. Need also document for QA manager (setting a QA cycle, read the result, managing the QA cycle)
• write an easy-to-read Quality Engineering section in Moodledocs (include Code Testing as well).
• we could create a testing program as Netbeans [1]
• create a Moodle package including test data for people testing in local
• We don't have a long build process, so smoke testing could seem not required. However it could be good to have it if we create a testing program.

Jira as Test Case Management System

The main reason to use Jira as Test Case Management System is that the Moodle community is familiar with. It will also be easy to link test case issues to bug issues.

• The search tool of Jira will provide the QA results.
• we will create a main issue containing thousand of sub-tasks. These subtasks will be the test cases. Then for every new QA cycle we will clone this main issue. Need to identify how to change subtask version quickly/easily.
• re-write the Jira as a Test Case Management Software documentation [2]

Other QA from popular open-source projects

• Netbeans wrote specification tests [3]. They have a QA program [4]
• Firefox use an integrated testcase management and QA tool called Litmus [5]. An example of a test case: [6]. They've got more than 7000 test cases.
• OpenOffice has a QA page [7]. I found it a bit a hassle to navigate into these pages. Make some succinct documentation for Moodle QA tester in order to start quickly and easily.