Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Quality assurance: Difference between revisions

From MoodleDocs
m (81 revisions)
 
(47 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{stub}}[[Category:Quality Assurance]]
{{Template:Work in progress}}[[Category:Quality Assurance]]


=What is QA=
==What is Quality assurance (QA)?==
Here few documents related to QA/Testing on Wikipedia: [http://en.wikipedia.org/wiki/Software_quality_assurance], [http://en.wikipedia.org/wiki/Software_testing]


=Introduction=
QA is generally one of the last stages of the software development. However QA can start even before the developers write any code. From the functional specification and use case, a QA test plans writer writes test cases, usually one test case per use case. Once the test cases are written, QA testers will test the software following these test cases. When they find a bug, they will report the issue in the tracker. Once all test cases have been completed, the QA manager can review the results to determine how ready the software is for release. A test case management system may be used for managing test cases and reviewing results.
The document described the Moodle QA. Any participation is welcome. Please use the tab "page comments" in order to participate. A good Moodle QA will be a great benefit for Moodle project but also for any company using Moodle.


=QA Objective=
==QA objective==
Main objective: is a major version ok for release?


A major version is ok when (TBD):
The goals of QA testing are to:
* critical bug number < 5%
* major bug number < 20%
* new functionalities has been implemented


A successful QA cycle assures that we have tested all major Moodle functionalities.
# Bring more clarity and thoroughness to the process of testing a new Moodle release.
# Ensure that all significant functionality has been tested before a new version is released.
# Give people a new way to easily contribute to the Moodle project.


=Need to be identified=
A major version is ready for release when there are:
QA Cost (in persons-hours) for:
* < 1% open blocker issues
* writting test cases (the first time)
* < 5% open critical issues
* preparing the QA cycle (update test case, get a new QA system ready for testing, update test data)
* < 20% open major issues
* create a test data package for people testing on local?
* running all test cases


When should a QA cycle be run?<br>
Also, all test cases should have been written/updated for new features and all test cases marked as 'Passed'.
What function specs/Use Cases have we got?<br>
Who can write test cases?<br>
Who can run test cases?<br>


=Draft/Ideas/Opinion/Requirements/Notes=
A successful QA cycle assures all major Moodle functionality has been tested.
* We need to make a clear statement of the Moodle docs in order to make a difference between: User Manuals, Function Specifications, Technical Specifications, Requirements, ... and many mixed document. (The moodle docs are a powerful resource of information, however at the current time Google is your best friend in order to find information.)
 
* There is a lack of detailed functional specifications and Use Cases => at this moment only developers and experimented user would be able to write functionality with missing use cases. (not so good)
==The QA cycle==
* use JIRA tracking system: we will use Jira as Test Case Managment system. Jira as not been developed for Test Case Management system but it can be used in order to manage a simple QA environment. The main reason to use Jira is to use a software that the community is familiar with.
 
* I need to study Quality Assurance into open source world: how does the other projects manage their QA department?
=== Setting a QA cycle ===
* Moodle doesn't have deadline. It is released when it will be ready. But avoid no-end QA phase.
 
* there are many experimented users that can test Moodle.
Test cases should be created in the tracker then updated for each new QA cycle. Ideally, poeple writing/updating the functional specs would write/update test cases at the same time. Alternatively a 'Cannot be run test' case status could be used when a tester cannot run a test case because the feature changed. This would alert the QA manager that the test case needs to be updated.
* some people/company using Moodle probably have already written test plans, test cases, maybe even use cases.
 
* need documentation on how to write a test cases and how to run test case. Need also document for QA administrator (setting a QA cycle, read the result, managing the QA cycle)
=== Running a QA cycle ===
* need to link Moodledocs QA documents with Moodle docs tracker documents, organize an easy to read Quality Engineering section into Moodledocs (include Code Testing as well).
 
Testers will choose a feature to test (with status 'Not run' or 'To retest').
 
* If the tester finds a blocker/critical/major issue, the test case is set to 'Failed'. The tester then writes a bug issue and links the test case to the bug issue. Once the bug is fixed, the bug fixer changes the test case status from 'Failed' to 'To retest'.
* If the bug is minor and the feature is working, the tester still writes a bug issue and still links the test case to the bug issue. However, the test case status may be set to 'Passed'.
 
== Validating a QA cycle ==
 
The test case management system needs to display:
* The number of failed, passed, not run, to retest test case issues for a particular version or component
* The number of critical/major issues for a specific version or component
 
The QA manager can review these results to determine how ready the software is for release.
 
==For further consideration==
 
Need to be identify the time cost of writing test cases, preparing a QA cycle (updating test cases, getting a new QA system ready for testing) and running all test cases.
 
Additional questions:
*When should a QA cycle be run?
*What functional specs/use cases do we have?
*Who can write test cases?
*Who can run test cases?
*Do we need test data?
 
Jerome's ideas/notes:
* it would be good to identify the different kind of Moodledocs: User Manuals, Function Specifications, Technical Specifications, Requirements, ... due to the wiki collaborative aspect, it is understandable that sometimes documents are a mix of User Manual, Functional and Technical specs. Maybe could we create these specific category: User Manuals, Function Specifications, Technical Specifications, Requirements
* There is a lack of detailed functional specifications and Use Cases => at this moment only developers and highly experimented users would be able to write test cases with missing use cases.
* Moodle doesn't have deadline. It is released when it will be ready. However we have to take care about no-end QA phase.
* there are many experimented users who can test Moodle.
* some people/companies using Moodle probably have already written test plans, test cases, maybe even use cases.
* need documentation on how to write a test case and how to run test cases. Need also document for QA manager (setting a QA cycle, read the result, managing the QA cycle)
* write an easy-to-read Quality Engineering section in Moodledocs (include Code Testing as well).
* we could create a testing program as Netbeans [http://qa.netbeans.org/processes/cat/65/index.html]
* we could create a testing program as Netbeans [http://qa.netbeans.org/processes/cat/65/index.html]
* create a Moodle package including test data for people testing in local
* We don't have a long build process, so smoke testing could seem not required. However it could be good to have it if we create a testing program.
==Jira as Test Case Management System==
The main reason to use Jira as Test Case Management System is that the Moodle community is familiar with. It will also be easy to link test case issues to bug issues.
* The search tool of Jira will provide the QA results.
* we will create a main issue containing thousand of sub-tasks. These subtasks will be the test cases. Then for every new QA cycle we will clone this main issue. Need to identify how to change subtask version quickly/easily.
* re-write the ''Jira as a Test Case Management Software documentation'' [https://docs.moodle.org/en/Jira_as_a_Test_Case_Management_Software]


=Jira=
==Other QA from popular open-source projects==
* The search tool of Jira will provide the QA results. Need to identify exatcly how.
* Netbeans wrote specification tests [http://wiki.netbeans.org/TestSpecifications]. They have a QA program [http://qa.netbeans.org/processes/cat/65/index.html]
* JIRA as a Test Case Management System: We'll create a main issue containing thousand of sub-tasks. These subtasks will be Test Case.
* Firefox use an integrated testcase management and QA tool called Litmus [https://litmus.mozilla.org/]. An example of a test case: [https://litmus.mozilla.org/show_test.cgi?id=5036]. They've got more than 7000 test cases.
* OpenOffice has a QA page [http://qa.openoffice.org/ooQAReloaded/ooQA-ManualTesting.html]. I found it a bit a hassle to navigate into these pages. Make some succinct documentation for Moodle QA tester in order to start quickly and easily.


=QA for other projects=
[[Category:Quality Assurance]]
Netbeans [http://wiki.netbeans.org/TestSpecifications]

Latest revision as of 08:37, 15 September 2011

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

What is Quality assurance (QA)?

QA is generally one of the last stages of the software development. However QA can start even before the developers write any code. From the functional specification and use case, a QA test plans writer writes test cases, usually one test case per use case. Once the test cases are written, QA testers will test the software following these test cases. When they find a bug, they will report the issue in the tracker. Once all test cases have been completed, the QA manager can review the results to determine how ready the software is for release. A test case management system may be used for managing test cases and reviewing results.

QA objective

The goals of QA testing are to:

  1. Bring more clarity and thoroughness to the process of testing a new Moodle release.
  2. Ensure that all significant functionality has been tested before a new version is released.
  3. Give people a new way to easily contribute to the Moodle project.

A major version is ready for release when there are:

  • < 1% open blocker issues
  • < 5% open critical issues
  • < 20% open major issues

Also, all test cases should have been written/updated for new features and all test cases marked as 'Passed'.

A successful QA cycle assures all major Moodle functionality has been tested.

The QA cycle

Setting a QA cycle

Test cases should be created in the tracker then updated for each new QA cycle. Ideally, poeple writing/updating the functional specs would write/update test cases at the same time. Alternatively a 'Cannot be run test' case status could be used when a tester cannot run a test case because the feature changed. This would alert the QA manager that the test case needs to be updated.

Running a QA cycle

Testers will choose a feature to test (with status 'Not run' or 'To retest').

  • If the tester finds a blocker/critical/major issue, the test case is set to 'Failed'. The tester then writes a bug issue and links the test case to the bug issue. Once the bug is fixed, the bug fixer changes the test case status from 'Failed' to 'To retest'.
  • If the bug is minor and the feature is working, the tester still writes a bug issue and still links the test case to the bug issue. However, the test case status may be set to 'Passed'.

Validating a QA cycle

The test case management system needs to display:

  • The number of failed, passed, not run, to retest test case issues for a particular version or component
  • The number of critical/major issues for a specific version or component

The QA manager can review these results to determine how ready the software is for release.

For further consideration

Need to be identify the time cost of writing test cases, preparing a QA cycle (updating test cases, getting a new QA system ready for testing) and running all test cases.

Additional questions:

  • When should a QA cycle be run?
  • What functional specs/use cases do we have?
  • Who can write test cases?
  • Who can run test cases?
  • Do we need test data?

Jerome's ideas/notes:

  • it would be good to identify the different kind of Moodledocs: User Manuals, Function Specifications, Technical Specifications, Requirements, ... due to the wiki collaborative aspect, it is understandable that sometimes documents are a mix of User Manual, Functional and Technical specs. Maybe could we create these specific category: User Manuals, Function Specifications, Technical Specifications, Requirements
  • There is a lack of detailed functional specifications and Use Cases => at this moment only developers and highly experimented users would be able to write test cases with missing use cases.
  • Moodle doesn't have deadline. It is released when it will be ready. However we have to take care about no-end QA phase.
  • there are many experimented users who can test Moodle.
  • some people/companies using Moodle probably have already written test plans, test cases, maybe even use cases.
  • need documentation on how to write a test case and how to run test cases. Need also document for QA manager (setting a QA cycle, read the result, managing the QA cycle)
  • write an easy-to-read Quality Engineering section in Moodledocs (include Code Testing as well).
  • we could create a testing program as Netbeans [1]
  • create a Moodle package including test data for people testing in local
  • We don't have a long build process, so smoke testing could seem not required. However it could be good to have it if we create a testing program.

Jira as Test Case Management System

The main reason to use Jira as Test Case Management System is that the Moodle community is familiar with. It will also be easy to link test case issues to bug issues.

  • The search tool of Jira will provide the QA results.
  • we will create a main issue containing thousand of sub-tasks. These subtasks will be the test cases. Then for every new QA cycle we will clone this main issue. Need to identify how to change subtask version quickly/easily.
  • re-write the Jira as a Test Case Management Software documentation [2]

Other QA from popular open-source projects

  • Netbeans wrote specification tests [3]. They have a QA program [4]
  • Firefox use an integrated testcase management and QA tool called Litmus [5]. An example of a test case: [6]. They've got more than 7000 test cases.
  • OpenOffice has a QA page [7]. I found it a bit a hassle to navigate into these pages. Make some succinct documentation for Moodle QA tester in order to start quickly and easily.