Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Bug triage

From MoodleDocs
Revision as of 04:21, 14 June 2013 by Michael de Raadt (talk | contribs) (Adding note about addon-candidate label)
alt Triage image

Triage is medical term referring to the process of prioritising patients based on the severity of their condition so as to maximise benefit (help as many as possible) when resources are limited.

Bug triage is a process where tracker issues are screened and prioritised. Triage should help ensure we appropriately manage all reported issues - bugs as well as improvements and feature requests.


The triage process

Initial screening

The following aspects should be checked first to ensure the issue can be worked on.

Has the issue been reported previously?
If so, link to a duplicate issue and close the newly reported issue with no fix version set. Encourage the reporter to search before reporting. If a newer issue has a patch or more voters/watchers, consider closing the older issue. Checking for duplicates first will save you having to check the rest of the issue.

Here is a comment that can be used when closing a duplicate.

Thanks for your report.

It seems you're not the only one to have come across this bug, as it's been reported previously - see MDL-xyz. I'm going to close this issue now so we can focus on MDL-xyz. Please watch, vote ir comment on MDL-xyz if there is any additional information you can provide.

Does the problem seem rational?
If not, then the problem may simply be an misunderstanding on the part of the reporter. It might be a problem exclusive to the reporter's server setup. If you can replicate the problem quickly, do so. If you can't replicate the problem, ask the reporter to attempt to replicate the problem on demo.moodle.net. If the problem seems persistent but strange, consider asking a developer with experience working in the area to consider the problem and determine if it could be a real problem.
Can the problem be replicated?
There should be sufficient information in the issue so that someone else could replicate it. Ideally this would include:
  • replication steps (with expected results and actual results),
  • error messages (if applicable) and
  • screenshots (if applicable).
If more information is needed, do not complete the triage. Instead, watch the issue and refer it back to the reporter. If the reporter does not respond with more detail in a reasonable amount of time (say one week), close the issue as incomplete, but keep a watch on the issue.

Here is an example comment that could be used to elicit further information from a reporter.

Thanks for reporting this.

Could you please add more information to your report such as replication instructions, error messages and screenshots. If you are able to suggest a workaround, that would be immediately helpful to others experiencing this problem. If you can determine the cause of the problem or even determine a solution, that will be very valuable.

If the report clearly relates to a problem with the reporter's server or local set-up, you could use the following statement. (It's easy to slip into help-desk mode, but that is not the purpose of the Tracker.)

It sounds like this is a problem with your server set-up or the way you have configured Moodle. It doesn't sound like this is a problem in the Moodle code.

You might find help on the Installation forum.

Components

Are the correct components selected?
Listing components correctly is important as they are the primary variable used for searching for issues.
  • If the issue is listed with more than one component or if you change the components, the default assignee might not be set correctly.
  • If the issue covers more than one component determine the main component and ensure the assignee is set in relation to that component. Add component leads of other components as watchers if there is a significant overlap between components.

Security level

Is the issue a security issue?
If so, this should be determined as soon as possible.
Usually security issues should be bugs; improvements that improve security need not have a security level set (unless they reveal a weakness in Moodle, in which case they should be a bug).
Issues are often raised with a tentative "Could be..." security level. Ideally this should be set to "none", "minor" or "serious" as soon as possible to control the visibility of the issue.
Minor
  • problem could inappropriately reveal personal information to existing site users
  • problem could give existing users an inappropriate level of access
Serious
  • problem could inappropriately reveal personal information to non-users
  • problem could be exploited to attack a site, potentially altering the site or denying access

Priority

Is the priority level reasonable and representing the issues real priority in respect to the community?
Some reporters over-state an issues priority. Some reporters don't know they can set a priority.
  • Security issues should be blockers.
  • Backup/restore issues should have a high priority.
  • Other issues should follow the Tracker guide.

Note: Improvements/new features cannot be blockers and should usually be left with a Minor priority (though the dev team can raise the priority later)

Affects version

Does the Affects version make sense now and into the future?
This field should include one or more released versions of Moodle that are affected by the issue, with the following exceptions:
  • The issue is a bug in code that is present in the Master branch only, in which case the next major version should be used. (The next major version should not be used in conjunction with previous released versions, this won't make sense later.)
  • The issue is a new feature and is unrelated to any existing code in Moodle, in which case the 'Future dev' version should be used.
If the reported Affects version is unsupported, consider whether the issue could affect currently supported versions. In some cases it is simple close such an issue when a component has been replaced in a recent version, but otherwise this might require a quick replication test in a supported version. If the issue only affects unsupported versions, the issue should be closed as 'Won't fix'.

The following is a comment that can be used when closing an issue that affects only unsupported versions.

Thanks for reporting this.

I'm closing this issue because I believe it affects only unsupported versions of Moodle. This issue will remain here in case other users have the same problem.

If you haven't already done so, I encourage you to upgrade to a supported version.

Fix version

Is the Fix version set correctly?
When an issue is reported, it can be added to a backlog of work that a developer at Moodle HQ can handle. That does not preclude other developers, who are not working at Moodle HQ, from working on the issue.
The Fix version will later become the version of Moodle that the issue is fixed in, but that value should only be set at the time of integration.
In the Moodle project (MDL issues), the Fix version can be set to the BACKEND backlog, FRONTEND backlog or no backlog, depending on the primary component of the issue. Being on a backlog does not ensure work will be completed, equally not being on a backlog does not mean work will not be done, it's just a helpful way of allocating work to devs at Moodle HQ.
BACKEND FRONTEND Handled in a different way
  • Administration
  • Authentication
  • Automated functional testing
  • Backup
  • Caching
  • Cohorts
  • Database SQL/XMLDB
  • Enrolments
  • Events API
  • Files API
  • Gradebook
  • Grading methods
  • Groups
  • Installation
  • Libraries
  • Licensing
  • Logging
  • Messages
  • Performance
  • phpdoc
  • Portfolio API
  • Repositories
  • Roles / Access
  • RSS
  • Tags
  • Unicode
  • Unit tests
  • Web Services
  • Accessibility
  • Assignment
  • Assignment (2.2)
  • Blocks
  • Blog
  • Book
  • Calendar
  • Chat
  • Choice
  • Commenting
  • Course
  • Database activity module
  • Filepicker
  • Filters
  • Forms Library
  • Forum
  • Global search
  • Glossary
  • HTML Editor
  • HTML and CSS
  • IMS-CP resource type
  • Lesson
  • MNet
  • My home
  • Navigation
  • Ratings
  • Reports
  • Resource
  • Survey
  • Themes
  • Usability
  • Wiki (2.x)
  • Workshop
  • Activity completion
  • AJAX and JavaScript
  • Backup: IMS-CC
  • Badges
  • Conditional activities
  • Course completion
  • Database MS SQL
  • Documentation
  • External Tool (IMS-LTI)
  • Feedback
  • General
  • Language
  • Maths filters
  • Other
  • Plagiarism
  • Policy
  • Questions
  • Quiz
  • SCORM
  • Security Alert
  • Survey 2
  • Translation
  • Unknown

Assignee and watchers

Prior to May 2013 issues were automatically assigned to developers, even though they might never work on the issue. This created a false sense of progress and notionally denied others from becoming involved. Currently issues are not automatically assigned. Developers should only assign an issue to themselves when they have a definite intention to complete the issue.

Some components are automatically watched by users. If you think a user should be involved, add them to the list of watchers and comment on the issue.

Testing instructions

Has the reporter provided testing instructions?
The Testing instructions field is used during testing, after an issue has been resolved. If the reporter has provided appropriate testing instructions they can be left in place and updated by the developer who works on the issue.
Often issue reporters will include reproduction steps in this field. It is good that this information is provided by the reporter, but it should be moved to the description unless it is seen to constitute an appropriate set of testing instructions.

Labels

Are standard labels labels used?
If not, remove superfluous labels (if necessary, they can be copied to the description).
  • Issues are labelled triaged to indicate that all the above points have been covered.
  • Issues with proposed fixes are labelled patch so that they can be found easily and given attention. When this is the case, consider whether moving the issue to the 'Waiting for peer review' state in the workflow might be more appropriate. Also consider if the user should be promoted to a jira-developer so they can submit patches more easily in future.
A non-standard label that can be useful is 'addon-candidate', which can be used to signify that an improvement or new feature could be achieved as an add-on. This way, people looking for projects could search for issues with this label.

Gratitude and encouragement

After triaging many issues, it's easy to lose sight of the fact that the reporter has contributed their time and energy to report an issue for the benefit of the community.

It is easy to become defensive of Moodle if reports are seen as criticism (and sometimes reporters may use phrasing that suggests this), however the triagers response must always be one of sincere gratitude.

It is also important to encourage reporters to continue being involved with the issue after it is triaged. We must not give the sense that we are taking the issue ownership away from the reporter. Instead the reporter should be encouraged to discover the cause of the problem and present a solution; this is appropriate in an open-source project. It is amazing that such a challenge can lead to a sense of purpose for the reporters.

Here is a comment thanking a user for reporting a bug and encouraging them to continue working on the issue.

Thanks for reporting this.

I've put that on the backlog.

In the meantime feel free to help us work on this issue. If you are able to provide a patch or links to your Git repository branch, please add a patch label so we will spot it.

Here is a comment thanking a user for suggesting an improvement and encouraging them to continue working on the issue.

Thanks for reporting this.

Adding more detail to your suggestion will make it easier to work on.

If you can propose a code solution, that will help others who may have the same need and will increase the chance of this improvement/feature coming about sooner. If you are able to provide a patch or links to your Git repository branch, please add a patch label so we will spot it. Otherwise, how about posting in a forum on moodle.org and encouraging people to vote, comment and/or come up with a patch.

Creating a filter and gadget for triaging

If you are a component lead you are responsible for triaging issues that involve your component. A good way to monitor newly created issues is to create a filter in the Tracker and add a gadget on your dashboard to show the results of the filter.

Adding a filter

In Tracker...

  1. Select Issues > Search for Issues.
  2. Create a search for untriaged issues in your component, for example...
    component in (Assignment) AND resolution = Unresolved AND (labels is EMPTY OR labels not in (triaged)) ORDER BY created DESC
  3. Run the search query by pressing Enter or clicking the magnifying glass icon to the right of the search box.
  4. When the search query results are displayed, click the Save as button.
  5. Save the search query with an appropriate name, like "Untriaged Assignment issues".

Adding a filter in a gadget to your dashboard

In Tracker...

  1. Click Dashboards > Dashboard for XXX (where XXX is your name).
  2. Click + Add Gadget.
  3. Find the Filter Results gadget.
  4. Click Add it Now.
  5. Click Finished.
  6. In the Saved Filter input, search for and select your newly created filter.
  7. Click Save.

The most recent untriaged issues should appear in reverse-date order.

Triaging priorities and the Triaging Dashboard

The following are the priorities for ordering issues to be triaged that are reflected on the Triaging dashboard.

  1. Security issues - should always be reduced to 0
  2. High-priority issue - aim for blockers and critical issues to be reduced to 0
  3. Partner issues - aim for partner issues to be reduced to 0
  4. Patched issues - aim to triage as soon as possible
  5. Developer-reported issues (HQ and non-HQ) - should be quick to triage
  6. Recent community bugs - should be triaged last-in-first-out

See also