Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Process: Difference between revisions

From MoodleDocs
No edit summary
(Major rewrite (draft))
Line 1: Line 1:
We are planning a new development process for future core work on Moodle to integrate reviews at many levels throughout the process of code development.
This document summarises the development processes used in developing Moodle.


==Terms==
==Terms==
Line 9: Line 9:
* DEV backlog = everything to be looked at for upcoming major versions (eg 2.X)
* DEV backlog = everything to be looked at for upcoming major versions (eg 2.X)


==Overview of the process==


[[Image:Devprocess.jpg]]
==Development Processes==


We have a number of different processes that operate at different levels.


==Roles==


Let's look at the roles in the new system.  Note that one person may perform more than one of these roles, depending on their job description.  However, no-one should ever be reviewing their own code.
===Integration workflow in the tracker===


The Moodle tracker keeps track of the status of all bug fixes and new features.


===User role===
We use a workflow that ensures that new code receives multiple reviews by different people before it is included into the core Moodle code.


# Uses Moodle
[[Image:Workflow.jpg]]
# Finds issues (both "bugs" and "suggestions")
# Report issues to tracker


A number of roles make this work:


===[[Bug triage|Triage role]]===
====Users====


Initial staff: Eloy, Helen
Users report bugs and make feature requests directly in the tracker, by creating new issues with a summary and a description.


# Reads new issues coming into Moodle Tracker
====Developers====
# Performs housekeeping, fixes spelling, removes duplicates and spam, updates priorities/assignees/components/descriptions/statuses.
# For items in core:
## If the issue is a bug in the current stable version (eg 2.0), assign "Fix version" as STABLE backlog
## If the issue is related to backup and restore, assign "Fix version" to STABLE backlog with high priority
## If the issue is a bug in the current stable version requiring database changes, assign "Fix version" to DEV backlog
## If the issue is a feature request, assign "Fix version" to DEV backlog


===Product Owner role===
Developers work on the issues in the tracker to specify solutions and write code that implements these solutions.  They will often ask other developers to "peer review" their code in the early stages to avoid problems later on.


Initial staff: Martin
While many of the developers work for Moodle.com, a large number are part of the global development community around Moodle.


# Monitors DEVBACKLOG, updates priorities to produce a queue
====Component leads====
# Manages community discussions/feedback
# Maintains overall roadmap and goals based on issues + community
# Develops detailed specifications


===Scrum Master role===
Component leads are developers with some responsibility for particular components (plugins or modules) in Moodle.  They have the authority to decide that a particular fix is suitable and complete enough to be considered for integration in Moodle core, by adding their git branches to the integration queue.  This can happen at any time.


Initial staff: ???
====Integrators====


# Coaches and helps the team to finish each sprints
On Monday of each week, the integration team (a small team of senior developers employed by Moodle HQ) conducts a code-level review of all issues in the integration queue.  This is often called the "pull" process.  If the fix is judged appropriate they will integrate the code into our git integration repository for further testing and it gets added to the testing queue.
# Holds sprint restrospective meetings
# Keeps documentation updated


===Moodle HQ Stable Maintenance team===
If they find problems they reject the issue and send it back to the developer for further work.


This SCRUM team generally works on the bugs important for improving the current stable release (STABLEBACKLOG).  Just before any major release they also work on backup/restore for that release.  This ensures a good familiarity with the code-base of the next release.
====Testers====


Initial staff: Andrew, Aparup, Dongsheng, Jerome, Rossi, Sam
On Tuesday each week, testers look at all the issues in the testing queue, trying each fix and feature to make sure that it does actually fix the problem it was supposed to, and that there are no regressions.


# Selects one month's work (STABLEBACKLOGSPRINT) from the highest priority in STABLEBACKLOG
If they find problems they reject the issue and integrators may remove it from the integration repository and push it back to the developer for further work.
# Assigns X to them so they are visible/manipulable in Greenhopper as a Sprint, eg "Jan 2011", and also get assigned "Fix version" of the next minor release (eg 2.0.1).
# Meets with Product Owner to resolve any questions, agree on details
# Assigns work among team members
# Meets daily with Scrum Master to keep on track, resolve issues
# Develops solutions on local git installs
# Pushes solutions as issue-named branches (eg wip-MDL-12345) to a public repository (moodle.com ?)
# Publish suggested code via links in comments in Jira
# Conduct basic peer-reviews of code from other team members (direct communication, Jira, Crucible?)
# Rebase WIP branches and remove "wip-" from the name so it's called MDL-12345
# Create PULL requests for each issue or group of related issues for Integration review


====Committers====


===Moodle HQ New Developments team===
On Wednesday each week, committers merge all the issues that passed testing into the git production repository, and it becomes available for use on production systems via git, cvs, and download packages.


Initial staff: David,


# Selects one month's work (DEVBACKLOGSPRINT) from the highest priority in DEVBACKLOG
===Stable maintenance cycles===
# Assigns X to them so they are visible/manipulable in Greenhopper as a Sprint, eg "Jan 2011", and also get assigned "Fix version" of the next major release (eg 2.1).
# Meets with Product Owner to resolve any questions, agree on details
# Extend/deepen specifications as necessary
# Assigns work among team members
# Develops solutions on local git installs
# Pushes work-in-progress solutions as issue-named branches (eg wip-MDL-12345) to a public repository (moodle.com, github.com ?)
# Publish suggested code via links in comments in Jira
# Conduct basic peer-reviews of code from other team members (direct communication, Jira, Crucible?)
# Rebase WIP branches and remove "wip-" from the name so it's called MDL-12345
# Create PULL requests for each issue or group of related issues for Integration review


Moodle releases regular updates of the stable version of the software to fix bugs and other issues.  Releases like 2.0.1, 2.0.2, 2.0.3 etc only include fixes based on the latest major release (2.0) and never any significant new features or database changes.


===External Developers===
At Moodle HQ there is a team of developers using a SCRUM-based process to work on these releases.


# Develop whatever they like, however they like!
====Triage team====
# Publish to public git repositories (github, gitorious, whatever)
# Create/find issues in Jira
# Suggest code via links in comments in Jira


Triagers evaluate new issues, making sure that they are labelled correctly.  One of the most important jobs they do is to identify issues that should be fixed in the stable branch.  These are added to the STABLE backlog with a priority ranging from "Trivial" up to "Blocker".


Other larger issues go into the DEV backlog for consideration as part of the next major release.


===Component maintainer role===
====Scrum team====


# Reviews and selects final solution for each issue in their control
Every three weeks, the Stable Scrum team takes a number of the most urgent issues from the STABLE backlog  (approx 30, but it varies) and starts working on solving them.  The team meets daily to discuss solutions and progress, as well as conducting peer reviews of code.  The team has a "Scrum master" to help everyone stay organised and to "unblock" any barriers to progress.  Everything is documented publically in the tracker.
# Possibly rebases, cleans up
# Create PULL requests for each issue or group of related issues for Integration review


Whenever a solution for an issue is finished, it is submitted into to the standard integration workflow process described above.




===Integration reviewer role===
===Core release cycles===


Initial staff: Petr, Eloy, Sam
Since Moodle 2.0, we have a policy of release major versions (eg 2.1, 2.2) every six months: in June and December.


Only these people have write access to the integration repository '''git.moodle.org/integration.git'''.
Each release can be different, but generally the cycles work like this:


# At set times per week, the Integration Review team goes through the current queue of “pull requests” (reviewers must not review/commit their own code)
====Roadmap is defined====
# Each new “pull request” is either accepted or rejected with comments.
# If accepted then the “pull request” is merged into the integration repository to Moodle branches and the “pull request” issue changes status to “QA review”
# If rejected then the reviewer marks the pull request as “Rejected” and the component maintainer is told why in the Jira issue


The product owner (Martin Dougiamas) defines the likely roadmap based on community wishes, third-party developments and important issues within the existing code.  The DEV backlog is also a factor.


===[[Testing|QA testing role]]===
Sometimes these might be based on earlier features, sometimes they may be evauluating something developed by a third party, sometimes it might be something completely new.


Initial staff: Helen, volunteers


# Go through the “QA review” issues in the “pull request” queue.
====Dev team====
# Test the new features/fixes on automatically-built Moodle sites (or their own)
# Sign off on the fix and change the status of the “pull request” to “QA success”.
# If the test fails, comment and change the status of the “pull request” to “QA fail”


===Production maintainer role===
The Dev team (employed at Moodle HQ) work on major new features one or two at a time.  The aim is always to try and finish things completely before moving on to the next features, as this helps to avoid delays in delivering a release on time.


Initial staff: Petr, Eloy
All this code goes through the standard weekly integration workflow described above.


Only these people have write access to the official production repository '''git.moodle.org/moodle.git''', and all commits come direct from the integration repository.
====QA Team====


# At set times per week, go through the list of “QA fail” and “QA review” pull requests and make sure that code is not in the integration server, or that the Integration reviewer fixes them.  All pull requests should be closed with “QA success” or “Rejected”.
Finally, in the last weeks of the release, our QA team performs a full functional test of Moodle features (manually, one at a time) to make sure there are no regressions caused by new features.  This list of features is extended on every release to include new developments.
# Push the integrated Moodle branches to the production server.
# Close the original Jira issues, say thanks and buy drinks.


==Commit messages==


Every commit should have a commit message to help other developers follow and review what is being changed. They are very helpful when tyring to understand a piece of code. The command <tt>git blame</tt> will tell you who last changed each line of code, and when they did it. Then you can look at the commit comments to tell you why that change was made.
===Feature development===


A commit message should contain, in order:
Major new features in Moodle usually should go through the following process.
* A couple of words indicating which part of moodle this change affects. For example 'forum backup', 'quiz attempting' or 'weblib'.
* A tracker issue id like MDL-12345. (If necessary, create an issue before committing, the only exception is for very minor typos.)
* A brief description of the problem you are fixing. (If you are feeling lazy, you may be able to get away with copying and pasting the issue summary from the tracker.)
* If it is not immediately obvious, a brief summary of why this change fixes the problem. If a longer description is necessary,  you may also want to add more information to the tracker issue, or Moodle Docs, but the code changes + commit message should provide enough clues to how the fix works on their own.


Here is an example of a commit message from a simple change:
====Specification====


user selection: MDL-17072 Polishing the role assign page. Do processing before print_header in line with best practice.
The developer should create a detailed spec (here in Moodle Docs) outlining their goals for the development and their design for meeting those goals.  The more detail the better.
 
Actually, the ideal commit message would be a bit shorter than that. It is good if the first three items (area of Moodle, tracker id and brief description) will fit in the subject line of an email message, so the first line should be shorter than about 70 characters if possible.
 
Here is an example from a bigger commit (would it have been better if this could have been done as several smaller commits?):
role overrides: MDL-17070 Improve override roles page to match the recent usability improvements on the assign page.
Including:
MDL-11529 When assigning/overriding roles, the dropdown for switching to
another role should have a number in brackets
MDL-16549 Should not be able to edit the permission associated with
moodle/site:doanything on any role.
 
Here is an example of a commit (from a while ago, which is why the issue number is in the wrong place) with some explanation of the change:
 
accesslib: get_user_by_capability() - Handle complex rolecap resolution
With this patch, get_user_by_capability() can handle the cases where
users have multiple role assignments to the same course, and PREVENTs
and PROHIBITs affect the rolecaps of this course.
   
Without stored procedures we cannot resolve this entirely on the
server side - so in the complex cases we do as much as we can on SQL,
and post-process the data on the PHP side, including SQL-style
pagination.
MDL-12452
 
==Workflow in tracker==
 
Proposed new workflow for MDL issues.
 
[[Image:Workflow.jpg]]


==Scheduling==
You should also create an issue in the tracker (linking to your docs) to keep track of the project status.


Here are some suggested scheduling times for different parts of the process.
====Community consultation====


===Integration review===
Get the community involved in looking at the spec to see if it meets their needs and to get further feedback.  The forums on moodle.org are good for this, but you could also blog/tweet about it etc.


* Monday - Review team divides PULL backlog and reviews all code -> Accept/Reject, commit to integration repo
You might also want to talk with HQ core developers to make sure the ideas make sense, and possibly get some review on database design and so on.
* Tuesday - QA testing on all code accepted on integration repository
* Wednesday - code pushed to main repository


===Stable scrum sprints===
====Develop the code using git====


* Day 1, Sprint planning meeting, choosing and assigning high-priority tasks from the backlog to the sprint
Develop your code on an open git repository, like github.com.  That enables people to see your code and to help you as it develops.  Testers and early adopters also have the opportunity to try it early in the process and give you more valuable feedback.
* Day 2-19, daily stand-up meeting, developing solutions, add requests to the PULL backlog
* Day 20, Sprint Review meeting, Sprint Retrospective meeting


Here are the tasks for a typical scrum, in progress, as represented in the tracker:
====Submit the code for integration====


[[Image:Scrumtasks.png]]
This is much the same as for any Moodle code.  See the information about the integration workflow above.




Revision as of 16:40, 26 April 2011

This document summarises the development processes used in developing Moodle.

Terms

Here are some terms we'll use in this document:

  • STABLE = the next stable minor version of Moodle (eg 2.0.1, 2.0.2 ... etc) Note that database changes and new features are not allowed in STABLE
  • STABLE backlog = the queue of bugs to be fixed in STABLE (eg 2.0.x)
  • DEV = the next major version of Moodle (eg 2.1, 2.2, 2.3 ... etc)
  • DEV backlog = everything to be looked at for upcoming major versions (eg 2.X)


Development Processes

We have a number of different processes that operate at different levels.


Integration workflow in the tracker

The Moodle tracker keeps track of the status of all bug fixes and new features.

We use a workflow that ensures that new code receives multiple reviews by different people before it is included into the core Moodle code.

Workflow.jpg

A number of roles make this work:

Users

Users report bugs and make feature requests directly in the tracker, by creating new issues with a summary and a description.

Developers

Developers work on the issues in the tracker to specify solutions and write code that implements these solutions. They will often ask other developers to "peer review" their code in the early stages to avoid problems later on.

While many of the developers work for Moodle.com, a large number are part of the global development community around Moodle.

Component leads

Component leads are developers with some responsibility for particular components (plugins or modules) in Moodle. They have the authority to decide that a particular fix is suitable and complete enough to be considered for integration in Moodle core, by adding their git branches to the integration queue. This can happen at any time.

Integrators

On Monday of each week, the integration team (a small team of senior developers employed by Moodle HQ) conducts a code-level review of all issues in the integration queue. This is often called the "pull" process. If the fix is judged appropriate they will integrate the code into our git integration repository for further testing and it gets added to the testing queue.

If they find problems they reject the issue and send it back to the developer for further work.

Testers

On Tuesday each week, testers look at all the issues in the testing queue, trying each fix and feature to make sure that it does actually fix the problem it was supposed to, and that there are no regressions.

If they find problems they reject the issue and integrators may remove it from the integration repository and push it back to the developer for further work.

Committers

On Wednesday each week, committers merge all the issues that passed testing into the git production repository, and it becomes available for use on production systems via git, cvs, and download packages.


Stable maintenance cycles

Moodle releases regular updates of the stable version of the software to fix bugs and other issues. Releases like 2.0.1, 2.0.2, 2.0.3 etc only include fixes based on the latest major release (2.0) and never any significant new features or database changes.

At Moodle HQ there is a team of developers using a SCRUM-based process to work on these releases.

Triage team

Triagers evaluate new issues, making sure that they are labelled correctly. One of the most important jobs they do is to identify issues that should be fixed in the stable branch. These are added to the STABLE backlog with a priority ranging from "Trivial" up to "Blocker".

Other larger issues go into the DEV backlog for consideration as part of the next major release.

Scrum team

Every three weeks, the Stable Scrum team takes a number of the most urgent issues from the STABLE backlog (approx 30, but it varies) and starts working on solving them. The team meets daily to discuss solutions and progress, as well as conducting peer reviews of code. The team has a "Scrum master" to help everyone stay organised and to "unblock" any barriers to progress. Everything is documented publically in the tracker.

Whenever a solution for an issue is finished, it is submitted into to the standard integration workflow process described above.


Core release cycles

Since Moodle 2.0, we have a policy of release major versions (eg 2.1, 2.2) every six months: in June and December.

Each release can be different, but generally the cycles work like this:

Roadmap is defined

The product owner (Martin Dougiamas) defines the likely roadmap based on community wishes, third-party developments and important issues within the existing code. The DEV backlog is also a factor.

Sometimes these might be based on earlier features, sometimes they may be evauluating something developed by a third party, sometimes it might be something completely new.


Dev team

The Dev team (employed at Moodle HQ) work on major new features one or two at a time. The aim is always to try and finish things completely before moving on to the next features, as this helps to avoid delays in delivering a release on time.

All this code goes through the standard weekly integration workflow described above.

QA Team

Finally, in the last weeks of the release, our QA team performs a full functional test of Moodle features (manually, one at a time) to make sure there are no regressions caused by new features. This list of features is extended on every release to include new developments.


Feature development

Major new features in Moodle usually should go through the following process.

Specification

The developer should create a detailed spec (here in Moodle Docs) outlining their goals for the development and their design for meeting those goals. The more detail the better.

You should also create an issue in the tracker (linking to your docs) to keep track of the project status.

Community consultation

Get the community involved in looking at the spec to see if it meets their needs and to get further feedback. The forums on moodle.org are good for this, but you could also blog/tweet about it etc.

You might also want to talk with HQ core developers to make sure the ideas make sense, and possibly get some review on database design and so on.

Develop the code using git

Develop your code on an open git repository, like github.com. That enables people to see your code and to help you as it develops. Testers and early adopters also have the opportunity to try it early in the process and give you more valuable feedback.

Submit the code for integration

This is much the same as for any Moodle code. See the information about the integration workflow above.

Retrieved from "https://docs.moodle.org/dev/index.php?title=Process&oldid=18548"
Powered by MediaWiki