User:Sam Hemelryk/My Moodle Git workflow

Jump to: navigation, search

This document is a sort of tutorial/inspection of my personal development workflow when working on Moodle with Git.

Creating the repositories

Ok so the very first step is to create several Moodle git repositories, one for each major branch of Moodle being worked on.

At the time of writing the major branches are MOODLE_19_STABLE, MOODLE_20_STABLE and master.

The reason for this is that that way we have a separate site for each Moodle branch that we can test on easily. Yes you will have three sites but you won't have to worry about swtiching your site when you change branches.

Step 1: Create a directory to create the repositories within

Personally I use Ubuntu (Linux) and I chose to create my repositories within a folder I created in /var/www. If you are using windows I would instead suggest creating C:/www/

cd /var/www
mkdir repositories
cd repositories

Step 2: Create a master repository

This is the easiest of the branches to create a repository for.

Within your repositories directory create a new directory called master and move into it:

mkdir master
cd master

Next we want to clone the Moodle source from the offical Moodle git repository as follows:

git clone git://git.moodle.org/moodle.git moodle

This will create a new directory called moodle within the master directory we created just before.

That new moodle directory should contain the Moodle source code as a git repository.

Now that we've got our main git repository we need to add our github account as a remote so that we can push to it later:

cd moodle
git remote add github git@github.com:yourname/moodle.git

This adds your moodle repository on github as a remote called github that you can push to later.

Now, could be a good time to identify yourself as the code author, which is responsible for the code changes on this local git repository.

git config --global user.name 'John Doe'
git config --global user.email johndoe@example.com

These setting will sets your name and email for commit messages.

Finally before we move onto the next repository we need to create a data directory because we know we will need that:

cd ..
mkdir data
sudo chmod -R 777 data

Step 3: Create a MOODLE_20_STABLE repository

The next repository we create will be for the MOODLE_20_STABLE branch. To start with get back to the repositories directory that we created earlier and create a directory called MOODLE_20_STABLE within that.

cd /var/www/repositories
mkdir MOODLE_20_STABLE
cd MOODLE_20_STABLE

Now create a new clone of the main Moodle repository like we did for the master branch above.

git clone git://git.moodle.org/moodle.git moodle

Once we have done that move into the moodle directory.

By default the master branch is the branch that is created, however we don't want that for this repository. Instead we want the MOODLE_20_STABLE branch.

To do this we checkout local branch based upon the official MOODLE_20_STABLE branch as follows:

cd moodle
git checkout -b MOODLE_20_STABLE origin/MOODLE_20_STABLE

Once we have created that branch we can delete the local master branch so that we don't accidentally ever work on it.

git branch -D master

Then like we did for master we add our github repository as a remote:

git remote add github git@github.com:yourname/moodle.git

And finally create a data directory we can use when we have install the site:

cd ..
mkdir data
sudo chmod -R 777 data

Step 4: Create a MOODLE_19_STABLE repository

This step is identical to the above step except we are using the MOODLE_19_STABLE branch instead of the MOODLE_20_STABLE branch. The commands for this are as follows:

cd /var/www/repositories
mkdir MOODLE_19_STABLE
cd MOODLE_19_STABLE
git clone git://git.moodle.org/moodle.git moodle
cd moodle
git checkout -b MOODLE_19_STABLE origin/MOODLE_19_STABLE
git branch -D master
git remote add github git@github.com:yourname/moodle.git
cd ..
mkdir data
sudo chmod -R 777 data

Step 5: Install the sites

Now I can't tell you how to do this on your own machine however what I would recommend is that you create a link from your web root to the moodle directory of each branch.

If you are using linux and your webroot is /var/www/localhost you would do it as follows:

cd /var/www/localhost/
ln -s /var/www/repositories/master/moodle master
ln -s /var/www/repositories/MOODLE_20_STABLE/moodle MOODLE_20_STABLE
ln -s /var/www/repositories/MOODLE_19_STABLE/moodle MOODLE_19_STABLE

With that done you should be able to browse to each repository with the following URL's

http://localhost/master/
http://localhost/MOODLE_20_STABLE/
http://localhost/MOODLE_19_STABLE/

Once that is done you need to browse to each site and complete the installation.

Under this method your will need to ensure you set the configuration option sessioncookiepath to the URI of your site. You can either do this in the admin interfaces or add the following to your config.php file for each site.

// For the master site
$CFG->sessioncookiepath = '/master/';
// For the MOODLE_20_STABLE site
$CFG->sessioncookiepath = '/MOODLE_20_STABLE/';
// For the MOODLE_19_STABLE site
$CFG->sessioncookiepath = '/MOODLE_19_STABLE/';

Failure to do this will mean that when you log in at one site and then visit the next you will have to log in again and your session on the first site will no longer work.

Now that you have got 3 repositories, one for each main branch, and set up a site for each you are ready to start development.

Before you start work each day

Before you start work each day you should update each of the repositories you have created to ensure that you are always working with up to date versions of Moodle.

You should also take this opportunity to update the repositories on your github account to ensure they are always up to date.

To start with lets update the master repository:

cd /var/www/repositories/master/moodle
git fetch --all --prune

This moves to the master repository and then tells git for fetch all of the changes from your remote repositories and clean up any old branches it is aware of.

Once that operation has completed you should update your master branch

Before you do that it pays to make sure you don't have any uncommitted changes, if you do DON'T TRY IT. There would be a good chance you will encounter problems. Instead finish what you are working on, commit your changes and then update your master branch.

trying to git checkout master before committing uncommitted changes will, probably, result in:

error: Your local changes to the following files would be overwritten by checkout:
file1.php
file2.php
...                   
Please, commit your changes or stash them before you can switch branches.                                                                                                                                  
Aborting  

so please git commit you modified files and then continue to:

git checkout master
git pull

Next we will update the github repositories by running the following command:

git push github refs/remotes/origin/master:master

This tells git to push the master branch from origin (the offical Moodle repository) to the master branch on your github account.

Now that you've updated the master repository and your github account, its time to update the MOODLE_20_STABLE and MOODLE_19_STABLE branches.

cd /var/www/repositories/MOODLE_20_STABLE/moodle
git fetch --all --prune
git checkout MOODLE_20_STABLE
git pull
git push github refs/remotes/origin/MOODLE_20_STABLE:MOODLE_20_STABLE

And

cd /var/www/repositories/MOODLE_19_STABLE/moodle
git fetch --all --prune
git checkout MOODLE_19_STABLE
git pull
git push github refs/remotes/origin/MOODLE_19_STABLE:MOODLE_19_STABLE

Working on a Moodle issue

This part of the document looks at how I go about working on an MDL issue from the Moodle tracker.

At Moodle HQ we use the scrum methodology we sees us choose several issues to include in a scrum which we then work on for the period of the scrum. As a community member its more likely you will just be choosing issues that you are passionate about.

Either way once you have found an issue to work on you are ready to start.

For the purpose of this part of the document lets say I am going to work on bug MDL-12345.

Fixing a bug on the master repository - MDL-12345

The first step is to create a local branch to make changes on, to do this we will start our work on the master repository.

When fixing an issue I find it easiest to fix the issue on the master branch first and then move my changes to the other branches.

The VERY first thing you should do when you start working on a Moodle issue is make sure you have assigned it to yourself and then click the start progress button.

This way other users know that you are activly working on the bug and no one from HQ will steal it from you.

Step 1: Creating a local branch

So first get to the master repository:

cd /var/www/repositories/master/moodle

Once there we create a new branch as follows:

git checkout -b wip-MDL-12345-master origin/master

This command checks out a new branch called wip-MDL-12345-master that is based upon origin/master.

origin/master is of course the master branch on the official Moodle repository (git.moodle.org/moodle.git).

The name of the branch is also very important. It is essentially telling us three things.

wip 
This stands for work in progress, it helps people see that you are currently working on this branch and that they shouldn't base there work on it because it will likely change.
MDL-12345 
This is of course the Moodle bug number. It helps people quickly identify what you are working on.
master 
This is the branch we are making changes on. It helps people quickly understand where the changes are being made.

You can optionally add more to the end of the branch name such as a key word that identifies the area or a date at which you started work e.g.

That is up to you, personally I'm not a fan of it, however the rest is all required to help integrate your work.

Either way now you have a branch, mine is called wip-MDL-12345-master.

Step 2: Make changes

Now that you have created you branch you are ready to make changes.

I'll let you make whatever changes you are need, once you have made changes move onto the next step.

Before you do move on however make sure that your code meets the high quality of code Moodle requires.

You can find information about that on the moodle docs here: https://docs.moodle.org/en/Development:Coding_style

The following are common mistakes:

  • Incorrect white space
    • Tabs instead of spaces
    • Extra spaces at the end of lines
    • Multiple new lines
    • Forgetting to put spaces between function arguments
  • Incorrect commit messages (read on to find out about writing a good commit message)
  • Incorrect variable or function names

Making these sort of mistakes can lead to your code being rejected despite it working correctly.

I would strongly suggest while learning Moodle development that you use Tim Hunt's fantastic Code Checker plugin.
You'll find information on how to use it within its README file and it will pick up many of the things that may lead to your work being rejected.

Or course it will be reviewed in regards to security, usability and the other important factors of code development.

Step 3: Commit your changes

Now that you have made your changes you are getting ready to commit them.

The first thing to do is check your changes. To do this run the following command:

git status

You should see all of the files you have changes highlighted in red.

Next you need to stage all of the files you want to commit.

To do this you add the files to the stage using the following command.

git add lib/modifiedfile.php
git add lib/newfile.php
git add theme/base/style/core.css

Why not just commit all changes. Many of the git guides you read will tell you above git commit -a which commits all of the changes you have made.

We've hardly being using git here at Moodle and already I've seen several people accidentally commit changes that they didn't mean to.

Staging files like this ENSURES you only commit things you intend to.

Now that you have staged your changes/new files you are ready to commit. The following command commits your changes:

git commit -m "MDL-12345 enrol - Fixed up a couple of enrolment bugs"

This command commits your changes with the commit message "MDL-12345 enrol - Fixed up a couple of enrolment bugs". It's very important to Moodle that you format your commit messages like this. They consist of three parts:

  1. The MDL bug number in this case MDL-12345
  2. The area you made changes to in this case enrol
  3. A short description of what you did, in this case I fixed a couple of enrolment bugs.

When you commit your message you will see something like the following:

[master 19a484e] changes
 1 files changed, 16 insertions(+), 18 deletions(-)

There is one thing here you need to note down for later, that is the number that appears within the square brackets after the branch name, in my case it is 19a484e. This is the commit id of the commit I just made.

Step 4: Pushing your changes to your github account

Now that you have made changes to your local branch you should push it to your github account so that it can be reviewed and hopefully integrated into Moodle.

Doing this is very simple just run the following command:

git push github wip-MDL-12345-master

Here we are telling git to push the branch wip-MDL-12345-master to the remote called github which we added when setting up our git repositories. And just like that you've made a branch that contains your work and pushed it to github ready to get it reviewed.

Moving your changes to another branch

Now that you have made changes to the master branch you need to decide whether it is appropriate make those changes on any of the other branches. Most likely if you are fixing a bug you will need to make the changes on the MOODLE_20_STABLE branch as well, and perhaps the MOODLE_19_STABLE branch if it is a security bug.

Lets assume for MDL-12345 that I fixed earlier that it should be ported to MOODLE_20_STABLE as well.

Step 1: Creating a local MOODLE_20_STABLE branch

To start with I need to move to the MOODLE_20_STABLE repository as follows:

cd /var/www/repositories/MOODLE_20_STABLE/moodle

Once there I need to create a new branch based upon the MOODLE_20_STABLE branch that is going to contain my changes.

git checkout -b wip-MDL-12345-MOODLE_20_STABLE origin/MOODLE_20_STABLE

This is very similar to the branch we created for the changes to the master branch except that I have subsituted MOODLE_20_STABLE in place of master. This is because the new branch we are creating is based upon the MOODLE_20_STABLE branch on the offical Moodle git repository and contains changes for that branch.

Step 2: Cherry-picking my changes

Once we have a branch for our changes I am ready to make them again. There is however with git an easy way to do this, cherry-picking.

Cherry-picking is the process of copying a single commit from one branch to another. In this case I want to cherry pick the commit I made earlier onto the branch I've just created.

However before I can do this I need to fetch the changes I made from the github repository. I need to do this because I made that changes of a totally separate local repository remember.

So to update my repository with the latest changes I run the following command.

git fetch --all --prune

This command tells git to look at each remote and get any new changes (it also removes any branches that have been deleted). Once it has completed you will be ready to cherry pick the commit. To do so you use the following command but replace my commit is with yours.

git cherry-pick 19a484e

Providing there are no conflicts your branch will now contain a copy of the changes you made to the master branch. If there are conficts that you will need to resolve them and commit your changes however you can search for another tutorial about how to do that,

Now I know at this point that there will be some of you who are saying 'Whoops I forgot what my commit id was!'. If this is you don't worry, its pretty easy to find it providing you used a proper commit message like I described above. Simply run the following command:

git log --oneline origin/master..github/wip-MDL-12345-master

This command shows you all of the commits that are in the wip-MDL-12345-master at your github account but are not in the master branch at the offical Moodle repository. You can then cherry-pick the commits shown there (the commit ID's should be highlighted in yellow).

Once you have cherry-picked your commits its time to check that everything worked, run the command:

git status

You should see something like the following:

# On branch wip-MDL-12345-MOODLE_20_STABLE
# Your branch is ahead of 'origin/MOODLE_20_STABLE' by 1 commit.
#
nothing to commit (working directory clean)

This is very handy as it tells you that your branch is ahead of the offical MOODLE_20_STABLE branch by one commit. You can also check that your commit contains the changes you think by running the following command:

git diff origin/MOODLE_20_STABLE

This command shows you all of the changes you have made as a diff.

Now that you know that your commit is there you and that it is correct it is time to push it to your github account.

git push github wip-MDL-12345-MOODLE_20_STABLE

And thats it.

You github account should now have two branches on it:

wip-MDL-12345-master 
This branch is the changes you have made for the master branch
wip-MDL-12345-MOODLE_20_STABLE 
This branch is those same changes but for the MOODLE_20_STABLE branch

Now you are ready to create PULL requests so that your work gets reviewed and hopefully integrated.

Creating PULL requests

Now that you've got a branch with changes for your MDL issue its time to create some PULL requests to get it integrated.

Creating a PULL request for the master branch

So you have successfully fixed the bug MDL-12345, you have created two branches one for master and one for MOODLE_20_STABLE. It is time to create a PULL request for each of these branches.

First lets create a PULL request for the master branch:

  1. Browse to tracker.moodle.org/browse/MDL-12345
  2. When the page loads log in if you haven't already.
  3. Click on the button in the top right labelled `Create issue`
  4. Change Project to `Pull Requests`
  5. Change Issue type to `Pull Request`
  6. Click create

This takes you to a screen where you can enter the details for the PULL request.

You should fill it out in the following way:

Summary

This is the summary for the PULL request, you should copy and paste the summary from the MDL issue here. For MDL-12345 the summary is `alphabetization of UI different on left and right sides` so that's what I will use.

Affects Versions

This is the version that your branch is changing, in the case of this PULL request I will select `master`.

Security level

If the issue you are working on is a security issue you should set this to the same level that the issue is set to. If its not a security issue or you don't know just leave it as none.

Pull from repository

This is the repository that the branch is at, for you this will be your github accont. If you head to http://github.com/yourname/moodle you will see a box that has three options SSH, HTTP, and Git read only, click Git Read Only and then copy the contents of the text box into the repositry field of the PULL request. For me that is git://github.com/yourname/moodle.git

Pull branch

This is the name of the branch that contains your changes. For me that is wip-MDL-12345-master.

Pull Diff URL

This is the URL to a page that shows the changes that you've made on your branch. Github is nice in that it has a pretty interface for that. To get to the interface follow these steps:

  1. Browse to https://github.com/youraccount/moodle
  2. Click `Switch Branches` and then click on the branch you changes are based upon, in my case master.
  3. Click the button labelled `Branch List`
  4. Locate your branch in the list and click the `Compare button` for me this was the compare button to the right of wip-MDL-12345-master.
  5. Copy the URL in your browse and paste it into the Pull Diff URL of the PULL request.

For me this URL was: https://github.com/yourname/moodle/compare/master...wip-MDL-23532-master For the MOODLE_20_STABLE branch this would be: https://github.com/yourname/moodle/compare/MOODLE_20_STABLE...wip-MDL-23532-MOODLE_20_STABLE

Description

The description should contain several bits of important information:

  1. A short description of what your patch does
  2. Notes about that changes you've made that might help the integrator understand what you did when he/she is reviewing it.
  3. Testing instructions that guide the testers through testing your changes.

Components

Select general from the components list, general is the only option.

Finishing up

Once you have filled in the fields as above click `Create`. This should create you PULL request and then take you to view it.

Its important to remember that there is still one thing you need to do and that is link to the original issue. While you are still logged in click the `More Actions` button and select `Link issue`, in the dialoug that pops up change the `This issue` field to `will help resolve`, type MDL-12345 into the issues field, and then type `Linking to MDL-12345` into the comments box. Once you've done that simple click Link.
I should add that the comment isn't really required. It's just something I personally like doing.

Time to create the PULL request for the other branches.

Creating the PULL request for MOODLE_20_STABLE

Once you've created the PULL request for the master branch it is time to create the PULL request for the MOODLE_20_STABLE branch. This is as easy as can be, click Create Issue in the top right, select PULL project and choose Pull Request and then click create. On the next screen enter the following details:

Summary 
Copy the summary from the previous PULL Request
Affects Versions 
Select MOODLE_20_STABLE
Security level 
Same as the MDL
Pull from repository 
Copy from the previous PULL request
Pull Branch 
wip-MDL-12345-MOODLE_20_STABLE
Pull Diff URL 
https://github.com/yourname/moodle/compare/MOODLE_20_STABLE...wip-MDL-23532-MOODLE_20_STABLE
Description 
Copy from the previous PULL request
Component 
general

Then click create to create the PULL request.

Once the PULL request has been create don't forget to link to the MDL issue like we did for the previous PULL request.

Resolving the issue

The final step for the MDL issue now that you have created the two PULL requests is to Resolve the issue (do not close it, just resolve).

You can do this by logging into tracker, browsing to the MDL issue, and clicking the resolve button.

If your fixes are integrated the tester will close the issue, otherwise if it doesn't get integrated someone will reopen the MDL.

Rebasing after the weekly release

As many of you will be aware on Wednesday after the latest weekly has been released a comment will be added to all PULL requests that are still open.

The main moodle.git repository has just been updated with latest weekly modifications.
You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

TIA and ciao :)

This is a message that is bulk added as is done so as a polite request. While it's not essential for you to rebase your work it really does help the integrators as its much easier to see what is going on and greatly reduces the chance of conflicts (because you'll solve them when you rebase).

The good news is that rebasing your work is EASY!

For the following example lets assume I created the two PULL requests above on Wednesday morning and the weekly was released on Wednesday afternoon. This means that my branch doesn't have the changes that have just been released and I need to rebase them.

Step 1: Rebasing the wip-MDL-12345-master

The first step is to rebase my work that is based upon the master branch. So first move to the master repository.

cd /var/www/repositories/master/moodle

The next thing we have to do is get all of the changes from the remote repositories, this will fetch down all of the latest changes.

git fetch --all --prune

Now before the next step make sure that you don't have any uncommit changes, if you do finish that work, commit it, then proceed to the next step which is updating your local master.

git checkout master
git pull
git push github master

Now that your local master is up to date, and has you've updated you master branch at github you are ready to rebase! At this point we want to checkout the branch we are going to rebase, in this case wip-MDL-12345-master and then we are going to rebase origin/master which is the official Moodle master branch (and the branch our branch is based upon).

git checkout wip-MDL-12345-master
git rebase origin/master

Once that command has completed your branch wip-MDL-12345-master will be completely up to date. Before you are done however you need to push the rebased wip-MDL-12345-master up to your github account so that its the branch the integrator sees.

git push -f github wip-MDL-12345-master

Now that command is just about identical to the command we used when we first pushed our branch to our github account, there is however one VERY important difference, the -f option. The -f option tells git to force the push, this is required because when you rebase all of your commit id's will change and unless you tell git to force it will see that they have changed and not allow you to push your work.

Thats it! you've now rebased your work and the integrators will be happy again :)

Step 2: Rebasing wip-MDL-12345-MOODLE_20_STABLE

Now that we've rebased our work that was based upon the master branch it is time to rebase the MOODLE_20_STABLE version. This step is just about identical to the previous step except we substitute master for MOODLE_20_STABLE.

cd /var/www/repositories/MOODLE_20_STABLE/moodle
git fetch --all --prune
git checkout MOODLE_20_STABLE
git pull
git push github MOODLE_20_STABLE
git checkout wip-MDL-12345-MOODLE_20_STABLE
git rebase origin/MOODLE_20_STABLE
git push -f github wip-MDL-12345-MOODLE_20_STABLE

Done!

So your work got rejected... what now?

First up don't give up, we (Moodle integrators) don't just reject code willy-nilly, we always provide a reason.

There are a couple of different paths that this may lead you down:

  1. More work needed: This happens if you have fixed half of the bug, or some of the bugs, but not all/completely. In this case you just need to finish off the MDL and then create new PULL requests.
  2. Bugs + white space issues: If this is the case you need to fix up what ever was spotted and then create new PULL requests.
  3. Not appropriate for a stable branch: This happens if the integrator decides that your work shouldn't get into the STABLE branch but is still OK for the master branch. Things like new features or improvements are likely to head down this path. If this is the case then everything is done, your work will have been integrated into master and you will just have to wait until the next major release (2.1, 2.2 etc).
  4. Not appropriate for core: If this happens it means that the integrator has decided that the work you've done isn't suitable to go into the main Moodle code base. If this is the case and you still want to use your branches that is fine, just leave them up on your github account and whenever you need to use them just merge your branch into your sites repository.

Deleting branches

Get a list of local branches already integrated

git branch --merged master
git branch --merged MOODLE_20_STABLE
git branch --merged MOODLE_19_STABLE

Note, that base branch itself also appears in the list.

Get a list of remote branches already integrated

git branch -r --merged master

Delete a local branch

First switch to another branch, because you can not remove branch you are currently on. Than attempt to delete a branch:

git checkout master
git branch -d wip-MDL-12345-master

If the branch has not been integrated, this command will throw an error. In this case use -D instead of -d

Delete a remote branch

In order to delete a remote branch you need to push empty content into it:

git push github :wip-MDL-12345-master

The extra mile

Now as many of you may have guessed by the time you are working on several different issues and managing several repositories the process I've described here seems VERY repetitive. And it IS!.

If you talk to any HQ developer they will tell you they deal with it in their own way, anything from scripts, to custom config files, through to virtual machine snapshots.

Personally I have a couple of bash scripts that I wrote to help me manage my workflow.

The first script I use I run after each weekly release and it does the following things:

  • Moves to each repository and does the following
    1. Fetches all remote changes
    2. Checks whether it is safe to update my local main release branches and does that if it is safe
    3. Updates all main release branches on my github account

The second script I run on every site that I have installed, it does the following

  • Goes to each site I have installed and does the following
    1. Overwrites the database with a stable snapshot
    2. Overwrites the moodle data directory with a stable snapshot taken at the same time as the database one.
    3. Upgrades the moodle site if required
    4. Takes a snapshot of the database and Moodle data directory.

The third script I run whenever I start working on a new issue and it restores the stable snapshot of a sites database and moodle data directory.

These three scripts allow me to very easily ensure I am working with the latest changes and that I can easily restore to a point I know is stable and safe.

There was a fourth script I used to use but don't any more that also rebased any unmerged branches however I found it to be far to problematic and I prefer to do it myself so that I know exactly what is going on.

Useful links

The following are links I found useful while learning Git.

I you're looking to read/buy a book on Git I'd strongly recommend Pro Git. I certainly found that the most useful book I read.