Moodle Production Server with GIT
GIT Repository scheme
This is the description of how we maintain our Moodle2 code at TU Berlin. Our Moodle 1.9 has ca. 30000 active user with average 500 online user 24/7 (based on active users the last 15 minutes). This moves slowly to Moodle 2 now. We have a lot of code changes and plugins, so it's essential for us to maintain the code as automated as possible.
Our Moodle is called ISIS. So further on everything what is named ISIS belongs to us, everything what is called MOODLE belongs to moodle.org.
We use several GIT-repositories. The scheme with branches looks like this
The main purpose for the ISIS remote repository is for backups. One can imagine a scenario without this remote server only with local repositories involved. If you use gitolite for your remote repository, it's even more comfortable to keep the code clean, because you are able the restrict access to branches for developer and integration manager individually.
The puzzle about the isis_rebase branch will be explained, when we come to a new moodle release.
Install the repositories
We start with the original moodle code. For orientation we suggest to read the very useful description Git_for_Administrators first. Instead of the advise there we won't clone the moodle.org-repository in the first place. We start over with creating an empty repository at the ISIS Integration server.
wendt@integ:~$ mkdir isis wendt@integ:~$ cd isis/ wendt@integ:~/isis$ git init Initialized empty Git repository in /home/wendt/isis/.git/ wendt@integ:~/isis$
Now we add moodle.org as remote repository.
wendt@integ:~/isis$ git remote add moodle git://git.moodle.org/moodle.git wendt@integ:~/isis$ git fetch moodle remote: Counting objects: 597378, done. remote: Compressing objects: 100% (141938/141938), done. remote: Total 597378 (delta 442383), reused 597378 (delta 442383) Receiving objects: 100% (597378/597378), 223.38 MiB | 11.16 MiB/s, done. Resolving deltas: 100% (442383/442383), done. From git://git.moodle.org/moodle * [new branch] MOODLE_13_STABLE -> moodle/MOODLE_13_STABLE * [new branch] MOODLE_22_STABLE -> moodle/MOODLE_22_STABLE . . . * [new branch] MOODLE_23_STABLE -> moodle/MOODLE_23_STABLE * [new branch] master -> moodle/master * [new tag] v1.3.5 -> v1.3.5 . . . * [new tag] v2.3.2 -> v2.3.2 wendt@integ:~/isis$
Next we create a local branch of the moodle 2.3 code.
wendt@integ:~/isis$ git branch --track MOODLE_23_STABLE remotes/moodle/MOODLE_23_STABLE Branch MOODLE_23_STABLE set up to track remote branch MOODLE_23_STABLE from moodle. wendt@integ:~/isis$ git checkout MOODLE_23_STABLE Switched to branch 'MOODLE_23_STABLE' wendt@integ:~/isis$
From this we fork our initial ISIS code.
wendt@integ:~/isis$ git checkout -b isis_contrib Switched to a new branch 'isis_contrib' wendt@integ:~/isis$
At this point we expect an empty ISIS remote repository, where we can initially push the ISIS code. First we add it as a remote repository to our local git-repository. then we push the isis_contrib branch into the different branches of the remote repository.
wendt@integ:~/isis$ git remote add isis git://isis.remote.tu-berlin.de/isis.git wendt@integ:~/isis$ git push isis isis_contrib:isis_contrib Counting objects: 472320, done. Delta compression using up to 4 threads. Compressing objects: 100% (117914/117914), done. Writing objects: 100% (472320/472320), 199.01 MiB | 17.92 MiB/s, done. Total 472320 (delta 347911), reused 464510 (delta 342938) To git://isis.remote.tu-berlin.de/isis.git * [new branch] isis_contrib -> isis_contrib wendt@integ:~/isis$ git push isis isis_contrib:isis_productive Total 0 (delta 0), reused 0 (delta 0) To git://isis.remote.tu-berlin.de/isis.git * [new branch] isis_contrib -> isis_productive wendt@integ:~/isis$ git push isis isis_contrib:isis_rebase Total 0 (delta 0), reused 0 (delta 0) To git://isis.remote.tu-berlin.de/isis.git * [new branch] isis_contrib -> isis_rebase wendt@integ:~/isis$
To stay in sync with the remote ISIS repository we track the remote branches to local ones now.
wendt@integ:~/isis$ git branch --track isis_productive remotes/isis/isis_productive Branch isis_productive set up to track remote branch isis_productive from isis. wendt@integ:~/isis$ git branch --track isis_rebase remotes/isis/isis_rebase Branch isis_rebase set up to track remote branch isis_rebase from isis. wendt@integ:~/isis$ git checkout isis_contrib Already on 'isis_contrib' wendt@integ:~/isis$ git branch --set-upstream remotes/isis/isis_contrib Branch remotes/isis/isis_contrib set up to track local branch isis_contrib. wendt@integ:~/isis$
Now we are ready to bring the code to the productive Server. You can pull the code from the remote ISIS repository, but you will have a lot of overhead this way. So we simply rsync the code to the server.
wendt@integ:~/isis$ git checkout isis_productive Switched to branch 'isis_productive' wendt@integ:~/isis$ cd .. wendt@integ:~$ rsync -a --delete --exclude=config.php --exclude=.htaccess --exclude=.git/ isis admin@www.isis.tu-berlin.de:/var/www/ admin@www.isis.tu-berlin.de's password: wendt@integ:~$
This rsync command will reappear later updating the production code as well. At this point you can start using your moodle and follow the install procedure of moodle.
Develop and change local code
Our developers clone the ISIS remote repository to their local home directories, prepare and test their changes and push the results back to isis_contrib branch of the ISIS remote repository.
develop1@mylaptop:~$ git clone git://isis.remote.tu-berlin.de/isis.git Cloning into isis... remote: Counting objects: 472320, done. remote: Compressing objects: 100% (112942/112942), done. remote: Total 472320 (delta 347910), reused 472320 (delta 347910) Receiving objects: 100% (472320/472320), 199.03 MiB | 23.88 MiB/s, done. Resolving deltas: 100% (347910/347910), done. warning: remote HEAD refers to nonexistent ref, unable to checkout. develop1@mylaptop:~$
To be aware of changes we track the remote isis_contrib branch to a local branch.
develop1@mylaptop:~$ cd isis/ develop1@mylaptop:~/isis$ git branch -av remotes/origin/isis_contrib 562dbe4 weekly release 2.3.2+ remotes/origin/isis_productive 562dbe4 weekly release 2.3.2+ remotes/origin/isis_rebase 562dbe4 weekly release 2.3.2+ develop1@mylaptop:~/isis$ git branch --track isis_contrib remotes/origin/isis_contrib Branch isis_contrib set up to track remote branch isis_contrib from origin. develop1@mylaptop:~/isis$
Lets fork a branch, where we put and test our code changes.
develop1@mylaptop:~/isis$ git checkout isis_contrib Switched to branch 'isis_contrib' develop1@mylaptop:~/isis$ git checkout -b isis_my_code1 Switched to a new branch 'isis_my_code1' develop1@mylaptop:~/isis$
Make code changes.
develop1@mylaptop:~/isis$ emacs theme/base/style/pagelayout.css develop1@mylaptop:~/isis$
After commiting it, we can push it to the ISIS remote repository. Of course there are 1000 ways of using git to develop and change code and to push it to remote repositories. This is just one of it. Feel free to use, whatever you are used to.
develop1@mylaptop:~/isis$ git commit -a -m "base theme updated" [isis_my_code1 b5ec6cc] base theme updated 1 files changed, 1 insertions(+), 1 deletions(-) develop1@mylaptop:~/isis$ git push origin isis_my_code1:isis_contrib Counting objects: 11, done. Delta compression using up to 4 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 511 bytes, done. Total 6 (delta 5), reused 0 (delta 0) To git://isis.remote.tu-berlin.de/isis.git 562dbe4..b5ec6cc isis_my_code1 -> isis_contrib develop1@mylaptop:~/isis$
Don't forget to update your isis_contrib branch before the next code change.
develop1@mylaptop:~/isis$ git checkout isis_contrib Switched to branch 'isis_contrib' Your branch is behind 'origin/isis_contrib' by 1 commit, and can be fast-forwarded. develop1@mylaptop:~/isis$ git pull Updating 562dbe4..b5ec6cc Fast-forward theme/base/style/pagelayout.css | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) develop1@mylaptop:~/isis$
Integrate code changes to productive
Back on the integration server, the integration manager has to review and test the new code contributions. So lets get the new code changes from the ISIS remote repository.
wendt@integ:~/isis$ git checkout isis_contrib Switched to branch 'isis_contrib' wendt@integ:~/isis$ git pull remote: Counting objects: 11, done. remote: Compressing objects: 100% (6/6), done. remote: Total 6 (delta 5), reused 0 (delta 0) Unpacking objects: 100% (6/6), done. From isis.remote.tu-berlin.de/isis 562dbe4..b5ec6cc isis_contrib -> isis/isis_contrib Updating 562dbe4..b5ec6cc Fast-forward theme/base/style/pagelayout.css | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) wendt@integ:~/isis$
If you go for the code changes, you can merge them into the production branch and cherry-pick it on top of the rebase branch. I prefer cherry-picking the code, because it gives more awareness of what I do. Although there exists more ways of merging the code on top of the isis_rebase-branch.
wendt@integ:~/isis$ git checkout isis_productive Switched to branch 'isis_productive' wendt@integ:~/isis$ git merge isis_contrib Updating 562dbe4..b5ec6cc Fast-forward theme/base/style/pagelayout.css | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) wendt@integ:~/isis$ git checkout isis_rebase Switched to branch 'isis_rebase' wendt@integ:~/isis$ git cherry-pick b5ec6cc Finished one cherry-pick. [isis_rebase ed9bd24] base theme updated 1 files changed, 1 insertions(+), 1 deletions(-) wendt@integ:~/isis$
Now we push our changes to the ISIS remote repository.
wendt@integ:~/isis$ git push Counting objects: 11, done. Delta compression using up to 4 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 517 bytes, done. Total 6 (delta 5), reused 0 (delta 0) To git://isis.remote.tu-berlin.de/isis.git 562dbe4..b5ec6cc isis_productive -> isis_productive 562dbe4..ed9bd24 isis_rebase -> isis_rebase wendt@integ:~/isis$
And finally we apply the rsync again to copy the source code to the ISIS productive server.
wendt@integ:~/isis$ git checkout isis_productive Switched to branch 'isis_productive' wendt@integ:~/isis$ cd .. wendt@integ:~$ rsync -a --delete --exclude=config.php --exclude=.htaccess --exclude=.git/ isis admin@www.isis.tu-berlin.de:/var/www/ admin@www.isis.tu-berlin.de's password: wendt@integ:~$
Merge the weekly moodle release
Now, having a working Moodle instance running at the Server, we have to merge the weekly Moodle releases, To be on the safe side, you can always fork new test branches and test the merging process there, before you apply these to your productive branches. After practising it for several month everything still merges smoothly without any conflicts.
First we get the recent Moodle release in our tracked Moodle-branch.
wendt@integ:~/isis$ git checkout MOODLE_23_STABLE Switched to branch 'MOODLE_23_STABLE' wendt@integ:~/isis$ git pull remote: Counting objects: 4304, done. remote: Compressing objects: 100% (776/776), done. remote: Total 3014 (delta 2310), reused 2841 (delta 2155) Receiving objects: 100% (3014/3014), 733.24 KiB | 276 KiB/s, done. Resolving deltas: 100% (2310/2310), completed with 532 local objects. From git://git.moodle.org/moodle a8d6719..9fa159e MOODLE_21_STABLE -> moodle/MOODLE_21_STABLE a8ce907..10c3c37 MOODLE_22_STABLE -> moodle/MOODLE_22_STABLE 562dbe4..b1c5155 MOODLE_23_STABLE -> moodle/MOODLE_23_STABLE ccd90e7..7e8ae12 master -> moodle/master Updating 562dbe4..b1c5155 Fast-forward admin/cli/install.php | 14 +- admin/renderer.php | 2 +- admin/tool/customlang/locallib.php | 2 +- . . . version.php | 4 +- webservice/lib.php | 2 +- 104 files changed, 2211 insertions(+), 1248 deletions(-) create mode 100644 install/lang/ms/admin.php create mode 100644 install/lang/ms/install.php create mode 100644 install/lang/ms/moodle.php delete mode 100644 mod/lesson/importppt.php delete mode 100644 mod/lesson/importpptlib.php create mode 100644 pix/f/epub.png wendt@integ:~/isis$
Now we merge the new release into our isis branches.
wendt@integ:~/isis$ git checkout isis_contrib Switched to branch 'isis_contrib' wendt@integ:~/isis$ git merge MOODLE_23_STABLE Removing mod/lesson/importppt.php Removing mod/lesson/importpptlib.php Merge made by recursive. admin/cli/install.php | 14 +- admin/renderer.php | 2 +- admin/tool/customlang/locallib.php | 2 +- admin/tool/customlang/styles.css | 4 + . . . user/selector/search.php | 13 +- version.php | 4 +- webservice/lib.php | 2 +- 104 files changed, 2211 insertions(+), 1248 deletions(-) create mode 100644 install/lang/ms/admin.php create mode 100644 install/lang/ms/install.php create mode 100644 install/lang/ms/moodle.php delete mode 100644 mod/lesson/importppt.php delete mode 100644 mod/lesson/importpptlib.php create mode 100644 pix/f/epub.png wendt@integ:~/isis$ git checkout isis_productive Switched to branch 'isis_productive' wendt@integ:~/isis$ git merge MOODLE_23_STABLE Removing mod/lesson/importppt.php Removing mod/lesson/importpptlib.php Merge made by recursive. admin/cli/install.php | 14 +- admin/renderer.php | 2 +- . . . user/selector/search.php | 13 +- version.php | 4 +- webservice/lib.php | 2 +- 104 files changed, 2211 insertions(+), 1248 deletions(-) create mode 100644 install/lang/ms/admin.php create mode 100644 install/lang/ms/install.php create mode 100644 install/lang/ms/moodle.php delete mode 100644 mod/lesson/importppt.php delete mode 100644 mod/lesson/importpptlib.php create mode 100644 pix/f/epub.png wendt@integ:~/isis$
As the name suggests, we rebase our code changes from the isis_rebase branch on top of the MOODLE_23_STABLE branch.
wendt@integ:~/isis$ git checkout isis_rebase Switched to branch 'isis_rebase' wendt@integ:~/isis$ git rebase MOODLE_23_STABLE First, rewinding head to replay your work on top of it... Applying: base theme updated wendt@integ:~/isis$
Finally we push everything to the ISIS remote repository.
wendt@integ:~/isis$ git push Counting objects: 1062, done. Delta compression using up to 4 threads. Compressing objects: 100% (159/159), done. Writing objects: 100% (661/661), 151.24 KiB, done. Total 661 (delta 501), reused 623 (delta 488) To git://isis.remote.tu-berlin.de/isis.git b5ec6cc..df8a1ca isis_contrib -> isis_contrib b5ec6cc..7b75c8e isis_productive -> isis_productive ! [rejected] isis_rebase -> isis_rebase (non-fast-forward) error: failed to push some refs to 'git://isis.remote.tu-berlin.de/isis.git' To prevent you from losing history, non-fast-forward updates were rejected Merge the remote changes (e.g. 'git pull') before pushing again. See the 'Note about fast-forwards' section of 'git push --help' for details. wendt@integ:~/isis$
Of course pushing the isis_rebase branch fails, so we push it with force.
wendt@integ:~/isis$ git push --force isis isis_rebase:isis_rebase Counting objects: 11, done. Delta compression using up to 4 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 515 bytes, done. Total 6 (delta 5), reused 0 (delta 0) To git://isis.remote.tu-berlin.de/isis.git + ed9bd24...eeeb43f isis_rebase -> isis_rebase (forced update) wendt@integ:~/isis$
Rebase on new Moodle version
Because the developing branches of Moodle 2.4 are forked from Moodle 2.3.0 every merge from our recent productive branch with the new developing branch will fail because of conflicts. This is, why we collect all of our code changes on top of the isis_rebase branch. Once a new Moodle-release is there, we can simply put our code changes again on top of this release and have hopefully less work with conflicts.
wendt@integ:~/isis$ git branch -v MOODLE_23_STABLE e6aac11 weekly release 2.3.3+ MOODLE_24_dev 87e9331 weekly release 2.4dev isis_rebase 6cc7e47 Ver-/Entschlüsselung der Backup-Archive @author: Jan Eberhardt, 10/08/2012 wendt@integ:~/isis$ git checkout isis_rebase wendt@integ:~/isis$ git rebase --onto MOODLE_24_dev MOODLE_23_STABLE isis_rebase wendt@integ:~/isis$
