Moodle development environment with Git submodules
This page describes how to set up a development environment for a Moodle instance with external plugins. Over time the number of plugins may increase in such a way, that they can't be managed manually by developers anymore. Without any precautions this can lead to various problems, for instance:
- Missing plugins in cloned repositories
- Diverged versions in existing repositories (can lead to fatal errors when trying to update the site)
- Messed up Git excludes
You'll learn in this guide how to use Git submodules in order to achieve the following setup:
Note: The extension mod_mylittleextension is a fake plugin, which was created for testing purposes only.
By using Git submodules all extensions of a repository can be simply synchronized with other projects. All repositories are based on the so called superproject repository and the developers' projects will be sometimes referred as 'local' repository. Anyone who is interested in maintaining the superproject and core changes by Git can visit Moodle Production Server with GIT.
Note: This guide won't cover basic Git commands and how to install/maintain Git submodules. It is assumed, that you have read Git for Administrators where these topics are covered. The superproject will be seen as a Moodle repository where several plugins have been already installed via Git submodules.
Cloning a superproject
First a short explanation of events which are triggered, when a superproject is cloned. There are some things in Git submodules, which aren't obvious and can be quite surprising. Note: Ensure that you've understood the how-to at Installing and maintaining contributed extensions using Git submodules before continuing.
Assume the plugin mod_mylittleextension (remember: MLE is a fake plugin) was installed in the superproject and these changes have been commited:
$ cd /path/to/your/superproject $ git submodule add /local/repositories/mle/ mod/mylittleextension $ git commit -a -m "New module MLE installed"
Now a developer has to type in the following commands, in order to clone the superproject (maybe from a remote device).
$ cd /path/to/local/directory $ git clone <source> moodle $ cd moodle $ git submodule update --init --recursive Submodule 'mod/mylittleextension' (/local/repositories/mle) registered for path 'mod/mylittleextension' Cloning into 'mod/mylittleextension'... done. Submodule path 'mod/mylittleextension': checked out '89d9eae3d5142474d8452128e8df5720d89012cd' $ cd mod/mylittleextension mod/mylittleextension $ git branch -av * (no branch) 89d9eae Initial commit master 89d9eae Initial commit remotes/origin/HEAD -> origin/master remotes/origin/dev 3a2d487 First commit remotes/origin/master 89d9eae Initial commit
The new main repositories will automatically know about the new extension, but it won't be initialized, which means that the submodule's directory will be empty in the beginning. For initializing the submodule, the developer needs to execute git submodule update --init --recursive. The --recursive option means, that Git will walk through nested submodules as well (not obligatory in this example).
After the intialization the submodule's repository will be in detached HEAD state. This is due the fact, that Git receives following data from the superproject:
* source url * local path * HEAD reference (hash)
See this figure, for a better understanding:
This means, git submodule update --init will first do a clone of source url into the path and then check out the reference, leaving the submodule in detached HEAD state. You and your developers need to keep in mind, that any change of the HEAD's reference of your submodule will be noticed by the main repository. This means also, that everyone can switch branches unnoticedly as long as the HEAD's hash reference doesn't change. E.g. a checkout of the branch master won't be noticed by the main repository, but a checkout of dev will:
mod/mylittleextension $ git checkout master mod/mylittleextension $ git branch -av * master 89d9eae Initial commit remotes/origin/HEAD -> origin/master remotes/origin/dev 3a2d487 First commit remotes/origin/master 89d9eae Initial commit $ cd ../.. $ git status # On branch master nothing to commit (working directory clean) $ cd mod/mylittleextension mod/mylittleextension $ git checkout dev Branch dev set up to track remote branch dev from origin. Switched to a new branch 'dev' mod/mylittleextension $ cd ../.. $ git status # On branch master # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: mod/mylittleextension (new commits) # no changes added to commit (use "git add" and/or "git commit -a")
There is a shorter command to initialize the submodules, but the outcome will be the same as above. Instead of changing the moodle directory and use git submodule update --init after the clone, you can use git clone with the --recursive option as well.
$ cd /path/to/new/location $ git clone --recursive <source> moodle
In other words, you may use git submodule update --init in the case you forget the --recursive option in your git clone command.
Upgrading your submodules via the superproject
We now assume, that the external plugin is updated by it's own repository only and no customization is needed. See Installing and maintaining contributed extensions using Git submodules for further information on the topic of maintaining submodules of your superproject.
Pull and upgrade
In case a submodule was updated inside the superproject, your developers have to be careful. git pull won't upgrade the submodules at once and an additional command is needed. This is what git pull would look like:
$ cd /path/to/local/repository $ git pull remote: Counting objects: 5, done. remote: Compressing objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 0 (delta 0) Unpacking objects: 100% (3/3), done. From /local/repositories/super/ b547f6c..7f0c348 master -> origin/master Fetching submodule mod/mylittleextension remote: Counting objects: 7, done. remote: Compressing objects: 100% (5/5), done. remote: Total 6 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (6/6), done. From /local/repositories/mle 3a2d487..5e0e66a dev -> origin/dev 89d9eae..5e0e66a master -> origin/master Updating b547f6c..7f0c348 Fast-forward mod/mylittleextension | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) $ cd mod/mylittleextension mod/mylittleextension $ git branch -av * (no branch) 89d9eae Initial commit dev 3a2d487 [behind 1] First commit master 89d9eae [behind 2] Initial commit remotes/origin/master 5e0e66a Second commit remotes/origin/dev 5e0e66a Second commit
As you see, git pull triggers git fetch' inside the submodules repository, but nothing else. Therefore, your developers have to update the submodules afterwards:
$ cd /path/to/local/repository $ git submodule update Submodule path 'mod/mylittleextension': checked out '5e0e66aa787f4ebe9f61a940969a6e94abf01a1e'
Figure: Updates without customization
Known issues
Since git submodule upgrade checks out the reference, given by your main repository's commit, your submodule's repository will turn into detached HEAD state again. The command is equivalent to git checkout of the reference inside each submodule's path. That means, that conflicts will abort the checkout, like entering the directory and trying to do the checkout manually. So the submodule's repository should be clean in order to prevent problems with the submodule update.
A special case of this problem happens, if you try to clone a superproject, whose submodule's reference is not known by the origin. Your submodule's directory would be left blank (since there's nothing to check out) and you would have to resolve this issue manually (e.g. add your superprojects submodule repository as a remote). A situation, you don't want to get, for sure.
Upgrading customized plugins
Figure: Suggested workflow
In a normal workflow you don't touch external plugins and therefore there is no need to alter the submodules in any way. But what, if you want to alter them. In some cases the submodule is your own plugin or the extension is somehow insufficient but the original contributor doesn't want to merge your pull requests.
As you know from the previous section git submodule update tries to checkout a specific reference. Therefore, this reference must exist in the origin or you will get a problem while updating them. If you're not the contributer of the extension it is necessary to fork these repositiories. A suggested workflow would be like this:
Your developers are responsible for updating your forked plugin. Therefore they must merge the original changes into your forked code and resolve conflicts, if necessary. Your superproject pulls from the fork only. If you're setting up your first superproject, it doesn't have to be connected to the original repositiory at all, since you'll get all updates from your fork. So instead of adding the extension's original remote, you may add the fork only (after adding you can use git remote rename origin fork before your initial commit of the plugin).
Setting up a new submodule
In case you haven't installed the extension yet, you can add and rename the repository afterwards (to avoid misunderstanding). On your superproject's machine do the following commands:
$ cd /path/to/moodle $ git submodule add <source:fork> mod/mylittleextension Cloning into 'mod/mylittleextension'... done. $ cd mod/mylittleextension mod/mylittleextension $ git remote rename origin fork mod/mylittleextension $ git branch -avv * master 5e0e66a [fork/master] Second commit remotes/fork/HEAD -> fork/master remotes/fork/dev 5e0e66a Second commit remotes/fork/master 5e0e66a Second commit mod/mylittleextension $ cd ../.. $ git commit -a -m "New extension 'MLE' installed" [master adecc96] New extension 'MLE' installed 2 files changed, 4 insertions(+) create mode 100644 .gitmodules create mode 160000 mod/mylittleextension
Your developers will get your new submodule using git clone or git pull and can rename the remote likewise. In addition, they have to fetch updates from the original remote and have to add a remote repository manually. On your developers machine type in (output messages from Git are omitted):
cd /path/to/local/repository $ git pull $ git submodule update --init $ cd mod/mylittleextension mod/mylittleextension $ git remote rename origin fork mod/mylittleextension $ git remote add origin <source:origin> mod/mylittleextension $ git fetch origin
Now your developers can add branches, edit code, do commits and contribute to your fork. Details about that topic will follow after the next section.
Changes for an existing submodule
If the submodule is already installed and your superproject cloned, you must edit the .gitmodules file. Since the path and url of your submodule are saved (reference is saved in your commit), you may edit the file manually. But you can also use a Git command for that purpose. On the superproject's machine type in (output messages from Git are omitted):
cd /path/to/moodle/ $ git config --file=.gitmodules submodule."mod/mylittleextension".url "<source:fork>"
or edit the file .gitmodules respectively.
Afterwards type in:
$ git submodule sync Synchronizing submodule url for 'mod/mylittleextension' $ git commit -a -m "Changed source url of 'MLE'" $ cd mod/mylittleextension mod/mylittleextension $ git remote rename origin fork
The sync command will alter the remotes of your submodule's repository to match the settings of your .gitmodules. It is to mention, that changes in the remotes (like rename or add) of a submodule wont be noticed by the Moodle repository. In fact, unless you edit a file or change the submodule's HEAD reference, you can do whatever you want with your branches.
At last your developer has to update his project as well. On the developers machine type in (output messages are omitted again):
$ cd /path/to/local/repository $ git pull $ git submodule update $ git submodule sync $ cd mod/mylittleextension mod/mylittleextension $ git remote rename origin fork mod/mylittleextension $ git remote add origin <source:origin> mod/mylittleextension $ git fetch origin
Again, the sync command will change the remote settings of the submodule to match them with the .gitmodules.
Contributing to the superproject
This section will explain how to contribute patches for your superproject. Assume your developer has changed an extension and commit his changes in the submodule's repository.
$ cd /path/to/local/repository $ cd mod/mylittleextension mod/mylittleextension $ git branch -avv * dev 9701a0c [fork/dev: ahead 1] Third commit master 5e0e66a [fork/master] Second commit remotes/fork/HEAD -> fork/master remotes/fork/dev 5e0e66a Second commit remotes/fork/master 5e0e66a Second commit
Push the changes into the fork.
mod/mylittleextension $ git push fork dev:master
You may organize your forked repository in another way, e.g. let the developers push into a seperate branch and merge them only, when it is satisfying. For now, we assume the developers can push into your master (or, in general, the branch, which is tracked by your superproject). Your developer's Moodle repository will notice the changes from their commit:
$ cd /path/to/moodle $ git status # On branch master # Changes not staged for commit: # (use "git add <file>..." to update what will be committed) # (use "git checkout -- <file>..." to discard changes in working directory) # # modified: mod/mylittleextension (new commits) # no changes added to commit (use "git add" and/or "git commit -a")
Do not commit the changes. We presume, that every change in your main repository is done through a commit from your superproject. The status is just saying, that your submodule is on another state than your superproject. But this is not a problem at all. We will fix this later. Note: When your developers need to commit changes, which are not related to the submodule, they have to avoid to stage the submodule until the superproject has been updated.
Update the superproject with the following commands on the superproject's machine:
(On the superproject's machine) $ cd /path/to/moodle $ git submodule foreach git pull $ git commit -a -m "Plugin updates"
Your developers can now fetch the updated superproject.
(On the developer's machine) $ cd /path/to/local/repository $ git pull $ git status # On branch master nothing to commit (working directory clean)
As you can see, your developer don't need to commit the changed status of the submodule. Your developer's repository will be clean again after a fetch.
Figure: Workflow of an update of your submodules
See also
- Moodle Docs
- Git for Administrators
- Git for developers guide
- Windows installation using Git
- Git for Mac
- Case study Git + Moodle from Technical University Berlin
- External resources
