View source for Moodle development environment with Git submodules

This page describes how to set up a development environment for a Moodle instance with external plugins.
Over time the number of plugins may increase in such a way, that they can't be managed manually by developers anymore.
Without any precautions this can lead to various problems, for instance:
* Missing plugins in cloned repositories
* Diverged versions in existing repositories (can lead to fatal errors when trying to update the site)
* Messed up Git excludes

You'll learn in this guide how to use Git submodules in order to achieve the following setup:

[[Image:superprojects1.png|frame|center|Scheme of a development environment]]

'''Note: The extension '''mod_mylittleextension''' is a fake plugin, which was created for testing purposes only.'''

By using Git submodules all extensions of a repository can be simply synchronized with other projects.
All repositories are based on the so called '''superproject''' repository and the developers' projects will be sometimes referred as 'local' repository.
Anyone who is interested in maintaining the superproject and core changes by Git can visit [[Moodle Production Server with GIT]].

'''Note: This guide won't cover basic Git commands and how to install/maintain Git submodules.
It is assumed, that you have read [[Git for Administrators]] where these topics are covered.
The superproject will be seen as a Moodle repository where several plugins have been already installed via Git submodules.'''

== Cloning a superproject ==

First a short explanation of events which are triggered, when a superproject is cloned.
There are some things in Git submodules, which aren't obvious and can be quite surprising.
'''Note: Ensure that you've understood the how-to at [[Git for Administrators#Installing_and_maintaining_contributed_extensions_using_Git_submodules|Installing and maintaining contributed extensions using Git submodules]] before continuing.'''

Assume the plugin '''mod_mylittleextension''' (remember: MLE is a fake plugin) was installed in the superproject and these changes have been commited:
<pre>
$ cd /path/to/your/superproject
$ git submodule add /local/repositories/mle/ mod/mylittleextension
$ git commit -a -m "New module MLE installed"
</pre>

Now a developer has to type in the following commands, in order to clone the superproject (maybe from a remote device).
<pre>
$ cd /path/to/local/directory
$ git clone <source> moodle
$ cd moodle
$ git submodule update --init --recursive
Submodule 'mod/mylittleextension' (/local/repositories/mle) registered for path 'mod/mylittleextension'
Cloning into 'mod/mylittleextension'...
done.
Submodule path 'mod/mylittleextension': checked out '89d9eae3d5142474d8452128e8df5720d89012cd'
$ cd mod/mylittleextension
mod/mylittleextension $ git branch -av
* (no branch)           89d9eae Initial commit
  master                89d9eae Initial commit
  remotes/origin/HEAD   -> origin/master
  remotes/origin/dev    3a2d487 First commit
  remotes/origin/master 89d9eae Initial commit
</pre>

The new main repositories will automatically know about the new extension, but it won't be initialized, which means that the submodule's directory will be empty in the beginning.
For initializing the submodule, the developer needs to execute '''git submodule update --init --recursive'''.
The '''--recursive''' option means, that Git will walk through nested submodules as well (not obligatory in this example).

After the intialization the submodule's repository will be in detached HEAD state.
This is due the fact, that Git receives following data from the superproject:
  * source url
  * local path
  * HEAD reference (hash)
See this figure, for a better understanding:

[[Image:superprojects2.png|frame|center|Dotted line: information obtained from the superproject, but already saved locally (by pull)]]

This means, '''git submodule update --init''' will first do a clone of '''source url''' into the '''path''' and then check out the '''reference''', leaving the submodule in detached HEAD state.
You and your developers need to keep in mind, that any change of the HEAD's reference of your submodule will be noticed by the main repository.
This means also, that everyone can switch branches unnoticedly as long as the HEAD's hash reference doesn't change.
E.g. a checkout of the branch '''master''' won't be noticed by the main repository, but a checkout of '''dev''' will:
<pre>
mod/mylittleextension $ git checkout master
mod/mylittleextension $ git branch -av
* master                89d9eae Initial commit
  remotes/origin/HEAD   -> origin/master
  remotes/origin/dev    3a2d487 First commit
  remotes/origin/master 89d9eae Initial commit
$ cd ../..
$ git status
# On branch master
nothing to commit (working directory clean)
$ cd mod/mylittleextension
mod/mylittleextension $ git checkout dev
Branch dev set up to track remote branch dev from origin.
Switched to a new branch 'dev'
mod/mylittleextension $ cd ../..
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   mod/mylittleextension (new commits)
#
no changes added to commit (use "git add" and/or "git commit -a")
</pre>

There is a shorter command to initialize the submodules, but the outcome will be the same as above.
Instead of changing the moodle directory and use '''git submodule update --init''' after the clone, you can use '''git clone''' with the '''--recursive''' option as well.
<pre>
$ cd /path/to/new/location
$ git clone --recursive <source> moodle
</pre>

In other words, you may use '''git submodule update --init''' in the case you forget the '''--recursive''' option in your '''git clone''' command.

== Upgrading your submodules via the superproject ==

We now assume, that the external plugin is updated by it's own repository only and no customization is needed.
See [[Git for Administrators#Installing_and_maintaining_contributed_extensions_using_Git_submodules|Installing and maintaining contributed extensions using Git submodules]] for further information on the topic of maintaining submodules of your superproject.

=== Pull and upgrade ===

In case a submodule was updated inside the superproject, your developers have to be careful.
'''git pull''' won't upgrade the submodules at once and an additional command is needed.
This is what '''git pull''' would look like:
<pre>
$ cd /path/to/local/repository
$ git pull
remote: Counting objects: 5, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 2), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From /local/repositories/super/
   b547f6c..7f0c348  master     -> origin/master
Fetching submodule mod/mylittleextension
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
From /local/repositories/mle
   3a2d487..5e0e66a  dev        -> origin/dev
   89d9eae..5e0e66a  master     -> origin/master
Updating b547f6c..7f0c348
Fast-forward
 mod/mylittleextension |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
$ cd mod/mylittleextension
mod/mylittleextension $ git branch -av
* (no branch)           89d9eae Initial commit
  dev                   3a2d487 [behind 1] First commit
  master                89d9eae [behind 2] Initial commit
  remotes/origin/master 5e0e66a Second commit
  remotes/origin/dev    5e0e66a Second commit
</pre>

As you see, '''git pull'' triggers '''git fetch''' inside the submodules repository, but nothing else.
Therefore, your developers have to update the submodules afterwards:
<pre>
$ cd /path/to/local/repository
$ git submodule update
Submodule path 'mod/mylittleextension': checked out '5e0e66aa787f4ebe9f61a940969a6e94abf01a1e'
</pre>

=== Figure: Updates without customization ===

[[File:superprojects3.png|frame|center|A workflow for an upgrade from the source repository]]

=== Known issues ===

Since '''git submodule upgrade''' checks out the reference, given by your main repository's commit, your submodule's repository will turn into detached HEAD state again.
The command is equivalent to '''git checkout''' of the reference inside each submodule's path.
That means, that conflicts will abort the checkout, like entering the directory and trying to do the checkout manually.
So the submodule's repository should be clean in order to prevent problems with the submodule update.

A special case of this problem happens, if you try to clone a superproject, whose submodule's reference is not known by the origin.
Your submodule's directory would be left blank (since there's nothing to check out) and you would have to resolve this issue manually (e.g. add your superprojects submodule repository as a remote).
A situation, you don't want to get, for sure.

== Upgrading customized plugins ==

=== Figure: Suggested workflow ===

In a normal workflow you don't touch external plugins and therefore there is no need to alter the submodules in any way.
But what, if you '''want to alter''' them.
In some cases the submodule is your own plugin or the extension is somehow insufficient but the original contributor doesn't want to merge your pull requests.

As you know from the previous section '''git submodule update''' tries to checkout a specific reference.
Therefore, this reference must exist in the origin or you will get a problem while updating them.
If you're not the contributer of the extension it is necessary to fork these repositiories.
A suggested workflow would be like this:

[[File:superprojects4.png|frame|center|Suggested workflow with customized plugins]]

Your developers are responsible for updating your forked plugin.
Therefore they must merge the original changes into your forked code and resolve conflicts, if necessary.
Your superproject pulls from the fork only.
If you're setting up your first superproject, it doesn't have to be connected to the original repositiory at all, since you'll get all updates from your fork.
So instead of adding the extension's original remote, you may add the fork only (after adding you can use '''git remote rename origin fork''' before your initial commit of the plugin).

=== Setting up a new submodule ===

In case you haven't installed the extension yet, you can add and rename the repository afterwards (to avoid misunderstanding).
On your superproject's machine do the following commands:
<pre>
$ cd /path/to/moodle
$ git submodule add <source:fork> mod/mylittleextension
Cloning into 'mod/mylittleextension'...
done.
$ cd mod/mylittleextension
mod/mylittleextension $ git remote rename origin fork
mod/mylittleextension $ git branch -avv
* master              5e0e66a [fork/master] Second commit
  remotes/fork/HEAD   -> fork/master
  remotes/fork/dev    5e0e66a Second commit
  remotes/fork/master 5e0e66a Second commit
mod/mylittleextension $ cd ../..
$ git commit -a -m "New extension 'MLE' installed"
[master adecc96] New extension 'MLE' installed
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 mod/mylittleextension
</pre>

Your developers will get your new submodule using '''git clone''' or '''git pull''' and can rename the remote likewise.
In addition, they have to fetch updates from the original remote and have to add a remote repository manually.
On your developers machine type in (output messages from Git are omitted):
<pre>
cd /path/to/local/repository
$ git pull
$ git submodule update --init
$ cd mod/mylittleextension
mod/mylittleextension $ git remote rename origin fork
mod/mylittleextension $ git remote add origin <source:origin>
mod/mylittleextension $ git fetch origin
</pre>

Now your developers can add branches, edit code, do commits and contribute to your fork.
Details about that topic will follow after the next section.

=== Changes for an existing submodule ===

If the submodule is already installed and your superproject cloned, you must edit the '''.gitmodules''' file.
Since the '''path''' and '''url''' of your submodule are saved ('''reference''' is saved in your commit), you may edit the file manually.
But you can also use a Git command for that purpose.
On the superproject's machine type in (output messages from Git are omitted):
<pre>
cd /path/to/moodle/
$ git config --file=.gitmodules submodule."mod/mylittleextension".url "<source:fork>"
</pre>
'''or''' edit the file '''.gitmodules''' respectively.

Afterwards type in:
<pre>
$ git submodule sync
Synchronizing submodule url for 'mod/mylittleextension'
$ git commit -a -m "Changed source url of 'MLE'"
$ cd mod/mylittleextension
mod/mylittleextension $ git remote rename origin fork
</pre>

The '''sync''' command will alter the remotes of your submodule's repository to match the settings of your '''.gitmodules'''.
It is to mention, that changes in the remotes (like '''rename''' or '''add''') of a submodule wont be noticed by the Moodle repository.
In fact, unless you edit a file or change the submodule's HEAD reference, you can do whatever you want with your branches.

At last your developer has to update his project as well.
On the developers machine type in (output messages are omitted again):
<pre>
$ cd /path/to/local/repository
$ git pull
$ git submodule update
$ git submodule sync
$ cd mod/mylittleextension
mod/mylittleextension $ git remote rename origin fork
mod/mylittleextension $ git remote add origin <source:origin>
mod/mylittleextension $ git fetch origin
</pre>

Again, the '''sync''' command will change the remote settings of the submodule to match them with the '''.gitmodules'''.

=== Contributing to the superproject ===

This section will explain how to contribute patches for your superproject.
Assume your developer has changed an extension and commit his changes in the submodule's repository.
<pre>
$ cd /path/to/local/repository
$ cd mod/mylittleextension
mod/mylittleextension $ git branch -avv
* dev                 9701a0c [fork/dev: ahead 1] Third commit
  master              5e0e66a [fork/master] Second commit
  remotes/fork/HEAD   -> fork/master
  remotes/fork/dev    5e0e66a Second commit
  remotes/fork/master 5e0e66a Second commit
</pre>

Push the changes into the fork.
<pre>
mod/mylittleextension $ git push fork dev:master
</pre>

You may organize your forked repository in another way, e.g. let the developers push into a seperate branch and merge them only, when it is satisfying.
For now, we assume the developers can push into your '''master''' (or, in general, the branch, which is tracked by your superproject).
Your developer's Moodle repository will notice the changes from their commit:
<pre>
$ cd /path/to/moodle
$ git status
# On branch master
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#	modified:   mod/mylittleextension (new commits)
#
no changes added to commit (use "git add" and/or "git commit -a")
</pre>

'''Do not commit the changes.'''
We presume, that every change in your main repository is done through a commit from your superproject.
The status is just saying, that your submodule is on another state than your superproject.
But this is not a problem at all. We will fix this later.
'''Note: When your developers need to commit changes, which are not related to the submodule, they have to avoid to stage the submodule until the superproject has been updated.'''

Update the superproject with the following commands on the superproject's machine:
<pre>
(On the superproject's machine)
$ cd /path/to/moodle
$ git submodule foreach git pull
$ git commit -a -m "Plugin updates"
</pre>

Your developers can now fetch the updated superproject.
<pre>
(On the developer's machine)
$ cd /path/to/local/repository
$ git pull
$ git status
# On branch master
nothing to commit (working directory clean)
</pre>

As you can see, your developer don't need to commit the changed status of the submodule.
Your developer's repository will be clean again after a fetch.

=== Figure: Workflow of an update of your submodules ===

[[Image:superprojects5.png|frame|center|The 5th command ('''git submodule update [--init]''') is only necessary, when other submodules are added or updated.]]

== See also ==

; Moodle Docs
* [[Git for Administrators]]
* [[:dev:Git for developers|Git for developers guide]]
* [[Windows installation using Git]]
* [[Git for Mac]]
* [[Moodle_Production_Server_with_GIT|Case study Git + Moodle from Technical University Berlin]]

; External resources
* [https://git-scm.com/book/en/v2/Git-Tools-Submodules Submodules at the Git book]