How to set up a CVS mirror

This document will outline the exact steps to set up a CVS mirror that Moodle users can use to download Moodle and maintain their web sites. (It could use some help to flesh it out!)

Get permission

The cvsmirror.moodle.org rsync server implements a whitelist of host IPs so that only official Moodle mirrors can mirror the entire repository. To become a new mirror you should contact support@moodle.com with the name of your server.

Hardware

Just as a reference, es.cvs.moodle.org is running on an old Athlon XP 3000 with 512 MB of RAM, RAID 5 IDE disks (3 UltraATA 100 disks), with Debian Etch. It doesn't get a lot of daily connections, but the machine is completely idle 99'999% of the time.

Operating System

• es.cvs.moodle.org is running on Debian Etch, using cvs, cvsd (to be able to chroot the repository access and reduce exposure in case of security breach through cvs bugs), rsync and viewcvs (to be able to browse the repository from the web).

CVS Server

A standard CVS install is fine if you can use chroot to restrict access a little, otherwise look at cvsd as detailed below.

CVS Repository

Please make sure that your Moodle repository is available under /cvsroot/moodle (for compatibility with all the other mirrors).

Mirroring the Moodle repository

You can easily mirror the Moodle repository (cvsmirror.moodle.org) to your own machine using rsync to copy recently changed files. You can simply run these commands (every 15 minutes, using cron):

cd [your local directory]

rsync -av --delete --exclude '#cvs.*' cvsmirror.moodle.org::moodle .

It is a good idea to wrap this up as a shell script, and use a lock or semaphore to prevent a pileup of processes if there's a snag anywhere. Last thing we want is to end up placing too much load on SF again. The script below is what MartinL uses on Eduforge.org --

#!/bin/bash

# get the lock,
# - don't retry too hard, it's ok if we miss a run
# - single-server so -p means we'll consider the lock stale
#   if the pid isn't around
# 
dotlockfile -p -r2 /var/tmp/moodle-sync.lock

# bring all of the repo
cd ~/moodle-cvsmirror
rsync -a --delete --exclude '#cvs.*' --exclude 'CVSROOT/config' cvsmirror.moodle.org::moodle .

# release me!
dotlockfile -u  /var/tmp/moodle-sync.lock

cvsd setup (a more secure alternative to plain cvs server)

cvsd is a cvs wrapper that adds a chroot jail to cvs, so a security breach in cvs can be contained inside the chroot. But it needs the real cvs server to do all the hard work, so you'll need to have cvs installed.

1. If your Linux distribution has cvsd already packaged, use that package and skip the download and compilaton steps and go to the configuration steps.

2. Download cvsd from http://ch.tudelft.nl/~arthur/cvsd/downloads.html. Grab the latest available version.

3. Go to the directory where you downloaded the file and extract it with:

 tar xzf cvsd-1.X.YY.tar.gz

4. Enter the newly created directory:

 cd cvsd-1.X.YY

5. Read the README file. Really, do :-)

6. Configure cvsd to put the configuration files under /etc/cvsd and the binary files under /usr/bin and /usr/sbin. We'll use the same set of configuration values that the Debian version uses:

 ./configure --prefix=/usr \
       --mandir=/usr/share/man \
       --infodir=/usr/share/info \
       --sysconfdir=/etc \
       --enable-warnings \
       --disable-dependency-tracking \
       --with-cvs=/usr/bin/cvs \
       --with-perl=/usr/bin/perl

7. Compile cvsd:

 make

8. From here on, we need to perform the rest of the steps as the root user. First install the files:

 make install

9. Then we need to create a user and group to run the cvsd daemon:

 groupadd cvsd
 useradd -m -d /var/lib/cvsd -s /bin/false -c "cvs pserver daemon" -g cvsd cvsd

10. Now we need to create the chroot jail:

 cvsd-buildroot /var/lib/cvsd

VERY IMPORTANT: You should rerun this script if your cvs binary changes or the libraries that it depends upon. (for example, when you upgrade your system or packages).

11. Once the chroot jail is created, we need to create an empty repository and setup some files before putting the real Moodle repository there:

 mkdir -p /var/lib/cvsd/cvsroot/moodle
 cvs -d /var/lib/cvsd/cvsroot/moodle init

12. We need to make sure that we allow anonymous access to the repository and that nobody has write access to it. So we execute:

 cvsd-passwd /var/lib/cvsd/cvsroot/moodle +anonymous
 touch /var/lib/cvsd/cvsroot/moodle/CVSROOT/writers

13. Now we need to create a directory where cvs can create the locks it uses internally, and make sure the user running cvsd is the owner of it:

 mkdir -p /var/lib/cvsd/cvsroot/cvs-locks/moodle
 chown cvsd:cvsd /var/lib/cvsd/cvsroot/cvs-locks/moodle

14. Now we need to edit /var/lib/cvsd/cvsroot/moodle/CVSROOT/config and specify that we don't want to use system users for authentication purposes, and that we want to use the locks dir created above. So we set the following options:

 add "SystemAuth=no"
 add "PamAuth=no"       (on systems that have this option)
 add "LockDir=/cvsroot/cvs-locks/moodle"

15. Now we need to configure cvsd. Edit /etc/cvsd/cvsd.conf and specify at least:

 RootJail /var/lib/cvsd
 Uid cvsd
 Gid cvsd
 Nice 10                (optional)
 MaxConnections 50      (optional)
 Listen * 2401
 Repos /cvsroot/moodle

This will allow a maximum of 50 concurrent connections, each one with a nice value of 10, listening on port 2401 of all the IP addresses of the machine. There will be just one available repository: /cvsroot/moodle.

16. Now we need to make sure there is nothing running on port 2401. So you should stop and disable plain cvs pserver access in your system if you are already using it.

17. Now that everything is in place, we are going to mirror the real Moodle CVS repository inside the chroot jail. This is the script that es.cvs.moodle.org uses (run by a root cron job). It's a slightly modified version of the one used by Martin Langhoff, to make it work with the chroot jail:

 #!/bin/bash
 
 # 2007.04.30 Moodle anon-cvs mirror syncing from cvs.moodle.org
 #            Slightly modified by iarenaza to make it work inside
 #            a cvsd chroot jail.
 
 # get the lock,
 # - don't retry too hard, it's ok if we miss a run
 # - single-server so -p means we'll consider the lock stale
 #   if the pid isn't around
 #
 dotlockfile -p -r2 /var/tmp/moodle-sync.lock
 
 # bring all of the repo including CVSROOT
 #  
 # Beware!!! Don't overwrite the CVSROOT subdirectory under /var/lib/cvsd/cvsroot/moodle
 # as it contains the chroot jail users and permissions. Otherwise, no one will
 # be able to use the repository.!!!!!!!!
 #
 cd /var/lib/cvsd/cvsroot/moodle
 rsync -a --quiet --delete --exclude CVSROOT --exclude '#cvs.*' cvsmirror.moodle.org::moodle .
 
 # release me!
 dotlockfile -u /var/tmp/moodle-sync.lock

18. Finally start the cvsd daemon so we can access the repository.

 /etc/init.d/cvsd start

By the way, this starts cvsd this time, but you'll need to configure it to start/stop as the system starts up/shuts down. Have a look at update-rc.d, chkconfig or the tool your distribution uses to configure starting/stopping services to see what you need to do. If you are using Debian, or Debian based distros, just do a:

 update-rc.d cvsd defaults

19. Now check that you can access the repository:

 cvs -d :pserver:anonymous@localhost:/cvsroot/moodle login
 cvs -z3 -d :pserver:anonymous@localhost:/cvsroot/moodle checkout moodle

20. If it doesn't work, and you have re-checked everything, you can have a look at the cvsd FAQ.

Letting us know

To inform the community and have your server's details added to our mirror network, contact Martin Dougiamas at moodle.com via the helpdesk.