mirroring using gitolite

v2 mirroring users:

There are several important differences in mirroring; it's best to start from a clean slate on all slaves, using this documentation.

If you're not willing to do that, you may be able to use the older, more detailed, documentation here to manage the migration. However, I cannot support that (mainly due to lack of time).

quick intro

Mirroring is simple: you have one "master" server and one or more "slave" servers. The slaves get updates only from the master; to the rest of the world they are at best read-only.

In the following picture, each box (A, B, C, ...) is a repo. The master server for a repo is colored red, slaves are green. Users only push to a master server (red), and the master server then does a git push --mirror to the slaves. The arrows show this mirror push.

Here's a more complete description of what gitolite can do:


setting up mirroring

This is in two parts: the initial setup and the rc file, followed by the conf file settings and syntax.

the initial setup and the rc file

For each server:

conf file settings and syntax

Mirroring is defined by the following options. You can have different settings for different repos, and of course some repos may not have any mirror options at all -- they are then purely local.

repo foo
    ...access rules...

    option mirror.master        =   mars
    option mirror.slaves        =   phobos deimos
    option mirror.redirectOK    =   all

The first line is easy, since a repo can have only one master.

The second is a space separated list of hosts that are all slaves. You can have several slave lists, as long as the config key starts with 'mirror.slaves' and is unique. For example.

    option mirror.slaves-1   =   phobos deimos
    option mirror.slaves-2   =   io europa
    option mirror.slaves-3   =   ganymede callisto

Do not repeat a key; then only the last line for that key will be effective.

(v3.6+) preventing automatic sync

Sometimes you don't want a repo to be mirrored automatically (as soon as someone pushes to the master) to all the slaves. For whatever reasons, you have some slaves for whom you would like to trigger the sync later (and you don't mind the fact that those slaves are out of sync until then).

To make that happen, use option lines like this instead of those shown above:

    option mirror.slaves.nosync-1   =   phobos deimos

Except for the addition of a .nosync just after slaves, all the other rules are the same as before.

(v3.6.1+) mirroring failures

Since mirror pushes happen asynchronously (i.e, the user who originally pushed does not have to wait for the mirrors to be synced), any mirror push failures are not immediately visible to a human being, although you will find them if you look in gitolite's log files.

Note: since only a successful push can clear the error status, it follows that if a mirror push failed due to an invalid hostname, that status file will need to be manually deleted from the server. Look in the bare repo directory on the server, for one or more files whose names start with 'gl-slave' and delete the appropriate one.

Therefore, when the output of the mirror push to some slave contains the word "fatal", gitolite saves the output. This saved output is printed to STDERR when any user attempts to clone/fetch/push the repo on the master server for that repo. The hope is that someone will alert an admin to look at the problem. This will continue to happen until the error condition is cleared (i.e., a successful mirror push happens to that specific slave).

If you don't want these unexpected reports confusing users (or programs!), simply create a new rc variable called HUSH_MIRROR_STATUS and set it to 1. (If you're not sure where in the rc file this should go, I suggest putting it right after the HOSTNAME variable).

You can see the mirror status of any repo using the 'mirror status' command; the command line help for the mirror command ('gitolite mirror -h' or 'ssh git@host mirror -h') has details.

manually synchronising a slave repo

You can use the gitolite mirror push command on a master to manually synchronise any of its slaves. Try it with -h to get usage info.

Tip: if you want to do this to all the slaves, try this:

for s in `gitolite mirror list slaves reponame`
    gitolite mirror push $s reponame

This command can also be run remotely; run ssh git@host mirror -h for details.

Note: if your version of the mirror command does not support 'list slaves', use gitolite git-config -r reponame mirror.slave | cut -f3 instead.

redirected pushes

Please read carefully; there are security implications if you enable this for mirrors NOT under your control.

Normally, a master, (and only a master), pushes to a slave, and the slaves are "read-only" to the users. Gitolite allows a slave to receive pushes from a user and transparently redirect them to the master.

This simplifies things for users in complex setups, letting them use their local mirror for both fetch and push access to all repos.

The syntax for enabling this is one of these:

    option mirror.redirectOK    =   all
    option mirror.redirectOK    =   phobos deimos

The first syntax trusts all valid slaves to redirect user pushes, while the second one trusts only some slaves.


appendix A: HOSTNAME substitution

Wherever gitolite sees the word %HOSTNAME, it will replace it with the HOSTNAME supplied in the rc file, if one was supplied. This lets you maintain configurations for all servers in one repo, yet have them act differently on different servers, by saying something like:

include "%HOSTNAME/*.conf"

(See include for more on the 'include' command).

You can use it in other places also, for example:

    RW+     VREF/NAME/subs/%HOSTNAME/       =   @%HOSTNAME-admins

(you still have to define @mars-admins, @phobos-admins, etc., but the actual VREF is now one line instead of one for each server!)

appendix B: efficiency versus paranoia

If you're paranoid enough to use mirrors, you should be paranoid enough to set this on each server, despite the possible CPU overhead:

git config --global receive.fsckObjects true

appendix C: moving the admin repo to a different master

Moving only some repos (other than the gitolite-admin repo) to a different master is easy. Just make the change in the gitolite.conf file, add, commit, and push.

Even for the gitolite-admin repo, if the current master is ok, it's the same thing; just make the change and push to the current master. Subsequent pushes will go to the new master, of course.

But if the current master is already dead, there's a bit of a catch-22. You can't push to the master because it is dead, and you can't push to any slave because they won't accept updates from anywhere but the server they think is the master.

Here's how to resolve this:

  1. On each slave:

    • Edit ~/.gitolite/conf/gitolite.conf to change the master and slave options for the gitolite-admin repo.

    • Run gitolite setup.

  2. Now clone the admin repo from the new master to your workstation, change the options for the rest of the repos (if needed), then add/commit/push.

And that should be all you need to do.

appendix D: fixing a broken slave

Let's say you had

option mirror.master        =   mars
option mirror.slaves        =   phobos deimos

and you then accidentally removed phobos and pushed.

You quickly realise your error and put the line back, then push again, only to realise that phobos is now unreachable. It took the previous push to heart and updated its internal state accordingly, and any subsequent attempts to touch the gitolite-admin repo on phobos results in the message:

FATAL: phobos: 'gitolite-admin' is mirrored but not here

This is basically saying, "yes I see that repo in the config, but I (phobos) am neither the master nor one of the slaves for it, so please don't ask me about it!". You're basically locked out of phobos now.

The solution is this:

That should do it.