Software Development at Mocana — Version Control

Note: This post is part 1 of a two-part series. The second post in this series is here.

In this post, I’d like to give an overview of some of the tools we use for software development at Mocana, specifically around version control.

Mocana has a large Subversion-based source tree. Newer projects are using git, but it’s still somewhat of a mix. We have a large codebase, dating back to our founding in 2002 (though the first Subversion commit is an import from somewhat later). Managing all the moving pieces can be a challenge, and I wanted to share some of our experiences.

Migration from Subversion: A Phased Approach

The git-svn tool makes this fairly easy. On the server hosting the souce control, assuming it hosts both the git and Subversion repositories, you can first create a git-svn clone of the Subversion repository you want to migrate.

A git-svn repository is interesting in that you can actually use it to commit back to the original Subversion repository, but only if you do so from the same git repository that you originally cloned the Subersion source from. (you cannot commit to a Subversion server using a ‘git clone’ of a git-svn cloned repository) So this will be a one-way migration; in some ways, just what we want!

We created a special mirror user on the source control server, which has access to check out from Subversion, and push to git. Then we created a git-svn directory inside its home directory, checked out each Subversion project we wanted to migrate, (using git-svn clone) and verified that the repository contained what we expected it to contain.

Imagine the following directory structure:

/home/mirror/git-svn/project1
/home/mirror/git-svn/project2
...

Add a shell function as follows:

update_and_mirror ()
{
pushd $1
git svn fetch --all
git svn rebase


# Note: not using --mirror because it deletes
# refs in the remote repository.
# This'push' makes each Subversion branch
# a full-fledged git branch!
git push /git/$1 +refs/remotes/*:refs/heads/*
popd
}

The way you run this shell function is to change to the ~/git-svn directory and run it against each Subversion checkout you want mirrored to git. (note: an [empty] repository under /git that corresponds to the name of each git-svn clone must already be created.)

This can be run periodically (for example, in a crontab) to continually push any changes in the Subversion repository to git.

The beauty of this approach is that it allows teams who are working on git-based projects to push to their own branches, while retaining the ability to merge in changes from teams who are still using Subversion. (of course, if the teams using git create something that the teams using Subversion want to use, that’s a motivation for the Subversion-based teams to make the jump to git!)

Organizing git repositories

Complex software projects today often pull together components from many different source control repositories, all of which you may want to modify and/or build from source.

At Mocana, we had the same problem, and we turned to our experience working on the Android OS to help. When working on the source code for Android, you use a tool called repo to work with the source tree. The repo tool initializes a source code sandbox from a “manifestt.xml” file. (which itself exists inside a git repository)

We used the repo tool to gather together all the software repositories we needed in order to integrate the product. Here’s an example manifest.xml file with just a few projects:


<?xml version="1.0" encoding="UTF-8"?>
<manifest>

<remote name="origin"
fetch="ssh://git.company.com/git/" />

<default revision="myproject-dev"
remote="origin"
sync-j="4" />

<project path="modules/project1"
name="project1"/>

<project path="modules/project2"
name="project2" />

<project path="third-party/otherproject"
name="otherproject" />

</manifest>

This tells repo four important things. First, the source code should be fetched from ssh://git.company.com/git/. (Note: you can also use a relative URL here. The Android project uses .. which would make it easier to mirror the repository to other servers. However, we only needed a single central server to be defined.)

Second, the default revision (branch to check out) is myproject-dev. You can also override this on a per-project basis, but usually it’s best to keep it simple. By using a branch specific to your project, you can increase the predictability of your project, assuming changes are going into that repository that you do not have direct control over. Smaller projects may want to simply use master to get the latest from each repository, but using a consistent branch name specific to your project also makes it easier to branch the project later.

Third, three git repositories should be checked out:

ssh://git.company.com/git/project1
ssh://git.company.com/git/project2
ssh://git.company.com/git/otherproject

Fourth, the layout of the source tree after running “repo init” and “repo sync” will look like this:

modules/project1
modules/project2
third-party/otherproject

Simply save your manifest XML in a file called manifest.xml and commit it to a git repository of its own. Run repo init -u to fetch the manifest, and then run repo sync to get a sandbox.

The repo tool makes it easy to organize a set of git repositories in a flexible, predictable way, without directly specifying dependencies on specific versions of the other git repositories in your project. If you want to do a build, you can always run repo manifest -r to output a version of the manifest with “hard-coded” repository revisions. You could subsequently replace your local manifest.xml with that file and do a repo sync to restore your sandbox to a previous build.

Another factor in choosing to use repo was the potential to use Gerrit for code review, an excellent code review tool that integrates with repo.

Continuous Integration

One of the tools we use for doing builds of our products is Jenkins. There is arepo plugin available for Jenkins, which makes it easy to incorporate repo into your continuous integration workflow.

Jenkins has a wide variety of plugins that can assist with other aspects of the software development process, as well.

Mike Pontillo is a Principal Engineer at Mocana.