This is an updated version of version 1 of the proposal to support using multiple git repositories for source. We had some additional discussion on IRC. Major changes:

  • Instead of changing the existing git action, add a new git_mirror action. Keep the old action as-is, so that users may migrate gradually.

  • Mirror git repositories into .mirror in workspace. Checkouts can happen from there into wherever in the workspace.

  • Ick will come with a pipeline, tentatively called get_sources which does both the mirroring and the checkouting.

Status quo

Currently the git action works like this: the user sets the project parameters git_url and git_dir, and uses the git action in a pipeline:

- action: git
  where: host

The git_url specifies the repository URL and git_dir specifies the directory relatvie to the workspace where the repository is to be cloned. The git action does the network operations needed to clone the repository (if the git_dir directory doesn't exist), or to update an existing clone (if it does), by git remote update.

This is an action that's run on the host, not in a container, for two reasons:

  • it has access to the worker's SSH key, and so can access private repositories

  • it is expected that in the future anything that runs in the container will have no network access at all

The git action is meant to be followed by a shell or Pythona action, run in a container, which checks out the ref in the git_ref parameter.

Problems with the status quo

Status quo works, but only for single-repository projects. Daniel would like to use multiple repositories. His specific use case is that he separates the upstream part and the Debian packaging parts of his projects into separate repositories (or separate branches of the same repository). Additionally, his projects share build scripting, which are yet another git repository.

Lars notes that this is probably not a use case unique to Daniel and it would be good for ick to support constructing the workspace from any number of repositories.

Proposal

Add a new action git_mirror, which uses the project parameter sources, which is a list of dicts:

sources:
  - name: foo
    repo: git://git.example.com/foo.git
    ref: master
    location: src
  - name: tools
    repo: git://git.example.com/tools.git
    ref: master
    location: icktools

The git_mirror action will iterate over the list of dicts, and does git clone --mirror $repo .mirror/$name for each repository. If there is already a mirror, it does git remote update --prune in .mirror/$name. The action ignores the ref and location fields in each dict.

The git_mirror action needs to be run with where: host, so that it has network access and can use the worker's ssh key (in case the repository needs to be accessed over ssh).

Because the repository URLs are often on the same server, the git_mirror action supports shortening them. If the URL is relative, it is joined with the value of the git_repo_base parameter.

Example: assume the following project parameters:

git_repo_base: git://git.example.com
sources:
  - name: foo
    repo: foo.git
    ref: master
    location: src
  - name: tools
    repo: git://other-git.example.com/tools.git
    ref: master
    location: icktools

The git_mirror action would construct the following tree in the workspace:

.mirror/
    foo/        mirror of git://git.example.com/foo.git
    tools/      mirror of git://other-git.example.com/tools.git

Checkouts

Again, the git action only does the operations requiring network. It is expected that the user will provide additional shell or Python actions to update the working tree afterwards. Ick will provide a pipeline doing that.

Backwards compatibility

The old git action will work unchanged. No-one need use the new action, for now.