Ick is a continuous integration (CI) system. It may some day evolve into a continuous deployment (CD) system.

This document describes the technical architecture of ick. Specifically, the architecture for the ALPHA-1 release, but not further than that.

Background and justification

This section should be written some day. In short, Lars got tired of Jenkins, and all competitors seem insufficient or somehow unpleasant. Then Daniel suggested a name and Lars is incapable of not starting a project if given a name for it.


A continuous integration (CI) or continuous deployment (CD) system is, at its most simple core, an automated system that reacts to changes in a program's source code by doing a build of the program, running any of its automated tests, and then publishing the results somewhere. A CD system continues from there to also installing the new version of the program on all relevant computers. If a build or an automated test fails, the system notifies the relevant parties.

Ick aims to be a CI system. It deals with a small number of concepts:

  • projects, which consist of source code in a version control system (only git right now, but support for others may be added later)
  • pipelines, which are sequences of steps aiming to convert source code into something executable, or to test the program
  • workers, which do all the heavy lifting

The long-term goal for ick is to provide a CI/CD system that can be used to build and deploy any reasonable software project, including building packages of any reasonable type. In our wildest dreams it'll be scalable enough to build a full, large Linux distribution such as Debian. Also, it should be painless to deploy, operate, and use. We'll see.


We will be returning to this example throughout this document. Imagine a static website that is built using the ikiwiki software. The source of the web pages is stored in a git repo, and the generated HTML pages are published on a web server.

This might be expressed as an Ick configuration like this:

  - project:
        git_url: git://
        git_ref: master
    - build_ikiwiki_site


  - pipeline: build_ikiwiki_site
    - git_url
    - git_ref
    - rsync_target
    - python: |
        import subprocess
        def R(*args):
          subprocess.check_call(*args, stdout=None, stderr=None)
        R(['git', 'clone', '-vb', params['git_ref],
           params['git_url'], 'src'])
        R(['ql-ikiwiki-publish', 'src', params['rsync_target']])

Note that pipelines are defined in the configuration. Eventually, Ick may come with libraries of pre-defined pipelines that can easily be reused, but it will always be possible for users to define their own.

Ick ALPHA-1 architecture

The architecture of Ick is a collection of mutually recursive self-modifying microservices. (That's intended to scare you off.)

  • A project consists of one or more pipelines to be executed when triggered to do so, and some parameters given to the pipelines. Each pipeline needs to be triggered individually. Each pipeline acts in the same workspace. The entire pipeline is executed on the same worker. All workers are considered equal (for now).

  • There is currently no separate workspace description. Each pipeline needs to construct the workspace itself, if it needs to. (This probably will change.)

  • The project's pipelines do things like: prepare workspace, run actual build, publish build artefacts from worker to a suitable server. The controller keeps track of where in each pipeline a build is.

  • Workers are represented by worker-managers, which request work from the controller and perform the work by running commands locally (later also over ssh on the actual worker host).

  • Worker-builders register themselves with the controller.

  • A pipeline is a sequence of steps (such as shell or python snippets to run), plus some parameters that the shell (or whatever) can reference.

  • If a pipeline step fails, the controller will mark the pipeline execution as having failed and won't schedule more steps to execute.

Ick ALPHA-1 components

Ick consists of several independent services. This document describes how they are used individually and together.

  • The controller keeps track of projects, build pipelines, workers, and the current state of each. It decides which build step is next, and who should execute it. The controller provides a simple, unconditional "build this pipeline" API call, to be used by the trigger service (see below).

  • A worker-manager represents a build host. It queries the controller for work, and makes the build host (the actual worker) execute it, and then reports results back to the controller.

  • The trigger service decides when a build should start. It polls the state of the universe, or gets notifications of changes of the same. (Trigger services don't exist for ALPHA-1. They'll be added later.)

  • The controller and trigger services provide an API. The identity provider (IDP) takes care of the authentication of each API client, and what privileges each should have. The API client authenticates itself to the IDP, and receives an access token. The API provider gets the token in each request, validates it, and inspects it to see what the client is allowed to do. (There is no IDP for ALPHA-1. Each API client generates its own access tokens. This will change later.)

On an implementation level, the various services of Ick may be implemented using any language and framework that works. However, to keep things simple. initially we'll be using Python 3, Bottle, and Green Unicorn. Also, the actual API implementation ("backend") will be running behind haproxy, such that haproxy de-crypts TLS and sends the actual HTTP request over unencrypted localhost connections to the backend.

The API providing services will be running in a configuration like this:

Individual APIs

This chapter covers interactions with individual APIs.

On security

All APIs are provided over TLS only. Access tokens are signed using public key encryption and the public part of the signing keys is provided to all API providers at deployment time.

Getting an access token

(We don't have an IDP for handing out access tokens. Each API client gets the RSA key pair to sign tokens itself. This will be fixed later.)

The API client (user's command line tool, a putative web app, git server, worker-manager, etc) authenticates itself to the IDP, and if successful, gets back a signed JSON Web Token. It will include the token in all requests to all APIs so that the API provider will know what the client is allowed to do.

The privileges for each API client are set by the sysadmin who installs the CI system, or a user who's been given IDP admin privileges by the sysadmin.

All API calls need a token. Getting a token happens the same way for every API client.

Worker (worker-manager) registration

(Currently worker-manager only runs commands locally, and needs to run on each worker host. This will be changed later.)

The sysadmin arranges to start a worker-manager for every build host. They may run on the same host, or not: the Ick architecture doesn't really care. If they run on the same host, the worker manager will start a sub-process. If on different hosts, the sub-process will be started using ssh.

The CI admin may define tags for each worker. Attributes may include things like whether the worker can be trusted with credentials for logging into other workers, or for retrieving source code from the git server. Workers may not override such tags. Workers may, however, provide other tags, to e.g., report their CPU architecture or Debian release. The controller will eventually be able to use the tags to choose which worker should execute which pipeline steps.

The worker manager runs a very simple state machine.

Add project to controller

The CI admin (or a user authorised by the CI admin) adds projects to the controller to allow them to be built. This is done using an "CI administration application", which initially will be a command line tool, but may later become a web application as well. Either way, the controller provides API endpoints for this.

A full build

Next we look at how the various components interact during a complete build, using a single worker, which is trusted with credentials. We assume the worker has been registered and projects added.

The sequence diagrams in this chapter have been split into stages, to make them easier to view and read. Each diagram after the first one continues where the previous one left off.

Although not shown in the diagrams, the same sequence is meant to work if having multiple projects running concurrently on multiple workers.

Trigger build by pushing changes to git server

The first pipeline has now been started by the trigger service.

Pipeline 1: get sources

The first pipeline uses the trusted worker to fetch source code from the git server (we assume that requires credentials), and push them to the powerful worker.

The first pipeline finished, and the website building can start. That's the second pipeline, which has just been started.

Pipeline 2: Build static web site

The second pipeline runs on the same worker. The source is already there and it just needs to perform the build.

At the end of the second pipeline, we start the third one.

Pipeline 3: Publish web site to web server

The third pipeline copies the built static website from the trusty worker to the actual web server.

The website is now built and published.

Ick APIs

APIs follow the RESTful style

All the Ick APIs are RESTful. Server-side state is represented by a set of "resources". These data objects that can be addressed using URLs and they are manipulated using HTTP methods: GET, POST, PUT, DELETE. There can be many instances of a type of resource. These are handled as a collection. Example: given a resource type for projects Ick should build, the API would have the following calls:

POST /projects -- create a new project, giving it an ID
GET /projects -- get list of all project ids
GET /projects/ID -- get info on project ID
PUT /projects/ID -- update project ID
DELETE /projects/ID -- remove a project

Resources are all handled the same way, regardless of the type of the resource. This gives a consistency that makes it easier to use the APIs.

Note that the server doesn't store any client-side state at all. There are no sessions, no logins, etc. Authentication is handled by attaching (in the Authorization header) a token to each request. An Identity Provider gives out the tokens to API clients, on request.

Note also the API doesn't have RPC style calls. The server end may decide to do some action as a side effect of a resource being created or updated, but the API client can't invoke the action directly. Thus, there's no way to "run this pipeline"; instead, there's a resource showing the state of a pipeline, and changing that resource to say state is "triggered" instead of "idle" is how an API client tells the server to run a pipeline.

Ick controller resources and API

A project consists of a workspace specification, and an ordered list of pipelines. Additionally the project has a list of builds, and for each build a build log, and metadata (time and duration of build, what triggered it, whether it was successful or not). Also, a current state of the workspace.

A project resource:

    "project": "",
    "parameters": {
        "rsync_target": ""
    "pipelines" [

A pipeline resoure:

    "name": "ikiwiki-config",
    "actions": [
        { "shell": "cat src/ikiwiki.setup.template > ikiwiki.setup" },
        { "shell": "echo \"destdir: {{ workspace }}/html\" >> ikiwiki.setup" },
        { "name": "mkdir", "dirname": "html" }


  • the pipeline consists of a sequence of steps
  • each step is a shell snippet, a Python3 snippet, or a built-in operation implemented by the worker-manager directly
  • project parameters may be used by pipeline steps

A pipeline status resource at /projects/PROJECTNAME/pipelines/PIPELINENAME, created automatically when a project resource is updated to include the pipeline:

    "status": "idle/triggered/running/paused"

To trigger a pipeline, PUT a pipeline resource with a status field of "triggered". It is an error to do that when current status is not idle.

A build resource is created automatically, at /projects/PROJECTNAME/builds, when a pipeline actually starts (not when it's triggered). It can't be changed via the API.

    "build": "12765",
    "project": "",
    "pipeline": "ikiwiki-run",
    "worker": "bartholomew",
    "status": "running/success/failure",
    "started": "TIMESTAMP",
    "ended": "TIMESTAMP",
    "triggerer": "WHO/WHAT",
    "trigger": "WHY"

A build log is stored at /projects/ as a blob. The build log is appended to by the worker-manager by reporting output.

Workers are registered to the controller by creating a worker resource. Later on, we can add useful metadata to the resource, but for now we'll have just the name.

    "worker": "bartholomew"

A work resource resource tells a worker what to do next:

    "project": "",
    "pipeline": "ikiwiki-run",
    "step": {
        "shell": "ikiwiki --setup ikiwiki.setup"
    "parameters": {
        "rsync-target": "..."

The controller provides a simple API to give work to each worker:

GET /work/bartholomew
PUT /work/bartholomew

The controller keeps track of which worker is currently running which pipeline

Work output resource:

    "worker": "bartholomew",
    "project": "",
    "pipeline": "ikiwiki-run",
    "exit_code": null,
    "stdout": "...",
    "stderr": "...",
    "timestamp": "..."

When exit_code is non-null, the step has finished, and the controller knows it should schedule the next step in the pipeline.