Introduction

Ick is a continuous integration (CI) system. It may some day evolve into also being a continuous deployment (CD) system.

This document describes the technical architecture of ick. Specifically, the architecture for the upcoming ALPHA-6 release, but not further than that. ALPHA-6 is meant to usable by people other than the primary developer.

Background and justification

This section should be written some day. In short, Lars got tired of Jenkins, and all competitors seem insufficient or somehow unpleasant. Then Daniel suggested a name and Lars is incapable of not starting a project if given a name for it.

Overview

A continuous integration (CI) or continuous deployment (CD) system is, at its most simple core, an automated system that reacts to changes in a program's source code by doing a build of the program, running any of its automated tests, and then publishing the results somewhere. A CD system continues from there to also installing the new version of the program on all relevant computers. If a build or an automated test fails, the system notifies the relevant parties.

Ick aims to be a CI system. It deals with a small number of concepts:

  • projects, which consist of source code in a version control system
  • pipelines, which are reuseable sequences of steps aiming to convert source code into something executable, or to test the program, or to achieve some other goal
  • workers, which do all the actual work by executing pipeline actions
  • artifact store, which holds results of project builds, and intermediary results used by the build

The long-term goal for ick is to provide a CI/CD system that can be used to build and deploy any reasonable software project, including building packages of any reasonable type. In our wildest dreams it'll be scalable enough to build a full, large Linux distribution such as Debian. Also, it should be painless to deploy, operate, and use.

Example project

We will be returning to this example throughout this document. Imagine a static website that is built using the ikiwiki software, using a wrapper that also pushes the generated HTML files to a web server over rsync. The source of the web pages is stored in a git repo, and the generated HTML pages are published on a web server.

This might be expressed as an Ick configuration like this:

projects:
  - project: ick.liw.fi
    parameters:
        git_url: git://git.liw.fi/ick.liw.fi
        git_ref: master
        rsync_target: ickliwfi@ick.liw.fi:/srv/http/ick.liw.fi
    pipelines:
    - get_source
    - build_ikiwiki_site
    - publish_html

pipelines:

  - pipeline: get_source
    parameters:
    - git_url
    - git_ref
    actions:
    - python: |
        import subprocess
        def R(*args):
          subprocess.check_call(*args, stdout=None, stderr=None)
        R(['git', 'clone', '-vb', params['git_ref],
           params['git_url'], 'src'])

  - pipeline: build_ikiwiki_site
    actions:
    - python: |
        import subprocess
        def R(*args):
          subprocess.check_call(*args, stdout=None, stderr=None)
        R(['ikiwiki', 'src/ikiwiki.setup'])

  - pipeline: publish_html
    parameters:
    - rsync_target
    actions:
    - shell: |
        tgt="$(params | jq .)"
        rsync -a --delete html/. "$tgt"

Note that pipelines are defined in the configuration by the user. Eventually, Ick will come with libraries of pre-defined pipelines that can easily be reused, but it will always be possible for users to define their own.

Ick architecture

The architecture of ick is a collection of mutually recursive self-modifying microservices. (That's intended to scare you off.)

  • A project consists of one or more pipelines to be executed when triggered to do so. A project defines some parameters given to the pipelines. The user (or some other entity, such as a version control server) triggers a project, and ick will execute all the pipelines. Each pipeline acts in the same workspace. The entire pipeline is executed on the same worker. All workers are considered equal.

  • There is no separate workspace description. Each project needs to construct the workspace itself, if it needs to. Each build starts with an empty directory as the workspace. The project needs to populate it by, say, git clone or by telling ick to fetch the contents of the prevoius build's workspace from the artifact store.

  • The project's pipelines do things like: prepare workspace, run actual build, publish build artifacts from worker to a suitable server. The controller keeps track of where in each pipeline a build is.

  • Each worker is represented by a worker-manager, which requests work from the controller and perform the work by running commands locally (later also over ssh on the actual worker host).

  • Worker-builders register themselves with the controller.

  • A pipeline is a sequence of actions (such as shell or python snippets to run), plus some parameters that the actions can reference.

  • If a pipeline action fails, the controller will mark the pipeline execution as having failed and won't schedule more steps to execute.

Ick components

Ick consists of several independent services. This document describes how they are used individually and together.

  • The controller keeps track of projects, pipelines, workers, builds, and the current state of each. It decides which build action is next, and who should execute it. The controller provides a simple, unconditional "build this project" API call, which the user can use.

  • A worker-manager represents and directly controls a build host. It queries the controller for work, and executes the related action on its build host, and then reports results back to the controller. Results consist of any output (stdout, stderr) and exit code.

  • An artifact store stores individual files (which may be tar files). As an example, the container system tree (see below) will be stored in the artifact store.

  • The controller and artifact store provide an API. The identity provider (IDP) takes care of the authentication of each API client, and what privileges each should have. The API client authenticates itself to the IDP, and receives an access token. The client includes the access token with each call to an API, the API provider validates the token, and inspects it to see what the client is allowed to do.

  • The icktool command line tool provides the ick user interface. It get an access token from the identity provider, and uses the controller and artifact store APIs to manage project and pipeline descriptions, build artfifacts, trigger builds, and view build status.

On an implementation level, the various services of ick may be implemented using any language and framework that works. However, to keep things simple. initially we'll be using Python 3, Bottle, and Green Unicorn. Also, the actual API implementation ("backend") will be running behind haproxy, such that haproxy de-crypts TLS and sends the actual HTTP request over unencrypted localhost connections to the backend.

The API providing services will be running in a configuration like this:

Individual APIs

This chapter covers interactions with individual APIs.

On security

All APIs are provided over TLS only. Access tokens are signed using public key encryption and the public part of the signing keys is provided to all API providers at deployment time.

Getting an access token

Ick uses Qvisqve as the IDP solution.

The API client (icktool, worker-manager) authenticates itself to the IDP, and if successful, gets back a signed JSON Web Token. It will include the token in all requests to all APIs so that the API provider will know what the client is allowed to do.

The privileges for each API client are set by the sysadmin who installs the CI system.

All API calls need a token. Getting a token happens the same way for every API client.

The exception is the API call to trigger a project build. This is un-authenticated, to avoid having to distribute API credentials to git servers. We will add a safer approach for that later.

The worker-manager

The sysadmin arranges to start a worker-manager on every build host and installs IDP credentials for each worker-manager.

The worker manager runs a very simple state machine.

Add project to controller

The CI admin (or a user authorised by the CI admin) adds projects to the controller to allow them to be built. This is done using icktool. The controller provides API endpoints for this.

Pipeline descriptions happen in the same way, except using different resources.

A full build

Next we look at how the various components interact during a complete build, using a single worker, which is trusted with credentials to external systems. We assume the worker has been registered and projects added.

The sequence diagrams in this chapter have been split into stages, to make them easier to view and read. Each diagram continues where the previous one left off.

Although not shown in the diagrams, the same sequence is meant to work if having multiple projects running concurrently on multiple workers.

Trigger build by pushing changes to git server

The project has now been marked by the controller as triggered.

Pipeline 1: get sources

The first pipeline uses the trusted worker to fetch source code from the git server (we assume that requires credentials), and push them to the powerful worker.

The first pipeline finished, and the website building can start.

Pipeline 2: Build static web site

The second pipeline runs on the same worker. The source is already there and it just needs to perform the build.

At the end of the second pipeline, we start the third one.

Pipeline 3: Publish web site to web server

The third pipeline copies the built static website from the trusty worker to the actual web server.

The website is now built and published. The controller won't give anything else to do to the worker until a new build is started.

Ick APIs

APIs follow the RESTful style

All the Ick APIs are RESTful. Server-side state is represented by a set of "resources". These data objects that can be addressed using URLs and they are manipulated using HTTP methods: GET, POST, PUT, DELETE. There can be many instances of a type of resource. These are handled as a collection. Example: given a resource type for projects ick should build, the API would have the following calls:

  • POST /projects – create a new project, giving it an ID
  • GET /projects – get list of all project ids
  • GET /projects/ID – get info on project ID
  • PUT /projects/ID – update project ID
  • DELETE /projects/ID – remove a project

Resources are all handled the same way, regardless of the type of the resource. This gives a consistency that makes it easier to use the APIs.

Except for blobs, all resources are in the JSON format. Blobs are just sequences of bytes and don't have structure. Build artifacts and build logs are blobs.

Note that the server doesn't store any client-side state at all. There are no sessions, no logins, etc. Authentication is handled by attaching (in the Authorization header) a token to each request. The identity provider gives out the tokens to API clients, on request.

Note also the API doesn't have RPC style calls. The server end may decide to do some action as a side effect of a resource being created or updated, but the API client can't invoke the action directly. Thus, there's no way to "run this pipeline"; instead, there's a resource showing the state of a pipeline, and changing that resource to say state is "triggered" instead of "idle" is how an API client tells the server to run a pipeline.

Ick controller resources and API

A project consists of a list of pipelines and parameters for them:

{
    "project": "liw.fi",
    "parameters": {
        "rsync_target": "www-data@www.example.com/srv/http/liw.fi"
    },
    "pipelines" [
        ""
    ]
}

A pipeline resoure:

{
    "name": "ikiwiki-config",
    "actions": [
        { "shell": "cat src/ikiwiki.setup.template > ikiwiki.setup" },
        { "shell": "echo \"destdir: {{ workspace }}/html\" >> ikiwiki.setup" },
        { "name": "mkdir", "dirname": "html" }
    ]
}

Here:

  • the pipeline consists of a sequence of actions
  • each action is a shell snippet, a Python3 snippet, or a built-in operation implemented by the worker-manager directly
  • project parameters may be used by pipeline actions
  • each pipeline should declare what parameters it expects, but nothing checks that

A pipeline status resource at /projects/PROJECTNAME/pipelines/PIPELINENAME, created automatically when a project resource is updated to include the pipeline:

{
    "status": "idle/triggered/running/paused"
}

To trigger a pipeline, PUT a pipeline resource with a status field of triggered. It is an error to do that when current status is not idle.

A build resource is created automatically, at /builds/PROJECTNAME/BUILDNUMBER, a pipeline is triggered. It can't be changed via the API.

{
    "project": "liw.fi",
    "build_id": "liw.fi/12765",
    "build_number": 12765,
    "log": "logs/liw.fi/12765",
    "parameters": {},
    "pipeline": "ikiwiki-run",
    "worker": "bartholomew",
    "status": "building",
}

A build log is stored at /logs/liw.fi/12765 as a blob. The build log is appended to by the worker-manager by reporting output.

Workers are registered to the controller by creating a worker resource. Later on, we can add useful metadata to the resource, but for now we'll have just the name.

{
    "worker": "bartholomew"
}

A work resource resource tells a worker what to do next:

{
    "project": "liw.fi",
    "pipeline": "ikiwiki-run",
    "step": {
        "shell": "ikiwiki --setup ikiwiki.setup"
    },
    "parameters": {
        "rsync-target": "..."
    }
}

The controller provides a simple API to give work to each worker:

GET /work

The controller identifies the worker from the access token.

The controller keeps track of which worker is currently running each pipeline.

Work output resource:

{
    "worker": "bartholomew",
    "project": "liw.fi",
    "pipeline": "ikiwiki-run",
    "exit_code": null,
    "stdout": "...",
    "stderr": "...",
    "timestamp": "..."
}

When exit_code is non-null, the step has finished, and the controller knows it should schedule the next step in the pipeline. If exit_code is a non-zero integer, the action failed.