This page provides an RB'S and an Atom feed of new blog entries, including the text of the entries, not just titles.

Distributed CI and threat modelling

Some thoughts about making a distributed CI system secure.

  • outline of system

    • version control system hold source code
    • IDP authenticates and authorizes users, system components
    • controller co-ordinates builds, collects build logs
    • artifact store holds build artifacts
    • workers (many) do the actual building, are told by controller what to do, fetch source from version control system, upload artifacts to artifact store
  • entitites in the system that need to be protected:

    • the person using CI
    • the person running the IDP, controller, and artifact store (for simplicity, assume they're all run by the same person, although they could each be run by separate people)
    • the people running runners
  • threats to person using CI

    • malicious workers, which embeds unwanted code in build artifact
      • mitigation: use reproducible builds and build on at least two workers to detect unwanted changes in artifacts; this would work OK, if there are relatively few malicious workers
    • many malicious workers, or workers that become malicious after a long period of working fine
      • mitigation: have at least one trusted worker, which might be slow, but whose output is required for a build to be trusted
        • artifacts from maybe-trusted workers can't be used for deployment, but could be used with sufficient isolation to speed things up, e.g., to do heavy testing: if the trusted worker later confirms the binaries are trustworthy (bitwise identical), then the test results can be trusted, too
      • variant of mitigation: require at least N maybe-trusted workers to produce bitwise identical build artifacts, where N is set by the person running the CI or whose project is being built
      • rejected: a karma or reputation system based on past behaviour: this makes long-lived workers valuable targets, and years of good behaviour won't protect if the worker gets hijacked
  • threats to person running IDP, controller, artifact store

    • there are no new threats to these that come due to the distributed nature of CI
    • all the usual threats apply, of course
  • threats to those running workers

    • build uses too much CPU or RAM
      • mitigation: enable person running worker to set limits and priorities so that the build doesn't use resources needed by other things
    • build attacks remote hosts (e.g., DDoS)
      • mitigation: prevent build from accessing any network hosts, except version control server, controller, artifact store
    • build attacks host where worker runs
      • mitigation: run build in a VM, using the best avilable isolation techniques, such as carefully configured qemu/KVM to implement the VM, and keeping all related software up to date
liw
Planning meeting (iteration 34)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

The controller+Muck task is carried over from the previous iteration.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Implement login/logout in webui Ivan+Pyry 4
Change the controller to use Muck Lars 4
Total Ivan+Pyry 4
Total Lars 4

Task descriptions

  • Change the controller to use Muck: Change the controller to use Muck for all persistent data storage. Add Muck to the Ick cluster. Create a new Ick instance with the new controller and Muck. Add two users to the Ick instance, one for Lars and another test account.

    Acceptance criteria: The new Ick instance can build all of Lars's projects using his existing .ick files from git.

  • Implement login/logout in webui: Add login and logout functionality to the webui. A user who is not logged in can log in, and a logged in user can log out. Log in happens using OIDC using the IDP.

    Acceptance criteria: The webui clearly tells the user if they are logged in, and lets the user log in and out as they want.

Posted
liw
Retrospective meeting (iteration 33)

People

  • liw
  • Ivan
  • Pyry

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Write an API test for facade /version Ivan+Pyry yes
Change the controller to use Muck Lars partially

What went well?

  • One task got completed.

What didn't go well?

  • Lars over-allocated himself. Again.

What can we improve for the future?

  • Lars to be more realistic.
Posted
liw
Planning meeting (iteration 33)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration aims at taking further steps towards a web UI and using Muck to store controller data persistently.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write an API test for facade /version Ivan+Pyry 4
Change the controller to use Muck Lars 4
Total Ivan+Pyry 4
Total Lars 4

Task descriptions

  • Write an API test for facade /version: The facade now runs and has an endpoint /version. Pick a suitable implementation approach and write an automated test for the endpoint. The test should be possible to test any facade instance, given it's base URL.

    Acceptance criteria: The test runs successfully against ickdev-api.vm.liw.fi when Lars runs it.

  • Change the controller to use Muck: Change the controller to use Muck for all persistent data storage. Add Muck to the Ick cluster. Create a new Ick instance with the new controller and Muck. Add two users to the Ick instance, one for Lars and another test account.

    Acceptance criteria: The new Ick instance can build all of Lars's projects using his existing .ick files from git.

Posted
liw
Retrospective meeting (iteration 32)

People

  • liw
  • Ivan
  • Pyry

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting for the iteration.

what Who Done
Write trivial facade for ickui Ivan+Pyry yes
Make Muck benchmark to simulate log storage Lars yes

What went well?

  • We got everything done.
  • Ivan and Pyry were proactive in arranging an Ick hack day.

What didn't go well?

  • Life events make it difficult to commit to significant amounts of time. Nothing to be done about that.

What can we improve for the future?

  • No real change needed, but keep tasks minimal and clearly defined, as that makes it easier to get them done.

  • Maybe arrange hack days more often?

Posted
Hackday in Helsinki

Three of us (Lars, Ivan, Pyry) met today and worked on getting the new Ick web UI started. Due to time constraints, we didn't very far, but there's now the scaffolindg of a facade. The facade will provide the JS frontend (running in the browser) with a convenient API, and will make any requests needed for the Ick controller that are necessary. The facade will also participate in the user authentication, as part of the OpenID Connect protocol.

Posted
liw
Planning meeting (iteration 32)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about moving Ick forward to using Muck.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write trivial facade for ickui Ivan+Pyry 1
Make Muck benchmark to simulate log storage Lars 1
Total Ivan+Pyry 1
Total Lars 1

Task descriptions

  • Write trivial facade for ickui: Write a facade that provides a trivial /version endpoint, which reports the facade's version, without requiring a login.

    Acceptance criteria: Facade /version can be tested manually with curl. The output includes the git commit, if the facade is running from a git clone.

  • Make Muck benchmark to simulate log storage: The design of how build logs should be stored in Muck was discussed on the Ick mailing list. The design that the discussion ended up with should be validated by having a benchmark show that it works acceptably fast with a 100,000 lines of build log.

    Acceptance criteria: The benchmark simulates how the controller would store logs, and reports a speed for re-assembling the log from snippets. (It is not necessary to make it fast; that's out of scope for the task.)

Posted
liw
Retrospective meeting (iteration 31)

People

  • liw
  • Ivan
  • Pyry

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Write trivial facade for ickui Ivan+Pyry no
Add Muck and Yuck to the Ick project Lars yes
Draft plan for using Muck in Ick controller Lars yes

What went well?

  • Process mostly worked.

What didn't go well?

  • Life events happend.

What can we improve for the future?

  • Smaller tasks.
Posted
Netsurf and Ick

Netsurf (http://www.netsurf-browser.org/) is a lightweight and portable web browser. It targets several older operating systems, for which it is often the only graphical web browser.

Netsurf is cross-built for many older operating systems which can not support native build. It is natively built on a number of esoteric systems for which cross-building is not possible. For some more modern operating systems, NetSurf is built for several different toolkits and target kinds, including some pure-test-related builds whose outputs are coverage reports, test summaries, etc, rather than executable artifacts.

NS currently uses Jenkins, but finds it problematic. A particular problem is that current versions of Jenkins want a version of Java that is newer than what is supported on some of the systems NS targets. Jenkins has also proven to be inflexible to use. NS is interested in replacing Jenkins with Ick, eventually.

The NetSurf project has sunk a significant amount of effort into making the older Jenkins fit-for-use, so while replacement is desirable, there will need to be an easy pathway to transition to Ick, or to have Ick run alongside Jenkins while jobs are migrated piecemeal over time.

Another significant issue with Jenkins for the NetSurf project is that of configuration management. Jenkins' ability to be configured without use of its web-UI is, while not limited as such, unpleasant to work with. Significant security issues surround the JNLAPI and as such, given the age of the Jenkins in use, NetSurf are unable to automate configuration management. A feature of Ick which attracts the project is the possibility of git-based configuration of the CI system. However, the project does find the web-UI useful from time to time in order to make small tweaks to configuration (for example when debugging a CI job). Ick will need to support some configuration changes through a web-UI (ideally making and pushing git commits behind the scenes).

NS is divided into many components, sub-projects, each of which builds something like a library used by other components. The various components thus have build dependencies on each other, and if a dependency changes, any components that build-depend on it need to be built and tested as well.

Some components are host-based tools which are consumed by many downstream variants of component builds. For example, the binding generator is built for each host (e.g. x86_64 Linux vs. i686 Haiku) and is then consumed by any downstream build running on that host, regardless of its target (e.g. on x86_64 Linux, a build of NetSurf targetting AmigaOS).

The NS project hosts its own builders, and there is at minimum ssh access to them. They may need to have each builder run multiple workers, to use hardware efficiently. The builders are not all co-located in the same physical location: being frugal with bandwidth use is a concern.

All of the builders are linked to a virtual LAN (single broadcast domain across multiple sites). This means that in situations where bandwidth is a concern, non-encrypted protocols are considered just as secure as encrypted ones. While SSH is available to all hosts, some are perhaps less reliable than others. It would behoove Ick to not rely on communication purely via SSH if at all possible. Sadly Python is not necessarily available on all hosts either. A small C-based agent may be required for maximum flexibility and minimum risk of TTY/SSH related issues.

NetSurf has CI notifications routed to a number of targets including the project IRC channel, a mailing list, and an RSS feed from the Jenkins. In order for these routes to be useful, NetSurf would find the ability to construct rules in order to filter and route notifications very useful indeed. For example, every successful or failed build should not be notified to IRC, instead only when a job changes state from success to failure or back. Ensuring that notifications carry useful semantic information and links will also be desirable.

NetSurf makes use of Jenkins' history of builds, allowing the project to trace non-fatal issues through builds. These link back to the git repositories at the specific commits built, and this is considered a critical feature of the UI.

NetSurf would very much like it if the CI could be easily configured to deploy branch-based builds in a way which will not confuse artifact handling, and will allow developers to build a branch across many repositories in order to simplify the testing of the development of a new feature which crosses libraries. Such a concept is often referred to as "system branches".

It would be best if these builds were not created by default for any branch, but rather that there was a simple way for a developer to activate a set of jobs for a branch. Perhaps by adding the branch name to a configuration in Ick via either the web-UI or via the git configuration. Or perhaps by pattern of branch name.

Posted
liw
Planning meeting (iteration 31)

People

  • liw
  • Ivan
  • Pyry

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • Ick development has stalled in recent weeks, due to life and work happening. This iteration is about trying to move things forward, at least a little.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write trivial facade for ickui Ivan+Pyry 1
Add Muck and Yuck to the Ick project Lars 1
Draft plan for using Muck in Ick controller Lars 2
Total Ivan+Pyry 1
Total Lars 3

Task descriptions

  • Write trivial facade for ickui: Write a facade that provides a trivial /version endpoint, which reports the facade's version, without requiring a login.

    Acceptance criteria: Facade /version can be tested manually with curl. The output includes the git commit, if the facade is running from a git clone.

  • Add Muck and Yuck to the Ick project: Add pages to the ick.liw.fi website that describe Muck and Yuck, and their architecture and (planned) features.

    Acceptance criteria: The new pages are announced on the ick-discuss mailing list.

  • Draft plan for using Muck in the Ick controller: Draft a plan for how the Ick controller should store data in Muck, including a transition plan, and how access control should work.

    Acceptance criteria: A blog post on ick.liw.fi with the plan has been published, and announced on the ick-discuss mailing list.

Posted
Ick hack day in Helsinki

Today, three people met for most of the day in Helsinki to work on Ick. One of these was new to the project: Pyry. Also there was Ivan and me. We spent the day talking. No actual code was written, but we inducted Pyry to the project, and had extensive and detailed discussions about authentication.

Pyry and Ivan will be working together on the Ick web UI. They'll be using React for the parts that run in the browser, and probably Python for the parts that form the "facade". The facade will be necessary for OpenID Connect authentication using the authorization code flow, since that requires there to be a trusted component, the facade, to hold the access and refresh tokens. The browser isn't to be entrusted with these.

We went over the OpenID Connect protocol in some detail. This will hopefully allow Pyry and Ivan to work on implementing the web ui effectively.

I would classify the day as a success, even if no code was written. The authentication part is a little tricky and it's good to have more people understand it.

Posted
liw
Retrospective meeting (iteration 30)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Update arch doc about OIDC authentication Lars yes
Review updated arch doc Daniel yes
Give Ivan access to ickui server Lars yes
Write trivial facade for ickui Ivan carried over
Create ick pipeline to deploy ickui facade Ivan carried over

What went well?

  • Some things got done.

What didn't go well?

  • Life interfered and ruined estimations.

What can we improve for the future?

  • Have no life.
Spelling of project name: 'Ick' with a capital I

After a brief discussion on IRC, we've decided that the proper spelling of the project name is "Ick", with a capital I. The set of software is spelled the same way: "Ick". An individual program is spelled with a lower case initial: "icktool".

For example, "Today, the Ick project is proud to release a new version of the Ick continuous integration and deployment system. It includes a rewritten implmentation of the 'icktool' command line program."

liw
Planning meeting (iteration 30)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about advancing the ick web UI that Ivan is working on: we want to add a facade, which runs between the frontend (Javascript in the browser) and the backend (controller and other ick components with and API). The facade is necessary for authentication security.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Update arch doc about OIDC authentication Lars 2
Review updated arch doc Daniel 1 (timebox)
Give Ivan access to ickui server Lars 0
Write trivial facade for ickui Ivan 1
Create ick pipeline to deploy ickui facade Ivan 2
Total Lars 2
Total Daniel 1
Total Ivan 3

Task descriptions

  • Update arch doc about OIDC authentication: The architecture document currently claims only OAuth2 authetication is used. Update it to explain how OpenID Connect if used to authenticate end-users, and why a separate facade is needed.

    Acceptance criteria: Daniel says OK.

  • Review updated arch doc: Review changes to the ərchitecture document and either say they're OK, or provide feedback on what needs improvement. Repeat with Lars until OK.

    Acceptance criteria: Daniel says OK.

  • Give Ivan access to ickui server: Ivan will need to deploy the ickui facade on the ickui server. For this, he needs to log in and have root access. Make it so.

    Acceptance criteria: Ivan confirms he can do ssh ivan@ick-ui.vm.liw.fi sudo id and have it report the root user. No password is needed.

  • Write trivial facade for ickui: Write a facade that provides a trivial /version endpoint, which reports the facade's version, without requiring a login.

    Acceptance criteria: Facade /version can be tested manually with curl. The output includes the git commit, if the facade is running from a git clone.

  • Create ick pipeline to deploy ickui facade: Set up a project on the demo ick instance, which deploys the ickui facade on the ickui server, on an empty freshly installed one, or upgrading an existing server.

    This can be done by cloning the facade git repo to a suitable location on the server. Additionally, any systemd units, haproxy, etc, need to be set up, and any missing dependencies installed, by the ick project.

    Acceptance criteria: When the facade git repo is updated, the ick CD project is triggered, and after it's done, the running facade reports a new version. This works even if the ickui server has been deleted and a new one created.

Posted
liw
Retrospective meeting (iteration 29)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Draft spec for API tokens in Qvisqve Lars yes
Draft spec for variable expansion in .ick files Lars yes
Set up a server to host ick web UI Lars yes
Prepare React App framework repository and populate with Hello World app Ivan yes
Verify independent ability to build and run webapp Daniel yes

What went well?

  • We got everything done.

What didn't go well?

  • ?

What can we improve for the future?

  • ?
Posted
Qvisqve API tokens: a design

Continuing previous musings on API tokens. Discussed this briefly with Daniel on IRC, and we came to the decision that going for the approach of using standard client credentials is the better way over inventing a non-standard API token format and protocol for using them.

Thus the approach is as follows:

  • User logs into web application to create a new API client. User gives them a name, which gets prefixed by their username. Thus, if user "igor" creates a client called "slab", the actual client id is "igor/slab". Qvisqve creates the client, and shows the user the client id and client secret. The secret is generated by Qvisqve.

    The client id prefix is there to make it easier to see whose client it is.

    The client gets given a list of allowed scopes. This is a subset of the ones the user has at the time of client creation. The user may choose to allow only some of the scopes for the client.

    The client gets an attribute "sub", which gets assigned a value identifying the user who created the client.

  • The Qvsiqve API for creating clients also sets the sub attribute from the token. (The web application uses the same API endpoint.)

  • An API client with a suitable scope gets to set or remove the sub field of a client it creates or updates.

  • When a client requests an access token use OAuth2 client credentials, the token's sub field is set according to the client's sub attribute, if it is set.

    An access token will have scopes that are the intersection of scopes the client requested, that the client is allowed to get, and the user is allowed to get, at the time of the token being granted.

  • Qvisqve will eventually allow users to manage their own applications: remove the client, reset the client secret (Qvisqve generates and sets a new one), add and remove scopes (from the set of scopes the client has).

  • Also eventually, Qvisqve will add groups, but that's not relevant yet.

Added (suggested by Daniel): Clients should NOT be given a scope to add new clients. It may be worthwhile to have Qvisqve enforce that. Only users should be able to do that.

Specification: variable expansion in .ick files

Ick currently doesn't support variable expansion in .ick files at all. Sometimes it would be practical for reducing repetition. For example: it would be nice to have a pipeline like this:

pipeline: build_systree
actions:
- debootstrap: {{debian_codename}}
- archive: workspace
  artifact_name: "{{artifact_name}}-{{debian_codename}}"

Currently, the debootstrap action implementation knows about the debian_codename project parameter. With variable expansion, we don't need to have magic, hardcoded parameter, and can be more explicit.

The archive: workspace action likewise can avoid using a magic, hardcoded variable name.

The Mustache templating language has been suggested and looks OK. I especially like that it isn't Turing complete, making it easier to understand the templates. (Jinja2 is Turing complete, and tries to do too much. It's also Python specific.)

To add variable expansion support, I propose that we do the following:

  • .ick files may use variable expansion in any string value, but not in dictionary keys. This keeps things orthogonal and simple.

  • Expansion happens when a build starts. The build resource created will have all values expanded.

  • Variables for expansion are all project parameters, and some additional variables defined by the controller. Such variables will initially be:

    • project — the name of the project
    • buildno — the build number
    • buildid — the build id
  • The value of a variable is converted to a string upon expansion, if it isn't already. Booleans and integers are straigtforward. List and dict values are converted into a normalised JSON representation (but it's not expected to be actually useful).

  • We add a "project variable" concept. Each project will have an associated dict, which contains variables the user may set, and some that the ick controller will add or change, such as the next build number for the project. This is distinct from "project parameters", which are set in the project definition. We'll add API calls for this:

    • GET /projects/foo/vars — return current variables dict
    • PUT /projects/foo/vars — update variables dict; this updates the whole dict, and will remove variables not in the update

    The controller will add/update the project, buildid, buildno variables in this dict, when a build is triggered. The user may add any variables to the dict they wish.

Posted
API tokens for Qvisqve

This is not directly part of ick, but Qvisqve, but it's something ick needs, so I'm putting this in the ick blog.

Access control to ick APIs happens using OAuth2 and OpenID Connect (OIDC) access tokens. The API client and end-user authenticate themselves to Qvisqve, and the API client gets an access token in return. The API grants access based on the access token. The token is digitally signed by Qvisqve, and only grants specific access. The signature prevents anyone from modifying the token to gain extra access.

Qvisqve currently implements two methods for getting access tokens:

  • The OAuth2 "client credentials grant" is for non-human API users and are based on "client id" and "client secret"; this is what icktool uses.

  • The OIDC "authorization code flow" is for interactive authentication of humans.

The problems I am current worried about are:

  • Ick needs to start isolating its users from each other. If Alice has a project, Bob should not be able to modify it, delete it, see it, or even know it exists. Because of this, icktool needs to identify the end-user, but the client credentials grant doesn't do that.

  • Non-interactive use of the ick APIs (icktool in a cron job, for example) needs to use instance specific credentials that can be monitored and revoked individually. It doesn't seem good for all machines using icktool to use the same credentials: if those leak, all machines lose access at once. Also, different machines may need different types of access, and shouldn't get any more than they need.

I can see two solutions:

  • Create a new "client" for each user and use instance: if Alice needs to run icktool on her laptop and a server, that's two clients, with different client ids and secrets (alice_on_laptop and alice_on_server). This creates a separate identity for each instance, allowing separate monitoring and revocation: if Alice's laptop gets stolen, only the alice_on_laptop identity needs to be revoked (or secret changed). Also, each of the clients gets tied to a user. The user would manage their API clients' access via a web application, as a logged-in user, granting and revoking client access as they need to.

    Pro: this is standard OAuth2 and requires no changes for clients to support it.

    Con: this results in a fair number of API client identities, but that's not a serious problem.

    Con: users need to give API clients two things (client id, client secret), which also isn't a very serious problem.

  • Add support for an "API token". This would be a new kind of token that can only be used for getting an access token, in lieu of client credentials. The API token would identify the user, and what subset of the user's access the token grants. API clients would give the API token to Qvisqve, and get a normal access token. The API token would optionally not expire.

    Pro: slightly easier for the user: just one thing to handle, instead of two.

    Con: non-standard OAuth2 extension, would require all API clients to support it. Might be an issue for OAuth2 libraries.

I'm currently not sure which of these is better, but I put out this blog post to document my thinking on this. Feedback is welcome.

liw
Planning meeting (iteration 29)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about moving ick towards being hostable. The current roadmap for that is reproduced below, and is on the roadmap page as well.

  • There are currently to major aspects that need work: isolating users from each other, and having a friendly web user interface.

  • For isolation, we will make each "resource" in the ick API be owned by a user, and for that we will need to change API use to rely on access tokens that identify the user. This means moving form the OAuth2 client credential authentication to OpenID Connect, extended with "API tokens".

  • For the web user interface, Ivan and Daniel have agreed to collaborate. Lars will stay out of it, since he's not a front-end developer. The current plan is to use React, and to build the static JS files in a container, using npm, and deploying the generated files to a server.

Roadmap for hostable ick

Roadmap for this iteration

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Draft spec for API tokens in Qvisqve Lars 1
Draft spec for variable expansion in .ick files Lars 1
Set up a server to host ick web UI Lars 1
Prepare React App framework repository and populate with Hello World app Ivan 4
Verify independent ability to build and run webapp Daniel 2
Total Lars 3
Total Daniel 2
Total Ivan 4

Task descriptions

  • Draft spec for API tokens in Qvisqve: Draft a specification for how API tokens would work in Qvisqve, and how they would be used by icktool. Offer draft for review.

    Acceptance criteria: At least one person says +1 and nobody has serious objections.

  • Draft spec for variable expansion in .ick files: To let users reduce repetition in .ick files, and to shorten them, it would be nice to be able to use variables. Draft a specification for how that would work, with examples. Probably based on the Mustache specification.

    Acceptance criteria: At least one person says +1 and nobody has serious objections.

  • Set up a server to host ick web UI: Set up a VM, domain name, and TLS certificate to serve the static files generated from the ick web UI. Later on, this will also host the ick UI facade application (which will be necessary to do logins securely). The demo ick workers, and Daniel, and Ivan, should have ssh logins and passwordless sudo access on the host.

    Acceptance criteria: Daniel and Ivan accept the server setup and confirm they can log in and become root with sudo.

  • Prepare React App framework repository and populate with Hello World app: The application should say "hello, world" (or some similar greeting), and use the React machinery to do so: npm to install dependencies, whatever the React build command is chosen to do the build.

    Acceptance criteria: Ivan says OK.

  • Verify independent ability to build and run webapp: Review the work by Ivan to make sure the application can be built and run by someone else than Ivan. This is a necessary step to automate build and deployment.

    Acceptance criteria: Daniel says OK.

Posted
Lars Wirzenius
New voting member: Daniel Silverstone

Status quo is that I, Lars, am the only voting member of the ick project. I am also the secretary.

Decision: I have arranged a vote amongst myself to add Daniel Silverstone as a voting member of the project. Daniel has accepted this. This vote also means that my own membership is extended by one year. See the ick project governance page for details:

Daniel's membership, and my extended membership are set to expire on 2019-09-01, so just over a year from now. These will be automatically extended if we vote before then.

Daniel has been helping the ick project since it started: he gave ick its name. He also provides ick with mailing lists, and has provided me with valuable feedback and assistance with architectural decisions, as well as using ick and providing feedback on that.

Welcome, Daniel.

liw
Retrospective meeting (iteration 28)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Make make-it-so accept any number of input files Lars yes (was alreday done)
Add a "archive: systree" action Lars yes
Make controller check pipelines get all their parameters Lars yes
Make controller, icktool allow deleting resource Lars yes (was already done)
Write a dummy controller program that can be profiled Lars yes
Specify a lint command to icktool Daniel yes

What went well?

  • Pretty much everything.

What didn't go well?

  • It's too hot.

What can we improve for the future?

  • Live inside a cave.
Posted
icklint finds problems in your .ick files

I've recently written a new tool: icklint. You give it your ick project and pipeline descriptions, and it looks for problems. Currently it checks for the following things:

  • Each pipeline must have actions.
  • There is not pipeline field called "parameter" (the correct spelling is "parameters", in the plural).
  • Each project has pipelines and parameters.
  • Each project defines all the parameters its pipelines expect, and no parameters none of its pipelines expect.

To use:

  • Get the script from http://git.liw.fi/icklint/tree/ (you can just download the script, or run it from a cloned repository). Requires the Python YAML library (python3-yaml in Debian).

  • Download all projects and pipeline descriptions from your ick controller:

      icktool show /projects > projects.json
      icktool show /pipelines > pipeline.json
    
  • Run: ./icklint projects.json pipelines.json

All errors are printed to stdout.

If you can think of more checks for icklint to have, do suggest!

Posted
liw
Planning meeting (iteration 28)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about cleaning up rough edges.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Make make-it-so accept any number of input files Lars 1
Add a "archive: systree" action Lars 1
Make controller check pipelines get all their parameters Lars 1
Make controller, icktool allow deleting resource Lars 1
Write a dummy controller program that can be profiled Lars 4
Specify a lint command to icktool Daniel 1
Total Lars 8
Total Daniel 8

Task descriptions

  • Make make-it-so accept any number of input files: Currently you have to run make-it-so separately for each input file, in a loop. Put the loop inside make-it-so, for user convenience.

    Acceptance criteria: Lars can run icktool make-it-so pipelines/*.ick in the ick source tree, to update the "standard library of pipelines" on his instance.

  • Add a "archive: systree" action: Currently, to build a systree, you have to run the entire debootstrap, which is rather slow. Add an action to the worker manager, which archives the systree. This allows one to build a systree by building on top of another systree.

    Acceptance criteria: Lars can build two systrees (build-essential and python-all), the latter of which is built on top of the former.

  • Make controller check pipelines get all their parameters: Currently, the controller does not check at all, in any way, that pipelines get all the parameters they declare they want. Change the controller so that it checks that all pipelines used in a build get all the parameters they declare they want, and fail the build (before actually executing any actions) if that is not so.

    Acceptance criteria: Trigger a project build with a missing parameter results in a failed build. A yarn scenrio tests this.

  • Make controller, icktool allow deleting resource: Add to controller API endpoints to delete resources: project, pipelines, build, logs. Add to icktool a command to use the endpoints.

    Acceptance criteria: There are yarns to check that one can delete a project, pipeline, build, and log, and they pass. It is not necessary for deleting a project to also delete any builds or logs associated with it, or to delete the log if a build is deleted.

  • Write a dummy controller program that can be profiled: Currently, we can't use the Python profiler to see where time is spent in the controller, because of complications with gunicorn. Write a little dummy controller, which reads dummy API requests from a file, and executes them, as if it were the full controller, but not running as a service under gunicorn. Run it always under the Python profiler.

    Acceptance criteria: Running the dummy controller results in Python profiler output.

  • Specify a lint command to icktool: Write a short specification for an icktool command lint, which checks projects and pipelines in the controller for things that look fishy, but aren't errors, as such. For example pipelines that aren't used by any projects.

    Acceptance criteria: Review on the ick-discuss list is positive. The spec should list any checks lint should do, when it gets implemented. The spec is added to ick.liw.fi as a blog post.

Posted
liw
Retrospective meeting (iteration 27)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Implement the git_mirror action Lars yes
Implement a get_sources pipeline Lars yes
Document the get_sources pipeline Lars yes
Review get_sources documentation Daniel yes
Add a way to install pipelines with Ansible Lars no

Excuses: Lars realised that installing pipelines with Ansible leads to complications, and dediced not to do that. For example: if sysadmin installs pipeline foo, and user modifies foo, should foo be reset to the version from ick2.git when ick is upgraded? Should users be able to modify foo in the first place?

Instead, Lars changed the controller to handle projects and pipelines with slashes, and added to ick2.git some sample pipelines that are prefixed with ick/ in their names. We can revisit how these can be installed automatically later on.

What went well?

  • Most things.

What didn't go well?

  • Slight snafu in the planning of the Ansible task.

What can we improve for the future?

  • All fine for now.
Posted
liw
Planning meeting (iteration 27)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The goal of this iteration is to implement the git_mirror specification. This includes implementing a pipeline that also does the "checkout" part.

  • Additionally, this iteration we start a "standard library" of ick pipelines: pipelines that are installed to ick when it's installed, before any users create their pipelines.

Roadmap for this iteration

  • N/A

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Implement the git_mirror action Lars 2
Implement a get_sources pipeline Lars 2
Document the get_sources pipeline Lars 1
Review get_sources documentation Daniel 0
Add a way to install pipelines with Ansible Lars 1
Total Lars 6

Task descriptions

  • Implement the git_mirror action: Add a new action git_mirror to the worker manager, according to spec.

    Acceptance criteria: It's deployed to the demo site, and a test project using it builds OK and mirrors (first run) or updates (additional runs) the mirrors.

  • Implement a get_sources pipeline: See the spec. The get_sources pipeline should use git_mirror to update mirrors of the relevant source code, and additional actions to check out the relevant source code version.

    Acceptance criteria: A test project using the pipeline correctly checks out a non-master branch of at least two git repositories.

  • Document the get_sources pipeline: Put the pipeline in the ick source repository, and add YAML comments to explain how it works.

    Acceptance criteria: Someone else reviews the comments, and says they're sufficient to understand how to use the pipeline.

  • Review get_sources documentation: Review pipeline documentation: does it seem clear how the pipeline should be used, after a quick skim?

    Acceptance criteria: Have read the docs, told Lars if they're any good.

  • Add a way to install pipelines with Ansible: Install any pipelines in the ick source tree into the Debian packages, and change initial startup so that they're available for users to use in their projects.

    Acceptance criteria: The get_sources pipeline is usable by a test project in a fresh ick instance.

Posted
liw
Retrospective meeting (iteration 26)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Write spec for "multi-git" feature Daniel, Lars yes
Change icktool make-it-so to do PUT before POST Lars skipped
Fix icktool trigger to give better error messages Lars yes
Change icktool to download from the artifact store Lars yes
Add an action to archive only parts of workspace Lars yes

The multi-git spec.

The icktool PUT-before-POST change was attempted, but turned out to not improve things much.

What went well?

  • First iteration with another participant!
  • We got everything done.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
Lars Wirzenius
Version 2: git action: multiple repository support

This is an updated version of version 1 of the proposal to support using multiple git repositories for source. We had some additional discussion on IRC. Major changes:

  • Instead of changing the existing git action, add a new git_mirror action. Keep the old action as-is, so that users may migrate gradually.

  • Mirror git repositories into .mirror in workspace. Checkouts can happen from there into wherever in the workspace.

  • Ick will come with a pipeline, tentatively called get_sources which does both the mirroring and the checkouting.

Status quo

Currently the git action works like this: the user sets the project parameters git_url and git_dir, and uses the git action in a pipeline:

- action: git
  where: host

The git_url specifies the repository URL and git_dir specifies the directory relatvie to the workspace where the repository is to be cloned. The git action does the network operations needed to clone the repository (if the git_dir directory doesn't exist), or to update an existing clone (if it does), by git remote update.

This is an action that's run on the host, not in a container, for two reasons:

  • it has access to the worker's SSH key, and so can access private repositories

  • it is expected that in the future anything that runs in the container will have no network access at all

The git action is meant to be followed by a shell or Pythona action, run in a container, which checks out the ref in the git_ref parameter.

Problems with the status quo

Status quo works, but only for single-repository projects. Daniel would like to use multiple repositories. His specific use case is that he separates the upstream part and the Debian packaging parts of his projects into separate repositories (or separate branches of the same repository). Additionally, his projects share build scripting, which are yet another git repository.

Lars notes that this is probably not a use case unique to Daniel and it would be good for ick to support constructing the workspace from any number of repositories.

Proposal

Add a new action git_mirror, which uses the project parameter sources, which is a list of dicts:

sources:
  - name: foo
    repo: git://git.example.com/foo.git
    ref: master
    location: src
  - name: tools
    repo: git://git.example.com/tools.git
    ref: master
    location: icktools

The git_mirror action will iterate over the list of dicts, and does git clone --mirror $repo .mirror/$name for each repository. If there is already a mirror, it does git remote update --prune in .mirror/$name. The action ignores the ref and location fields in each dict.

The git_mirror action needs to be run with where: host, so that it has network access and can use the worker's ssh key (in case the repository needs to be accessed over ssh).

Because the repository URLs are often on the same server, the git_mirror action supports shortening them. If the URL is relative, it is joined with the value of the git_repo_base parameter.

Example: assume the following project parameters:

git_repo_base: git://git.example.com
sources:
  - name: foo
    repo: foo.git
    ref: master
    location: src
  - name: tools
    repo: git://other-git.example.com/tools.git
    ref: master
    location: icktools

The git_mirror action would construct the following tree in the workspace:

.mirror/
    foo/        mirror of git://git.example.com/foo.git
    tools/      mirror of git://other-git.example.com/tools.git

Checkouts

Again, the git action only does the operations requiring network. It is expected that the user will provide additional shell or Python actions to update the working tree afterwards. Ick will provide a pipeline doing that.

Backwards compatibility

The old git action will work unchanged. No-one need use the new action, for now.

Posted
Lars Wirzenius
Ick git action: multiple repository support

This is a proposal to change ick's git action to support multiple repositories. It's been discussed on IRC and on the ick-discuss mailing list: ticket STANDARD APOLLO REFORM. This page is based on those discussions.

Status quo

Currently the git action works like this: the user sets the project parameters git_url and git_dir, and uses the git action in a pipeline:

- action: git
  where: host

The git_url specifies the repository URL and git_dir specifies the directory relatvie to the workspace where the repository is to be cloned. The git action does the network operations needed to clone the repository (if the git_dir directory doesn't exist), or to update an existing clone (if it does), by git remote update.

This is an action that's run on the host, not in a container, for two reasons:

  • it has access to the worker's SSH key, and so can access private repositories

  • it is expected that in the future anything that runs in the container will have no network access at all

The git action is meant to be followed by a shell or Pythona action, run in a container, which checks out the ref in the git_ref parameter.

Problems with the status quo

Status quo works, but only for single-repository projects. Daniel would like to use multiple repositories. His specific use case is that he separates the upstream part and the Debian packaging parts of his projects into separate repositories (or separate branches of the same repository). Additionally, his projects share build scripting, which are yet another git repository.

Lars notes that this is probably not a use case unique to Daniel and it would be good for ick to support constructing the workspace from any number of repositories.

Proposal

Change the git action to expect a project parameters git and git_defaults. The former is a list of key/value pairs (dicts), and the latter gives defaults for keys not specified in those dicts. Example:

- project: foo
  parameters:
    git_defaults:
      repo_base: git://git.example.com/
      ref: master
    git:
    - repo: foo.git
      dir: src
    - repo: foo/debian.git
      dir: src/debian
      ref: debian/unstable
    - repo: git://other-git.example.com/scripts.git
      dir: scripts

The git action would construct the following tree in the workspace:

src/            clone of git://git.example.com/foo.git
  .git/
  debian/       clone of git://git.example.com/foo/debian.git
    .git/
scripts/        clone of git://other-git.example.com/scripts.git
  .git/

In the dicts in the git parameter, if repo is a relative URL, it is joined with the repo_base value from git_defaults, or from the same variable in the same git dict.

Again, the git action only does the operations requiring network. It is expected that the user will provide additional shell or Python actions to update the working tree afterwards. Ick will provide such an action, later, when we've honed one to work well.

Backwards compatibility

If there is no git parameter, the git action will look for the git_url and git_dir parameters, and act as if a git parameter is defined with those values:

git_url: git://git.example.com/foo.git
git_dir: src

With these parameters, the git action would pretend as if the folloging parameter exists:

git:
- repo: git://git.example.com/foo.git
  dir: src

Additionally, the git action will add to the build log a deprecation warning suggesting the use of the git parameter.

Posted
liw
Planning meeting (iteration 26)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about making ick more usable by Daniel and others.

Roadmap for this iteration

  • N/A

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write spec for "multi-git" feature Daniel, Lars 1
Change icktool make-it-so to do PUT before POST Lars 1
Fix icktool trigger to give better error messages Lars 1
Change icktool to download from the artifact store Lars 1
Add an action to archive only parts of workspace Lars 1
Total Daniel 1
Total Lars 5

Task descriptions

  • Write spec for "multi-git" feature: Daniel, and others, will want to construct the source tree from overlaying multiple git repositories. Currently the git action only allows one. Write a spec for a new git action that can do what Daniel wants.

    Acceptance criteria: Both Lars and Daniel are happy with the spec: it should allow Daniel to do what he wants, and Lars is happy to implement it.

  • Change icktool make-it-so to do PUT before POST: Currently, make-it-so tries to POST (create) each project, and if it fails, does a PUT (update) instead. For most use, this means the POST fails, and that takes time. Change it to do PUT first, saving half the HTTP requests.

    Acceptance criteria: Lars can use make-it-so on his production ick and it is not slower than before the change.

  • Fix icktool trigger to give better error messages: Triggering a project that doesn't exist, or a project that references a pipeline that does not exist, currently gives very confusing error messages. Change the controller to give a more humane error message.

    Acceptance criteria: The error messages explain what is wrong, not using HTTP status codes and Python stack traces.

  • Change icktool to download from the artifact store: There is currently no good way for a user to download an artifact from the artifact store, without using ssh to log into the server. Change qvisqve configuration to allow the necessary scopes, and icktool to request those scopes, and icktool to have a command for getting a blob from the artifact store and write it to stdout.

    Acceptance criteria: Lars can download an artifact manually from the artifact store.

  • Add an action to archive only parts of workspace: It would be nice to store, in the artifact store, only some of the files from the workspace. For example, only the Debian packages built by ick. This would allow a user to download only those, and not have to download a very large artifact.

    Acceptance criteria: Lars can create a project that builds a file in /workspace, archives that in the artifact store, and Lars can then download that artifact.

Posted
Ick ALPHA-6 released: usable by others

After some updates to the ick website, but no significant changes to the software, ick has now reached ALPHA-6. It has, demonstrably, been deployed and used by other people than its primary developer.

Ick can, right now:

  • Build system trees for containers.
  • Use system trees to run builds in containers.
  • Build Debian packages.
  • Publish Debian packages via its own APT repository.
  • Deploy to a production server.

There's still many missing features. Ick is by no means ready to replace your existing CI/CD system, but if you'd like to have a look at ick, and help us make it the CI/CD system of your dreams, now is a good time to give it a whirl.

Posted
liw
Retrospective meeting (iteration 25)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Update existing installation docs Lars yes
Update existing user docs Lars yes
Clean up Ansible playbooks Lars yes
Invite people to try Lars yes

What went well?

  • Mostly everything.

What didn't go well?

  • A little difficulty getting people to install or try ick.

What can we improve for the future?

  • Onboarding? Make ick more attractive.
Posted
liw
Planning meeting (iteration 25)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The goal of this iteration is to produce enough ick installation instructions and user manual to allow Daniel to try setting it up for himself for building Gitano.

Roadmap for this ALPHA cycle

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Update existing installation docs Lars 2
Update existing user docs Lars 2
Clean up Ansible playbooks Lars 2
Invite people to try Lars 0
Total Lars 6

Task descriptions

  • Update existing installation docs: Review the existing installation docs, in the INSTALL file and on the website. Combine them to one doc on the website, and update that.

    Acceptance criteria: Lars can mechanically follow the installation steps to set up a new ick instance.

  • Update existing user docs: Review the existing docs (mainly on the website). Add material to cover concepts (copy from arch doc, maybe). Add example project definitions and icktool example command lines. Add a walkthrough of an example from a fresh, empty ick instance until a simple project has been built and published in the ick APT repository.

    Acceptance criteria: Lars can mechanically follow the example steps steps to set up and build a new project.

  • Clean up Ansible playbooks: Review the Ansible playbooks for setting up ick. Refactor them so they're easy for someone else to use, by only providing a fresh "vars" file to Ansible. Document all the variables that need to be defined.

    Acceptance criteria: Lars's personal instances are set up using the same playbooks, using a vars file for his needs. Also, the updated installation instructions use the updated Ansible playbooks.

  • Invite people to try: Invite at least Daniel to set up and use an ick instance, following the updated documentation.

    Acceptance criteria: At least one person tries and tells us how far they got.

Posted
Ick ALPHA-5 released: notifications

Ick is a continuous integration (CI) system. It is starting to be ready for you to use for real, at least for simple projects. See http://ick.liw.fi/ for more information.

The fifth ALPHA version (0.52) of ick has now been released. The hightlight feature is notifications: ick can now send an email with the build log after a build finishes. Other changes can be seen in the NEWS file snippet below.

The ALPHA-6 cycle of ick development will concentrate on improving documentation (both sysadmin and user documentation), as well as making the deployment process smoother. By the end of ALPHA-6 we will invite people to try ick for real.

Development continues, see the roadmap for what's planned.

NEWS for ick2, a CI server

Version 0.52, released 2018-06-11

  • Add a notification service, which sends the build log when a build ends. Also, make the controller invoke the notification at the end of each build.

Version 0.50, released 2018-04-28

  • Add populate_workspace action.

Version 0.49, released 2018-04-27

  • The rsync action now deletes files from the target if they're not in the source.

Version 0.48, released 2018-04-27

  • New action to do git clone or git remote update.

  • New action to do rsync -av from a directory in the workspace to a remote server.

  • New action to do dput *.changes in the root of the workspace, to upload Debian packages to the APT repository that is part of an ick cluster.

Version 0.47, released 2018-04-25

  • Worker manager now sets LC_ALL and DEBIAN_FRONTEND variables inside the container.

  • Each action in a pipeline MUST now have a where field. Previously you could leave it out and it would act as if where: host was specified.

  • All commands run by worker manager now get their stdin redirected from /dev/null.

  • icktool delete command can now delete projects and pipelines.

Version 0.46, released 2018-04-22

  • The populate_systree action now takes the name from the systree_name parameter, if no systree_name field is given, or its value is auto.

  • New command: icktool show-latest-log PROJECT.

  • icktool make-it-so now accepts filename from which to read project and pipeline descriptions.

  • icktool --no-verify-tls now actually works.

Version 0.45, released 2018-04-21

  • Force new release, since the previous one failed to build right. Again. This is why I am writing CI software.

Version 0.44, released 2018-04-21

  • Force new release, since the previous one failed to build right.

Version 0.43, released 2018-04-21

  • icktool status is now faster. Previously it was linear to the number of projects, now it is as fast for one as for a hundred. Also, the list of projects is now sorted by project name.

  • icktool show-log has been added.

  • icktool show can show any JSON resource (not logs) and needs to be given the URL path component.

  • icktool make-it-so now updates existing projects and pipelines.

  • icktool trigger now takes any number of projects to trigger on the command line.

Version 0.42, released 2018-04-20

  • icktool status, icktool trigger commands added.

  • icktool build-graph PROJECT [BUILD-ID] command added. It writes out a .dot file for showing the build graph for a build. Feed that to dot(1) for processing. Works for a running build as well as older builds.

Version 0.41, released 2018-04-20

  • icktool is now the rewritten version.

  • The controller now allows triggering an entire project, instead of a pipeline in a project. Additionally, when the project build starts, all actions from all pipelines are queued for the build.

Posted
liw
Retrospective meeting (iteration 24)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Add a notification action to worker-manager Lars yes
Change controller to create notification actions Lars yes
Change Lars's ick deployments to use an SMTP server Lars yes

What went well?

  • We have an ick that notifies about builds.

What didn't go well?

  • Adding the notification actions to the build graph was much more difficult than anticipated.

  • Also, life interfered.

What can we improve for the future?

  • Don't have a life.

  • Have a way to handle unexpectedly large tasks.

Posted
Changing ick to prepare for notification actions

I'm going to change ick to manage actions in build graphs in way to make it sensible to add actions to do notifications later on. Here's what I'm planning.

  • Each build will have a state: triggered, building, done, failed. Later on, I will add states notifying-ok, and notifying-fail, for doing notifications.

  • Each action in the build graph will have dependencies (which actions must succeed before the action can be done). Dependencie will refer to other actions via an build-local identifier, so that will be added as well. Actions will also have a state: blocked (not all dependencies are satisfied), ready (action can be executed), building (a worker is executing the action), done (action succeeded), failed (action failed).

  • When actions are created, the first action of the first pipeline will be in state ready, everything else will be blocked and depending on the previous action.

  • Currently builds have a current-action attribute. This will be removed. Instead, the controller will scan the build graph for available actions that it can assign to a worker (state is ready), when a worker needs something to do.

  • When an action fails, the controller marks it to have state failed, and also marks the build as state failed.

  • When an action succeeds, the controller marks it as done. It then scans all the actions, and marks any actions that are in state blocked as ready, if all the actions it depends on are now in state done. If all actions are now in state done, the controller marks the build as done.

Later on, I will change this so that when all actions are done, the build moves to state notifying-ok, and the controller adds notification actions. If an action fails, the build is marked as notifying-fail, and adds appropriate notification actions.

The build state diagram will be like this:

Actions state diagrams will be like this:

Posted
liw
Planning meeting (iteration 24)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about adding notifications about builds.

  • Lars already accidentally added a rudimentary notification service, which can send the emails. What remains is to have the controller have those notifications triggered at opportune moments.

Roadmap until ALPHA-6

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Add a notification action to worker-manager Lars 1
Change controller to create notification actions Lars 1
Change Lars's ick deployments to use an SMTP server Lars 1
Total Lars 3

Task descriptions

  • Add a notification action to worker-manager: Add an action: notify to the worker-manager, with fields recipients, build, and log, to be given to the notification service. Also add the field url, which has the URL to the notification service endpoint.

    Add a controller configuration variable for the notification endpoint. Change Ansible playbooks to set that variable.

    The action shall do a POST to the given URL, with a JSON body with the three fields.

    Add to ickdev.yaml a manully triggered notification action.

    Acceptance criteria: Running test-ick, which uses ickdev.yaml, results in a notification email being sent.

  • Change controller to create notification actions: Change the controller to look for a notify field in project specifications, and if it's there, to create a notification action at the end of each build of the project. The notify project field should have list of email addresses, to be given to the notify action as the recipients field.

    Add notify fields to projects in ickdev.yaml. Remove manually triggered notification actions.

    Acceptance criteria: Running test-ick results in notification emails from each project builds ending.

  • Change Lars's ick deployments to use an SMTP server: The ick-cluster playbook should add a notification service to the controller host, and configure its SMTP access entirely via Ansible variables.

    Acceptance criteria: test-ick passes and notification emails are received by the recipients.

Posted
liw
Retrospective meeting (iteration 23)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Add action populate_workspace Lars yes
Modify ci-prod.yaml to do incremental builds Lars yes

What went well?

  • Most everything.

What didn't go well?

  • The worker-manager is a bit too slow to test.

What can we improve for the future?

  • N/A.
Posted
liw
Planning meeting (iteration 23)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about implementing incremental builds, by adding actions to allow user implement persistent workspaces.

Roadmap until ALPHA-6

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Add action populate_workspace Lars 1
Modify ci-prod.yaml to do incremental builds Lars 1
Total Lars 2

Task descriptions

  • Add action populate_workspace: Add a new action to worker-manager, which retrieves an artifact from the artifact store and unpacks it into the workspace. The actions gets the name of the artifact from the workspace_artifact parameter. If the artifact doesn't exist, the action should not fail, merely not do anything.

    Modify ickdev.yaml to archive the workspace at the end of a build, and to populate it from the artifact store at the start of the build.

    Acceptance criteria: test-ick passes after the changes.

  • Modify ci-prod.yaml to do incremental builds: Modify Lars's production ick projects and pipelines to do incremental builds: the git repositories shouldn't be cloned from scratch each time, ikiwiki should do incremental builds instead of full builds. Debian package building should ensure packages from the previous build do not get get re-uploaded.

    Upgrade ick on the production instance.

    Acceptance criteria: All projects build in the production instance.

Posted
liw
Retrospective meeting (iteration 22)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Add action to run git clone or git remote update Lars yes
Add action to run rsync Lars yes
Change ick-cluster.yml to caret dput.cf for the cluster apt repo Lars yes
Add action to run dput Lars yes

What went well?

  • Most things.

What didn't go well?

  • Esimates were wildly over the top.

What can we improve for the future?

  • All good.
Posted
liw
Planning meeting (iteration 22)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about adding built-in actions to the worker manager so that Lars's pipelines no longer need to copy in the worker ssh keys into the container to do network operations, such as git, rsync, and dput. After this itertion, all such operations that require ssh keys will be done outside the container.

Roadmap until ALPHA-6

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Add action to run git clone or git remote update Lars 4
Add action to run rsync Lars 2
Change ick-cluster.yml to caret dput.cf for the cluster apt repo Lars 1
Add action to run dput Lars 1
Total Lars 8

Task descriptions

  • Add action to run git clone or git remote update: Add an action action: git to the worker manager. It should use the project parameter git_dir to get the name of a subdirectory in the workspace. If that directory does not exist, it should do git clone git_url git_dir, where git_url is another project parameter. If the directory does exist, the action should run git remote update in the directory.

    Acceptance criteria: A version of ickdev.yaml modified to use the new action works with test-ick.

  • Add action to run rsync: Add an action action: rsync which uses the project parameters rsync_src and rsync_target and runs rsync -av --delete rsync_src rsync_target/. on the host at the root of the workspace.

    Add to ickdev.yaml a project that creates a dummy static website and uses the rsync action to publish it. The web server should be provided by the test-ick script.

    Acceptance criteria: test-ick passes after the changes.

  • Change ick-cluster.yml to caret dput.cf for the cluster apt repo: The playbook knows the domain for the APT repository in the cluster. It should create a dput configuration file (in /etc/dput.cf) that can upload to the cluster APT repository. The configuration should define an upload target named ick.

    Acceptance criteria: Manual testing.

  • Add action to run dput: Add an action action: dput which runs dput ick *.changes at the root of the workspace. It should assume dput has an upload target ick defined, which is the cluster's APT repository.

    Modify ickdev.yaml to use this new action.

    Acceptance criteria: test-ick passes after the changes.

Posted
liw
Retrospective meeting (iteration 21)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Write a deploy-and-test script Lars yes
Change APT to keep old versions Lars yes
Fix WM to fail if artifact upload fails Lars yes
Get systree name from parameter Lars yes
Fix WM to really set env vars Lars yes
Run actions with stdin from /dev/null Lars yes
Change the debmirror action to have a default mirror Lars yes
Remove unnecessary scripts from source Lars yes
Identify worker from token, not path Lars yes
Require an explicit where on all actions Lars yes
Add icktool show-latest-log command Lars yes
Add icktool delete command Lars yes
Change icktool make-it-so to accept filenames Lars yes

What went well?

  • The script to deploy and test to a set of test VMs took much longer than estimated.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
liw
Planning meeting (iteration 21)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about cleaning things up and fixing small bugs that don't take a lot of effort.

Roadmap until ALPHA-6

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write a deploy-and-test script Lars 4
Change APT to keep old versions Lars 1
Fix WM to fail if artifact upload fails Lars 1
Get systree name from parameter Lars 1
Fix WM to really set env vars Lars 1
Run actions with stdin from /dev/null Lars 1
Change the debmirror action to have a default mirror Lars 1
Remove unnecessary scripts from source Lars 1
Identify worker from token, not path Lars 1
Require an explicit where on all actions Lars 1
Add icktool show-latest-log command Lars 1
Add icktool delete command Lars 1
Change icktool make-it-so to accept filenames Lars 1
Total Lars 16

Task descriptions

  • Write a deploy-and-test script: Write a script that builds ick from a git commit, creates VMs, installs and configures ick, creates some representative projects, and verifies all of those can be built. This may take some time to write, but it will save time when testing changes.

    Acceptance criteria: Manual testing.

  • Change APT to keep old versions: The APT configuration currently set up does not keep old versions. It is sometimes useful to keep them. Change the reprepro configuation to keep old versions.

    Acceptance criteria: Set up an ick cluster, trigger a build twice. Check that both versions are in the package pool in the APT repository.

  • Fix WM to fail if artifact upload fails: Currently, it seems, when the worker-manager uploads an artifact (workspace) to the artifact store, with curl, and the upload fails, it doesn't fail the whole build. Fix.

    Acceptance criteria: Code review.

  • Get systree name from parameter: Change the populate_systree action to get the name of the systree artifact from the systree_name parameter, when not given one explicitly, or when it is given as auto.

    Acceptance criteria: Set up an ick cluster with a pipeline using the new feature, and check that the project builds.

  • Fix WM to really set env vars: The worker-manager tries, but fails, to set LC_ALL and DEBIAN_FRONTEND for all shell and python actions. Fix this.

    Acceptance criteria: Run a pipeline with shell and python actions that print the whole environment and check the variables are set.

  • Run actions with stdin from /dev/null: Change worker-manager to redirect stdin for all actions to be from /dev/null. This prevents (buggy) programs from blocking, when they try to read from "the terminal".

    Acceptance criteria: Set up an ick instance with my usual projects and pipelines, and check that everything builds.

  • Change the debmirror action to have a default mirror: The default mirror should be deb.debian.org.

    Acceptance criteria: Set up an ick cluster with a project that builds a systree, without specifying a mirror, and check that it works.

  • Remove unnecessary scripts from source: The source tree has some "helper" or "debug" scripts for Lars's use. Remove any obsolete ones.

    Acceptance criteria:

  • Identify worker from token, not path: When a worker requests work, it currently does GET /work/foo, where foo is the name of the worker. This is silly. Change things so the endpoint is just /work and the controller identifies the worker from the access token's aud claim.

    Acceptance criteria: Deploy a new ick cluster with my usual projects, verify that everything builds.

  • Require an explicit where on all actions: Change worker-manager to require an explicit where on each actions and to refuse a new pipeline where that isn't true.

    Acceptance criteria: Unit tests check for this and pass.

  • Add icktool show-latest-log command: This should take a list of project names as arguments, and output the latest build log for each.

    Acceptance criteria: Manual testing.

  • Add icktool delete command: This should take a list of resource paths (/projects/foo) and delete them. Should work for at least projects and pipelines.

    Acceptance criteria: Manual testing.

  • Change icktool make-it-so to accept filenames: Currently it reads from the standard input. Change it to accept number of filenames, and to read each as it currently reads from sdin.

    Acceptance criteria: Manual testing.

Posted
liw
Retrospective meeting (iteration 20)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Add a project build graph to build resource Lars yes
Change controller to trigger whole project Lars yes
Drop "pipeline instance" resource Lars yes
Change icktool to produce dot file for build graph Lars yes
Make release, deploy Lars yes
Refactor liw-ci to use nicer pipeline structrue Lars yes

What went well?

  • All OK.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
liw
Planning meeting (iteration 20)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about changing the controller to run all pipelines for a project, sequentially. At the start of a build, a build graph will be constructed of all the actions that need to be taken for the build to be finished.

Roadmap until ALPHA-6

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Add a project build graph to build resource Lars 4
Change controller to trigger whole project Lars 1
Drop "pipeline instance" resource Lars 1
Change icktool to produce dot file for build graph Lars 4
Make release, deploy Lars 2
Refactor liw-ci to use nicer pipeline structrue Lars 4
Total Lars 16

Task descriptions

  • Add a project build graph to build resource: Add a build graph representation to the build resource, when a build starts. Construct the build graph from all the actions in all the pipelines in the project, sequentially. The build graph will thus be a linear list of actions. Mark each graph node with a state: ready, blocked, building, finished, failed. Add or update unit tests to test construction of build resource and build graph, and scheduling of actions from the build graph.

    Acceptance criteria: Unit tests pass.

  • Change controller to trigger whole project: Change the build trigger endpoint in the controller to apply for the whole project, instead of a pipeline. Change the controller to construct the whole build graph at that time, and to create the build resource. Update automated tests to use new wayt to trigger a build.

    Acceptance criteria: All automated tests pass.

  • Drop "pipeline instance" resource: The "pipeline instance" resource type should now be unnecessary. Drop it.

    Acceptance criteria: All automated tests pass.

  • Change icktool to produce dot file for build graph: Change icktool to get the build graph from a build ressource (for a specific build, or the latest build), and to output a .dot file representation of the build graph. Each node should have a suitable shape and colour. The .dot file should be input to the dot command to produce an SVG graph.

    Acceptance criteria: Manual testing shows a pretty graph is produced.

  • Make release, deploy: Make a release of ick and deploy it to Lars's "ci-prod" cluster.

    Acceptance criteria: Manual testing shows all projects can still be built.

  • Refactor liw-ci to use nicer pipeline structure: Change the project and pipeline specifications in the "liw-ci" git repository to use cleaner pipelines, aimed at being re-useable.

    Acceptance criteria: Manual testing shows all projects can still be built.

Posted
Ick ALPHA-4 released: build and publish .deb packages

Ick is a continuous integration (CI) system. It is starting to be ready for you to use for real. See http://ick.liw.fi/ for more information.

The fourth ALPHA version (0.40) of ick has now been released. Ick now defaults to verifying TLS certificates (all praise Let's Encrypt), code has been cleaned up a bit (especially icktool and worker-manger),l and builds get the BUILD_NUMBER environment variable set.

Thanks to the build number, and changes to the ick2-ansible repository of Ansible playbooks, it's possible to set up an ick instance, with an APT package repository, and have projects build Debian packages, and upload those to the APT repository. The ick2-ansible repository has an example, and liw-ci has an example of an ick project description that does that. This is in production-like use: ick CI builds gets updated whenever that ick master branch changes on the git server.

See NEWS, below, for a more detailed summary of the changes.

Development continues, see the roadmap for what's planned.

NEWS for ick2, a CI server

Version 0.40, released 2018-04-16

  • This is the ALPHA-4 version.

Version 0.39.2, released 2018-04-16

  • Fix build problem.

Version 0.39.1, released 2018-04-16

  • Fix build problem.

Version 0.39, released 2018-04-16

  • Fix bug in worker-manager in how BUILD_NUMBER is set in the environment when running command in container.

Version 0.38, released 2018-04-15

  • Fix bug in worker-manager to set BUILD_NUMBER in environment when running command in container.

Version 0.37, released 2018-04-15

  • icktool trigger now works. There was a bug in how the controller response was printed out.

  • Worker-manager now gives all the child processes it runs the build number as the BUILD_NUMBER environment variable. This lets pipeline actions do things like embed it in Debian package versions for CI builds.

Version 0.36, released 2018-04-08

  • icktool is again mostly functional. Deleting resources (projects, pipelines, etc) hasn't been implemented yet. Neither has handling blobs. These will be added later, when some one needs them.

Version 0.35, released 2018-04-07

  • Controller now knows the URL to the identity provider's authentication endpoint. This is reported via /version, which now doesn't require authentication.

  • Worker-manager and icktool now fetch an access token from the IDP, instead of generating token themselves.

  • icktool has been substantcially rewritten, with much loss of functionality. Lost functionality will be added back as the loss is noticed by users.

Version 0.34, released 2018-04-05

  • icktool --verify-tls now works as intended. worker-manager now has a --verify-tls option. For both programs, the default is "verify". Use --no-verify-tls to turn it off.
Posted
liw
Retrospective meeting (iteration 19)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Add BUILD_NUMBER env vars to builds Lars yes
Set up an ick cluster that CI builds ick Lars yes

What went well?

  • Everything.

What didn't go well?

  • N/A.

What can we improve for the future?

  • N/A.
Posted
liw
Planning meeting (iteration 19)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about making ick do CI builds of itself, including producing and publishing .deb packages of itself. Note, we aim only at CI builds, and not yet release builds. It turns out that release builds are trickier, partly because they'll need to be built from signed git tags, and that old tags should be ignored. The first time a "release build" runs, it should only remember what release tags exist, and not actually do any building.

  • CI builds are more straighforward: they should always happen, when triggered, and the only tricky part is that the version number for the Debian package should be incremented for each build. Otherwise each build produces packages with the same version number, and that doesn't lead to anything good.

Roadmap until ALPHA-6

Current projects

  • N/A.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Add BUILD_NUMBER env vars to builds Lars 1
Set up an ick cluster that CI builds ick Lars 4
Total Lars 5

Task descriptions

  • Add BUILD_NUMBER env vars to builds: Change worker-manager to set the BUILD_NUMBER environment variable for all child-processes it starts. It should get the value from the work resource.

    Acceptance criteria: Manual testing shows it works.

  • Set up an ick cluster that CI builds ick: The cluster should have a project to do CI builds of the ick repository (master branch), such that it mangles the version number to include the build number. Thus, each successful build should result in a new .deb package, with a version number that's strictly larger than the previous one. The project should do uploads to the unstable-ci "release" in the APT repository of the cluster.

    Acceptance criteria: Manual testing shows new .debs get uploaded.

Posted
liw
Retrospective meeting (iteration 18)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Install Qvisqve in ick2-ansible Lars yes
Add IDP URL to controller /version Lars yes
Change client.py to fetch token from IDP Lars yes
Change icktool to use client.py Lars yes
Set up test instance, check that it works Lars yes

What went well?

  • Everything.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
liw
Planning meeting (iteration 18)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about introducing a real IDP component to get rid of having each API client creating its own access tokens.

Roadmap until ALPHA-6

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Install Qvisqve in ick2-ansible Lars 1
Add IDP URL to controller /version Lars 1
Change client.py to fetch token from IDP Lars 1
Change icktool to use client.py Lars 1
Set up test instance, check that it works Lars 1
Total Lars 5

Task descriptions

  • Install Qvisqve in ick2-ansible: Add a role to ick2-ansible.git for installing Qvisqve on a host, and use that role in a playbook. The deployed Qvisqve should allow the user to define at least one pre-configured API client via parameters. The token signing key should of course also be provided by an Ansible variable.

    Acceptance criteria: Manually test that a Qvisqve server can be configured and that it grants tokens to a client.

  • Add IDP URL to controller /version: Add another field to the /version result, similar to artifact_store, but for the URL to the IDP. Call it auth_url. The URL will be provided by the controller configuration file.

    Acceptance criteria: Unit and integration tests check for the IDP URL in the /version result. Tests pass.

  • Change client.py to fetch token from IDP: Change the client.py module to retrieve the IDP URL from the controller, and fetch an access token from the IDP. Change worker-manager to use the new functionality and drop the token generation code.

    Acceptance criteria: Suitable unit tests have been added to the worker manager and they pass.

  • Change icktool to use client.py: icktool currently implements its own version for accessing the APIs and for generating tokens. Replace all of that with client.py instead.

    Acceptance criteria: Manually test that icktool can fetch a token from an IDP.

  • Set up test instance, check that it works: Test the playbook, icktool and worker-manager changes by setting up a fresh test instance, and adding projects to build a systree and to run something in a container using the systree.

    Acceptance criteria: Manually check that the builds pass in the test instance.

Posted
liw
Retrospective meeting (iteration 17)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Change icktool, worker-manager to verify TLS certs Lars yes
Add Let's Encrypt certs to Ansible playbook Lars yes

What went well?

  • Everything went smoothly.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
Notifications in ick: the MVP

Ick will need a way to notify users about build failures. Later on, the same notification mechanism can be used for other things, but let's start with the minimal thing that is useful. Here's a sketch.

  • Notifications will be configured in the project resource in the controller.

  • The controller will trigger a notification service to actually send the notification. Don't want to burden the controller with all the details of how notifications are sent, but the controller needs to trigger a notification. (It will do this by telling a worker to do it, so that we don't need to have the controller do blocking HTTP calls to other services.)

  • Notifications will be triggered only on failures. Later on, this needs to be configurable and include options like a "build now succeeds, when it previously failed", and possibly also "always notify".

  • Later on, ick needs to be able to flexibly notify via a number of channels: email, IRC, Matrix, SMS, RSS/Atom, etc. Probably also proprietary systems like Slack will need to be supported. However, we'll start with just email.

  • The notification emails will always be sent with the same sender information, and this will be configured by the sysadmin. Also, it will always be sent with the same mail server, also configured by the sysadmin. Probably using /usr/sbin/sendmail rather than SMTP. Aiming for minimal necessary configuration to start with.

  • The template for the emails will be hard-coded in the source code (for now).

The mails will containt the following information:

  • name of project that was built
  • build number
  • build id
  • log id (for use with icktool)
  • last 1000 lines of build log
  • exit code of build

Later on, this will need to extended, but the above should suffice for now.

liw
Planning meeting (iteration 17)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is all about making use of Let's Encrypt for TLS certificates.

Roadmap

Current projects

  • N/An

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Change icktool, worker-manager to verify TLS certs Lars 1
Add Let's Encrypt certs to Ansible playbook Lars 4
Total Lars 5

Task descriptions

  • Change icktool, worker-manager to verify TLS certs: Currently all places in ick where HTTP queries are made, checking of TLS certificates is turned off. Specifically, icktool and worker-manager. Change both so that it's optional: enabled by default, but possible to disable via a configuration option. Enabled is the sensible default, but to support those who for whatever reason can't get a CA-signed certificate, it should be possible to turn it off.

    Acceptance criteria: Running icktool and worker-manager with TLS certificate checking turned off means nothing really works. Test this manually.

  • Add Let's Encrypt certs to Ansible playbook: Change the Ansible playbook(s) in the ick2-ansible repository to automatically set up a TLS certificate by LE for the controller and artifact store.

    Acceptance criteria: Deploy a new ick instance with separate controller and artifact store hosts. Run icktool, with certificate checking enabled, to add a project, and to trigger its build. Check manually that it gets built OK.

Posted
Ick ALPHA-3 released: concurrent building, for real

Ick is a continuous integration (CI) system. It is starting to be ready for you to use for real. See http://ick.liw.fi/ for more information.

The third ALPHA version (0.32) of ick has now been released. It fixes building with multiple workers, and expecially the artifact store, which didn't work over a real network connection. See NEWS for a summary of the changes. Download it from the website.

Development continues, see the roadmap for what's planned.

Posted
liw
Retrospective meeting (iteration 16)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Write artifact store Lars yes

What went well?

  • Mostly everything.

What didn't go well?

  • Tried to do too much in one step.
  • The apifw library is a bit of a pain.

What can we improve for the future?

  • Do things in smaller steps.
  • Fix underlying issues before building more on rotten dependencies.
Posted
liw
Planning meeting (iteration 16)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about fixing the blog service and renaming it to the artifact store.

Roadmap for this iteration

Current projects

  • N/A.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Write artifact store Lars 8h
Total Lars 8h

Task descriptions

  • Write artifact store: The current blob service doesn't work well across actual networks (mostly works over localhost). Also, the name is not great. Write a replacement, called artifact store, which has the same API, but can handle large blobs.

    Acceptance criteria: The artifact store is a drop-in replacement for the blob service. A client, running on a different host, can upload a large blob (at least 10 GiB) and download it again. The downloaded blob must be bitwise identical to the uploaded one. Transfer speeds must be at least 90% of wire speed.

Posted
Thoughts on concurrency in ick

I met a friend yesterday, and discussed ick at length. The main topic we covered was how to let ick maximise concurrency during building and optimise how soon the user gets notified of any problems in the build.

As an example, imagine a project, written in a language that gets compiled to native code, has some unit tests, and that ick builds Debian packages for it. When the user pushes changes to git, ick will do roughly the following things, in order:

  • fetch the source (or updates to it)
  • build the code
  • run unit tests
  • run any integration tests that can be run from the build tree without installing the software
  • create a Debian source package (.dsc)
  • from the source package, build the binary package (.deb)
  • upload the source and binary packages to an APT repository
  • deploy the uploaded binary packages to a test server
  • test the installed software on the test server, possibly heavily (e.g., benchmarks, load tests)
  • deploy the uploaded binary packages to a production server

What the user wants is to be notified of any problem as soon as possible. For example, if there is an error during building, they should hear about it as soon as it happens.

Ick lets the user specify projects, each of which has one or more pipelines, each of which has one or more actions. When all the actions of all the pipelines are executed in order, the project is built and the result it something the user wants.

For simple cases, this is fine, and there's usually not much room for concurrency. If the worker has multiple CPUs, the action that runs the project's build system ("make") can exploit that by running multiple compilers concurrently ("make -j128"). That is however not of interest to ick.

For ick, what matters is building different parts of the project on different workers at the same time, and not waiting for things unnecessarily. For the simple example above, this is simple enough: execute the steps in sequence, and tell the user after each step if it failed.

A more interesting case is when the program needs to be built on more than one CPU architecture. For example, amd64 (fast) and armhf (slow). Even more intereting is when the build takes a long time. Thus, we extend the example above with the following changes:

  • the project source code is large (think LibreOffice large)
  • it takes hours to build even of a fast machine, days on a slow machine
  • feedback should still come as soon as possible

The simplistic way would be to do the build, unit test, and package building steps sequentially, once per CPU type, starting with the fastest first. However, this would slow things down too much: the slow workers won't even start and if there's several slow architectures, the last one might take many days, even weeks. Not acceptable.

The solution is to start all builds at once. Basically, each build will effectively "fork" the pipeline for each type of build target, and excute actions within the fork sequentially.

Some things will need to wait for all builds from forks to finish. Forac example, and automatic notification of a new build should probably not be sent out until the software is built for all types of targets.

The tricky part, then, is to give the user a way to express this in way that's easy to get right and difficult to get wrong, and that still allows ick to things with as much concurrency as possible. Here's a suggestion.

A pipeline's list of actions implies a dependency order: each action implicitly depends on the previous action in the same list. Depending means that the depended-on action must finish successfully before the depepending action is started.

A "gate" action implies that all previous actions for the same build must finish successfully before the gate is passed. All gates must be explicit: there is no implicit gate at the end of a pipeline. This is so that a fast worker can continue opportunistically with the next pipeline, without having to wait (for days!) for a slow architecture build to finish.

A pipeline, or each of its actions, can specify that it should be run once per build, once per each type of target, or only for a specific type of target. This is done by using tags. Each worker will have a set of tags that describe it, and ick will automatically detect types of workers from the tags.

As a concrete example, a project and pipeline and workers might be specified as follows. We'll have three workers, two for amd64, and one for armel.

workers:
  - worker: cheetah
    tags:
      - amd64-arch
  - worker: cheetch2
    tags:
      - amd64-arch
  - worker: turtle
    tags:
      - armhf-arch

We'll have one project, which we specify should be built for every architecure. More specifically, ick will automatically run multiple forks of the build, one fork for each set of available workers that matches the tag wildcards. In our example workers, the two amd64 workers are considered equal, and ick will choose randomly one of them. With the example workers, ick will start two forks for this project when it starts building.

projects:
  - project: gcc
    pipelines:
      - clone
      - configure
      - build
      - check
      - dsc
      - deb
      - dput
      - announce
    target-tags:
      - *-arch

Here are simplistic versions of the pipelines. The actual build actions are uninteresting, we concentrate on the way concurrencty possibilities are expressed.

pipelines:
  - pipeline: clone
    actions:
      - shell: git clone git://git.example.com/gcc
    build: once

  - pipeline: configure
    actions:
      - shell: ./configure
    build: each

  - pipeline: build
    actions:
      - shell: make
    build: each

  - pipeline: check
    actions:
      - shell: make check
    build: each

  - pipeline: dsc
    actions:
      - shell: dpkg-buildpackage -S
    build: each

  - pipeline: deb
    actions:
      - shell: dpkg-buildpacakge -b
    build: each

  - pipeline: dput
    actions:
      - shell: dput *.changes
    build: each

  - pipeline: announce
    actions:
      - action: gate
      - shell: echo GCC HAS BEEN UPDATED AND ALL IS WELL
    build: once

The "build" field in a pipeline should have one of the following values:

  • once – the pipeline should be executed once for all forks; if a fast worker executes the pipeline successfully, and slower one arrives to it, the controller will just skip the pipeline for the slow worker

  • each – the pipeline should be executed by each fork

We might want to allow build for each action in a pipeline as well.

A gate action means that the controller will wait for all preceding actions to finish successfully before it gives any further actions to any workers to execute.

Ick will automatically archive the workspace in the blob service after each action, and each action will start by fetching the archived workspace from the previous action, whether from the same fork or not. (Obviously, the archiving and fetching will only be done if actions are given to different workers.)

Conceptually, the ick controller will start a build by building a directed acyclic graph of actions that need to be executed, and label each node in the graph with the type of worker needed to execute it. Something like the graph below.

When the build is actually executing, the controller will start keeping track of the build at the "start" node, and when a worker wants something to do, the controller will pick the next node in the graph suitable for that worker. If it's not the worker that did the previous action, the controller will automatically tell the new worker to fetch the correct workspace blob from the blob service.

I believe this will result in a nice amount of concurrency in builds orchestrated by ick. Also, the build graph will make for a nice visualisation in the UI, each node turning into green when the action is finished, or something.

Ick ALPHA-2 released: concurrent building

Ick is a continuous integration (CI) system. It is not yet ready for you to use for real. See http://ick.liw.fi/ for more information.

The second ALPHA version (0.28) of ick has now been released. It allows an installation with multiple workers, and building different projects at the same time. Download it from the website.

Development continues, see the roadmap for what's planned.

Posted
liw
Retrospective meeting (iteration 15)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Try concurrent building with two workers Lars yes

What went well?

  • Everything.

What didn't go well?

  • Nothing.

What can we improve for the future?

  • We're perfect, except for too few kittens.
Posted
liw
Planning meeting (iteration 15)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about actually doing concurrent building.

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Try concurrent building with two workers Lars 4h
Total liw

Task descriptions

  • Try concurrent building with two workers: Set up an ick cluster with at least two workers and at least two projects. Trigger both projects. Do both build at the same time? Do both build correctly?

    Acceptance criteria: Manual testing shows things work OK. Any issues either get fixed or reported.

Posted
liw
Retrospective meeting (iteration 14)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Fix icktool to use blob service URL Lars yes
Write a yarn scenario for concurrent builds Lars yes

What went well?

  • Life (or work) got in the way of hobbies. Otherwise, all OK.

What didn't go well?

  • N/A.

What can we improve for the future?

  • Win the lottery, retire, hack on ick.
Posted
liw
Planning meeting (iteration 14)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about making progress on building on multiple workers concurrently.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Fix icktool to use blob service URL Lars 1
Write a yarn scenario for concurrent builds Lars 3
Total Lars 4

Task descriptions

  • Fix icktool to use blob service URL: Currently icktool assumes it can get /blobs on the controller. It no longer can. But it can get the blob service URL from the controller. Fix it to get that URL and to get/upload blobs using that URL.

    Acceptance criteria: Manual testing shows this works.

  • Write a yarn scenario for concurrent builds: Add a yarn scenerio that tests that the controller can handle concurrent builds using several workers.

    Acceptance criteria: Scenario is written and all new steps are implemented. The scenario does not need to pass.

Posted
liw
Retrospective meeting (iteration 13)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done
Package each component separately Lars yes

I decided to change the task. Instead of having a separate .deb for each component, plus an ick-common.deb with shared parts, I went for one .deb, which doesn't start any services. Ansible will then configure and enable the services. This achieves the same result, but is easier to mangage.

What went well?

  • Everything.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
liw
Planning meeting (iteration 13)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about splitting up the ick Debian packaging so its various components can be installed separately.

Roadmap for this iteration

  • N/A

Current projects

  • N/A

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Package each component separately Lars 4
Total Lars 4

Task descriptions

  • Package each component separately: Currently the ick Debian packaging produces a single package, ick2.deb. Re-do the packaging so that there is a separate package for each component: ick-controller.deb, ick-blob-service.deb, ick-worker-manager.deb, icktool.deb. Also, an ick-common.deb with the common parts, which is mainly the ick2 Python package. Each of the component packages should be installable separately.

    Also update the Ansible playbook in ick2-ansible.git so that the packages, except the worker manager, can be installed on the same host, and then install additional hosts as worker managers.

    Acceptance criteria: Manually test the Ansible playbook by setting up an ick setup with a controller host and two workers. It's enough if the workers can be set up and register themselves to the controller, and can run one build at a time. It's not necessary to run concurrent builds.

Posted
liw
Retrospective meeting (iteration 12)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what Who Done?
Have workspace as /workspace in containers Lars yes
Sensible default for debootstrap mirror Lars yes
Debootstrap suite should be parameter Lars yes
Worker manager startup race, crash restart Lars yes
Mount /proc into chroot actions Lars yes
Add env vars to exec actions Lars yes
Set up a global Gitano hook to notify CI Lars yes

What went well?

  • Everything.

What didn't go well?

  • Lars didn't wait for others to help.

What can we improve for the future?

  • Get others to help.
Posted
Setting up a global post-receive hook in Gitano to trigger ick

Gitano is a Git server, which happens to be the one that hosts ick's own source code. Gitano supports post-receive hooks: bits of code that get run after a user does git push. This can be used to trigger a CI build, whenever a repository gets modified.

Here is an example. It's a Lua script. To use it, clone the Gitano admin repository (gitano-admin), and create the file global-hooks/post-receive.lua in the master branch. Put the code below into that file. Commit and push to the server. After this, any git repository on that Gitano instance that has the Gitano per-repository configuratution variable ci.notify_url causes the configured URL to be fetched (using GET), which in this case triggers ick to build the relevant project.

To set the configuration variable:

url=https://ick.example.com
project=ick.liw.fi
pipeline=build_ikiwiki_site
ssh git@gitano config ick.liw.fi set ci.notify_url \
    "$url/projects/$project/pipelines/$pipeline/+trigger"

(Split into multiple statements so it fits in 80 columns.)

The post-receive Lua hook script:

--
-- This is a global hook for a Gitano instance.
-- It should be installed as gitano-admin::global-hooks/post-receive.lua
--
-- It's meant as a simple CI trigger


local project_hook, repo, updates = ...

local do_notify = false

for ref in pairs(updates) do
   if not ref:match("^refs/gitano/") then
      do_notify = true
   end
end

local ci_notify_url = repo:get_config("ci.notify_url")

if ci_notify_url then
   log.state("Notifying CI at: " .. ci_notify_url)
   local code, msg, headers, content = fetch(ci_notify_url)
   if code ~= "200" then
      log.state("CI notification failed: " .. tostring(code))
   else
      log.state("CI notification succeeded.")
   end
   for line in (content or ""):gmatch("([^\r\n]*)\r?\n") do
      log.state("CI: " .. line)
   end
end

-- Chain to the project hook if there is one
return project_hook(repo, updates)
liw
Planning meeting (iteration 12)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The goal of this iteration is to fix bugs and warts in ick.

Roadmap for this iteration

  • None.

Current projects

  • None

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what Who estimate(h)
Have workspace as /workspace in containers Lars 0..1
Sensible default for debootstrap mirror Lars 0..1
Debootstrap suite should be parameter Lars 0..1
Worker manager startup race, crash restart Lars 0..1
Mount /proc into chroot actions Lars 0..1
Add env vars to exec actions Lars 0..1
Set up a global Gitano hook to notify CI Lars 0..1
Total Lars 0..7

Task descriptions

  • Have workspace as /workspace in containers: When running in a container, bind-mount the workspace (/var/lib/ick/workspace) into the container as /workspace, where it's easy to refer to.

    Acceptance criteria: Manual testing shows this works.

  • Sensible default for debootstrap mirror: There is currently no default mirror for the debootstrap action. There should be. The deb.debian.org CDN based mirror sees like a sensible default.

    Acceptance criteria: Manual testing shows this works.

  • Debootstrap suite should be parameter: Currently the debootstrap action requires the suite argument hardcoded into the pipeline step. This is awkward. Change worker manager so it accepts the value auto instead of a suite name, and looks up the suite from the debian_codename pipeline paramaeter.

    Acceptance criteria: Manual testing shows this works.

  • Worker manager startup race, crash restart: When worker manager starts, it immediately talks to the controller and if the controller is not yet up, this fails. That makes the worker manager crash. Fix this by having systemd restart the worker manager upon crash.

    Acceptance criteria: Manual testing shows this works.

  • Mount /proc into chroot actions: When running an action with where: chroot, sometimes the thing that gets run requires /proc or /sys to be mounted Change worker manager to mount those into the workspace unconditionally, creating the mount point directories if need be.

    Acceptance criteria: Manual testing shows this works.

  • Add env vars to exec actions: When running a shell or python action, give it the LC_ALL and DEBIAN_FRONTEND evironment variables. This will make it easier to install packages.

    Acceptance criteria: Manual testing shows this works.

  • Set up a global Gitano hook to notify CI: Develop a sample global Gitano post-receive hook to alert CI of a change. The hook should be configurable: the repository config should have a variable ci_notify_url and the hook should fetch that URL to notify CI. If the variable is not set, the hook should do nothing. Discuss the design with Gitano developers and change the spec if need be.

    Acceptance criteria: git.liw.fi uses the new hook and manual testing shows it works.

Posted
Consensus decision on preferred English variant

After a short discussion on IRC, the following was decided by consensus:

The preferred variant of English for the ick project is British English. It's a weak preference, and contributions in American English, or "international English" are accepted.

Pondering ALPHA-2 architecture

I've been thinking about what the ALPHA-2 version of ick should be like. One thing that's fairly clear to me is that ick should, very soon, be able to build its own .deb packages. I'm likely to want to make that a goal for ALPHA-2.

A critical concern about this is handling of secrets. If nothing else, if .deb packages get uploaded, they should be PGP signed. The signing requires a private key, which must be kept secret. Ick gets what is essentially arbitrary, untrusted code and runs that, and it would be unfortunate if the private key gets stolen by the code that gets built.

Thus a design goal for ick needs to be that there are no secrets on the worker host at all, except the secret that the worker-manager needs to authenticate itself to the controller and other ick components. That secret should not be useful for anything else. This way, any repercussions from the secret leaking is minor and easily controllable.

Instead, we introduce the concept of "trusted worker", which never runs user-supplied code, and can hold secrets needed for external access, such as ssh and PGP private keys.

As an architecture, it might look like this:

The trusted worker has an ssh key that it uses to fetch source code from the git server. It packages up the source code as a blob, and uploads the blob to the blob service, as a "workspace".

The normal, untrusted worker gets the workspace from the blog service and, after building, uploades any build artifacts (.deb files) back to the build service.

After that, the trusted worker downloads the artifacts from the blob service, PGP signs them, and uploads them to an APT repository.

Altenatively, the developer downloads the .debs from the blob service, does the signing themselves with their personal PGP key, and uploads them to the APT repository. This is what needs to be done if ick is building packages that are to be uploaded to Debian proper, since those uploads need to be signed by a key Debian trusts, and that means it can't be a key put on some shady service not under direct control by the developer.

In this design, only the trusted worker has ssh and PGP keys that need to known and trusted by external parties.

Posted
liw
Retrospective meeting (iteration 11)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Check blob uploads, downloads n/a Lars yes
Write icktool manual n/a Lars yes
Make ALPHA-1 release n/a Lars yes
Draft public announcement n/a Lars yes
Gather feedback of announcement n/a Lars yes
Post announcement n/a Lars yes

What went well?

  • Everything.

What didn't go well?

  • Nothing.

What can we improve for the future?

  • Be awesome.
Posted
Ick ALPHA-1 released

TL;DR: Ick is a continuous integration or CI system. See http://ick.liw.fi/ for more information.

More verbose version follows.

First public version released

The world may not need yet another continuous integration system (CI), but I do. I've been unsatisfied with the ones I've tried or looked at. More importantly, I am interested in a few things that are more powerful than what I've ever even heard of. So I've started writing my own.

My new personal hobby project is called ick. It is a CI system, which means it can run automated steps for building and testing software. The home page is at http://ick.liw.fi/, and the download page has links to the source code and .deb packages and an Ansible playbook for installing it.

I have now made the first publicly advertised release, dubbed ALPHA-1, version number 0.23. It is of alpha quality, and that means it doesn't have all the intended features and if any of the features it does have work, you should consider yourself lucky.

Invitation to contribute

Ick has so far been my personal project. I am hoping to make it more than that, and invite contributions. See the governance page for the constitution, the getting started page for tips on how to start contributing, and the contact page for how to get in touch.

Architecture

Ick has an architecture consisting of several components that communicate over HTTPS using RESTful APIs and JSON for structured data. See the architecture page for details.

Manifesto

Continuous integration (CI) is a powerful tool for software development. It should not be tedious, fragile, or annoying. It should be quick and simple to set up, and work quietly in the background unless there's a problem in the code being built and tested.

A CI system should be simple, easy, clear, clean, scalable, fast, comprehensible, transparent, reliable, and boost your productivity to get things done. It should not be a lot of effort to set up, require a lot of hardware just for the CI, need frequent attention for it to keep working, and developers should never have to wonder why something isn't working.

A CI system should be flexible to suit your build and test needs. It should support multiple types of workers, as far as CPU architecture and operating system version are concerned.

Also, like all software, CI should be fully and completely free software and your instance should be under your control.

(Ick is little of this yet, but it will try to become all of it. In the best possible taste.)

Dreams of the future

In the long run, I would ick to have features like ones described below. It may take a while to get all of them implemented.

  • A build may be triggered by a variety of events. Time is an obvious event, as is source code repository for the project changing. More powerfully, any build dependency changing, regardless of whether the dependency comes from another project built by ick, or a package from, say, Debian: ick should keep track of all the packages that get installed into the build environment of a project, and if any of their versions change, it should trigger the project build and tests again.

  • Ick should support building in (or against) any reasonable target, including any Linux distribution, any free operating system, and any non-free operating system that isn't brain-dead.

  • Ick should manage the build environment itself, and be able to do builds that are isolated from the build host or the network. This partially works: one can ask ick to build a container and run a build in the container. The container is implemented using systemd-nspawn. This can be improved upon, however. (If you think Docker is the only way to go, please contribute support for that.)

  • Ick should support any workers that it can control over ssh or a serial port or other such neutral communication channel, without having to install an agent of any kind on them. Ick won't assume that it can have, say, a full Java run time, so that the worker can be, say, a micro controller.

  • Ick should be able to effortlessly handle very large numbers of projects. I'm thinking here that it should be able to keep up with building everything in Debian, whenever a new Debian source package is uploaded. (Obviously whether that is feasible depends on whether there are enough resources to actually build things, but ick itself should not be the bottleneck.)

  • Ick should optionally provision workers as needed. If all workers of a certain type are busy, and ick's been configured to allow using more resources, it should do so. This seems like it would be easy to do with virtual machines, containers, cloud providers, etc.

  • Ick should be flexible in how it can notify interested parties, particularly about failures. It should allow an interested party to ask to be notified over IRC, Matrix, Mastodon, Twitter, email, SMS, or even by a phone call and speech syntethiser. "Hello, interested party. It is 04:00 and you wanted to be told when the hello package has been built for RISC-V."

Please give feedback

If you try ick, or even if you've just read this far, please share your thoughts on it. See the contact page for where to send it. Public feedback is preferred over private, but if you prefer private, that's OK too.

Posted
liw
Planning meeting (iteration 11)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration aims to finish the ALPHA-1 development cycle and culminates in its release.

Roadmap for this iteration

Current projects

  • None

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Check blob uploads, downloads n/a Lars 0..2
Write icktool manual n/a Lars 0..1
Make ALPHA-1 release n/a Lars 0
Draft public announcement n/a Lars 0..1
Gather feedback of announcement n/a Lars 0..1
Post announcement n/a Lars 0
Total Lars 0..5

Task descriptions

  • Check blob uploads, downloads: Set up an ick instance, build a systree, make sure it works. This is to test recent changes that do uploads and downloads via shell pipelines instead of keeping the whole blob in memory at once.

    Acceptance criteria: Build using systree blobs works.

  • Write icktool manual: Write a manual for icktool on the ick.liw.fi website. Enough to allow somone with access to an ick instance to set up a project and build it.

    Acceptance criteria: Lars is happy with it.

  • Make ALPHA-1 release: Tag and build the release that will be announced as ALPHA-1.

    Acceptance criteria: Debian packages are available on code.liw.fi.

  • Draft public announcement: Draft the announcement of ALPHA-1 and circulate it among friends for feedback.

    Acceptance criteria: URL to draft is shared with #ick, and some friends.

  • Gather feedback of announcement: If there's feedback on the announcement draft, gather it and update the draft as needed.

    Acceptance criteria: Lars is happy with the new announcemnt version.

  • Post announcement: Post the ALPHA-1 announcement to suitable places.

    Acceptance criteria: It's posted to Lars's blog and in the ick.liw.fi news section, and possibly elsewhere.

Posted
liw
Retrospective meeting (iteration 10)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Configure StrictHostKeyChecking to no Lars yes
Rename pipeline name field to pipeline Lars yes
Fix build numbering Lars yes
Change icktool to show latest/current log Lars yes
Change icktool to show build ok/fail Lars yes
Write HOWTO for deploying ick Lars yes
Use containers in a real build Lars yes

What went well?

  • Got everything done and within estimated time.

What didn't go well?

  • Nobody volunteered to help set up an ick of their own.

What can we improve for the future?

  • Dunno.
Posted
liw
Planning meeting (iteration 10)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about bug fixing and enabling others to deploy ick for themselves.

Roadmap for this iteration

Current projects

  • None.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Configure StrictHostKeyChecking to no Lars 0..1
Rename pipeline name field to pipeline Lars 0..1
Fix build numbering Lars 0..2
Change icktool to show latest/current log Lars 1..2
Change icktool to show build ok/fail Lars 0..1
Write HOWTO for deploying ick Lars 1..3
Use containers in a real build Lars 2..3
Total Lars 4..11

Task descriptions

  • Configure StrictHostKeyChecking to no: Lars's CI instances mostly build websites and those need to rsync the results over ssh to a web server. For this to work smoothly on a new instance, ssh should accept the host key of the web server the first time. This is perhaps not a great security decision, but good enough for now.

    Acceptance criteria: Lars can set up a new ick instance with Ansible and have it build ick.liw.fi without any manual changes.

  • Rename pipeline name field to pipeline: Currently it's called "name" and that's inconsistent and inconsistency is annoying. It's early, there's only one user, so there's time to change this.

    Acceptance criteria: Lars has changed all his personal pipelines to use the new field name and it works.

  • Fix build numbering: Currently all builds for all projects number builds sequentially and this leads to clashes. Change build numbering to be foo/n, where foo is the project name, and n is the build number for that project.

    Acceptance criteria: Yarns have been changed to expect the new build numbering and code has been changed to the yarns pass.

  • Change icktool to show latest/current log: Currently, the user has to manually specify which log to show. When the log number changes every run, this is an unnecessary bit of friction. Change icktool to show the latest build log for a project.

    Acceptance criteria: Manual testing shows this works.

  • Change icktool to show build ok/fail: The "icktool status" command does not currently show if the latest build for a project/pipeline combo was successful or failed. The user has to check that manually by viewing the log. Change icktool to show the ok/fail status.

    Acceptance crieria: Manual testing shows this works.

  • Write HOWTO for deploying ick: Write a page on ick.liw.fi to document how one would go about setting up an ick instance.

    Acceptance criteria: If there's a volunteer, they can set up their own ick instance by following the document. If there's no volunteer, Lars is happy with it.

  • Use containers in a real build: Containers may or may not work, they've not been used in anger yet. Set up a project and pipeline to run unit tests for some smal project in a container, installing any build dependencies into the container.

    Acceptance criteria: Manual testing shows this works.

Posted
Tentatively ALPHA-1 feature complete

As of the 0.22 release, ick2 should be feature complete for the ALPHA-1 release. This means all the features planned for the release should be there, but there may still be bugs or other things that need to be fixed.

Posted
liw
Retrospective meeting (iteration 9)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Implement "where: chroot" in actions Lars yes
Add an archive action Lars yes
Set up systree for a pipeline Lars yes
Implement "where: container" Lars yes
Make a relase Lars yes
Upgrade production instance Lars mostly

What went well?

  • Significant new functionality was added.

  • Adopted a logo and a constitution for the project.

What didn't go well?

  • Last minute mysterious problems with actually using new functionality. May have been deployment issue.

  • Had to back off from already implemented changes, undo those, and rewrite some functionality. Specifically, I had originally implemented a way for the controller to tell the worker manager to empty the workspace as a flag in the work resource. This made things needlessly complicated, when adding more functionality (populating a systree). So I ripped that off and added a "create workspace" step explicitly, not as a flag. Better architecure and implementation this way.

What can we improve for the future?

  • Test releases better. Automatically and manually.
Posted
Ick logo

I've decided to use ick-logo.svg as the ick project logo, unless there's objections.

The font is from the fonts-femkeklaver package in Debian.

Adopting constitution 1.0; first vote and decisions

In order to bootstrap the constitution into effect, the following happens now:

  • I started the Ick project. I've written all the code. It is entirely and only my project at this moment. I am the supreme leader and dictator of the project.

  • Using my dictatorial powers, I declare unilaterally that version 1.0 of the constitution (see governance) is now in effect. I name myself as the the voting member and project secretary. From here-on, the constitution applies and any further changes will be made following rules set in the constitution. As a consequence, I am no longer the dictator. Down with the tyrant! Power to the people!

  • As the first formal decision, I as secretary have held a vote by asking myself what the voting procedure will be. The result is that as long as there are no other voting members, I just ask myself. When we have a few other voting members, we'll hold votes on IRC or email. If the project grows larger, we'll arrange something more secure, but it's premature to do that now.

  • As the first consensus decision I declare that the project will be using git.liw.fi as the git server, and use ikiwiki as the static site generator to create the ick.liw.fi website.

liw
Planning meeting (iteration 9)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about doing builds in containers. See the design for more information.

Roadmap for this iteration

Current projects

  • None.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Implement "where: chroot" in actions Lars 1h
Add an archive action Lars 1h
Set up systree for a pipeline Lars 2h
Implement "where: container" Lars 2h
Make a relase Lars 1h
Upgrade production instance Lars 4h
Total Lars 11h

Task descriptions

  • Implement "where: chroot" in actions: Allow pipeline actions to specify that the shell or python code should be run in a chroot, using the workspace as the root directory.

    Acceptance criteria: Manually test a pipeline that runs debootstrap and then an ls command in a chroot in the workspace, and it outputs the right thing.

  • Add an archive action: Add to worker-manager a built-in action to archive the workspace into a tarball, and upload it to the blob service. Take the blob name from the systree_name parameter.

    Acceptance criteria: Manually test that a pipeline with the archive action uploads a blob with the given name, and that the blob looks like it has the right content.

  • Set up systree for a pipeline: Change worker-manager to download the blob named in the pipeline systree field, unpack that into /var/lib/ick/systree.

    Acceptance criteria: A manual test indicates this works.

  • Implement "where: container": Change worker-manager to allow a shell or python action to be run in a container, if there is a field where: container in the action. The systree should already be created in its directory. Bind mount the workspace directory. Run bash or python3 in the container.

    Acceptance criteria: A manual test indicates this works.

  • Make a relase: Make a relase of Ick. Build .deb packages.

    Acceptance criteria: Packages are available on code.liw.fi.

  • Upgrade production instance: Upgrade the Ick2 server that builds the ick.liw.fi website to the new release. Then change the projects and pipelines to use containers to build using containers.

    Acceptance criteria: A manual test indicates this works.

Posted
Design for building systrees and using them for containers

I've written before that I want Ick to do builds in an environment isolated both from the host and, optionally, from the network.

  • Isolation from host makes it easier to install any build dependencies needed to build a project. They can be installed inside a container. Installing things directly on the build host (worker) is risky, and may result in a non-functional build host. Also, if each project can install whatever it wants on a build host, different projects may keep altering the build environment in ways that make other projects fail to build. They will certainly not built reproducibly, if there's no guarantee about what's in the build environment.

  • Isolation from the network means that builds can't depend on things outside the build environment. This is good for the safety and security of the build. (Some builds inherently need to access the network, of course, so that needs to be possible.)

I've decided to use containers, as provided by systemd-nspawn, for the isolation, at least for now. This is not set in stone, and can be re-evaluated after ALPHA-1 is out. nspawn acts more or less like the chroot command, except it provides better isolation. It's a very easy interface to running thing in isolation.

I will add the concept of systree (system tree) to Ick. A systree is a directory tree in which all the software (operating system, applications, libraries, etc) that are needed for a build are installed. This directory will form the root directory inside the container.

In addition, the container will bind-mount a workspace directory from the host system, as /workspace inside the container. The workspace starts out empty, but the pipeline may fill it with stuff. Typically it will contain the source code of a project, but it can be anything. In this case, it is where the systree gets built.

Ick will allow a pipeline action to be executed in one of three ways: on the build host directly (where: host), in a chroot using the workspace (where: chroot), in a container (where: container). In the latter case Ick will construct the systree, create an empty directory as the workspace, and execute the action inside a container using the systree as the root directory, and the workspace as /workspace.

The systree for a container is constructed by running debootstrap on the host, to fill the workspace, and then archiving the workspace. The archive will be stored somewhere else. I've added the blob service component for this. When it's needed, Ick (worker-manager) will get it from the blob service, and unpack it into the systree directory. Voila!

The archive can, of course, be re-used any number of times, on any number of worker hosts.

I will further make Ick allow the user to specify how the systrees get constructed. The user will specify a project to build a systree:

project: systree_stretch
parameters:
  debian_codename: stretch
  packages:
    - git
    - ikiwiki
    - rsync
    - python3
  systree_name: stretch-ikiwiki
pipelines:
  - build_systree

In the above example, the project defines three parameters (debian_codename, packages, and systree_name), and a pipeline. The pipeline looks like this:

name: build_systree
parameters:
  - debian_codename
  - packages
  - systree_name
actions:
  - debootstrap: auto
    where: host
  - python:
      import subprocess
      packages = params['packages']
      subprocess.check_call(['apt', 'install', '-y'] + packages)
    where: chroot
  - archive: /workspace
    where: host

This pipeline takes the same three parameters, and runs three actions. The debootstrap action is run on the host, and uses the debootstrap program to install Debian into a directory. The action will instruct the program to install into the workspace directory. The auto parameter means the action will get the name of the Debian release from the debian_codename parameter.

The python action then installs additional packages into the workspace. It runs in a chroot, not container, as there isn't a systree yet in which to run. Running in a chroot means apt will install packages in the chroot, not the build host.

The archive action creates a tarball of the workspace, and uploads it to the blob service. The action takes the name of the blob from the systree_name parameter.

After this, we have a systree blob in the blob service. We next use it to run a build.

project: ick.liw.fi
parameters:
  git_url: git://git.liw.fi/ick.liw.fi.git
  git_ref: master
  rsync_target: www-data@ick.liw.fi:/srv/http/ick.liw.fi
pipelines:
  - ikiwiki_website

This example again defines some parameters, and then specifies a pipeline. There's nothing about systrees or containers here yet. That goes into the pipeline.

name: ikiwiki_website
parameters:
  - git_url
  - git_ref
  - rsync_target
systree: stretch-ikiwiki
actions:
  - shell: |
      url="$(params | jq -r .git_url)"
      ref="$(params | jq -r .git_ref)"
      tgt="$(params | jq -r .rsync_target)"
      git clone "$url" -b "$ref" src
      mkdir html
      cd src
      ikiwiki --setup ikiwiki.setup --verbose
      cd ../html
      rsync -av --delete . "$tgt/."
    where: container

This specifies that the stretch-ikiwiki blob is to be retrieved from the blob service, and unpacked as a systree. Then the shell code is to be run inside a container using the unpacked systree.

Voila! We have a way to construct systrees and use them to run other things in containers.

Compared to my current CI system, this means I don't need to keep installing build dependecies of my various personal projects to the CI build hosts, which is a small relief.

liw
Retrospective meeting (iteration 8)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Document how parameters work, are implemented Lars yes
Implement project, pipeline parameters Lars yes
Make a release Lars yes
Convert Lars's pipelines into one Lars yes

What went well?

  • Got everything done.

What didn't go well?

  • Process-wise, everything went OK.
  • Code quality-wise, looks like we may need a test suite for worker-manager.

What can we improve for the future?

  • Start getting more people involved.
Posted
Parameterised pipelines

The problem

Currently, Ick has a very simple model of projects and pipelines. Pipelines are defined independently of projects, and project just list, by name, which pipelines they consist of. A pipeline is a sequence of actions, where an action is a snippet of shell or Python code. The snippets do not get any parameters. I have currently have two projects, which both build a static website with ikiwiki. Both projects have nearly identical pipelines (expressed here as YAML, but equivalent to JSON):

name: build_static_site
actions:
  - python: |
      git_url = 'git://git.liw.fi/ick.liw.fi'
      rsync_target = 'ickliwfi@pieni.net:/srv/http/ick.liw.fi'
      import os, sys, cliapp
      def R(*args, **kwargs):
        kwargs['stdout'] = kwargs['stderr'] = None
        cliapp.runcmd(*args, **kwargs)
      R(['git', 'clone', git_url, 'src'])
      R(['ql-ikiwiki-publish', 'src', rsync_target])

The other pipeline is otherwise identical, but it defines differerent git_url and rsync_target values.

This code duplication is silly and I want to fix this.

Possible approaches

Code duplication between pipelines can be addressed in various ways. Here's a short summary of the ones I have considered.

  • Use jinja2 or similar templating for the code snippet. The project would define some variables, which get interpolated into the code snippet in some way when it gets run. Jinja2 is a well-regarded Python templating library that could be used.

    Pro: simple; straightforward; not dependent on the programming language of the snippet.

    Con: not well suited for non-simple values, e.g., lists and structs; snippets need to be written carefully to add appropriate quoting or escaping so that, for example, template interpolation a shell snippet does not in-advertantly introduce syntax or semantic errors

  • Use a generator utility to create un-parameterised pipelines. A tool reads a language built on top of un-parameterised pipelines and projects and generates pipelines that embed the expanded parameters.

    Pro: this moves the parameters complexity completely outside the controller and worker-manager.

    Con: requires a separate languages outside core Ick; adds a separate "compliation" phase when managing project and pipeline specifications, which seems like an unnecesary step.

  • Add to the controller an understanding of pipeline paramaters, which get provided by projects using the pipelines. Implement a way to pass in parameter values for each type of snippet (Python and shell, at the moment).

    Pro: makes Ick's project/pipeline specifications more powerful.

    Con: more complexit in Ick, though it's not too bad; requires more effort to add a new languges for pipeline action snippets.

Overall, I find the approach of dealing with parameters natively in project and pipeline specifications the best one. So I choose that. If it turns out to be a problem, it's a decision that can be re-visited later.

Specification

Project parameters

I will add a way for a project to specify parameters. These apply to all pipelines used by the project. Parameters will be defined as a dict:

project: ick.liw.fi
parameters:
    git_url: git://git.liw.fi/ick.liw.fi
    rsync_target: ickliwfi@pieni.net:/srv/http/ick.liw.fi
pipelines:
  - build_static_site

A parameters value can any thing that JSON allows:

project: hello
parameters:
    gits:
      - url: git://git.liw.fi/hello
        branch: master
        subdir: src
      - url: git://git.liw.fi/hello-debian
        branch: debian/unstable
        subdir: src/debian
pipelines:
  - build_debian_package

In the above example, the Debian packacing part of the source tree comes from its own git repository that gets cloned into a sub-directory of the upstream part.

Pipeline parameters

I will add a way for pipelines to declare the parameters they want, by listing them by name.

name: build_static_site
parameters:
  - git_url
  - rsync_target
actions:
  - python: |
      git_url = params['git_url']
      rsync_target = params['rsync_target']
      import os, sys, cliapp
      def R(*args, **kwargs):
        kwargs['stdout'] = kwargs['stderr'] = None
        cliapp.runcmd(*args, **kwargs)
      R(['git', 'clone', git_url, 'src'])
      R(['ql-ikiwiki-publish', 'src', rsync_target])

When the controller give an action for the worker-manager to execute, the work resource will have the parameters:

{
    "parameters": {
        "git_url": "git://git.liw.fi/ick.liw.fi",
        "rsync_target": "ickliwfi@pieni.net:/srv/http/ick.liw.fi"
    },
    ...
}

The actual step will access the parameters in a suitable way.

  • If the step is implemented by the worker-manager directly, it can directly access the parameters directly.
  • If the step is implemented by a Python snippet, worker-manager will prepend a bit of code to the beginnig of the snippet to set a global Python dict variable, params, which can be used by the snippet.
  • If the step is implemented by a shell snippet, worker-manager will prepend a bit of code to the beginning of the snippet to define a shell function, params, that outputs a JSON object that defines the parameters. The snippet can pipe that to the jq utility, which can extract the desired value. jq is a small, but powerful utility (installation size about 100 KiB) for processing JSON programmatically from shell. It will need to be installed on the workers.

jq examples

To get a simple value:

params | jq -r .foo

To get a simple value from inside a more complicated on:

params | jq -r '.gits|.[1]|.url'

Considerations

There will be no type safety, at least for now. If the pipeline expects a list and gets a plain string, tough luck.

Requiring jq on workers is a compromise, for now. It avoids having to implement the same functionality in another way. It's small enough to hopefully not be a size problem on workers.

liw
Planning meeting (iteration 8)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The goal of this iteration is to have parametrised pipelines. Currently, Lars is running ick2 to update two static websites, with near-identical copies of the pipeline. The only differences are the git URL from which the website source gets retrieved, and the rsync target to which the result gets sent. This duplication needs to be fixed, and that is done by adding pipeline parameters.

Current projects

  • None.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Document how parameters work, are implemented Lars 1h
Implement project, pipeline parameters Lars 2h
Make a release Lars 1h
Convert Lars's pipelines into one Lars 1h
Total Lars 5h

Task descriptions

  • Document how parameters work, are implemented: Update the architecture document on the website with a description of how project, pipeline parameters work, in enough detail that someone else could do the implementation.

    Acceptance criteria: Lars is happy to do the implemention based on the documentation.

  • Implement project, pipeline parameters: Implement the parameter design.

    Acceptance criteria: The yarns have been updated to use parameters, and to test them, and they pass.

  • Make a release: Make a new release with support for parameters.

    Acceptance criteria: The release is available on code.liw.fi.

  • Convert Lars's pipelines into one: Upgrade ick on Lars's instance, and update the project and pipeline definitions to use parameters.

    Acceptance criteria: Updating git://git.liw.fi/ick.liw.fi.git results in http://ick.liw.fi getting updated automatically.

Posted
liw
Retrospective meeting (iteration 7)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Register worker automatically Lars yes
Give worker sudo via Ansible Lars yes
Define blob service API Lars yes
Implement blob service Lars yes

What went well?

  • All planned tasks got done.

What didn't go well?

  • Other things affected productivity.

What can we improve for the future?

  • Avoid other things.
Posted
liw
Planning meeting (iteration 7)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The main goal for this iteration is to get a very rudiemntary blob service working.

Roadmap to ALPHA-1

Current projects

  • None.

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Register worker automatically Lars 1h
Give worker sudo via Ansible Lars 1h
Define blob service API Lars 1h
Implement blob service Lars 4h
Total Lars 7h

Task descriptions

  • Register worker automatically: Change the worker-manager to register itself to the controller when it starts up.

    Acceptance criteria: Deploy a test instance, check manually that there is a registered worker after the test instance has started.

  • Give worker sudo via Ansible: Change the Ansible playbook that deploys an Ick instance to give the worker-manager user (_ickwm) passwordless sudo access to run debootstrap.

    Acceptance criteria: Deploy a test instance with a project and pipeline use the debootstrap pipeline step and check that it successfully builds a system tree.

  • Define blob service API: Write an API specification, in the form of a yarn scenario, for the simplest possible blob service API. It should be able to store a blob under a name, and retrieve the blob (same bytes, disregarding metadata such as content-type) using that name later.

    Acceptance criteria: Yarn scenario exists, is implamented, and as far as it can, it runs successfully. It's OK for it to fail because the blob service hasn't yet been implemented.

  • Implement blob service: Implement a blob service according to the minimal specification. It needs to work, it does not need to be performant in this iteration. Also add an Ansible playbook for deployment. Add support for storing and retrieving blobs to icktool.

    Acceptance criteria: The blob service yarns pass, locally and against a deployed instance. Deploying an ick2 instance results in a blob service also running.

Posted
liw
Retrospective meeting (iteration 6)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Add project, pipeline parameters Lars yes
Make pipelines be named resources Lars yes
Use temp dir for workspace Lars yes
Allow pipeline steps to be in Python Lars yes
Add a built-in debootstrap pipeline step Lars yes
Make release Lars yes
Upgrade Lars's two ick2 instances Lars yes

What went well?

  • Everything.

What didn't go well?

  • Nothing.

What can we improve for the future?

  • Try to attract other developers.
Posted
liw
Tentative API plans for blob service

We will soon be needing a blob service to store archives of systrees. Later on, we will want to use this for storing build artifacts.

I intend the blob service to be quite simple. The first version should be able to manage with the following API:

  • POST /id with an application/json body with the pipeline parameters that were used to build the blob. Computes a blob id from the body and returns it.

  • PUT /blobs/id where id is a blob id calculcated by by the previous request. Body is the blob to be stored. The blob service stores the body, ignoring content type for now. The service does not check hte id, at least for now.

  • GET /blobs/id where ìd is again a blob id. Returns the blob, with a content type of application/octet-stream.

This is enough to implement systree building and storing. It's not good enough long-term but we don't need to worry about that yet.

The id is computed by normalising the incoming JSON body and calculating the SHA256 of that. It explicitly not the SHA256 of the blob itself, since that would mean you'd need to know that to retrieve it, and we don't. We do know the pipeline parameters, though, so that's an easy way of identifying the blob we need.

Longer term plans

I'm thinking that longer term the blob service needs to start doing things like these:

  • store content type, creation timestamp, last use timestamp
  • possibly allow clients to specify arbitrary metadata, such as "filename" and a project/pipeline instance id
  • list all blobs, with metadata, including size of each blob
  • delete blobs
  • possibly have a way to specifying a retention policy, such as "keep the most recently used blobs up to N gigabytes of data for each project", and a way of deleting anything that doesn't fit the policy
  • do aggressive de-duplication of blob content: it's likely there's chunks of data that are identical across blobs, and there's no need to store those chunks more than once
    • de-duplicating backup software can offer tips here, such as calculating a weak rolling checksum and ending a chunk whenever its lowest N bits are 0 (with N=16, chunks will on average be around 64 KiB)
    • that kind of chunking doesn't care of the type of data, and doesn't mind if chunks are at different offsets in different blobs
    • it may be worthwhile to have the client do the chunking so that it can avoid uploading chunks that already exist in the service; likewise, possibly the client should download chunks directly, so it can avoid downloading chunks it already has; we'll see

All of this is tentative, and not meant for the first version, which will be as simple as possible so we get into use earlier rather than later.

Planning for ALPHA-1

Here is a very rough, and probably wrong, plan for getting to ALPHA-1. White is ready to be done, grey is blocked, green is done.

Posted
Ick2 build environments and concurrency and complicated mess

Background: ick2 can now run simple pipelines, which are really only lists of string that get run by a local shell instance. This is fine for things like running ikiwiki and publishing the results, and I have that running now. But for more complicated projects, such as building Debian packages of Ick2 itself, that's not really good enough:

  • all build dependencies need to be pre-installed on each worker, for all projects

  • builds have network access, which is not nice in general

  • there's no good way to express things like "build this project against these Debian release on these hardware architectures" (e.g., Debian 8, Debian 9, and unstable, on amd64, i386, armhf, and arm64); this would require specifying each combintion as a separate project

So I started thinking what it would take to support automatically maintained build environments that are isolated from the host operating system and the network, when needed, and also to provide "matrix builds".

Let me walk through my thining so far.

A project is a thing the user wants to build, test, deliver, deploy, or otherwise do something to, automatically, triggered by some event. It might be a program, a static web site, or an analysis of the Debian archive. For simpliciy, I call the thing that the user wants done building the project, whether it's actually compiling anything or not. The trigger mechanism is not relevant here, but the trigger should trigger the whole project to be built.

A project build should happen in suitable pieces, which I call pipelines. A pipeline is a unit of work to be done when building. A project can be split into multiple pipelines for several reasons:

  • modularity: to enable sharing code between projects, and re-arranging how parts of the build are combined for each project
  • concurrency: instances of a pipeline can be run at the same time
  • clarity: a long pipeline becomes easier to understand if it's broken up into several smaller pipelines

A pipeline consists of a sequence of steps, with no branches, or loops. Steps are taken in sequence, from the first to the last, unless a step fails, in which case the pipeline run aborts. For simplicity, think of a step as a snippet of shell or Python code provided by the user. Pipelines are specified independently of the project, and each project lists one or more of them to run.

Pipelines have parameters, supplied by the project. This allows sharing of the pipeline code (steps) between projects. (If the project doesn't supply all the parameters the pipeline needs, that's an error and the project can't be built.) Parameters are key/value pairs, similar to Python dicts or JSON objects. One set of parameters is a single dict; a project may provide a list of sets of parameters, for concurrent building, see below.

Each pipeline runs in some environment:

  • a container on the worker host
  • the worker host itself
  • a remote host accessed over ssh
  • some day, some other way of running commands (maybe remote host over a serial port?)

The primary goal is to run pipeline steps in containers, where the environment can have any build dependencies installed easily, but running command locally is necessary for some things.

A container requires a systree (system tree), which is like the root filesystem of an operating system. It might be produced with debootstrap or similar tools. Ick2 will provide a way for the user to specify how the systrees get constructed.

A build operates on some source code or other data that the user specifies. During a build, this is stored in a workspace. The user specifies how the workspace is populated at the beginning of a project build. This might be, for example, by running git clone to fetch the source code from a git server, or by downloading artifacts from previous builds. The project workspace gets populated at the beginning of a build, by running a named pipeline. The populated workspace is available inside a container, or other build environment.

A workspace archive is a snapshot of the workspace taken after a pipeline finishes successfully. The archive is named automatically based on the names of the parameters the pipeline declares that it needs. Systrees are constructed by unpacking archives: the archive is created by running a pipeilne that uses debootstrap to create a small Debian installation in the workspace, and then unpacked when a container is needed.

Archives are stored in a blob server (which will be a new component for ick2), and the worker-manager will upload them automatically when an archive is created, and download them when a build needs a systree. There will be some caching to avoid unnecessary transfers, and careful storage management to avoid the blob storage from using up all disk space. The goal is for archives to be reproducible in effect, even if not bit-by-bit, so that if an archive is deleted, it can always be re-built.

Once we start archiving build artifacts separately, they will probably also go to the blob server.

Archives are also used for distributing workspaces among worker hosts when builds happen concurrently on multiple workers. These archives get created and downloaded automatically, when ick2 decides that a pipeline needs to run on a different worker than the previous one, or on more than worker concurrently.

A project is built as follows:

  • create an empty, temporary directory as the workspace
  • run specified pipeline to populate workspace
  • run all named pipelines, one after the other, giving the workspace to each successive pipeline
  • if any pipeline step fails, abort

A pipeline may specify the systree it needs for running. A project may specify zero or more systrees. If both a project and its pipeline specifies a systree, the pipeline's (single) systree is used. Pipelines that do not specify a systree are run concurrently with each systree specified by the project, if the project specifies more than one. This results in concurrency.

Project may also specify more than one set of parameters. If they do, all pipelines are run concurrently with each set of parameters. Multiple systrees and parameter sets results in potentially a large matrix build.

When pipelines are run concurrently, all instances need to finish before the next pipeline or set of concurrent instances start. For example, there might be concurrent building of a project's Debian packages against several Debian releases and on several CPU architectures, but they all need to finish before the next pipeline, which uploads all of them to an APT repository.

In pseudo-code (ignoring error handling and other boring details):

for each pipeline in the project: for each set of project parameters: for each systree the project defines: start_pipeline_instance(params, systree, pipeline) wait until all those pipeline instances have finished

That is, all the pipeline instances for one set of parameters run concurrently, but the next pipeline doesn't start until they've all finished, successfully. Thus, all the .deb packages will finish building before the package uploading pipeline starts.

Not sure yet how to handle the handover of the workspace from a concurrent set of pipeline instances to the next pipeline. What version of the workspace should be handed over? Maybe require that each pipeline specifies what parts of the workspace to preserve and after concurrent pipeline instances, construct a union of the saved parts?

Examples

These are examples of the kind of YAML files I am thinking of for the thing I described earlier. Obviously I've not tested these, so there may be horrific mistakes. If so, gently point them out to me.

A list of systrees, and a pipeline to build systrees, and a project to build all systrees. Python steps can get the list of systrees, thus the build_systree pipeline gets a list of systree names, instead of the actual description.

systrees:
  # A systree for running ikiwiki and rsync
  debian9-ikiwiki:
    debian_codename: stretch
    packages:
      - ikiwiki
      - libhighlight-perl
      - graphviz
      - rsync

  # Another systree, for cloning git repositories.
  debian9-git-client:
    debian_codename: stretch
    packages:
      - git
      - openssh-client
      - python3
      - python3-cliapp

pipelines:

  # A pipeline to build a systree whose name is given as a parameter.
  build_systree:
    parameters:
      - systree_name
    systrees: []
    actions:
      - where: local
        python: |
          systree = get_systree(systree_name)
          runcmd([
              'debootstrap',
              systree['debian_codename'],
              '.',
              'http://deb.debian.org/debian',
          ])
      - where: local
        python: |
          for p in packages:
            runcmd(['sudo', 'chroot', 'apt', 'install', p])
    archive: yes


  # Pipeline to populate the workspace by cloning git repos.
  populate_workspace:
    parameters:
      - gits
    systrees:
      - debian9-git-client
    actions:
      - where: container
        network: yes
        python: |
          for spec in gits:
            if os.path.exists(spec['dir']):
              git_pull(spec['dir'])
              git_checkout(spec['branch'])
            else:
              git_clone(spec['git'], spec['dir'], spec['branch'])
    archive: no


  # Pipeline to run ikiwiki on a website source, publish it with rsync.
  # The source is assumed to be in the "src" subdir, and the generated
  # HTML will go to "html".
  website_with_ikiwiki:
    systrees:
      - debian9-ikiwiki
    parameters:
      - rsync_target
    actions:
      - where: container
        network: yes
        shell: |
          cd src
          sed -i '/^destdir:/' ikiwiki.setup
          echo "destdir: ../html" >> ikiwiki.setup
          ikiwiki --setup ikiwiki.setup
          rsync -av --delete ../html/. "{{ rsync_target }}/."

projects:

  # A project to build all systrees listed in parameter sets.
  build_all_systrees:
    parameters:
      - systree_name: debian9-build-essential
      - systree_name: debian-unstable-build-essential
      - systree_name: debian9-ikiwiki
    pipelines:
      - build_systree

  # Project to update the ick.liw.fi website from source in git, using
  # ikiwiki.
  ick.liw.fi:
    workspace:
      pipeline: populate_workspace
      gits:
      - git: git://git.liw.fi/ick.liw.fi
        branch: master
        dir: src
    parameters:
      rsync_target: ick@pieni.net:/srv/http/ick.liw.fi
    systrees: []
    pipelines:
      - website_with_ikiwiki
Posted
liw
Planning meeting (iteration 6)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • The goal of this iteration is to start moving ick2 towards the vision expressed in vision. This iteration won't reach there, but takes the first steps in that direction. There will be backwards-incompatible changes, but that's unavoidable in pre-ALPHA software.

  • I tried to keep tasks for this iteration to be as small as possible, but it's still a bunch of work.

  • After this iteration, project specification syntax will have changed to follow the example, which is still far from my vision, but closer than status quo.

      pipelines:
          update_ick.liw.fi:
              parameters:
              - unused_parameter
          actions:
              - where: local
                python: |
                  runcmd(['ikiwiki', '--setup', 'ikiwiki.setup'])
              - where: local
                shell: rsync -a --delete . ick@remote.site:
    
    
      projects:
          ick.liw.fi:
          parameters:
              unused_parameter: nonsense-value
          pipelines:
          - update_ick.liw.fi
    

Current projects

  • None.

Tasks for this iteration

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Add project, pipeline parameters Lars 1h
Make pipelines be named resources Lars 2h
Use temp dir for workspace Lars 1h
Allow pipeline steps to be in Python Lars 2h
Add a built-in debootstrap pipeline step Lars 1h
Make release Lars 1h
Upgrade Lars's two ick2 instances Lars 2h
Total Lars 19h

Task descriptions

  • Add project, pipeline parameters: Allow user to specify a parameters section for projects (as key/value pairs), and also for pipelines (as a list of key names). Give those of projects's parameters that a pipeline wants to the worker-manager in each work resource, and only those parameters. The values should be any value encodeable as JSON. The parameters need not be used for anything yet.

    Acceptance criteria: The yarns for the controller have been updated to verify that the projects can give parameters and that the work resources for a pipeline get the right ones. Also that if there are two pipelines in a project, which want different parameters, they get the right ones from the project. All yarns pass.

  • Make pipelines be named resources: Add a pipeline resource type to the API. Change project specs to refer to them by name.

    Acceptance criteria: Yarns for the controller have been updated to test that pipeline resources can be used via the API, and the yarns' project tests have been updated to use named pipelines, instead of embedded the step lists. All yarns pass.

  • Use temp dir for workspace: Make controller tell worker-manager to create a directory to act as the workspace and remember its location, and to run every pipeline step in the workspace directory. The workspace directory may be temporary or fixed, depending on what makes sense during implementation.

    Acceptance criteria: A test instance runs builds successfully.

  • Allow pipeline steps to be in Python: Add a way to add a snippet of Python code to implement a step, in addition to a shell snippet. Change worker-manager to run them with Python3. Document how the snippet gets run, and what imports and globals it can rely on.

    Acceptance criteria: A test instance runs builds successfully with pipeline steps implemented in Python.

  • Add a built-in debootstrap pipeline step: Add to worker-manager a way to run debootstrap directly, instead of invoking it via shell or Python. A step description would look like this:

      debootstrap: stretch
      mirror: https://deb.debian.org/debian
    

    The step gets run in the workspace directory.

    Acceptance criteria: A test instance of ick2 can run debootstrap and manual inspection shows that afterwards the workspace contains a directory tree than can be used to create a container: sudo systemd-nspan -D wrkspc pwd reports the root directory.

  • Make release: Update Debian packaging, NEWS, and other files. Tag and build a release. Update Ansible playbook.

    Acceptance criteria: Set up a test instance and check that it can run builds.

  • Upgrade Lars's two ick2 instances: I have two instances running ick2, updating static ikiwiki sites. Delete them, set up a one that has two projects, one per site to update. Update gitano hooks as necessary.

    Acceptance criteria: The instance still updates both ikiwiki sites when their backing git repositories change.

Posted
liw
Retrospective meeting (iteration 5)

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Add icktool manpage liw mostly
Clean up controller code liw yes
Fix PUT to nonexistent resource liw yes
Make icktool read YAML, send JSON liw yes
Make icktool show all piplines, their latest builds liw yes
Set up ick2 to update ick.liw.fi triggered by gitano liw yes

The icktool manpage isn't yet very useful, but improvements will happen later.

What went well?

  • Got everything done.

What didn't go well?

  • Nothing.

What can we improve for the future?

  • Keep working.
Posted
Lars Wirzenius
Planning meeting (iteration 5)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is for cleanup and bug fixing.

Current projects

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Add icktool manpage liw 1h
Clean up controller code liw 2h
Fix PUT to nonexistent resource liw 1h
Make icktool read YAML, send JSON liw 1h
Make icktool show all piplines, their latest builds liw 1h
Set up ick2 to update ick.liw.fi triggered by gitano liw 1h
Total liw 7h

Task descriptions

  • Add icktool manpage: icktool does not have a manpage. Write one. Have setup.py install it.

    Acceptance criteria: The manpage is installed by the ick2 .deb package.

  • Clean up controller code: The controllerapi.py module is a bit messy. Break it into smaller modules, clean up each to be clean code. Add missing unit tests.

    Acceptance criteria: Lars is happy with the code.

  • Fix PUT to nonexistent resource: Add a yarn test, and possibly unit test, to verify PUT to a non-existent resource fails. Fix the code so it does fail.

    Acceptance criteria: Test is implemented and code passes.

  • Make icktool read YAML, send JSON: Writing JSON manually is awkward (trailing commas!), but YAML is much more user-friendy. It would be easy for icktool to read YAML (the subset that is JSON-equivalent) and send JSON to the controller API. Make it so.

    Acceptance criteria: There are examples of icktool input in YAML in the source tree. They've been manually tried.

  • Make icktool show all piplines, their latest builds: Modify icktool to list all pipelines in all projects and the status of their current build or latest finished build if they're currently idle.

    Acceptance criteria: Manual testing indicates this works.

  • Set up ick2 to update ick.liw.fi triggered by gitano: Lars already has an ick2 instance on his laptop, with a project that generates and publishes the ick.liw.fi website from markdown in git using ikiwiki. This works, but the build needs to be triggered manually. Set up an instancein the cloud such that it can be triggered by the git server automatically upon changes being pushed to master.

    Acceptance criteria: The ick.liw.fi site is updated within a minute of changes being pushed to the master branch of the git repository backing the site.

Posted
Lars Wirzenius
Retrospective meeting

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Write yarn scenario for build build-scenario liw yes
Fix controller so build scenario passes build-scenario liw yes
Try new controller with worker-manager build-scenario liw yes
Make, test release with iteration changes build-scenario liw yes

What went well?

  • Got everything done.

What didn't go well?

  • Life got in the way.

What can we improve for the future?

  • Have no life.
Posted
Lars Wirzenius
Planning meeting (iteration 4)

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration is about getting the controller side of running a build working.

  • Due to my other time commitments this may become a fairly long iteration, but I would rather have a longer iteration than try to break this iteration down to smaller pieces that don't make sense on their own.

Current projects

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Write yarn scenario for build build-scenario liw 2h
Fix controller so build scenario passes build-scenario liw 4h
Try new controller with worker-manager build-scenario liw 2h
Make, test release with iteration changes build-scenario liw 1h
Total liw 9h

Task descriptions

  • Write yarn scenario for build: Write a yarn test scenario that simulates a worker-manger and uses controller API to get work to do, and reports results. The scenario need only test a single worker-manager, with a single project and pipeline, and only one build running at the same time. The scenario need not pass, but all steps must be implemented.

    Acceptance criteria: The test scenario follows the design in the architecture document and seems like it should work.

  • Fix controller so build scenario passes: Make any necessary changes to the controller to make the build scenario pass.

    Acceptance criteria: Test scenario passes.

  • Try new controller with worker-manager: Install newly changed controller on a test machine, and set up a worker (on the same or a different machine) and try running an actual build using ikiwiki to render a static web site.

    Acceptance criteria: Either the build works or I know where it fails, and have reported the failure as a ticket.

  • Make, test release with iteration changes: Make a release with the controller changes. If the test build didn't work, document this in the NEWS file. Make any necessary changes to ick-ansible to deploy controller.

    Acceptance criteria: I can set up a VM with the new controller.

Posted
Lars Wirzenius
Retrospective meeting

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Add icktool commands for project icktool-projects liw yes
Add /workers API ednpoint workers liw yes
Add icktool commands for workers workers liw yes

What went well?

  • Everything seems to be going smoothly.

What didn't go well?

  • All good.

What can we improve for the future?

  • Continue the same way.
Posted
Lars Wirzenius
Planning meeting

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • This iteration I want to make icktool useful for the project resources (create, update, list, show, delete), and add a way to manage worker resources (including with icktool).

Current projects

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Add icktool commands for project icktool-projects liw 1h
Add /workers API ednpoint workers liw 1h
Add icktool commands for workers workers liw 1h
Total liw 3h

Task descriptions

  • Add icktool commands for project: A subcommands to icktool to allow listing projects, creating, showing, updating, and deleting a project.

    Acceptance criteria: It's possible to create project, list all projects, show a particular project's info, update a project, and delete a project, using icktool.

  • Add /workers API ednpoint: Add API endpoint to the controller to manage worker information.

    Acceptance criteria: Workers can be managed (POST /workers, GET /workers, GET /workers/name, PUT /workers/name, DELETE /workers/name), and there are yarn tests for the API endpoint.

  • Add icktool commands for workers:A subcommands to icktool to allow listing workers ,creating, showing, updating, and deleting a worker.

    Acceptance criteria: It's possible to create project, list all workers, show a particular worker's info, update a worker, and delete a worker, using icktool.

Posted
Lars Wirzenius
Retrospective meeting

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Write icktool that can generate token icktool liw yes
Review yarns for /persons projects-endpoint liw yes
Reviaw code for /persons projects-endpoint liw yes
Clean up yarns, code for /persons projects-endpoint liw yes

What went well?

  • Got everything done.

What didn't go well?

  • N/A

What can we improve for the future?

  • N/A
Posted
Review of /projects yarn and code

I've reviewed the yarns for the /projects endpoint, and the corresponding code that implements it. Here are my thoughts:

  • 000.yarn
    • Add links to Ick and yarn home pages.
    • explain the the yarns can be run in either local mode (yarns start a local instance for each scenario) or remote mode (existing remote instance) and how to use ./check to run in either mode
  • 100-projects.yarn
    • the yarn and the architecture spec differ radically in what the project resource should contain; the arch spec seems better
    • otherwise, the scenarios seems OK
  • 200-version.yarn
    • add a note that the yarns are expected to run from a source tree, and the the remote controller is expected to have a compatible version
  • controllerapi.py
    • the code is not very clear, especially when it comes to using local functions to capture local variable/method args; it'd be clearer, I think, to not use that trick
    • would it be clearer and easier to use a more helpful backend than a directory with files? something like Qvarn comes to mind, though it'd require running a Postgres instance and that seems a bit heavyweight
    • decision: stick with the current approach, it's fewer moving parts
    • how strictly should the api validate incoming json resources? it'd be sensible to validate against a specified schema, and json schema seems like it might be plausible, but needs some research.

Tasks:

  • Add link to Ick home page to yarn introduction.
  • Add link to yarn home page to yarn introduction.
  • Add to yarn introduction an explanation of ./check can be used to run in local or remote modes and what that means.
  • Update 100-projects.yarn to use resources that match the arch doc.
  • Update 200-version.yarn with a note about source tree and remote controller version matching.
  • Refactor controllerapi.py to not use nested functions.
  • Add a future task to add resource validation to the controller.
Posted
Lars Wirzenius
Planning meeting

People

Agenda

  • Discuss and decide goals for this iteration.

Notes

  • My aim for this iteration is to get the controller to manage project information properly. This means reviewing the yarns and code related to the /projects end point, and fixing anything that needs fixing. This will be time-boxed, so that if I run out of time to fix, I will at least make a list of things that need to be fixed.

  • Additionally, I will start a new tool, icktool, which will provider a command-line interface to accessing the Ick APIs. Thinks curl, but with token generation added, and output formatting.

Current projects

Tasks for this week

Tasks may be part of a project or be random small ones (max an hour) that just need doing.

what project Who estimate(h)
Write icktool that can generate token icktool liw 2h
Review yarns for /persons projects-endpoint liw 2h
Reviaw code for /persons projects-endpoint liw 2h
Clean up yarns, code for /persons projects-endpoint liw 2h
Total liw

Task descriptions

  • Write icktool that can generate token: Add to ick2.git a new script, icktool, which can generate a token, and query /version. It should read a configuration file to get the necessary information so that the user doesn't need to repeat things every time.

    Acceptance criteria: Assuming a correct configuration file, user can run icktool token to generate a token, and icktool version to get a nicely formatted vesion of the /version resource. A token is generated automatically, as needed, or can be provided by the user via a command line option, to avoid re-generating every time.

  • Review yarns for /persons: Read the current yarns for /projects, and make note of anything that seems it would be worth fixing.

    Acceptance criteria: I'm happy that after fixing the things I find, the yarns will be good, meaning they're understandable by others, correctly specify the needed functionality, and completely test all the relevant aspects of the controllers handling of project information.

  • Reviaw code for /persons:Read the current code implementing /projects, and make note of anything that seems it would be worth fixing.

    Acceptance criteria: I'm happy that after fixing the things I find, the code will be good, meaning it's clean, correct, has good unit tests, and has no major architectural or other flaws.

  • Clean up yarns, code for /persons: This is time-boxed: only spend the allocated time this iteration, and if there's more to do, leave that for a future iteration.

    Acceptance criteria: I've fixed at least one thing off the list, and it's committed, merged, and passes CI.

Posted
Lars Wirzenius
Retrospective meeting

People

Agenda

  • How did we do this iteration? What went well? What't didn't? What can we improve for the future?

Notes

This is the retrospective for planning meeting:

what project? Who Done
Set up ick in my old CI release-version-endpoint liw yes
Build release of master release-version-endpoint liw yes
Write Ansible playbook to set up controller release-version-endpoint liw yes

What went well?

  • I got everything done.

What didn't go well?

  • Debugging is unpleasant.

What can we improve for the future?

  • Nothing springs to mine at this time.
Posted