Some thoughts about making a distributed CI system secure.

  • outline of system

    • version control system hold source code
    • IDP authenticates and authorizes users, system components
    • controller co-ordinates builds, collects build logs
    • artifact store holds build artifacts
    • workers (many) do the actual building, are told by controller what to do, fetch source from version control system, upload artifacts to artifact store
  • entitites in the system that need to be protected:

    • the person using CI
    • the person running the IDP, controller, and artifact store (for simplicity, assume they're all run by the same person, although they could each be run by separate people)
    • the people running runners
  • threats to person using CI

    • malicious workers, which embeds unwanted code in build artifact
      • mitigation: use reproducible builds and build on at least two workers to detect unwanted changes in artifacts; this would work OK, if there are relatively few malicious workers
    • many malicious workers, or workers that become malicious after a long period of working fine
      • mitigation: have at least one trusted worker, which might be slow, but whose output is required for a build to be trusted
        • artifacts from maybe-trusted workers can't be used for deployment, but could be used with sufficient isolation to speed things up, e.g., to do heavy testing: if the trusted worker later confirms the binaries are trustworthy (bitwise identical), then the test results can be trusted, too
      • variant of mitigation: require at least N maybe-trusted workers to produce bitwise identical build artifacts, where N is set by the person running the CI or whose project is being built
      • rejected: a karma or reputation system based on past behaviour: this makes long-lived workers valuable targets, and years of good behaviour won't protect if the worker gets hijacked
  • threats to person running IDP, controller, artifact store

    • there are no new threats to these that come due to the distributed nature of CI
    • all the usual threats apply, of course
  • threats to those running workers

    • build uses too much CPU or RAM
      • mitigation: enable person running worker to set limits and priorities so that the build doesn't use resources needed by other things
    • build attacks remote hosts (e.g., DDoS)
      • mitigation: prevent build from accessing any network hosts, except version control server, controller, artifact store
    • build attacks host where worker runs
      • mitigation: run build in a VM, using the best avilable isolation techniques, such as carefully configured qemu/KVM to implement the VM, and keeping all related software up to date