Building a Docker-based Development Environment for Concourse-CI

Nate Catelli
A write-up on creating a local development environment for concourse-ci.
March 25, 2018
dockerci/cdconcourse

Introduction:

Jenkins has predominantly been the bread and butter CI/CD tool for technology organizations, where very few tools have been able to compete with the expressiveness of its Groovy-based DSL and the extensibility of its plugin ecosystem. That being said, its tool API is not very straightforward and its configuration lends itself to eventually becoming a snowflake server on an organization’s network. Because of this, I’m always looking for new CI/CD tools to play with. concourse-ci caught my eye with its simple YAML-based configuration DSL and modular architecture. One thing I like about concourse-ci is how easy it is to integrate with VCS, enabling a high degree of automation.

Concourse-ci offers a few options for turning up a development environment. Many of them use a tool written by its parent organization, Cloud Foundry, called bosh. There is a docker-compose tutorial that offers an environment to play with the UI, but it’s missing many core components that prevents it from being usable for large-scale testing. I’ve made some modifications to their docker-compose environment, which makes it easier to experiment and develop concourse pipelines.

Requirements:

In order to proceed with this tutorial, you will need to install the following tools:

Setup:

You will need to clone the concourse dev environment repo.

$ git clone https://github.com/ncatelli/concourse-development-environment
$ cd concourse-development-environment
$ docker-compose up

Services:

The docker-compose.yml in the repository defines 3 core services: web, worker and db. It also includes a sidecar to handle key generation, a service for the fly cli utility, and a synchronization service to wrap it all together.

Network and volumes:

I’ve defined a frontend and backend network in order to separate the fly and worker services from postgres. I’ve also defined a flyrc volume for persisting the fly configurations across subsequent runs of the fly service.

volumes:
  flyrc:
  web_keys:
  worker_keys:

networks:
  frontend:
  backend:

Web API/UI Service:

The web API/UI service is a stateless service that handles build scheduling, user interaction, and worker managemement. This service primarily interacts with end users, workers, and any polled resources to determine if a build should be scheduled. In our docker-compose environment, it is configured to communicate with the postgres database and is in both the frontend and backend networks.

  web:
    image: concourse/concourse:3.9.2
    command: 
      - web
    ports: 
      - "8080:8080"
    volumes: 
      - "web_keys:/concourse-keys:ro"
    restart: unless-stopped 
    environment:
      CONCOURSE_BASIC_AUTH_USERNAME: concourse
      CONCOURSE_BASIC_AUTH_PASSWORD: changeme
      CONCOURSE_EXTERNAL_URL: "${CONCOURSE_EXTERNAL_URL}"
      CONCOURSE_POSTGRES_HOST: 'db'
      CONCOURSE_POSTGRES_USER: concourse
      CONCOURSE_POSTGRES_PASSWORD: changeme
      CONCOURSE_POSTGRES_DATABASE: concourse
    networks:
      - frontend
      - backend
    depends_on: 
      - db
      - ready

Worker Service:

The worker service is responsible for executing builds. It polls the web-api (ATC) for jobs. These jobs are configured to run within Docker containers, so it is important that the worker can access to the Docker engine. The problem with the default docker-compose tutorial is that Docker has not been added to the runner. We will extend both the compose file and the Dockerfile to mount the local host’s Docker socket into the worker container.

  worker:
    build:
      context: ./worker
    privileged: true
    command: worker
    volumes: 
      - "worker_keys:/concourse-keys:ro"
      - "/var/run/docker.sock:/var/run/docker.sock"
    environment:
      CONCOURSE_TSA_HOST: web
    networks:
      - frontend
    depends_on:
      - web

We will need to install Docker on the local host.

FROM concourse/concourse:3.9.2

LABEL maintainer="Nate Catelli <ncatelli@packetfire.org>"
LABEL description="Containerized version of a concourse worker running docker."

VOLUME /var/lib/docker

RUN apt-get update -y && \
    apt-get install curl -yq && \
    curl -sSL https://get.docker.com/ | sh && \
    apt-get clean

Since our goal is to invoke the concourse worker, we will simply extend the concourse image by triggering the Docker install shell script. We should now be able to schedule builds on our worker.

Keygen sidecar:

Both the worker and web containers require keys in order to operate. Before we can start using our containers, we will need to create a sidecar container to generate these keys. This can be accomplished with an alpine container and openssh.

FROM alpine:3.7

LABEL description='Key generation sidecar for concourse-ci'
LABEL maintainer='Nate Catelli <ncatelli@packetfire.org>'

ENV KEY_DIR='/data'

COPY start.sh /usr/local/bin/start.sh
RUN chmod +x /usr/local/bin/start.sh && \
    apk add --no-cache openssh

VOLUME ${KEY_DIR}
WORKDIR ${KEY_DIR}

CMD ["/usr/local/bin/start.sh"]

We will create a small alpine image to generate our keys. Then we will invoke a bash script provided by the concourse team that will generate our keys.

#!/bin/sh

mkdir -p ./web ./worker

ssh-keygen -t rsa -f ./web/tsa_host_key -N ''
ssh-keygen -t rsa -f ./web/session_signing_key -N ''

ssh-keygen -t rsa -f ./worker/worker_key -N ''

cp ./worker/worker_key.pub ./web/authorized_worker_keys
cp ./web/tsa_host_key.pub ./worker

Finally, we will mount volumes for each service’s keys.

  keygen_sidecar:
    build:
      context: ./keygen_sidecar
    working_dir: "/data"
    volumes:
      - "worker_keys:/data/worker"
      - "web_keys:/data/web"

Fly service:

The fly cli is used to interact with the web API and will be our main point of interaction with concourse. It can be used to create and trigger pipelines, inspect workers and poll the states of jobs. Since fly is a static binary, we can wrap it in a small alpine image.

FROM alpine:3.7

ARG VERSION="3.9.2"

LABEL description='Command container for concourse fly cli'
LABEL maintainer='Nate Catelli <ncatelli@packetfire.org>'

VOLUME /root

ADD https://github.com/concourse/concourse/releases/download/v${VERSION}/fly_linux_amd64 /usr/local/bin/fly
RUN chmod +x /usr/local/bin/fly

ENTRYPOINT [ "/usr/local/bin/fly" ]
CMD [ "-h" ]

Our main point of persistence for fly is the .flyrc file. Since our image is run as the root user, we can simply persist the state of our fly service by making the /root directory of our fly service a volume. We can then invoke this service any number of times without losing our login credentials.

  fly:
    build:
      context: ./fly
      args:
        VERSION: "3.9.2"
    volumes:
      - flyrc:/root
    networks:
      - frontend

Putting it all together:

Using all of these services we can now start our cluster by running docker-compose up. This should bring up each of our dependent services followed by the web-ui. This can be viewed by browsing to port 8080 on your localhost which should present you with and empty version of the web UI, showing that no pipelines are configured.

Configuring a pipeline:

Let’s push a simple hello world task to the concourse api using our fly service. We will begin by authenticating fly with the service. The following command connects to our concourse api using the basic auth credentials under the main team name.

$ docker-compose run --entrypoint sh fly
$ fly login -c http://web:8080 -u concourse -p changeme -t main
$ fly ts
name  url              team  expiry
main  http://web:8080  main  Tue, 10 Apr 2018 01:22:41 UTC

We can then create a basic hello world pipeline using the following simple pipeline. Which we should save locally to test-task.yml.

---
jobs:
- name: job-hello-world
  public: true
  plan:
    - task: hello-world
      config:
        platform: linux
        image_resource:
          type: docker-image
          source:
            repository: ubuntu
        run:
          path: echo
          args:
            - hello world

We can finally push it to the concourse api with the following command. This applies our configuration and unpauses the pipeline. After running the following commands you should now see the pipeline in your web UI, which you can manually trigger by clicking the job and clicking the + symbol in the top right corner.

$ fly -t main sp -c test-task.yaml -p helloworld
apply configuration? [yN]: y
pipeline created!
you can view your pipeline here: http://web:8080/teams/main/pipelines/helloworld

the pipeline is currently paused. to unpause, either:
  - run the unpause-pipeline command
  - click play next to the pipeline in the web ui
$ fly -t main up -p helloworld

Optionally, you can run the job via fly with the following trigger job command.

$ fly -t main gp -p helloworld
groups: []
resources: []
resource_types: []
jobs:
- name: job-hello-world
  public: true
  plan:
  - task: hello-world
    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ubuntu
      run:
        path: echo
        args:
        - hello world
$ fly -t main tj -j helloworld/job-hello-world
started helloworld/job-hello-world #2

Conclusion:

This simple docker environment should be enough to get you started running your first concourse pipelines. To expand on your pipeline’s complexity, I recommend referencing the great tutorials at concource tutorials as well as working your way through the documentation on the various components involved in creating a pipeline.