special_sauce.md

Docker Build Pipeline with Remote Asset Compilation

Problem

To compile code into assets/artifacts and build a Docker image with the resulting code only, and no compilers/devel packages/libraries, you have a workflow like so:

1. Clone a git repo
2. Run a docker container to compile assets from code in said repo
3. Run a docker image build with resulting assets

This works fine with a local Docker daemon, but what if your CI environment connects to remote Docker daemons on worker nodes?

In this case the Docker build context is local to your CI environment. The CI client connects to the remote host, and send the code and build context to the remote host's Docker daemon. The remote host docker daemon runs the container, compiling assets
and putting them into a persistent volume mounted from the remote host's filesystem.

Then, your CI environment attempts to launch a Docker build job, but the context is still in the CI environment, and the compiled code cannot be made available to to the build job.

Solution

Dependencies -

  1. Image with Docker client & Git installed ("Docker/Git" image)
  2. Image with compilers, dev tools, libraries ("Builder" image)
  3. CI Server has remote access to Build Server via Docker API

Process -

1. CI server creates "Docker/Git" container on Build Server

   Important options:

      -v /var/run/docker.sock:/var/run/docker.sock:Z
      -v /scratch:/scratch:Z  <--could be any place with sufficient space
      --privileged
   
3. Docker/Git container clones repo with source code
4. Docker/Git container runs "Builder" container on the Build server via the 
   docker socket, but mounts the /scratch volume from itself Builder container's
   output directory
   
   Important options:
     
        -v /scratch:/build_output:Z
        --privileged
   
 5. Builder contianer compiles code/assets and drops them into output directory
 6. /scratch inisde Docker/Git container now contains compiled code/assets
 7. Docker/Git container builds a Docker image on the Build server via the Docker 
    socket with /scratch as the build context
    
       cd /scratch ; docker build -t myimage .  <-- the "." is the build context (/scratch)
    
 8. Code/assets from /scratch now available to Docker build process
 9. Resulting image exists on Build server, with code from compiled assets
 10. CI can do whatever with image now

Example

Note: tabs and ">>" used to try to denote containers in containers

It's turtles all the way down, man...

Contents of Dockerfile-docker_git:

FROM centos:centos7
MAINTAINER Chris Collins <[email protected]>
RUN yum install -y docker git

Build the docker-git image

docker -H remote_host.example.org:2376 build -f Dockerfile-docker_git -t docker-git .

Contents of Dockerfile-builder:

FROM centos:centos7
MAINTAINER Chris Collins <[email protected]>

# Build tools, etc
RUN yum install -y gcc make autoconf mysql-devel etc. etc.

# This is a stand-in for compiling code
ENTRYPOINT /bash -c "echo 'hi' > /build_output/output.txt"

Build the builder image

docker -H remote_host.example.org:2376 build -f Dockerfile-builder -t builder .

Run docker-git image to clone repo

docker -H remote_host.example.org:2376 run \
       -v /scratch:/scratch:Z \
       -v /var/run/docker.sock:/var/run/docker.sock:Z \
       --privileged \
       --entrypoint git \
       docker-git clone https://my_repo/ /scratch

/scratch on remote_host.exmaple.org now containes contents of git repo

Run the docker-git image to:

run the builder image to:

"compile" the code

docker -H remote_host.example.org:2376 run \
       -v /scratch:/scratch:Z \
       -v /var/run/docker.sock:/var/run/docker.sock:Z \
       --privileged \
       --entrypoint docker \
       -it docker-git run \
                      -v /scratch:/build_output:Z \
                      --privileged \
                      -it builder

/scratch on remote_host.example.org now contains contens of git repo "compiled"

ie: output.txt with "hi" in it (the "compile" output from the builder container)

Run the docker-git image to:

build the final_image

docker -H remote_host.example.org:2376 run \
       -v /scratch:/scratch:Z \
       -v /var/run/docker.sock:/var/run/docker.sock:Z \
       --privileged \
       --entrypoint docker \
       --workdir /scratch \
       --it docker-git build \
                       -t final_image .

"." is /scratch, and is the build context of the docker build.

Docker image "final_image" now exists on remote_host.example.org, but from the contents of /scratch, including our "compiled" outputs.txt file.

Tag and Push the final_image

 docker -H remote_host.example.org:2376 tag final_image registry.example.org/final_image:latest
 docker -H remote_host.example.org:2376 tag push registry.example.org/final_image:latest

Image with compiled code now in the registry available to other hosts/users/etc.