compose-on-kubernetes running on GKE

compose-on-kubernetes.md

Docker just opensourced compose-on-kubernetes (COK for short), and I wanted to run on GKE. This is the jurney:

Architecture

You will be able to use plain-old-compose files against k8s. Actually the docker-compose.yml have to be transformed to a Stack CustomResourceDefinition, and than a controller will take care of creating the all the necesseary k8s resources (deployments, services)

You can create those CRDs in 2 ways:

• run docker stack deploy --orchestrator kubernetes which will translate your docker-compose.yaml into an intermediate format, and send it to the compose-api runing on k8s
• create the CRD the cloudnative way: kubectl create -f stack.yaml

First you have to deploy COK on k8s, by creating a few k8s resources:

• deployment.extensions/compose - Controller for the Stack CRD
• deployment.extensions/compose-api - The Compose API itself
• service/compose-api - this where docker stack deploy --orchestrator kubernetes will talk to

Deployment

To be able to deploy the components above, the repo contains a Makefile, but it will work only on Docker4Mac or against DockerEE.

But there is a cli installer which can be parametrized. To build the installer, just call make binaries. Than you can install:

IMAGE_REPO_PREFIX=docker/kube-compose- ./bin/installer -kubeconfig=$KUBECONFIG

Create a stack

As I couldn't yet figure out how to tell docker stack deploy to talk to GKE. I just created teh CRD with kubectl:

kubect apply -f - <<EOF
apiVersion: compose.docker.com/v1beta2
kind: Stack
metadata:
  name: k8sweb
  namespace: docker
spec:
  services:
  - deploy:
      placement: {}
      resources: {}
    image: nginx
    name: myweb
    ports:
    - published: 80
      target: 80
EOF

The controller created the following resources:

$ kubectl get all -l com.docker.service.id
NAME                         READY   STATUS    RESTARTS   AGE
pod/myweb-7d7885c559-k2l88   1/1     Running   0          59m

NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/myweb             ClusterIP      None            <none>          55555/TCP      59m
service/myweb-published   LoadBalancer   10.19.243.204   35.204.58.104   80:31711/TCP   59m

NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/myweb   1         1         1            1           59m

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/myweb-7d7885c559   1         1         1       59m

The main resources:

• deployment.apps/myweb - the k8s deployment taking care of nginx
• service/myweb - the ClusterIP type k8s service representing, so that other components can refer to it by friendly dns name
• service/myweb-published - a LoadBalancer type service as I was exposing a service port.

tl;dr

Without specifying the IMAGE_REPO_PREFIX the deployments will try to use docker/kube-compose-XXX:latest images (XXX=controller,api-server), which doesnt exists.

But still if you use the right images, first you will face this:

panic: --etcd-servers must be specified

goroutine 1 [running]:
main.main()
        /go/src/github.com/docker/kamoulox-compose/cmd/api-server/main.go:17 +0x5a

So after I've watched some french youtube video as it was the first reference to kamoulox ;) I've digged deeper, and figured that compose-api needs access to an etcd. For Docker4Mac, where all k8s components are running in containers, you can just use the etcd server used by the k8s api-server (hint: docker run --net=host installer

In the applyEtcdOptions there is a special case, when the installed even deploys a basic etcd server. I wasn't able to reach that if section by changing configurations, so I made a small change in the code. After recompiling the installer, I got a bit further, but still no cigar. The installer creates a secret holding the ca/cert/key for etcd communication, but it was empty.

So i just used the certification script from istio, to create a valid cert for compose-api.docker.svc