kidn - k3s in dockerd's namespaces

_readme.md

kidn

k3s in dockerd's namespaces

This enables a convenient development setup where images built locally with docker build are available to the kubernetes cluster without needing to push the image to a registry from where kubernetes can pull it.

The version of Docker Desktop for macOS includes a kubernetes cluster that already does the above. This script enables similar on Linux hosts, and also works in macOS. More accurately, the nsenter hack is necessary for running this on macOS and also happens to work on Linux hosts (where a simpler version without the hacks could work).

Tested on

• macOS 10.14
• Ubuntu 18.04 (failed with docker installed via snap; worked with docker installed from docker.com's packages)
• NixOS (if using firewall, requires allowing all traffic from cni0 network)

1. A privileged docker container is started in the host's pid namespace
2. That namepsace is scanned for 'dockerd'
3. The namspaces of that dockerd are entered, and a shell is executed
4. Those namespaces are assumed to include commands: sh, env, and k3s (via docker volume)

$ ./kidn
...

$ kubectl --kubeconfig ./kubeconfig.yaml get pods -A
NAMESPACE     NAME                         READY   STATUS      RESTARTS   AGE
kube-system   coredns-b7464766c-76ll7      1/1     Running     0          20m
kube-system   helm-install-traefik-dthwg   0/1     Completed   0          20m
kube-system   svclb-traefik-4856w          2/2     Running     0          9m5s
kube-system   traefik-56688c4464-zkwbh     1/1     Running     0          9m5s

$ docker build -t localimage .

$ kubectl --kubeconfig ./kubeconfig.yaml run test --restart=Never --image=localimage --image-pull-policy=Never
pod/test created

$ kubectl --kubeconfig ./kubeconfig.yaml logs test
it worked

On macOS, where the underlying Linux VM is normally not accessible, we can use the nsenter hack to run vpnkit directly, connecting port 80 on the mac to port 80 on the Linux VM (where svclb has directed traffic to traefik). This is the same technique used when exposing ports via docker run --publish....

$ ./kidn nsenter vpnkit-expose-port -proto tcp -host-ip 127.0.0.1 -host-port 80 -container-ip 127.0.0.1 -container-port 80

Dockerfile

FROM busybox:1.31.0
ENTRYPOINT ["echo", "it worked"]

kidn

#! /usr/bin/env bash

# The reason k3s server is run separately and in a docker container
# (rather than in dockerd's namespace) is so we can use docker's
# ability to forward a port, which works on Mac even though
# docker-client is not on the same machine as dockerd.

set -uo pipefail

warn() { >&2 printf "%s\\n" "$*"; }
abort() { warn "$@"; exit 1; }

NSENTER_IMAGE="${NSENTER_IMAGE:-busybox:1.31.0}"
K3S_CLUSTER_SECRET="${K3S_CLUSTER_SECRET:-changethissecret}"
PORT="${PORT:-6443}"

K3S_VERSION="${K3S_VERSION:-0.8.0}"
K3S_IMAGE="rancher/k3s:v$K3S_VERSION"
K3S_CHECKSUM="8cc31a89adbf61a92d0548cd9e278ec685367fafced3659cbbf894aa75005dac"

# 1. A privileged docker container is started in the host's pid namespace
# 2. That namepsace is scanned for 'dockerd'
# 3. The namspaces of that dockerd are entered, and a shell is executed
# 4. Those namespaces are assumed to include commands: sh, env, and k3s (via docker volume)
#
# Note: the user namespace is not entered (possibly because CLONE_FS
# was used? see setns(2) for possible failure reasons).

script='
  pid="$(pgrep dockerd)"
  if [ -z "$pid" ]; then
    >&2 echo no pid for dockerd
    exit 1
  fi

  # prepend dockerd PATH to our PATH, since it is more likely to have
  # useful things like coreutils and dockerd, particularly on NixOS
  entry="$(</proc/$pid/environ xargs -n1 -0 echo | grep ^PATH=)"
  if [ -n "$entry" ]; then path="${entry#PATH=}:$PATH"; else path="$PATH"; fi

  exec nsenter -t "$pid" -m -u -i -n -p -r -w -F -- env "PATH=$path" "$@"
'

if [[ "${1:-}" = nsenter ]]; then
  shift
  warn "executing in dockerd's namespaces: $*"
  exec docker run -ti --rm --pid host --privileged "$NSENTER_IMAGE" \
    sh -c "$script" -- "$@"
fi

docker pull "$K3S_IMAGE" || abort "failed to pull image: $K3S_IMAGE"

docker volume inspect --format='{{.Name}}' kidn >/dev/null ||
  docker volume create -d local kidn ||
  abort "failed to create volume"

docker inspect --format='{{.ID}}' kidn-server >/dev/null ||
  docker run --detach \
    --name kidn-server \
    -v "kidn:/var/lib/rancher/k3s" \
    -p "$PORT:$PORT" \
    "$K3S_IMAGE" server \
    --write-kubeconfig /kubeconfig.yaml \
    --cluster-secret "$K3S_CLUSTER_SECRET" \
    --https-listen-port "$PORT" \
    --disable-agent || abort "failed to run server"

SERVER_IP=$(docker inspect kidn-server --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' | head -n1) || abort "failed to find server IP"

for i in {1..10}; do
  docker cp kidn-server:/kubeconfig.yaml ./kubeconfig.yaml && break
  sleep 2
done || abort "failed to copy kubeconfig.yaml"

vol() {
  printf %s/%s "$(docker volume inspect kidn --format='{{.Mountpoint}}')" "$1"
}

fetch-k3s() {
  local URL="https://github.com/rancher/k3s/releases/download/v${K3S_VERSION}/k3s"
  
  docker run --rm -v kidn:/kidn "$NSENTER_IMAGE" sh -c "
    set -e

    [ -x /kidn/bin/k3s ] && exit 0

    wget -O /tmp/k3s https://github.com/rancher/k3s/releases/download/v${K3S_VERSION}/k3s
    sum=\"\$(sha256sum /tmp/k3s | cut -f1 -d' ')\"
    [ \"$K3S_CHECKSUM\" = \"\$sum\" ] || {
      >&2 echo \"bad checksum: \$sum expected: $K3S_CHECKSUM\"
      exit 1
    }

    mkdir -p /kidn/bin
    chmod +x /tmp/k3s
    mv /tmp/k3s /kidn/bin/k3s
  "
}

fetch-k3s || abort "failed to install k3s into volume: $(vol bin/k3s)"

docker inspect --format='{{.ID}}' kidn-node >/dev/null ||
  docker run --detach \
    --name kidn-node \
    --pid host \
    --privileged "$NSENTER_IMAGE" \
    sh -c "$script" -- "$(vol bin/k3s)" agent \
    --cluster-secret "$K3S_CLUSTER_SECRET" \
    --server "https://$SERVER_IP:$PORT" \
    --data-dir "$(vol run/k3s)" \
    --kubelet-arg "root-dir=$(vol run/kubelet)" \
    --kubelet-arg "experimental-dockershim-root-directory=$(vol dockershim)" \
    --docker ||
  abort "failed to run agent"