Build images directly into the K3s containerd instance with BuildKit

Build images directly into the K3s containerd instance with BuildKit.md

I wanted to iterate on some images and have them available to K3s immediately, much like the Docker Desktop experience.

I also wanted to use docker buildx bake for this, but buildctl and nerdctl build should work fine too. You can omit Docker components if you don't want Buildx.

• Ubuntu 22.04
• K3s latest (v1.24.2+k3s1)
• BuildKit v0.10.3
• Docker Engine 20.10.17 (required for Buildx)
• Docker Buildx master (required for new 'remote' driver)
• Nerdctl 0.21.0

# Install K3s
curl -sfL https://get.k3s.io | INSTALL_K3S_CHANNEL=latest sh -s -

# Install BuildKit
curl -L https://github.com/moby/buildkit/releases/download/v0.10.3/buildkit-v0.10.3.linux-amd64.tar.gz \
| tar xzf - -C /usr/local/

# Install nerdctl
curl -L https://github.com/containerd/nerdctl/releases/download/v0.21.0/nerdctl-0.21.0-linux-amd64.tar.gz \
| tar xzf - -C /usr/local/bin

# Install buildkitd.service (below) into /etc/systemd/system/buildkit.service
systemctl daemon-reload
systemctl enable --now buildkitd.service

# Set up Docker + Buildx (skip this if you prefer to just use buildctl)
# Install Docker (https://docs.docker.com/engine/install/ubuntu/)

# Install Buildx (need the released version to build master)
mkdir -p /usr/local/lib/docker/cli-plugins
curl -Lo /usr/local/lib/docker/cli-plugins/docker-buildx \
  https://github.com/docker/buildx/releases/download/v0.8.2/buildx-v0.8.2.linux-amd64
chmod +x /usr/local/lib/docker/cli-plugins/docker-buildx

# Build Buildx master
docker buildx bake https://github.com/docker/buildx.git
mv ./bin/docker-buildx /usr/local/lib/docker/cli-plugins/docker-buildx

# Create a Buildx context
docker buildx create --name k3s --driver remote unix:///run/buildkit/buildkitd.sock --use

# Build something with Buildx
docker buildx build -t docker.io/some-namespace/some-image . --output type=image,unpack=true

# Manage built images with nerdctl
nerdctl --address /run/k3s/containerd/container.sock --namespace k8s.io image (ls/rm/inspect)

⚠️ Important arguments when building images

I found that if I omitted the docker.io part of the image tag and just built an image as some-namespace/some-image, it failed to find the image (or tried to pull it) when I tried to run it in K3s.

I also found that the unpack=true was required in the --output option to ensure the image was fully resolved. Without it, ctr image check showed them as 'unavailable' and they wouldn't run in K3s.

buildkitd.service

[Unit]
Description=BuildKit
Wants=k3s.service
After=k3s.service

[Install]
WantedBy=multi-user.target

[Service]
Restart=always
RestartSec=5s
ExecStart=/usr/local/bin/buildkitd \
  --oci-worker=false \
  --containerd-worker-addr=/run/k3s/containerd/containerd.sock \
  --containerd-worker-namespace=k8s.io

The method with just Buildkit and no Docker is not working for me (maybe something in buildkit changed) with K3s. I have documented my method here:
moby/buildkit#3629