Buildx setup on docker for linux and performing a multiarch build.

Preface

Docker buildx is a build utility for docker that lets you build and push images with multiple processor architectures. In my case I'm going to build images that are ultimately going to run on my RPI4 (linux/aarch64).

The way this works is that docker executes the build sequences with moby-buildkit docker images that rely on qemu, which emulates the architecture you're going to build for, but do not run locally.

CHECK THE DATE THIS ARTICLE WAS PUBLISHED. STUFF, SPECIALLY THE ONES ABOUT THE CONSTRAINTS OF THE LOCAL REGISTRY MAY BE DEPRECATED AT THE TIME YOU READ IT

Requirements (Running the QEMU Build sequence in the moby-buildkit docker image)

To be able to run docker buildx on your linux machine, you should meet the requirements below: This thing runs QEMU, and as long as you're not doing anything weird, like a PCI passthrough which needs KVM acceleration (Only applicable to X86_64) you don't even need VT-d/VT-x enabled or whatever AMD calls it.

Docker >= 19.03
Linux kernel >= 4.8
binfmt_misc fs should be mounted (Systemd's responsible for mounting the FS)

Check the versions yourself and do something about them if they did not meet the requirements. Docker 19.03 to some point later did not have the manifest and buildx features enabled by default, so please enable the experimental features in your ~/.docker/config.json if you have one of those older versions installed. The version I have installed at this point of time (20.10.12, build e91ed57) has these features enabled by default.

Setup

We need to first make sure the binfmt_misc filesystem is mounted, so run the line of command below:

ls /proc/sys/fs/binfmt_misc/ #this should return you with 'register status'

Then run

docker buildx install #Adds a line to your ~/.docker/config.json that aliases buildx as the build module.

Now you should be ready for starting your work with Buildx itself.

Getting Started with Buildx

To get started with buildx you should instruct the buildx submodule to get the right build images (which obviously should be in your target architecture's supported architecture.)

Spawning buildx buildkit images

With running this line of command you'll instruct buildx to spawn up a new buildkit docker image when you'll start building your first image (it will stick around after your first build and you should destroy it when you no longer wanted it.)

docker buildx create --platform linux/amd64/,linux/arm64,linux/arm/v7 --name buildimage

You may find the list of platforms in this page And the the buildx configuration reference in this page

Now we should change our dockerfile to be able to perform a multiarch build:

Changing the dockerfile for a multiarch build:

Change your From statement to include --platform=$BUILDARCH like the sample below:

FROM --platform=$BUILDARCH alpine:3.14
LABEL maintainer "Khashayar Danesh [email protected]"

# Install
RUN apk --no-cache add ca-certificates curl minidlna tini shadow su-exec alpine-conf inotify-tools

Now let's get to building the image.

Running a Multiarch build

So now we should pass a couple of extra options to our docker build cli to be able to do the the build, and there are a few thing I would like to make you aware of:

Note: The image build happens in a docker image, so you should ask it to export the build results to somewhere because it does not automatically store the build results on your docker daemon image registry. You have a couple of options; May it be your local docker daemon or a registry. Though there are some constraints about loading multiarch builds to your local registry which will be explained below.

Note: This multiarch build process does not support exporting multiarch images to your local docker daemon, because the registry bit doesn't support manifests as of now, though there are somewhat workaround for this and you can simply push multiarch builds to a docker registry and if you're keen on keeping the images on your local daemon, you can do a single arch build for whatever arch and keep it there, but that's alt-arch rather than multiarch.

With that out of the way, let's do a multiarch build:

# Multiarch build and push it to a registry (requires setting up the registry auth beforehand) (and change the registry ref)
docker buildx build . --platform linux/arm64,linux/amd64 --tag minidlna/multiarch:latest --export-to=type=registry,ref=localhost:5000/myrepo:buildcache

# Single alt build and importing the results to the local docker daemon:
docker buildx build . --platform linux/aarch64 --tag minidlna/altarch-arm64:latest --load 

# Single host-arch build and pushing the results to the local docker daemon:
docker buildx build . --tag minidlna/hosttarch:latest --load 
# Note: 'docker build .' doesn't store the single hostarch builds in the local image registry either (because we swapped the build module entirely earlier)
#        so just use the line above.

What about the buildkit images we spawned earlier?

# Checking the buildkit container's status: 
docker buildx ls 

# Stopping the builder image
docker buildx stop <name>

You can also inspect the builders and many other things. Just check the buildx reference.

KhashayarDanesh/BuildxSetup.md