Docker for Mac

On M1 machines, Docker for Mac is running a lightweight linux ARM VM, then running containers within that, so containers are essentially running natively. Don't be fooled by the fact the UI or binary CLI tools (e.g. docker) might require Rosetta.

Within that VM is an emulation layer called QEmu. This can be used by docker to run Intel containers. This does not use Rosetta at all, and has a roughly 5-6X performance penalty. (If you just upgraded your CPU this may result in a similar performance to your old machine!)

Pulling and running with Docker

Many images in public registries are multi-architecture. For instance at the time of writing on Docker Hub the php:8.0-cli image has the following digests:

d98d657e4314 linux/386    162.19 MB
02b32d43112f linux/amd64  159.59 MB
8c4e84d860e3 linux/arm/v5 138.22 MB

If I want to use this image on an ARM machine I can docker pull php:8.0-cli and the image 8c4e84d860e3 will be tagged with that in my local registry. I can then docker run php:8.0-cli and get an ARM container.

From that perspective, everything will 'just work' for remote images that support ARM.

The most important understanding is that the local registry can only have one image with a particular architecture for each tag at any given time.

You can override which platform to pull using the --platform flag, or by setting the DOCKER_DEFAULT_PLATFORM environment variable. This has two effects:

During a pull it will set which architecture we want to pull to the local registry, and produce a hard error if that architecture is not found remotely
During a run it will specify which architecture we want to run, and produce a hard error if the image in the local registry doesn't match

(Note that a run where there's no local image is the same as a pull + run so the flag will be used in both places)

So, if I docker pull --platform=linux/amd64 php:8.0-cli I will get the image 02b32d43112f tagged locally and then I can run in a few ways:

docker run php:8.0-cli will run it using emulation, but show a warning. Running a different architecture to native is not an error if no platform was specified
docker run --platform=linux/amd64 php:8.0-cli will run it using emulation, with no warnings
docker run --platform=linux/arm64 php:8.0-cli will cause a hard error message

Note: you can end up in a situation where you pulled a non-native image and forgot, and because you don't specify platform when running you get poor performance but only a few warnings that you could ignore. It's best to be specific.

Some images don't have an ARM manifest, for instance on Docker Hub mysql is AMD-only. In those cases you will need to specify AMD when pulling (or on a first run if you've never pulled)

(There are AMD mysql images available, notably mysql/mysql-server so if in this situation you can either migrate or build your own new one)

Building images with Docker

When you build an image, the same sort of flags apply. docker build only builds a single platform which will default to ARM.

For the most part builds are simple, as long as you appreciate you will be getting ARM images as the result. For instance if your target says FROM php:8.0.0-cli then by default it will use the ARM base image and build an ARM image, using the steps defined.

If the base image does not have ARM support

For instance if your target says FROM mysql, you will get a hard error unless you specify platform i.e. docker build --platform=linux/amd64 .. The resulting image will be an AMD one and can be run under emulation as described above.

If the image will not build under ARM

There are various ways your images may be hard-coded to be AMD-only, so you may need to make modifications to be able to build natively.

A common thing to look for is binaries being downloaded:

RUN wget -o https://somesite.com/download/mylibrary_amd64.so mylibrary.so

Buildkit provides some handy build arguments that can be used here:

ARG TARGETARCH
RUN wget -o https://somesite.com/download/mylibrary_${TARGETARCH}.so mylibrary.so

For more complex situations you make need to throw in conditional expressions in bash and so forth.

A handy tip is that you can use the platform build args combined with multiple targets to add extra layers.

ARG TARGETARCH

FROM whatever AS base

# do stuff that's common to both architectures

FROM base AS build-arm64

# ARM stuff

FROM base AS build-amd64

# AMD stuff

FROM build-${TARGETARCH} as final

Then build with e.g. docker build --target=final .

Pushing to a registry

If your workflow involves building and pushing images locally (rather than building in CI), and there are a mix of AMD and ARM users you probably want to build AMD images explicitly. If you accidentally push an ARM image it'll overwrite the last AMD one pushed etc.

Multi-arch builds

The other alternative is to switch builder to something like docker buildx which can build multi-architecture images. You need to change your workflow slightly however, as it can't build two architectures for local use, it has to push them somewhere instead

This is done like docker buildx --platform=linux/amd64,linux/arm64 --tag ciaranmcnulty/whatever:latest --push .

Which would build both platforms and push as one tag to a registry

Docker Compose

The compose equivalent of the above is the platform key. You can pass this per-service, and run a mixture of architecutres:

services:
    nginx:
        image: nginx
    mysql:
        image: mysql
        platform: linux/amd64 # if you don't have this you'll get an error

Unfortunately for some older versions of docker-compose you may have issues with this flag, but as long as your tooling is up to date you'll be fine.

Unlike docker where it's possible to have different --platform for pull and run, with compose this flag will ensure you pull, build and run the same architecture for that service (unless you have separately manually pulled via docker)

Compose does not support multi-arch builds.

Wow, this is an incredible guide! We discovered this problem a while back, and that's why we started building Depot, a remote container build service that is backed by an optimized version of BuildKit (the same engine that backs Docker).

With Depot, we can build an image up on cloud VMs running native Intel & Arm CPUs so that you avoid the entire qemu emulation chain altogether.

The other alternative is to switch builder to something like docker buildx which can build multi-architecture images. You need to change your workflow slightly however, as it can't build two architectures for local use, it has to push them somewhere instead

Multi-architecture or multi-platform images are generally pretty painful as they require you to run your own builder for the non-host architecture you want to build for. It's also problematic when you want to run the image that is built as docker buildx build --load won't work with Docker when a multi-platform image is involved. You're stuck with pushing the image to a registry and then running docker pull to get the image back you want to run on your host architecture.

With Depot, we take over the buildx portion and route a true multi-platform image (i.e. linux/amd64,linux/arm64) to two separate builders that build their respective architectures on native CPUs and push the result to an image registry. We also handle the --load case. We can send back the architecture of your host machine when you perform a multi-platform image build and specify --load.

Great guide, and thank you again for sharing your knowledge!

ciaranmcnulty/notes.md