Fast Intel-on-ARM Docker on macOS with Lima and Rosetta

README.md

macOS 13 Ventura introduced support of running amd64 binaries with Rosetta inside of arm64 Linux VMs when using Apple Virtualization framework.

Lima VM v0.14.0 and later support the new feature.

Setup:

# Install Docker client and Lima
brew install docker docker-compose docker-credential-helper lima

# Link docker-compose plugin
mkdir -p ~/.docker/cli-plugins
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

# Start VM
limactl start --name=default ubuntu-docker-lima.yaml

# Lima 0.14.2 panics on first start, run start again
limactl start default

# Run the ouput of the command above to setup Docker ^
docker context create lima-default --docker "host=unix:///Users/$USER/.lima/default/sock/docker.sock"
docker context use lima-default

# Docker client can now run arm64 containers
docker run hello-world

# and amd64 containers with the help of Rosetta
docker run --platform=linux/amd64 hello-world

Note: the yaml Lima config below is a combination of docker.yaml and vz.yaml with customized mounts.

ubuntu-docker-lima.yaml

vmType: "vz"
rosetta:
  # Enable Rosetta for Linux.
  # Hint: try `softwareupdate --install-rosetta` if Lima gets stuck at `Installing rosetta...`
  enabled: true
  # Register rosetta to /proc/sys/fs/binfmt_misc
  binfmt: true
images:
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"
  arch: "aarch64"
mounts:
  - location: "~/go"
    writable: true
  - location: "~/.cache"
    writable: true
mountType: "virtiofs"

# Docker config
containerd:
  system: false
  user: false
provision:
- mode: system
  # This script defines the host.docker.internal hostname when hostResolver is disabled.
  # It is also needed for lima 0.8.2 and earlier, which does not support hostResolver.hosts.
  # Names defined in /etc/hosts inside the VM are not resolved inside containers when
  # using the hostResolver; use hostResolver.hosts instead (requires lima 0.8.3 or later).
  script: |
    #!/bin/sh
    sed -i 's/host.lima.internal.*/host.lima.internal host.docker.internal/' /etc/hosts
- mode: system
  script: |
    #!/bin/bash
    set -eux -o pipefail
    command -v docker >/dev/null 2>&1 && exit 0
    export DEBIAN_FRONTEND=noninteractive
    curl -fsSL https://get.docker.com | sh
    # NOTE: you may remove the lines below, if you prefer to use rootful docker, not rootless
    systemctl disable --now docker
    apt-get install -y uidmap dbus-user-session
- mode: user
  script: |
    #!/bin/bash
    set -eux -o pipefail
    systemctl --user start dbus
    dockerd-rootless-setuptool.sh install
    docker context use rootless
probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    if ! timeout 30s bash -c "until command -v docker >/dev/null 2>&1; do sleep 3; done"; then
      echo >&2 "docker is not installed yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until pgrep rootlesskit; do sleep 3; done"; then
      echo >&2 "rootlesskit (used by rootless docker) is not running"
      exit 1
    fi
  hint: See "/var/log/cloud-init-output.log". in the guest
hostResolver:
  # hostResolver.hosts requires lima 0.8.3 or later. Names defined here will also
  # resolve inside containers, and not just inside the VM itself.
  hosts:
    host.docker.internal: host.lima.internal
portForwards:
- guestSocket: "/run/user/{{.UID}}/docker.sock"
  hostSocket: "{{.Dir}}/sock/docker.sock"
message: |
  To run `docker` on the host (assumes docker-cli is installed), run the following commands:
  ------
  docker context create lima-{{.Name}} --docker "host=unix://{{.Dir}}/sock/docker.sock"
  docker context use lima-{{.Name}}
  docker run hello-world
  ------

Unable to configure Rosetta: Rosetta is unsupported on non-ARM64 hosts

@runeksvendsen It looks like you have an amd64 version of Lima installed (also perhaps amd64 version of Homebrew?). You need arm64 Lima to use Visualization Framework with Rosetta. There was a commit two days ago that reports an error when Lima runs under Rosetta.

What does file $(which limactl) output? It should be Mach-O 64-bit executable arm64.

@akrylysov thanks! This is indeed the issue:

➜  ~ file $(which limactl)
/usr/local/bin/limactl: Mach-O 64-bit executable x86_64

I tried it once and it successfully for me into a Docker container where I could run x86 executables. But trying it a second time I get this segmentation fault:

$ limactl start --name=default ubuntu-docker-lima.yaml
? Creating an instance "default" Proceed with the current configuration
INFO[0000] Attempting to download the image from "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-arm64.img"  digest=
INFO[0001] Using cache "/Users/rune/Library/Caches/lima/download/by-url-sha256/17dad398d7fab7f11627da46d973b43a812847ace3b3216a590667981417c25d/data"
INFO[0003] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/rune/.lima/default/serial.log")
INFO[0003] [hostagent] Setting up Rosetta share
INFO[0003] SSH Local Port: 60022
INFO[0003] [hostagent] panic: runtime error: invalid memory address or nil pointer dereference
INFO[0003] [hostagent] [signal SIGSEGV: segmentation violation code=0x2 addr=0x20
INFO[0003] [hostagent] goroutine 94 [running]:
INFO[0003] [hostagent] github.com/containers/gvisor-tap-vsock/pkg/tap.(*Switch).Accept(0x1400060e000, {0x101094ed8, 0x140000a4140}, {0x10109c910, 0x1400012a248})
INFO[0003] [hostagent] 	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/containers/[email protected]/pkg/tap/switch.go:83 +0x4c
INFO[0003] [hostagent] github.com/containers/gvisor-tap-vsock/pkg/virtualnetwork.(*VirtualNetwork).AcceptBess(...)
INFO[0003] [hostagent] 	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/containers/[email protected]/pkg/virtualnetwork/bess.go:9
INFO[0003] [hostagent] github.com/lima-vm/lima/pkg/networks.run.func1()
INFO[0003] [hostagent] 	/private/tmp/lima-20221223-12555-f6y7sz/lima-0.14.2/pkg/networks/gvisor.go:90 +0x64
INFO[0003] [hostagent] golang.org/x/sync/errgroup.(*Group).Go.func1()
INFO[0003] [hostagent] 	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x5c
INFO[0003] [hostagent] created by golang.org/x/sync/errgroup.(*Group).Go
INFO[0003] [hostagent] 	/Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:72 +0xa4
FATA[0003] host agent process has exited: exit status 2

$ cat /Users/rune/.lima/default/serial.log

Thank you for sharing. It looks so promising for my use case.

Had no blips and docker run hello-world works great.

Issue

docker pull image
no matching manifest for linux/arm64/v8 in the manifest list entries

Is there something special I need to do to tell docker to use ARCH=x86_64 instead of aarch?

I did see some people tried to run it with a different arch, but it didn't so so well.

One thing I noticed is limactl list has it as ARCH=aarch64 instead of x86/amd64.

limactl list
NAME             STATUS     SSH                VMTYPE    ARCH       CPUS    MEMORY    DISK      DIR
dockerr86        Running    127.0.0.1:57201    vz        aarch64    4       4GiB      100GiB    ~/.lima/dockerr86

Is there a way to tell docker that the ARCH is different from the one listed in limactl?
Any debugging tips would be great.

UPDATE: Solution

ARCH=amd64 docker pull image
docker pull --platform=linux/amd64 image

Issue: This requires I pass --platform or ARCH= everywhere.

---

apple m1 max
Sonoma 14.5

$ limactl --version
limactl version 0.23.2
$ docker --version
Docker version 27.2.0, build 3ab4256958
$ file $(which limactl)
/opt/homebrew/bin/limactl: Mach-O 64-bit executable arm64