Docker VM on osx using lima support

Docker tooling preparation

# Install necessary docker related tooling using brew
brew install docker # install docker command line tools
brew install docker-compose # install docker-compose plugin 
brew install docker-buildx # install docker-buildx plugin (for multi-arch builds)

# if previously rancher docker was installed, it might need additional cleanup.
# check if ~/.docker/cli-plugins/ contains docker-compose and docker-buildx
# and linked to correct/existing files
ls -la ~/.docker/cli-plugins
total 0
drwxr-xr-x@ 4 mar  staff  128 May  5 09:34 .
drwxr-xr-x@ 5 mar  staff  160 May  5 09:33 ..
lrwxr-xr-x@ 1 mar  staff   49 May  5 09:34 docker-buildx -> /opt/homebrew/opt/docker-buildx/bin/docker-buildx
lrwxr-xr-x@ 1 mar  staff   51 May  5 09:34 docker-compose -> /opt/homebrew/opt/docker-compose/bin/docker-compose
# if links are missing or linked to something like ~/.rd/bin/docker-compose
# remove them and link to the correct files
unlink ~/.docker/cli-plugins/docker-compose
unlink ~/.docker/cli-plugins/docker-buildx
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose
ln -sfn /opt/homebrew/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx

Docker host preparation

brew install lima # install lima to run docker locally
limactl start --name=docker template://docker # start lima with docker preconfigured template
# configure docker to use lima VM you have just created
docker context create lima-docker --docker "host=unix://${HOME}/.lima/docker/sock/docker.sock"
docker context use lima-docker
docker run hello-world

Optional configuration

OSX Keychain support for docker credentials store

brew install docker-credential-helper # install docker-credential-helper to use with osxkeychain
# for docker to use osxkeychain for credentials, we need to configure it if it's not done yet automagically
# check if osxkeychain is configured in ~/.docker/config.json
# it should looks similar too
cat ~/.docker/config.json
{
    "auths": {},
    "credsStore": "osxkeychain",
    "currentContext": "lima-docker"
}%
# if "credsStore": "osxkeychain", is missing, just add it manually

Minikube installation in lima docker vm

brew install minikube
brew install kubectl
minikube start --driver=docker --container-runtime=containerd

note: You can use this guide to deply kubernetes in different VM, just switch context in docker or use DOCKER_HOST env variable before installation and You are good to go. export DOCKER_HOST=unix://${HOME}/.lima/docker/sock/docker.sock

Host network for lima VM

Current limitations of lima is no support for forwarding UDP traffic from host to VM. To workaround this issue you can use host network for VM.

limactl stop docker # stop the instance before editing
limactl edit docker # You can also open vm config file directly using text editor `code ~/.lima/docker/lima.yaml`

search for the network: section in the config file or if it doesn't exist simply add it:

networks:
# Lima can manage daemons for networks defined in $LIMA_HOME/_config/networks.yaml
# automatically. The socket_vmnet binary must be installed into
# secure locations only alterable by the "root" user.
# The same applies to vde_switch and vde_vmnet for the deprecated VDE mode.
- lima: host

limactl start docker # start the instance

get new IP of lima VM

limactl shell docker ip addr show lima0 |grep -v inet6| grep inet | cut -d ' ' -f6 | cut -d '/' -f1
192.168.106.4

note: as to access docker we are using docker.sock, IP address manipulations will not affect existing configuration. after IP is resolved You can access docker host as regular host in the network.

ping 192.168.106.4
PING 192.168.106.4 (192.168.106.4): 56 data bytes
64 bytes from 192.168.106.4: icmp_seq=0 ttl=64 time=9.518 ms

UDP port forwarding using socat ( can be used in rancher desktop)

# on the host
socat -d -d -T15 udp4-recvfrom:8125,reuseaddr,fork tcp:localhost:8126
# on the lima docker 
socat -d -d tcp4-listen:8126,reuseaddr,fork UDP:127.0.0.1:8125
# test if packets are comming. Run from the host
echo "deploys.test.myservice:100|c" | nc -w 1 -u 127.0.0.1 8125 # example of statsd metric

note: -d -d for debug only

HOST:UDP:8125 --> HOST:TCP:8126 --> VM:TCP:8126 --> VM:UDP:8125

note: to ssh into ranchers lima VM

## ssh shell to rancher VM
LIMA_HOME="${HOME}/Library/Application Support/rancher-desktop/lima" "/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl" shell 0

Reference materials

Lima default VM template

minikube install in lima

marijus-ravickas/macosx_docker_lima.md