Unix socket between k8s pod and local host

README.md

Aim

We want to have a container running in kubernetes that is able to connect to a local process via a unix domain socket that it's listening on.

Local Host                 +-------------------------------------+
                           | Local Kubernetes Cluster            |
                           | +---------------------------------+ |
                           | | Pod                             | |
  server                   | |          client                 | |
    |                      | |            |                    | |
    |  listen              | |            |  connect+read+write| |
    |                      | |            |                    | |
    v                      | |            v                    | |
tmp/listen.sock <- - - - - - - - -> /app/tmp/listen.sock       | |
                           | |                                 | |
                           | +---------------------------------+ |
                           +-------------------------------------+

Simplified, with just Docker

Let's start with a simplified scenario where we just run the client in a docker container.

There is a simple client and server in this repository, written in go.

First we build the client as a docker image

docker build -t client .

We build the server as a local binary

go build server.go

We run the server in a second terminal and leave it running

## in a separate terminal
./server
# 2025/04/06 08:40:58 server: listening on tmp/listen.sock

## back in our first terminal
docker run -it --rm -v $PWD/tmp/listen.sock:/app/tmp/listen.sock client
# 2025/04/06 06:41:05 client: got message: Hello
# 2025/04/06 06:41:06 client: got message: Hello
# 2025/04/06 06:41:07 client: got message: Hello

One observation to make is that the mount selects the socket specifically. If we try to mount the containing directory, we get an error (todo: check linux)

% docker run -it --rm -v $PWD/tmp/:/app/tmp/ client
# 2025/04/06 06:44:20 client: dial: dial unix tmp/listen.sock: connect: operation not supported

Using kind

First, we have to start the server process, so that the socket exists.

## in the second terminal
mkdir /tmp/kind-share
SOCKET_PATH=/tmp/kind-share/listen.sock ./server

Then we can create our cluster

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraMounts:
  - hostPath: /tmp/kind-share/listen.sock
    containerPath: /tmp/host-mount/listen.sock
EOF

Because we've not set up a registry for kind, we have to load the image into the cluster.

kind load docker-image client:latest

Then we can apply the manifest for our client pod

k apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: client
spec:
  containers:
  - image: client
    imagePullPolicy: Never  # because we loaded the image into the cluster
    name: client
    volumeMounts:
    - name: local-share
      mountPath: /app/tmp/listen.sock
  volumes:
  - name: local-share
    hostPath:
      path: /tmp/host-mount/listen.sock
      type: Socket
EOF

kubectl logs client
# 2025/04/06 07:23:13 client: got message: Hello
# 2025/04/06 07:23:14 client: got message: Hello
# 2025/04/06 07:23:15 client: got message: Hello
# 2025/04/06 07:23:16 client: got message: Hello
# 2025/04/06 07:23:17 client: got message: Hello

Now, to clean up we can delete the cluster with

kind delete cluster

Using k3d

Warning

This doesn't work :(

First we will create a mountable share and start the server

## in second terminal
mkdir /tmp/k3d-share/
SOCKET_PATH=/tmp/k3d-share/listen.sock ./server

To create the cluster

k3d cluster create --no-lb \
    --k3s-arg "--disable=traefik@server:0" \
    --volume /tmp/k3d-share/listen.sock:/tmp/k3d-share/listen.sock@all
# ...
# Are you trying to mount a directory onto a file (or vice-versa)?
# ...

To load the image

k3d image import client:latest

To clean up, at the end

k3d cluster delete

client.go

package main

import (
	"bufio"
	"fmt"
	"log"
	"net"
	"os"
	"strings"
	"time"
)

func main() {
	log.SetFlags(log.LstdFlags | log.Lmsgprefix)
	log.SetPrefix("client: ")
	if err := xmain(); err != nil {
		log.Fatalln(err)
	}
}

func xmain() error {

	addr := os.Getenv("SOCKET_PATH")
	if addr == "" {
		addr = "tmp/listen.sock"
	}

	var d = net.Dialer{Timeout: 3*time.Second}

	for {
		conn, err := d.Dial("unix", addr)
		if err != nil {
			return fmt.Errorf("dial: %w", err)
		}
		defer conn.Close()
		conn.SetReadDeadline(time.Now().Add(3 * time.Second))
		message, err := bufio.NewReader(conn).ReadString('\n')
		if err != nil {
			return fmt.Errorf("read: %w", err)
		}
		message = strings.TrimSuffix(message, "\n")
		log.Printf("got message: %v\n", message)

		time.Sleep(1*time.Second)
	}
}

Dockerfile

FROM golang:1.23 AS builder
    WORKDIR /work
    COPY client.go .
    RUN go build client.go

FROM debian:bookworm-slim
    WORKDIR /app
    COPY --link --from=builder /work/client /app/client
    CMD ["./client"]

server.go

package main

import (
	"fmt"
	"log"
	"net"
	"os"
)

func main() {
	log.SetFlags(log.LstdFlags | log.Lmsgprefix)
	log.SetPrefix("server: ")

	addr := os.Getenv("SOCKET_PATH")
	if addr == "" {
		addr = "tmp/listen.sock"
	}
	err := os.RemoveAll(addr)
	if err != nil {
		log.Fatalf("removing %v: %v\n", addr, err)
	}

	log.Println("listening on", addr)
	l, err := net.Listen("unix", addr)
	if err != nil {
		log.Fatalln("listen:", err)
	}
	defer l.Close()

	for {
		conn, err := l.Accept()
		if err != nil {
			log.Fatalln("accept:", err)
		}
		go func(conn net.Conn) {
			defer conn.Close()
			_, err := fmt.Fprintln(conn, "Hello\n")
			if err != nil {
				log.Printf("write failed: %v\n", err)
			}
		}(conn)
	}
}