Skip to content

Instantly share code, notes, and snippets.

@dims

dims/k8s-e2e-node-lambda-cloud.md

Created April 2, 2026 02:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dims/b033e62eab37b0a13807d1ef05304bee to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dims/b033e62eab37b0a13807d1ef05304bee to your computer and use it in GitHub Desktop.
Download ZIP
Running Kubernetes GPU e2e Tests on Lambda Cloud
Raw
k8s-e2e-node-lambda-cloud.md

Running Kubernetes GPU e2e Tests on Lambda Cloud

Step-by-step guide to launch a Lambda Cloud GPU instance, set up a single-node Kubernetes cluster with NVIDIA GPU support, and run GPU e2e tests like [sig-node] [Feature:GPUDevicePlugin] [Serial] Test using a Job should run gpu based jobs.

Prerequisites (on your Mac)

  1. lambdactl CLI at ~/go/src/github.com/dims/lambdactl/bin/lambdactl
  2. Lambda API key stored at ~/.config/lambda/.key (or set LAMBDA_API_KEY env var)
  3. SSH key registered with Lambda (see step 1b below)

1. Launch an Instance

1a. Check availability

lambdactl types

This shows all instance types and how many are available. Pick one with >0 available. You can also watch for any available instance:

# Watch for ANY available GPU (launches cheapest found)
lambdactl watch --ssh dims --dry-run

# Watch for a specific type
lambdactl watch --gpu gpu_1x_a10 --ssh dims --dry-run

Remove --dry-run to actually launch when found.

1b. Register your SSH key (one-time setup)

# List existing keys
lambdactl ssh-keys

# Add your key
lambdactl ssh-keys add <your-name> ~/.ssh/id_ed25519.pub

1c. Launch

lambdactl start --gpu gpu_1x_a10 --ssh <your-key-name>

Wait for it to print Ready! ssh ubuntu@<IP>. Note the IP address.

2. Set Up the Instance

SSH in:

ssh ubuntu@<IP>

2a. Install base packages and Go

set -eux

sudo apt-get update -qq
sudo apt-get install -y -qq \
  build-essential pkg-config libseccomp-dev libseccomp2 \
  iptables iproute2 conntrack ebtables kmod socat ethtool \
  jq rsync psmisc curl wget git

# Go (match the version in kubernetes/go.mod)
GO_VERSION=1.26.0
curl -sSL "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" | sudo tar -C /usr/local -xzf -
echo 'export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"' | sudo tee /etc/profile.d/golang.sh
export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"
go version  # verify

2b. Install CNI plugins

if [ ! -f /opt/cni/bin/bridge ]; then
  CNI_VERSION=$(curl -s https://api.github.com/repos/containernetworking/plugins/releases/latest | grep tag_name | cut -d'"' -f4)
  sudo mkdir -p /opt/cni/bin
  curl -sL "https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-amd64-${CNI_VERSION}.tgz" \
    | sudo tar -C /opt/cni/bin -xz
fi

2c. Configure CNI networking

sudo mkdir -p /etc/cni/net.d
sudo tee /etc/cni/net.d/10-containerd-net.conflist > /dev/null <<'EOF'
{
  "cniVersion": "1.0.0",
  "name": "containerd-net",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "promiscMode": true,
      "ipam": {
        "type": "host-local",
        "ranges": [[{"subnet": "10.88.0.0/16"}]],
        "routes": [{"dst": "0.0.0.0/0"}]
      }
    },
    {
      "type": "portmap",
      "capabilities": {"portMappings": true}
    }
  ]
}
EOF

2d. Configure containerd with NVIDIA runtime

Lambda ships containerd with CRI disabled and no NVIDIA runtime configured. We need both.

# Generate fresh containerd config with CRI enabled + SystemdCgroup
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml > /dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# Add NVIDIA runtime and set it as default (required for GPU device plugin)
sudo nvidia-ctk runtime configure --runtime=containerd --set-as-default
sudo systemctl restart containerd

2e. Enable IP forwarding and bridge netfilter

sudo modprobe br_netfilter
echo 1 | sudo tee /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
sudo tee /etc/sysctl.d/99-kubernetes.conf > /dev/null <<'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
sudo swapoff -a || true

3. Clone Kubernetes and Build Binaries

export PATH="/usr/local/go/bin:$HOME/go/bin:$PATH"

mkdir -p ~/go/src/k8s.io
cd ~/go/src/k8s.io
git clone --depth 1 https://github.com/kubernetes/kubernetes.git
cd kubernetes

# Fetch tags so kubeadm can parse the version
git fetch --tags --depth 1 origin
KUBE_GIT_VERSION=$(git describe --tags --match='v*' 2>/dev/null || echo "v1.34.0")

# Build kubeadm, kubelet, kubectl, e2e test binary, and ginkgo
make WHAT="cmd/kubeadm cmd/kubelet cmd/kubectl test/e2e/e2e.test vendor/github.com/onsi/ginkgo/v2/ginkgo" \
  KUBE_GIT_VERSION="$KUBE_GIT_VERSION"

sudo cp _output/local/go/bin/kubeadm /usr/local/bin/
sudo cp _output/local/go/bin/kubelet /usr/local/bin/
sudo cp _output/local/go/bin/kubectl /usr/local/bin/

kubeadm version  # verify

4. Set Up Single-Node Kubernetes Cluster

4a. Create kubelet systemd service

sudo tee /etc/systemd/system/kubelet.service > /dev/null <<'EOF'
[Unit]
Description=kubelet: The Kubernetes Node Agent
Wants=network-online.target
After=network-online.target

[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo mkdir -p /etc/systemd/system/kubelet.service.d
sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf > /dev/null <<'EOF'
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_EXTRA_ARGS
EOF

sudo systemctl daemon-reload
sudo systemctl enable kubelet

4b. Initialize the cluster

sudo kubeadm init \
  --pod-network-cidr=10.88.0.0/16 \
  --cri-socket=unix:///run/containerd/containerd.sock \
  --ignore-preflight-errors=NumCPU,Mem,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables,SystemVerification

4c. Configure kubectl and untaint the node

mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# Allow pods to schedule on the control-plane node
kubectl taint nodes --all node-role.kubernetes.io/control-plane-

4d. Fix pod-to-pod networking

The default iptables FORWARD policy is DROP, which blocks pod traffic through the bridge. Add explicit ACCEPT rules:

sudo iptables -I FORWARD -i cni0 -j ACCEPT
sudo iptables -I FORWARD -o cni0 -j ACCEPT
sudo iptables -I FORWARD -i cni0 -o cni0 -j ACCEPT
sudo iptables -t nat -A POSTROUTING -s 10.88.0.0/16 ! -o cni0 -j MASQUERADE

4e. Install NVIDIA device plugin

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.1/deployments/static/nvidia-device-plugin.yml

4f. Verify everything is working

# Wait for device plugin to roll out
kubectl -n kube-system rollout status daemonset/nvidia-device-plugin-daemonset --timeout=120s

# Check GPU appears in node capacity (may take ~30s)
kubectl get nodes -o jsonpath='{.items[0].status.capacity.nvidia\.com/gpu}'
# Should print: 1

# Check all pods are running
kubectl get pods -A

Expected output should show CoreDNS (1/1 Running), etcd, apiserver, controller-manager, scheduler, kube-proxy, and nvidia-device-plugin all running.

5. Run GPU Tests

All tests are in test/e2e/node/gpu.go. They are [Serial] and share a single GPU, so run them one at a time.

Important: --provider=aws is needed because the tests have SkipUnlessProviderIs("aws", "gce"). This is just a string check; it doesn't actually use AWS APIs.

cd ~/go/src/k8s.io/kubernetes
export KUBECONFIG=$HOME/.kube/config
mkdir -p /tmp/gpu-test-artifacts

Test 1: GPU Job (cupy vector addition, 5 completions)

_output/local/go/bin/ginkgo \
  -timeout=30m \
  -focus="should run gpu based jobs" \
  -skip="\[Flaky\]" \
  -v \
  _output/local/go/bin/e2e.test \
  -- \
  --provider=aws \
  --kubeconfig=$KUBECONFIG \
  --report-dir=/tmp/gpu-test-artifacts

Test 2: GPU matrix multiplication (TensorFlow 5000x5000)

_output/local/go/bin/ginkgo \
  -timeout=30m \
  -focus="should run gpu based matrix multiplication" \
  -skip="\[Flaky\]" \
  -v \
  _output/local/go/bin/e2e.test \
  -- \
  --provider=aws \
  --kubeconfig=$KUBECONFIG \
  --report-dir=/tmp/gpu-test-artifacts

Test 3: nvidia-smi + CUDA demo suite (deviceQuery, vectorAdd, bandwidthTest, busGrind)

_output/local/go/bin/ginkgo \
  -timeout=30m \
  -focus="should run nvidia-smi and cuda-demo-suite" \
  -skip="\[Flaky\]" \
  -v \
  _output/local/go/bin/e2e.test \
  -- \
  --provider=aws \
  --kubeconfig=$KUBECONFIG \
  --report-dir=/tmp/gpu-test-artifacts

Run all GPU tests sequentially

_output/local/go/bin/ginkgo \
  -timeout=60m \
  -focus="\[Feature:GPUDevicePlugin\]" \
  -skip="\[Flaky\]" \
  -v \
  _output/local/go/bin/e2e.test \
  -- \
  --provider=aws \
  --kubeconfig=$KUBECONFIG \
  --report-dir=/tmp/gpu-test-artifacts

Validated Results (Lambda A10, 2026-04-02)

Test Result Time
should run gpu based jobs PASSED ~104s
should run gpu based matrix multiplication PASSED ~129s
should run nvidia-smi and cuda-demo-suite PASSED ~211s

Find available GPU test names

grep -r 'It("' test/e2e/node/gpu.go

6. Gotchas

Problem Cause Fix
Device plugin: "Incompatible strategy detected auto" NVIDIA runtime not set as default in containerd Run nvidia-ctk runtime configure --runtime=containerd --set-as-default and restart containerd, then restart the device plugin pod
nvidia.com/gpu missing from node capacity Device plugin started before NVIDIA runtime was configured Delete the device plugin pod and let the daemonset recreate it
CoreDNS crash-looping Pod-to-pod networking broken (iptables FORWARD DROP) Add explicit iptables FORWARD ACCEPT rules for cni0 (see step 4d)
kubeadm: "Unable to parse output from Kubelet" Shallow git clone missing tags, version is v0.0.0-master Run git fetch --tags --depth 1 origin and rebuild with KUBE_GIT_VERSION
_output/ permission denied during build Previous sudo make test-* created root-owned files Run sudo chown -R ubuntu:ubuntu _output/
Test shows "0 of N Specs" Focus pattern didn't match, or [Serial] test skipped Check exact test name; use ginkgo directly for [Serial] tests
"PS1: unbound variable" Harmless bashrc warning Ignore it

7. Stop the Instance

When done, terminate to stop billing:

# From your Mac
lambdactl instances          # find the instance ID
lambdactl stop <instance-id>

Or check instances and stop from the Lambda web console.

Cost reminder: gpu_1x_a10 is $0.86/hr. Don't forget to stop it!

Quick Reference: What Lambda Provides vs What You Install

Component Lambda A10 ships with You install/configure
NVIDIA driver 570.x (already there)
nvidia-container-toolkit 1.17.x (already there)
containerd 1.7.x (CRI disabled) Enable CRI + set nvidia as default runtime
runc 1.2.x (already there)
CNI plugins (not installed) Install + configure bridge network
Go (not installed) Install matching kubernetes/go.mod version
kubeadm/kubelet/kubectl (not installed) Build from k8s source
NVIDIA device plugin (not installed) Deploy as DaemonSet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment