Romil Bhardwaj romilbhardwaj

Using local GPUs with SkyPilot + Kubernetes

This is a guide to using GPUs on your local machine with SkyPilot. This guide sets up a Kubernetes cluster (using KinD) so you can use SkyPilot's Kubernetes support to get it running.

Inspired by Klueska's comment and Sam Stoelinga's blog post.

Prerequisites

Docker
SkyPilot
NVIDIA Container Toolkit. If not installed, follow guide below.

Rancher Installation Guide for SkyPilot

Rancher should be super easy to use to setup a k8s cluster. It gives you one line commands to run on your worker nodes to connect them to your k8s cluster.

Their official docs are a little overwhelming, so here's a compressed guide:

On the master node (lets call it master), run sudo docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher . This will launch the rancher web UI.
To access this from your laptop, you may need to port forward port 443 to your laptop. Do so with ssh -L 8888:localhost:443 test
Open localhost:8888 in your browser. Complete the initial setup and set any password for the webui. It will also ask for an address accessible from all nodes. This field will be prepopulated with https://localhost:8888. Change this to the ip address of the node (get with ip a, should be accessible from other nodes as well) and don't specify port. E.g., https://10.35.0.21/.
You'll see the rancher UI. Click on Create clus

	import boto3
	import paramiko
	import time

	# AmzLinux Deep learning AMI: ami-0407d462d1919a19a
	# Amazon linux AMI: ami-0c2ab3b8efb09f272

	private_key_path = '<PATH_TO_PEM>' # Set to the path of your private key
	aws_keypair = '<KEYPAIR_NAME>' # Should already exist in your AWS account
	ami = 'ami-0407d462d1919a19a'

	sudo usermod -aG docker $USER
	# Install KIND
	# For AMD64 / x86_64
	[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
	# For ARM64
	[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-arm64
	chmod +x ./kind
	sudo mv ./kind /usr/local/bin/kind

	# Install Kubectl

	apiVersion: kyverno.io/v1
	kind: ClusterPolicy
	metadata:
	name: mutate-dws-pod-template
	annotations:
	policies.kyverno.io/title: Remove incompatible resources from PodTemplates
	policies.kyverno.io/category: Other
	policies.kyverno.io/severity: low
	kyverno.io/kyverno-version: 1.10.1
	policies.kyverno.io/minversion: 1.6.0

	https://github.com/skypilot-org/skypilot/blob/master/examples/torch_ddp_benchmark/torch_ddp_benchmark.yaml
	2x A100:8 nodes on GCP.
	$ sky launch -c a100 examples/torch_ddp_benchmark/torch_ddp_benchmark.yaml

	With gVNIC
	(head, rank=0, pid=7056) -----------------------------------
	(head, rank=0, pid=7056) PyTorch distributed benchmark suite
	(head, rank=0, pid=7056) -----------------------------------
	(head, rank=0, pid=7056)
	(head, rank=0, pid=7056) * PyTorch version: 2.4.1+cu121