Custom Driver Install for NVIDIA on GKE

daemonset-preloaded.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nvidia-driver-installer-cos
  namespace: kube-system
  labels:
    k8s-app: nvidia-driver-installer-cos
spec:
  selector:
    matchLabels:
      k8s-app: nvidia-driver-installer-cos
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        name: nvidia-driver-installer-cos
        k8s-app: nvidia-driver-installer-cos
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cloud.google.com/gke-accelerator
                operator: Exists
            - matchExpressions:
              - key: cloud.google.com/gke-os-distribution
                operator: In
                values:
                - "cos"
      tolerations:
      - operator: "Exists"
      hostNetwork: true
      hostPID: true
      volumes:
      - name: dev
        hostPath:
          path: /dev
      - name: vulkan-icd-mount
        hostPath:
          path: /home/kubernetes/bin/nvidia/vulkan/icd.d
      - name: nvidia-install-dir-host
        hostPath:
          path: /home/kubernetes/bin/nvidia
      - name: root-mount
        hostPath:
          path: /
      - name: cos-tools
        hostPath:
          path: /var/lib/cos-tools
      initContainers:
      - image: "cos-nvidia-installer:fixed"
        imagePullPolicy: Never
        name: nvidia-driver-installer
        resources:
          requests:
            cpu: 0.15
        securityContext:
          privileged: true
        env:
        - name: NVIDIA_DRIVER_VERSION
          value: "440.64.00"
        - name: NVIDIA_INSTALL_DIR_HOST
          value: /home/kubernetes/bin/nvidia
        - name: NVIDIA_INSTALL_DIR_CONTAINER
          value: /usr/local/nvidia
        - name: VULKAN_ICD_DIR_HOST
          value: /home/kubernetes/bin/nvidia/vulkan/icd.d
        - name: VULKAN_ICD_DIR_CONTAINER
          value: /etc/vulkan/icd.d
        - name: ROOT_MOUNT_DIR
          value: /root
        - name: COS_TOOLS_DIR_HOST
          value: /var/lib/cos-tools
        - name: COS_TOOLS_DIR_CONTAINER
          value: /build/cos-tools
        volumeMounts:
        - name: nvidia-install-dir-host
          mountPath: /usr/local/nvidia
        - name: vulkan-icd-mount
          mountPath: /etc/vulkan/icd.d
        - name: dev
          mountPath: /dev
        - name: root-mount
          mountPath: /root
        - name: cos-tools
          mountPath: /build/cos-tools
      containers:
      - image: "gcr.io/google-containers/pause:2.0"
        name: pause

I used above code but it is not going into ready state

NAMESPACE     NAME                          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR           
kube-system   nvidia-driver-installer-cos   1         1         0       1            0           <none>    

But if iam removing NVIDIA_DRIVER_VERSION from env,it is running...

can you please help me with this @allanlei

@adityaSharma369 Unfortunately, you haven't provided any information for me to help you. It would also be very beneficial to know how to manage and debug kubernetes before using GPUs.

I tested this with "510.47.03" but it does not do much, it still installs the default version "450.119.04".