device-plugin.md

Let's say our device plugin (The daemon set running on a node) sends a Register() call to the kubelet with RegisterRequest being:

{
        version: "1"
        endpont: "/var/run/gpu/intel.sock"
        resource_name: "intel.com/gpu"
}

kubelet gets the registration request and does not update the node status yet. It sends a ListAndWatch() request to our plugin to know more about the devices that we support. From the device plugin we look at the number of Intel GPUs available on the node and find that there are 2 of them. We then send to the kubelet a ListAndWatchResponse that will look like:

{
        {
                ID: "gpu-1234"
                health: "healthy"
        },
	{
                ID: "gpu-5678"
                health: "healthy"
        },
}

From there, kubelet will update the node status back to the API server to let it know that there are 2 intel.com/gpu devices available on the node.

Then at some point a user sends a pod spec that looks like this:

apiVersion: v1
kind: Pod
metadata:
  name: gpu-example
  spec:
  containers:
  - name: foobar
    image: sameo/gpu-example
    resources:
      requests:
        intel.com/gpu: "1"

The scheduler knows that there is such resource available in our node and may send it to our kubelet. If that's the case kubelet picks a device from the available list of devices it build from our last ListAndWatchResponse it got from our device plugin and calls our device plugin Allocate() gRPC with the following AllocateRequest:

{
        {
                "gpu-5678" // It could be gpu-1234 as well
        },
}

So the bottom line is: kubelet picks the device out of the list of devices the device plugin advertised.