gitlab-runner configuration file with docker runner for using NVIDIA GPUs (nvidia-docker)

README.md

Use Gitlab-CI with GPU support

Since gitlab-runner cannot be forced to use nvidia-docker wrapper, follow this steps:

1. Install all required software: docker, nvidia-docker, gitlab-ci-multi-runner
2. Execute: curl -s http://localhost:3476/docker/cli
3. Use that data to fill devices/volumes/volume_driver fields in /etc/gitlab-runner/config.toml

config.toml

concurrent = 1
check_interval = 0

[[runners]]
  name = "Docker runner <---complete-me--->"
  url = "https://<---complete-me---->"
  token = "28ce17edc8ea7437f3e49969c86341"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "nvidia/cuda"
    privileged = false
    disable_cache = false
    devices = ["/dev/nvidiactl", "/dev/nvidia-uvm", "/dev/nvidia-uvm-tools", "/dev/nvidia3", "/dev/nvidia2", "/dev/nvidia1", "/dev/nvidia0"]
    volumes = ["/cache", "nvidia_driver_384.81:/usr/local/nvidia:ro"]
    volume_driver = "nvidia-docker"
    shm_size = 0
  [runners.cache]

Do you guys know how to make it work with docker v19.03.2 which integrates native support for nvidia gpus?
The runtime = "nvidia"does not work anymore, containers should be executed with --gpus flag now.

docker run -it --rm --gpus all ubuntu nvidia-smi

it is an open issue and, looking at the comments, it does not seem to be fixed soon.

I am using Docker 19.03 together with nvidia-docker2. This provides the new --gpu switch, while keeping the compatibility with the old --runtime switch (refer to https://github.com/NVIDIA/nvidia-docker/tree/master#upgrading-with-nvidia-docker2-deprecated).