Gitlab runner: Rootless Docker Executor on Ubuntu 22.04

installation.md

Setup a rootless Docker build compatible Gitlab runner

Install Docker engine (tested with v20.10.17) and Gitlab runner (tested with v15.2.1) as described here and here.

Identify the UID of the gitlab-runner user.

$ id gitlab-runner
uid=999(gitlab-runner) gid=998(gitlab-runner) groups=998(gitlab-runner)

Do not forget to install the namespace packages described here and to specify UID ranges for the gitlab-runner user.

$ cat /etc/subuid
gitlab-runner:100000:65536
$ cat /etc/subgid
gitlab-runner:100000:65536

Install a rootless Docker service for a user. Take the rootless Docker service configuration file from this user (/home/the-user/.config/systemd/user/docker.service) and extend it by the following options.

User=gitlab-runner
Group=gitlab-runner
Environment=XDG_RUNTIME_DIR=/run/user/999
Environment=HOME=/home/gitlab-runner

Then, install it as a Docker rootless service in a system location.

$ cat /etc/systemd/system/docker-rootless.service 
[Unit]
Description=Docker Application Container Engine (Rootless)
Documentation=https://docs.docker.com/go/rootless/

[Service]
Environment=PATH=/usr/bin:/sbin:/usr/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
ExecStart=/usr/bin/dockerd-rootless.sh 
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
Type=simple
KillMode=mixed
User=gitlab-runner
Group=gitlab-runner
Environment=XDG_RUNTIME_DIR=/run/user/999
Environment=HOME=/home/gitlab-runner

[Install]
WantedBy=default.target

Make sure that the gitlab-runner is always "logged in" so that XDG_RUNTIME_DIR expected by the Docker rootless daemon does exist. (I do not know if the loginctl-approach "survives" machine restarts.)

$ sudo loginctl enable-linger gitlab-runner

Finally, "patch" the Gitlab runner systemd service using the following options.

User=gitlab-runner
Group=gitlab-runner
Environment=DOCKER_HOST=unix:///run/user/999/docker.sock
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/home/gitlab-runner/config.toml" "--service" "gitlab-runner" "--user" "gitlab-runner"

Which leads to this configuration in a system location.

$ cat /etc/systemd/system/gitlab-runner.service 
[Unit]
Description=GitLab Runner
ConditionFileIsExecutable=/usr/bin/gitlab-runner
 
After=syslog.target network.target 

[Service]
StartLimitInterval=5
StartLimitBurst=10
User=gitlab-runner
Group=gitlab-runner
Environment=DOCKER_HOST=unix:///run/user/999/docker.sock
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/home/gitlab-runner/config.toml" "--service" "gitlab-runner" "--user" "gitlab-runner"

Restart=always

RestartSec=120
EnvironmentFile=-/etc/sysconfig/gitlab-runner

[Install]
WantedBy=multi-user.target

You can then proceed to register Gitlab runners (sudo gitlab-runner register) that will use the rootless Docker daemon as their Docker executor.

$ sudo cat /home/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-build-rootless"
  url = "..."
  token = "..."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:20.10.17"
    privileged = true # for docker-service approach
    #privileged = false # for docker.sock approach
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    #volumes = ["/run/user/999/docker.sock:/var/run/docker.sock", "/cache"] # for docker.sock approach
    shm_size = 0

Note, these rootless Docker limitations, which in terms of functionality might not be a problem for the Docker build use case considered here.

usage.md

.gitlab-ci.yml example

image: docker:20.10.17

variables:
  DOCKER_HOST: tcp://dockerservice:2375
  DOCKER_TLS_CERTDIR: ""

services:
  - name: docker:20.10.17-dind
    alias: dockerservice

explore_the_docker_host_system:
  script:
    - docker run -v /:/root/host ubuntu:22.04 ls -la /root/host

build_some_docker_image:
  script:
    - docker build --no-cache -t test-unpriv-docker-ci .

Hi! First of all, thanks for this excellent script! However, I’m a bit confused about some of the steps in your guide.

Insert lines to /home/the-user/.config/systemd/user/docker.service.

You advise adding the following lines to /home/the-user/.config/systemd/user/docker.service:

User=gitlab-runner
Group=gitlab-runner
Environment=XDG_RUNTIME_DIR=/run/user/999
Environment=HOME=/home/gitlab-runner

After doing so, I was unable to restart the Docker service for my user, and it failed with the following error:

$ systemctl --user status docker.service
× docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/my-user/.config/systemd/user/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2025-01-17 08:45:32 UTC; 42s ago
       Docs: https://docs.docker.com/go/rootless/
    Process: 4816 ExecStart=/home/my-user/bin/dockerd-rootless.sh --iptables=false (code=exited, status=216/GROUP)
   Main PID: 4816 (code=exited, status=216/GROUP)
        CPU: 1ms

Jan 17 08:45:30 my-glrunner02 systemd[4240]: docker.service: Main process exited, code=exited, status=216/GROUP
Jan 17 08:45:30 my-glrunner02 systemd[4240]: docker.service: Failed with result 'exit-code'.
Jan 17 08:45:30 my-glrunner02 systemd[4240]: Failed to start Docker Application Container Engine (Rootless).
Jan 17 08:45:32 my-glrunner02 systemd[4240]: docker.service: Scheduled restart job, restart counter is at 3.
Jan 17 08:45:32 my-glrunner02 systemd[4240]: Stopped Docker Application Container Engine (Rootless).
Jan 17 08:45:32 my-glrunner02 systemd[4240]: docker.service: Start request repeated too quickly.
Jan 17 08:45:32 my-glrunner02 systemd[4240]: docker.service: Failed with result 'exit-code'.
Jan 17 08:45:32 my-glrunner02 systemd[4240]: Failed to start Docker Application Container Engine (Rootless).
my-user@my-glrunner02:~$ journalctl --user -xeu docker.service

I understand the intention behind this code snippet, but is there an extra step required to make it work? My GitLab Runner user, of course, does not have access to the services of my systemd configuration—and vice versa. Could this mismatch be causing the issue?

Therefor I just continued your guide which leads me to another confusion;

Docker rootless service in a system location

Next, I’m confused by your instruction to:

Then, install it as a Docker rootless service in a system location.

What exactly does this mean, and how can I achieve it? Since I have Docker installed as a rootless instance, there is no system-wide Docker service. Additionally, Docker explicitly states on their documentation page:

Starting Rootless Docker as a systemd-wide service (/etc/systemd/system/docker.service) is not supported, even with the User= directive.

I attempted to create the file and paste your configuration into it, but this didn’t resolve the issue. Could you clarify how to handle this step?

"patch" the Gitlab runner systemd service

Lastly, you mention:

Finally,` "patch" the Gitlab runner systemd service using the following options.

User=gitlab-runner
Group=gitlab-runner
Environment=DOCKER_HOST=unix:///run/user/999/docker.sock
ExecStart=/usr/bin/gitlab-runner "run" "--working-directory" "/home/gitlab-runner" "--config" "/home/gitlab-runner/config.toml" "--service" "gitlab-runner" "--user" "gitlab-runner"

Which leads to this configuration in a system location.

Could you provide more details about this step? Are you referring to a user-based GitLab Runner service? If so, how does this configuration affect the existing system-wide service?

Thank you for your time, and I appreciate any clarifications you can provide!