nftables with docker

docker-nftables.conf

# /etc/systemd/system/docker.service.d/docker-nftables.conf
# disable iptables in docker, allowing nftables to do work
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false

nftables.conf

#!/usr/bin/nft -f
# /etc/nftables.conf

table inet filter {
  chain input {
    type filter hook input priority 0;

    # allow established/related connections
    ct state {established, related} counter accept

    # early drop of invalid connections
    ct state invalid counter drop

    # allow from loopback
    iifname lo counter accept

    # allow icmp
    ip protocol icmp counter accept
    ip6 nexthdr icmpv6 counter accept

    # allow ssh
    # tcp dport ssh counter accept

    # everything else
    counter reject with icmp type port-unreachable
  }
  chain forward {
    type filter hook forward priority 0;
#    drop
  }
  chain output {
    type filter hook output priority 0;
  }

}

table ip nat {
  chain prerouting {
    type nat hook prerouting priority 0;
  }
  chain postrouting {
    type nat hook postrouting priority 0;
    oifname "eno1" counter masquerade
  }
}

prep_forwarding.sh

#!/bin/sh

cat > /etc/systemd/network/ipforward.network <<EOF
[Network]
IPForward=ipv4
EOF

cat > /etc/systemd/network/99-docker.conf <<EOF
net.ipv4.ip_forward = 1
EOF

sysctl -w net.ipv4.ip_forward=1

The /etc/systemd/system/docker.service.d/docker-nftables.conf that you have causes systemctl to just hang in RHEL 8.1.

For one:

ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false

should be

ExecStart=/usr/bin/dockerd daemon -H fd:// --iptables=false

But even if you do that and then try to start the service:

[ec2-user@ip-172-31-77-155 ~]$ sudo systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
[ec2-user@ip-172-31-77-155 ~]$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─docker-nftables.conf
   Active: failed (Result: exit-code) since Tue 2020-03-24 14:04:04 UTC; 15ms ago
     Docs: https://docs.docker.com
  Process: 5624 ExecStart=/usr/bin/dockerd daemon -H fd:// --iptables=false (code=exited, status=1/FAILURE)
 Main PID: 5624 (code=exited, status=1/FAILURE)

Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: docker.service: Service RestartSec=2s expired, scheduling restart.
Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: Stopped Docker Application Container Engine.
Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: docker.service: Start request repeated too quickly.
Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 24 14:04:04 ip-172-31-77-155.ec2.internal systemd[1]: Failed to start Docker Application Container Engine.

I don't think you're supposed to have ExecStart again in that file. Those files supplement the systemd unit file, not override it:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-Managing_Services_with_systemd-Unit_Files#sect-Managing_Services_with_systemd-Unit_File_Modify

I just added the --iptables=false to the main docker.service file.

I'm not sure that you can. Docker still uses IPTables so there has be to a shim somewhere. I this gist is trying to do that but I was not able to get it to work. I did get docker-ce and iptables to work on RHEL 8:

https://gist.github.com/dmc5179/2f55cd54a6fdd103ab1873d52e3464a8

Change ExecStart= on the second line to ExecStart=/usr/bin/dockerd -H fd:// --iptables=false and it should work.

Also should use /etc/docker/daemon.json, settings {"iptables": false}, rather than using a custom systemd config file (or service).