docker centos/rhel recommendations

gistfile1.txt

# Docker 17.03.2 on RHEL/CentOS 7

Rancher has discovered a few issues when running Upstream Docker 17.03.2 on RHEL/CentOS 7. This document is being written to document recommendations for Docker configuration in order to ensure reliability while operating Kubernetes and Rancher with RHEL/CentOS 7.

### Overlay2 Storage Driver

Currently, Upstream Docker 17.03.2 performs kernel version validations to enable overlay2. As official overlay2 support from the upstream Linux kernel was not enabled until version 4.0 or higher than the kernel. Red Hat backported overlay2 support to 3.10.0-514 of their kernel.

Overlay2 support can be enabled by setting the following contents in the `/etc/docker/daemon.json` file and restarting Docker. Please note that you must do this on a fresh docker installation, or remove all running containers before performing this action.
```{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}```

### Cgroup Driver

Docker 17.03.2 running on RHEL/CentOS 7 with a containerized kubelet has an issue where cgroup `memory_kmem_usage` leaks for pods that are in constant `oom` state. This causes the kernel to eventually immediately kill all child processes for the container, which causes the kubelet to thrash while creating the new container processes. Eventually, docker can enter a hung/race-condition state.

This can be avoided by changing the cgroup driver for Docker from `cgroupfs` to the `systemd` driver by enabling the following setting in the `/etc/docker/daemon.json` file and restarting Docker.

```{
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ]
}```


### Setting these Options

Both of the above options can both be set in the `/etc/docker/daemon.json` file with the following contents:
```{
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ],
  "exec-opts": [
    "native.cgroupdriver=systemd"
  ]
}```