Docker + nginx-proxy + let's encrypt + watchtower + fail2ban

Readme.md

Complete solution for websites hosting

This gist contains example of how you can configure nginx reverse-proxy with autmatic container discovery, SSL certificates generation (using Let's Encrypt) and auto updates.

Features:

• Automatically detect new containers and reconfigure nginx reverse-proxy
• Automatically generate/update SSL certificates for all specified containers.
• Watch for new docker images and update them.
• Ban bots and hackers who are trying to bruteforce your website or do anything suspicious.

Techonolgy stack:

• Docker, docker-compose
• jwilder/nginx-proxy - nginx reverse proxy
• jrcs/letsencrypt-nginx-proxy-companion - Let's encrypt SSL
• containrrr/watchtower - auto updater for containers
• crazymax/fail2ban - dynamic firewall
• nginx - sample service based on the nginx image

Watchtower

With watchtower you can update the running version of your containerized app simply by pushing a new image to the Docker Hub or your own image registry. Watchtower will pull down your new image, gracefully shut down your existing container and restart it with the same options that were used when it was deployed initially.

In this example we run watchtower with the following command: "--interval 60 --cleanup", so basically we ask watchtower to check for new images every minute and remove old images when udpate is performed.

Fail2Ban

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs -- too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured.

In this example I created two configuration rules - "wplogin" and "phpmyadmin".

• "wplogin" rule: if someone will make 3 POST call to "wp-login.php" within 10 minutes - consider it as a brute force attack and block such IP for 10 minutes.
• "phpmyadmin" rule: I don't host any "phpmyadmin" implementation on the website, so if someone (or some vulnerability scanner to be preciese) will try reach such URL - block his IP for an hour.

How to use fail2ban client?

# check status
$ sudo docker exec -t fail2ban fail2ban-client status
Status
|- Number of jail:	2
`- Jail list:	phpmyadmin, wplogin

# Check status of specific jail
$ sudo docker exec -t fail2ban fail2ban-client status phpmyadmin
Status for the jail: phpmyadmin
|- Filter
|  |- Currently failed:	0
|  |- Total failed:	2
|  `- File list: /container-logs/[....]-json.log
`- Actions
   |- Currently banned:	0
   |- Total banned:	2
   `- Banned IP list:	


# ban specific IP
$ sudo docker exec -t fail2ban fail2ban-client set <JAIL> banip <IP>

# unban specific IP
$ sudo docker exec -t fail2ban fail2ban-client set <JAIL> banip <IP>

custom-nginx.conf

# Context - http
# Location insdie docker container - /etc/nginx/conf.d/custom-nginx.conf:ro

client_max_body_size 32m;
gzip on;

# Restore real user IP if using Clouflare
# If you are using Clouflare or any other reverse-proxy - uncomment the following lines.
# Because Clouflare replaces user IP address we need to convert it back for correct work of fail2ban.
#set_real_ip_from 103.21.244.0/22;
#set_real_ip_from 103.22.200.0/22;
#set_real_ip_from 103.31.4.0/22;
#set_real_ip_from 104.16.0.0/12;
#set_real_ip_from 108.162.192.0/18;
#set_real_ip_from 131.0.72.0/22;
#set_real_ip_from 141.101.64.0/18;
#set_real_ip_from 162.158.0.0/15;
#set_real_ip_from 172.64.0.0/13;
#set_real_ip_from 173.245.48.0/20;
#set_real_ip_from 188.114.96.0/20;
#set_real_ip_from 190.93.240.0/20;
#set_real_ip_from 197.234.240.0/22;
#set_real_ip_from 198.41.128.0/17;
#set_real_ip_from 2400:cb00::/32;
#set_real_ip_from 2606:4700::/32;
#set_real_ip_from 2803:f800::/32;
#set_real_ip_from 2405:b500::/32;
#set_real_ip_from 2405:8100::/32;
#set_real_ip_from 2c0f:f248::/32;
#set_real_ip_from 2a06:98c0::/29;
#real_ip_header CF-Connecting-IP;
#real_ip_header X-Forwarded-For;

docker-compose.yml

version: '3'

services:

  # NGINX reverse-proxy
  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: nginx-proxy
    restart: unless-stopped
    ports: ["80:80", "443:443"]
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - ./custom-nginx.conf:/etc/nginx/conf.d/custom-nginx.conf:ro
      - ./nginx-data/conf.d:/etc/nginx/conf.d
      - ./nginx-data/vhost.d:/etc/nginx/vhost.d
      - ./nginx-data/html:/usr/share/nginx/html
      - ./nginx-data/certs:/etc/nginx/certs:ro
      - ./nginx-data/htpasswd:/etc/nginx/htpasswd:ro
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    environment:
      HTTPS_METHOD: "noredirect"
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "10"

  # Automatically add SSL certificates from Let's Encrypt
  nginx-letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: nginx-letsencrypt
    restart: unless-stopped
    environment:
      NGINX_PROXY_CONTAINER: nginx-proxy
      # Uncoment to test on Staging
      #ACME_CA_URI: https://acme-staging.api.letsencrypt.org/directory
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./nginx-data/conf.d:/etc/nginx/conf.d
      - ./nginx-data/vhost.d:/etc/nginx/vhost.d
      - ./nginx-data/html:/usr/share/nginx/html
      - ./nginx-data/certs:/etc/nginx/certs:rw
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "10"

  watchtower:
    image: containrrr/watchtower
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
    command: --interval 60 --cleanup
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "10"

networks:
  default:
    external:
      name: edge

fail2ban-docker-compose.yml

version: '3'

services:

  fail2ban:
    image: crazymax/fail2ban:latest
    container_name: fail2ban
    restart: "unless-stopped"
    network_mode: "host"
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - "./fail2ban-data:/data"
      - "/var/log:/var/log:ro"
      - "/var/lib/docker/containers/:/container-logs/:ro"
    env_file:
      - "./fail2ban.env"
    logging:
      driver: "json-file"
      options:
        max-size: "5m"
        max-file: "10"

fail2ban.env

TZ=America/Chicago

F2B_LOG_TARGET=STDOUT
F2B_LOG_LEVEL=INFO
F2B_DB_PURGE_AGE=1d
F2B_MAX_RETRY=3
F2B_DEST_EMAIL=RECEIPIENT_EMAIL
F2B_SENDER=SENDER_EMAIL
F2B_ACTION=%(action_mwl)s             <---- action_mwl - block + send email
F2B_IPTABLES_CHAIN=DOCKER-USER

SSMTP_HOST=smtp.sendgrid.net
SSMTP_PORT=587
SSMTP_USER=apikey
SSMTP_PASSWORD=YOUR_API_KEY
SSMTP_TLS=YES

jail.local

# File:
# "./fail2ban-data/jail.d/jail.local"

# The DEFAULT allows a global definition of the options.
# They can be override in each jail afterwards.

[DEFAULT]
# Number of seconds that a host is banned.
bantime  = 18000 # ban for 5 hours

# "ignoreip" can be an IP address, a CIDR mask or a DNS host.
# Fail2ban will not ban a host which matches an address in this list.
# Several addresses can be defined using space separator.
ignoreip = 127.0.0.1/8 73.176.154.35

# attempts must occur within the 10-minute
findtime = 600

# How many attempts can be made before a ban is imposed
maxretry = 3


[wplogin]
enabled = true
port = http,https
logpath = /container-logs/*/*-json.log
filter = wplogin
bantime = 600
maxretry = 3


[phpmyadmin]
enabled = true
port = http,https
logpath = /container-logs/*/*-json.log
filter = phpmyadmin
bantime = 3600
maxretry = 1

phpmyadmin.conf

# File:
# ./fail2ban-data/filter.d/wplogin.conf

[Definition]
failregex = {"log":"<HOST> -.*phpmyadmin.*
ignoreregex =

sample-service.yml

# Sample service
# VIRTUAL_HOST is used by "jwilder/nginx-proxy" to redirect traffic to right docker container.
# LETSENCRYPT_HOST and LETSENCRYPT_EMAIL are used by "jrcs/letsencrypt-nginx-proxy-companion" to create SSL certificates.
version: '3'

services:

  nginx:
    image: nginx
    container_name: my_super_service
    restart: unless-stopped
    environment:
      VIRTUAL_HOST: my-domain.com
      LETSENCRYPT_HOST: my-domain.com
      LETSENCRYPT_EMAIL: [email protected]

networks:
  default:
    external:
      name: edge

wplogin.conf

# File:
# ./fail2ban-data/filter.d/wplogin.conf

[Definition]
failregex = {"log":"<HOST> -.*POST.*wp-login.php.*
ignoreregex =

Thanks for the write-up, looking very good!

You might want to add your user to the docker group so you don't have to prefix sudo on every command:

$ sudo docker ...

becomes just

$ groups
... docker ...
$ docker ...

I had the problem that fail2ban banned IPs and added a correct iptables rule, though the rule was not processed correctly - banned IPs came through anyway. It seemed like the rule was not in the correct order. The solution for me was to specifically configuring the chain DOCKER-USER to be used in the fail2ban jail like this:

[FILTERNAME]
enabled = true  
chain = DOCKER-USER

Source: fail2ban/fail2ban#2292

PS: thanks for this awesome guide which really helped me for my project!

Incredibly helpful!

For me instead of defining F2B_ACTION and F2B_IPTABLES_CHAIN. I used the following in my applications jail.d (eg. app.jail)

action = iptables[name=app, port=https, protocol=tcp, chain=FORWARD]

Thanks for the inspiration, but this seems to have each jail process the logs for All docker containers on the system. That seems wasteful, and would lead to an infinite loop, since fail2ban docker will be processing it's own logs??

Also, in my case, one of my dockers had a different timezone configured, which seems to mess up all of the jails.

2021-08-10 21:43:33,649 fail2ban.filter         [1]: WARNING [nginx-http-auth] Simulate NOW in operation since found time has too large deviation None ~ 1628646213.6496086 +/- 60
2021-08-10 21:43:33,650 fail2ban.filter         [1]: WARNING [nginx-http-auth] Please check jail has possibly a timezone issue. Line with odd timestamp: {"log":"2021-08-10 21:43:33,641 fail2ban.filter         [1]: ERROR   Failed to process line: '{\"log\":\"Initializing files and folders...\\\\n\",\"stream\":\"stdout\",\"time\":\"2021-08-11T01:43:32.295184913Z\"}', caught exception: IndexError('string index out of range')\n","stream":"stdout","time":"2021-08-11T01:43:33.642316898Z"}

How can I configure the jails to only process the appropriate log file?

Thanks for the inspiration, but this seems to have each jail process the logs for All docker containers on the system. That seems wasteful, and would lead to an infinite loop, since fail2ban docker will be processing it's own logs??

Logs are placed on the docker host under /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log so I guess the easiest fix for making fail2ban process its own logs is using a different logging driver (like local or syslog) for the fail2ban container (see https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers)