High Available Pi-hole failover cluster using Keepalived and Orbital Sync

High Available Pi-hole failover cluster using Keepalived and Orbital Sync.md

High Available Pi-hole failover cluster using Keepalived and Orbital Sync

Part of collection: Hyper-converged Homelab with Proxmox

Introduction

Keepalived is a Loadbalancer to add ‘high availability` to Linux systemen. See the Keepalived documentatie for more background information.

This is my implementation of two Pi-hole servers in a high availability configuration, with Keepalived for failover purposes and Orbital Sync for synchronizing the Pi-hole configuration.

Orbital Sync

For this to work and make sense, the two Pi-hole servers need to be in sync, I have Orbital Sync running as a Service inside my Docker Swarm. I only edit my Pi-hole configuration on the Master Node (192.168.1.155), from where the config synchronizes every 15 minutes to the Slave Node (192.168.1.166).

To use Oribital Sync; spin up a Docker Container based on the orbital-sync-yml configuration below.

Note: The Google SMTP Service I use, requires an App Passwords. To create it go to Google Account and 'Select Security / "Signing in to Google," / 2-Step Verification' At the bottom of that page, select App passwords and add one.

Setup Keepalived

This setup is using a virtual ip address: 192.168.1.3, which is the only ip address that DNS clients need to have configured as their DNS server.

Pi-hole - Master Node

On Pi-hole01: 192.168.1.155

Execute these commands:

sudo apt-get install keepalived -y

sudo mkdir /etc/scripts
sudo sh -c "curl https://gist.githubusercontent.com/Drallas/e61e4ebc1eaa2e52f21597fc8f2f43b8/raw/eca7ce1061e7fc4b2c801fe693aa02c34f758963/script-chk_ftl -o /etc/scripts/chk_ftl"
sudo chmod +x /etc/scripts/chk_ftl

# Add the configuration to the server
sudo curl https://gist.githubusercontent.com/Drallas/e61e4ebc1eaa2e52f21597fc8f2f43b8/raw/08ad959f6db894b0402a3ce7178d4da86f4f2d21/script-keepalived-master.conf -o /etc/keepalived/keepalived.conf

# Start keepalived
systemctl enable --now keepalived.service
systemctl status keepalived.service

Pi-hole - Slave Node

On Pi-hole02: 192.168.1.166

Execute these commands:

sudo apt-get install keepalived -y

sudo mkdir /etc/scripts
sudo sh -c "curl https://gist.githubusercontent.com/Drallas/e61e4ebc1eaa2e52f21597fc8f2f43b8/raw/eca7ce1061e7fc4b2c801fe693aa02c34f758963/script-chk_ftl | tr -d '\r' > /etc/scripts/chk_ftl"
sudo chmod +x /etc/scripts/chk_ftl

# Add the configuration to the server
sudo curl https://gist.githubusercontent.com/Drallas/e61e4ebc1eaa2e52f21597fc8f2f43b8/raw/08ad959f6db894b0402a3ce7178d4da86f4f2d21/script-keepalived-slave.conf -o /etc/keepalived/keepalived.conf

# Start keepalived
systemctl enable --now keepalived.service
systemctl status keepalived.service

Keepalived should now be active, time to test if it's working correctly.

Note: on Vm's vrrp_track_process can also be used as an alternative for vrrp_script, but on RaspberryPI PC's and LXC containers this doens't work!

Testing

Open the Pi-hole interface on the virtual IP we configured previously (e.g. http://192.168.1.3/admin) it should show the hostname of the Master Pi-hole (pihole01).

Shut down the Master Pi-hole (or systemctl stop pihole-FTL) and refresh the browser, it should now show the hostname of the Slave Pi-hole (pihole02).

Test Script

From a workstation use this commandline script:

while true; do nslookup gist.github.com 192.168.1.3; sleep 1; done

To test if DNS lookups are not interrupted when the Master Pi-hole Node is taken offline, stop the Pi-hole service systemctl stop pihole-FTL on the Pi-hole Master Node.

The nslookup command should keep on resolving uninterrupted.

Failover

To visualize the failover on Pi-hole Nodes; execute ip -br a and check if the virtual-ip (192.168.1.3), moves from the Master to the Slave Node, a few seconds after stopping the pihole-FTL service.

# Master Node
ip -br a
lo               UNKNOWN        127.0.0.1/8
eth0@if62        UP             192.168.1.155/24 192.168.1.3/24
ip -br a
lo               UNKNOWN        127.0.0.1/8
eth0@if62        UP             192.168.1.155/24


# Slave Node
ip -br a
lo               UNKNOWN        127.0.0.1/8
eth0@if26        UP             192.168.1.166/24
ip -br a
lo               UNKNOWN        127.0.0.1/8
eth0@if26        UP             192.168.1.166/24 192.168.1.3/24

Make sure to start the pihole-FTL service again: systemctl start pihole-FTL and that it's healthy systemctl status pihole-FTL.

orbital-sync.yml

# Orbital Sync
# <https://github.com/mattwebbio/orbital-sync#configuration>
# Synchronize multiple Pi-hole instances

# create an .env from the variable template below, inserting your own values.

# HOST_PASSWORD=yourpass
# PRIMARY_HOST_BASE_URL=<http://192.168.1.xxx/admin/>
# SECONDARY_HOST_1_BASE_URL=<http://192.168.1.xxx/admin/>
# SMTP_HOST=smtp.gmail.com
# EMAIL=youremail
# SMTP_PASSWORD=your-app-password
# TZ=Europe/Amsterdam
# SMTP_HOST=smtp.gmail.com

      PRIMARY_HOST_BASE_URL: ${PRIMARY_HOST_BASE_URL}
      PRIMARY_HOST_PASSWORD: ${HOST_PASSWORD}
      PRIMARY_HOST_PATH: /
      SECONDARY_HOST_1_BASE_URL: ${SECONDARY_HOST_1_BASE_URL}
      SECONDARY_HOST_1_PASSWORD: ${HOST_PASSWORD}
      SECONDARY_HOST_1_PATH: /
      INTERVAL_MINUTES: 15
      NOTIFY_ON_SUCCESS: 'false'
      NOTIFY_ON_FAILURE: 'true'
      NOTIFY_VIA_SMTP: 'true'
      VERBOSE: 'true' # On / Off tp Troubleshot
      SMTP_HOST: ${SMTP_HOST}
      SMTP_PORT: 587
      SMTP_USER: ${EMAIL}
      SMTP_PASSWORD: ${SMTP_PASSWORD}
      SMTP_TO: ${EMAIL}
      TZ: ${TZ}
    restart: "unless-stopped"

script-chk_ftl

#!/bin/bash

STATUS=$(ps ax | grep -v grep | grep pihole-FTL)

if [ "$STATUS" != "" ]
then
    exit 0
else
    exit 1
fi

script-keepalived-master.conf

global_defs {
    router_id pihole-dns-01
    script_user root
    enable_script_security
}

vrrp_script chk_ftl {
    script "/etc/scripts/chk_ftl"
    interval 1
    weight -10
}

vrrp_instance PIHOLE {
    state MASTER
    interface eth0
    virtual_router_id 55
    priority 150
    advert_int 1
    unicast_src_ip 192.168.1.155
    unicast_peer {
        192.168.1.166
    }

    authentication {
        auth_type PASS
        auth_pass 8kgEDPp3
    }

    virtual_ipaddress {
        192.168.1.3/24
    }

    track_script {
        chk_ftl
    }
}

script-keepalived-slave.conf

global_defs {
    router_id pihole-dns-02
    script_user root
    enable_script_security
}

vrrp_script chk_ftl {
    script "/etc/scripts/chk_ftl"
    interval 1
    weight -10
}

vrrp_instance PIHOLE {
    state BACKUP
    interface eth0
    virtual_router_id 55
    priority 145
    advert_int 1
    unicast_src_ip 192.168.1.166
    unicast_peer {
        192.168.1.155
    }

    authentication {
        auth_type PASS
        auth_pass 8kgEDPp3
    }

    virtual_ipaddress {
        192.168.1.3/24
    }

    track_script {
        chk_ftl
    }
}