Alert manager cluster mode

README.md

Alert manager cluster mode

Given 2 servers,

1 - Configure the Two Alertmanager Instances to Form a Cluster

1. On both servers, edit the Alertmanager unit file:

sudo vi /etc/systemd/system/alertmanager.service

2. In the ExecStart section of the Prometheus Server, add the private IP address of the Alertmanager 2 server using the cluster.peer flag:

    ExecStart=/usr/local/bin/alertmanager \
      --config.file /etc/alertmanager/alertmanager.yml \
      --storage.path /var/lib/alertmanager/ \
      --cluster.peer=10.0.1.102:9094

Note: When you add the cluster.peer line, be sure to add the \ at the end of the storage.path line.

3. Save and exit the file by pressing Escape followed by :x!.

4. SSH to AlertManager 2 server with:

ssh 10.0.1.102

5. In the ExecStart section of the Alertmanager 2 server, add the private IP address of the Prometheus Server using the cluster.peer flag:

$ sudo vi /etc/systemd/system/alertmanager.service

    ExecStart=/usr/local/bin/alertmanager \
      --config.file /etc/alertmanager/alertmanager.yml \
      --storage.path /var/lib/alertmanager/ \
      --cluster.peer=10.0.1.101:9094

Note: When you add the cluster.peer line, be sure to add the \ at the end of the storage.path line.

6. Save and exit the file by pressing Escape followed by :x!.

7. On both servers, reload the unit file:

sudo systemctl daemon-reload

8. On the Prometheus Server, restart Alertmanager:

sudo systemctl restart alertmanager

9. On the Alertmanager 2 server, enable and start Alertmanager:

sudo systemctl enable alertmanager
sudo systemctl start alertmanager

10. On both servers, check the status of Alertmanager:

sudo systemctl status alertmanager

11. They should both show a status of active (running).

12. On both servers, press Ctrl+C to exit the process.

13. Access the AlertManager on Prometheus Server instance in a new browser tab: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9093

14. Click Silences.

15. Click New Silence.

16. Set the following values:

    • Name: test
    • Value: 1
    • Creator: me
    • Comment: This is a test

17. Click Create.

18. Access the Alertmanager 2 instance in a new browser tab: http://<ALERTMANAGER_2_PUBLIC_IP>:9093

19. Click Silences, and verify the silence you created on the Prometheus Server instance appears.

20. Click View, and we should then see it's the one we created.

2 - Configure Prometheus to Use Your Multi-Instance Alertmanager Setup

1. Close out of the Alertmanager 2 terminal, as we will no longer be working in it.

2. On the Prometheus Server, edit the Prometheus configuration file:

sudo vi /etc/prometheus/prometheus.yml

3. Add the new Alertmanager (10.0.1.102:9093) to the list of Alertmanager targets:

alerting:
    alertmanagers:
    - static_configs:
    - targets:
        - localhost:9093
        - 10.0.1.102:9093

4. Save and exit the file by pressing Escape followed by :x!.

5. Restart Prometheus to reload the config:

sudo systemctl restart prometheus

6. Access the Prometheus server in the browser: http://<PROMETHEUS_SERVER_PUBLIC_IP>:9090

7. Click Status > Runtime & Build Information.

8. Verify both of your Alertmanagers appear under the Alertmanagers section.