Monitoring Nginx with InfluxDB - Telegraf - Prometheus - Grafana

README.md

Monitoring Nginx from Grafana with InfluxDB-Telegraf & Prometheus

InfluxDB for Time Database
Telegraf for Metrics Server Agent
Prometheus for Metrics Collector
Grafana for Metrics Visualizator

I just want to exploration about monitoring some metrcis Nginx with LoadBalancer Topology on Grafana. 
This is an continous project from [https://gist.github.com/gilangvperdana/53a70f9a64d3f7becebfa5f3a3d54c1f]

Environment

Ubuntu Server 20.04 LTS

Docker
Nginx
InfluxDB
Telegraf
Prometheus
Grafana

Installation

Docker

apt install -y docker.io
docker network create --subnet=172.19.0.0/16 nginxnet

Nginx

docker run --net nginxnet --ip 172.19.0.10 --name nginx-lbs -d -p 80:80 -p 443:443 docker pull gilangvperdana/research:nginx-lb-monitoring
docker run --net nginxnet --ip 172.19.0.11 --name nginx-1 -d gilangvperdana/research:nginx-1
docker run --net nginxnet --ip 172.19.0.12 --name nginx-2 -d gilangvperdana/research:nginx-2
docker run --net nginxnet --ip 172.19.0.13 --name nginx-3 -d gilangvperdana/research:nginx-3
docker run --net nginxnet --ip 172.19.0.14 --name nginx-final -d gilangvperdana/research:nginx-f

the basic difference about "nginx-lbs-metrics" and "nginx-lbs" is the difference in enabling "/nginx_status".
this is the addition :
---
# ACCESS METRICS ON /nginx_status
server {
    listen 80;
    listen [::]:80;
    server_name 172.19.0.10 172.19.0.10;

    location /nginx_status {
        stub_status on;
        access_log off;
    }
}
---

InfluxDB

docker run -d --name=influxdb \
 -p 8086:8086 \
 -v  /tmp/nginxnet/influx:/root/.influxdb2 \
      --net=nginxnet \
      --ip 172.19.0.2 \
      influxdb:2.0
docker exec -it influxdb influx setup

Telegraf

docker run -d --name=telegraf \
    -v /tmp/nginxnet/telegraf/telegraf.conf:/var/lib/influxdb \
    --net=nginxnet \
    --ip=172.19.0.3 \
    telegraf

Telegraf Configuration

wget https://gist.githubusercontent.com/gilangvperdana/12673aeb389763b292bba4455803437d/raw/cbbfc21359efde20dd412f0fac37c0f2758dde86/telegraf.conf
docker cp telegraf.conf telegraf:/etc/telegraf/

The difference lies in [[inputs.nginx]] , [[inputs.tail]] , [[outputs.prometheus_client]] , [[outputs.influxdb_v2]]
So, you must change "token" variable on [[outputs.influxdb_v2]]. You can define the variable with execute this on InfluxDB container :
$ influx auth list

then, copy token and define on telegraf container : /etc/telegraf/telegraf.conf. Define on "token" in [[outputs.influxdb_v2]].

docker exec -it telegraf bash
tail -f /var/log/telegraf/telegraf.log

Prometheus

docker run --name prometheus --net nginxnet --ip 172.19.0.50 -d prom/prometheus

Prometheus Configuration

wget https://gist.githubusercontent.com/gilangvperdana/12482197b63063774a5631284a4a81a9/raw/bee12112a65eb451893cd4186fde9a47dfa325d0/prometheus.yml
docker cp prometheus.yml prometheus:/etc/prometheus/

Grafana

 docker run --name grafana --net nginxnet --ip 172.19.0.20 -d grafana/grafana:8.4.4

Access Grafana Dashboard, add Prometheus for Data Source with 172.19.0.50:9090 then save.
Import dashboard with code 14900 or https://grafana.com/grafana/dashboards/14900 then use Prometheus for Data Source, save.
Happy monitoring !

Access

This is defined on Container IP which is can't access on public env, but you can access with SOCKSv5 technique.

https://labs.com/ for Frontend
http://172.19.0.10/nginx_status for Nginx Metrics Exposed

http://172.19.0.2:8086 for InfluxDB
http://172.19.0.3:8125/metrics for Telegraf
http://172.19.0.50:9090/ for Prometheus
http://172.19.0.20:3000 for Grafana

Inject Traffic

apt install -y siege

siege https://labs.com