Example of using Azure Monitor to scrape Prometheus metrics from NGINX

nginx-prometheus-azmon-example.md

Example of using Azure Monitor to scrape Prometheus metrics from NGINX

NGINX Ingress Controller w/ Helm

1. Enable metrics endpoint:

helm install stable/nginx-ingress --name my-release --set controller.metrics.enabled=true

2. Update Azure Monitor agent ConfigMap:

...
  prometheus-data-collection-settings: |-
    [prometheus_data_collection_settings.cluster]
      interval = "30s"
      kubernetes_services = ["http://nginx-ingress-controller-metrics.namespace:9913/metrics"]
...

3. View Data

InsightsMetrics
| extend tags=parse_json(Tags)
| where tostring(tags.controller_class) == "nginx" 

Calculate requests per minute:

InsightsMetrics
| where Name == "nginx_ingress_controller_nginx_process_connections_total" 
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc 
| extend RequestsPerMinute = Val - prev(Val) 
| sort by TimeGenerated desc 

Get a chart

 InsightsMetrics
| where Name == "nginx_ingress_controller_nginx_process_connections_total" 
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc 
| project RequestsPerMinute = Val - prev(Val), TimeGenerated 
| render barchart  

NGINX + NGINX Prometheus Exporter:

1. Edit default.conf to expose metrics.

server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    location = /stub_status {
        stub_status;
    }

}

Update Dockerfile:

FROM nginx
COPY default.conf /etc/nginx/conf.d/default.conf

2. Deploy updated NGINX container and NGINX Prometheus Exporter with annotations and args:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-exporter
  namespace: nginx-prometheus
spec:
  selector:
    matchLabels:
      app: nginx-exporter
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx-exporter
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9113"
    spec:
      containers:
        - name: nginx-exporter
          image: "nginx/nginx-prometheus-exporter:0.4.2"
          imagePullPolicy: "Always"
          args: ["-nginx.scrape-uri", "http://nginx/stub_status"]
          ports:
            - name: http
              containerPort: 9113
              protocol: TCP
          resources:
            {}

3. Update Azure Monitor agent ConfigMap:

...
  prometheus-data-collection-settings: |-
    [prometheus_data_collection_settings.cluster]
      interval = "30s"
      monitor_kubernetes_pods = true
...

4. View data:

InsightsMetrics
| extend tags=parse_json(Tags)
| where Name == "nginx_connections_accepted"

Calculate requests per minute:

InsightsMetrics
| where Name == "nginx_http_requests_total" 
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc 
| extend RequestsPerMinute = Val - prev(Val) 
| sort by TimeGenerated desc 

Get a chart

 InsightsMetrics
| where Name == "nginx_http_requests_total" 
| summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m)
| sort by TimeGenerated asc 
| project RequestsPerMinute = Val - prev(Val), TimeGenerated 
| render barchart