Setting Up a Monitoring Stack with Grafana, Prometheus, Node Exporter, and cAdvisor in Docker Compose, Exposed via NGINX

1. GRAFANA.md

Setting Up a Monitoring Stack with Grafana, Prometheus (Mimir), Node Exporter, cAdvisor and Loki in Docker Compose, Exposed via NGINX

This guide provides instructions on setting up a comprehensive monitoring stack using Grafana, Prometheus, Node Exporter, cAdvisor and Loki. These components are orchestrated with Docker Compose and exposed via an NGINX reverse proxy, making them accessible through a single domain.

Components

• Grafana: The analytics and monitoring solution with support for multiple data sources, including Prometheus.
• Prometheus: The monitoring and alerting toolkit, collecting metrics from configured targets at specified intervals.
• Mimir: A horizontally-scalable, highly-available, multi-tenant log aggregation system.
• Node Exporter: A Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.
• cAdvisor: Analyzes resource usage and performance characteristics of running containers.
• Loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system.
• NGINX: Used as a reverse proxy to expose Grafana on the internet securely.

Prerequisites

• Docker and Docker Compose installed on your host machine.
• Domain name configured to point to the host machine (for NGINX configuration).

Configuration Files Overview

• docker-compose.yml: Defines the services, networks, and volumes for the Docker containers.
• prometheus.yml: Configuration file for Prometheus to define scrape targets and intervals.
• mimir.yml: Configuration file for Mimir to define the storage backend and other settings.
• grafana.ini: Configuration file for Grafana's settings, including SMTP for email notifications.
• loki-config.yml: Configuration file for Loki to define the storage backend and other settings.
• domain.conf: NGINX configuration for proxying requests to Grafana and handling WebSocket connections.

Installation Steps

1. Configure Docker Daemon: Enable Docker metrics by adding the following to /etc/docker/daemon.json and restarting Docker:

   {
     "metrics-addr": "127.0.0.1:9323"
   }
   # Then reload Docker configurations:
   sudo systemctl restart docker

2. Install Loki plugin for Docker: Install the Loki plugin for Docker to enable log collection:

   docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

3. Clone the Repository: Clone the repository to your host machine:

   git clone https://gist.github.com/398e94fb0aebb994b25223d99e7ae769.git monitoring-stack
   cd monitoring-stack

4. Start the Services: Start the monitoring stack using Docker Compose:

   docker-compose up -d

Note

The version with Mimir is available in the docker-compose-basic.yml file. Additionaly nginx and loki are not running in this version.

5. Access Grafana: Access Grafana at http://<your-domain> and log in with the default credentials (username: admin, password: admin).

Note

Important Notes:

• Replace placeholder values in grafana.ini with your actual SMTP credentials to enable email notifications.

• Adjust the domain.conf to match your domain and specific requirements.

• Review Docker and NGINX logs in case of any issues during setup.

docker-compose-basic.yml

volumes:
  prometheus_data: {}
  grafana_data: {}
  mimir_data: {}

services:
  node-exporter:
    image: prom/node-exporter:latest
    network_mode: host
    container_name: node-exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - "--path.procfs=/host/proc"
      - "--path.rootfs=/rootfs"
      - "--path.sysfs=/host/sys"
      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    expose:
      - 9100

  prometheus:
    image: prom/prometheus:latest
    network_mode: host
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=15d"
      - "--web.console.libraries=/etc/prometheus/console_libraries"
      - "--web.console.templates=/etc/prometheus/consoles"
      - "--web.enable-lifecycle"
      - "--web.enable-remote-write-receiver"
    expose:
      - 9090

  mimir:
    image: grafana/mimir:latest
    network_mode: host
    container_name: mimir
    restart: unless-stopped
    volumes:
      - ./mimir.yml:/etc/mimir.yml
      - mimir_data:/data
    command: ["-config.file=/etc/mimir.yml"]
    ports:
      - 9009

  grafana:
    image: grafana/grafana:latest
    network_mode: host
    container_name: grafana
    restart: unless-stopped
    volumes:
      - ./grafana.ini:/etc/grafana/grafana.ini
      - grafana_data:/var/lib/grafana
    ports:
      - 3000

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: host
    container_name: cadvisor
    restart: unless-stopped
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
    ports:
      - 8080

docker-compose.yml

version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    network_mode: "host"
    volumes:
      - "./loki-config.yml:/mnt/config/loki-config.yml"
      - "loki-data:/loki"
    command: -config.file=/mnt/config/loki-config.yml
    restart: unless-stopped
    expose:
      - "3100"
      - "9096"

  node-exporter:
    image: prom/node-exporter:latest
    network_mode: "host"
    pid: "host"
    volumes:
      - "/:/host:ro,rslave"
    command:
      - '--path.rootfs=/host'
    restart: unless-stopped
    expose:
      - "9100"

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    network_mode: "host"
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
    restart: unless-stopped
    expose:
      - "8080"

  prometheus:
    image: prom/prometheus:latest
    network_mode: "host"
    volumes:
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "prometheus-data:/prometheus"
    restart: unless-stopped
    expose:
      - "9090"

  grafana:
    image: grafana/grafana:latest
    network_mode: "host"
    volumes:
      - "./grafana.ini:/etc/grafana/grafana.ini"
      - "grafana-storage:/var/lib/grafana"
    restart: unless-stopped
    expose:
      - "3000"

  nginx:
    image: nginx:latest
    network_mode: "host"
    volumes:
      - "./domain.conf:/etc/nginx/conf.d/domain.conf:ro"
    logging:
      driver: "loki"
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
    restart: unless-stopped
    expose:
      - "80"

volumes:
  prometheus-data:
  grafana-storage:

domain.conf

map $http_upgrade $connection_upgrade {
  default upgrade;
  '' close;
}

upstream grafana {
  server localhost:3000;
}

server {
  listen 80;
  listen [::]:80;

  server_name grafana.diegocornejo.com;

  location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }

  # Proxy Grafana Live WebSocket connections.
  location /api/live/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
  location /api/datasources/proxy/ {
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;
    proxy_set_header Host $host;
    proxy_pass http://grafana;
  }
}

grafana.ini

[server]
domain = grafana.diegocornejo.com
root_url = https://grafana.diegocornejo.com
[smtp]
enabled = true
host = email-smtp.us-east-1.amazonaws.com:587
user = <username>
password = <password>
skip_verify = false
from_address = grafana@diegocornejo.com
from_name = Grafana
[live]
max_connections = 100
allowed_origins = https://grafana.diegocornejo.com

loki-config.yml

# This file has been downloaded from https://raw.githubusercontent.com/grafana/loki/v3.0.0/cmd/loki/loki-local-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
#  reporting_enabled: false

mimir.yml

multitenancy_enabled: false

server:
  http_listen_port: 9009

ingester:
  ring:
    replication_factor: 1

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
  - job_name: 'docker'
    static_configs:
      - targets: ['localhost:9323']
  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:8080']
 
 remote_write:
  - url: "http://remote.prometheus.com:9090/api/v1/write"
    write_relabel_configs:
      - source_labels: ["__address__"]
        target_label: "instance"
        replacement: "my-instance-name"
  # This is for send metrics to mimir      
  - url: "http://127.0.0.1:9009/api/v1/push"
  # On Grafana append /prometheus path for mimir datasource: http://localhost:9009/prometheus