Grafana Loki logging Setup and Forward logs using Promtail

This page describes how to setup Grafana Promtail and Loki for sending the contents of local logs to the centralized Grafana Loki server or Grafana Cloud.

Prerequisties

Docker Engine
Docker Compose

Setup Grafana Loki

Grafana Loki is a log aggregation system designed to store and query logs from all your apps.

Docker Compose Structure

grafana-loki
├── docker-compose.yml
├── loki-config.yaml

Create a Docker compose configuration file named docker-compose.yml.

services:
  loki:
    image: grafana/loki:3.0.0
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml
      - /data:/loki/data
    entrypoint: >-
      loki -config.file=/etc/loki/loki-config.yaml
    networks:
      - logging-net

  grafana:
    image: grafana/grafana:11.3.0
    container_name: grafana
    ports:
      - "3000:3000"
    environment:
      - GF_PATHS_DATA=/var/lib/grafana
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD="admin"
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - loki
    networks:
      - logging-net

networks:
  logging-net:

volumes:
  grafana-data:
    driver: local

Create a Loki configuration file named loki-config.yaml.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /loki/data

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  filesystem:
    directory: /loki/data/chunks

compactor:
  retention_enabled: true
  delete_request_store: filesystem

limits_config:
  retention_period: 168h

Setup Grafana Promtail Agent

Grafana Promtail is an agent that ships and forward the contents of local logs to the Grafana Loki server. It is usually deployed to every server that runs apps.

Docker Compose Structure

promtail
├── docker-compose.yml
└── promtail-config.yaml

Create a Docker compose configuration file named docker-compose.yml.

services:
  grafana-promtail:
    container_name: grafana-promtail
    image: grafana/promtail:3.0.0
    volumes:
      - ./promtail-config.yaml:/etc/promtail/promtail-config.yaml
      - /var/log:/var/log
    entrypoint: >-
      promtail -config.file=/etc/promtail/promtail-config.yaml

Forwarding Logs from file to Grafana Loki

Create a Promtail configuration file named promtail-config.yaml.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: "http://loki:3100/loki/api/v1/push"

scrape_configs:
  - job_name: odoo-system
    static_configs:
      - targets:
          - localhost
        labels:
          job: dnf-logs
          __path__: /var/log/dnf5.log
      - targets:
          - localhost
        labels:
          job: local-odoo-logs
          __path__: /var/log/odoo.log

Forwarding Container stdout Logs to Grafana Loki

Install the Docker Loki logging driver client on each Docker host.

$ docker plugin install grafana/loki-docker-driver:2.9.2 --alias loki --grant-all-permissions

$ docker plugin ls
ID             NAME          DESCRIPTION           ENABLED
6405b734a30c   loki:latest   Loki Logging Driver   true

Add the Loki logging driver in the /etc/docker/daemon.json file.

For example,

{
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "http://172.16.x.x:3100/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}

(OR)

Add the Loki logging drive in the Docker compose configuration file.