Docker Container Logging using Promtail

promtail_docker_logs.md

Inspired By: grafana/loki#333

• docker-compose.yml

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:1.4.1
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  promtail:
    image: grafana/promtail:1.4.1
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers
      - /home/ubuntu/docker-config.yml:/etc/promtail/docker-config.yml
    command: -config.file=/etc/promtail/docker-config.yml
    networks:
      - loki

  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"
    networks:
      - loki

• docker-config.yml

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

- job_name: containers
  entry_parser: raw

  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  # --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
  pipeline_stages:

  - json:
      expressions:
        stream: stream
        attrs: attrs
        tag: attrs.tag

  - regex:
      expression: (?P<image_name>(?:[^|]*[^|])).(?P<container_name>(?:[^|]*[^|])).(?P<image_id>(?:[^|]*[^|])).(?P<container_id>(?:[^|]*[^|]))
      source: "tag"

  - labels:
      tag:
      stream:
      image_name:
      container_name:
      image_id:
      container_id:

$ docker-compose up -d

$ docker run -itd --name nginxapp  -p 8080:80 --log-driver json-file --log-opt tag="{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}" nginx
$ curl http://localhost:8080/?foo=bar

Screenshot:

Setup with less labels:

• docker-config.yml

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: containers
  entry_parser: raw

  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      cluster: multipass-cluster
      __path__: /var/lib/docker/containers/*/*log

  # --log-opt tag="{{.Name}}"
  pipeline_stages:

  - json:
      expressions:
        stream: stream
        attrs: attrs
        tag: attrs.tag

  - regex:
      expression: (?P<container_name>(?:[^|]*[^|]))
      source: "tag"

  - labels:
      #tag:
      stream:
      container_name:

$ docker run -itd --name nginxapp3  -p 8080:80 --log-driver json-file --log-opt tag="{{.Name}}" nginx
$ curl -XGET -A "Mozilla" --refer http://bot.com/scrape.html http://localhost:8080/?foo=barx

Screenshot:

Files aren't copied in /var/lib/docker/containers. Do we need sudo privileges for it?

This worked for me for Docker Mac & Linux...

volumes:
  - /var/lib/docker/:/var/lib/docker:ro    

How can I cut the name into multiple and assign them to different tags, such as my container_name="k8s_httpbin_httpbin-2_lgy_ae8b8ca4-586e-4f60-b3b0-e80e1a4d0e72_0", but I want to convert this name into
pod="httpbin- 2"
namespace="lgy" this format

Since only root has access to /var/lib/docker and I'm running the Grafana agent under its own user on the host itself, changing the permissions to that directory seemed like a bad idea to me. After a bit of research I figured out how to make this work using the journald driver, which works nicely.

Docker Compose:

x-logging:
  &default-logging
  driver: "journald"
  options: 
    tag: "{{.Name}}"

services:
  # for each service you want to log
  some-service:
    # ...
    logging: *default-logging

Promtail (or Grafana agent) config:

      scrape_configs:
        - job_name: journal
          journal:
            labels:
              job: journal
          relabel_configs:
            # services
            - source_labels:
                - __journal__systemd_unit
              target_label: unit
            # docker containers
            - source_labels:
                - __journal_container_name
              target_label: container  # use whatever label you like

Hint: Promtail adds a __journal_ prefix to all fields - use something like this to see what's available:

journalctl -o json-pretty -u docker -n 10

Hi.
Thanks for your code. I appreciate it !

The question is about multiline stage. I configured promtail with less labels. But loki does not shows all the logs, shows only the last row.

• For example the actual log is:

warn: Ocelot.Responder.Middleware.ResponderMiddleware[0]
      requestId: GM:02, previousRequestId: no previous request id, message: Error Code: ConnectionToDownstreamServiceError Message: Error connecting to downstream service, exception: System.Net.Http.HttpRequestException: Connection refused (microservice:port)
       ---> System.Net.Sockets.SocketException (111): Connection refused
         at System.Net.Http.ConnectHelper.ConnectAsync(Func`3 callback, DnsEndPoint endPoint, HttpRequestMessage requestMessage, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
         at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
         at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
         at Ocelot.Requester.HttpClientHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/api/ping, request method: GET

• Shows only:

 at Ocelot.Requester.HttpClientHttpRequester.GetResponse(HttpContext httpContext) errors found in ResponderMiddleware. Setting error response for request path:/api/ping, request method: GET

Could you help me about that?

There is an approach the leaves the tag and the unnecessary any per-service logging: options: settings.

It requires an extra entry in daemon.json however - the addition of a log-opts:labels, or log-opts:labels-regex to allow the com.docker.* labels that docker attaches to containers when doing stack, service or compose deployments.

  "log-driver": "json-file",
  "log-opts": {
    "labels-regex": "^.+",
  }

Once this is done, one can verify the new labels being added to logs using --details

docker logs <container_id> --details

Now that these labels are present in the logs, promtails config can be:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

This

• sets the container_id from the path
• sets the timestamp correctly
• passes "tag" through so that it can be used for extra stack defined filtering
• generates "compose_project", "compose_name" labels for docker compose deployed containers
• generates "swarm_service_name", "swarm_task_name" for swarm services
• generates "stack_name" for containers deployed from a stack.

Thank you.
I'll try it and let you know the result

@chrisbecke this works great.

Thanks!

There is an approach the leaves the tag and the unnecessary any per-service logging: options: settings.

It requires an extra entry in daemon.json however - the addition of a log-opts:labels, or log-opts:labels-regex to allow the com.docker.* labels that docker attaches to containers when doing stack, service or compose deployments.

  "log-driver": "json-file",
  "log-opts": {
    "labels-regex": "^.+",
  }

Once this is done, one can verify the new labels being added to logs using --details

docker logs <container_id> --details

Now that these labels are present in the logs, promtails config can be:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

This

• sets the container_id from the path
• sets the timestamp correctly
• passes "tag" through so that it can be used for extra stack defined filtering
• generates "compose_project", "compose_name" labels for docker compose deployed containers
• generates "swarm_service_name", "swarm_task_name" for swarm services
• generates "stack_name" for containers deployed from a stack.

Hi!
I tried but it doesn't work for me.

I solved it by multiline

There is an approach the leaves the tag and the unnecessary any per-service logging: options: settings.

It requires an extra entry in daemon.json however - the addition of a log-opts:labels, or log-opts:labels-regex to allow the com.docker.* labels that docker attaches to containers when doing stack, service or compose deployments.

  "log-driver": "json-file",
  "log-opts": {
    "labels-regex": "^.+",
  }

Once this is done, one can verify the new labels being added to logs using --details

docker logs <container_id> --details

Now that these labels are present in the logs, promtails config can be:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

This

* sets the container_id from the path

* sets the timestamp  correctly

* passes "tag" through so that it can be used for extra stack defined filtering

* generates "compose_project", "compose_name" labels for docker compose deployed containers

* generates "swarm_service_name", "swarm_task_name" for swarm services

* generates "stack_name" for containers deployed from a stack.

Thanks for your post! Freakin' awesome - great to use promtail and avoid the loki plugin. I had some minor tweaks for my setup but everything works like a dream. Hats off for ya!

Only thing I'll call out - you need a Docker CE version new than early 2021. They added the labels-regex option to the JSON logs about that time.

There is an approach the leaves the tag and the unnecessary any per-service logging: options: settings.

It requires an extra entry in daemon.json however - the addition of a log-opts:labels, or log-opts:labels-regex to allow the com.docker.* labels that docker attaches to containers when doing stack, service or compose deployments.

  "log-driver": "json-file",
  "log-opts": {
    "labels-regex": "^.+",
  }

Once this is done, one can verify the new labels being added to logs using --details

docker logs <container_id> --details

Now that these labels are present in the logs, promtails config can be:

server:
  http_listen_address: 0.0.0.0
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:

- job_name: containers
  static_configs:
  - targets:
      - localhost
    labels:
      job: containerlogs
      __path__: /var/lib/docker/containers/*/*log

  pipeline_stages:
  - json:
      expressions:
        log: log
        stream: stream
        time: time
        tag: attrs.tag
        compose_project: attrs."com.docker.compose.project"
        compose_service: attrs."com.docker.compose.service"
        stack_name: attrs."com.docker.stack.namespace"
        swarm_service_name: attrs."com.docker.swarm.service.name"
        swarm_task_name: attrs."com.docker.swarm.task.name"
  - regex:
      expression: "^/var/lib/docker/containers/(?P<container_id>.{12}).+/.+-json.log$"
      source: filename
  - timestamp:
      format: RFC3339Nano
      source: time
  - labels:
      stream:
      container_id:
      tag:
      compose_project:
      compose_service:
      stack_name:
      swarm_service_name:
      swarm_task_name:
  - output:
      source: log

This

• sets the container_id from the path
• sets the timestamp correctly
• passes "tag" through so that it can be used for extra stack defined filtering
• generates "compose_project", "compose_name" labels for docker compose deployed containers
• generates "swarm_service_name", "swarm_task_name" for swarm services
• generates "stack_name" for containers deployed from a stack.

working, ty!

entry_parser: raw
it does not work

@chrisbecke you can remove entry_parser: raw if you want to use new versions of promtail
@Pl8tinium thank you very much, best solution for me!!!

@Pl8tinium dude you saved my life, no joke! thanks. I was dealing with the docker plugin but it locked all my containers, thanks to promtail and your answer i could implement the solution.

The docker plugin has a hard dependency to a loki instance. So if the Loki instance is down, you aren't able to start your containers, which is an absolute nogo. I would rather miss the log messages than miss my running containers...