Example filebeat monitoring setup

.env

ES_HOSTS=https://(PRODUCTION ES):443
ES_API_KEY=(API KEY FOR WRITING LOGS)
ES_UUID=(PRODUCTION ES UUID)
ES_MONITOR_HOSTS=https://(MONITOR ES):443
ES_MONITOR_API_KEY=(API KEY FOR WRITING MONITORING DATA)

README.md

Post: https://discuss.elastic.co/t/filebeat-monitoring-metrics-not-visible-in-elasticsearch/315596/1

docker-compose.yml

version: "3.8"
services:
  logger:
    image: debian:stable-slim
    command: /usr/local/bin/logger.sh
    volumes:
      - ./logger.sh:/usr/local/bin/logger.sh
      - logs:/logs
  filebeat:
    image: elastic/filebeat:8.4.3
    environment:
      - ES_HOSTS
      - ES_API_KEY
      - ES_UUID
      - ES_MONITOR_HOSTS
      - ES_MONITOR_API_KEY
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
      - logs:/logs

volumes:
  logs:

filebeat.yml

filebeat.inputs:
- type: filestream
  id: logs
  paths:
    - /logs/*.log

monitoring:
  enabled: true
  cluster_uuid: "${ES_UUID}"
  elasticsearch:
    hosts: "${ES_MONITOR_HOSTS}"
    api_key: "${ES_MONITOR_API_KEY}"

output.elasticsearch:
  hosts: "${ES_HOSTS}"
  api_key: "${ES_API_KEY}"

logger.sh

#!/usr/bin/env bash

while true; do
    echo "$(date): mark" >> /logs/logger.log
    sleep 1
done

Hi Matt,

Is there a way to programmatically obtain the UUID of the cluster? (https://www.elastic.co/guide/en/beats/filebeat/current/monitoring-metricbeat-collection.html)

I'm trying to follow your conversation about Filebeat not showing up in the Stack Monitoring page (https://discuss.elastic.co/t/filebeat-monitoring-metrics-not-visible-in-elasticsearch/315596/6), it took me here.

The only way that I can think of getting that value is by logging into elastic and getting it from the response, which is not a good way if I want to automate the whole creation of the monitoring setup.

Hey, it’s been a year or so since I left elastic, so no idea if this is up to date info, but pretty sure we would just query the cluster with the monitoring role to get the uuid.

I’d recommend downloading and running a trial of elastic cloud enterprise and looking at how the monitoring components work there. You should be able to mirror the setup.