monitoring docker containers in mesos with prometheus and cadvisor and grafana

mesos_cadvisor_prometheus_grafana

# This marathon.json deploys cadvisor to each node and exposes it on port 3002.

cadvisor/marathon.json
{
  "id": "cadvisor",
  "cpus": 0.1,
  "mem": 100,
  "disk": 0,
  "instances": 8,                // equal to number of agent nodes you have
  "constraints": [["hostname", "UNIQUE"]],
  "container": {
    "type": "DOCKER",
    "volumes": [
      {
        "containerPath": "/rootfs",
        "hostPath": "/",
        "mode": "RO"
      },
      {
        "containerPath": "/var/run/",
        "hostPath": "/var/run",
        "mode": "RW"
      },
      {
        "containerPath": "/sys",
        "hostPath": "/sys",
        "mode": "RO"
      },
      {
        "containerPath": "/var/lib/docker",
        "hostPath": "/var/lib/docker",
        "mode": "RO"
      }
    ],
    "docker": {
      "image": "google/cadvisor:latest",
      "network": "BRIDGE",
      "portMappings": [
        {
          "containerPort": 8080,
          "hostPort": 3002,
          "protocol": "tcp"
        }
      ],
      "forcePullImage": true
    }
  }
}



# I'm deploying prometheus as a docker container.  I have a drone job that builds it on commit to github, then I just restart the service in mesos to grab the new config.
# prometheus.yml
  - job_name: 'mesos_cadvisor'
    scrape_interval: 1m
    static_configs:
      - targets: ['10.0.0.1:3002','10.0.0.2:3002'...]
      
      
      
      
# Dockerfile
FROM ubuntu:16.04

RUN apt-get update
RUN apt-get install golang git make -y
RUN export GOPATH='/' \
  && mkdir -p $GOPATH/src/github.com/prometheus \
  && cd $GOPATH/src/github.com/prometheus && git clone https://github.com/prometheus/prometheus.git \
  && cd prometheus && make build
COPY prometheus.yml .
CMD /src/github.com/prometheus/prometheus/prometheus -config.file /prometheus.yml



I'm running grafana external to mesos for no particular reason that I remember, but that's pointed as prometheus.
I grabbed a nice pre-defined docker-container dashboard here: https://grafana.net/dashboards/193


The resulting dashboard has container names like mesos-<impossibly long uuid>.  It's not pretty and I haven't yet found an easy way to find which task is associated with which container, but it's a start.