Monitor all system usage and by process with Prometheus and Grafana.

system-monitor-setup.md

Install Prometheus

sudo apt install prometheus prometheus-node-exporter prometheus-process-exporter

Configure Prometheus

Process exporter

Create config dir:

sudo mkdir /etc/process-exporter

Create config file /etc/process-exporter/config.yml:

process_names:
  - name: "{{.Comm}}"
    cmdline:
      - '.+'

Add config file to ARGS file of process-exporter /etc/default/prometheus-process-exporter:

ARGS="-config.path /etc/process-exporter/config.yml"

Restart service:

sudo systemctl restart prometheus-process-exporter.service

Prometheus

The node_exporter is already added to the config by installation. Add process-exporter to the Prometheus config.

• In /etc/prometheus/prometheus.yml add the following config to the end::

  - job_name: process_exporter
    static_configs:
      - targets: ['localhost:9256']

Restart service:

sudo systemctl restart prometheus

Check config

Go to: http://localhost:9090/classic/service-discovery

If everything is OK, you should see as the following:

Install Grafana

sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update
sudo apt install grafana -y
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

Grafana login

Go to: http://localhost:3000

• For the first login, the username and password is `admin.
• You will be asked to change them.

Add Prometheus as data source

Go to: http://localhost:3000/connections/datasources/new

• Choose Prometheus with everything else is default.

Add Grafana dashboards

Go to: http://localhost:3000/dashboard/import

Add following dashboards IDs by community:

• 1860 (node_exporter, full system usage)

• 22161 (process_exporter, by process)

• 13882 (process_exporter, by process)
• 249