Skip to content

Instantly share code, notes, and snippets.

@vladak

vladak/1-grafana-truenas.md

Last active September 11, 2022 20:38
Show Gist options
  • Save vladak/e3707bfc2f81307de0268530691e9572 to your computer and use it in GitHub Desktop.
Save vladak/e3707bfc2f81307de0268530691e9572 to your computer and use it in GitHub Desktop.
Download ZIP
TrueNAS dashboard for Grafana
Raw
1-grafana-truenas.md

TrueNAS dashboard for Grafana

This setup is basically using the configuration from https://crashlaker.github.io/2021/10/09/truenas_snmp_exporter.html

It assumes Prometheus as data backend. Also, nas.local is the NAS machine hostname used throughout this Gist.

This note is a bit of a experiment with Gists.

The end result looks like this: grafana-NAS

Configure SNMP on TrueNAS

In my case SNMPv3 is used: TrueNAS-SNMP

The TrueNAS SNMP MIBs live in the /usr/local/share/snmp/mibs on the TrueNAS machine, so copy them over to the matchine that will be running the SNMP exporter:

scp -r nas.local:/usr/local/share/snmp/mibs truenas-mibs

and verify the basic functionality (assumes the net-snmp is installed):

$ snmptranslate -M +truenas-mibs -On FREENAS-MIB::hddTempTable
.1.3.6.1.4.1.50536.3

$ snmpwalk -M +truenas-mibs -l authPriv \
    -a SHA -A 'INSERT_AUTH_PASSWORD_HERE' -x AES -X 'INSERT_PRIV_PASSWORD_HERE' \
    -u grafana -v 3 nas FREENAS-MIB::hddTempTable
FREENAS-MIB::hddTempDevice.1 = STRING: da0
FREENAS-MIB::hddTempDevice.2 = STRING: ada0
FREENAS-MIB::hddTempDevice.3 = STRING: ada2
FREENAS-MIB::hddTempDevice.4 = STRING: ada3
FREENAS-MIB::hddTempDevice.5 = STRING: ada1
FREENAS-MIB::hddTempDevice.6 = STRING: nvd0
FREENAS-MIB::hddTempDevice.7 = STRING: ada4
FREENAS-MIB::hddTempValue.1 = Gauge32: 43000
FREENAS-MIB::hddTempValue.2 = Gauge32: 41000
FREENAS-MIB::hddTempValue.3 = Gauge32: 40000
FREENAS-MIB::hddTempValue.4 = Gauge32: 38000
FREENAS-MIB::hddTempValue.5 = Gauge32: 38000
FREENAS-MIB::hddTempValue.6 = Gauge32: 56000
FREENAS-MIB::hddTempValue.7 = Gauge32: 42000

Setup SNMP exporter

This assumes that SNMP exporter runs as a service.

Use the generator.yml as source file for snmp_exporter and store it under ~/truenas-snmp directory.

cp -R truenas-mibs/* ~/.snmp/mibs/
cd ~/truenas-snmp
~/golang/bin/generator generate

This should generate the snmp.yml file. Append the file to the snmp.yml file used by the SNMP exporter service and restart the SNMP exporter service:

sudo systemctl restart snmp_exporter

Now test if the changes were properly put into effect:

curl 'http://localhost:9116/snmp?target=nas.local&module=freenas'

Setup Prometheus

Add the following to the scrape_configs section in /etc/prometheus/prometheus.yml :

  - job_name: nas
    static_configs:
      - targets:
        - nas.local
    scrape_interval: 15s
    metrics_path: /snmp
    params:
      module: [freenas]
    relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9116

and restart the service:

sudo systemctl restart prometheus

Setup Grafana dashboard

Provision Grafana with the NAS JSON file.

The TrueNAS logo needs to be stored on the Grafana instance itself per https://community.grafana.com/t/using-text-panel-to-show-image-where-to-store-image/6642

sudo mv /tmp/TrueNAS\ logo.png /usr/share/grafana/public/img/TrueNAS-logo.png

Note on alerting

It would be nice if single stat or gauge panels supported alerting however currently (2022) this is not the case. There were issues filed and one PR however all were closed to much despair of users.

Raw
generator.yml
modules:
freenas:
walk:
- zpoolDescr
- memTotalFree
- memTotalReal
- memAvailReal
#- ssCpuRawIdle
- ssCpuIdle
- laEntry # load average
# host resource metrics
- hrSystemUptime
- hrSystemProcesses
- hrSystemMaxProcesses
- hrMemorySize
- hrStorageSize
- hrStorageUsed
- hrProcessorLoad # non idle percent past minute
# interface metrics
- ifAdminStatus # 1=up,2=down,3=testing
- ifOperStatus # 1=up,2=down,3=testing,4=unknown,5=dormant,6=notPresent,7=lowerLayerDown
- ifLastChange
- ifInDiscards
- ifInErrors
- ifOutDiscards
- ifOutErrors
- ifHCInOctets
- ifHCOutOctets
# tcp metrics
- tcpMaxConn
- tcpActiveOpens
- tcpPassiveOpens
- tcpEstabResets
- tcpCurrEstab
#- zpoolDescr #.3 = STRING: oricoA
- zpoolSize #.1 = INTEGER: 25034752
- zpoolAllocationUnits
- zpoolUsed #.1 = INTEGER: 366581
- zpoolAvailable #.4 = INTEGER: 899712777
- zpoolHealth # online(0), degraded(1), faulted(2), offline(3), unavail(4), removed(5)
#- zvolDescr #.1 = STRING: nvme/esxi6-flash-ds
- zvolSize #.1 = INTEGER: 131072004
- zvolUsed #.1 = INTEGER: 12742140
- zvolAllocationUnits
- zvolAvailable #.1 = INTEGER: 208719694
- zvolReferenced #.1 = INTEGER: 7553203
#- datasetDescr #.1 = STRING: boot-pool/ROOT
- datasetAllocationUnits
- datasetSize #.1 = INTEGER: 24251018
- datasetUsed #.1 = INTEGER: 365180
- datasetAvailable #.1 = INTEGER: 23885838
#- hddTempDevice # string da0 da1 ada0 nvd0
- hddTempValue # 41000
- zfsArcSize #.0 = Gauge32: 26288777
- zfsArcMeta #.0 = Gauge32: 638834
- zfsArcData #.0 = Gauge32: 25647720
- zfsArcHits #.0 = Gauge32: 2127255356
- zfsArcMisses #.0 = Gauge32: 24503533
- zfsArcC #.0 = Gauge32: 26335347
- zfsArcP #.0 = Gauge32: 24565288
- zfsArcMissPercent #.0 = STRING: 0.22811610704906116
- zfsArcCacheHitRatio #.0 = STRING: 99.77
- zfsArcCacheMissRatio #.0 = STRING: 0.23
- zfsL2ArcHits #.0 = Counter32: 4629098
- zfsL2ArcMisses #.0 = Counter32: 19863879
- zfsL2ArcRead #.0 = Counter32: 40120202
- zfsL2ArcWrite #.0 = Counter32: 3900941555
- zfsL2ArcSize #.0 = Gauge32: 205686406
#lookups:
#- old_index: ifIndex
# new_index: ifName
lookups:
- source_indexes: [ifIndex]
lookup: ifName
- source_indexes: [zvolIndex]
lookup: zvolDescr
- source_indexes: [zpoolIndex]
lookup: zpoolDescr
- source_indexes: [datasetIndex]
lookup: datasetDescr
- source_indexes: [hddTempIndex]
lookup: hddTempDevice
- source_indexes: [laIndex] #loadAverageIndex
lookup: laNames
- source_indexes: [hrStorageIndex]
lookup: hrStorageDescr
- source_indexes: [hrDeviceIndex]
lookup: hrDeviceDescr
version: 3
auth:
username: "grafana"
security_level: authPriv
auth_protocol: SHA
password: "INSERT_AUTH_PASSWORD_HERE"
priv_protocol: AES
priv_password: "INSERT_PRIV_PASSWORD_HERE"
Raw
NAS-dashboard.json
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
}
],
"__elements": {},
"__requires": [
{
"type": "panel",
"id": "gauge",
"name": "Gauge",
"version": ""
},
{
"type": "grafana",
"id": "grafana",
"name": "Grafana",
"version": "9.0.3"
},
{
"type": "datasource",
"id": "prometheus",
"name": "Prometheus",
"version": "1.0.0"
},
{
"type": "panel",
"id": "stat",
"name": "Stat",
"version": ""
},
{
"type": "panel",
"id": "text",
"name": "Text",
"version": ""
},
{
"type": "panel",
"id": "timeseries",
"name": "Time series",
"version": ""
}
],
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": {
"type": "grafana",
"uid": "-- Grafana --"
},
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"target": {
"limit": 100,
"matchAny": false,
"tags": [],
"type": "dashboard"
},
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "celsius"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 12,
"x": 0,
"y": 0
},
"id": 2,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "hddTempValue / 1000",
"legendFormat": "{{hddTempDevice}}",
"range": true,
"refId": "A"
}
],
"title": "HDD temperature",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"max": 100,
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 80
},
{
"color": "red",
"value": 90
}
]
},
"unit": "percent"
},
"overrides": []
},
"gridPos": {
"h": 9,
"w": 5,
"x": 12,
"y": 0
},
"id": 6,
"options": {
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"showThresholdLabels": false,
"showThresholdMarkers": true,
"text": {}
},
"pluginVersion": "9.0.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "((zpoolSize{zpoolDescr=\"uloziste\"} - zpoolAvailable{zpoolDescr=\"uloziste\"}) / zpoolSize{zpoolDescr=\"uloziste\"}) * 100",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "uloziste capacity",
"type": "gauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "text",
"value": null
}
]
},
"unit": "bytes"
},
"overrides": []
},
"gridPos": {
"h": 5,
"w": 4,
"x": 17,
"y": 0
},
"id": 16,
"options": {
"colorMode": "value",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "9.0.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "code",
"expr": "zpoolAvailable{instance=\"nas.local\", zpoolDescr=\"uloziste\"} * zpoolAllocationUnits{instance=\"nas.local.home.kotalovi.cz\", zpoolDescr=\"uloziste\"}",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "uloziste available space",
"type": "stat"
},
{
"datasource": {
"type": "datasource",
"uid": "grafana"
},
"gridPos": {
"h": 2,
"w": 3,
"x": 21,
"y": 0
},
"id": 4,
"options": {
"content": "<img src=\"/public/img/TrueNAS-logo.png\">",
"mode": "html"
},
"pluginVersion": "9.0.3",
"type": "text"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [
{
"options": {
"0": {
"color": "green",
"index": 0,
"text": "OK"
},
"1": {
"color": "red",
"index": 1,
"text": "BAD"
}
},
"type": "value"
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 3,
"w": 3,
"x": 21,
"y": 2
},
"id": 8,
"options": {
"colorMode": "background",
"graphMode": "area",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"text": {},
"textMode": "value"
},
"pluginVersion": "9.0.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "zpoolHealth{instance=\"nas.local\", zpoolDescr=\"uloziste\"} != bool 0",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "uloziste status",
"transformations": [
{
"id": "reduce",
"options": {
"includeTimeField": false,
"mode": "reduceFields",
"reducers": [
"last"
]
}
}
],
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "thresholds"
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "blue",
"value": null
}
]
},
"unit": "timeticks"
},
"overrides": []
},
"gridPos": {
"h": 4,
"w": 7,
"x": 17,
"y": 5
},
"id": 18,
"options": {
"colorMode": "none",
"graphMode": "none",
"justifyMode": "auto",
"orientation": "auto",
"reduceOptions": {
"calcs": [
"lastNotNull"
],
"fields": "",
"values": false
},
"textMode": "auto"
},
"pluginVersion": "9.0.3",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "hrSystemUptime{instance=\"nas.local\"}",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "system uptime",
"type": "stat"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "Bps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 9
},
"id": 10,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "rate(ifHCInOctets{instance=\"nas.local\"}[$__rate_interval])",
"legendFormat": "{{ifName}}",
"range": true,
"refId": "A"
}
],
"title": "inbound traffic",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "binBps"
},
"overrides": []
},
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 9
},
"id": 12,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "rate(ifHCOutOctets{instance=\"nas.local\"}[$__rate_interval])",
"legendFormat": "{{ifName}}",
"range": true,
"refId": "A"
}
],
"title": "outbound traffic",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
}
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 12,
"x": 0,
"y": 17
},
"id": 14,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "laLoadFloat{instance=\"nas.local\", laNames=\"Load-15\"}",
"legendFormat": "__auto",
"range": true,
"refId": "A"
}
],
"title": "system load average over 15 minutes",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
"fillOpacity": 0,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"lineInterpolation": "linear",
"lineWidth": 1,
"pointSize": 5,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"spanNulls": false,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "off"
}
},
"mappings": [],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 80
}
]
},
"unit": "deckbytes"
},
"overrides": []
},
"gridPos": {
"h": 6,
"w": 12,
"x": 12,
"y": 17
},
"id": 20,
"options": {
"legend": {
"calcs": [],
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "single",
"sort": "none"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"editorMode": "builder",
"expr": "avg(memAvailReal{instance=\"nas.local\"})",
"legendFormat": "available",
"range": true,
"refId": "A"
}
],
"title": "available RAM",
"type": "timeseries"
}
],
"refresh": "1m",
"schemaVersion": 36,
"style": "dark",
"tags": [],
"templating": {
"list": []
},
"time": {
"from": "now-24h",
"to": "now"
},
"timepicker": {},
"timezone": "",
"title": "NAS",
"uid": "_IvT0TM4z",
"version": 11,
"weekStart": ""
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment