Skip to content

Instantly share code, notes, and snippets.

@mman

mman/Victron Energy Grafana Notes.md

Last active February 7, 2023 13:07
Show Gist options
  • Save mman/45b37f7fbec79f4fc7aa38909fc7c54a to your computer and use it in GitHub Desktop.
Save mman/45b37f7fbec79f4fc7aa38909fc7c54a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
Victron Energy Grafana Notes.md

Improving Victron Energy InfluxDB + Grafana Implementation

State of Affairs as of Feb 7 2023

  • Advanced users of Victron hardware can optionally install Venus Docker Server + Influx + Grafana to get:

    • access to super granular (2 sec) data from any Venus OS based device (any Victron GX device, or custom Venus OS device).
    • ability to create custom Grafana dashboards and alerts
    • ability to archive data for extended periods of time

  • Installation of Influx + Grafana currently uses sample docker-compose and requires:

    • official influxdb docker image
    • Victron built venus-docker-server - a node.js based container watching MQTT and capturing all data into InfluxDB
    • Victron built venus-docker-upnp - a node.js based UPNP browser that auto detects all VenusOS based devices on the local network.
    • Victron built venus-docker-grafana - a custom built grafana docker image with pre configured sample dashboards

Issues

  • All data obtained via MQTT from Venus OS device, and stored into InfluxDB live by default inside of Docker volume. This is typically an opaque file living somewhere on the system, that is hard to examine, backup, and restore, and that is prone to accidental loss for users not experienced with docker and docker-compose.

  • All Grafana dashboards and configuration live inside a SQLite database that is by default burried inside of Docker volume. This makes it hard to examine/backup/restore all Grafana customizations in case of system upgrade, restore.

  • Source code for docker images of venus-docker-server, venus-docker-upnp, venus-docker-grafana is mixed together with source code of actual venus-docker-grafana-server npm module (https://www.npmjs.com/package/venus-docker-grafana-server).

  • venus-docker-server, and venus-docker-upnp docker images actually use release of venus-docker-grafana-server npm modulle that is publicly available, instead of code from the repository, which makes it hard to iterate on both.

  • venus-docker-grafana-server npm module actually vends two binaries, venus-server that listens for MQTT and stores data to InfluxDB. And venus-upnp-browser that scans local network for available Venus OS devices and communicates discovered devices over HTTP to venus-server.

  • hostnames and ports used for communication between the venus-upnp-browser and venus-server are hard coded in the code and rely on proper naming of stuff in docker-compose.yaml

Goals

  • Make it safe by default, and easy to understand/customize where InfluxDB data are stored. So that it is never accidentally lost.
  • Make it safe by default, and easy to understand/customize where Grafana configuration is stored. So that it is never accidentally lost.
  • Make the whole venus-server + influx + grafana work out of the box with little to no customization when used in a standard setting: Runing on the same network, with one (or more?) Venus OS devices present.
  • Ship with nice sample dashboards for typical Victron use cases (ESS, OffGrid, Yacht, RV, ???)
  • Make each component documented and working independently with clear interface, so that advanced scenarios can be achieved quickly.

Proposed Changes

InfluxDB

No issues, stock docker image of 1.7 used, upgrade to 1.8 smooth. Need to test 2.6 which is the latest.

Grafana

Custom Victron Grafana image is created by taking official Grafana docker image, and adding:

  • InfluxDB datasource to visualize data stored in InfluxDB.
  • simpod-json-datasource plugin, and data source to display VenusOS instances that are being monitored by venus-server.
  • sample Grafana dashboard displaying live Venus OS instances
  • sample Grafana dashboard to visualize battery SOC, AC Load, etc...

This is currently achieved by taking an existing SQLite configuration database from a sample system and slapping it onto the fresh Grafana installation. This still works but feels inapropriate and prone to breaking in the future. New version of Grafana takes that existing SQLite DB and migrates it to the new version automatically, potentially breaking things along the way.

TODO: I propse to drop the custom Victron Grafana Docker image altogether, and use official Grafana Docker Image together with official documented approach of Grafana Provisioning (https://grafana.com/docs/grafana/latest/administration/provisioning/). This makes it easy to store required datasources and dashboards in a textual representation in a git repo insted of in an opaque SQLite database. These sample dashboards and data source configuration should probably live in https://github.com/victronenergy/venus-docker-grafana because they relate to how the grafana talks to influxdb and venus-server, and this is specified in docker-compose.yaml.

Venus Docker Server + UPNP

Venus Docker Server NPM module currently produces two funcional artifacts, the venus-server, and venus-upnp-browser. First via binary (https://github.com/victronenergy/venus-docker-grafana-images/blob/master/server/bin/venus-server), second via source file deep in the hierarchy (https://github.com/victronenergy/venus-docker-grafana-images/blob/master/server/lib/upnp.js).

It vends an unofficial HTTP API by which venus-upnp-browser talks to venus-server when Venus OS instances are discovered. It vends an unofficial HTTP API by which Grafana simpod-json-datasource talks to venus-server to obtain list of Venus OS instances being monitored. It provides browser based UI to discover and configure: What Venus OS instances to monitor, where to store the data.

TODO: Cleanup the repository hosting this npm module to align with npm best practices, document it, and align with the way https://github.com/victronenergy/node-red-contrib-victron is made.

TODO: Produce two binaries venus-server, and venus-upnp-browser.

TODO: Figure out why UPNP browser and Venus Server run as separate docker instances - UPNP is configured with docker host networking mode, in order to access LAN directly. Not sure if this is needed, Venus Server can also run in docker host networking mode to have access to LAN for browsing and have access to InfluxDB and Grafana. Need to investigate this.

TODO: Think about ranaming this component from venus-server to something more appropriate like venus-live-data-exporter, or venus-influxdb-ingester, or something similar, because the primary purpose of this component is to actually live dump all data from Venus OS to InfluxDB (or TimescaleDB) - potentially located externally from VenusOS. Unfortunately venus-server currently also vends /grafana-api endpoint for Grafana, as well as couple more endpoints for UPNP browser and couple endpoints for its own UI.

TODO: Improve docker build to use code from repository directly, to iterate faster.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment