GX10-83FB Telemetry (Prometheus + Grafana)

TELEMETRY.md

GX10-83FB Telemetry (Prometheus + Grafana)

Telemetry from the GX10-83FB host is exported to Prometheus and visualized in Grafana. Prometheus and Grafana run in the eudorus observatory stack in Docker on Zane. Access from any machine on the network (GX10, vonnegut, etc.) must use Zane’s Tailscale hostname, not localhost.

Observatory on Zane (Docker + Nginx)

The observatory stack runs in Docker on Zane. Nginx is the router in front; deployment path on Zane:
/Users/lou/migration/rootfs/home/lou/observatory-deployment
Config: observatory/nginx/nginx.conf; compose: docker-compose.yml.

macOS Docker → host: Zane is macOS. In Docker for Mac, containers run in a Linux VM and cannot reach the host via localhost or 127.0.0.1. Any container that must access a service exposed by the host (e.g. Neo4j on 7474) must use host.docker.internal. The compose file gives every service extra_hosts: host.docker.internal:host-gateway so they can reach host services when needed. Host-side services that containers should reach must listen on 0.0.0.0 (or the appropriate interface), not only 127.0.0.1.

Use these URLs (Tailscale; no subdomain DNS required):

Service	URL
Grafana	http://zane.tail3dac72.ts.net/grafana/
Prometheus	http://zane.tail3dac72.ts.net/prometheus/
Observatory (web app)	http://zane.tail3dac72.ts.net/observatory/
Redis	zane.tail3dac72.ts.net:6379 (direct)
Neo4j primary (GX10)	Browser: neo4j.zane... or gx10-83fb...:7474. Bolt (socket): gx10-83fb...:7687 only — do not put Bolt behind Nginx.
Neo4j reporting replica (GX10)	Browser: neo4j-reporting.zane... or gx10-83fb...:17474. Bolt (socket): gx10-83fb...:17687 only — do not put Bolt behind Nginx.

Nginx does path-based routing on zane.tail3dac72.ts.net so only that hostname has to resolve (Tailscale MagicDNS).

Real DNS (Cloudflare): If you prefer a dedicated hostname and no subpath, add DNS in Cloudflare for the host-based server names already in Nginx. Point each name at the same backend (Zane’s IP or the Cloudflare tunnel endpoint). Then use root URLs with no redirect/subpath handling:

• Grafana → grafana.nkllon.com (or e.g. grafana.yourdomain.com) — Nginx server_name already includes grafana.nkllon.com.
• Prometheus → prometheus.nkllon.com
• Observatory → observatory.nkllon.com

Create CNAME (or A) records in Cloudflare for those names; the existing Nginx server blocks will serve them at / with no subpath. If you use the host-based Grafana URL, set Grafana to root in docker-compose.yml: GF_SERVER_ROOT_URL=http://grafana.nkllon.com/ and GF_SERVER_SERVE_FROM_SUB_PATH=false (then use only the host-based URL for Grafana, not the path-based one).

Apply changes (on Zane, after editing nginx, compose, prometheus, or Grafana config):

cd /Users/lou/migration/rootfs/home/lou/observatory-deployment
docker compose up -d --force-recreate nginx grafana

After changing observatory/prometheus.yml or files in observatory/prometheus-rules/ (e.g. alert_rules.yml), restart Prometheus so it picks up new scrape jobs and rules: docker compose restart prometheus. Grafana will auto-load dashboards from observatory/grafana-config/dashboards/json/ (e.g. GX10-83FB Telemetry in folder Observatory).

Verify Grafana (run after deploy; no redirect loop = success):

curl -sI -L --max-redirs 3 "http://zane.tail3dac72.ts.net/grafana/" | head -20

Expect a final 200 or 302 to a login/dashboard path, not repeated 301 to /grafana/. If you see 502, check that the Grafana container is up and Nginx can reach it.

Components

Component	Location	Purpose
Metrics exporter	telemetry/metrics_exporter.py (this repo)	Runs on GX10; exposes /metrics on port 9092; optional Redis push
Redis transport	Same exporter when REDIS_HOST set	Pushes METRICS_UPDATE to beast:observatory:observatory-consumer:in; observatory-consumer forwards to Pushgateway
Prometheus scrape	eudorus observatory/configs/prometheus/prometheus.yml	Job gx10-83fb scrapes gx10-83fb.tail3dac72.ts.net:9092 (or Pushgateway when using Redis)
Grafana dashboard	eudorus observatory/configs/grafana/dashboards/gx10-83fb.json	Pre-provisioned dashboard "GX10-83FB Telemetry"

Metrics exposed

• Host: gx10_host_memory_bytes (total, free, available, cached, buffers), gx10_host_uptime_seconds, gx10_host_load (1m, 5m, 15m).
• GPU: gx10_gpu_memory_bytes (total, used, free), gx10_gpu_utilization_percent, gx10_gpu_info, gx10_gpu_temperature_celsius, gx10_gpu_power_watts (temp and power from nvidia-smi; power may be N/A on some systems).
• Exporter health: gx10_exporter_last_collect_timestamp_seconds — Unix time of last successful collect; use for “no data for N minutes” alerts.
• Services: gx10_service_up (1=up, 0=down) for llm_8000, neo4j_bolt_7687, neo4j_http_7474.
• LLM status (from GET /status): gx10_llm_state{port="8002"|"8003", state="up"|"loading"|"down"} = 1 for current state (0 for others); gx10_llm_desired{model="7b"|"120b"} = 1 for desired model. Exporter calls GX10_LLM_STATUS_URL (default http://127.0.0.1:8000/status) each collect.

Possible future metrics

• Disk: Free/used bytes for / or model paths (e.g. node_filesystem_*-style or custom) to detect full disk or swap pressure.
• LLM request metrics: Tokens/s, request count, latency percentiles would require instrumentation in the LLM proxy or backends (8002/8003), not just the exporter.
• Network: Interface bytes in/out if client traffic or Redis push volume matters.

Grafana: A dashboard that shows these LLM metrics is in telemetry/grafana/gx10-83fb-llm.json. Import it in Grafana (Dashboards → Import → upload/paste the JSON) or copy it into your observatory Grafana provisioned dashboards dir and reload. See telemetry/grafana/README.md.

Dashboard correctness and flatline

The GX10-83FB Telemetry Grafana dashboard (e.g. observatory/configs/grafana/dashboards/gx10-83fb.json) is correct: every panel queries the same metric names and labels the exporter produces, with job="gx10-83fb". If the dashboard is flatline, Prometheus has no series for that job — the pipeline is broken before Grafana.

Check in order:

1. Exporter running on GX10
   On gx10: curl -s http://127.0.0.1:9092/metrics | head -20. If that fails, the exporter is not running (no systemd, or not started).

2. Data reaching Prometheus
   Prometheus on Zane often cannot scrape GX10 directly (Docker container can’t use Tailscale). So metrics must be pushed from GX10 to the Pushgateway on Zane (PUSHGATEWAY_URL=http://zane.tail3dac72.ts.net:9091), and Prometheus must scrape the Pushgateway (job pushgateway, honor_labels: true). If the exporter never has PUSHGATEWAY_URL set (or Redis push to observatory-consumer → Pushgateway), Prometheus will never see job="gx10-83fb" series.

3. Prometheus and Grafana running
   On Zane, ensure the observatory stack is up: Prometheus and Grafana containers running (not Exited). Restart if needed so Prometheus scrapes Pushgateway and Grafana queries Prometheus.

4. Confirm in Prometheus
   In Prometheus UI (e.g. http://zane.tail3dac72.ts.net/prometheus/), run: gx10_host_uptime_seconds or {job="gx10-83fb"}. If no results, no gx10 metrics are in Prometheus (fix exporter + push path and/or stack).

5. Grafana time range
   Dashboard time range must include “now”; use e.g. “Last 1 hour” or “Last 15 minutes”.

Run the exporter on GX10

On the GX10 host (or from this workspace with SSH):

# From gx10-83fb repo on the host (or copy telemetry/ there)
cd /path/to/gx10-83fb
python3 -m venv .venv-telemetry
source .venv-telemetry/bin/activate
pip install -r telemetry/requirements.txt
export GX10_METRICS_PORT=9092   # default
export GX10_METRICS_HOST=0.0.0.0
python3 telemetry/metrics_exporter.py

Systemd (on GX10): Copy deploy/gx10-telemetry-exporter.service to /etc/systemd/system/, fix User/paths if needed, optionally create deploy/gx10-telemetry.env from deploy/gx10-telemetry.env.example with REDIS_HOST=zane.tail3dac72.ts.net, then:

sudo systemctl daemon-reload
sudo systemctl enable --now gx10-telemetry-exporter

Ad-hoc: ./telemetry/run_exporter.sh (uses .venv-telemetry or .venv if present).

Pushgateway push (recommended when Prometheus can't scrape GX10)

Prometheus runs in Docker on Zane and cannot reach GX10 over Tailscale from inside the container. Use Pushgateway so GX10 pushes metrics to Zane; Prometheus scrapes Pushgateway.

On Zane the observatory stack includes Pushgateway (port 9091). From GX10 (or any host running the exporter), set:

export PUSHGATEWAY_URL=http://zane.tail3dac72.ts.net:9091
export GX10_PROMETHEUS_JOB=gx10-83fb
export GX10_PROMETHEUS_INSTANCE=gx10-83fb
# optional: export GX10_PUSH_INTERVAL=15
python3 telemetry/metrics_exporter.py

Then open the GX10-83FB Telemetry dashboard in Grafana; data appears after the first push and Prometheus scrape (~15–30 s).

Exporter in Docker on GX10: If the exporter runs inside a container on GX10, set GX10_SERVICE_CHECK_HOST=host.docker.internal so service checks (Neo4j, LLM) hit the host where those ports are published; otherwise 127.0.0.1 is the container and Neo4j will show Down.

Redis transport (optional)

If the observatory stack’s Redis is reachable from GX10 (e.g. via Tailscale to the host running observatory), you can push metrics through Redis so the existing observatory-consumer forwards them to the Pushgateway. Prometheus then scrapes the Pushgateway (no need for Prometheus to reach GX10 directly).

Set Redis env vars and run the same exporter; it will keep serving HTTP and also push every GX10_REDIS_PUSH_INTERVAL seconds (default 15):

export REDIS_HOST=zane.tail3dac72.ts.net
export REDIS_PORT=6379
# export REDIS_PASSWORD=...             # if required
# export REDIS_DB=0
export GX10_REDIS_PUSH_INTERVAL=15
export GX10_PROMETHEUS_JOB=gx10-83fb
export GX10_PROMETHEUS_INSTANCE=gx10-83fb
python3 telemetry/metrics_exporter.py

• Stream: beast:observatory:observatory-consumer:in (override with GX10_OBSERVATORY_STREAM_PREFIX if your observatory uses another prefix).
• The eudorus metrics_consumer uses metadata.job and metadata.instance when pushing to Pushgateway, so GX10 metrics appear under job gx10-83fb.

Prometheus and Grafana (eudorus)

• Prometheus is configured in eudorus at observatory/configs/prometheus/prometheus.yml with a scrape job gx10-83fb for gx10-83fb.tail3dac72.ts.net:9092. Reload or restart Prometheus after config changes.
• Grafana datasource is the default Prometheus instance. The dashboard GX10-83FB Telemetry is provisioned from observatory/configs/grafana/dashboards/gx10-83fb.json. Rebuild the Grafana image (see eudorus Dockerfile.grafana) to bake in new dashboards.

If Prometheus runs inside Docker on Zane and cannot resolve Tailscale hostnames, either run Prometheus with host network, use a Tailscale sidecar, or set the scrape target to GX10’s Tailscale IP in a copy of the config used at deploy time. From other machines, open Grafana at http://zane.tail3dac72.ts.net/grafana/ (path-based via Nginx).

Runbook: flatline triage and remediation

Triage order (check in this sequence):

1. Exporter on GX10 — On gx10: curl -s http://127.0.0.1:9092/metrics | head -5. If this fails, start or fix the exporter (systemd unit, env file with PUSHGATEWAY_URL).
2. Push path — Exporter must push to Pushgateway (or Redis). Ensure PUSHGATEWAY_URL (or REDIS_HOST) is set in deploy/gx10-telemetry.env and the unit is restarted.
3. Observatory stack on Zane — Prometheus, Pushgateway, Grafana containers must be running. On Zane: docker ps; restart with docker compose up -d in the observatory-deployment directory.
4. Prometheus scrape — Pushgateway job must be configured with honor_labels: true. Query Prometheus: {job="gx10-83fb"}; if no series, fix push or scrape config.
5. Grafana time range — Dashboard time range must include "now" (e.g. Last 15 minutes).

Remediation matrix:

Symptom	Likely cause	Action
No data in Prometheus for job="gx10-83fb"	Exporter not pushing or not running	Set PUSHGATEWAY_URL in env; restart exporter; run scripts/telemetry_pipeline_probe.sh
Exporter exits repeatedly	Bad env, missing venv, or script error	Check journalctl -u gx10-telemetry-exporter; fix env or paths
Pushgateway unreachable from GX10	Network or Zane down	Verify Zane Tailscale and Pushgateway port 9091
Prometheus has no pushgateway job	Config not loaded	Copy eudorus prometheus.yml to deployment; restart Prometheus
Alerts not loading	Rules not in deployment	Copy alert_rules.yml into observatory/prometheus-rules/; restart Prometheus

Ownership / escalation: GX10 telemetry is owned by the operator of the gx10-83fb host and the Zane observatory stack. Escalate to whoever maintains Zane Docker and eudorus observatory configs if Pushgateway or Prometheus are down or config drift is found. For host-level config drift or compromise detection on GX10, see FILE_INTEGRITY_MONITORING.md (AIDE).

Rollback: (1) On GX10: sudo systemctl stop gx10-telemetry-exporter. (2) On Zane: restore previous Prometheus/alert config from backup; restart Prometheus. (3) Dashboard will show no new data; no impact on GX10 workload.

Evidence: Recovery report and dashboard screenshot requirements are in docs/TELEMETRY_RECOVERY_REPORT.md. Pipeline probe: TELEMETRY_PROBE_OUTPUT_DIR=docs/evidence ./scripts/telemetry_pipeline_probe.sh. Drift check: ./scripts/telemetry_drift_check.sh /path/to/observatory-deployment.

Eudorus paths (reference)

• Primary path: /Volumes/lemon/gemini/eudorus or mirror /Volumes/lemon/cursor/eudorus; local clone may be at ~/Documents/eudorus.
• Prometheus config: observatory/configs/prometheus/prometheus.yml
• Grafana dashboards: observatory/configs/grafana/dashboards/
• Grafana Dockerfile: Dockerfile.grafana (copies observatory/configs/grafana/dashboards into the image)