Tickets where /var was >=99% full (bytes or inodes) in the past year.
When /var is at 100%, Salt cannot write to /var/cache/salt/minion/ and saltutil.sync_all fails with SyncRet {}.
fsenrich falls back to Grafana-only enrichment — no directory breakdown.
Total: 44 tickets | 28 unique hosts | 22 WPs | 26 at exactly 100%
When a machine's filesystem is 100% full, Salt can't execute because it needs
to write to /var/cache/salt/minion/ before running any command. We wait 8
minutes retrying, then give up and post a degraded note with no directory
breakdown — exactly when the triage team needs it most.
Real case: sdpnmd-tt-038 /var, June 10. NetBackup was writing 3.58GB .img
Comparing what Triage Ops sees in the DRQS note between old and new implementations.
Old: triagercsvc → SSH as op user (port 2222) → sudo /opt/bbinfra/bin/tree --du -xfuDsnipQ
New: Salt orchestration → sudo safe_fsenrich_find.sh → find -printf + awk pipeline
Date: 2026-06-02 | Epic: ENGTRIASVC-2898 (Migrate fsenrich off CRADMN)
| Jira | Summary | State | Blocking? |
|---|---|---|---|
| ENGTRIASVC-3104 | DF_BIN fallback for hosts without bbinfra | Merged ✅ triatm#97 | — |
Period: Mon May 25 00:00 UTC → Sun Jun 1 11:00 UTC
Verdict: PROMOTE — 11/11 comparable pairs validated, 0 regressions, 1 improvement over prod
| Field | Value |
|---|---|
| Beta DRQS | 183110294 |
| Period | 2026-05-25 → 2026-06-01 (7 days) |
| Stage filter | bpaasStage = "beta-s0a" |
| Total beta notes | 85 |
Date: 2026-05-19
Discovered via: Beta vs prod enrichment comparison during ENGTRIASVC-3067 validation
Affects: All prod disk space enrichment on high-inode filesystems
Salt file: automation-tenancies/triatm — salt/orch/triatm/fsenrich.sls