valkey.md

Valkey Sentinel as a Unified Infrastructure Backbone

HomeFlare Cluster -- 3-node Proxmox homelab running BGP anycast, Ceph, Nomad, and 85+ API endpoints.

A single Valkey Sentinel cluster, running on a 40 Gbps Thunderbolt 4 mesh with sub-millisecond latency, serves as the shared data plane for 8 distinct infrastructure concerns -- eliminating the need for 8 separate systems.

Note: Summarized with Claude

---

Architecture

graph TB
    subgraph TB4["Thunderbolt 4 Mesh (40 Gbps)"]
        V1["Valkey<br/>n2:62120"]
        V2["Valkey<br/>n3:62120"]
        V3["Valkey<br/>n4:62120"]
        S1["Sentinel<br/>n2:62130"]
        S2["Sentinel<br/>n3:62130"]
        S3["Sentinel<br/>n4:62130"]
    end

    V1 <--> V2 <--> V3
    S1 --- S2 --- S3

    subgraph Consumers["Consumers (all via TLS + Sentinel)"]
        A1["Agent API<br/>85 endpoints"]
        WS["WebSocket API<br/>live state + events"]
        CI["CI Runners<br/>turbo cache"]
        PUB["Data Publisher<br/>14 providers"]
        LE["Leader Election<br/>single-writer coordination"]
        SD["Service Discovery<br/>per-node registration"]
        DOC["Docs System<br/>45+ auto-updated sections"]
        EB["Event Bus<br/>real-time cluster events"]
    end

    A1 --> V1
    WS --> V1
    CI --> V1
    PUB --> V1
    LE --> V1
    SD --> V1
    DOC --> V1
    EB --> V1

Loading

What Valkey Replaces

Concern	Without Valkey	With Valkey	Key Pattern
Cluster State	Poll each node, aggregate	Single read from hfc:state:{node}	15s TTL, 3 keys
Leader Election	External coordinator (etcd, Consul)	SET hfc:leader NX EX 30	Lease-based, 10s refresh
Service Discovery	Static config or DNS	hfc:svc:{node} with 15s TTL	Auto-expires on node failure
Event Bus	Message broker (RabbitMQ, NATS)	Pub/sub on hfc:event + sorted set hfc:events	Real-time + queryable history
CI Build Cache	S3 bucket or filesystem	hfc:turbo:{hash} with 7-day TTL	Turbo Remote Cache API
Data Bus	Per-request API calls	hfc:data:{provider}:{query}	4 TTL tiers (30s to 24h)
Docs Cache	Rebuild on every deploy	hfc:docs:{key} with 1h TTL	45+ auto-updated MDX sections
WebSocket State	Server-side memory per connection	Valkey pub/sub + state reads	Stateless agents, any-node gateway

Data Flow

sequenceDiagram
    participant Agent as Agent (any node)
    participant Valkey as Valkey Sentinel
    participant Runner as CI Runner
    participant WS as WebSocket Client

    Note over Agent,Valkey: Cluster State Broadcasting (every 5s)
    Agent->>Valkey: SETEX hfc:state:n2 15 {cpu, memory, load}
    Agent->>Valkey: SETEX hfc:svc:n2 15 {services}

    Note over Agent,Valkey: Leader Election
    Agent->>Valkey: SET hfc:leader n2 NX EX 30
    Valkey-->>Agent: OK (elected) or nil (not leader)

    Note over Agent,Valkey: Data Publisher (leader-only queries)
    Agent->>Valkey: SETEX hfc:data:nomad:jobs 30 {job data}
    Agent->>Valkey: SETEX hfc:data:netbox:devices 3600 {devices}

    Note over Runner,Valkey: Turbo Remote Cache
    Runner->>Agent: PUT /v8/artifacts/{hash} (Bearer auth)
    Agent->>Valkey: SETEX hfc:turbo:{hash} 604800 {binary}
    Runner->>Agent: GET /v8/artifacts/{hash}
    Agent->>Valkey: GET hfc:turbo:{hash}
    Valkey-->>Agent: {binary}
    Agent-->>Runner: 200 application/octet-stream

    Note over WS,Valkey: Real-Time Events
    Agent->>Valkey: PUBLISH hfc:event {deploy, alert, ...}
    Valkey-->>WS: Forwarded via agent WebSocket

Loading

Key Namespaces

Prefix	Purpose	TTL	Writer
hfc:state:{node}	Per-node health (CPU, memory, load, uptime)	15s	Every agent
hfc:svc:{node}	Per-node service list (ports, health)	15s	Every agent
hfc:leader	Current cluster leader node name	30s	Leader only
hfc:data:{provider}:{query}	Cached provider results (14 providers, 45+ queries)	30s -- 24h	Leader or local
hfc:docs:{key}	Documentation section cache	1h	Docs system
hfc:turbo:{hash}	Turborepo build artifacts (binary)	7 days	CI runners
hfc:turbo:{hash}:tag	Artifact integrity tags	7 days	CI runners
hfc:events	Cluster event history (sorted set)	Rolling	Any agent
hfc:event	Pub/sub channel for real-time events	Ephemeral	Any agent

TTL Tiers (Data Bus)

The data publisher runs on every agent and schedules provider fetches across 4 frequency tiers:

Tier	TTL	Refresh	Providers	Rationale
Fast	30s	15s	ceph, bgp, nomad, system, valkey	Changes frequently, stale data is misleading
Medium	5m	5m	caddy, certs, tunnel, ups	Rarely changes mid-day
Slow	1h	30m	netbox, pve	External systems, expensive queries
Static	24h	6h	static, repo	Reference data, almost never changes

Leader-only providers (nomad, netbox) run on one node to avoid duplicate API calls. Local-only providers (system, ceph, bgp) run on every node since data is node-specific.

Turbo Remote Cache: CI Impact

graph LR
    subgraph Before["Before: Independent Caches"]
        R1B["Runner 1<br/>Local cache only"]
        R2B["Runner 2<br/>Local cache only"]
        R1B -.->|"cache miss"| B1["Build 17s"]
        R2B -.->|"cache miss"| B2["Build 17s"]
    end

    subgraph After["After: Shared Valkey Cache"]
        R1A["Runner 1"]
        R2A["Runner 2"]
        VK["Valkey<br/>hfc:turbo:*"]
        R1A -->|"PUT artifact"| VK
        VK -->|"GET artifact"| R2A
        R1A -.->|"first build"| B3["17s"]
        R2A -.->|"cache hit"| B4["19ms"]
    end

Loading

Metric	Before	After
Cold build (CLI)	17s	17s (first run)
Warm build (cache hit)	17s per runner	19ms (shared)
Cache sharing	None	Cross-runner via Valkey
Storage	Local filesystem x2	Valkey with 7-day TTL
Auth	None	Bearer token (HFC_WS_TOKEN)

Security

All Valkey connections use TLS with cluster CA certificates and password authentication via Sentinel. The Turbo Remote Cache routes require Bearer authentication on every request (GET, PUT, HEAD). No unauthenticated access to cached artifacts.

Layer	Protection
Transport	TLS 1.3 with cluster-issued certificates
Authentication	Password (Valkey) + Bearer token (API)
Network	Sentinel on TB4 mesh (not routable from LAN)
Failover	Sentinel quorum (2/3), auto-promote replica in ~5s

Why This Matters

One Valkey Sentinel cluster replaces what would otherwise require:

• etcd or Consul for leader election and service discovery
• RabbitMQ or NATS for event pub/sub
• S3 or Artifactory for CI build cache
• Redis or Memcached for application caching
• Custom polling for cluster state aggregation

All running on a 40 Gbps direct-attach mesh with sub-millisecond latency, automatic failover, and zero additional infrastructure to deploy or maintain.

52 source files across the CLI reference Valkey. 14 data providers publish to it. 85 API endpoints read from it. 2 CI runners share build cache through it. 1 service to operate.