XRPL — validation relay loss on mainnet: one-ledger cohort drops at the s1/s2 vantage

validation-relay-loss-mainnet.md

Validation relay loss on mainnet — one-ledger cohort drops at the s1/s2 vantage

Status: investigated (2026-08-02), monitor cross-check shipped (xrplf-release-notifier, uncommitted), root-cause candidates ranked from code; perfnet repro pending Origin: observatory LOW_QUORUM alert, ledger 106013031 ("25/35 UNL validations, need 28"). Investigation proved the network was healthy and the alert was a vantage artifact — which surfaced the real finding: Ripple's public s1/s2 clusters intermittently lose a cohort of validators' validations for exactly one ledger. Relevant to the capacity campaign because relay redundancy, not consensus, was the layer that degraded under load.

---

1. The incident (2026-08-02 04:22 UTC, ledger 106013031)

• Our observatory (subscribed to wss://s1.ripple.com + wss://s2.ripple.com validations streams) saw 78 validation messages for ledger 106013031 vs ~100 for every neighbor; 25/35 UNL. The missing validations never arrived — not even hours later.
• Independent vantage (xrplwin xPOP Validation Store, xpop.xrplwin.com/0/…) had 33/35 UNL validations on disk for the same ledger, including 8 of our 10 "missing" UNL validators (Berkeley, Bitrue, Interledger, Anodos, Arrington, Swarthout, xrp.vet, AureusOx) — all signed on time (04:22:23), full=true, correct hash. Network quorum was never in danger.
• The lost cohort = the latest signing bucket of that ledger's validation burst: everything signed after ~04:22:23.6 at that vantage was lost; the 25 delivered UNL validations span :22.59–:23.65 arrival.
• Adjacent load: ledger 106013032 carried 529 txns (495 AccountSet/OfferCancel from 153 distinct senders, all at 12 drops — queue-drain signature of a cheap burst).

2. It recurs — every observed "quorum dip" is a relay gap

Sweep of the full observatory capture (~15.6k ledgers over ~40h): every ledger with <90 observed validations, after excluding monitor-restart artifacts and one 20-ledger plateau at 88–89 (a single validator's outage), leaves six genuine dips. All six are confirmed relay gaps — the xPOP store holds 32–34 UNL validations for each:

ledger	when (UTC)	our feed	xPOP store	nearby load
105997679	08-01 11:49	69	32	quiet (~30 tx/ledger)
105999951	08-01 14:16	81	33	moderate (28–55)
106000071	08-01 14:23	78	33	burst: 203+165 tx flush
106002428	08-01 16:55	89	34	burst: 269 tx prior ledger
106002453	08-01 16:56	87	34	sustained 51–163
106013031	08-02 04:22	78	33	burst: 529 tx next ledger

Zero genuine low-quorum events in the capture. Rate ≈ one dip per 6–7h; bursts are associated in 4/6 cases but not required (105997679 was quiet).

Cadence correction: the "9s round then 1s closes" pattern in ledger headers around these events is an artifact of close_time_resolution = 10 plus the effCloseTime +1s floor (include/xrpl/ledger/LedgerTiming.h:133-144). Validation-arrival timestamps show a steady ~4.0s real cadence through every dip. There were no slow rounds; the loss is purely relay-layer.

3. Mechanism (code-ranked, rippled @ develop)

Structural facts that make a one-ledger cohort loss possible:

• Squelching thins per-validator relay redundancy to ~5 peers. With reduce_relay, a node selects kMaxSelectedPeers = 5 source peers per validator and squelches all others (overlay/Slot.h:314-398, ReduceRelayCommon.h:31); squelch is enforced sender-side before queueing (PeerImp.cpp:280-286), for 300s–1h. Unsquelch-on-idle needs 8s of silence checked every 4s (Slot.h:256, Tuning::kCheckIdlePeers) — slower than a ledger, so a one-ledger source hiccup cannot self-heal in time.
• The peer send queue never drops individual messages — it is FIFO and unbounded (PeerImp.cpp:315-318); loss happens only when the whole connection is torn down, discarding the entire queued tail at once ("Large send queue" disconnect: queue ≥128 across four 60s timer ticks, PeerImp.cpp:713-717, Tuning.h:25,31). There is no broadcast-vs-critical prioritization and no retransmit path for validations — fire-and-forget.
• The WS validations stream publishes every validation that reaches recvValidation (pubValidation unconditional, NetworkOPs.cpp:2605). So a validation absent from both s1 and s2 streams never arrived at those nodes — it was not "received but unrelayed".
• Receive-side filters (isCurrent ±3min windows), job-queue saturation (defers, never drops), and relay gating (30s HashRouter window) are all ruled out for on-time trusted validations.

Best-fit narrative: under burst load, the outbound queue of a selected (unsquelched) relay source backs up behind bulk traffic; its tail — the latest-signed validations for one ledger — is delayed past the ledger or discarded by a connection teardown. All other peers are squelched for those validators, so nothing refills the gap, and fire-and-forget means the loss is permanent at that vantage. s1/s2 share upstream topology, so one upstream event starves both.

4. Why this matters

1. Monitoring/reputation infrastructure is built on this vantage. Anyone measuring validator agreement from a single cluster's streams (VHS-style scorers, dashboards, our own monitor) records false misses. Validators get blamed for infrastructure loss.
2. Capacity angle (Weakness #0 adjacency): consensus held, but a modest cheap-tx burst measurably degraded validation propagation. kMaxSelectedPeers = 5 redundancy per validator is the thin edge; a node whose 5 sources for many validators route through one congested hub can transiently lose quorum-relevant signal. A node relying on those validations for checkAccept would locally delay full validation.
3. It composes badly with real events: during a genuine partial outage, vantage loss subtracts further from an already-reduced count, making panic-level readings possible on a healthy-ish network.

5. What shipped (xrplf-release-notifier, uncommitted)

monitors/crawler now cross-checks LOW_QUORUM against the xPOP store before posting (crosscheck.rs): missing validations found there → recategorized RELAY_GAP (WARNING, "network quorum OK, relays lost upstream of our sources"); missing there too → escalates to CRITICAL (two vantages agree). Store unreachable → original alert + error note. Live-verified against ledger 106013031 (recovers the 8 lost UNL validators). Deploy to the observatory VM pending.

Vantage-diversity upgrade (same commit): the monitor now subscribes to wss://xrplcluster.com (community infra) alongside s1/s2 — the detection engine unions endpoints, so LOW_QUORUM only fires when every vantage missed — and a per-source tracker (sources.rs) compares which UNL validators each endpoint delivered per finalized ledger, emitting VANTAGE_LOSS (WARNING, per-source 24h dedup) when one endpoint is ≥5 UNL validations short of the others' union. That turns the s1/s2 loss events from false alarms into live-measured relay-gap telemetry with no third-party dependency. Endpoint reconnects back off exponentially (5s→60s) so the pre-DNS validator endpoint doesn't spam the journal.

6. Next steps

• Deploy the cross-check + multi-vantage monitor (observatory deploy.sh) and commit the notifier changes.
• Add our own validator as a third monitor vantage: Cloudflare DNS-only A record node.unl.xrpl.foundation → 79.110.60.134 (do NOT touch bare unl.xrpl.foundation — it serves the UNL blob from GCS and the box's nginx does not), re-run the ansible nginx/certbot phase, redeploy observatory (endpoint already added to the systemd unit).
• Report to Ripple ops with evidence: s1/s2 streams lost 24 validators' validations for ledger 106013031 at 04:22:23Z (and 5 more events listed above) that independent stores hold.
• Perfnet repro for the capacity campaign: reduce_relay on, tx burst, instrument per-node validation receipt — measure how squelch redundancy behaves under queue pressure and whether kMaxSelectedPeers/unsquelch timing deserve upstream tuning.
• Optional: AI-generated alert narratives (see notifier src/ai/ pattern — haiku, fixed prompt, small token cap): a Rust-side Messages-API call enriching RELAY_GAP/LOW_QUORUM posts with a 2–3 sentence evidence-based explanation; key via /etc/observatory.env.