Skip to content

Instantly share code, notes, and snippets.

@dangell7

dangell7/weakness-0-consensus-liveness.md

Created August 11, 2026 16:04
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dangell7/7fe6769563e099b31c70fcf355d674cc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dangell7/7fe6769563e099b31c70fcf355d674cc to your computer and use it in GitHub Desktop.
Download ZIP
XRPL Weakness 0 — consensus liveness under job-queue saturation (load-fee escalation dead code)
Raw
weakness-0-consensus-liveness.md

Weakness #0 — consensus liveness under job-queue saturation

Status: investigated (2026-08-01), fix PR in progress Origin: the "XRPL falls over at 20–25M accounts" stress campaign. On a real 5-validator perf net under sustained live account creation, consensus lost liveness — LoadMonitor:WRN Job: AcceptLedger run: 19s, validators fell behind, dropped quorum, and the network deep-forked with no self-heal (only a genesis reset recovered it). Masked operationally by resizing validators 8→16 vCPU; the protocol weaknesses remain.

Three read-only investigations (code refs against Transia develop). Weakness #0 is not one bug — it is two independent, complementary defects plus one correct behavior that must NOT be "fixed".

Defect 1 — local load-fee escalation is dead code (PR2, the root cause)

File: src/xrpld/app/main/LoadManager.cpp, LoadManager::run().

The per-tick load-feedback block — if (getJobQueue().isOverloaded()) raiseLocalFee(); else lowerLocalFee(); if (change) reportFeeChange(); — sits outside the while (true) loop, at function scope after the loop's closing brace. The loop's only exit is break when stop_ is set (shutdown). So the fee adjustment runs exactly once, at server shutdown, where it is useless. During live operation the local load fee is never raised under load.

Consequence: localTxnLoadFee_ stays pinned at normal (LoadFeeTrack.cpp), so scaleFeeLoad never escalates the open-ledger cost. The node keeps advertising near-normal fees while its job queue saturates — there is no fee-based backpressure telling clients/relays to back off. The raiseLocalFee/lowerLocalFee implementations are correct but orphaned (their only callers are this dead block).

Regression: commit d36024394d ("Report job queue data if a deadlock is detected", Nik Bougalis, 2021-11-16) — a logging refactor that moved the loop's closing brace before the fee block. Latent since Nov 2021; every subsequent rename/tidy preserved the broken structure. This is upstream rippled, not Transia-specific — mainnet nodes have had dead local-load-fee escalation for ~4 years.

Fix (PR2, dangell7/load-fee-escalation): relocate the fee block back inside the loop so it runs every ~1 s tick; the break still exits at shutdown. One-block move.

Validation: the rig. On the unpatched build, saturate the queue and watch server_info load_factor/local fee stay pinned; on the patched build it rises and sheds load. This is the campaign's "break it, then show the fix holds" phase — not a unit test (the loop is a 1 s-tick threaded, Application-coupled path; forcing isOverloaded() in a unit test is decay-timing fragile and would test LoadMonitor plumbing, not the placement fix).

Defect 2 — JobQueue has no consensus-thread protection (PR3)

Even with backpressure restored, scheduling can still starve consensus under a transient flood.

  • Fixed worker pool, no preemption. ~6 threads (down to ~4 on small boxes; Application.cpp:330-353). A worker runs one job to completion (JobQueue.cpp:342-390).
  • Priority is selection-order only (Job.cpp:83-93, getNextJob JobQueue.cpp:296): JtAccept=30 outranks transaction/client/ledger-data work — but only when a thread is free.
  • Flood types have unlimited concurrency. JtTransaction, JtBatch, JtClient*, JtRpc, JtValidationT, JtWrite are maxLimit (INT_MAX) in the JobTypes table (JobTypes.h:54-100). With no per-type cap and no reserved thread, a flood can occupy every worker on long low-priority jobs; a freshly-enqueued AcceptLedger waits behind them (the wait component).
  • JtAccept has 0 ms / 0 ms latency targets (JobTypes.h:81); isOverTarget returns false when targets are 0 (LoadMonitor.cpp:133-139). So AcceptLedger's own 19 s never trips isOverloaded() and never raises the fee — the overload signal is structurally blind to it.

Fix (PR3): (A) reserve worker slots for consensus-critical jobs in getNextJob — don't let low-priority work consume the last 1–2 threads; and/or (B) cap flood-type concurrency (finite limit reusing the existing deferred/finishJob path). Complementary one-liner: give JtAccept non-zero targets so a slow AcceptLedger feeds the (now-live) fee signal. Higher risk than PR2 — needs rig tuning of the reserve count. Separate PR so the trivially-correct PR2 regression fix isn't held hostage to a behavioral scheduling change.

Note: the AcceptLedger body itself (applying the full canonical set synchronously, RCLConsensus.cpp:520-527) can take many seconds on a large set — that's a distinct apply-throughput problem, out of scope here.

Not a PR — deep-fork self-heal would break finality

Deliberately not pursuing auto-recovery from a deep fork. Findings:

  • The quorum stall is correct. 5 validators → quorum 4 (ValidatorList.cpp:1888, max(ceil(0.8·eff), ceil(0.6·unl))). Lose 2 → <4 → no ledger fully validates → validLedgerSeq_ freezes. This is the intended 80% Byzantine-safety / halt-rather-than-diverge tradeoff. Negative UNL is a slow preventive tool (one change per 256-ledger flag ledger, needs validation to commit) and structurally cannot rescue an already-stalled net.
  • Non-reconvergence is downstream of that. "Preferred ledger by branch" (getPreferred, LedgerTrie) only counts branches whose ledgers were acquired (Validations.h:461-477); a stalled/partitioned node can't fetch the majority chain (the RCLValidations.cpp:131 "Need validated ledger" log), so the trie never points it home. Compounded by online_delete pruning the intervening ledgers on peers.
  • areCompatible is a genuine finality invariant (View.cpp:134-189, gated in NetworkOPs.cpp:1983-1991). A node that fully-validated a ledger past the divergence point will refuse to follow a supermajority — correctly. An automatic "abandon my validated ledger" path would trade away XRPL's finality guarantee. Do not build it.

Remedy is operational/observability: quorum-loss + fork alerting, adequate validator count, and a documented (optionally admin-RPC-gated) operator resync/reset. Fits the monitoring roadmap, not a consensus code change.

Summary

# Defect Correct? Action
1 LoadManager fee-escalation dead (outside loop, since d36024394d) bug PR2 — move block into loop
2 JobQueue: no consensus-thread reservation; flood types unlimited; JtAccept 0 ms targets bug PR3 — reserve threads / cap floods / non-zero targets
— Deep-fork non-self-heal correct (finality + quorum) operational alerting only; do NOT auto-abandon validated ledgers

Related: [[project_capacity_hardening_prs]], the generational-GC PR1 (proposals/online-delete-generational-gc.md), and the observability roadmap (proposals/network-capacity-6-month-roadmap.md).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment