Goal: remove the master-mutex serial-apply ceiling. Target: 5–10× sustained TPS lift (from ~159 TPS payment baseline to 800–1500+ on commodity multi-core hardware, scaling further with cores).
Headline approach: Solana-style static conflict-graph scheduling, not Block-STM. XRPL transactions already carry a small, statically-determinable access set per tx type; we extract it, build a conflict graph, dispatch independent groups to a worker pool, and merge per-account writes at the end of the round. No speculation, no abort cost.
Spike coverage: 4 of 68 transactor source files (~6%).
Spiked and validated (plan-1-spike-findings.md):
- ✅
payment/Payment.cpp— fully static access set, all variants (XRP, IOU, MPT, paths, credentials, MPT, DepositPreauth) - ✅
dex/OfferCreate.cpp— fully static including AMM auto-bridging keylets - ✅
dex/AMMDeposit.cpp— partially static: pool keylet derivable from tx, but the pool's pseudo-account ID is stored INSIDE the pool SLE, so accessSetOf needs a closed-ledger snapshot read - ✅
system/Change.cpp— confirmedtouchesGlobalfor SetFee/EnableAmendment/UNLModify
NOT verified — 64 of 68 transactor files: the algorithmic plan assumes all transactors have static access sets (with the snapshot-read pattern for the AMM-style cases). Several transactor categories post-date AMM and may have similar or worse dynamic patterns:
| Category | Transactor files | Risk |
|---|---|---|
vault/ |
XLS-65 Single-Asset Vault — recent | HIGH: vault pseudo-account ID likely follows AMM pattern |
lending/ |
XLS-66 Lending Protocol — brand new (3.1.0) | HIGH: interacts with AMM + Vault. Compound dynamism risk |
bridge/ |
XChain bridges | UNKNOWN: cross-chain claim/commit semantics not audited |
token/ |
MPT (Multi-Purpose Tokens) | MEDIUM: issuance + holder objects, likely static |
permissioned_domain/ |
XLS-80/81 | MEDIUM: domain root + credential lookups |
nft/, credentials/, did/, oracle/, escrow/, check/, payment_channel/, delegate/, account/ |
various | MEDIUM-LOW: appear straightforward but unaudited |
Phase 1.5 of the plan must spike all 17 categories before Phase 1 commits. A finding like the AMM pseudo-account-ID dependency in any of the high-risk categories doesn't break the plan — the snapshot-read pattern handles it — but enough touchesGlobal over-approximations across these would reduce the lift estimate. The current 5-10× TPS claim assumes most workload can parallelize; if AMM/Vault/Lending workloads all serialize via touchesGlobal, the lift on those workloads is closer to 1×.
This doesn't invalidate the plan but does add measurement work to Phase 1 that the original phasing understated.
The unit the scheduler reasons about. One per transaction.
struct AccessSet {
flat_set<AccountID> accounts; // AccountRoot entries read or written
flat_set<Keylet> trustlines; // RippleState entries
flat_set<Keylet> offerBooks; // book directory roots
flat_set<Keylet> ammPools; // AMM root entries
flat_set<Keylet> nftPages; // NFToken page roots
flat_set<Keylet> miscObjects; // escrow / check / paychan / signerlist / etc.
bool touchesGlobal; // SetFee / EnableAmendment / UNLModify / flag-ledger logic
};Conflict rule: two AccessSets conflict iff any of their non-empty corresponding fields intersect OR either has touchesGlobal == true. Same-account transactions always conflict (sequence ordering).
// Input: CanonicalTXSet (already deterministically ordered by consensus)
// Output: groups of mutually-independent txs + per-account sequence chains
struct ConflictGroup {
std::vector<TxRef> sequenced; // ordered, same-conflict-set; apply serially
};
struct Schedule {
std::vector<ConflictGroup> groups; // dispatched to workers; independent across groups
std::vector<TxRef> globalSerial; // touchesGlobal txs; applied last, serially
};Worker policy: bounded work-stealing thread pool, cores - 2 workers (reserve cores for consensus + I/O threads). Groups dispatched in arbitrary order; within-group is sequence-ordered.
struct WriteJournal {
// Each worker's local view of writes performed by its group.
// Reads of unmodified state pass through to the closed-ledger snapshot.
flat_map<Keylet, std::shared_ptr<SLE>> writes;
std::vector<Keylet> deletions;
};At end of round, journals are merged into a single new OpenView in deterministic group order (which doesn't matter — they're independent by construction, so any order produces the same final state).
Each transactor must produce a static AccessSet from the signed transaction body alone — no state lookups. Pessimistic where dynamic discovery would be needed.
| Transactor | AccessSet |
|---|---|
Payment (XRP→XRP) |
{accounts: {src, dst}} |
Payment (IOU, no path) |
{accounts: {src, dst, issuer}, trustlines: {src-issuer, dst-issuer}} |
OfferCancel |
{accounts: {src}, miscObjects: {offer keylet}} |
AccountSet |
{accounts: {src}} |
SetRegularKey |
{accounts: {src}} |
SignerListSet |
{accounts: {src}, miscObjects: {signerlist keylet}} |
TrustSet |
{accounts: {src, issuer}, trustlines: {(src,issuer)}} |
TicketCreate |
{accounts: {src}, miscObjects: {ticket keylets}} |
EscrowCreate |
{accounts: {src, dst}, miscObjects: {escrow keylet, src.dir, dst.dir}} |
EscrowFinish |
{accounts: {src, dst}, miscObjects: {escrow keylet}} |
EscrowCancel |
{accounts: {src, dst}, miscObjects: {escrow keylet}} |
PaymentChannelCreate |
{accounts: {src, dst}, miscObjects: {paychan keylet}} |
PaymentChannelFund |
{accounts: {src}, miscObjects: {paychan keylet}} |
PaymentChannelClaim |
{accounts: {src, dst}, miscObjects: {paychan keylet}} |
CheckCreate |
{accounts: {src, dst}, miscObjects: {check keylet}} |
CheckCash |
{accounts: {src, dst, issuer}, trustlines: optional, miscObjects: {check keylet}} |
CheckCancel |
{accounts: {src}, miscObjects: {check keylet}} |
DepositPreauth |
{accounts: {src, auth}, miscObjects: {preauth keylet}} |
AccountDelete |
{accounts: {src, dst}, miscObjects: {all src-owned objects}} |
NFTokenMint |
{accounts: {src}, nftPages: {src.nft pages}} |
NFTokenBurn |
{accounts: {src, owner}, nftPages: {owner.nft pages}} |
NFTokenCreateOffer |
{accounts: {src}, miscObjects: {nft offer keylet, nft directory}} |
NFTokenCancelOffer |
{accounts: {src}, miscObjects: {offer keylets…}} |
NFTokenAcceptOffer |
{accounts: {src, seller, buyer}, miscObjects: {offer keylets, nft pages}} |
| Transactor | AccessSet | Note |
|---|---|---|
Payment w/ paths |
{accounts: {src, dst, all path accounts}, trustlines: {all path trustlines}, ammPools: {all path AMMs}} |
Paths are in the tx body — static |
OfferCreate |
{accounts: {src}, offerBooks: {target book keylet}} |
Crosses other offers; pessimistically lock the whole book |
OfferCreate w/ paths |
Same + path access | Rare; treat as global if path is complex |
AMMDeposit |
{accounts: {src}, ammPools: {pool}, trustlines: {src ↔ each token, src ↔ LP-token, pool ↔ each token}} |
LP token trustline is critical |
AMMWithdraw |
Same as Deposit | |
AMMVote |
{accounts: {src}, ammPools: {pool}} |
|
AMMBid |
{accounts: {src}, ammPools: {pool}, trustlines: {src ↔ LP-token}} |
|
AMMCreate |
{touchesGlobal: true} |
Creates new pool keylet; pessimistically global |
AMMDelete |
{ammPools: {pool}, trustlines: pool's …} |
|
Clawback |
{accounts: {issuer, holder}, trustlines: {holder ↔ issuer}} |
|
XChainAccountCreateCommit etc. (bridge) |
{accounts: {…}, miscObjects: {bridge keylet, claim keylet}} |
Bridge state |
DIDSet / DIDDelete |
{accounts: {src}, miscObjects: {did keylet}} |
|
Permissioned* (XLS-80/81) |
Domain root + accountRoots; pessimistic about domain | |
MPT* (XLS-33) |
{accounts: {src, holder}, miscObjects: {issuance keylet, holder issuance keylet}} |
|
VaultDeposit/Withdraw (XLS-65) |
Vault root + associated trustlines | Recent — needs deep audit |
LoanSet/Pay/Default (XLS-66 Lending) |
Loan keylet + AMM + vault interactions | Recent — needs deep audit |
| Transactor | Reason |
|---|---|
SetFee |
Fee voting; ledger-wide effect |
EnableAmendment |
Amendment state |
UNLModify |
NegativeUNL state |
Every 256 ledgers ("flag ledger"), RCLConsensus::onClose runs fee voting, amendment voting, NegativeUNL voting (RCLConsensus.cpp:360-383). On flag ledgers, short-circuit to fully serial apply. The marginal lift loss is 1/256 of throughput — negligible.
OpenLedger.cpp:40-51 — every modification:
- Copy the entire current
OpenView(full COW) - Mutate the copy
- Atomically swap into
current_
This is serial by construction and allocation-heavy.
┌─────────────────────────────────────┐
│ Closed-ledger snapshot (read) │
│ (immutable for the round) │
└────────────┬────────────────────────┘
│
┌──────────────┼──────────────┐
▼ ▼ ▼
┌───────────┐ ┌───────────┐ ┌───────────┐
│ Worker 1 │ │ Worker 2 │ │ Worker N │
│ Journal A │ │ Journal B │ │ Journal Z │
│ {writes} │ │ {writes} │ │ {writes} │
└───────────┘ └───────────┘ └───────────┘
│ │ │
└──────────────┴──────────────┘
│ merge (deterministic order)
▼
New OpenView (committed)
Read semantics: a worker reading a Keylet k checks its own journal first, then the closed-ledger snapshot. It never reads from another worker's journal — that's why disjoint AccessSets are the prerequisite.
Write semantics: writes go to the worker's local journal. At end of round, journals are merged. Because groups are conflict-free by construction, merge order doesn't affect the result — but we pick a deterministic order anyway (sorted by lowest tx hash in the group) to keep the path that produced the state observable.
This is the part of Plan 1 that needs the most design review. Alternatives considered:
- MVCC (Block-STM style): every write produces a versioned entry. Too memory-heavy for XRPL's ledger sizes; we don't need speculation.
- Per-account COW (intermediate): each AccountRoot copies on write; trustlines/offers/AMMs do too. Possible but more bookkeeping than single-journal-per-worker.
- Lock-free: appealing but the SHAMap operations beneath aren't lock-free. Worker-journal approach sidesteps that.
The worker-journal approach is the chosen design unless P2 (scheduler isolation testing) reveals it's wrong.
Goal: end Phase 1 with every transactor declaring its access set correctly, validated by a runtime assertion against actual state touched.
- 1.1 Define
AccessSetstruct ininclude/xrpl/protocol/AccessSet.h. Header-only, allocation-free, comparable. - 1.2 Add
static AccessSet accessSetOf(STTx const&)per transactor undersrc/xrpld/app/tx/. One file per tx type; each <100 LOC. - 1.3 Add DEBUG-build assertion in
Transactor::apply(): track everypeek(),update(),insert(),erase()call against the open view; at end-of-apply assert the touched-keylet set ⊆ declaredAccessSet. - 1.4 Run the full test suite. Every assertion violation is either:
- A bug in
accessSetOf(fix) - A transactor that legitimately touches state it can't predict statically (mark as
touchesGlobal)
- A bug in
- 1.5 Audit the hard transactors (table above): Payment-with-paths, OfferCreate, all AMM types, Clawback, MPT, Vault, Lending. Pair with the transactor author or owner where possible.
- 1.6 Document the access-set contract in
src/xrpld/app/tx/README.md. Make it a CI gate: new transactors must implementaccessSetOfor explicitly marktouchesGlobal.
Phase 1 deliverable: PR adding access-set extraction across all transactors, with DEBUG-build assertion. No behavioral change to the network. Mergeable as soon as it's green.
Risk in this phase: any transactor we mis-classify creates a future correctness bug. Mitigation: the DEBUG assertion catches under-specification; AMM/Vault/Lending pair-review catches over-specification (which is safe but kills parallelism).
Goal: build the prio-graph scheduler as a pure function, validate against mainnet history.
- 2.1 Implement
Schedule scheduleApply(CanonicalTXSet const&)insrc/xrpld/app/ledger/Schedule.{h,cpp}. Builds conflict graph; topologically partitions into groups + sequence chains. - 2.2 Replay test harness: for a corpus of mainnet ledgers, run the scheduler to produce
Schedule, then serially apply in the scheduler's group order, and assert the resulting state root equals the original mainnet state root. - 2.3 Stress harness: synthetic workloads with known contention patterns:
- 10,000 independent payments between disjoint accounts (target: 10,000 single-tx groups)
- 1,000 OfferCreates on the same book (target: 1 sequenced group)
- Mixed AMM deposit/withdraw/swap on the same pool (target: 1 sequenced group)
- Payments with overlapping paths
- 2.4 Metrics: average group size, max group size, fraction of txs that are
touchesGlobal, scheduler latency per ledger.
Phase 2 deliverable: scheduler module + replay harness in CI + corpus of 6 months of mainnet ledgers replayed with byte-identical state roots.
Risk: scheduler bug that produces a different canonical order than the conservative serial order. Mitigation: differential state-root CI gate (X4 in todo.md) is mandatory.
Goal: actually parallelize.
- 3.1 Implement
WriteJournaland per-worker apply path. Workers see anApplyViewthat consults journal → closed-ledger snapshot. - 3.2 Worker pool:
boost::asio::thread_poolor equivalent,cores - 2workers. Bounded work-stealing. - 3.3 End-of-round merge: walk journals in deterministic order, apply writes to a single new
OpenView. - 3.4 Feature flag
parallel_applyin Config.h (config-level, not amendment-gated). Off by default. - 3.5 Differential state-root assertion in CI — every test run executes both serial and parallel paths, asserts equality.
- 3.6 Devnet deployment with
parallel_apply = truefor 4 weeks. Watch for: state-root divergences (zero tolerance), apply latency distribution, worker utilization.
Phase 3 deliverable: parallel apply running on devnet, byte-identical to serial, behind operator-config flag.
Risk: subtle non-determinism — anything that depends on apply order within a conflict-free group (e.g., transient fee snapshots, log-line ordering that other code parses, RNG seeded from now()). These are the bugs that take longest to find. The replay harness is the primary defense.
Goal: prove the win and prove safety at scale.
- 4.1 Run RippleX Lending Performance harness on testnet with
parallel_applyenabled. Target: ≥1000 TPS sustained payment baseline (vs the 159 TPS measured on serial). Measure: payment TPS, mixed-workload TPS, consensus latency distribution, memory growth, worker CPU utilization. - 4.2 Replay last 12 months of mainnet against the parallel path. Zero state-root divergences required.
- 4.3 Antithesis-style stochastic testing — deliberately inject thread scheduling perturbations to surface races. (RippleX is already using Antithesis per the dev.to article; extend their harness.)
- 4.4 Adversarial workloads: a single account spamming txs with the same access set (proves serial-chain handling); 10,000 accounts each spamming independent payments (proves parallelism); mixed AMM contention (proves the pessimistic AMM access set doesn't kill throughput catastrophically).
Phase 4 deliverable: testnet running parallel apply for ≥6 weeks under continuous load, no incidents, measured TPS at target.
Goal: ship.
- 5.1 Submit amendment
parallelApply(or similar name). Activate behavior gated on amendment, with a fallback path that keeps serial apply if the amendment is disabled (so a downgrade path exists). - 5.2 Operator communication 8 weeks before vote: documented config, monitoring guidance, rollback procedure.
- 5.3 Vote, activation, monitor.
- 5.4 Post-activation: instrument apply latency distribution, worker utilization, group-size histogram. Tune worker pool size if needed.
Phase 5 deliverable: parallel apply on mainnet, ≥1000 TPS sustained available to the network.
- Determinism across validators. Any non-determinism — work-stealing order leaking into observed state, FP rounding in cross-account fee distribution, RNG that isn't deterministically seeded — forks the network. The replay harness + differential CI is mandatory; no merge without it.
- AMM internal state. AMMs hold their own trustlines and own an LP token issuer. An
AMMDepositreads/writes the pool root + the depositor's LP trustline + the depositor's underlying trustlines + the pool's underlying trustlines. The full enumeration is non-trivial; pair-review with AMM authors. - Cross-currency Payment paths. A path can traverse arbitrary trustlines and AMM pools. The path is static in the tx body, but discovery of which trustlines exist on the path requires reading state. Either: enumerate every keylet the path could touch (over-approximate; safe), or treat path-Payments as
touchesGlobal(simple; kills parallelism for a meaningful workload). Phase 1.5 decides. - OfferCreate book locking. Pessimistic full-book lock means all offers on the same book serialize. For active books (XRP/USD on Bitstamp issuer) this is the entire book traffic. Fine for v1; v2 could refine to per-price-level locking.
- Flag ledgers. Short-circuit to serial — verified.
- Failed-tx handling. A tx that fails to apply still consumes its sequence number on its source account. The journal must record this even on failure.
- Retry passes (
LEDGER_TOTAL_PASSES = 3). The retry loop exists because some txs fail-then-succeed when state changes. Under parallel apply this is more subtle — a tx in group A might depend on a tx in group B, but if the AccessSet is correct, this can't happen by construction. Validate that the retry pass becomes unnecessary post-parallelization, or scope it to within-group sequence chains.
- Worker pool sizing. Too few workers → no parallelism. Too many → context-switch overhead.
cores - 2is the starting point; tune in Phase 5. - Journal merge cost. End-of-round merge is itself serial. If groups are tiny (e.g., 4000 single-tx groups), merge cost may eat the gains. Measure in Phase 4.4.
- Cache effects. Cross-thread reads of the closed-ledger snapshot through
intrusive_ptrreference counts may cause cache-line ping-pong on hot objects (e.g., the issuer's AccountRoot for a popular IOU). May need read-onlyconst&snapshots that don't touch refcounts.
- Validator hardware heterogeneity. UNL validators run different hardware. Parallel apply makes throughput more variable across validators. As long as all of them stay within
ledgerMinConsensus, this is fine — but watch for slow validators dropping out under heavier loads. - Memory. Per-thread journals + larger speculative ledger size = more memory. RippleX is already chasing 20+ GB SHAMap memory; budget for this.
- Rollback. If a problem appears post-activation, the amendment can be disabled, but old serial-apply code must remain in the tree as the fallback path. Don't delete the old path until the amendment has run clean for 6+ months.
6. Test infrastructure (X1 from todo.md)
These need to exist before Phase 2 can land:
- X1.1 Mainnet ledger corpus: download last 12 months of mainnet ledger headers + transaction sets + state diffs. Storage: ~hundreds of GB.
- X1.2 Replay engine: pure-function
(closedLedger, txSet) → newLedger. Already exists in xrpld test paths; harden it for batch use. - X1.3 State-root differential:
assert(replay(closed, txs).hash() == mainnet.hash()). Run as nightly CI on the corpus. - X1.4 Stochastic test harness (Antithesis-style): inject thread scheduling perturbations into Phase 3+ builds. RippleX already has this contract — extend.
Team composition (recommended):
- 1 architect / tech lead (owns the design decisions across all 5 phases)
- 2 senior engineers (split: scheduler + worker pool / OpenLedger refactor + journal)
- 1 engineer on the access-set audit per transactor (Phase 1 owner, then transitions to consensus integration)
- 1 engineer on test infrastructure (X1.*, replay harness, differential CI) — runs in parallel from month 1
Total: ~36 person-months over 12 calendar months (including amendment governance overlap).
Single biggest schedule risk: Phase 1.5 — the audit of hard transactors (AMM, Vault, Lending, MPT, paths). If we discover that AMM access sets cannot be expressed statically without touchesGlobal, parallel apply lifts substantially less for DEX/AMM workloads, and we're back to "great for payments, not for AMM." Front-load this audit; don't wait until month 4.
Single biggest correctness risk: non-determinism in the apply path leaking into state. Mitigation is X1.3 — differential state-root CI gate. Make this non-bypassable.
- Not Block-STM. No speculation, no abort, no multi-version state. We have static access sets; we use them.
- Not sharded execution across machines (Aptos Shardines). Single-machine, multi-core. If we later need horizontal sharding, P1's worker-journal design extends naturally, but that's a separate quarter-scale project.
- Not a consensus-layer change. Consensus already produces a
CanonicalTXSet; we change how that set is applied, not how it's agreed. - Not a hash-function change. SHA-512Half stays. Plan 5 is independent.
- Not a fast-path bypass. Sui-Lutris-style consensus skipping for owned-account txns is a separate, larger plan (Tier S4 in the audit). Do this first.
- Amendment naming.
parallelApply/fasterApply/ something else — what conforms to XRPL convention? - Worker-pool config. Should it be operator-tunable (config file) or fixed to
cores - 2? - Rollback policy. Keep serial apply in the tree as the fallback path indefinitely? Or remove after N months of clean amendment operation?
- Hard-case scope: are we okay with
touchesGlobalover-approximation for path-Payments + OfferCreate book-lock in v1, accepting that the AMM/DEX TPS lift is more modest than the payment TPS lift? - Test corpus retention. 12 months of mainnet history vs longer? Storage cost vs replay confidence trade-off.