Goal: materialize XRPL state as a flat keylet→SLE map. Apply does dual-write (SHAMap stays authoritative, flat map indexes reads). Reads go to the flat map only — no fallback path. This is the pure NoSQL 2-writes-for-1-read pattern. Direct TPS lift is moderate-to-large; indirect value is the larger story — this unlocks downstream consumers (indexers, light clients, hooks, AMM bots, sidecar services) that currently can't query XRPL state at scale.
Why this is Plan 6: the SHAMap is a NoSQL substrate (keylet→SLE Merkle trie) currently being used in a SQL access pattern (descend the index on every read). Plan 6 finishes what the substrate was designed for: materialize the read view. Plan 5 (incremental hashing) becomes mostly obsolete once Plan 7 (deferred SHAMap, the structural follow-on to this plan) lands.
Today every state read in rippled/xrpld descends the SHAMap. The descent is:
- Take the root, do a spinlock-protected lookup of branch[nibble₀] → child node
- If child is an inner node, recurse to branch[nibble₁]
- After ~6–10 levels (practical, log₁₆ of ~10M state entries), reach the leaf
- Read the SLE from the leaf
Per descent cost:
- 6–10 inner-node lookups
- Each lookup acquires a
PackedSpinlock(one bit in anatomic<uint16_t>) — at high contention this is real - Each inner node may need to be fetched from the TaggedCache; cache misses go to NodeStore (disk!)
- ~6–10 cache-line touches even on a warm cache
Reads-per-tx vary by tx type but typical Payment reads ~5–10 SLEs (src AccountRoot, dst AccountRoot, src trustline, dst trustline, src OwnerDir, fee settings, etc.). For 4000 tx/ledger, that's 20k–40k SHAMap descents per ledger close on the apply path alone. RPC + ledger queries multiply this.
The flat lookup replaces each descent with one unordered_map::find().
Two separate structures:
| Structure | Role | Updates |
|---|---|---|
| SHAMap (existing) | Authoritative — produces the state root that consensus agrees on. Provides Merkle proofs. | On apply; full COW snapshot per closed ledger. Stays as-is. |
| FlatStateMap (new) | Read-side index. unordered_map<uint256, shared_ptr<SLE const>> keyed by SHAMap key. |
Maintained alongside SHAMap; same lifecycle. |
The SHAMap is unchanged. Its hash, its Merkle proof generation, its consensus role, its on-wire serialization — all unchanged. No amendment required.
The FlatStateMap is a purely auxiliary cache. If it's wrong or absent, the system falls back to SHAMap descent. This is the safety property that lets Plan 6 ship without an amendment.
Lifecycle (per Ledger object):
┌──────────────────────────────┐
Closed Ledger N ──>│ SHAMap (authoritative) │
│ FlatStateMap (read fast) │
│ Pointers share SLE objects │
└──────────────────────────────┘
│
│ close N+1
▼
┌──────────────────────────────┐
Closed Ledger N+1 │ SHAMap (COW from N + diff) │
│ FlatStateMap (COW from N + │
│ diff; shares unchanged SLEs│
│ with N's flat map) │
└──────────────────────────────┘
The COW property is preserved via shared SLE pointers. If ledger N+1 only modifies 100 SLEs out of 10M, ledger N+1's FlatStateMap has 100 new pointers and 9,999,900 pointers shared with ledger N (via a persistent / copy-on-write hash map; see §4).
ReadView interface stays unchanged. The internal implementation changes; consumers see the speedup transparently:
// Existing API — semantics unchanged, implementation faster
std::shared_ptr<SLE const> ReadView::read(Keylet const& k) const;
bool ReadView::exists(Keylet const& k) const;For external consumers (the "other project" enablement):
// New public API — exposes the flat map directly for bulk consumers
class FlatStateView {
public:
// O(1) keylet lookup, no SHAMap descent
std::shared_ptr<SLE const> read(uint256 const& key) const;
// Iterate all SLEs of a given type — backed by a secondary index
template <typename F>
void forEachOfType(LedgerEntryType type, F&& visitor) const;
// Diff between two ledger versions — for indexers
StateDiff diff(LedgerIndex from, LedgerIndex to) const;
// Subscribe to state changes per ledger — for streaming consumers
Subscription subscribeStateUpdates(StateUpdateCallback);
};The "other project" likely needs one or more of:
- Indexers / explorers — bulk read at ledger close
- Light clients — Merkle proofs (still served from SHAMap) + fast reads (served from flat map)
- Hooks / smart contract execution — random-access state reads at execution time
- AMM/DEX bots — sub-millisecond pool state queries
- Sidecar services mirroring state in a different store (Postgres, Redis, columnar)
The subscribeStateUpdates + StateDiff API gives external services an authoritative incremental feed without polling.
| Option | Reads | Writes | COW | Memory | Notes |
|---|---|---|---|---|---|
A. std::unordered_map, rebuilt per ledger |
O(1) | O(1) | Re-allocate everything per ledger | High churn | Simplest; OK if rebuild cost is small |
B. std::unordered_map + COW per ledger via shared_ptr to buckets |
O(1) | O(1) avg, O(N) worst (rehash) | Per-bucket COW | ~2× pointers | Awkward to make truly COW |
| C. Persistent HAMT (Hash Array Mapped Trie) | O(log₃₂ N) ~ effectively O(1) | O(log₃₂ N) | Structural sharing native | +~5–10% pointers | Best COW story; Clojure / Scala use this |
| D. Two-tier: immutable "base" map per closed ledger + mutable overlay for open ledger | O(1) | O(1) | Snapshot-on-close | Low overhead | Pragmatic; matches existing OpenLedger/Closed semantics |
Recommendation: Option D. Matches how XRPL ledgers already work — open ledger is mutable, closed ledger is immutable. The flat map's open-ledger overlay holds in-flight writes; on close, the overlay merges into a new immutable base. Read path: check overlay first, then base.
Option C (HAMT) is the elegant long-term answer if we end up needing many concurrent historical ledger snapshots, but D ships sooner and covers 99% of read traffic.
Once we have a flat map, secondary indexes by LedgerEntryType become trivial:
flat_hash_map<LedgerEntryType, intrusive::list<SLE_entry>> by_type_;This makes queries like "all AMM pools" / "all offers on this book" sub-millisecond, which is exactly what indexer-class consumers need. Currently these queries require full SHAMap traversal.
The flat map is read-mostly during apply. Writes happen from one thread (the apply thread) per tx; reads happen from many threads (RPC, peer-overlay validators verifying proposals, etc.).
For Plan 1 (parallel apply) this becomes multi-writer. The recommended design:
- Reads always lock-free via
std::atomic<std::shared_ptr<FlatStateMap>>swap-on-close. RPC threads load the current pointer once and read against it; reads never block writes. - Writes during apply go to a per-worker overlay (matching Plan 1's per-worker journal), merged at end-of-round.
This works because the flat map mirrors the SHAMap, and Plan 1's scheduler already proves conflict-free access sets — so two workers can't both be writing the same flat-map entry.
Goal: the flat map is the only read source. Reads never descend the SHAMap. This is the 2w/1r pattern in full.
- P6.1 Add
FlatStateMapdata structure (src/libxrpl/ledger/FlatStateMap.{h,cpp}). Option D layout — immutable base per closed ledger + mutable overlay for open ledger. - P6.2 Eager startup populate. On node startup (after the most-recent closed ledger is loaded from NodeStore), walk the SHAMap once and build the flat map. Estimated ~60–100 s for current mainnet state (~10M SLEs). Optionally persist the built flat map to a sidecar store keyed by ledger hash so warm restarts skip this work entirely.
- P6.3 Dual-write on apply. Every
view.peek()/update()/insert()/erase()updates the flat-map overlay in lockstep with the SHAMap. This is the "2× write" cost — small (one hash-map insert per mutation vs. ~8 inner-node hashes today). - P6.4 Reads go to flat map only.
ReadView::read(Keylet)reads fromflat.find(key). No SHAMap fallback. A miss is a bug — assert in debug, log + crash in release. The SHAMap and flat map are invariants of each other. - P6.5 Differential invariant check runs in CI and in DEBUG builds at runtime: after every
OpenLedger::accept(), walk both structures and assert byte-identical(keylet → SLE hash)mappings. This is the non-bypassable correctness gate that replaces the runtime fallback. - P6.6 On
OpenLedger::accept(), freeze the overlay into a new immutable base for the now-closed ledger.
Deliverable: every read in the system is O(1). SHAMap descent is dead code on the read path — only consulted during startup populate and Merkle-proof generation. Zero behavior change observable to the network.
Why no fallback is the right call:
- A fallback path hides drift. A flat-map miss that silently SHAMap-descends means a real bug runs undetected, possibly for weeks, until something else exposes the inconsistency.
- The startup cost is bounded (~minutes) and amortized over node-session lifetime (days/weeks).
- The differential invariant check (P6.5) catches divergence at the source — at the close boundary — instead of via mysterious read latency spikes.
Goal: keep flat maps for the last N closed ledgers (default N=20) so recent-history queries are fast.
- P6.6 Closed ledgers retain their immutable flat-map base. Old ledgers evict their flat map (SHAMap stays in the LedgerHistory cache).
- P6.7 Memory measurement: per closed ledger flat map size =
8 bytes pointer × number of SLEs in ledger ≈ 8 × 10M ≈ 80 MB. For N=20 ledgers retained: ~1.6 GB. Verify against the RippleX memory budget (current SHAMap+cache is ~20 GB; +8% is acceptable). - P6.8 Operator config:
flat_state_retained_ledgers = 20(range 5–256).
Deliverable: RPC queries against recent ledgers (~30 seconds of history) are 5–10× faster.
Goal: expose the flat map to external consumers and provide type-indexed iteration.
- P6.9
class FlatStateView : public ReadView— public-facing read view with O(1) reads and type-indexed iteration. - P6.10 Add secondary indexes by
LedgerEntryType. Update on every flat-map write. ExposeforEachOfType()andforEachInRange(). - P6.11
StateDiff(from, to)— returns the set of(keylet, before, after)between two ledger versions. Implementable asflat_map[to].diff(flat_map[from]). - P6.12
subscribeStateUpdates()— streaming callback API; called once per ledger close with the StateDiff. The "other project" consumes this. - P6.13 RPC endpoint
state_diffandstate_by_typefor off-process consumers.
Deliverable: external consumers can mirror XRPL state at sub-second latency via streaming callbacks or RPC.
Goal: multi-writer overlay consistent with Plan 1's worker-journal model.
- P6.14 Per-worker flat-map overlay (matches Plan 1 §3's per-worker
WriteJournal). - P6.15 Merge per-worker overlays into the round's overlay at end-of-round, in the same deterministic order Plan 1 uses.
- P6.16 Verify byte-identical SHAMap roots: the flat map's writes are validated against SHAMap writes via the differential CI gate already required by Plan 1.
If Plan 1 ships first, this is a small extension. If Plan 6 ships first, Plan 1 inherits the overlay design.
- P6.17 Run the mainnet replay corpus through the new read path. Every
ReadView::read()result must be byte-identical to the SHAMap-descent result. - P6.18 Devnet/testnet for 4 weeks under load. Measure: read throughput, flat-map hit rate, memory delta, RPC latency distribution.
- P6.19 Ship as a feature in the next minor release; no amendment needed (the flat map is purely auxiliary).
Per closed ledger, the flat map adds:
overhead per SLE:
- hash bucket entry: ~16 bytes (key + pointer + next-bucket)
- shared_ptr control block: shared with existing SLE (no extra)
- secondary index node: ~24 bytes
total: ~40 bytes per SLE
At ~10M SLEs in current mainnet state: ~400 MB per ledger flat map. For N=20 retained ledgers: ~8 GB.
That's substantial. Mitigations:
- The SLE objects themselves are shared via
shared_ptracross ledger versions; only the pointers + bucket overhead are duplicated - Persistent HAMT (Option C in §4) cuts this dramatically by structurally sharing unchanged buckets across versions
- Lower default
flat_state_retained_ledgersto 5 if 8 GB is too much; that's ~2 GB
Trade-off: validators with tight memory budgets can disable Plan 6 entirely (operator config) and fall back to SHAMap descent. The flat map is opt-in (config-gated, not amendment-gated). External consumers running indexer / RPC-heavy nodes get the speedup; validators with memory pressure don't pay for it.
- Synchronization with SHAMap. Every SHAMap mutation must produce the corresponding flat-map mutation. Mitigation: route all writes through a single
StateWriterabstraction that updates both; differential CI gate (compareflat_map.read(k)vsshamap.read(k)for all k after every test) catches drift. - Lifetime of cached SLEs. Flat map holds
shared_ptr<SLE const>. SHAMap also holds them (via SHAMap leaf nodes). Both must agree on lifetime; SLEs are immutable once published, so this is straightforward but easy to get wrong with COW semantics. - Subscription correctness. If a subscriber goes silent, do we buffer or drop? Default: drop (lossy stream; consumer reconnects with
StateDiff(last_seen_ledger, current)).
- Memory growth. Per-ledger flat-map overhead is the dominant cost. Mitigation: low retention default, operator config, persistent HAMT as a v2.
- Hot-bucket contention. During parallel apply, two workers writing to the same hash bucket (different keys, same bucket) would serialize. Mitigation: per-worker overlays merged at end of round — no concurrent bucket access during apply.
- Feature flag complexity. Plan 6 ships as
flat_state = enabled(default true). Validators with memory pressure can disable. If disabled, reads fall back to SHAMap descent — no other behavioral change.
Plan 5 (incremental SHAMap hashing) addresses the per-write SHAMap-inner-node hash cost. With Plan 6 alone, this cost still exists — apply does dual-write to SHAMap + flat map, and the SHAMap update still walks dirtyUp(). Plan 5's ~1.05–1.10× lift is real on top of Plan 6.
Plan 7 (deferred SHAMap construction) is the structural follow-on: the SHAMap is no longer updated during apply. Writes touch the flat map only; the SHAMap is rebuilt at close from the flat map's modified set. See plan-7-deferred-shamap.md. When Plan 7 ships, Plan 5 becomes obsolete — there's no per-write SHAMap hashing to optimize. The inner-node hashing moves to a single batched bottom-up build at close, which is embarrassingly parallel by subtree.
Recommended sequence:
- Plan 6 (this plan) — 2w/1r in full, no fallback. ~1 month. Lifts reads.
- Plan 7 — deferred SHAMap. ~3–4 months including amendment. Lifts writes by removing per-write SHAMap maintenance.
- Plan 5 — only if Plan 7 doesn't ship for whatever reason. Otherwise deprecated.
Xahau ships an in-memory database called RWDB (github.com/Xahau/xahaud RWDBFactory.cpp). A reviewer asked whether Plan 6 duplicates it. It does not — they sit at different layers and compose.
| Layer | What lives here | Xahau status | This plan |
|---|---|---|---|
| NodeStore backend (storage medium) | Compressed node blobs keyed by node hash | RWDB replaces NuDB/RocksDB with std::map<uint256, vector<uint8_t>> in RAM. NOT persistent — wiped on restart. |
Untouched. |
| SHAMap (Merkle structure) | The authoritative state-root trie | Unchanged in Xahau — still maintained incrementally per-write | Plan 7 makes it deferred. |
| Read pattern (how an SLE is found) | ReadView::read(Keylet) → SHAMap descent → 6–10 spinlocked inner-node fetches |
Unchanged in Xahau — RWDB still requires SHAMap descent and per-inner-node decompression | Plan 6 makes it O(1). |
Plan 6 is most impactful on top of RWDB. RWDB removes disk latency but still pays per-inner-node mutex + decompression per read. Plan 6 skips the descent entirely. RWDB + Plan 6 + Plan 7 is the natural endpoint:
- RWDB: where the bytes live (RAM)
- Plan 7: when the Merkle authority is built (at close, not per-write)
- Plan 6: how state is looked up (O(1) flat map, not SHAMap descent)
Three distinct optimizations; Xahau did one; this plan + Plan 7 do the other two.
- External consumer authentication. If the
subscribeStateUpdatesAPI is exposed via gRPC or WebSocket, what auth model? Likely defer to existing RPC auth. - Snapshot retention vs. memory. The default
flat_state_retained_ledgers = 20is a guess. Tune from operator feedback in Phase 5. - Persistent HAMT (Option C) vs. snapshot-array (Option D). Option D ships first; Option C is a v2 if memory pressure justifies it.
- gRPC streaming vs. WebSocket. Streaming endpoint format for external consumers. gRPC is more efficient for high-throughput indexers; WebSocket matches existing
subscribepatterns. Probably both.
- Phase 1 (1 month) — biggest win, smallest risk. Ships independently. Lifts every read in the system.
- Phase 3 (overlap Phase 1 by 2 weeks) — design the public API alongside the internal change so the "other project" can integrate against the API before Phase 5.
- Phase 2 (1 month) — memory-resident historical ledgers. Validates the memory budget assumption.
- Phase 5 (1 month) — rollout.
Total: ~3 months for Phases 1+2+3+5 (Phase 4 deferred unless Plan 1 is concurrent).
Smallest meaningful slice if the team can only afford 1 month: Phase 1 alone — every internal read becomes O(1); the external API is deferred. Even this slice delivers a 5–10× speedup on the apply-path read load and ships with zero protocol risk.
Code-complete and verified at the library level. Default-off. Not yet wired into the live node.
| Piece | Where | Tests |
|---|---|---|
FlatStateMap (flat keylet→SLE, shared_mutex, deep-copy snapshot()) |
FlatStateMap.h / .cpp | unit + concurrency + bench |
Write-through mirror (mirrorRaw*) wired into Ledger::raw{Insert,Replace,Erase} (P6.3) |
Ledger.cpp | mirror + integration |
Flat-only read routing in Ledger::read (P6.4) |
Ledger.cpp | LedgerReadFlatPathMatchesShaMapPath |
Close-time differential invariant validateFlatStateMapMatchesShaMap (P6.5) |
Ledger.cpp | InvariantFailsOn{Phantom,Missing}Entry (failure-direction proven) |
| Multi-ledger lifecycle: child inherits an INDEPENDENT snapshot of the parent's map | Ledger.cpp:255 | ChildInheritsIndependentSnapshot, ChildWritesDoNotCorruptParentSnapshot, PropagationChainsAcrossGenerations — red-proven (sharing instead of snapshotting fails them) |
When no map is attached (the default), every path is a no-op ⇒ byte-for-byte the prior behavior. Full libxrpl suite (8 targets) green.
The v2 design (eager full mirror, reads flat-only) supersedes the original P6.1/P6.2 LRU positive+negative caches — a full materialization needs no separate negative cache (a miss IS absence). Those checklist items are obsolete under v2.
- Live-app wiring (operator flag →
ApplicationstartupattachFlatStateMapTo→ close-invariant gate in BuildLedger.cpp:46). Touches the consensus-critical close path; compile-only-verifiable here (full-node build), not runtime-verifiable without a running node. - P6.7 mainnet replay gate — the non-negotiable safety gate (byte-identical state roots over months of history). Requires the replay harness + infra; cannot be run here. Flat-only reads must NOT be enabled in production before this passes (a flat-map bug = wrong state served = consensus divergence).
- P6.8/P6.9 devnet→testnet rollout, default-on. Ops + time.
- Persistent/HAMT map — the deep-copy
snapshot()is O(N)/ledger; correct but too costly to enable as-is. A structurally-sharing map is the prerequisite for the feature to be a net win, and is a scoped follow-on data-structure task.
Bottom line: all of Plan 6 that is code and can be proven correct in-repo is done and green. What remains is integration into a running node plus the multi-month replay/rollout validation — gated on infrastructure and time, not on more code.