The L2Book path used to prepare every supported aggregation variant for every flush, even when no client had subscribed to most of those variants. This made each L2 flush scale with the full variant set instead of the active subscriptions.
This change added subscription-aware tracking for active L2 snapshot parameters. The listener now derives the requested parameters from active L2 subscription keys and only computes those shapes.
let requested_params: HashSet<L2SnapshotParams> =
active_keys.iter().map(|key| key.params).collect();This reduces wasted L2 snapshot work when clients only subscribe to one or a few nSigFigs / mantissa combinations.
The dirty-coin set could contain coins that had changed in the order book but had no active L2Book subscribers. Those changes still woke the L2 flush path and could trigger snapshot work that no client would consume.
This change made the L2 subscription registry track active coins as well as active parameter shapes. Before computing L2 snapshots, the listener filters pending dirty coins down to subscribed coins and exits early when none match.
This keeps unrelated market activity from delaying the subscribed L2Book feed.
Previously, an L2 flush could carry entries that did not actually change for a subscribed shape. The websocket layer still had per-client dedupe, but unchanged entries still cost memory, map work, and dispatch overhead before being dropped.
This change made the prepared L2 update map contain only entries whose snapshot payload changed. When a coin or subscription shape is unchanged, it is omitted from that flush.
The result is smaller internal broadcasts and less work per websocket connection.
Universe broadcasts and L2Book flushes were coupled too closely. That meant work related to listing or market-universe changes could run as part of the same path as the latency-sensitive L2 snapshot cycle.
This change split universe updates into their own Universe internal message and only recomputes the universe when the coin count or changed coins suggest it might have changed.
That keeps the L2 flush focused on L2 data and avoids adding universe maintenance cost to the 50ms L2 cadence.
The websocket dispatch path used to do more work per connection to turn snapshots into outbound L2Book messages. With multiple clients subscribed to the same L2 shape, that repeated the same payload preparation.
This change introduced prepared L2Book payloads that are built once per flush and shared through Arc.
PreparedL2Book {
version,
payload,
}Each websocket connection can now reuse the prepared payload instead of rebuilding it independently. This moves shared work out of the per-client path.
L2 dedupe was pushed closer to the websocket side, which meant the system could still prepare entries that every client would later recognize as unchanged.
This change centralized L2 dedupe in the listener using per-subscription-key payload hashes and monotonically increasing prepared versions. The listener only emits a prepared update when the exported levels hash changes.
The websocket layer still tracks the last version sent per connection, but the expensive global decision is now made once.
Computing multiple L2 aggregation variants used to walk the same side of the book once per requested variant. For example, default L2, nSigFigs=5, and nSigFigs=4 each repeated similar level traversal work.
This change added OrderBook::to_l2_snapshots, which builds all requested variants from a shared pass over the bid and ask totals.
let bids = map_to_l2_levels_many(&self.bid_totals, Side::Bid, n_levels, params);
let asks = map_to_l2_levels_many(&self.ask_totals, Side::Ask, n_levels, params);This makes the cost closer to "walk the book once and fill the requested outputs" instead of "walk the book once per output".
The L2 path was still folding order lists to compute total size and order count at each price level. BBO was much faster because it already had a direct best-level path, while L2 had to aggregate from individual orders.
This change added cached per-price LevelTotal maps for bids and asks. Order add, cancel, modify, and match operations now maintain those totals incrementally.
L2 snapshot generation can read the price-level totals directly instead of folding every linked list during each flush. This narrows the performance gap between BBO and L2Book.
Even with a 50ms L2 throttle, the scheduled flush could be delayed behind a busy stream of file events. That allowed L2Book intervals to drift above the intended throttle even when the snapshot work itself was ready to run.
This change moved L2 flushing onto a scheduler branch that is selected before file-event processing.
tokio::select! {
biased;
_ = l2_flush_ticker.tick() => {
// flush L2 first
}
Some(event) = tokio_rx.recv() => {
// process file events
}
}This reduces event-loop starvation and makes the 50ms throttle behave more like a real cadence under load.
Preparing L2 payloads includes truncation, export, hashing, and payload construction. Doing all of that while holding the listener lock blocks incoming order-book events and can increase latency for both L2Book and other streams.
This change split the flush into two steps:
- Take a lightweight
L2FlushJobwhile holding the listener lock. - Prepare payloads outside the lock, then reacquire the lock only to publish the prepared result.
This keeps the listener lock held for less time and prevents L2 payload preparation from blocking the event ingestion path.
The L2 flush tick ran often, so even small allocations on every tick added overhead. The path was building active parameter and coin sets before checking whether there were any pending L2 changes.
This change added an early return before subscription-set allocation when pending_l2_changed_coins is empty. It also derives active params and coins from the single active-key set instead of calling multiple registry methods that allocate separate sets.
That makes idle or no-op ticks cheaper and keeps the hot scheduler path lean.
The aggregate L2 broadcast latency metric showed that L2 flushes were slow, but it did not show which part of the flush was responsible.
This change added l2_flush_phase_latency_seconds with snapshot, prepare, and publish labels.
l2_flush_phase_latency_seconds{phase="snapshot"}
l2_flush_phase_latency_seconds{phase="prepare"}
l2_flush_phase_latency_seconds{phase="publish"}
This makes the next performance investigation more direct: if L2Book still misses the 50ms target, the metrics should show whether time is going into snapshot computation, payload preparation, or publish overhead.