Load Test Analysis: Two Distinct Failure Modes

gistfile1.txt

---
  Load Test Analysis: Two Distinct Failure Modes

  ★ Insight ─────────────────────────────────────
  The Datadog CSV export only includes the Content (message) column, but the critical diagnostic data — the revert selector and
  classification — is logged as tracing structured fields (classification = ?classification), which Datadog stores as attributes, not in
   the message string. This is why every "non-recoverable" line looks identical. We're diagnosing blind.
  ─────────────────────────────────────────────────

  Failure Mode 1: Two-Phase on Base Sepolia — 100% On-Chain Submission Failure

  Numbers (from gateway-1):
  - 20 tasks created
  - 40 BLS aggregations completed (all succeed)
  - 16 batch submissions → 20 items classified as "non-recoverable"
  - 0 successes. 0 receipt timeouts.

  Timeline for a single task:
  17:21:03.730  BLS aggregation initialized at source block 10617581
  17:21:06.357  Prepare phase completed (1 operator responded, quorum reached)
  17:21:07.904  Commit phase completed (1 operator signed)
  17:21:08.297  BLS aggregation completed in 4567ms
  17:21:09.363  Flushing batch
  17:21:10.278  Submitting batch
  17:21:10.358  BatchPartialFailure → non-recoverable   ← 80ms, INSTANT
  17:21:10.359  "combined create+respond failed"

  The entire two-phase consensus works perfectly (Prepare → Consensus → Commit → BLS aggregation). The failure is exclusively in the
  on-chain submission. The revert happens within ~80ms of the eth_sendTransaction — this is a gas estimation revert, not a mined tx that
   failed. The contract immediately rejects every item.

  Root cause hypothesis: The error is inside _createAndRespond() which calls taskManager.createNewTask(task) then
  taskManager.respondToTask(task, response, signatureData). Since the BatchTaskManager.onlyAuthorized check passes (the method body
  executes, items are processed individually), the revert is from the underlying TaskManager:

  ┌──────────────────────────────┬─────────────┬────────────────────────────────────────────────────────────────────────────────────┐
  │          Candidate           │ Probability │                                        Why                                         │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ Base Sepolia uses BN254CertificateVerifier. The logs show "Building merkle tree    │
  │ BLS certificate verification │ High        │ from 2 operators for 1 non-signers" — if the cert verifier's stored merkle root    │
  │  failure in respondToTask    │             │ doesn't match what the gateway builds, every cert fails. This is systematic (100%  │
  │                              │             │ failure).                                                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ If BatchTaskManager address isn't registered via addTaskGenerator() on Base        │
  │ OnlyTaskGenerator on the     │ Medium      │ Sepolia's OperatorRegistry. The deployment script handles this                     │
  │ underlying TaskManager       │             │ (NewtonCrossChainDeployer.s.sol:260), but a partial deployment or redeployment     │
  │                              │             │ without re-registration would break this.                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ taskCreatedBlock uses the destination chain's block (sync.rs:208-213, Base Sepolia │
  │ TaskCreatedBlockTooOld       │ Lower       │  gets offset 5), so it should be valid. But if the chain service resolves the      │
  │                              │             │ wrong chain's block, this would fire.                                              │
  └──────────────────────────────┴─────────────┴────────────────────────────────────────────────────────────────────────────────────┘

  We literally cannot confirm without the revert selector. This is the #1 reason the observability fix is critical.

  Failure Mode 2: Centralized on Sepolia — Nonce Contention Across 10 Gateways

  Numbers (from gateway-1 of 10):
  - 20 tasks created
  - 4 succeeded (2 direct + 2 idempotent "already on-chain")
  - 11 receipt timeouts (all at exactly 30s)
  - 8 items classified as "poison" by simulation
  - 80% failure rate on this gateway

  Timeline:
  16:47:02  Batch 1 submitted
  16:47:31  Batch 1: all items succeeded         ← 29s to mine (Sepolia congestion)
  16:47:32  Batch 2 submitted
  16:47:53  Batch 2: "already on-chain" ×2       ← Other gateways submitted these!
  16:48:02  Batch 3 submitted
  16:48:33  Batch 3: RECEIPT TIMED OUT (30s)     ← Nonce race lost, tx dropped
  16:48:33  → Retry submission...
  16:49:03  Retry: RECEIPT TIMED OUT (30s)
  16:49:04  → Retry submission...
  16:49:34  Retry: RECEIPT TIMED OUT → max retries → fail all

  From this point, every single batch from gateway-1 follows the same pattern: submit → 30s timeout → retry → 30s timeout → max retries
  → fail.

  Root cause: 10 gateway replicas all use the same ECDSA signer key. Each gateway's CachedNonceManager calls
  eth_getTransactionCount("pending") independently, gets the same nonce. Only one gateway's tx lands per nonce. The rest are dropped
  from the mempool. The "already on-chain" items at 16:47:53 confirm other gateways are submitting the same tasks and winning the nonce
  race.

  Secondary issue: After a 30s timeout, simulate_and_classify() runs. At this point, ~30s have elapsed since the original
  taskCreatedBlock was set. On Sepolia (12s block time, buffer window 10 blocks = 120s), this should still be within the buffer. But the
   simulation itself reverts for these items — meaning either (a) the items genuinely can't be submitted (some other contract error) or
  (b) the simulation encounters its own nonce/gas estimation issue from the congested RPC.

  ★ Insight ─────────────────────────────────────
  The "already on-chain" log entries are the smoking gun for nonce contention. They prove that another gateway instance already
  submitted the same task successfully — the RPC request hit gateway-1, but gateway-3 (or whichever) submitted the batch tx that
  actually mined. This is expected behavior when N gateways share one signing key: the task itself is fine, but each gateway
  independently builds and submits its own batch tx. Only one can win the nonce.
  ─────────────────────────────────────────────────

  Both Modes: Zombie BLS Aggregator Slots

  Both logs show aggregator slots expiring with "processed 0 signatures" after 360 seconds. These are tasks where the on-chain
  submission failed but the aggregator slot was never cleaned up via cancel_task(). Under load, this wastes memory proportional to
  failure_rate × 360s TTL.

  ---
  What Needs to Happen

  There are three layers of problems — observability, two-phase contract reverts, and multi-gateway nonce contention. The priorities
  are:

  P0 — Observability (unblocks diagnosis of the two-phase issue):
  The revert selector and classification MUST appear in the log message string, not just as structured tracing fields. Without this, we
  can't diagnose the Base Sepolia 100% failure rate.

  P0 — Two-phase Base Sepolia diagnosis:
  Once we can see the revert selector, we can confirm whether it's certificate verification (NEWT-845 pattern), OnlyTaskGenerator, or
  something else. Independently, the engineer can call cast call with the same batch calldata to get the revert reason directly.

  P1 — Multi-gateway nonce contention:
  With 10 gateways sharing one signing key, at most 1 gateway can land a tx per nonce. This is the fundamental bottleneck. The
  architectural options are: per-gateway signing keys, nonce coordination, or leader election for the submission path.

  P1 — Simulation "poison" misclassification after timeout:
  After a receipt timeout, simulate_and_classify() classifies items that are genuinely healthy (just need a fresh taskCreatedBlock) as
  "poison" because the simulation itself may revert with TaskCreatedBlockTooOld.

  Shall I start with the observability fix and the investigation tooling (a diagnostic command that can identify the exact revert reason
   on Base Sepolia), or do you want to discuss the multi-gateway architecture first?

---

Load Test Results — Post-PR #508 (2026-04-08)

Ran 200 tasks on prod (10 gateways) in two modes after deploying the batch submission hardening from #508. Results are better than
the previous 60% success rate on stagef, but not production-ready yet.

Centralized mode (Sepolia): 103/200 succeeded, 97/200 timed out. Base Sepolia was much better — 199/200 succeeded, 1 timed out.

Two-phase mode (Base Sepolia): 0/200 succeeded. Every task completes BLS aggregation correctly but fails at on-chain submission with
a non-recoverable contract revert. 100% failure rate.

---

Centralized (Sepolia) — Sepolia congestion + simulation bug

Sepolia was congested during the test — the first batch took 29s to mine (normally 12-24s). With 10 gateways all submitting large
batch txs concurrently, subsequent batches started exceeding the 30s receipt timeout. Once a timeout fires, the batch submitter runs
an eth_call simulation to classify which items landed on-chain and which are retryable.

The simulation has a bug: the catch-all error handler in simulate_and_classify() treats ANY unrecognized error — including transient
RPC errors from the same congested Sepolia — as "poison" (permanently failed). So when the simulation itself times out or gets
rate-limited, healthy items get killed. This is why success drops off a cliff once congestion starts. Base Sepolia didn't have this
problem because 2s block time means 30s covers 15 blocks — plenty of room.

Fixes needed:

1. Simulation must check is_transient() before classifying as poison — transient errors should keep items in the retry pool, not kill
   them
2. Receipt timeout should be chain-adaptive: 60s for L1 (Sepolia), 30s for L2s (Base)

Two-phase (Base Sepolia) — 100% on-chain submission failure

BLS aggregation works perfectly end-to-end. Every task completes Prepare, Consensus, and Commit phases, produces a valid BLS
aggregation in ~5s. The failure is exclusively in the batchCreateAndRespondToTasks call on Base Sepolia — the contract instantly
reverts (within 80ms) with BatchPartialFailure where every item is classified as non-recoverable.

We don't know the exact revert reason yet. The revert selector and classification are logged as tracing structured fields
(classification = ?classification), which Datadog stores as attributes but strips from the CSV message export. Every
"non-recoverable" log line looks identical — no revert details visible.

The most likely candidates are: certificate verification failure on the destination chain's BN254CertificateVerifier (stale operator
table / merkle proof mismatch), or OnlyTaskGenerator if the BatchTaskManager isn't registered on Base Sepolia's OperatorRegistry.
Both would produce a 100% systematic revert.

Next steps:

1. Fix the log format — embed the revert selector and classification in the message string, not just as structured fields. This is
   blocking diagnosis.
2. Independently, we can reproduce the revert via cast call with the same batch calldata against Base Sepolia to get the exact error
   without waiting for a redeploy.

Zombie aggregator slots

Both modes show BLS aggregator slots lingering for 360s (6 minutes) with "processed 0 signatures" after tasks fail at submission. The
cancel_task() cleanup isn't being called in all error paths. Not blocking but wastes memory under sustained load.

---

Want me to adjust the tone, add/remove detail, or restructure anything?

⏺ ★ Insight ─────────────────────────────────────
All 5 tasks complete. Here's the change summary across 7 files (+63/-26):

1. Observability (batch_submitter.rs): Revert selector and classification now appear in log message strings, not just structured
   tracing fields. This unblocks diagnosis of the two-phase Base Sepolia 100% failure.
2. Poison misclassification bug (batch_submitter.rs): simulate_and_classify() now checks e.is_transient() before the catch-all.
   Transient RPC errors during simulation keep items as healthy for retry instead of permanently killing them.
3. Chain-adaptive receipt timeout (writer.rs + 6 callers): AvsWriter now has a chain_id field. batch_receipt_timeout() returns 60s
   for L1 (Ethereum/Sepolia) and 30s for L2s. This gives congested Sepolia ~5 blocks instead of ~2.5.
4. Zombie aggregator slot cleanup (sync.rs): cancel_task() now called on success and idempotent success paths — not just failure.
   Every resolved task frees its aggregator slot immediately instead of lingering 360s.
5. Lessons learned (lessons.md): Four new entries documenting simulation misclassification, Datadog CSV observability gap,
   chain-adaptive timeout, and aggregator cleanup.
   ─────────────────────────────────────────────────

What this does NOT fix — the two-phase Base Sepolia 100% failure. That's a contract-level revert on the destination chain whose root
cause requires the observability fix to be deployed first (or an independent cast call reproduction). The most likely candidates
remain certificate verification failure or BatchTaskManager not registered as a task generator on Base Sepolia's OperatorRegistry.