Load Test Analysis: Two Distinct Failure Modes

gistfile1.txt

---
  Load Test Analysis: Two Distinct Failure Modes

  ★ Insight ─────────────────────────────────────
  The Datadog CSV export only includes the Content (message) column, but the critical diagnostic data — the revert selector and
  classification — is logged as tracing structured fields (classification = ?classification), which Datadog stores as attributes, not in
   the message string. This is why every "non-recoverable" line looks identical. We're diagnosing blind.
  ─────────────────────────────────────────────────

  Failure Mode 1: Two-Phase on Base Sepolia — 100% On-Chain Submission Failure

  Numbers (from gateway-1):
  - 20 tasks created
  - 40 BLS aggregations completed (all succeed)
  - 16 batch submissions → 20 items classified as "non-recoverable"
  - 0 successes. 0 receipt timeouts.

  Timeline for a single task:
  17:21:03.730  BLS aggregation initialized at source block 10617581
  17:21:06.357  Prepare phase completed (1 operator responded, quorum reached)
  17:21:07.904  Commit phase completed (1 operator signed)
  17:21:08.297  BLS aggregation completed in 4567ms
  17:21:09.363  Flushing batch
  17:21:10.278  Submitting batch
  17:21:10.358  BatchPartialFailure → non-recoverable   ← 80ms, INSTANT
  17:21:10.359  "combined create+respond failed"

  The entire two-phase consensus works perfectly (Prepare → Consensus → Commit → BLS aggregation). The failure is exclusively in the
  on-chain submission. The revert happens within ~80ms of the eth_sendTransaction — this is a gas estimation revert, not a mined tx that
   failed. The contract immediately rejects every item.

  Root cause hypothesis: The error is inside _createAndRespond() which calls taskManager.createNewTask(task) then
  taskManager.respondToTask(task, response, signatureData). Since the BatchTaskManager.onlyAuthorized check passes (the method body
  executes, items are processed individually), the revert is from the underlying TaskManager:

  ┌──────────────────────────────┬─────────────┬────────────────────────────────────────────────────────────────────────────────────┐
  │          Candidate           │ Probability │                                        Why                                         │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ Base Sepolia uses BN254CertificateVerifier. The logs show "Building merkle tree    │
  │ BLS certificate verification │ High        │ from 2 operators for 1 non-signers" — if the cert verifier's stored merkle root    │
  │  failure in respondToTask    │             │ doesn't match what the gateway builds, every cert fails. This is systematic (100%  │
  │                              │             │ failure).                                                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ If BatchTaskManager address isn't registered via addTaskGenerator() on Base        │
  │ OnlyTaskGenerator on the     │ Medium      │ Sepolia's OperatorRegistry. The deployment script handles this                     │
  │ underlying TaskManager       │             │ (NewtonCrossChainDeployer.s.sol:260), but a partial deployment or redeployment     │
  │                              │             │ without re-registration would break this.                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ taskCreatedBlock uses the destination chain's block (sync.rs:208-213, Base Sepolia │
  │ TaskCreatedBlockTooOld       │ Lower       │  gets offset 5), so it should be valid. But if the chain service resolves the      │
  │                              │             │ wrong chain's block, this would fire.                                              │
  └──────────────────────────────┴─────────────┴────────────────────────────────────────────────────────────────────────────────────┘

  We literally cannot confirm without the revert selector. This is the #1 reason the observability fix is critical.

  Failure Mode 2: Centralized on Sepolia — Nonce Contention Across 10 Gateways

  Numbers (from gateway-1 of 10):
  - 20 tasks created
  - 4 succeeded (2 direct + 2 idempotent "already on-chain")
  - 11 receipt timeouts (all at exactly 30s)
  - 8 items classified as "poison" by simulation
  - 80% failure rate on this gateway

  Timeline:
  16:47:02  Batch 1 submitted
  16:47:31  Batch 1: all items succeeded         ← 29s to mine (Sepolia congestion)
  16:47:32  Batch 2 submitted
  16:47:53  Batch 2: "already on-chain" ×2       ← Other gateways submitted these!
  16:48:02  Batch 3 submitted
  16:48:33  Batch 3: RECEIPT TIMED OUT (30s)     ← Nonce race lost, tx dropped
  16:48:33  → Retry submission...
  16:49:03  Retry: RECEIPT TIMED OUT (30s)
  16:49:04  → Retry submission...
  16:49:34  Retry: RECEIPT TIMED OUT → max retries → fail all

  From this point, every single batch from gateway-1 follows the same pattern: submit → 30s timeout → retry → 30s timeout → max retries
  → fail.

  Root cause: 10 gateway replicas all use the same ECDSA signer key. Each gateway's CachedNonceManager calls
  eth_getTransactionCount("pending") independently, gets the same nonce. Only one gateway's tx lands per nonce. The rest are dropped
  from the mempool. The "already on-chain" items at 16:47:53 confirm other gateways are submitting the same tasks and winning the nonce
  race.

  Secondary issue: After a 30s timeout, simulate_and_classify() runs. At this point, ~30s have elapsed since the original
  taskCreatedBlock was set. On Sepolia (12s block time, buffer window 10 blocks = 120s), this should still be within the buffer. But the
   simulation itself reverts for these items — meaning either (a) the items genuinely can't be submitted (some other contract error) or
  (b) the simulation encounters its own nonce/gas estimation issue from the congested RPC.

  ★ Insight ─────────────────────────────────────
  The "already on-chain" log entries are the smoking gun for nonce contention. They prove that another gateway instance already
  submitted the same task successfully — the RPC request hit gateway-1, but gateway-3 (or whichever) submitted the batch tx that
  actually mined. This is expected behavior when N gateways share one signing key: the task itself is fine, but each gateway
  independently builds and submits its own batch tx. Only one can win the nonce.
  ─────────────────────────────────────────────────

  Both Modes: Zombie BLS Aggregator Slots

  Both logs show aggregator slots expiring with "processed 0 signatures" after 360 seconds. These are tasks where the on-chain
  submission failed but the aggregator slot was never cleaned up via cancel_task(). Under load, this wastes memory proportional to
  failure_rate × 360s TTL.

  ---
  What Needs to Happen

  There are three layers of problems — observability, two-phase contract reverts, and multi-gateway nonce contention. The priorities
  are:

  P0 — Observability (unblocks diagnosis of the two-phase issue):
  The revert selector and classification MUST appear in the log message string, not just as structured tracing fields. Without this, we
  can't diagnose the Base Sepolia 100% failure rate.

  P0 — Two-phase Base Sepolia diagnosis:
  Once we can see the revert selector, we can confirm whether it's certificate verification (NEWT-845 pattern), OnlyTaskGenerator, or
  something else. Independently, the engineer can call cast call with the same batch calldata to get the revert reason directly.

  P1 — Multi-gateway nonce contention:
  With 10 gateways sharing one signing key, at most 1 gateway can land a tx per nonce. This is the fundamental bottleneck. The
  architectural options are: per-gateway signing keys, nonce coordination, or leader election for the submission path.

  P1 — Simulation "poison" misclassification after timeout:
  After a receipt timeout, simulate_and_classify() classifies items that are genuinely healthy (just need a fresh taskCreatedBlock) as
  "poison" because the simulation itself may revert with TaskCreatedBlockTooOld.

  Shall I start with the observability fix and the investigation tooling (a diagnostic command that can identify the exact revert reason
   on Base Sepolia), or do you want to discuss the multi-gateway architecture first?

Worked for 6m 59s

❯ Root cause: 10 gateway replicas all use the same ECDSA signer key. Each gateway's CachedNonceManager calls
eth_getTransactionCount("pending") independently, gets the same nonce. Only one gateway's tx lands per nonce. The rest are dropped
from the mempool. The "already on-chain" items at 16:47:53 confirm other gateways are submitting the same tasks and winning the
nonce
race. --> incorrect. Analyze how we deploy our gateway in prod @~/projects/newton-prover-avs-deploy/ . We have 10 different task
generator signer keys each being assigned to each of the 10 gateway
⎿  Listed directory ../newton-prover-avs-deploy/

⏺ I stand corrected — let me look at the actual deployment architecture to understand the 10-gateway setup correctly.

⏺ Explore(Analyze gateway deployment architecture)
⎿  Done (22 tool uses · 105.9k tokens · 58s)
(ctrl+o to expand)

⏺ Thank you for the correction — that fundamentally changes the analysis. Each gateway has its own signer key, so the nonce contention
hypothesis is wrong. Let me re-analyze.

★ Insight ─────────────────────────────────────
The deployment uses AWS ECS Fargate with 10 fixed replicas. Each gateway gets TASK_GENERATOR_KEY_{1-10} from Secrets Manager,
injected as GATEWAY__TASK_GENERATOR__SIGNER__PRIVATE_KEY. All 10 gateways submit to the same BatchTaskManager contract. ALB
round-robins RPC requests with no sticky sessions.
─────────────────────────────────────────────────

Revised Analysis: Centralized (Sepolia) — NOT Nonce Contention

With per-gateway signing keys, each gateway has independent nonce management. No cross-gateway contention. Let me re-examine the
evidence:

The "already on-chain" entries at 16:47:53 — these are NOT from another gateway's submission. They're from this gateway's previous
batch that timed out but actually mined. The tx was sent, the 30s receipt timer fired before the receipt arrived, and when the
simulation ran, it found the items on-chain.

Revised timeline for gateway-1:
16:47:02 Batch 1: submitted
16:47:31 Batch 1: mined, success ← 29s to mine (Sepolia congestion)
16:47:32 Batch 2: submitted
16:47:53 Batch 2: partial — 2 items "already on-chain" ← prior tx DID land
16:48:02 Batch 3: submitted
16:48:33 Batch 3: RECEIPT TIMEOUT (30s) ← tx stuck in mempool
16:48:33 → Retry...
16:49:03 Retry: RECEIPT TIMEOUT
16:49:34 → max retries, fail

Root cause: Sepolia congestion + receipt timeout too short + poison misclassification

1. Sepolia congestion: The first batch took 29s to mine. Normal Sepolia is 12-24s. With 10 gateways all submitting batch txs
   concurrently (large gas per batch: ~400-800K per item), blocks fill up fast.
2. 30s receipt timeout is too tight for congested Sepolia: At 12s/block, 30s is ~2.5 blocks. Under congestion, txs may need 3-5
   blocks to mine. The PR #508 reduced this from 60s to 30s to cut worst-case blocking, but 30s is insufficient for congested L1.
3. Critical bug — simulation misclassifies transport errors as "poison": After a timeout, simulate_and_classify() runs per-item
   eth_call. If the Sepolia RPC is also congested (rate-limited, slow), the simulation itself may fail with a transport/RPC error.
   Looking at the code:

Read 1 file (ctrl+o to expand)

⏺ There it is. Line 693 — the catch-all Err(e) in simulate_and_classify():

Err(e) => {
debug!( // ← debug!, not warn!
task_id = %item.task_id,
error = %e,
"combined item identified as poison via simulation"
);
// ... classified as poison, failed to caller
poison_count += 1;
}

This catch-all treats every unrecognized error — including transient RPC errors, transport timeouts, and connection failures — as
"poison". If Sepolia's RPC is congested and the eth_call simulation fails with a transport error, the item gets permanently killed
even though it's perfectly healthy.

Revised Root Cause Summary

Centralized (Sepolia) — Three compounding issues:

┌─────────────────────────────────────────────────────────────┬───────────────────────────────────┬──────────────────────────────┐
│ Issue │ Impact │ Severity │
├─────────────────────────────────────────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ Sepolia congestion (29s to mine first batch) │ Txs stuck in mempool → receipt │ External, but expected │
│ │ timeout │ │
├─────────────────────────────────────────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ 30s receipt timeout too short for L1 congestion │ Healthy txs that would mine in │ Config — needs │
│ │ 36-48s are killed │ chain-adaptive timeout │
├─────────────────────────────────────────────────────────────┼───────────────────────────────────┼──────────────────────────────┤
│ simulate_and_classify() catch-all classifies transient │ Healthy items permanently failed │ Bug — critical │
│ errors as poison (line 693) │ on RPC hiccups │ │
└─────────────────────────────────────────────────────────────┴───────────────────────────────────┴──────────────────────────────┘

Two-Phase (Base Sepolia) — 100% systematic contract revert:

┌──────────────────────────────────────────────────────────────────┬─────────────────────────────────┬───────────────────────────┐
│ Issue │ Impact │ Severity │
├──────────────────────────────────────────────────────────────────┼─────────────────────────────────┼───────────────────────────┤
│ Unknown contract revert in respondToTask on destination chain │ Every task fails on-chain │ Critical — unknown root │
│ │ submission │ cause │
├──────────────────────────────────────────────────────────────────┼─────────────────────────────────┼───────────────────────────┤
│ Revert selector invisible in logs (structured field stripped by │ Cannot diagnose the above │ P0 observability gap │
│ Datadog CSV) │ │ │
└──────────────────────────────────────────────────────────────────┴─────────────────────────────────┴───────────────────────────┘

Both modes — Zombie BLS aggregator slots:

┌───────────────────────────────────────────┬─────────────────────────────────────────────────┬──────────┐
│ Issue │ Impact │ Severity │
├───────────────────────────────────────────┼─────────────────────────────────────────────────┼──────────┤
│ 360s aggregator slot TTL for failed tasks │ Memory leak proportional to failure rate × 6min │ Medium │
└───────────────────────────────────────────┴─────────────────────────────────────────────────┴──────────┘

---

Proposed Fixes

Here's what I think needs to happen, in priority order. Want to discuss before I start?

P0 — Observability: Include revert details in log message string
Embed classification, error, task_id, and revert selector directly into the log message so they're visible in any export format. This
unblocks diagnosis of the two-phase Base Sepolia failure.

P0 — Bug fix: simulate_and_classify() must distinguish transient errors from poison
The catch-all at line 693 should check e.is_transient() — transient errors mean "simulation inconclusive, treat as healthy for
retry", not "item is poison". Only non-transient contract reverts should classify as poison.

P1 — Chain-adaptive receipt timeout
30s is fine for L2s (Base Sepolia: 2s block time = 15 blocks). For L1 (Sepolia: 12s block time), under congestion 30s gives only ~2.5
blocks. Make the timeout chain-specific: 60s for L1, 30s for L2s.

P1 — Diagnose Base Sepolia two-phase revert
Independently (can do right now via cast call): reproduce the batchCreateAndRespondToTasks call against Base Sepolia to get the exact
revert data. Or, with the observability fix deployed, the next load test will reveal it.

P2 — Reduce aggregator slot TTL for batch-submitted tasks
After a batch item is resolved (success or failure), cancel_task() should be called immediately. The 360s TTL is a safety net, not
the primary cleanup mechanism.

---

Load Test Results — Post-PR #508 (2026-04-08)

Ran 200 tasks on prod (10 gateways) in two modes after deploying the batch submission hardening from #508. Results are better than
the previous 60% success rate on stagef, but not production-ready yet.

Centralized mode (Sepolia): 103/200 succeeded, 97/200 timed out. Base Sepolia was much better — 199/200 succeeded, 1 timed out.

Two-phase mode (Base Sepolia): 0/200 succeeded. Every task completes BLS aggregation correctly but fails at on-chain submission with
a non-recoverable contract revert. 100% failure rate.

---

Centralized (Sepolia) — Sepolia congestion + simulation bug

Sepolia was congested during the test — the first batch took 29s to mine (normally 12-24s). With 10 gateways all submitting large
batch txs concurrently, subsequent batches started exceeding the 30s receipt timeout. Once a timeout fires, the batch submitter runs
an eth_call simulation to classify which items landed on-chain and which are retryable.

The simulation has a bug: the catch-all error handler in simulate_and_classify() treats ANY unrecognized error — including transient
RPC errors from the same congested Sepolia — as "poison" (permanently failed). So when the simulation itself times out or gets
rate-limited, healthy items get killed. This is why success drops off a cliff once congestion starts. Base Sepolia didn't have this
problem because 2s block time means 30s covers 15 blocks — plenty of room.

Fixes needed:

1. Simulation must check is_transient() before classifying as poison — transient errors should keep items in the retry pool, not kill
   them
2. Receipt timeout should be chain-adaptive: 60s for L1 (Sepolia), 30s for L2s (Base)

Two-phase (Base Sepolia) — 100% on-chain submission failure

BLS aggregation works perfectly end-to-end. Every task completes Prepare, Consensus, and Commit phases, produces a valid BLS
aggregation in ~5s. The failure is exclusively in the batchCreateAndRespondToTasks call on Base Sepolia — the contract instantly
reverts (within 80ms) with BatchPartialFailure where every item is classified as non-recoverable.

We don't know the exact revert reason yet. The revert selector and classification are logged as tracing structured fields
(classification = ?classification), which Datadog stores as attributes but strips from the CSV message export. Every
"non-recoverable" log line looks identical — no revert details visible.

The most likely candidates are: certificate verification failure on the destination chain's BN254CertificateVerifier (stale operator
table / merkle proof mismatch), or OnlyTaskGenerator if the BatchTaskManager isn't registered on Base Sepolia's OperatorRegistry.
Both would produce a 100% systematic revert.

Next steps:

1. Fix the log format — embed the revert selector and classification in the message string, not just as structured fields. This is
   blocking diagnosis.
2. Independently, we can reproduce the revert via cast call with the same batch calldata against Base Sepolia to get the exact error
   without waiting for a redeploy.

Zombie aggregator slots

Both modes show BLS aggregator slots lingering for 360s (6 minutes) with "processed 0 signatures" after tasks fail at submission. The
cancel_task() cleanup isn't being called in all error paths. Not blocking but wastes memory under sustained load.

---

Want me to adjust the tone, add/remove detail, or restructure anything?

⏺ ★ Insight ─────────────────────────────────────
All 5 tasks complete. Here's the change summary across 7 files (+63/-26):

1. Observability (batch_submitter.rs): Revert selector and classification now appear in log message strings, not just structured
   tracing fields. This unblocks diagnosis of the two-phase Base Sepolia 100% failure.
2. Poison misclassification bug (batch_submitter.rs): simulate_and_classify() now checks e.is_transient() before the catch-all.
   Transient RPC errors during simulation keep items as healthy for retry instead of permanently killing them.
3. Chain-adaptive receipt timeout (writer.rs + 6 callers): AvsWriter now has a chain_id field. batch_receipt_timeout() returns 60s
   for L1 (Ethereum/Sepolia) and 30s for L2s. This gives congested Sepolia ~5 blocks instead of ~2.5.
4. Zombie aggregator slot cleanup (sync.rs): cancel_task() now called on success and idempotent success paths — not just failure.
   Every resolved task frees its aggregator slot immediately instead of lingering 360s.
5. Lessons learned (lessons.md): Four new entries documenting simulation misclassification, Datadog CSV observability gap,
   chain-adaptive timeout, and aggregator cleanup.
   ─────────────────────────────────────────────────

What this does NOT fix — the two-phase Base Sepolia 100% failure. That's a contract-level revert on the destination chain whose root
cause requires the observability fix to be deployed first (or an independent cast call reproduction). The most likely candidates
remain certificate verification failure or BatchTaskManager not registered as a task generator on Base Sepolia's OperatorRegistry.