Load Test Analysis: Two Distinct Failure Modes

gistfile1.txt

---
  Load Test Analysis: Two Distinct Failure Modes

  ★ Insight ─────────────────────────────────────
  The Datadog CSV export only includes the Content (message) column, but the critical diagnostic data — the revert selector and
  classification — is logged as tracing structured fields (classification = ?classification), which Datadog stores as attributes, not in
   the message string. This is why every "non-recoverable" line looks identical. We're diagnosing blind.
  ─────────────────────────────────────────────────

  Failure Mode 1: Two-Phase on Base Sepolia — 100% On-Chain Submission Failure

  Numbers (from gateway-1):
  - 20 tasks created
  - 40 BLS aggregations completed (all succeed)
  - 16 batch submissions → 20 items classified as "non-recoverable"
  - 0 successes. 0 receipt timeouts.

  Timeline for a single task:
  17:21:03.730  BLS aggregation initialized at source block 10617581
  17:21:06.357  Prepare phase completed (1 operator responded, quorum reached)
  17:21:07.904  Commit phase completed (1 operator signed)
  17:21:08.297  BLS aggregation completed in 4567ms
  17:21:09.363  Flushing batch
  17:21:10.278  Submitting batch
  17:21:10.358  BatchPartialFailure → non-recoverable   ← 80ms, INSTANT
  17:21:10.359  "combined create+respond failed"

  The entire two-phase consensus works perfectly (Prepare → Consensus → Commit → BLS aggregation). The failure is exclusively in the
  on-chain submission. The revert happens within ~80ms of the eth_sendTransaction — this is a gas estimation revert, not a mined tx that
   failed. The contract immediately rejects every item.

  Root cause hypothesis: The error is inside _createAndRespond() which calls taskManager.createNewTask(task) then
  taskManager.respondToTask(task, response, signatureData). Since the BatchTaskManager.onlyAuthorized check passes (the method body
  executes, items are processed individually), the revert is from the underlying TaskManager:

  ┌──────────────────────────────┬─────────────┬────────────────────────────────────────────────────────────────────────────────────┐
  │          Candidate           │ Probability │                                        Why                                         │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ Base Sepolia uses BN254CertificateVerifier. The logs show "Building merkle tree    │
  │ BLS certificate verification │ High        │ from 2 operators for 1 non-signers" — if the cert verifier's stored merkle root    │
  │  failure in respondToTask    │             │ doesn't match what the gateway builds, every cert fails. This is systematic (100%  │
  │                              │             │ failure).                                                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ If BatchTaskManager address isn't registered via addTaskGenerator() on Base        │
  │ OnlyTaskGenerator on the     │ Medium      │ Sepolia's OperatorRegistry. The deployment script handles this                     │
  │ underlying TaskManager       │             │ (NewtonCrossChainDeployer.s.sol:260), but a partial deployment or redeployment     │
  │                              │             │ without re-registration would break this.                                          │
  ├──────────────────────────────┼─────────────┼────────────────────────────────────────────────────────────────────────────────────┤
  │                              │             │ taskCreatedBlock uses the destination chain's block (sync.rs:208-213, Base Sepolia │
  │ TaskCreatedBlockTooOld       │ Lower       │  gets offset 5), so it should be valid. But if the chain service resolves the      │
  │                              │             │ wrong chain's block, this would fire.                                              │
  └──────────────────────────────┴─────────────┴────────────────────────────────────────────────────────────────────────────────────┘

  We literally cannot confirm without the revert selector. This is the #1 reason the observability fix is critical.

  Failure Mode 2: Centralized on Sepolia — Nonce Contention Across 10 Gateways

  Numbers (from gateway-1 of 10):
  - 20 tasks created
  - 4 succeeded (2 direct + 2 idempotent "already on-chain")
  - 11 receipt timeouts (all at exactly 30s)
  - 8 items classified as "poison" by simulation
  - 80% failure rate on this gateway

  Timeline:
  16:47:02  Batch 1 submitted
  16:47:31  Batch 1: all items succeeded         ← 29s to mine (Sepolia congestion)
  16:47:32  Batch 2 submitted
  16:47:53  Batch 2: "already on-chain" ×2       ← Other gateways submitted these!
  16:48:02  Batch 3 submitted
  16:48:33  Batch 3: RECEIPT TIMED OUT (30s)     ← Nonce race lost, tx dropped
  16:48:33  → Retry submission...
  16:49:03  Retry: RECEIPT TIMED OUT (30s)
  16:49:04  → Retry submission...
  16:49:34  Retry: RECEIPT TIMED OUT → max retries → fail all

  From this point, every single batch from gateway-1 follows the same pattern: submit → 30s timeout → retry → 30s timeout → max retries
  → fail.

  Root cause: 10 gateway replicas all use the same ECDSA signer key. Each gateway's CachedNonceManager calls
  eth_getTransactionCount("pending") independently, gets the same nonce. Only one gateway's tx lands per nonce. The rest are dropped
  from the mempool. The "already on-chain" items at 16:47:53 confirm other gateways are submitting the same tasks and winning the nonce
  race.

  Secondary issue: After a 30s timeout, simulate_and_classify() runs. At this point, ~30s have elapsed since the original
  taskCreatedBlock was set. On Sepolia (12s block time, buffer window 10 blocks = 120s), this should still be within the buffer. But the
   simulation itself reverts for these items — meaning either (a) the items genuinely can't be submitted (some other contract error) or
  (b) the simulation encounters its own nonce/gas estimation issue from the congested RPC.

  ★ Insight ─────────────────────────────────────
  The "already on-chain" log entries are the smoking gun for nonce contention. They prove that another gateway instance already
  submitted the same task successfully — the RPC request hit gateway-1, but gateway-3 (or whichever) submitted the batch tx that
  actually mined. This is expected behavior when N gateways share one signing key: the task itself is fine, but each gateway
  independently builds and submits its own batch tx. Only one can win the nonce.
  ─────────────────────────────────────────────────

  Both Modes: Zombie BLS Aggregator Slots

  Both logs show aggregator slots expiring with "processed 0 signatures" after 360 seconds. These are tasks where the on-chain
  submission failed but the aggregator slot was never cleaned up via cancel_task(). Under load, this wastes memory proportional to
  failure_rate × 360s TTL.

  ---
  What Needs to Happen

  There are three layers of problems — observability, two-phase contract reverts, and multi-gateway nonce contention. The priorities
  are:

  P0 — Observability (unblocks diagnosis of the two-phase issue):
  The revert selector and classification MUST appear in the log message string, not just as structured tracing fields. Without this, we
  can't diagnose the Base Sepolia 100% failure rate.

  P0 — Two-phase Base Sepolia diagnosis:
  Once we can see the revert selector, we can confirm whether it's certificate verification (NEWT-845 pattern), OnlyTaskGenerator, or
  something else. Independently, the engineer can call cast call with the same batch calldata to get the revert reason directly.

  P1 — Multi-gateway nonce contention:
  With 10 gateways sharing one signing key, at most 1 gateway can land a tx per nonce. This is the fundamental bottleneck. The
  architectural options are: per-gateway signing keys, nonce coordination, or leader election for the submission path.

  P1 — Simulation "poison" misclassification after timeout:
  After a receipt timeout, simulate_and_classify() classifies items that are genuinely healthy (just need a fresh taskCreatedBlock) as
  "poison" because the simulation itself may revert with TaskCreatedBlockTooOld.

  Shall I start with the observability fix and the investigation tooling (a diagnostic command that can identify the exact revert reason
   on Base Sepolia), or do you want to discuss the multi-gateway architecture first?

⏺ ★ Insight ─────────────────────────────────────
All 5 tasks complete. Here's the change summary across 7 files (+63/-26):

1. Observability (batch_submitter.rs): Revert selector and classification now appear in log message strings, not just structured
   tracing fields. This unblocks diagnosis of the two-phase Base Sepolia 100% failure.
2. Poison misclassification bug (batch_submitter.rs): simulate_and_classify() now checks e.is_transient() before the catch-all.
   Transient RPC errors during simulation keep items as healthy for retry instead of permanently killing them.
3. Chain-adaptive receipt timeout (writer.rs + 6 callers): AvsWriter now has a chain_id field. batch_receipt_timeout() returns 60s
   for L1 (Ethereum/Sepolia) and 30s for L2s. This gives congested Sepolia ~5 blocks instead of ~2.5.
4. Zombie aggregator slot cleanup (sync.rs): cancel_task() now called on success and idempotent success paths — not just failure.
   Every resolved task frees its aggregator slot immediately instead of lingering 360s.
5. Lessons learned (lessons.md): Four new entries documenting simulation misclassification, Datadog CSV observability gap,
   chain-adaptive timeout, and aggregator cleanup.
   ─────────────────────────────────────────────────

What this does NOT fix — the two-phase Base Sepolia 100% failure. That's a contract-level revert on the destination chain whose root
cause requires the observability fix to be deployed first (or an independent cast call reproduction). The most likely candidates
remain certificate verification failure or BatchTaskManager not registered as a task generator on Base Sepolia's OperatorRegistry.