Skip to content

Instantly share code, notes, and snippets.

@koeppelmann

koeppelmann/deployment_report.md

Last active July 1, 2026 19:27
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save koeppelmann/6d41f3bf3c33d6df451173ff20573003 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save koeppelmann/6d41f3bf3c33d6df451173ff20573003 to your computer and use it in GitHub Desktop.
Download ZIP
EEZ two-composer deployment on 147.135.221.107 — issue report (2026-07-01)
Raw
deployment_report.md

Two-composer EEZ deployment on 147.135.221.107 — issue report

Chain: Chiado L1 (10200) + EEZ Rollup0 L2 (10201) • Registry 0xd3a59fc7…99317 • Composers C1 / C2 Date: 2026-07-01 • Updated: 2026-07-01 after live diagnosis with Edu

Correction to the original diagnosis of #1

The initial version of this report attributed the failing L1→L2 deposits to the two-composer bundle race described in eez-rollup0#20. That was wrong. After Edu identified the real cause on-chain and I re-tested, the actual behavior is described in the corrected #1 below. The two-composer race may still exist as a separate issue, but it is not what I was reproducing.

1. Composer admits under-gassed L1 txs without simulation (both branches)

Severity: blocker for anyone whose wallet picks a low default gas limit.

Root cause (per Edu's on-source analysis, confirmed by live retry): the composer accepts L1 txs into its pool without running an eth_estimateGas / simulation. If the user's tx specifies a gas limit below what the tx actually consumes on-chain, it enters the pool anyway. When the composer later builds a Flashbots bundle around it, the on-chain execution runs out of gas, the bundle fails at L1 inclusion, and the composer loops retries indefinitely. This is present in both Edu's and debjit's branches. Edu is opening an issue.

What that looks like from the outside:

  • User's L1 tx never mines on Chiado (nonce doesn't advance, balance doesn't debit).
  • L2 shows a repeating cycle of preconfirmed credit → reorg every ~5-10 s. Each cycle is the composer trying again.
  • getTransactionByHash on Chiado returns null for the user's hash; Chiado's mempool has already evicted / never held it.
  • If the user retries at higher gas via a new signed tx (replacement), the same cycle continues at the new hash.

Reproduction with correct gas: fresh L1 EOA 0x068340…fCF1, deposit 0.05 xDAI to L1 proxy 0x0DA637E9…D9F9 via C1 entry :18649, gas limit 500 000 @ 15 gwei →

  • L1 mined at block 21 890 872 in 8.3 s
  • Actual gasUsed = 112 106 — so anything under ~120k for a deposit-to-proxy transfer is going to hit this bug
  • L2 credit stable at 0.0503 ETH, no reorg cycle
  • Fresh nonce advanced 0 → 1

Reproduction with insufficient gas (same sender, prior attempts): gas limit 60 000 → tx accepted by composer, ~10 s preconf/reorg cycle indefinitely, tx never lands on L1, sender's nonce never advances. Repeated at three separate gas prices (10, 12, 15 gwei) and two senders (main and fresh EOAs). Same symptom every time.

Suggested fixes on the composer side:

  • Pre-admission simulation with the tx's own gas limit; reject at RPC time with a clear error (intrinsic gas too low / out of gas at pc=X).
  • Once admitted, if a bundle attempt fails with OOG, evict from the pool and error the user rather than retrying forever.
  • Both are cheap and would eliminate the "silent forever" failure entirely.

Suggested fix in the deployment page: raise the deposit UI's default gas limit above 200k (real cost is ~112k for a bare value transfer; the button in rabbit-remix-info.html currently uses whatever the injected wallet defaults to, which is usually 21k or estimateGas — and estimateGas against a receive() proxy often underestimates when combined with the L1 bridge state changes on the far side).

2. Speculative L2 credit → repeated shallow reorg on the depositing composer

Severity: high, but downstream of #1.

Every ~5-10 s cycle observed on C1's L2 was a direct consequence of the composer's retry-forever loop for the OOG'd bundle. Symptoms shown in the original report were real:

[  2.1s] C1=0.000500  ← preconfirmed
[  6.3s] C1=0.000000  ← reorged after bundle miss
[ 20.8s] C1=0.000500  ← retry
[ 31.3s] C1=0.000000
… every ~5-10s until manually cancelled

Fixing #1 makes this go away, as confirmed by the successful 500k-gas retry.

Even so, the pattern is worth surfacing separately: any wallet that reads balance at the moment of a preconfirmation and treats it as final will see the credit vanish. dApps talking to this L2 should either wait for the tx to appear in a batch that has been posted on L1 with a receipt, or add an is-this-preconf-only check.

3. C1 and C2 diverge on preconfirmed content at the same height

Severity: high, likely also downstream of #1 + the multi-bundle race.

While the deposit was stuck in the retry cycle:

  • C1 showed my 0.0005 ETH credit; C2 never did (same L2 head, same wall-clock second, different eth_getBalance).
  • Block hashes and state roots at the same height were equal — so the divergence lives in the composer-local preconfirmation layer above the canonical chain.
  • After the reorg wiped the credit both composers agreed again.

Edu noted the multi-bundle interaction is related and is filing a separate issue.

4. Any tx from the L2 system EOA collides with the composer's dispatcher

Severity: medium (operational footgun).

The L2 system address on this deployment is 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266 — a well-known Hardhat #0 key with ~1 000 000 ETH pre-funded on L2. The composer uses this same EOA to sign executeIncomingCrossChainCall on 0x4200…0007, as visible in tx 0x353f5e…b9124 (from = sys EOA, to = EEZL2, value = 0.0005e15).

The l2SystemAddress label in the deployment page is public. Any user who signs with the known key from a tool (e.g. a naive Rabby import) will consume system-EOA nonces out from under the composer's dispatcher. I did this once (0xcc290513…be97842, nonce 15 on C1) and observed an immediate L2 reorg spike that overlapped with the composer's executeIncomingCrossChainCall attempts at the same nonce. The chain recovered overnight, but the interference window was noticeable.

Suggested fixes: use a dedicated sequencer EOA (not Hardhat #0), or at minimum change the page label — right now the label reads as an invitation. Even better: reject external sends from the system EOA in the composer's mempool policy.

5. Blockscout retains reorged txs as "confirmed"

Severity: low, but misleading.

Tx 0x353f5e…b9124 is shown by Blockscout at http://147.135.221.107/tx/… with confirmations: 45089 in block 27140. Live RPC lookups on both composers return null for the same hash and receipt. Blockscout indexed the tx during a preconfirmation window and never noticed the reorg that dropped it. Any tool that reads Blockscout for finality will accept this ghost tx as final. Once #1 is fixed the source of the ghost txs goes away, but the indexer itself should still probably learn about reorgs.

6. L1 balance economics for deposits are punishing on Chiado's 10 gwei floor

Severity: low, but painful for testing.

  • createCrossChainProxy costs ~263k gas → 0.00263 xDAI at 10 gwei floor
  • Deposit-to-proxy value transfer: ~112k gas actual → 0.00112 xDAI at 10 gwei (was thought to be 21-30k; that estimate is wrong — see #1)
  • A user starting with 0.005 xDAI can afford one proxy creation and one small deposit, no room for retries.

Environmental facts collected while diagnosing (for cross-reference)

  • L1 batches landing at ~1.36/block by both posters (C1 poster nonce jumped 38 857 → 45 989 overnight; C2 28 309 → 35 225).
  • L2 average block time ~1 s (per Blockscout stats).
  • 0x4200…0007 (EEZL2 / CCM): 13 406 bytes, verified.
  • 0x4200…0008 (L2 bridge receiver): 64 bytes, tiny fallback contract, currently holds 0.001 ETH.
  • Registry on Chiado: 40 090 bytes of code, alive.
  • PostBatch tracker on :8080 is configured with fromBlock=21872008, explorer=https://gnosis-chiado.blockscout.com — it tracks real Chiado, not the local L2.
  • Successful reference deposit for anyone wanting to reproduce: gas ≥ 200k @ ≥ 10 gwei, 0.05 xDAI value. Actual gasUsed was 112 106.

Suggested triage order

  1. #1 — the composer needs a pre-admission gas simulation and an OOG-evict policy. Edu is opening the issue. Everything else user-visible drops out once this is fixed.
  2. #2 — will vanish along with #1.
  3. #3 — separate multi-bundle issue Edu is filing.
  4. #4 — independent, worth a same-day fix regardless of #1's timeline.
  5. #5 — file with the indexer team; low urgency, but bad for external observability.
  6. #6 — cosmetic once the deposit UI defaults change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment