k3mempool.txt

## Anomalies found and explained

### A. Post-block "invalid tx" bursts (benign race, very frequent)

When block N is applied, its transactions remain in the mempool for a few milliseconds.
`CandidateGenerator` (running on `ef-critical-dispatcher`) races ahead of mempool cleanup
(`ErgoMemPool` invalidation/removal on another thread), so `collectTxs` flags the just-mined
transactions as "double-spending or spending non-existing inputs"
(`src/main/scala/org/ergoplatform/mining/CandidateGenerator.scala:897`).

Verified id-by-id for the 13:23:51 burst: 5 of 6 "invalid" txs were included in block 1834539;
the 6th (`c2391a0c…`) was a genuine double-spend correctly eliminated.

Side-effect: 149× WARN "Can't invalidate transaction" / "pool.get failed" — `EliminateTransactions`
from the candidate generator arrives after the tx was already removed from the pool.

### B. Node's own emission tx flagged "double-spending" + ERROR "Proofs for 0 txs" (real anomaly, 7 occurrences)

Timeline at 12:52:01:

- `.298` — block 1834526 applied; new emission box `63c3a2df…`
- `.317` — candidate generator creates its emission tx for 1834527 spending `63c3a2df…`
- `.667` — block 1834527 applied **on another thread**, spending that same emission box
- `.669` — `inputsNotSpent` fails for the node's own emission tx →
  "Collected 0 transactions, invalid total:1" → `createCandidate` throws
  `IllegalArgumentException: Proofs for 0 txs` (`CandidateGenerator.scala:623`), logged as ERROR.

Root cause: `generateCandidate`'s `chainSynced` check (`CandidateGenerator.scala:415`) is evaluated
only once at the start, and `UtxoStateReader` is a live view over the mutable UTXO store, not a
snapshot, while `UtxoNodeViewHolder` and `CandidateGenerator` run on different threads of the
shared multi-threaded `critical-dispatcher`. A 6-burst of the same ERROR occurred at 12:08:18
during rapid catch-up (blocks 1833665→1833667 applied within ~1 s).

Impact: wasted candidate assembly (~20–330 ms each) and misleading ERROR/DEBUG logs; the candidate
is regenerated on the next `FullBlockApplied`. No consensus risk.

### C. Context-expiring mempool txs (expected behavior, 12 occurrences)

12 txs failed at candidate time with "Scripts of all transaction inputs should pass verification:
#0 => Success((false,50))". Example: `f461a897…` was valid at mempool entry (13:15:46), included in
candidates at 13:17–13:18, then failed at 13:19:20 with **no new block in between** — a
timestamp/height-dependent contract (e.g. expiring order) whose script turned false against the
*upcoming* preHeader used in candidate assembly. Such txs are properly eliminated from the mempool
via `EliminateTransactions`.

### D. "No fee proposition found" noise (84 occurrences)

The INFO message fires whenever the assembled txs carry no fee box — including the node's own
emission tx when the mempool is empty — and for zero-fee transactions, which are valid in Ergo and
are included in the candidate without a fee-collection tx.

## Tests written (test code only, no production code touched)

Added 3 properties to `src/test/scala/org/ergoplatform/mining/CandidateGeneratorPropSpec.scala`:

1. `stale emission tx is invalidated when its box was spent by concurrently applied block` — reproduces B.
2. `mempool transactions spent by applied block are invalidated at next candidate assembly` — reproduces A.
3. `zero-fee transactions are collected without creating fee transaction` — documents D.

All 11 tests in the suite pass:
`sbt "testOnly org.ergoplatform.mining.CandidateGeneratorPropSpec"` → succeeded 11, failed 0.

Note: `sbt scalafmtCheck` from AGENTS.md does not resolve — the scalafmt sbt plugin is not present
in `project/plugins.sbt`.

## Suggested fixes (not applied, per instructions)

1. Re-check `chainSynced` **after** `collectTxs` inside `createCandidate` (or capture a state
   version before assembly and verify it didn't change); on mismatch, retry instead of throwing.
2. Downgrade the "Proofs for 0 txs" `IllegalArgumentException` path to a retryable debug/warn when
   the cause is a mid-assembly state change.
3. Log invalidation of the node's own emission tx distinctly (it is never a "double-spend"), and
   downgrade "Can't invalidate transaction" WARNs caused by the double-removal race.

---

# Part 2 — `~/logs` analysis (Jul 20 00:00 → Jul 22 14:15, ~14.9M lines, mining node)

Volume overview: 25 ERROR, 22,931 WARN over ~2.6 days.

## Issue 1 (critical) — OrderedTxPool registry/tree desync producing immortal "zombie" transactions

**Symptoms:** 21,162× WARN `pool.get failed for <id>` (ErgoMemPool.scala:158), in bursts of
~150–230/minute recurring after blocks, plus 587× `Can't invalidate transaction`. Zero occurrences
of the opposite desync direction (`in registry but not ordered transactions`).

**Root cause chain:**

1. `OrderedTxPool.orderedTransactions` is a `TreeMap` keyed by `WeightedTxId` with ordering
   `(-weight, id)`, while `WeightedTxId.equals/hashCode` use **id only**
   (OrderedTxPool.scala:240-250) — so the same tx under two different weights is *two different
   tree keys*.
2. `updateFamily` (OrderedTxPool.scala:200-227) computes `parentTxs` **once** from the pre-fold
   state, then mutates weights inside the fold and its recursion. In diamond ("reconvergent")
   topologies — tx D spending outputs of A, B1 and B2 where B1,B2 also spend outputs of A — A can
   be bumped (≥2×) by recursion into B1/B2 **before** its own fold iteration with the now-stale
   `wtx`. Then `orderedTransactions - wtx` silently misses (ordering mismatch) and
   `+ (newWtx -> ut)` inserts a **duplicate** entry for the same tx id. The registry points to one
   duplicate; a later `remove`/`invalidate` cleans the registry + one entry, leaving a zombie in
   `orderedTransactions`.
3. The zombies are then **unremovable**: `ErgoMemPool.invalidate(id)` fallback finds the tx in
   `orderedTransactions` but delegates to `OrderedTxPool.invalidate`, whose `case None` branch
   (registry miss, OrderedTxPool.scala:161-162) only adds the id to the bloom filter and leaves
   `orderedTransactions` unchanged; `remove` no-ops entirely for registry-missing txs
   (OrderedTxPool.scala:128-129).

**Consequences:** zombies keep being served by `getAllPrioritized`/`take`/`getAll` to every
candidate assembly (wasted validation + recurring "invalid" flags), generate the ~200/minute WARN
bursts, can defeat capacity eviction in `put` (victim removal no-ops on a zombie → pool can grow
past `mempoolCapacity`), and skew mempool statistics. Note: the same fold also **inflates
weights** of ancestors bumped multiple times for one descendant (consistent but wrong ordering
signal). The diamond topology needed is exactly the one reproduced in the untracked test file
`src/test/scala/org/ergoplatform/nodeView/mempool/OrderedTxPoolSpec.scala`
(`NVOrderedTxPoolSpec`, "ReconvergentFixture").

## Issue 2 (real bug, 1 occurrence) — CandidateGenerator actor crash: `None.get`

At 09:00:50: `java.util.NoSuchElementException: None.get` at `CandidateGenerator.scala:224`
(`completeBlock(state.cachedPreviousCandidate.get.candidateBlock, solution)`), actor restarted by
supervision. Trigger: a miner-submitted `AutolykosSolution` completes the *current* candidate into
a block that **fails** `powScheme.validate` (e.g. malformed/wrong-version solution), and
`cachedPreviousCandidate` is `None` → unguarded `.get`. Fix: pattern-match instead of
`.getOrElse(... .get)`, reply `StatusReply.error` when neither candidate is usable.

## Issue 3 (external data, noisy logging) — EIP-27 violations from the network

23× ERROR `EIP-27 check failed: requirement failed: outputs contain reemission token`
(ErgoTransaction.scala:319 `verifyReemissionSpending`). Some wallet/node on mainnet keeps
broadcasting transactions moving reemission tokens outside the reemission contract. The node
correctly rejects them; ERROR level is excessive for invalid *peer* data (WARN/DEBUG suffices).

## Issue 4 (interpreter robustness) — `ArrayIndexOutOfBoundsException` during script verification

5 distinct mainnet txs (e.g. `db8a8de2…`, `2a09f6c3…`, `e021cc84…`) fail input-script verification
with a stackless `java.lang.ArrayIndexOutOfBoundsException: null` inside the sigma interpreter
(tx ids logged with full verification context). Txs are properly rejected/invalidated, but a
crafted script triggering AIOOBE inside the interpreter is worth reporting to the sigmastate
team; JIT `-OmitStackTraceInFastThrow` erases the stack, hindering diagnosis.

## Issue 5 (benign but noisy) — sync/mempool chatter

- 828× `Extension is empty while comparison is fork` — one 2-minute burst (06:06:27–06:08:36)
  during heavy incoming-connection churn; peers transiently on a competing tip; self-resolved,
  no local rollback.
- 192× `Trying to apply modifier … that's already in history` — duplicate deliveries, benign.
- 29× `Can not generate block candidate` — empty-mempool/unsynced retry path, benign.
- 84× NetworkController connection WARNs + repeated `too many incoming connections` denials
  (this log predates the `incomingLimit` improvements on master).
- Occasional small-fork `Rollback UtxoState` events (normal mainnet 1-block forks).
- Critical-dispatcher thread churn: pool threads named `#6` and `#60` simultaneously active at
  log start (pool size is 2) — ~58 threads were created/died earlier in the JVM's life,
  indicating uncaught task escapes (e.g. Issue 2-style crashes) killing pool workers.

## Correlation with the amit-node findings (Part 1)

The `Can't invalidate transaction` double-removal race (587×) and post-block invalid bursts exist
here too, but this log's dominant pathology is the OrderedTxPool zombie cycle (Issue 1), which
was only nascent on the amit node (149 WARNs). The A1+B1 fixes (implemented, see Part 3) reduce
the invalidation traffic that feeds the zombie-reprocessing loop but do **not** fix the pool
desync itself — that needs an `updateFamily` rework (re-read parents after each recursion, or
accumulate weight deltas per parent id first) plus tree cleanup for registry-missing ids in
`invalidate`/`remove`.

---

# Part 3 — Implemented fixes (A1 + B1), 2026-07-22

Production changes:

1. **A1** — `ErgoNodeViewSynchronizerMessages.scala`: `LocalBlockApplied`/`RemoteBlockApplied`
   now carry `txIds: Seq[ModifierId]`; populated for free at the publish site
   (`ErgoNodeViewHolder.scala`). Pattern-match sites updated (`ErgoNodeViewSynchronizer.scala`).
2. **A1** — `CandidateGenerator`: `CandidateGeneratorState.lastAppliedBlockTxs` stores
   `(headerId, txIds)` on every `FullBlockApplied`; new pure helper `excludeAppliedTxs` filters
   those txs out of `poolTransactions` when the applied block is still the best full block
   (rollback-safe). Effect: just-mined txs no longer enter assembly — no misleading
   double-spending logs, no spurious `EliminateTransactions`, no double-removal WARNs.
3. **B1** — `generateCandidate` re-evaluates `isChainSynced` (new testable predicate, based on
   live `historyStorage` reads via `bestFullBlockOpt`) **after** `createCandidate`; on mismatch
   the candidate (success *or* failure) is discarded with a debug log instead of producing the
   `Proofs for 0 txs` ERROR or, worse, a silently inconsistent candidate. Regeneration happens
   via the already-queued `FullBlockApplied` trigger.

Tests added (test code only): `CandidateGeneratorPropSpec` — `excludeAppliedTxs filters
transactions of the applied best block only`, `isChainSynced compares best full block id with
state context last header id` (plus the 3 race-behavior properties from Part 1).