Antithesis Amaru Insufficient leader stake crash excerpt

antithesis-amaru-insufficient-leader-stake-excerpt.md

Antithesis Amaru Crash Excerpt

Session id: 203a9ed69fe29b6f001cb0a7425cb385-50-7

Crash signature: HeaderValidationError: Insufficient leader stake.

Relevant broad search: https://cardano.antithesis.com/search?search=v5veyJxIjp7Im4iOnsiciI6eyJoIjpbeyJoIjpbeyJjIjpmYWxzZSwiZiI6ImdlbmVyYWwub3V0cHV0X3RleHQiLCJvIjoiY29udGFpbnMiLCJ2IjoiNTkxNC5hNTc0NjA2Mzg0ZWYxNDE3YjFmYzg5MTUzYTljZWM0YmIzZjlmMjRhOGFiMTc0YTkzY2I0NjIwZTlkNmU1YTA1In1dLCJvIjoib3IifV0sIm8iOiJhbmQifSwidCI6eyJnIjpmYWxzZSwibSI6IiJ9LCJ5Ijoibm9uZSJ9fSwicyI6IjIwM2E5ZWQ2OWZlMjliNmYwMDFjYjBhNzQyNWNiMzg1LTUwLTcifQ&report_name=&get_logs=false&get_logs_event_desc=

Ordered Excerpt

5487.099 amaru-relay-2
2026-05-06T18:18:39.000416Z INFO epoch_transition:compute_stake_distribution: amaru::ledger::state::stake_distribution: stake_distribution.snapshot epoch=22 accounts=3 pools=3 active_stake=614990902714152 dreps=2 dreps_voting_stake=0 pools_voting_stake=614990902714152 from=22 into=23 epoch=22

5487.099 amaru-relay-1
2026-05-06T18:18:39.000536Z INFO epoch_transition:compute_stake_distribution: amaru::ledger::state::stake_distribution: stake_distribution.snapshot epoch=22 accounts=3 pools=3 active_stake=614990902714152 dreps=2 dreps_voting_stake=0 pools_voting_stake=614990902714152 from=22 into=23 epoch=22

5739.100 amaru-relay-2
2026-05-06T18:22:51.001153Z INFO epoch_transition:compute_stake_distribution: amaru::ledger::state::stake_distribution: stake_distribution.snapshot epoch=23 accounts=3 pools=3 active_stake=615793407101371 dreps=2 dreps_voting_stake=0 pools_voting_stake=615793407101371 from=23 into=24 epoch=23

5739.100 amaru-relay-1
2026-05-06T18:22:51.001199Z INFO epoch_transition:compute_stake_distribution: amaru::ledger::state::stake_distribution: stake_distribution.snapshot epoch=23 accounts=3 pools=3 active_stake=615793407101371 dreps=2 dreps_voting_stake=0 pools_voting_stake=615793407101371 from=23 into=24 epoch=23

5830.100 amaru-relay-2
2026-05-06T18:24:22.001081Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5902 tip.hash=7be5090c3952742ec421d8b3e7c05aa5a731d5ef3b5ee904e54c24a43f386d8a tip.block_height=1133 max_block_height=1133 suppressed=0

5830.100 amaru-relay-1
2026-05-06T18:24:22.001106Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5902 tip.hash=7be5090c3952742ec421d8b3e7c05aa5a731d5ef3b5ee904e54c24a43f386d8a tip.block_height=1133 max_block_height=1133 suppressed=0

5831.100 amaru-relay-2
2026-05-06T18:24:23.001195Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5903 tip.hash=641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 tip.block_height=1134 max_block_height=1134 suppressed=0

5831.100 amaru-relay-1
2026-05-06T18:24:23.001251Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5903 tip.hash=641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 tip.block_height=1134 max_block_height=1134 suppressed=0

5842.100 amaru-relay-2
2026-05-06T18:24:34.000706Z INFO amaru_consensus::stages::track_peers: roll backward peer=p2.example:3001 current=5903.641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 highest=5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05

5842.100 amaru-relay-1
2026-05-06T18:24:34.001087Z INFO amaru_consensus::stages::track_peers: roll backward peer=p1.example:3001 current=5903.641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 highest=5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05

5842.100 amaru-relay-1
2026-05-06T18:24:34.000679Z ERROR amaru_consensus::stages::track_peers: chain_sync.validate_header.failed error=Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Insufficient leader stake. peer=p1.example:3001

5842.100 amaru-relay-2
2026-05-06T18:24:34.000724Z ERROR amaru_consensus::stages::track_peers: chain_sync.validate_header.failed error=Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Insufficient leader stake. peer=p2.example:3001

Interleaved With Producer Tracer Events

This is sorted by the log payload timestamp where present; Antithesis virtual time is kept on the left.

5813.100 amaru-relay-2
2026-05-06T18:24:05.000821Z INFO amaru_consensus::stages::validate_block: rolling back ledger to common ancestor point parent=5875.aee4cf9f6fe9f188934521151114367f4369ed58e9635284c4df473bcdfc1030 current=5878.0ea4ac31f6ad53ec5e6e6410bef758cc55ccf252d7bcac53ab6e44a34ad5f2ba

5813.100 amaru-relay-2
2026-05-06T18:24:05.000822Z INFO amaru_consensus::stages::validate_block: rolling forward ledger to reach parent parent=5875.aee4cf9f6fe9f188934521151114367f4369ed58e9635284c4df473bcdfc1030 current=5875.aee4cf9f6fe9f188934521151114367f4369ed58e9635284c4df473bcdfc1030 points=0

5813.100 amaru-relay-2
2026-05-06T18:24:05.000823Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5885 tip.hash=4076062c6d3810f9835e357860f7eba0fec5b87a5df046369f683b575124720f tip.block_height=1127 max_block_height=1127 suppressed=0

5813.100 amaru-relay-1
2026-05-06T18:24:05.000830Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5885 tip.hash=4076062c6d3810f9835e357860f7eba0fec5b87a5df046369f683b575124720f tip.block_height=1127 max_block_height=1127 suppressed=0

5817.761 log-tailer host=p3
{"at":"2026-05-06T18:24:08.001231968Z","ns":"Forge.Loop.DidntAdoptBlock","data":{"kind":"TraceDidntAdoptBlock","slot":5888},"sev":"Error","thread":"51","host":"p3"}

5817.789 log-tailer host=p3
{"at":"2026-05-06T18:24:08.001960933Z","ns":"Forge.Loop.DidntAdoptBlock","data":{"kind":"TraceDidntAdoptBlock","slot":5888},"sev":"Error","thread":"51","host":"p3"}

5830.100 amaru-relay-2
2026-05-06T18:24:22.001081Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5902 tip.hash=7be5090c3952742ec421d8b3e7c05aa5a731d5ef3b5ee904e54c24a43f386d8a tip.block_height=1133 max_block_height=1133 suppressed=0

5830.100 amaru-relay-1
2026-05-06T18:24:22.001106Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5902 tip.hash=7be5090c3952742ec421d8b3e7c05aa5a731d5ef3b5ee904e54c24a43f386d8a tip.block_height=1133 max_block_height=1133 suppressed=0

5831.100 amaru-relay-2
2026-05-06T18:24:23.001195Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5903 tip.hash=641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 tip.block_height=1134 max_block_height=1134 suppressed=0

5831.100 amaru-relay-1
2026-05-06T18:24:23.001251Z INFO amaru_consensus::stages::adopt_chain: adopted tip tip.slot=5903 tip.hash=641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 tip.block_height=1134 max_block_height=1134 suppressed=0

5842.100 amaru-relay-1
2026-05-06T18:24:34.000679Z ERROR amaru_consensus::stages::track_peers: chain_sync.validate_header.failed error=Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Insufficient leader stake. peer=p1.example:3001

5842.100 amaru-relay-2
2026-05-06T18:24:34.000706Z INFO amaru_consensus::stages::track_peers: roll backward peer=p2.example:3001 current=5903.641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 highest=5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05

5842.100 amaru-relay-2
2026-05-06T18:24:34.000724Z ERROR amaru_consensus::stages::track_peers: chain_sync.validate_header.failed error=Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Failed to validate header at 5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05: HeaderValidationError: Insufficient leader stake. peer=p2.example:3001

5842.100 amaru-relay-1
2026-05-06T18:24:34.001087Z INFO amaru_consensus::stages::track_peers: roll backward peer=p1.example:3001 current=5903.641205b41d186aa99d51761edba7e5cb69ad88791875790f44a42c51d2be3553 highest=5914.a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05

5843.845 log-tailer host=p2
{"at":"2026-05-06T18:24:34.008165299Z","ns":"Forge.Loop.DidntAdoptBlock","data":{"kind":"TraceDidntAdoptBlock","slot":5914},"sev":"Error","thread":"50","host":"p2"}

5867.772 log-tailer host=p1.example
{"at":"2026-05-06T18:24:58.002007842Z","ns":"Forge.Loop.DidntAdoptBlock","data":{"kind":"TraceDidntAdoptBlock","slot":5938},"sev":"Error","thread":"44","host":"p1.example"}

5867.801 log-tailer host=p1.example
{"at":"2026-05-06T18:24:58.003107428Z","ns":"Forge.Loop.DidntAdoptBlock","data":{"kind":"TraceDidntAdoptBlock","slot":5938},"sev":"Error","thread":"44","host":"p1.example"}

source=p1, source=p2, and source=p3 stdout had no matches for the exact failed point. The producer-side rows above come from source=log-tailer, which tails the cardano-node tracer files and emits Warning/Error/Critical entries.

Notes

• The relays had successfully adopted slot 5903, block height 1134, immediately before the failure.
• The failed header is slot 5914, hash a574606384ef1417b1fc89153a9cec4bb3f9f24a8ab174a93cb4620e9d6e5a05.
• Both relays fail on the same header, but against different upstream peers: p1.example:3001 and p2.example:3001.
• Log search counts observed:
  • Insufficient leader stake + amaru-relay-1: 149 matching events.
  • Insufficient leader stake + amaru-relay-2: 149 matching events.
  • Exact failed point 5914.a574...: 519 matching events.
  • Exact failed point + source=p1: 0 matching events.
  • Exact failed point + source=p2: 0 matching events.
  • Exact failed point + source=log-tailer: 0 matching events.
  • Minute window 2026-05-06T18:24: + source=log-tailer: 9 matching events.