Skip to content

Instantly share code, notes, and snippets.

@yorickdowne

yorickdowne/supergau.md

Last active March 10, 2025 19:28
Show Gist options
  • Save yorickdowne/7058c5799cd4ce60d2eafe137044596b to your computer and use it in GitHub Desktop.
Save yorickdowne/7058c5799cd4ce60d2eafe137044596b to your computer and use it in GitHub Desktop.
Download ZIP
Holesky Pectra recovery
Raw
supergau.md

Holesky has been recovered

As of 2024-03-10 19:30 UTC, Holesky finalized on the correct chain. You can use recent versions of your clients and sync normally.

Historical content below, no longer relevant

Recover Eth Docker Holesky setup

You need:

  • 64 GiB of RAM, or 32 GiB + 32 GiB swap. 16 GiB is unlikely to work
  • Special CL build, pandaops checkpoint, some flags
  • Fixed EL build, pandaops snapshot, full sync flags

CL fixed builds: Lighthouse, Lodestar, Prysm, Teku, Nimbus, Grandine

EL that need no changes: Erigon, Reth

EL that require resync from snapshot: Nethermind, Besu, Geth

Instructions last updated:

  • 2024-03-10 13:45 UTC, pandaops rescue CL
  • 2024-03-08 03:30 UTC, Lodestar non-finalized checkpoint state
  • 2024-03-06 21:00 UTC, update heap for Teku and Besu
  • 2024-03-05 12:00 UTC, update Lodestar
  • 2024-03-04 13:00 UTC, update Lodestar and Lighthouse
  • 2024-03-03 01:30 UTC, update Prysm
  • 2024-03-02 15:00 UTC, update Lodestar and Lighthouse
  • 2024-03-02 10:00 UTC, advise to remove slashing protection
  • 2024-03-01 00:30 UTC, update Lodestar
  • 2024-02-27 19:30 UTC, add Grandine; update Teku and Prysm
  • 2024-02-27 17:00 UTC, add Teku and Nimbus
  • 2024-02-27 16:30 UTC, add Prysm
  • 2024-02-27 15:00 UTC, initial

Point validator client at rescue CL

Use the ethpandaops rescue CL to start attesting. Remove the slashing protection DB from your validator client for this.

Assuming the CL and EL are local:

CL_NODE=http://consensus:5052,http://holesky-rescue.ethpandaops.io

If they are not local, just add ,http://holesky-rescue.ethpandaops.io to whatever you already have in CL_NODE

Set client images and flags

cp .env .env.pectra-backup

nano .env

Find these variables and set them, depending on the clients you are running (check COMPOSE_FILE at the top to see which your clients are)

Lighthouse

LH_DOCKER_TAG=v7.0.0-beta.2
LH_DOCKER_REPO=sigp/lighthouse
CL_EXTRAS=--state-cache-size 32
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io

Lodestar

LS_DOCKER_TAG=holesky-rescue-edab7f0
LS_DOCKER_REPO=ethpandaops/lodestar
LODESTAR_HEAP=--max-old-space-size=32768
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io

If Lodestar takes too long to sync or won't sync, you can try to load it from a non-finalized checkpoint state. This is very experimental!

RAPID_SYNC_URL=
CL_EXTRAS=--checkpointState https://pub-3cd22dca6ea14e7a83ce8eb86685b16e.r2.dev/0x8ace010000000000847adec23f86a48d8b9a1f70b850235cebf966d127e1ac2c0e26e1dbcc047592

Then ./ethd resync-consensus

This checkpoint state can be found at https://dora-holesky.pk910.de/slot/3789120 and uses this epoch/root:

{
  "epoch": "118410",
  "root": "0x847adec23f86a48d8b9a1f70b850235cebf966d127e1ac2c0e26e1dbcc047592"
}

Prysm

PRYSM_DOCKER_TAG=hackSync
PRYSM_DOCKER_REPO=ethpandaops/prysm-beacon-chain
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io

Teku

TEKU_DOCKER_TAG=develop
TEKU_DOCKERFILE=Dockerfile.binary
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io
TEKU_HEAP=-Xmx12g

Nimbus

NIM_SRC_BUILD_TARGET=feat/splitview
NIM_DOCKERFILE=Dockerfile.source
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io

Grandine

GRANDINE_SRC_BUILD_TARGET=holesky-recover
GRANDINE_DOCKERFILE=Dockerfile.source
RAPID_SYNC_URL=https://checkpoint-sync.holesky.ethpandaops.io

Besu

EL_EXTRAS=--sync-mode=FULL
BESU_HEAP=-Xmx8g

The following are defaults and if you didn’t change them, no need to do so now

BESU_DOCKER_TAG=latest
BESU_DOCKER_REPO=hyperledger/besu

Nethermind

EL_EXTRAS=--Sync.SnapSync false --Sync.FastSync false

The following are defaults and if you didn’t change them, no need to do so now

NM_DOCKER_TAG=latest
NM_DOCKER_REPO=nethermind/nethermind

Geth

EL_EXTRAS=--syncmode full

The following are defaults and if you didn’t change them, no need to do so now

GETH_DOCKER_TAG=stable
GETH_DOCKER_REPO=ethereum/client-go

Reth, Erigon

No changes needed

Update images

./ethd update

Resync consensus

./ethd resync-consensus

Resync execution

We will start a resync, stop the execution client, get and extract a snapshot, then start execution again

./ethd resync-execution

./ethd stop execution

sudo bash

apt update && apt install wget zstd

Go to the Docker volume of the client. This assumes the directory is eth-docker - adjust if it’s not

Nethermind

Pay close attention to these commands. In particular, some rf commands remove the contents of a directory and should not remove the directory itself.

Don’t skip any, fix something if it didn’t work or broke, and adjust paths as needed for your system

cd /var/lib/docker/volumes/eth-docker_nethermind-el-data/_data

wget https://snapshots.ethpandaops.io/holesky/nethermind/latest/snapshot.tar.zst

This will take 20-30 minutes. While it’s running, if you have 32 GiB of RAM, go to “Create swap file”, then come back here

Unpack the snapshot, chown, and remove the snapshot

Verify you are in the nethermind directory

pwd

ls

Only continue if you are in the right directory

rm -rf nethermind_db/*

tar -I zstd -xvf snapshot.tar.zst -C nethermind_db/

chown -R 10001:10001 nethermind_db

rm snapshot.tar.zst

Exit the root shell, and start the client again

exit

./ethd up

Besu

Pay close attention to these commands. In particular, some rf commands remove the contents of a directory and should not remove the directory itself.

Don’t skip any, fix something if it didn’t work or broke, and adjust paths as needed for your system

cd /var/lib/docker/volumes/eth-docker_besu-el-data/_data

wget https://snapshots.ethpandaops.io/holesky/besu/latest/snapshot.tar.zst

This will take 20-30 minutes. While it’s running, if you have 32 GiB of RAM, go to “Create swap file”, then come back here

Unpack the snapshot, chown, and remove the snapshot

Verify you are in the besu directory

pwd

ls

Only continue if you are in the right directory

rm -rf *

tar -I zstd -xvf snapshot.tar.zst

rm key

chown -R 10001:10001 .

rm snapshot.tar.zst

Exit the root shell, and start the client again

exit

./ethd up

Geth

Pay close attention to these commands. In particular, some rf commands remove the contents of a directory and should not remove the directory itself.

Don’t skip any, fix something if it didn’t work or broke, and adjust paths as needed for your system

cd /var/lib/docker/volumes/eth-docker_geth-el-data/_data

wget https://snapshots.ethpandaops.io/holesky/geth/latest/snapshot.tar.zst

This will take 20-30 minutes. While it’s running, if you have 32 GiB of RAM, go to “Create swap file”, then come back here

Unpack the snapshot, chown, and remove the snapshot

Verify you are in the geth directory

pwd

ls

Only continue if you are in the right directory

rm -rf geth/*

tar -I zstd -xvf snapshot.tar.zst -C geth/

rm geth/nodekey

chown -R 10001:10001 geth

rm snapshot.tar.zst

Exit the root shell, and start the client again

exit

./ethd up

Create swap file

If you have 64 GiB of RAM or more, skip this step

If you have 16 GiB of RAM, you should likely not attempt this recovery until after Holesky finalizes again and you can use stock images with stock parameters

If you have 32 GiB of RAM, adding 32 GiB of swap should allow you to follow along

sudo fallocate -l 32G /swapfile

sudo chmod 600 /swapfile

sudo mkswap /swapfile

sudo swapon /swapfile

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=1

sudo nano /etc/sysctl.conf
Find the end and add
vm.swappiness=1

Make a note to remove the swapfile again when you are done and Holesky has recovered

sudo nano /etc/fstab

Find the /swapfile line and comment it out or remove it

sudo swapoff /swapfile

sudo rm /swapfile

Observe sync

./ethd logs -f —tail 50 consensus

./ethd logs -f —tail 50 execution

What about my validators?

Remove the slashing protection DB, with the steps communicated by ethpandaops.

You can use sudo bash to navigate to the VC (validator client) volume and delete what pandaops told you to delete.

This will get your validators attesting again, to aid in the effort to finalize Holesky. They will likely get slashed eventually.

./ethd logs -f —tail 50 validator if you wish to see what it is doing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment