Setup guide: BSC snapshot + AWS + geth full sync

gistfile1.md

1. Create an EC2 instance for snapshot downloading

   • Depending of cpu cores, uncompress faster
   • Estimate instance costs there: https://aws.amazon.com/ru/ec2/pricing/on-demand/
   • Connect ssh -i key.pem [email protected]

2. Create and attach EC2 volume gp3, min 1.5tb, better 2tb, 8000 IOPS, 250MB/s

   • Estimate GB size and costs https://calculator.aws/#/createCalculator/EBS
   • Attach volume to low-cost instance

3. Prepare attached volume to usage:

   • Based on this guide https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
   • Short steps to do are:

      sudo mkfs -t ext4 /dev/nvme1n1
      sudo mount /dev/nvme1n1 /home/ubuntu/bsc
      sudo chown ubuntu:ubuntu /home/ubuntu/bsc

   • lsblk shows connected volumes

4. Download and uncompress bsc snapshot

   • cd /home/ubuntu/bsc
   • Take download urls here https://github.com/binance-chain/bsc-snapshots
   • nohup wget -q -O - "URL TO ENDPOINT" | tar -I lz4 -xvf - &; disown %1
   • Downloading of 1150GB could take 2-4hr, depends of cpu cores and network.
   • Downloading of 1500GB could take 5-6hr, depends of cpu cores and network.
   • cd /home/ubuntu/bsc; mv server/data-seed/geth/ node/

5. Prepare bsc-geth binary and default configs:

   • sudo snap install go --classic
   • 1.1.8 might become outdated after a time, be careful.

   cd /home/ubuntu/bsc
   
   wget -c https://github.com/binance-chain/bsc/archive/refs/tags/v1.1.8.tar.gz -O - | tar -xz
   cd ./bsc-1.1.8; sudo make geth
   sudo cp ./build/bin/geth /home/ubuntu/bsc/; cd /home/ubuntu/bsc
   
   wget -c https://github.com/binance-chain/bsc/releases/download/v1.1.8/mainnet.zip; unzip mainnet.zip; rm mainnet.zip
   mv mainnet/* ./
   
   ./geth --help

At this step we have uncomressed snapshot which ready for sync. Chance instance if needed for more capable now and attach volume with same aws'avalaibility zone'

6. (Optional) Create AMI

   • Once all is ok, think about required but not installed tools yet, for example htop iotop unzip etc
   • If you passed all previous steps you have installed tool set.
   • Based on this guide https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html create AMI of m5.large

7. Create instance for geth sync stage

   • Create EC2 m5.2xlarge, (custom AMI or ubuntu), 16gb storage.
   • Attach 1tb+ volume
   • cd ~/bsc
   • sudo mount /dev/nvme1n1 ~/bsc

8. Start sync

   • cd ~/bsc
   • Basic command: ./geth --config config.toml --datadir node --cache 30000 --ws
   • Modified command: nohup ./geth --config config.toml --datadir node --cache 30000 --ws &; disown %1
   • At geth-v1.1.3 added --diffsync, it could speedup sync, dont tested yet

9. Analyze sync logs

   • tail -f ~/bsc/node/bsc.log
   • Estimate speed of sync:
     • Start sync and remember "age=" of "Imported new chain segment"
     • Check same "age=" after 30min
     • In good conditions it might become less in 30min (so 60min synces for 30min)

Setting up geth as a systemd .service
note, that need replace mgeth to geth

cd ~/bsc
cat > mgeth.service <<EOF
[Unit]
  Description=mgeth
  StartLimitIntervalSec=500
  StartLimitBurst=5

[Service]
  Restart=on-failure
  RestartSec=5s
  WorkingDirectory=/home/ubuntu/bsc
  ExecStartPre=/bin/chmod +x /home/ubuntu/bsc/mgeth
  ExecStart=/home/ubuntu/bsc/mgeth --config /home/ubuntu/bsc/config.toml --datadir /home/ubuntu/bsc/node --cache 28000 --ws
  Type=simple
  User=ubuntu
  KillSignal=SIGINT
  TimeoutStopSec=120

[Install]
  WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable ~/bsc/mgeth.service

sudo service mgeth start
service mgeth status

Reducing 1.5tb (freshly uncompressed snapshot). Ensure that you did mv server/data-seed/geth/ node/

cd ~/bsc
sudo service mgeth stop
./mgeth snapshot prune-block --datadir ./node --datadir.ancient /home/ubuntu/bsc/node/chaindata/ancient --block-amount-reserved 10000
# important note to use obsolete path to ancient

1500gb -> 700gb, you can move to cheaper EBS volume now
cp ~/bsc/* ~/bsc2/

Update ext4 after aws EBS volume increased

sudo resize2fs /dev/nvme1n1

Start prune-state, 5-10hr needed, cpu and io intensive

./mgeth snapshot prune-state --datadir ./node

sample of linking remote port to use as localhost geth
ws://127.0.0.1:8546

ssh -f -N -L 8546:localhost:8546 -i aws/123.pem [email protected]

Use profiler

1. --pprof // add this at geth start args
2. wait geth sync
3. curl -sK -v "http://127.0.0.1:6060/debug/pprof/profile?seconds=60" > profile_60s.out
4. go tool pprof -http=127.0.0.1:6060 profile_60s.out