The O(1) Labs team is excited to announce the release of the tools required to migrate Devnet archive database! The release will enable archive node operators on Devnet to incrementally migrate their database before the Berkeley Upgrade on Devnet.
The new tools included are:
-
mina-archive-maintenance: A tool to verify that the devnet archive database is correct and up-to-date. If the database is missing blocks, then the tool patches using blocks exported as json files by mina daemon. Users can also find these blocks in gs://mina_network_block_data location. Please refer here for documentation on this tool. -
mina-berkeley-migration-script: This script uses multiple tools to migrate devnet database and verify the migrated database. For documentation on how to use the tooling, either the toplevel simplified script or understanding the individual components, please refer to the draft documentation or abbreviated reference information.
Note - it is recommended to use a clean machine/VM for this testing with no other installed mina instances, until the final devnet hardfork build is prepared.
Supported environments include macOS, Linux (Debian 10, 11 and Ubuntu 20.04 LTS), and any host machine with Docker.
Debian Repository:
First, set up and update the unstable Debian Repository for your platform with the following commands:
sudo rm /etc/apt/sources.list.d/mina*.list
sudo echo "deb [trusted=yes] http://packages.o1test.net $(lsb_release -cs) unstable" | sudo tee /etc/apt/sources.list.d/mina.list
sudo apt-get update
Then, install the package(s) that you need:
Mina Migration Tooling: sudo apt-get install --allow-downgrades -y mina-archive-migration=3.0.0devnet-tooling-5792b50
Mina Archive Maintenance Tooling: sudo apt-get install --allow-downgrades -y mina-archive-maintenance=1.4.0beta2-compatible-060f0a5
Mina Berkeley Archive Schema: create_schema.sql and zkapp_tables.sql (both needed)
Every docker image is built for each of the supported platforms, replace CODENAME in the images below with your choice of focal, buster, or bullseye to select the base docker image.
Mina Archive Migration Tooling: gcr.io/o1labs-192920/mina-archive-migration:3.0.0devnet-tooling-5792b50-CODENAME
Mina Archive Maintenance Tooling: gcr.io/o1labs-192920/mina-archive-maintenance:1.4.0beta2-compatible-060f0a5-CODENAME
Mina Berkeley Archive Schema: create_schema.sql and zkapp_tables.sql (both needed)
The Devnet canonical-only migration and replay verification process downloads around 80GB and takes from a few hours (local PostgreSQL and all blocks downloaded over gigabit using SSD) to over a week (50ms ping to PostgreSQL).
The Mainnet canonical-only migration and replay verification process downloads 240GB and takes between 14 hours (local PostgreSQL and all blocks downloaded over gigabit using SSD) and 20-30 days (50ms ping to PostgreSQL).
For this tooling release, all the same documentation as usual for connecting to devnet applies:
https://docs.minaprotocol.com/node-operators/connecting-to-devnet
However, a new requirement for the migration is enough disk space to store the entire chain history in a bloated JSON format. Yeah, a full node can sync without downloading it, but we can't trustlessly perform data migration on data we don't have :think-about-it:
If you are short on disk space and high on patience, the mina-berkeley-migration-script --delete-blocks switch is your friend. If you want the process to avoid alternating between download and migrating, the mina-berkeley-migration-script --prefetch-blocks switch is also your friend and will do the download up-front, at the expense of disk space but lets you rerun at any time.
A bundle of some testing blocks is available at gs://mina_network_block_data/devnet-bundle-3NKRsRWBzmPR8Z8ZmJb4u8FLpnSkjRitUpKZzVkHp11QuwP5i839.tar.gz. Downloading and extracting this will take around half the time as prefetching or streaming the blocks. It is 50GB and unpacks to around 80GB.
The recommended testing configuration if you wish to migrate with live data is to use one container/environment with the SQL server and migration tools, and another that has the daemon/archive node. Using a high-latency PostgreSQL server distantly connected via network can extend the process from 12-18 hours into several days. Multiply your ping time to the server by 20 million or so to estimate the total migration time.
If you would like to test the tools without bothering with a node, O(1) is providing a fork configuration from 2024-03-25.
If you do already have a devnet node, when you are ready to perform a final migration with the mina-berkeley-migration-script, connect to your 1.4.1 node. Export a state dump for the current best tip:
curl --location "http://localhost:3085/graphql" \
--header "Content-Type: application/json" \
--data "{\"query\":\"query MyQuery {\n fork_config\n}\n\",\"variables\":{}}" | jq '.data.fork_config' > best_tip_state_dump.json
The daemon will complain (probably into the file) if it is not yet Synced, and the devnet is known to have bad bootstrap performance (one of many reasons to hardfork). So be careful, best_tip_state_dump.json might actually be an error message.
mina-create-genesis --config-file best_tip_state_dump.json --genesis-dir new-devnet-ledgers --hash-output-file hashes.json
# adjusts data format compatibility between major versions
sed -i -e 's/"set_verification_key": "signature"/"set_verification_key": {"auth": "signature", "txn_version": "2"}/' best_tip_state_dump.json
FORK_CONFIG_JSON=best_tip_state_dump.json LEDGER_HASHES_JSON=hashes.json GENESIS_TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ" -d "10 mins") mina-hf-create-runtime-config > genesis_config.json
This genesis_config.json can be used to start a new isolated devnet fork, for testing.
The migration documentation has a worked example that walks through two phases of the process, for a shortened version of the full process. To understand the full three-phase process, scroll to the top :)
These notes are longer than usual and you deserve some entertainment. Noone on the team was prepared for this.
In the realm of our code, a curious tale unfurls,
With "buggy validations" and glitches that gave us whirls.
Processes failed, though all seemed right,
Yet it brought us laughter and sheer delight.
Can you fathom why, with mind's bright light,
why mainnet block 241082 brought us relief, a moment so right?