upgrade.md

Here's the full lifecycle of a coordinated upgrade on the Cosmos Hub. There are actually two distinct eras — the old hard-fork style (cosmoshub-1→4) and the modern in-place style (cosmoshub-4 onward). Both are "coordinated upgrades" but work very differently.

---

Era 1: Hard Fork with Chain ID Increment (cosmoshub-1 → 2 → 3 → 4)

Used when the SDK lacked an in-place upgrade module. The chain literally stops, exports state, and restarts as a new chain.

Step 1 — Social Coordination

Validators agree off-chain (forums, governance proposals) on:

• A halt height (e.g., block 500,000)
• A target Gaia version (e.g., v1.0.0 → v2.0.0)
• A new chain ID (e.g., cosmoshub-1 → cosmoshub-2)

Step 2 — Set halt-height

Each validator edits ~/.gaia/config/app.toml:

halt-height = 500000

At commit time, BaseApp.checkHalt() in cosmos-sdk/baseapp/abci.go compares the current block height against this value. When height >= halt-height, the node returns:

ERR CONSENSUS FAILURE!!! err="halt per configuration height 500000 time 0"

The node stops after committing that block — state is finalized up to that height.

Step 3 — Export State

An operator on the old binary runs:

gaiad export --for-zero-height --height=500000 > cosmoshub-2-genesis.json

This calls ExportAppStateAndValidators() in gaia/app/export.go, which does critical work via prepForZeroHeightGenesis():

1. Withdraws all validator commissions — iterates every validator, calls DistributionKeeper.WithdrawValidatorCommission()
2. Withdraws all delegator rewards — iterates every delegation, calls DistributionKeeper.WithdrawDelegationRewards()
3. Clears slash events — deletes all ValidatorSlashEvent records
4. Clears historical rewards — deletes all ValidatorHistoricalRewards
5. Resets current rewards — zeroes out outstanding rewards per validator
6. Resets delegator starting info — sets all DelegatorStartingInfo.Height = 0
7. Resets unbonding heights — sets all validators' UnbondingHeight = 0
8. Resets signing info — sets ValidatorSigningInfo.StartHeight = 0 for each validator
9. Resets redelegation/unbonding creation heights — all set to 0
10. Exports each module's genesis via mm.ExportGenesisForModules() — every module serializes its state to JSON

The output is an ExportedApp:

type ExportedApp struct {
    AppState        json.RawMessage       // all module states as JSON
    Validators      []tmtypes.GenesisValidator  // validator set sorted by power
    Height          int64                 // export height
    ConsensusParams *tmproto.ConsensusParams
}

Step 4 — Migrate the Genesis

If the new SDK version changed any module's state format, the genesis needs migration:

gaiad genesis migrate v2 cosmoshub-2-genesis.json \
  --chain-id cosmoshub-2 \
  --genesis-time 2019-12-11T18:00:00Z

This runs version-specific migration functions that transform the AppState JSON. The resulting genesis file (AppGenesis struct from cosmos-sdk/x/genutil/types/genesis.go) contains:

{
  "app_name": "gaiad",
  "app_version": "2.0.0",
  "genesis_time": "2019-12-11T18:00:00Z",
  "chain_id": "cosmoshub-2",            // <-- incremented
  "initial_height": 1,                   // <-- reset to 1
  "app_hash": "",
  "app_state": { ... },                  // migrated module states
  "consensus": {
    "validators": [ ... ],
    "params": { "block": {...}, "evidence": {...}, "validator": {...} }
  }
}

Step 5 — Reset and Restart

Every validator:

# Install new binary
cp gaiad-v2.0.0 $(which gaiad)

# Reset local chain data (CometBFT state + app DB)
gaiad unsafe-reset-all

# Replace genesis
cp cosmoshub-2-genesis.json ~/.gaia/config/genesis.json

# Clear halt-height
# app.toml: halt-height = 0

# Start
gaiad start

unsafe-reset-all wipes ~/.gaia/data/ (block store, state DB, WAL) but preserves config/. The node boots as if joining a brand-new chain, using the exported state as its initial state.

Step 6 — New Chain Begins

Once 2/3+ of voting power comes online with the new genesis, the new chain starts producing blocks from height 1 under cosmoshub-2. The old chain's block history is orphaned — new nodes never sync it.

---

Era 2: In-Place Governance Upgrade (cosmoshub-4 onward)

Starting with Cosmos SDK v0.43+, the x/upgrade module allows upgrades without restarting from genesis.

Step 1 — Submit Upgrade Proposal

A governance proposal schedules an upgrade plan. The Plan struct (cosmos-sdk/x/upgrade/types/upgrade.pb.go):

type Plan struct {
    Name   string    // e.g., "v23.0.0"
    Height int64     // e.g., 24666000
    Info   string    // optional JSON with binary download URLs
}

Once the proposal passes, keeper.ScheduleUpgrade() stores the plan under key 0x0 in the x/upgrade store.

Step 2 — Chain Approaches Upgrade Height

The chain runs normally. The PreBlock hook (cosmos-sdk/x/upgrade/abci.go) checks every block:

• Is there a plan? → yes
• Is plan.Height <= currentHeight? → not yet → continue
• Safety check: if the new binary is already running (handler registered but height not reached), it panics — this prevents running new code against old state

Step 3 — At Upgrade Height, Old Binary Panics

When currentHeight == plan.Height and the old binary has no handler registered for this plan name:

1. DumpUpgradeInfoToDisk() writes ~/.gaia/data/upgrade-info.json:

   {
     "name": "v23.0.0",
     "height": 24666000,
     "info": "{\"binaries\": {\"linux/amd64\": \"https://...\"}}"
   }

2. The node panics with:

   ERR UPGRADE "v23.0.0" NEEDED at height: 24666000
   

Step 4 — Binary Swap

Manual approach:

# Stop node (already stopped from panic)
# Replace binary
cp gaiad-v23.0.0 $(which gaiad)
# Restart
gaiad start

Cosmovisor approach (automated):

Cosmovisor watches for upgrade-info.json. Directory structure:

~/.gaia/cosmovisor/
├── current -> genesis/         # symlink to active binary
├── genesis/bin/gaiad           # original binary
└── upgrades/v23.0.0/bin/gaiad  # new binary (pre-placed or downloaded)

Cosmovisor reads upgrade-info.json, swaps the current symlink to upgrades/v23.0.0/, and restarts the daemon.

Step 5 — New Binary Applies Upgrade

The new binary starts and hits PreBlock again at the same height. This time:

1. Plan exists and plan.Height <= currentHeight → yes
2. Handler for "v23.0.0" is registered → yes (registered in gaia/app/app.go)
3. ApplyUpgrade() runs with an infinite gas meter:
   • Calls the upgrade handler (e.g., gaia/app/upgrades/v26_0_0/upgrades.go):

     func CreateUpgradeHandler(...) upgradetypes.UpgradeHandler {
         return func(ctx context.Context, plan Plan, fromVM module.VersionMap) (module.VersionMap, error) {
             // Run all module migrations
             vm, err := mm.RunMigrations(ctx, configurator, fromVM)
             // Custom state changes (e.g., init new module params)
             return vm, nil
         }
     }

   • RunMigrations compares each module's ConsensusVersion in the new binary against the stored version map, running registered migration functions for any that changed
   • Store upgrades (adding/removing module stores) are applied via UpgradeStoreLoader at load time, configured in gaia/app/app.go:

     StoreUpgrades: storetypes.StoreUpgrades{
         Added: []string{"tokenfactory"},  // new modules
     }

4. Updates the module version map in store (key prefix 0x2)
5. Increments the app version in consensus params
6. Clears the plan from store
7. Marks upgrade as done (key: 0x1 || height || name)
8. Returns ConsensusParamsChanged: true

Step 6 — Normal Operation Resumes

The block at the upgrade height completes with the new code. Subsequent blocks run on the new binary. No chain ID change, no height reset, no state export. The chain continues seamlessly.

---

Key Differences

	Hard Fork (Era 1)	In-Place (Era 2)
Chain ID	Incremented (cosmoshub-3 → 4)	Unchanged
Block height	Reset to 1	Continuous
State	Exported to JSON, re-imported	Migrated in-place in DB
Historical blocks	Lost (archived separately)	Preserved
Key file created	New genesis.json	upgrade-info.json
Coordination	halt-height in app.toml	Plan via governance
Data reset	unsafe-reset-all	None
Downtime	Hours (export + distribute + restart)	Seconds to minutes (binary swap)