We recently rolled out two changes to the Bitcoin block acceptance rules (BIP16 and BIP30); this document records the lessons learned and makes recommendations for handling future blockchain rule changes.
Note: there are "soft" rule changes and "hard" rule changes. "Soft" changes tighten up the rules-- old software will accept all the blocks and transactions created by new software, but the opposite may not be true. "Soft" changes do not require the entire network of miners and merchants and users to upgrade or be left behind.
"Hard" changes modify the rules in a way that old, un-upgraded software consider illegal. At this point it is much, much more difficult (some might say impossible) to roll out "hard" changes, because they require every miner and merchant and user to upgrade.
- Be sure to consider all 4 combinations of old/new software running before/after the majority of the network supports the new rule(s).
- Backwards compatibility with old clients/miners is extremely important.
- Arrange for changes to become active on testnet before the main network.
- Think about laying a solid foundation, and then rolling out changes in stages. Baby steps instead of change-it-all-at-once.
If I was doing BIP16 over again and knew everything then I know now, I would propose rolling out support in three stages:
- Tighten up the "IsStandard" rule so it checked for 'standard' inputs. This would have prevented old miners from accidently putting invalid-under-new-rules transactions into their blocks.
- Add n-of-3 CHECKMULTISIGs as 'standard' transactions.
- Add BIP16 "pay to script hash" as 'standard' transactions.
- Completely non-controversial changes can happen relatively quickly (1-2 weeks).
BIP30 is a good example of a completely non-controversial change.
- Controversial changes can take months to deploy, even after there is rough consensus among the core developer, users, and miners.
BIP16 is a good example of a controversial change.
- People are error-prone; expect mistakes if you rely on people to set a configuration switch to support a new feature.
During the BIP16 adoption period some BIP16-supporting miners failed to update the switchover date for some of their nodes, and wasted time hashing on a minority chain. Several users also had the same problem because they were using an older 'release candidate' and hadn't upgraded.
They quickly realized and fixed their mistake, but it would be better if the switchover date/time was automatic based on the blockchain history. Code to do that could be written and tested once and then re-used for future upgrades.
- Blocks and transactions have version numbers; use them!
Late in the BIP12/BIP16/BIP17 debate somebody suggested that the new transactions should have a new version number. Which is a very good idea that "we" should have thought of at the beginning of the process. We should have used both block and transaction version numbers to support the new transaction form.
Block version numbers will be a better way for miners to indicate the rules they're using for validation, and to judge whether or not there is majority support for some new validation rule (BIP16 put the string "/P2SH/" in the coinbase transaction to judge support).
Old software that sees a majority of recent blocks in the chain with a version number that they do not understand should warn the user that their software is out of date and should be upgraded, but will otherwise ignore the version number and perform normal block and transaction validation.
Transaction version numbers will be a better way to specify exactly what rues should be used to validate a transaction; transactions with old version numbers will be validated under the old rules, transactions with new version numbers will always be validated under the new rules before being relayed or included in mined blocks. To avoid being in the minority fork of a block chain split, transactions with new version numbers appearing in blocks will only be validated using the new rules when a super-majority of the network (as expressed by block version numbers in the last N blocks) supports the new feature.
A transaction's version number is part of it's hash (transaction ID) and is part of the signature hash, so cannot be changed by an attacker. However, software using new transaction features will need to make sure that transactions have the correct version number before asking users to provide signatures or telling users that the transaction has been properly signed, etc.
There are some changes to core Bitcoin that should be made now that will make future upgrades smoother:
- Consider a transaction non-standard (do not accept it into the memory pool) if it has an unknown version number.
- Implement code to keep track of counts of block version numbers over the last 1,000 blocks.
- If 55% or more of the last 1000 blocks have an unknown version number, warn the user that they need to upgrade.
Here are some thumbnail sketches of how possible future upgrades could be handled in a way that minimizes risk and disruption.
Example: re-define OP_NOP1 to be OP_Q_CHECKSIGVERIFY, using a quantum-resistant digital signature algorithm.
- The opcode must be designed so that either the transaction fails validation OR it behaves exactly like a no-op. Any other design (e.g. one that consumes or leaves values on the stack) lets an attacker craft transactions that will be valid under the new rules but invalid for old miners/clients, causing a blockchain split.
- Transactions that use the new opcode are given a new version number.
- Miners running the new code produce blocks with a new version number, so support for the new feature can be measured.
- Older nodes will not relay or mine the new transactions, or transactions that spend the new transactions.
- New nodes will interpret OP_NOP1 as OP_Q_CHECKSIGVERIFY only if the transaction version number is correct, and 75+% of blocks created over the last 1000 blocks have an updated version number (51% of the last 100 blocks if on testnet).
Note that the "75% of the blocks created...." rule automatically handles old parts of the old block chain that might have invalid-under-the-new-rules transactions and also handles fluctuations in support; parts of the block-chain with less than 75% support will be validated under the old rules, parts with more than 75% support under the new. Users should not rely on the new feature until there is clear, consistent support.
Example: increasing MAX_BLOCK_SIZE (a 'hard' blockchain split change)
Increasing the maximum block size beyond the current 1MB per block (perhaps changing it to a floating limit based on a multiple of the median size of the last few hundred blocks) is a likely future change to accomodate more transactions per block. A new maximum block size rule might be rolled out by:
- New software creates blocks with a new block.version
- Allow greater-than-MAX_BLOCK_SIZE blocks if their version is the new block.version or greater and 100% of the last 1000 blocks are new blocks. (51% of the last 100 blocks if on testnet)
100% of the last 1000 blocks is a straw-man; the actual criteria would probably be different (maybe something like block.timestamp is after 1-Jan-2015 and 99% of the last 2000 blocks are new-version), since this change means the first valid greater-than-MAX_BLOCK_SIZE-block immediately kicks anybody running old software off the main block chain.