ReGenesis Plan
Summary of ReGenesis
Before we get into the planning of how ReGenesis can be implemented and rolled out, here is a quick summary of what it is. Although the idea is not new, it has been proposed in the current form here: https://ethresear.ch/t/regenesis-resetting-ethereum-to-reduce-the-burden-of-large-blockchain-and-state/7582
There is also an explainer post: https://medium.com/@mandrigin/regenesis-explained-97540f457807
ReGenesis logically splits the Ethereum state into two disjoint parts - active state and inactive state. There are ReGenesis “events” occuring regularly, at pre-scheduled times, but not very frequently, for example, every 1 million blocks, which is roughly 6 months. At a ReGenesis event, the active state is reset to just the state root hash, and the entire state is considered inactive. After that, until the next ReGenesis event, new state items get added to the active state, existing state items in the inactive state are moved from inactive into the active state (activated), and there, in the active state, they may change their values.
The only two ways to activate a piece of state, or add new data into the active state are transaction witness and block witness.
Transaction witness
Transactions get a new attribute, witness, which is a string of bytes encoding a collection of merkle multiproofs. Any part of the these multiproofs that are still valid against the active state, are added to the active state before transaction is executed. (TODO: Explain more how and why only part of these multiproofs will be valid most of the time, and how to find the valid parts).
Block witness
Blocks also get a new attribute, witness, which has the same encoding and capabilities as transaction witnesses, except that the valid parts of the block witness are added to the active state before executing any transaction in the block. One reason block witnesses are required is activation of miner’s coinbase address in the state. Another reason is ensuring a more gradual transition to the ReGenesis. For example, miner may choose to make an otherwise failing transaction succeed by identifying pieces of state missing from the transaction’s witnesses, and adding these pieces into the block witness (transaction repair).
Gas charge for witnesses
Depending on where we are in the process of introducing ReGenesis, the extra bytes in the transaction witnesses and block witnesses are either free of gas charge (initial stages), and later charged from transaction sender (for transaction witness ony).
Change in execution semantics
The semantics of executing transactions changes slightly. Now whenever transaction attempts to read a part of state, but that part of state is inactive, transaction reverts and gas spent up to that point is consumed.
Another change in the semantics is that if at the end of the block’s execution, the miner’s coinbase is not in the active state, the mining rewards are not given (reward and transaction fees are essentially burnt).
Change of Chain ID
After each ReGenesis event, the chainID changes to a new value. This ensures that any transactions that were created before the ReGenesis event and are still “in flight”, will not simply fail on chain (because they will most probably have insufficient witness). Instead, transactions for the new ReGenesis “epoch” need to be explicitely recreated. This would help transaction creators have more robust processes for the ReGenesis events.
Percieved benefits
The hope is that by re-setting the active state regularly, and requiring that only the active state is accessible by transactions, we can keep the active state small enough so that Ethereum nodes can store it in RAM, or at least on very low-latency devices. This, in turn, will ensure that the actual cost of state accessing operations (SLOAD, BALANCE, EXTCODEHASH, etc.) does not keep increasing. That means we can safely increase block gas limit to much higher values that we have today.
ReGenesis makes it more explicit at which point any node can sync “from scratch”. This means that it will become a norm to have an implementation that does not implement all the historical rules for all Ethereum forks. Instead, only behaviour since the last ReGenesis is a must, and all historical rules can be moved into specialised historical tracing tools.
Transactions will become larger because they will need to include witnesses. We expect this extra weight to be the highest right after a ReGenesis event, when transactions will need to activate the parts of the state they are using. However, active state can saturate to some degree, and more and more transactions will not need to include any witnesses, or very small witnesses.
Downsides for some stakeholders
Altough the burden on the core Ethereum infrastructure is likely to decrease, the burden on transaction-sending intfrastructure will increase. In order to compose a successful transaction, one would potentially need access not just to active state, but also to inactive state. In order to assist with this task, we would like network participants to create state snapshot in suitable formats and distribute them over BitTorrent (and perhaps other networks). In the same way, sets of historical blocks and receipts will be distributed for the purpose of retracing past transactions.
Discussion of benefits vs downsides
ReGenesis changes the status quo in that more aggressive optimisations will need to be done on the boundaries of the Ethereum network (where transactions are being created and injected), rather than in the core (critical infrastructure, “center” of the network). It may be more fair because boundaries are usually the places where the value is captured. Also, there is more diversity and decentralisation on the boundaries (there are more users than critical node operators). Failure to optimise on the boundaries does not represent systemic risk (i.e. it does not bring down the entire system). In the status quo, failure to optimise (in the core of the network) does represent systemic risk, and can potentially bring the entire system down.
Soft fork and hard fork
Broadly, the proposed transition from where we are to ReGeneis is via a soft-fork, followed by a hard fork. After the soft-fork, most of the benefits for the core infrastructure can be made tangible, at the cost of running some amount of altruistic (or sponsored) nodes. Following hard-fork is required to bring about the extra costs and downsides, therefore removing the reliance on the altruistic or sponsored nodes.
Soft forks can be seen as extra restrictions on the validity of transactions and blocks. Some blocks and transactions that are valid before a soft fork, are treated as invalid under the soft fork. Soft forks have been used in Bitcoin (most famously to activate SegWit), but were deemed to be unworkable on Ethereum after an attempt to do one to restrict the transactions to the DAO in 2016: https://hackingdistributed.com/2016/07/05/eth-is-more-resilient-to-censorship/
The main reason soft forks are deemed to be unworkable on Ethereum is that transactions can be well formed, and funded, and thus propagating through the transaction pool, but then hit the miners, where they would need to be fully executed only to find out that they are invalid.
In order to overcome this challenge, the plan is to gradually create two auxillary networks:
- Transaction pool with ReGenesis witnesses
- Pre-consensus network
Transaction pool with ReGenesis witnesses
This transaction pool can be created as a devp2p subprotocol, or as a completely new network. It will serve the same purpose as the the transaction pool today, but with some extra restrictions and extra capabilities. Any transaction message relayed on this network will need to include witness information. This witness needs to have a root at any state root between the latest ReGenesis event and the latest current block. This witness needs to be sufficient to verify sender’s balance, nonce, and also fully execute the transaction without having any access to the Ethereum state. If there is no witness, or it does not satisfy the requirements, the transaction is deemed invalid and gets removed from the pool. There could be some extra temporary penalty logic to prevent the same account regularly creating invalid transactions and waste tx pool network’s resources. At any point in time, the witness of a transaction can be updated by anyone, to include more recent information and rely on a more recent state root (but not the other direction, updates to more distant root are not permitted), as long the update does not remove previously included items. This update mechanism is used to knock out the transactions that have invalid nonce because the same sender had some other transactions included into the blocks, or insufficient funds because they have been spent by the intervining transactions.
There are few benefits of this network:
- Most of the verifying and relaying activities can be performed without having Ethereum state, which simplifies development of more diverse independent implementations (of standalone tx pool nodes), and faster innovations.
- Since transactions are getting executed by the pool nodes (albeight perhaps on an older version of the state), a lot of simple DOS attacks can be absorbed and filtered out by this network - this can be a good incentive for miners to eventually source their transactions from it.
The transaction pool network can be bootstrapped by creating network bridge nodes. These bridge nodes connect to both eth network and tx pool network, and relay transaction from one to another. If the idea proves successful, transaction injection will gradually move from eth network, and wallets, exchanges, and dapp backends will start injecting transactions directly into the tx pool network.
Pre-consensus network
It is the main mechanism for the enforcement of the soft fork. Here, Ethereum nodes will run a variant of Avalanche protocol to quickly agree on the composition of the blocks to be mined by the miners. Blocks agreed by the pre-consensus network will not include transactions that would, under ReGenesis rules, have to be accompanied by a very large witness, so that either the sending account would not have enough funds to pay for it, or the block gas limit would not allow it. This is similar to what the transaction pool network is doing, but while transaction pool provides only coarse-grained enforcement, the pre-consensus network provides a fine-grained enforcement, because transactions are evaluated in the context of being run in a proper block. Once a block is prepared and agreed by the pre-consensus network, it is presented in a form that is useful for the miners, with most of the calculation and hashing done, so that a miner needs to simply add the coinbase and finalise the state root calculation.
The reason pre-consensus network can be run more efficiently than the main network, is that it will operate in ReGenesis mode, having the active state in RAM, and therefore limiting the actual cost of state accessing operations. Nodes in the pre-consensus network will rely on the transaction pool network to append witnesses to the transactions and filter out most of the transactions that would fail due to the insufficient witness, or very large witness.
Snapshots
In order to assist transaction injecting nodes with performing their functions, there will be a regular creation and distribution of various snapshots via BitTorrent:
- Header files. Each header file will contain headers for some standardiased range of blocks. For example,
[1;11m], (11m;12m], (12m;13m], etc.
- Block bodies files. Same as header files but containing block bodies.
- Receipt files. Similar to the two above.
- State files. Each state snapshot contains the entire state at block number when ReGenesis event happened. Since such events do not happen very frequently, there will not be many files, and therefore we can ensure a very high replication factor.
The content addresses of these files will be hard-coded in the implementation, because we can expect to make at least 1 code release during 1 ReGenesis cycle.
Thin blocks
One of the main critiques of Stateless Ethereum project so far was the potentially large block witnesses. With ReGenesis, the expectation is that at least for some time after each ReGenesis event, the blocks will be large due to all transactions in them having to carry witnesses for the state they are accessing. In order to deal with this problem, we can introduce a concept of “thin blocks”. The idea is that instead of having every block body include the entire transaction with its witness, we will have block bodies include only the list of transaction IDs, and block witness (which is much smaller for ReGeneis, since it only need to ensure coinbase is in the active state). Transactions can be obtained via transaction pool, or if they are gone, full size blocks can be downloaded from peers on demand (but we assume that most peers who need to receive blocks fast, will be able to reconstruct them from the list of transaction IDs and block witnesses).
Pre-consensus and Sybil attacks
We can construct the rules of pre-consensus such that:
- Fuller blocks (by amount of gas spent) are preferred to emptier blocks
- Blocks that include transactions according to “high gas price to low gas price” rule are preferred to the blocks that exclude certain transactions (censorship)
In order to check (2) we would need to use observations from the transaction pool network. With such rules, we can end up with the small set of candidate blocks such that we are ambivalent about which block should win in the pre-consensus, because the rules above ensure that they are all good. However, an adversary performing a Sybil attack, can make pre-consensus stall, and never make any decision. To overcome this issue, the “Snowglobe” proposal for Bitcoin Cash suggested using “coin age” to resist Sybil attacks: https://github.com/tyler-smith/snowglobe/blob/master/spec/snowglobe.md#sybil-resistance-via-coin-age
Coin age can be very easily observe in UTXO blockchains, but in Ethereum we might need to create a “coin age” contract to emulate that. In such contract, anyone can deposit some ETH, and register a public key that would be used for verifying handshake challenges when nodes join pre-consensus. Withdrawal can happen at any time, but it immediately invalidates the eligibility of the registered public key.
Even thinner blocks with pre-consensus
If pre-consensus is achieved for a block, and miner followed it, and produced the pre-agreed block with its coinbase and its mining nonce, then such block can be propagated to the peers using just 60 bytes of information (20 bytes for coinbase address, 8 bytes for the mining nonce, and 32 bytes for extraData). This is almost as small as a block hash. Currently, go-ethereum uses this logic to propagate and announce new blocks. If it has N peers, it will randomly select sqrt(N) peers, and propagate the new block to them (with full body). For the rest of the peers, the new block is just announced (only hash of the block is sent). If the mined block followed pre-consensus, then such block can be propagated to all peers, and reach all corners of the network faster. This creates a potential incentive for miners to obey the pre-consensus.
Rough list of tasks
So far these tasks are not arranged in a “topological” order, so it is mostly to give impression of may need to be done. This section is heavily biased to the turbo-geth and its architecture.
- Header files in BitTorrent
- Block bodies files in BitTorrent
- Receipt files in BitTorrent
- State files in BitTorrent
- Transaction injector for the new transaction pool network
- Well-documented header download process via devp2p
- Well-documented bodies download process via devp2p
- Interface definitions between header download process and devp2p, between bodies download process and devp2p
- Standalone devp2p components
- Stage (for staged sync) for generating transaction witnesses
- Backtest the block sizes (large transactions due to witnesses) as if ReGenesis happened in the past
- Subprotocol for transaction pool for transaction with witnesses only
- Bridge between
eth and transaction pool networks
- Specialised tx pool software to work in the stateless mode
- DOS protection inside the new tx pool
- Pre-consensus network based on Avalanche
- ReGenesis powered preconsesus preparing blocks for mining
ReGenesis Plan
Summary of ReGenesis
Before we get into the planning of how ReGenesis can be implemented and rolled out, here is a quick summary of what it is. Although the idea is not new, it has been proposed in the current form here: https://ethresear.ch/t/regenesis-resetting-ethereum-to-reduce-the-burden-of-large-blockchain-and-state/7582
There is also an explainer post: https://medium.com/@mandrigin/regenesis-explained-97540f457807
ReGenesis logically splits the Ethereum state into two disjoint parts - active state and inactive state. There are ReGenesis “events” occuring regularly, at pre-scheduled times, but not very frequently, for example, every 1 million blocks, which is roughly 6 months. At a ReGenesis event, the active state is reset to just the state root hash, and the entire state is considered inactive. After that, until the next ReGenesis event, new state items get added to the active state, existing state items in the inactive state are moved from inactive into the active state (activated), and there, in the active state, they may change their values.
The only two ways to activate a piece of state, or add new data into the active state are transaction witness and block witness.
Transaction witness
Transactions get a new attribute, witness, which is a string of bytes encoding a collection of merkle multiproofs. Any part of the these multiproofs that are still valid against the active state, are added to the active state before transaction is executed. (TODO: Explain more how and why only part of these multiproofs will be valid most of the time, and how to find the valid parts).
Block witness
Blocks also get a new attribute, witness, which has the same encoding and capabilities as transaction witnesses, except that the valid parts of the block witness are added to the active state before executing any transaction in the block. One reason block witnesses are required is activation of miner’s coinbase address in the state. Another reason is ensuring a more gradual transition to the ReGenesis. For example, miner may choose to make an otherwise failing transaction succeed by identifying pieces of state missing from the transaction’s witnesses, and adding these pieces into the block witness (transaction repair).
Gas charge for witnesses
Depending on where we are in the process of introducing ReGenesis, the extra bytes in the transaction witnesses and block witnesses are either free of gas charge (initial stages), and later charged from transaction sender (for transaction witness ony).
Change in execution semantics
The semantics of executing transactions changes slightly. Now whenever transaction attempts to read a part of state, but that part of state is inactive, transaction reverts and gas spent up to that point is consumed.
Another change in the semantics is that if at the end of the block’s execution, the miner’s coinbase is not in the active state, the mining rewards are not given (reward and transaction fees are essentially burnt).
Change of Chain ID
After each ReGenesis event, the chainID changes to a new value. This ensures that any transactions that were created before the ReGenesis event and are still “in flight”, will not simply fail on chain (because they will most probably have insufficient witness). Instead, transactions for the new ReGenesis “epoch” need to be explicitely recreated. This would help transaction creators have more robust processes for the ReGenesis events.
Percieved benefits
The hope is that by re-setting the active state regularly, and requiring that only the active state is accessible by transactions, we can keep the active state small enough so that Ethereum nodes can store it in RAM, or at least on very low-latency devices. This, in turn, will ensure that the actual cost of state accessing operations (
SLOAD,BALANCE,EXTCODEHASH, etc.) does not keep increasing. That means we can safely increase block gas limit to much higher values that we have today.ReGenesis makes it more explicit at which point any node can sync “from scratch”. This means that it will become a norm to have an implementation that does not implement all the historical rules for all Ethereum forks. Instead, only behaviour since the last ReGenesis is a must, and all historical rules can be moved into specialised historical tracing tools.
Extra costs
Transactions will become larger because they will need to include witnesses. We expect this extra weight to be the highest right after a ReGenesis event, when transactions will need to activate the parts of the state they are using. However, active state can saturate to some degree, and more and more transactions will not need to include any witnesses, or very small witnesses.
Downsides for some stakeholders
Altough the burden on the core Ethereum infrastructure is likely to decrease, the burden on transaction-sending intfrastructure will increase. In order to compose a successful transaction, one would potentially need access not just to active state, but also to inactive state. In order to assist with this task, we would like network participants to create state snapshot in suitable formats and distribute them over BitTorrent (and perhaps other networks). In the same way, sets of historical blocks and receipts will be distributed for the purpose of retracing past transactions.
Discussion of benefits vs downsides
ReGenesis changes the status quo in that more aggressive optimisations will need to be done on the boundaries of the Ethereum network (where transactions are being created and injected), rather than in the core (critical infrastructure, “center” of the network). It may be more fair because boundaries are usually the places where the value is captured. Also, there is more diversity and decentralisation on the boundaries (there are more users than critical node operators). Failure to optimise on the boundaries does not represent systemic risk (i.e. it does not bring down the entire system). In the status quo, failure to optimise (in the core of the network) does represent systemic risk, and can potentially bring the entire system down.
Soft fork and hard fork
Broadly, the proposed transition from where we are to ReGeneis is via a soft-fork, followed by a hard fork. After the soft-fork, most of the benefits for the core infrastructure can be made tangible, at the cost of running some amount of altruistic (or sponsored) nodes. Following hard-fork is required to bring about the extra costs and downsides, therefore removing the reliance on the altruistic or sponsored nodes.
Soft forks can be seen as extra restrictions on the validity of transactions and blocks. Some blocks and transactions that are valid before a soft fork, are treated as invalid under the soft fork. Soft forks have been used in Bitcoin (most famously to activate SegWit), but were deemed to be unworkable on Ethereum after an attempt to do one to restrict the transactions to the DAO in 2016: https://hackingdistributed.com/2016/07/05/eth-is-more-resilient-to-censorship/
The main reason soft forks are deemed to be unworkable on Ethereum is that transactions can be well formed, and funded, and thus propagating through the transaction pool, but then hit the miners, where they would need to be fully executed only to find out that they are invalid.
In order to overcome this challenge, the plan is to gradually create two auxillary networks:
Transaction pool with ReGenesis witnesses
This transaction pool can be created as a devp2p subprotocol, or as a completely new network. It will serve the same purpose as the the transaction pool today, but with some extra restrictions and extra capabilities. Any transaction message relayed on this network will need to include witness information. This witness needs to have a root at any state root between the latest ReGenesis event and the latest current block. This witness needs to be sufficient to verify sender’s balance, nonce, and also fully execute the transaction without having any access to the Ethereum state. If there is no witness, or it does not satisfy the requirements, the transaction is deemed invalid and gets removed from the pool. There could be some extra temporary penalty logic to prevent the same account regularly creating invalid transactions and waste tx pool network’s resources. At any point in time, the witness of a transaction can be updated by anyone, to include more recent information and rely on a more recent state root (but not the other direction, updates to more distant root are not permitted), as long the update does not remove previously included items. This update mechanism is used to knock out the transactions that have invalid nonce because the same sender had some other transactions included into the blocks, or insufficient funds because they have been spent by the intervining transactions.
There are few benefits of this network:
The transaction pool network can be bootstrapped by creating network bridge nodes. These bridge nodes connect to both
ethnetwork and tx pool network, and relay transaction from one to another. If the idea proves successful, transaction injection will gradually move fromethnetwork, and wallets, exchanges, and dapp backends will start injecting transactions directly into the tx pool network.Pre-consensus network
It is the main mechanism for the enforcement of the soft fork. Here, Ethereum nodes will run a variant of Avalanche protocol to quickly agree on the composition of the blocks to be mined by the miners. Blocks agreed by the pre-consensus network will not include transactions that would, under ReGenesis rules, have to be accompanied by a very large witness, so that either the sending account would not have enough funds to pay for it, or the block gas limit would not allow it. This is similar to what the transaction pool network is doing, but while transaction pool provides only coarse-grained enforcement, the pre-consensus network provides a fine-grained enforcement, because transactions are evaluated in the context of being run in a proper block. Once a block is prepared and agreed by the pre-consensus network, it is presented in a form that is useful for the miners, with most of the calculation and hashing done, so that a miner needs to simply add the coinbase and finalise the state root calculation.
The reason pre-consensus network can be run more efficiently than the main network, is that it will operate in ReGenesis mode, having the active state in RAM, and therefore limiting the actual cost of state accessing operations. Nodes in the pre-consensus network will rely on the transaction pool network to append witnesses to the transactions and filter out most of the transactions that would fail due to the insufficient witness, or very large witness.
Snapshots
In order to assist transaction injecting nodes with performing their functions, there will be a regular creation and distribution of various snapshots via BitTorrent:
[1;11m],(11m;12m],(12m;13m], etc.The content addresses of these files will be hard-coded in the implementation, because we can expect to make at least 1 code release during 1 ReGenesis cycle.
Thin blocks
One of the main critiques of Stateless Ethereum project so far was the potentially large block witnesses. With ReGenesis, the expectation is that at least for some time after each ReGenesis event, the blocks will be large due to all transactions in them having to carry witnesses for the state they are accessing. In order to deal with this problem, we can introduce a concept of “thin blocks”. The idea is that instead of having every block body include the entire transaction with its witness, we will have block bodies include only the list of transaction IDs, and block witness (which is much smaller for ReGeneis, since it only need to ensure coinbase is in the active state). Transactions can be obtained via transaction pool, or if they are gone, full size blocks can be downloaded from peers on demand (but we assume that most peers who need to receive blocks fast, will be able to reconstruct them from the list of transaction IDs and block witnesses).
Pre-consensus and Sybil attacks
We can construct the rules of pre-consensus such that:
In order to check (2) we would need to use observations from the transaction pool network. With such rules, we can end up with the small set of candidate blocks such that we are ambivalent about which block should win in the pre-consensus, because the rules above ensure that they are all good. However, an adversary performing a Sybil attack, can make pre-consensus stall, and never make any decision. To overcome this issue, the “Snowglobe” proposal for Bitcoin Cash suggested using “coin age” to resist Sybil attacks: https://github.com/tyler-smith/snowglobe/blob/master/spec/snowglobe.md#sybil-resistance-via-coin-age
Coin age can be very easily observe in UTXO blockchains, but in Ethereum we might need to create a “coin age” contract to emulate that. In such contract, anyone can deposit some ETH, and register a public key that would be used for verifying handshake challenges when nodes join pre-consensus. Withdrawal can happen at any time, but it immediately invalidates the eligibility of the registered public key.
Even thinner blocks with pre-consensus
If pre-consensus is achieved for a block, and miner followed it, and produced the pre-agreed block with its coinbase and its mining nonce, then such block can be propagated to the peers using just 60 bytes of information (20 bytes for coinbase address, 8 bytes for the mining nonce, and 32 bytes for
extraData). This is almost as small as a block hash. Currently, go-ethereum uses this logic to propagate and announce new blocks. If it hasNpeers, it will randomly selectsqrt(N)peers, and propagate the new block to them (with full body). For the rest of the peers, the new block is just announced (only hash of the block is sent). If the mined block followed pre-consensus, then such block can be propagated to all peers, and reach all corners of the network faster. This creates a potential incentive for miners to obey the pre-consensus.Rough list of tasks
So far these tasks are not arranged in a “topological” order, so it is mostly to give impression of may need to be done. This section is heavily biased to the turbo-geth and its architecture.
ethand transaction pool networks