In part 1, we define agents, their contributions to the Ethereum system, and their challenges. This part deals with causes for these challenges, and the solution that have been proposed for these causes.
Causes
We try to understand the main causes of each challenge. Where it adds to understanding, we talk about sub-causes
Solutions
Proposed solutions should be targeted at the causes but may have side-effects (some solutions may be causes for other challenges, though perhaps less critical than the ones they are trying to address). These side effects are shown as dotted lines.
Summary of challenges
Here we summarise the challenges we have identified in the part 1.
Long time to sync a new node
Cost of storage devices
High internet traffic
Complex DevOps to run nodes
Conflict between updating rules of Ethereum and maintaining backwards compatibility
Conflict between updating the rules of Ethereum and improving client implementations
Difficulty of writing new implementations
Limited transaction throughput
Assessing safety of transactions
Estimating cost of a transaction
Description of challenges
Long time to sync a new node
When a new computer joins the Ethereum network, it needs to obtain the current state before it can start processing arbitrary transactions. The current state can be obtained in two main ways.
Firstly, the current state can be computed from the Genesis block, by applying (executing all transactions within) all the blocks in sequence. This method is now computationally hard due to the number of blocks (and transactions in them) to apply, and due to the size of the state.
Secondly, the current state can be downloaded from the peers that already computed (or downloaded) it. This method is not as hard computationally, but it requires a lot of communication with the peers, due to the size and the structure of the state.
Cost of storage devices
One can divide the persistent data managed by an Ethereum client into two main categories. First category is the data that rarely changes after being written. This category includes the content of downloaded block headers, block bodies, transaction receipts. Since chain reorganisations are relatively rare and “shallow” (meaning that not many blocks are getting reverted), one can store these data in an append-only form. This makes it feasible to store such data in a lower-cost devices such as HDD (Hard Disk Drives), or even tape (which is the cheapest storage device at the moment of writing). In fact, the “Freezer” implementation in go-ethereum 1.9 is based on these observations.
Second category is the data that is expected to change constantly. This includes mostly the active state of Ethereum, but also some indices for the append-only data. The Ethereum state is special in this category, because not only it constantly changes, but practically any part of it needs to be available for access at a short notice. To achieve maximum performance, one would choose storage devices with lower latency (SSD instead of HDD), and/or with a higher capacity for parallel reads (NVM instead of SSD). As the state grows, cost of these devices may grow super-linearly.
High internet traffic
Ethereum nodes constantly communicate with their peers in the network. In any peer-to-peer distribution network, being a “fair” participant requires not only satisfy one’s own needs for data but also provide data for the others. Depending on how many new nodes are joining the network, and what is current transaction activity, the internet traffic usage can be unpredictable.
Complex operations to run a node
The art of operations is how to run the infrastructure in a way that causes the least disruption, is cost efficient, and satisfy the functional requirements of the individual, business, or any organisation owning that infrastructure. Most of the complexities of running Ethereum nodes are usually related to the management of persistent data. Here are some operations that can be challenging, considering the amount of data and its dynamic character:
creating replicas of data to bootstrap new nodes
maintaining backups
pruning old histories
recovering from database corruptions
upgrading to new database formats
representing data in a format convenient for analysis
Conflict between updating rules of Ethereum and maintaining backwards compatibility
Often, the rules of Ethereum need to be updated because:
Implement a missing but very useful feature
Fix a flaw in the original design
Adjust to a change in the technological landscape
Smart contracts that have been deployed using old rules, may come to rely on those rules, explicitly or implicitly. When such contracts are still in frequent use, the challenge is to balance their need to exist in their original form (often, code improvements and redeployments solve the issue), and the need to update the rules.
Conflict between updating rules of Ethereum and improving client implementations
For reasons mentioned earlier, it is often necessary to update rules of Ethereum. When this happens, there is mandatory work for every Ethereum client implementation team who want their implementation to include the support for the update and stay relevant in the ecosystem. This work may distract from other, equally important work on constantly improving and optimising the implementations to keep up with the growing demands of the system. The balance between these two types of work can be difficult to strike. Ideally, there should be no conflict and the two efforts should be aligned.
Difficulty of writing new implementations
Sice Ethereum’s launch in August 2015, many teams have attempted to produce working implementation of Ethereum. Initially, there was a great variety of such projects. However, as the system’s usage grew, most implementations became non-viable or too difficult to maintain, although the number of rule changes is still relatively modest.
Limited transaction throughput
For Dapp developers, it is important to understand limitations of the Ethereum as a platform. One of the main limitation is the transaction throughput. Not only it puts limits on how many users may interact with an application, but it also puts popular application in competition with each other, since this limitation is global, and not per-application. It is currently believed that the maximum theorerical throughput of Ethereum is around 50 transactions per second (assuming simple Ether transferring transaction, 10M block gas limit, and 15 seconds block time). Applications that require more throughput, usually try to utilise so-called “Layer 2” solutions, however, these ase still not very mature at present, and may require lifting other limits (for example, maximum allowed transaction size, or maximum allowed size of contract byte code).
Assessing safety of transactions
For simple Ether transferring transaction, safety usually comes down to how well the private key is protected, and how secure is the generation of the digital signature (i.e. that it does not leak information about the private key by generating bad random numbers).
For transactions sent to smart contracts, safety assessment is harder, because the code of the smart contract may trigger undesirable actions if it is flawed or misused, intentionally or not. Often, Dapp developers attempt to assure the users of the safety of their contracts by conduction or commissioning security audits, offering bug bounties, or attempting to formally prove some safety properties.
It is believed that design of the execution engine (EVM in case of Ethereum) can be done in a way to make security analysis and formal verfication of smart contracts significantly easier and cheaper.
Another important consideration for safety is the probability of a transaction being reverted due to the chain reorganisations. Unintentional chain reorganisations happen all the time, but their depth is usually quite low. Therefore, the rule of “wait X number of blocks and consider transaction final” works quite well for dealing with unintentional reorganisations. Intentional reorganisations are often referred to as “51% attacks”, though the term may be inaccurate. When dealing with transactions of large value, the safety assessment need to include the risk of intentional reorganisations.
Estimating cost of a transaction
Gas price is a means of auctioning the limited space within the Ethereum blocks to the transaction senders. Such auctons work well when the space in the blocks is not fully utilised. However, when the blocks are becoming consistently full, higher gas price volatility is observed, and the transaction senders tend to overpay for gas, realising that they are competing with other senders. High volatility of gas prices makes the estimation of correct price to pay more complex.
Prioritising of challenges
Although any prioritisation of challenges would appear subjective, the approach is to compare the impact of challenges becomes overwhelming. For example, if the challenge Long time to sync a new node becomes overwhelming, and the sync time keeps growing, we can predict that in the future, the network will become difficult or impossible to join for the new operators. Although this will not immediately cause the system to fall, it will make it less resilient with some node operators disappearing.
Long time to sync a new node
It this challenge becomes overwhelming, and the sync time keeps growing, we can predict that in the future, the network will become difficult or impossible to join for the new operators. Although this will not immediately cause the system to fall, it will make it less resilient with some node operators disappearing. Even insentivising such node operators is unlikely to stop this tendency.
Cost of storage devices
If this challenge becomes overwhelming, and the cost of storage devices keeps growing, the number of node operators may reduce, unless there is a way of compensating node operators.
High internet traffic
If this challenge becomes overwhelming, and the demands of internet traffic grows, it is unclear what consequences there will be, because the cost of internet traffic appears to be insignificant compared to other costs involved in running an Ethereum node.
Complex DevOps to run a node
If this challenge becomes overwhelming, and the running an Ethereum node becomes operationally challenging, it is not clear what consequences will be. There are already mitigations to this challenge, in the form of pre-packages solutions to run node (like DappNode), service providers who specialise in hosting Ethereum node, and some cloud operators potentially offering Ethereum nodes “as a service”.
Conflict between updating rules of Ethereum and maintaining backwards compatibility
If this challenge becomes overwheleming, there are at least two possible scenarios:
Insufficient change of rules
Too many changes breaking backwards compatibility
Both of these require further analysis. Possible consequences of insufficient change incude:
eventual imbalance of gas schedule and DOS attacks on the system
inability to respond to highly desired feature requests
inability to fix design flaws and allow performance breakthroughs
Possible consequences of too many changes breaking backwards compatibility include:
Some popular contracts become obsolete or require redesign and redeployment
Conflict between updating rules of Ethereum and improving client implementations
If this challenge becomes overwhelming, there are at least two scenarios:
Client implementations struggle to keep up with the growth of the system, while most of the time is spent on updating the rules.
Updating rules slows down while core developers are busy with optimisations.
In the first scenario, one potential consequence is the reduction of the block gas limit to protect the implementations from breaking. After that, either existing implementations catch up, new implementation appears, or the system fails.
In the second scenario, the downside is a prolonged “feature freeze”.
Difficulty of writing new implementations
If this challenge becomes overwhelming, the consequence
is the reduction of number of viable implementations, and increased reliance on the resposible teams for the decision making about the rules of Ethereum.
Limited transaction throughput
If this challenge becomes overwhelming, that can mean one of two situations:
Transaction throughput starts to decline
Transaction throughput stays the same despite an expectation that it will increase.
First situation is more serious, and it would mean the reduction in the block gas limit as response to DOS attack or low performance of client implementations.
Second situation is less serious, but lead to “disappointment” in Ethereum as a smart contract platform. Currently, there seems to be an expectation that greatly increased transaction throughput will come from Ethereum 2, and Ethereum 1 can at best deliver x10 improvement. As long as such expectation remains justified, the status quo appears to be sustainable.
Assessing safety of transactions
This challenge can become overwhelming in two ways:
Researchers in formal verification and security of smart contracts come to general conclusion that it does not make economic sense to try to advance this without fundamental change of the EVM design
Intentional reorganisations become possible and sometimes happen in practice
If the challenge becomes overwhelming in the first way, it would lead to potentially large use cases only being deployed on Ethereum for limited use (e.g. limitation no more than $10 million in a smart contract that was suggested straight after “The DAO incident”).
The second way is more immediately alarming, because it would affect even people, companies and organisation that do not utilise sophisticated smart contracts, but rely on simple Ether transfer transactions.
Estimating cost of a transaction
If this challenge becomes overwhelming, it would mean that the gas prices are consistenty very volatile and it is hard to estimate a required fee to ensure that the transactions get confirmed in timely manner. The worst affected would be people, companies, and organisations, whole operations depend on sending and confirming large number of transactions, for example, exchanges (for deposits and withdrawals, and for trading in case of decentralised exchanges), mining pools (for payouts), popular Dapps. When gas prices are volatile, it becomes more economically attractive to perform transaction front-running and use mechanisms like GasToken to “accumulate” gas at lower prices to release it at higher prices.
Ethereum 1x Definition (part 2)
Methodology (continuation from part 1)
In part 1, we define agents, their contributions to the Ethereum system, and their challenges. This part deals with causes for these challenges, and the solution that have been proposed for these causes.
Causes
We try to understand the main causes of each challenge. Where it adds to understanding, we talk about sub-causes
Solutions
Proposed solutions should be targeted at the causes but may have side-effects (some solutions may be causes for other challenges, though perhaps less critical than the ones they are trying to address). These side effects are shown as dotted lines.
Summary of challenges
Here we summarise the challenges we have identified in the part 1.
Description of challenges
Long time to sync a new node
When a new computer joins the Ethereum network, it needs to obtain the current state before it can start processing arbitrary transactions. The current state can be obtained in two main ways.
Firstly, the current state can be computed from the Genesis block, by applying (executing all transactions within) all the blocks in sequence. This method is now computationally hard due to the number of blocks (and transactions in them) to apply, and due to the size of the state.
Secondly, the current state can be downloaded from the peers that already computed (or downloaded) it. This method is not as hard computationally, but it requires a lot of communication with the peers, due to the size and the structure of the state.
Cost of storage devices
One can divide the persistent data managed by an Ethereum client into two main categories. First category is the data that rarely changes after being written. This category includes the content of downloaded block headers, block bodies, transaction receipts. Since chain reorganisations are relatively rare and “shallow” (meaning that not many blocks are getting reverted), one can store these data in an append-only form. This makes it feasible to store such data in a lower-cost devices such as HDD (Hard Disk Drives), or even tape (which is the cheapest storage device at the moment of writing). In fact, the “Freezer” implementation in go-ethereum 1.9 is based on these observations.
Second category is the data that is expected to change constantly. This includes mostly the active state of Ethereum, but also some indices for the append-only data. The Ethereum state is special in this category, because not only it constantly changes, but practically any part of it needs to be available for access at a short notice. To achieve maximum performance, one would choose storage devices with lower latency (SSD instead of HDD), and/or with a higher capacity for parallel reads (NVM instead of SSD). As the state grows, cost of these devices may grow super-linearly.
High internet traffic
Ethereum nodes constantly communicate with their peers in the network. In any peer-to-peer distribution network, being a “fair” participant requires not only satisfy one’s own needs for data but also provide data for the others. Depending on how many new nodes are joining the network, and what is current transaction activity, the internet traffic usage can be unpredictable.
Complex operations to run a node
The art of operations is how to run the infrastructure in a way that causes the least disruption, is cost efficient, and satisfy the functional requirements of the individual, business, or any organisation owning that infrastructure. Most of the complexities of running Ethereum nodes are usually related to the management of persistent data. Here are some operations that can be challenging, considering the amount of data and its dynamic character:
Conflict between updating rules of Ethereum and maintaining backwards compatibility
Often, the rules of Ethereum need to be updated because:
Smart contracts that have been deployed using old rules, may come to rely on those rules, explicitly or implicitly. When such contracts are still in frequent use, the challenge is to balance their need to exist in their original form (often, code improvements and redeployments solve the issue), and the need to update the rules.
Conflict between updating rules of Ethereum and improving client implementations
For reasons mentioned earlier, it is often necessary to update rules of Ethereum. When this happens, there is mandatory work for every Ethereum client implementation team who want their implementation to include the support for the update and stay relevant in the ecosystem. This work may distract from other, equally important work on constantly improving and optimising the implementations to keep up with the growing demands of the system. The balance between these two types of work can be difficult to strike. Ideally, there should be no conflict and the two efforts should be aligned.
Difficulty of writing new implementations
Sice Ethereum’s launch in August 2015, many teams have attempted to produce working implementation of Ethereum. Initially, there was a great variety of such projects. However, as the system’s usage grew, most implementations became non-viable or too difficult to maintain, although the number of rule changes is still relatively modest.
Limited transaction throughput
For Dapp developers, it is important to understand limitations of the Ethereum as a platform. One of the main limitation is the transaction throughput. Not only it puts limits on how many users may interact with an application, but it also puts popular application in competition with each other, since this limitation is global, and not per-application. It is currently believed that the maximum theorerical throughput of Ethereum is around 50 transactions per second (assuming simple Ether transferring transaction, 10M block gas limit, and 15 seconds block time). Applications that require more throughput, usually try to utilise so-called “Layer 2” solutions, however, these ase still not very mature at present, and may require lifting other limits (for example, maximum allowed transaction size, or maximum allowed size of contract byte code).
Assessing safety of transactions
For simple Ether transferring transaction, safety usually comes down to how well the private key is protected, and how secure is the generation of the digital signature (i.e. that it does not leak information about the private key by generating bad random numbers).
For transactions sent to smart contracts, safety assessment is harder, because the code of the smart contract may trigger undesirable actions if it is flawed or misused, intentionally or not. Often, Dapp developers attempt to assure the users of the safety of their contracts by conduction or commissioning security audits, offering bug bounties, or attempting to formally prove some safety properties.
It is believed that design of the execution engine (EVM in case of Ethereum) can be done in a way to make security analysis and formal verfication of smart contracts significantly easier and cheaper.
Another important consideration for safety is the probability of a transaction being reverted due to the chain reorganisations. Unintentional chain reorganisations happen all the time, but their depth is usually quite low. Therefore, the rule of “wait X number of blocks and consider transaction final” works quite well for dealing with unintentional reorganisations. Intentional reorganisations are often referred to as “51% attacks”, though the term may be inaccurate. When dealing with transactions of large value, the safety assessment need to include the risk of intentional reorganisations.
Estimating cost of a transaction
Gas price is a means of auctioning the limited space within the Ethereum blocks to the transaction senders. Such auctons work well when the space in the blocks is not fully utilised. However, when the blocks are becoming consistently full, higher gas price volatility is observed, and the transaction senders tend to overpay for gas, realising that they are competing with other senders. High volatility of gas prices makes the estimation of correct price to pay more complex.
Prioritising of challenges
Although any prioritisation of challenges would appear subjective, the approach is to compare the impact of challenges becomes overwhelming. For example, if the challenge Long time to sync a new node becomes overwhelming, and the sync time keeps growing, we can predict that in the future, the network will become difficult or impossible to join for the new operators. Although this will not immediately cause the system to fall, it will make it less resilient with some node operators disappearing.
Long time to sync a new node
It this challenge becomes overwhelming, and the sync time keeps growing, we can predict that in the future, the network will become difficult or impossible to join for the new operators. Although this will not immediately cause the system to fall, it will make it less resilient with some node operators disappearing. Even insentivising such node operators is unlikely to stop this tendency.
Cost of storage devices
If this challenge becomes overwhelming, and the cost of storage devices keeps growing, the number of node operators may reduce, unless there is a way of compensating node operators.
High internet traffic
If this challenge becomes overwhelming, and the demands of internet traffic grows, it is unclear what consequences there will be, because the cost of internet traffic appears to be insignificant compared to other costs involved in running an Ethereum node.
Complex DevOps to run a node
If this challenge becomes overwhelming, and the running an Ethereum node becomes operationally challenging, it is not clear what consequences will be. There are already mitigations to this challenge, in the form of pre-packages solutions to run node (like DappNode), service providers who specialise in hosting Ethereum node, and some cloud operators potentially offering Ethereum nodes “as a service”.
Conflict between updating rules of Ethereum and maintaining backwards compatibility
If this challenge becomes overwheleming, there are at least two possible scenarios:
Both of these require further analysis. Possible consequences of insufficient change incude:
Possible consequences of too many changes breaking backwards compatibility include:
Conflict between updating rules of Ethereum and improving client implementations
If this challenge becomes overwhelming, there are at least two scenarios:
In the first scenario, one potential consequence is the reduction of the block gas limit to protect the implementations from breaking. After that, either existing implementations catch up, new implementation appears, or the system fails.
In the second scenario, the downside is a prolonged “feature freeze”.
Difficulty of writing new implementations
If this challenge becomes overwhelming, the consequence
is the reduction of number of viable implementations, and increased reliance on the resposible teams for the decision making about the rules of Ethereum.
Limited transaction throughput
If this challenge becomes overwhelming, that can mean one of two situations:
First situation is more serious, and it would mean the reduction in the block gas limit as response to DOS attack or low performance of client implementations.
Second situation is less serious, but lead to “disappointment” in Ethereum as a smart contract platform. Currently, there seems to be an expectation that greatly increased transaction throughput will come from Ethereum 2, and Ethereum 1 can at best deliver x10 improvement. As long as such expectation remains justified, the status quo appears to be sustainable.
Assessing safety of transactions
This challenge can become overwhelming in two ways:
If the challenge becomes overwhelming in the first way, it would lead to potentially large use cases only being deployed on Ethereum for limited use (e.g. limitation no more than $10 million in a smart contract that was suggested straight after “The DAO incident”).
The second way is more immediately alarming, because it would affect even people, companies and organisation that do not utilise sophisticated smart contracts, but rely on simple Ether transfer transactions.
Estimating cost of a transaction
If this challenge becomes overwhelming, it would mean that the gas prices are consistenty very volatile and it is hard to estimate a required fee to ensure that the transactions get confirmed in timely manner. The worst affected would be people, companies, and organisations, whole operations depend on sending and confirming large number of transactions, for example, exchanges (for deposits and withdrawals, and for trading in case of decentralised exchanges), mining pools (for payouts), popular Dapps. When gas prices are volatile, it becomes more economically attractive to perform transaction front-running and use mechanisms like GasToken to “accumulate” gas at lower prices to release it at higher prices.