In part 1, we define agents, their contributions to the Ethereum system, and their challenges. This part deals with causes for these challenges, and the solution that have been proposed for these causes.
Causes
We try to understand the main causes of each challenge. Where it adds to understanding, we talk about sub-causes
Solutions
Proposed solutions should be targeted at the causes but may have side-effects (some solutions may be causes for other challenges, though perhaps less critical than the ones they are trying to address). These side effects are shown as dotted lines.
Summary of causes
Here we summarise the causes for the challenges. Most of them are technological, though some could be viewed as organisational.
Increasingly intricate data management that an Ethereum node needs to perform
Functional coupling on implementation level
Functional coupling on protocol level
Large semantic gap between EVM and many dapps
EVM design favours expressiveness over tractability
Transaction fee market behaves sub-optimally when blocks are at the block gas limit (which is currently almost always).
Possibility of long chain re-orgs.
Over-reliance of spontaneous contributions.
Description of causes
Large (and growing) state
Ethereum’s state is a data structure that needs to be implicitly constructed, stored and accessed in order to execute arbitrary transactions. This is because a transaction may theoretically access any item in the current state. The state size grew beyond the capacity of RAM (Random Access Memory) on average computers some time in 2017. After that point, RAM could only cache certain portions of the state, whereas the entire state resides on persistent storage devices. Any caching strategy apart from keeping a random portion of the accessible state, would be vulnerable to attack. Therefore, assuming that random caching strategy, the cache hit ratio would very close to the ratio of size of the cache to the size of the entire state. And cache miss would mean accessing devices with much higher latency.
Because of the further growth of the state, it became impractical to use HDD (Hard Disk Drives - storage devices with mechanically spinning disks) for storage of the state, due to the high latency of the access. Even SDD (Solid State Drives) are on the edge of being appropriate. Devices such as NVM (Non Volatile Memory) are now required to ensure good performance. However, such devices are still relatively expensive and their price proportional to capacity ($/Gb) is highly non linear after a certain point.
State size also places a significant burden on the new participants in the Ethereum network. Most popular way of joining the network at this moment is so-called “snapshot synchronisation”. It is the process in which the new-joiner downloads the entire state from the existing peers. The sheer size of the state puts a high demand on the bandwidth quality. Dealing with the network latencies requires sophisticated algorithms for downloading. And the ever-changing nature of the state (it keeps changing during the download) either puts snapshot sync at odds with state history pruning, or requires even more sophisticated algorithms for the downloading of the state.
Snapshot synchronisation (briefly described in the second ) can be implemented in a variety of ways. The dominant algorithm used at the moment is called “fast sync”. It has been designed in the circumstances when the state size was still reasonably small. The main advantage of this algorithm is its simplicity. However, implementations may also vary in efficiency. For example, they might be sensitive to network latencies (how much slower algorithm performs with an increase in latency), or require a lot of overhead traffic (how much more data algorithm transmits with an increase of state size).
Increasingly intricate data management that an Ethereum node needs to perform
The hardest part of a main-net capable implementation seems to be the data management. Ethereum node needs to transmit, process and store large amounts of data. To do so efficiently, it needs non-trivial techniques, like data caching strategies, partitioning data by usage type. Such techniques, however, are currently not considered in Ethereum specification documents or other literature.
Functional coupling on the implementation level
Example - persistent storage of state as a Particia tree, which couples logical structure needed to compute state root, and physical structure needed to store the state. Such coupling makes it harder to switch to binary Merkle tree for hashing the state.
Functional coupling on protocol level
In the design of Ethereum, there some crucial concepts with double or triple functions. First example: gas is used as a charge for resources, as well as means of restricting callbacks and recursion depth. Second example: nonce of accounts is used for both replay protection and as input for generating of the contract addresses. Third example (though this is a prevalent implementation choice rather than a requirement of the specification): Particia Merkle tree is used for both defining what the state root hash is, and for storing data. Functional coupling makes the design inflexible and causes issues when something about the concept needs to be changed. For example, changing gas cost of some operations, like reduction for SSTORE and increase for SLOAD clashes with the concept of call stipend, which is part of the restricting function of gas.
The “semantic gap” as it is often referenced in IT is the difference between high-level programming sets in various computer languages, and the simple computing instructions that microprocessors work with in machine language. This classic difference has compelled engineers and designers to look at different ways of mediating between high-level language and basic machine language.
In the past, engineers tried to bridge the semantic gap by making microprocessors more complex, as with Complex Instruction Set Computing (CISC) models. However, they found that it could be just as effective, if not more so, to design Reduced Instruction Set Computing (RISC) models. The philosophy is that microprocessors do not have to do complex work, but can break the high-level instructions down into simple steps. That resonates with how semantic programming is compiled or broken down into machine language. The semantic gap illustrates the difference between humans and computers and how they process data.
After EVM has been designed and implemented, it turned out that there are many use cases, specifically involving cryptographic primitives (e.g. hashing algorithms, digital signatures) whose implementation in EVM opcodes would consume prohibitive amounts of gas.
EVM design favours expressiveness over tractability
A virtual machine
Over-reliance on spontaneous contributions
By “spontaneous voluntary contributions” we understand contributions to core implementation by people who are not explicitly asked to do the work, but do it, because they found it interesting and/or important.
By “managed development” we understand development in the core implementations that are directed by some leadership, according to some implementation plan.
Both ways of development have their pros and cons. It seems that in the current circumstances we mainly rely on the spontaneous voluntary contributions and that seems to leave important gaps and technical debt.
Summary of solutions
At this moment, only solutions to the technological causes are listed and explained.
Finality gadget to link Ethereum 1x to the Beacon chain
Upgrade of the Proof Of Work algorithm (ProgPOW)
State management component (interface design)
Description of solutions
State rent
State rent is the only currently proposed solution to decreasing of the state size. All other solutions are concerned with how to deal with the large (and growing) state size more efficiently and prevent some of the most obvious failure modes.
Stateless clients
Stateless clients is one approach to improve performance of Ethereum client implementations while processing blocks of transactions. Specifically, it seeks to ease the increasing burden that is the state size. It does so by removing the need to download, or implicitly construct and maintain the state, for most of the participants in the Ethereum network. The requirement to access the state is removed by introducing another type of data packets (existing data packet types are, for example, blocks and transactions) to be gossipped around the p2p network. We call this data packets “block witnesses”. For each block we have one corresponding block witness. The two main properties that block witnesses have:
It is possible verify efficiently that the block witness is indeed constructed from the correct version of the Ethereum state.
Block witness has all the required information to make it possible to execute the corresponding block.
Ethereum 1x Definition (part 3)
Methodology (continuation from parts 1 and 2)
In part 1, we define agents, their contributions to the Ethereum system, and their challenges. This part deals with causes for these challenges, and the solution that have been proposed for these causes.
Causes
We try to understand the main causes of each challenge. Where it adds to understanding, we talk about sub-causes
Solutions
Proposed solutions should be targeted at the causes but may have side-effects (some solutions may be causes for other challenges, though perhaps less critical than the ones they are trying to address). These side effects are shown as dotted lines.
Summary of causes
Here we summarise the causes for the challenges. Most of them are technological, though some could be viewed as organisational.
Description of causes
Large (and growing) state
Ethereum’s state is a data structure that needs to be implicitly constructed, stored and accessed in order to execute arbitrary transactions. This is because a transaction may theoretically access any item in the current state. The state size grew beyond the capacity of RAM (Random Access Memory) on average computers some time in 2017. After that point, RAM could only cache certain portions of the state, whereas the entire state resides on persistent storage devices. Any caching strategy apart from keeping a random portion of the accessible state, would be vulnerable to attack. Therefore, assuming that random caching strategy, the cache hit ratio would very close to the ratio of size of the cache to the size of the entire state. And cache miss would mean accessing devices with much higher latency.
Because of the further growth of the state, it became impractical to use HDD (Hard Disk Drives - storage devices with mechanically spinning disks) for storage of the state, due to the high latency of the access. Even SDD (Solid State Drives) are on the edge of being appropriate. Devices such as NVM (Non Volatile Memory) are now required to ensure good performance. However, such devices are still relatively expensive and their price proportional to capacity ($/Gb) is highly non linear after a certain point.
State size also places a significant burden on the new participants in the Ethereum network. Most popular way of joining the network at this moment is so-called “snapshot synchronisation”. It is the process in which the new-joiner downloads the entire state from the existing peers. The sheer size of the state puts a high demand on the bandwidth quality. Dealing with the network latencies requires sophisticated algorithms for downloading. And the ever-changing nature of the state (it keeps changing during the download) either puts snapshot sync at odds with state history pruning, or requires even more sophisticated algorithms for the downloading of the state.
Snapshot sync algorithm implementation deficiencies
Snapshot synchronisation (briefly described in the second ) can be implemented in a variety of ways. The dominant algorithm used at the moment is called “fast sync”. It has been designed in the circumstances when the state size was still reasonably small. The main advantage of this algorithm is its simplicity. However, implementations may also vary in efficiency. For example, they might be sensitive to network latencies (how much slower algorithm performs with an increase in latency), or require a lot of overhead traffic (how much more data algorithm transmits with an increase of state size).
Increasingly intricate data management that an Ethereum node needs to perform
The hardest part of a main-net capable implementation seems to be the data management. Ethereum node needs to transmit, process and store large amounts of data. To do so efficiently, it needs non-trivial techniques, like data caching strategies, partitioning data by usage type. Such techniques, however, are currently not considered in Ethereum specification documents or other literature.
Functional coupling on the implementation level
Example - persistent storage of state as a Particia tree, which couples logical structure needed to compute state root, and physical structure needed to store the state. Such coupling makes it harder to switch to binary Merkle tree for hashing the state.
Functional coupling on protocol level
In the design of Ethereum, there some crucial concepts with double or triple functions. First example: gas is used as a charge for resources, as well as means of restricting callbacks and recursion depth. Second example: nonce of accounts is used for both replay protection and as input for generating of the contract addresses. Third example (though this is a prevalent implementation choice rather than a requirement of the specification): Particia Merkle tree is used for both defining what the state root hash is, and for storing data. Functional coupling makes the design inflexible and causes issues when something about the concept needs to be changed. For example, changing gas cost of some operations, like reduction for
SSTORE
and increase forSLOAD
clashes with the concept of call stipend, which is part of the restricting function of gas.Large semantic gap between EVM and many dapps
Definition of semantic gap from www.techopedia.com:
After EVM has been designed and implemented, it turned out that there are many use cases, specifically involving cryptographic primitives (e.g. hashing algorithms, digital signatures) whose implementation in EVM opcodes would consume prohibitive amounts of gas.
EVM design favours expressiveness over tractability
A virtual machine
Over-reliance on spontaneous contributions
By “spontaneous voluntary contributions” we understand contributions to core implementation by people who are not explicitly asked to do the work, but do it, because they found it interesting and/or important.
By “managed development” we understand development in the core implementations that are directed by some leadership, according to some implementation plan.
Both ways of development have their pros and cons. It seems that in the current circumstances we mainly rely on the spontaneous voluntary contributions and that seems to leave important gaps and technical debt.
Summary of solutions
At this moment, only solutions to the technological causes are listed and explained.
Description of solutions
State rent
State rent is the only currently proposed solution to decreasing of the state size. All other solutions are concerned with how to deal with the large (and growing) state size more efficiently and prevent some of the most obvious failure modes.
Stateless clients
Stateless clients is one approach to improve performance of Ethereum client implementations while processing blocks of transactions. Specifically, it seeks to ease the increasing burden that is the state size. It does so by removing the need to download, or implicitly construct and maintain the state, for most of the participants in the Ethereum network. The requirement to access the state is removed by introducing another type of data packets (existing data packet types are, for example, blocks and transactions) to be gossipped around the p2p network. We call this data packets “block witnesses”. For each block we have one corresponding block witness. The two main properties that block witnesses have:
More details can be found here: https://medium.com/@akhounov/data-from-the-ethereum-stateless-prototype-8c69479c8abc