by Piotr Rudnik, Data Engineer

Introduction

During our previous lesson, we took a deep dive into Ethereum's architecture and investigated the art of data decoding. This article will provide a deeper understanding of how Ethereum stores data in its persistent storage by examining storage memory layouts. Understanding these layouts is essential for optimizing smart contract design and ensuring efficient data management within the Ethereum network.

We will also show how Token Flow’s storage diff datasets make this easier to analyze.

How Does Ethereum Store Data in Storage?

To understand Ethereum's data storage, we must first familiarize ourselves with the concept of a “slot”. A slot represents the smallest unit of memory that the Ethereum Virtual Machine (EVM) can allocate, with a size always equal to 32 bytes. This concept resembles blocks in traditional file systems like ext4 or NTFS, which aim to divide and manage larger memory spaces efficiently.

Consider this example :

contract TestContract{
	int a;
	bool b;
	int c;
}

The storage memory layout for TestContract looks like this :

Let’s examine another example with three contracts - A, B, and C - which are defined in files A.sol, B.sol, and C.sol, respectively.

File a.sol :

import "./B.sol";
import "./C.sol";

struct TestStruct{} 

contract A is B, C {   }

File b.sol :

import { TestStruct } from \\\\"./C.sol\\\\";

contract B {   
	bool[2] public arrBools;
  bytes16 public varBytes2; 
}

File c.sol :

struct TestStruct {uint256 value;  address owner;}

contract C {   
	bytes16 public varBytes1; 
	TestStruct[] public arrStructs;  
}

Let's explore how these contracts are organized in memory by following these steps:

1. Resolve imports:

File A.sol imports files B.sol and C.sol, granting access to all definitions stored there.
File B.sol utilizes a single named import from C.sol for the structure TestStruct.
File C.sol has no imports.

2. Resolve inheritance:

Contract A inherits from B and C, in that order.
Contracts B and C are root contracts with no parents.

3. Gather variables:

As contract B comes first in the inheritance order, its variables are: arrBools and varBytes2.
Next is contract C with variables arrStructs and varBytes1.
Finally, contract A has no variables.

4. Place variables in slots:

The first variable, arrBools, is a static array of bools with a size of 2. Static arrays always occupy an entire slot, so it lands in slot 1 with no space left.
The second variable, varBytes2, has a size of 16. It starts in slot 1, takes 16 bytes, and leaves 16 bytes free.
Next is arrStructs, which is a dynamic array of TestStructs. TestStruct has a uint256 and an address inside, totaling 52 bytes. However, dynamic arrays always take up one slot. They do not store data itself, but an address to memory where data is allocated. Hence, this address is placed in slot 2.
Finally, varBytes1 occupies 16 bytes in slot 3.

This is the final storage layout:

We can draw a few observations from this layout :

Although file A.sol has an empty definition of TestStruct that is not used anywhere, we must keep in mind that each file has its own import table. Thus, we cannot arbitrarily select a definition with the same name. We must trace it back, file by file, to the one used by this specific file. This is not uncommon in production contracts spanning tens or hundreds of files where external libraries are embedded in the code.
It is apparent that we can save storage space. In contract C, we can swap variables arrStructs and varBytes1, resulting in varBytes1 and varBytes2 occupying one slot, saving us a slot.
Assuming that the code does not require this, varBools can be split into two normal bools rather than an array. This would not save space in contract A since the next variable takes up 32 bytes, but it could save space if contract C is inherited by other contracts or is used standalone.

The challenging aspect is determining the correct order of variables within the storage layout. Numerous rules and guidelines must be followed, such as structure nesting or handling dynamic/static arrays. However, the most crucial mechanism to comprehend is C3 Linearization and the Inheritance Mechanism.

Inheritance is a fundamental concept in object-oriented programming that allows for the creation of new classes based on existing ones. Ethereum's Solidity language employs the C3 Linearization algorithm, also known as Method Resolution Order (MRO), to manage multiple inheritance and resolve potential conflicts.

C3 Linearization ensures that the inheritance hierarchy is consistently linearized, simplifying reasoning about the order in which base contracts are initialized and functions are called. This mechanism guarantees that all base contracts are initialized in the correct order, particularly important when handling multiple inheritance scenarios.

The algorithm processes the inheritance graph from left to right, adhering to the rules of C3 Linearization. It enforces a strict order in which base contracts are initialized, ensuring a consistent execution flow is maintained.

Consider this example :

contract A{}
contract D is A{}
contract C is A{}
contract D is B, C {}

In this case, the C3 Linearization algorithm determines the inheritance order as follows: D, B, C, A. This means that contract A's constructor is called first, followed by constructors of contracts B and C, and finally, contract D's constructor.

Applications of Storage Memory Layouts

Storage memory layouts are crucial in optimizing smart contract development and analysis on the Ethereum network. By understanding how data is stored and organized, developers and analysts can enhance contract efficiency, troubleshoot issues, and analyze contract behavior more effectively.

Understanding the storage layout is vital when upgrading smart contracts to ensure seamless transitions without disrupting existing data. Developers can use their knowledge of storage layouts to design upgradeable contracts, preserving stored data integrity while implementing new features or improvements.

Memory layouts play a significant role in optimizing gas usage and storage costs within the Ethereum ecosystem. By understanding how data is organized and stored in smart contracts, users can implement efficient data structures and techniques that reduce the overall storage footprint. This, in turn, leads to decreased gas costs associated with contract execution and data manipulation. Moreover, a well-organized storage layout helps minimize the risk of unintended contract behavior, potentially resulting in excessive gas consumption or security vulnerabilities.

But there is one big problem with storage information - it is provided by nodes in a form that is not very useful. If you know the exact memory location, you can always ask for its value at any block using web3 getStorageAt function. But conventional understanding is that there is no way to get the storage changes within a block (if it was changed many times you will only see values before and after the whole block). And, what’s even more problematic - many memory locations are calculated as hashes of slot and key (e.g. for hashmaps). Which means that you need to know the value of the key in advance and you cannot just iterate over hashmaps.

However, Token Flow provides datasets with the full history of storage diffs in pre-hashed form and within blocks, which means you can see every change and you can easily extract all the keys ever used to address any hashmap. Up until now this has not been accessible to developers and analysts, but with Token Flow’s datasets new possibilities of accessing and using this data become available.

Obtaining Storage Layouts

Obtaining a storage layout for a single contract is relatively simple using the Solidity compiler :

solc --abi MyContract.sol. -o build/
abi-dumper --abi-file build/MyContract.abi --storage-layout

However, this method does not work for all contracts, and you need to match the contract's compiler version.

Another option is manually determining storage layouts using pen and paper. This method has been employed in our development process several times, primarily as a sanity check for various edge cases.

In Token Flow, a vast amount of data is parsed from numerous contracts. Traditional options are neither scalable nor fast or easy to maintain. Thus, a custom contract parser was developed. This parser analyzes a contract's source code, resolves all dependencies and inheritance orders, and allows for fast and painless storage layout extraction. Each contract takes, on average, less than 1 second of CPU time to create a ready layout while maintaining compatibility with all Solidity versions.

As a side note, some Vyper contracts are also parsed by our system. Although there are important contracts written in Vyper, maintaining a stable setup for them has proven to be challenging.

This custom solution has made it possible to create a comprehensive database of all public contracts and their storage layouts, resulting in a unique and valuable dataset.

This dataset is used to decode the whole history of storage changes as described in the previous article.

How to get access to our pre-hashed diffs data

We will soon be releasing AnyTx, our new transaction decoder (and successor to EthTx) which includes diffs s well as events and calls. You can see a preview here.

For users who want to query the full history of storage diffs, we have a decoded dataset available for Ethereum, with other chains soon being released.

You can find more information in our documentation, and can keep updated on our future releases via discord, twitter and our mailing list (links below).

Follow us for news and announcements

You can follow us, sign up to our newsletter, and join the conversation at:

Web: tokenflow.live

Documentation: docs.tokenflow.live

Careers: careers.tokenflow.live

Twitter: @InsightsFlow

Discord: discord.gg

#EthereumDecoded lesson 2 - Understanding Ethereum storage memory layout