by Karol Chojnowski, Senior Data Engineer

Introduction

Welcome to the first lesson in our #EthereumDecoded series.

In this post, we focus on what smart contract storage is, and how to decode it.

Later in the series we will go deeper into using Token Flow data, as well as looking at some more advanced topics and edge cases.

Ethereum Data Warehouse (EDW)

Token Flow makes available a Snowflake data warehouse with the full Ethereum history (including state and storage diffs).

We use a custom processor for this, based on @ethereumjs.

Fully decoded storage diffs will soon be available in the Ethereum Data Warehouse - EDW.

You can read more about how to get access to EDW in our documentation.

What is Storage?

Let's start with a little bit of theory. To explain this well, we need to look at EVM architecture:

https://ethereum.org/en/developers/docs/evm/

Storage is a global and persistent place used to store blockchain state changes.

It is a key-value data structure that holds the state variable values of smart contracts.

It is like a map of 2^256 slots, where each SLOT has 32 bytes.

Storage can be read and written, but also updated.

Due to the fact that it is on-chain, storing data is expensive and its cost is volatile - it consumes a lot of gas.

Worked example

Let's choose an example to practice with.

First, we'll find the top 10 smart contracts that have been used since block number 15,000,000.

We'll use the EDW for this (latest block during query execution: 15,954,222).

SELECT contract, count(*) as total
FROM edw.raw.storage_diffs
where block > 15000000
group by contract
order by total desc
limit 10;

The query is relatively simple. The search, even with a small Snowflake warehouse, is executed in only 22 seconds despite there being over 200 million interactions with these top 10 smart contracts since block 15 million! 🤯

Let's check what we obtained:

Top 10 used contracts from edw.raw.storage_diffs (block > 15000000)

The most used (by far) smart contract with over 60 million interactions is Wrapped Ether (WETH) - 0xc02a...56cc2. This is also a very simple contract, so it will be an ideal example to start our journey with storage diffs decoding.

Smart contract analysis

To start, let's do a simple analysis of the contract source code (available on Etherscan for example) to see how many storage slots it has:

pragma solidity ^0.4.18;

contract WETH9 {
    string public name     = "Wrapped Ether";
    string public symbol   = "WETH";
    uint8  public decimals = 18;

    event  Approval(address indexed src, address indexed guy, uint wad);
    event  Transfer(address indexed src, address indexed dst, uint wad);
    event  Deposit(address indexed dst, uint wad);
    event  Withdrawal(address indexed src, uint wad);

    mapping (address => uint)                       public  balanceOf;
    mapping (address => mapping (address => uint))  public  allowance;
    
    ...
}

The contract does not implement any abstraction, so we see that it has only 5 simple variables. There are no structures in this contract.

Going from the top (this is how the variables are organized in the storage) we see the following in the code extract above:

name - type: string, value: Wrapped Ether;
symbol - type: string, value: WETH;
decimals - type: uint8, value: 18;
balanceOf - type: mapping. It is a hashmap from address to uint;
allowance - type: double mapping. It is a double hashmap from pair of addresses to uint;

Now let's put it into some kind of structure to make it easier to decode later. Note that storage slot number starts at 0, and not at 1 (so the five variables above are slot 0 to 4 in order).

We'll use JSON for this structure:

{
  "contract_name": "WETH9",
  "slots": [
    {
      "slot_no": 0,
      "variables": [
        {
          "name": "name",
          "type": "string",
          "size": 32
        }
      ]
    },
    {
      "slot_no": 1,
      "variables": [
        {
          "name": "symbol",
          "type": "string",
          "size": 32
        }
      ]
    },
    {
      "slot_no": 2,
      "variables": [
        {
          "name": "decimals",
          "type": "uint8",
          "size": 1,
        }
      ]
    },
    {
      "slot_no": 3,
      "variables": [
        {
          "name": "balanceOf",
          "type": "mapping",
          "size": 32,
          "from_type": "address",
          "to_type": "uint"
        }
      ]
    },
    {
      "slot_no": 4,
      "variables": [
        {
          "name": "allowance",
          "type": "mapping",
          "size": 32,
          "from_type": "address",
          "to_type": {
            "type": "mapping",
            "size": 32,
            "from_type": "address",
            "to_type": "uint"

          }
        }
      ]
    }
  ],
  "structs": []
}

Looks good, doesn't it?

Token Flow's EDW has a ready-made similar storage layout of all the verified contracts in such a nice structure (which contains even more valuable information). 🤓

The Solidity types definitions (found here) allow us to find out how many bytes a variable occupies in a slot. We will explain more about defining contract storage layout in a future article in the #EthereumDecoded series.

To sum up: we found 5 slots and 0 structures. We will now use them for further decoding.

Storage decoding

EDW result

So let's choose slots 3 (balanceOf) and 4 (allowance) to use for decoding.

We don't include slots 0 to 2 (name, symbol and decimals) because, as we see in the contract, they are constant values.

We will select sample data from Token Flow's EDW, which contains all the changes in the storage over the entire Ethereum mainnet history for this specific contract for each of slot 3 and 4:

select * from edw.raw.storage_diffs where contract = '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2' and location like '3%' limit 1;
select * from edw.raw.storage_diffs where contract = '0xc02aaa39b223fe8d0a0e5c4f27ead9083c756cc2' and location like '4%' limit 1;

The result is as follows:

LOCATION,PREV_VALUE,CURR_VALUE	
3[0x980cd96ca20d257c2d38236ab404462a929e08f6].0,0x46c61bea48f514d0,0x2c97984e462734d0
4[0x9f7ff7431302ca6ae2a56e2bb50ccc0babffff55].0[0xe5c783ee536cf5e63e792988335c4255169be4e1].0,0x,0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff

We will discuss the meaning of the results for each slot separately.

Slot 3 - balanceOf

Location: 3[0x980...e08f6].0

Slot number 3 is a simple map from address to uint. In our raw datasets the mapping type has a schema: slot[key0].field0. Let's divide it into separate components and then decode it.

Raw:

location=3[0x980cd96ca20d257c2d38236ab404462a929e08f6].0
slot=3
type=MAPPING
key0=0x980cd96ca20d257c2d38236ab404462a929e08f6
field0=0

Decoded:

location=balanceOf[0x980cd96ca20d257c2d38236ab404462a929e08f6]
slot=balanceOf
key0=0x980cd96ca20d257c2d38236ab404462a929e08f6
field0=

We took the semantics of variable balanceOf from the contract.

It is a mapping from address, to uint. The key is already provided as an address so it does not change.

The .0 indicates a possible structure field, but in this contract we don't have structures, so the value is ignored.

Previous value: 0x46c61bea48f514d0

To begin with, let's left-pad each location with zeros to full 32 bytes (or 64 characters).

We always cut the size from the end. For this slot, we have a variable length of 32 bytes, so this means that the entire slot has been used. If we had a smaller size we would accordingly cut from the end, e.g.: size:20 means 40 characters.

According to solidity types, uint is an alias to uint256. How can we decode it? We used a helper function in python for this:

#uint_decoder.py

def decode_uint(value: str | int) -> int:
    return int(self.value, 16)

So the decoded value should be like this:

before=0x00000000000000000000000000000000000000000000000046c61bea48f514d0
after=5099794321103983824 (uint)

Current value: 0x2c97984e462734d0

We do exactly the same decoding for the current value:

before=0x0000000000000000000000000000000000000000000000002c97984e462734d0
after=3213204321103983824 (uint)

Result

balanceOf[0x980c...e08f6] indicates the number of tokens (value) owned by a given address. So the number of tokens belonging to this address changed from 5099794321103983824 to 3213204321103983824.

Slot 4 - allowance

Location: 4[0x9f7...fff55]. 0[0xe5c...be4e1].0

Slot 4 is a double hashmap that maps address to another hashmap that maps a second address to uint.

In our raw data such doublemappingtype has a schema: slot[key0].field0[key1].field1.

Let's divide the location into separate components.

Raw:

location=4[0x9f7ff7431302ca6ae2a56e2bb50ccc0babffff55].0[0xe5c783ee536cf5e63e792988335c4255169be4e1].0
slot=4
type=DOUBLE MAPPING
key0=0x9f7ff7431302ca6ae2a56e2bb50ccc0babffff55
field0=0
key1=0xe5c783ee536cf5e63e792988335c4255169be4e1
field1=0

Decoded:

location=allowance[0x9f7ff7431302ca6ae2a56e2bb50ccc0babffff55][0xe5c783ee536cf5e63e792988335c4255169be4e1]
slot=allowance
key0=0x9f7ff7431302ca6ae2a56e2bb50ccc0babffff55
field0=
key1=0xe5c783ee536cf5e63e792988335c4255169be4e1
field1=

We took the semantics of the variable allowance from the contract. This is a double map: address map to address map to uint. As in slot 3, we have no structures, so .0 values are omitted.

Previous value: 0x

Again we do padding and casting to the proper type.

before=0x0000000000000000000000000000000000000000000000000000000000000000
after=0 (uint)

In this case, the slot is also fully used, so, we convert the entire value to uint256, which will give us 0.

Current value: 0xfff...fffff

The current value equals 66 chars, so 0-padding is not needed.

before=0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
after=115792089237316195423570985008687907853269984665640564039457584007913129639935 (uint)

Result

allowance[0x9f7...fff55][0xe5c...be4e1] indicates the maximum number of tokens (value) that address1 (key1) can transfer from address0's (key0) account. So the maximum value is 11579...39935.

Summary

Why do we (you) need storage diffs?

They allow powerful querying of a full blockchain's history.

For the first time, with Token Flow's EDW they are all pre-hashed which means that you see all mapping keys and finally you can iterate through hashmaps. As Token Flow, we are happy to provide all of this in EDW.

We learned today about two simple types of locations:

MAPPING
DOUBLE MAPPING

These types, right next to SIMPLE (the slot number only) appear most often (we will provide various statistics from decoded storage diffs soon). They are very easy to decode, but there are also some edge cases, which we will also discuss in future lessons.

Decoding storage diffs requires a bit of knowledge about basic types in Solidity and Vyper, and the ability to parse contract codes. We use this to extract information from the contract about slots and structures. For decoding, we use quite simple Python scripts.

Once you have the decoded data then the analysis is much more straightforward.

If you still don't feel up to it, don't worry. More difficult examples are coming later in this "#EthereumDecoded" cycle. Be sure to sign up for our mailing list and follow us on Twitter to be notified when the next post in the series is available (links are in the footer below).

Also, you don't have to do it yourself - you can just use Token Flow's EDW to get all decoded storage diffs throughout the full history of Ethereum mainnet.

Follow us for news and announcements

You can follow us, sign up to our newsletter, and join the conversation at:

Web: tokenflow.live

Documentation: docs.tokenflow.live

Careers: careers.tokenflow.live

Twitter: @InsightsFlow

Discord: discord.gg

#EthereumDecoded lesson 1 - storage diff decoding with Token Flow