What is Blockchain Reorganization (Reorg)?
A blockchain is an ever-growing ledger that records transactions in blocks that are linked sequentially. Each block contains the cryptographic hash of the previous block, a timestamp, and transaction details. This sequential chaining ensures that once a transaction is recorded on the blockchain, modifying it would require changing all subsequent blocks, making blockchain transactions immutable.
Sometimes, due to two miners producing blocks simultaneously, the chain can temporarily fork. To resolve this, the blockchain protocol uses a "fork choice rule" mechanism. This rule evaluates all received blocks and messages to decide which chain is the valid main chain. This is critical because when two competing blocks are generated at the same time, there can be multiple candidate main chains. Blocks that are not chosen for the main chain become orphan blocks. As the chain grows longer, the chances of a particular fork being discarded reduce significantly, ensuring the blockchain's integrity.
Block reorganization (block reorganization) is a complex concept that can be explained through diagrams. Below is a simplified example illustrating how block reorganization occurs.
- In a regular situation: Under normal operations, blocks are added to the blockchain in sequence.
A → B → C → D
- Fork occurrence: Due to reasons such as network latency, software incompatibility, or malicious actions, a portion of the nodes in the network might create and accept blocks that are different from other nodes. This leads to a fork.
A → B → C → D
↘ E → F
- Block reorganization: If the forked chain is longer than the main chain or is accepted by the majority of nodes in the network in some other way, the original chain might be discarded, and the forked chain may become the new main chain. This is what's referred to as block reorganization.
A → B → E → F
In this example, blocks C and D are discarded, while blocks E and F are accepted.
What happens if a transaction is involved?
When a block reorganization occurs in an EVM network, a transaction can move from a discarded block to a new main chain block. Here are the detailed steps and explanations of this process:
1. The transaction is packaged into Block A.
Validators package the transaction into Block A and broadcast it to the network.
User → Transaction → Mempool → Validator → Block A
2. A fork occurs.
Due to network latency or other reasons, another validator might simultaneously create another Block B. Now there are two competing blocks.
Block A → Network
↘ Block B
3. Reorganization Occurs
As more validators cast their votes, Block B might gain more support and becomes the new main chain. Block A is discarded.
Block A (discarded)
Block B → Network (main chain)
4. How the Transaction "Moves" to Block B
How the transaction "moves" from Block A to Block B depends on how Block B was created:
- Scenario 1: Block B Contains the Same Transaction: If the validator of Block B picked the same transaction from the mempool and packaged it into Block B, the transaction will naturally transition from Block A to Block B.
- Scenario 2: Block B Doesn't Contain the Transaction: If Block B doesn't contain the transaction, then when Block A is discarded, the transaction might return to the mempool and be repackaged in subsequent blocks.
5. Transaction Gets Confirmed
As more blocks are added to the main chain, Block B and the transactions it contains eventually get confirmed.
Block B (finalized) → Network
In the reorganization process of Ethereum or any EVM chain, a transaction might move from a discarded block to a new main chain block. This can be achieved either by including the transaction in a new block or by returning the transaction to the mempool and repackaging it in subsequent blocks. Regardless of the method, the final outcome of the transaction should not be affected by the reorganization.
Despite the fact that a single transaction will not be affected by the reorg, the inconsistency of the transaction data hash and mapping block will definitely influence the data indexer and provider.
Why is block reorganization a problem for indexer?
- Data Consistency: Data indexers rely on the blockchain's data to be consistent. A block reorganization can change the state of the blockchain, meaning that the data previously indexed might no longer be valid. This requires the indexer to reprocess and reindex the affected blocks.
- Performance Overhead: Reindexing due to block reorganization can be resource-intensive. The indexer might need to roll back to a previous state and then reindex the new blocks, which can be time-consuming and can strain system resources.
- Data Integrity: If the indexer doesn't handle reorganizations correctly, it might serve outdated or incorrect data to applications or users relying on its indexed data.
- Increased Latency: To mitigate the effects of potential reorganizations, some indexers might introduce a delay in indexing new blocks to ensure they are part of the longest chain. This can increase the time it takes for new data to appear in indexed results.
How Chainbase handles these situations？
As an infrastructure that relies on blockchain data, Chainbase is acutely aware of the importance of effectively detecting and handling block reorganizations. Below is a detailed explanation of how Chainbase executes these two critical steps:
Detecting Block Reorganization:
In this step, Chainbase monitors potential block reorganizations on the chain while indexing block data. The advantage of this approach is that Chainbase can discover real-time changes in the chain's status, including normal block additions and abnormal block reorganizations.
This method allows us to pass reorganized data downstream (for example, to the application layer or user interface) while maintaining consistency with the chain. This ensures that Chainbase can quickly respond to changes on the chain and accurately reflect these changes in the client's applications or services.
Handling Block Reorganization:
In this step, we update and restore relevant data in real time based on detected block reorganization events. For example, if a user's asset balance needs to be rolled back to a specific block height due to a block reorganization, Chainbase can automatically perform this operation. For "unconfirmed" block data, Chainbase saves all versions of block heights with changes and makes the latest version of the data visible to the outside. When a block reorganization occurs, it rolls back to the specified version.
The advantage of this approach is that it ensures that Chainbase's data remains consistent with the actual state of the chain, even when faced with complex situations such as block reorganizations.
At the product module level, Chainbase's SYNC module can continuously synchronize the most up-to-date data from the chain, and sync the reorganized data to users when a reorganization occurs. This is a very important feature, as it ensures that users always receive accurate and up-to-date data.
More Technical Detail
1. Detecting the Starting Point of Block Reorganization:
First, we will check whether the hash of the first received block is consistent with the block_hash saved locally. This step is to determine whether a block reorganization (reorg) has occurred.
If a block reorganization is detected, we will update the last_sync_block_file, which records the information of the last synchronized block.
Once a reorganization is discovered, we will instruct the ETL (Extract, Transform, Load) process to start rewriting data from the block height where the reorganization occurred.
2. Ensuring Block Continuity:
Our algorithm will traverse and check each block in the export_items to ensure that its hash is consistent with the hash of the previous block.
This step is to ensure that the block_hash of this batch is continuous, thereby guaranteeing the integrity and continuity of the block data.
3. Final Verification and Response to Block Reorganization:
In the end, we will determine whether the hash of the last_block is consistent with the data on the chain.
If a block reorganization (reorg) is discovered, we will decide to discard this batch of data.
In this situation, we will instruct the ETL process to re-fetch data from the chain, thereby ensuring that the local data remains consistent with the state of the chain.
Conclusion
The issue of blockchain reorganization is an undeniable challenge for all data providers. To ensure the integrity and accuracy of data, we are obligated to confront and address this issue head-on. At the same time, deep communication and collaboration with industry experts become paramount. Only by combining collective efforts and wisdom can we hope to devise durable and efficient solutions, aiming to offer the most reliable blockchain data services to users. Chainbase seizes this opportunity to share our developmental approach transparently and aspires to collaborate with data indexers and providers across the industry, jointly crafting a superior web3 data platform for developers.
About Chainbase
Chainbase is an all-in-one web3 data infrastructure for indexing, transforming, and utilization of on-chain data at scale. By leveraging enriched on-chain data and streaming computing technologies across one data infrastructure, Chainbase automates the indexing and querying of blockchain data, enabling developers to accomplish more with less effort.
Want to learn more about Chainbase?
Visit our website chainbase.com Sign up for a free account, and Check out our documentation.
Website｜Blog｜Twitter｜Discord｜Link3
