ETL pipeline for the Ethereum blockchain using NodeJS, EthereumJS and any Ethereum node providing RPC interface.
For a given block, the block information, transactions and transaction receipts are fetched and processed by several processors which goal is to extract and structure information such as events, ERC20 transfers, NFTs transfers and load them in a database.
You are lost? Check the documentation on how to run your own DAppBoard!
This ETL reads the blockchain block by block in order to identify and save the following elements:
- Blocks: A block is the basic element of a 'blockchain'. It functions as an entry in a distributed ledger, recording a series of transactions together with a reference to the previous block. A block is chained to its preceding block by a cryptographic hash of its contents as a means of reference.
- Transactions: Transactions are messages between two accounts that may transfer Ether and may contain a payload. Transactions always originate from an external account that is controlled by an external actor by means of a private key.
- Events: Events are structures implemented in smart contracts. They cannot be accessed from the contract but can be used from the outside, e.g. by a web application that reacts to Events in a smart contract.
- Token transfers: Event generated when a token is transferred between 2 addresses.
- Tokens: A token is a virtual good that can be traded. They can be ERC20 or ERC721 (NFTs).
- Meta events: The name and parameters of events implemented by a smart contract.
This ETL is developed by the DAppBoard team and is used there. We're glad to offer our technology to the public so anyone can understand how data are gathered and stored before being displayed.
We'd like to thanks Blockchain ETL that inspired us and helped us organize data in an easy way to understand format and where tools can be used and developed across teams and projects.
This project is licensed under the MIT License.