Custom reth-based indexer

[naps62]

NOTICE: Indexer migrated to its own repo: ethui/indexer#1

---

We want to provide, as a SaaS, an indexer, possibly based on reth-indexer, to provide enhanced abilities for users.

Namely:

• faster indexing of transaction history, without relying on alchemy (aka: remove the need for a key in the onboarding flow, and bypass their rate limit restrictions)
• indexing additional data for wallets, such as ERC20 allowances (which would allow us to implement a revoke.cash like solution in the wallet)

The service should be available to a whitelist of addresses, comprised of:

• our own wallets, for testing purposes
• test first few addresses of the test test test ... junk mnemonic, as a way to showcase the feature to every incoming user
• to any address that signs up through a yet-to-be-defined sign up process, but which would be kept as simple as possible in initial stages (e.g.: by making a one-off donation to iron-wallet.eth)

Tracking

• binary / crate - indexer: initial setup #534
• HTTP server - indexer: basic actix setup #535
• HTTP auth via address signature
• postgres database connection
• db table to hold list of whitelisted addresses
• db data to hold sync progress per address
• db tables to hold indexed data
• syncing data from current block into db
• one-off jobs for syncing past data for new addresses

[ABZerra]

Looks good to me, I liked that you specified some usecases! Would only add the Revoke.cash link since you also linked the reth one

https://revoke.cash/

[joshstevens19]

This is super cool - would love to help change reth-indexer to be able to be consumed like a package instead of a singleton runner as it’s wrote now! Really open to changing the code and working with you on it!

Also loving what you guys are trying to build here let me know how I can help? 🔥

[naps62]

@joshstevens19 thanks! for now I'm trying to speedrun an initial implementation.
basic idea is to index up-to-date balances, as well as all relevant txs for a wallet's history (so not only from/to, but also any tx where a given address is mentioned in event logs).

I also have an additional requirement: the list of addresses to track grows at runtime.
so for new joiners, I also need to back-fill info from past blocks. I'm thinking here of launching an additional one-off task for each address, with much less priority and probably a cut-off date.
no idea what the real-world performance here will look like, but appreciate any input you may have

your work & docs were already helpful, particularly the CSV copy optimization, which I had no idea about