feat: index celo transactions #9713

fedor-ivn · 2024-03-24T09:05:54Z

Motivation

This PR adds support for custom fields from Celo blockchain and implements a
custom way to index Celo transactions.

Changelog

Enhancements

Environment variables

New value for CHAIN_TYPE env: celo
New variable: CELO_CORE_CONTRACTS

Transactions

New fields specific to CELO network are now supported. Here is the outline how the fields coming from RPC node are mapped to fields in transactions table:

feeCurrency → gas_token_contract_address_hash
gatewayFeeRecipient → gas_fee_recipient_address_hash (this field is
scheduled for deprecation as stated in CELO docs)
gatewayFee → gateway_fee (this field is
scheduled for deprecation as stated in CELO
docs)

The token used for gas payment (feeCurrency) is indexed within block fetcher process.

Token-duality feature

Native chain asset transfers are treated as CELO ERC-20 token transfers. These token transfers are artificial, meaning that there is not real T

API updates

Native chain asset transfers are treated as CELO ERC-20

Indexing of native chain asset transfers as CELO ERC-20 token transfers from transactions and internal transactions
- Implemented in block fetcher and internal transactions

Checklist for your Pull Request (PR)

If I added new functionality, I added tests covering it.
If I fixed a bug, I added a regression test to prevent the bug from silently reappearing again.
I checked whether I should update the docs and did so by submitting a PR to https://github.com/blockscout/docs
If I added/changed/removed ENV var, I submitted a PR to https://github.com/blockscout/docs to update the list of env vars at https://github.com/blockscout/docs/blob/master/for-developers/information-and-settings/env-variables.md and I updated the version to master in the Version column. Changes will be reflected in this table: https://docs.blockscout.com/for-developers/information-and-settings/env-variables.
If I add new indices into DB, I checked, that they are not redundant with PGHero or other tools

apps/indexer/test/indexer/block/fetcher_test.exs

fedor-ivn · 2024-03-26T10:11:19Z

apps/indexer/test/indexer/block/fetcher_test.exs

@@ -614,6 +607,39 @@ defmodule Indexer.Block.FetcherTest do
          assert fifth_address.fetched_coin_balance_block_number == block_number

        EthereumJSONRPC.Nethermind ->


For some reason, the tests in this file just ignore the EthereumJSONRPC variant.

In my case, this config sets the variant geth:

blockscout/apps/indexer/config/runtime/test.exs

Lines 1 to 11 in 509b897

import Config

alias EthereumJSONRPC.Variant

config :indexer, Indexer.Fetcher.Beacon.Blob.Supervisor, disabled?: true

config :indexer, Indexer.Fetcher.Beacon.Blob, start_block: 0

variant = Variant.get()

Code.require_file("#{variant}.exs", "#{__DIR__}/../../../explorer/config/test")

Code.require_file("#{variant}.exs", "#{__DIR__}/../../../indexer/config/test")

So, it loads this config, setting EthereumJSONRPC.Geth variant:

blockscout/apps/indexer/config/test/geth.exs

Lines 1 to 8 in 509b897

import Config

config :indexer,

json_rpc_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

]

blockscout/apps/explorer/config/test/geth.exs

Lines 1 to 13 in 509b897

import Config

config :explorer,

json_rpc_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

],

subscribe_named_arguments: [

transport: EthereumJSONRPC.Mox,

transport_options: [],

variant: EthereumJSONRPC.Geth

]

However, these json_rpc_named_arguments settings are just ignored in the tests. Why it could be so?

Because of this, I had nothing to do, but put my celo-changes to EthereumJSONRPC.Nethermind case branch

fedor-ivn · 2024-04-03T09:31:05Z

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex

I think hard-coded values are fine at least for v1, because they are really not a subject to change.

For instance, on Celo mainnet, the last transaction to Registry contract was done 4 years ago. On alfajores, Registry contract was not even updated -- e.g it returns nil-address (0x000...) for the core contract names.

nikitosing · 2024-04-11T16:25:14Z

apps/indexer/lib/indexer/fetcher/internal_transaction.ex

+      case tokens do
+        [token] ->


Why not to case token_transfers?

Yeah, make sense. Please, check 222feba to verify if I implemented your suggestions correctly

nikitosing · 2024-04-11T16:26:22Z

apps/indexer/lib/indexer/fetcher/internal_transaction.ex

+    defp to_address_token_balances(token_transfers, celo_token) do
+      %{token_transfers: token_transfers}
+      |> Addresses.extract_addresses()
+      |> Enum.map(fn %{fetched_coin_balance_block_number: block_number, hash: hash} ->
+        with {:ok, address_hash} <- Hash.Address.cast(hash),
+             {:ok, token_contract_address_hash} <- Hash.Address.cast(celo_token.contract_address_hash) do
+          %{
+            address_hash: address_hash,
+            token_contract_address_hash: token_contract_address_hash,
+            block_number: block_number,
+            token_type: celo_token.type,
+            token_id: nil
+          }
+        else
+          error -> Logger.error("Failed to cast string to hash: #{inspect(error)}")
+        end
+      end)
+      |> MapSet.new()
+    end
+
+    defp async_import_celo_token_balances(%{token_transfers: token_transfers, tokens: tokens}) do


I would suggest to define functions in order in which they are called. I.e. firstly define async_import_celo_token_balances and then to_address_token_balances

Also fixed in 222feba

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex

…ield

* master: (65 commits) fix: Add healthcheck endpoints for indexer-only setup (#10076) 6.6.0 fix: Rework revert_reason (#9212) fix: Eliminate from_address_hash == #{address_hash} clause for transactions query in case of smart-contracts (#9469) feat: Add optional retry of NFT metadata fetch in Indexer.Fetcher.Tok… (#10036) fix: Separate indexer setup (#10032) feat: Blueprint contracts support (#10058) chore: Update hackney pool size: add new fetchers accounting (#9941) fix: Disallow batched queries in GraphQL endpoint (#10050) feat: Clone with immutable arguments proxy pattern (#10039) fix: vyper contracts re-verificaiton (#10053) refactor: Refactor get_additional_sources/4 -> get_additional_sources/3 (#10046) chore: Bump credo from 1.7.5 to 1.7.6 (#10060) chore: Bump redix from 1.5.0 to 1.5.1 (#10059) chore: Bump ex_doc from 0.32.1 to 0.32.2 (#10061) fix: Fix Unknown UID bug at smart-contract verification (#9986) refactor: test database config (#9662) chore: remove `has_methods` from `/addresses` (#10051) feat: Improve retry NFT fetcher (#10027) chore: Add support of Blast-specific L1 OP withdrawal events (#10049) ...

fedor-ivn · 2024-05-18T17:02:10Z

apps/indexer/test/indexer/block/realtime/fetcher_test.exs

+      # In CELO network, there is a token duality feature where CELO can be used
+      # as both a native chain currency and as an ERC-20 token. Transactions
+      # that transfer CELO are also counted as token transfers, and the
+      # TokenInstance fetcher is called. However, for simplicity, we disable it
+      # in this test.
+      Application.put_env(:indexer, Indexer.Fetcher.TokenInstance.Realtime.Supervisor, disabled?: true)


What do you think about this workaround?

@vbaranov @nikitosing

* master: refactor: Remove hardcoded numResults from fetch_pending_transactions_besu (#10117) feat: Add window between balance fetch retries for missing balanceOf tokens (#10142) fix: Add missing preloads to tokens endpoints (#10072) Explicit message on token balance update error (#10129) fix: missing nil case for revert reason (#10136) fix: Hotfix for Indexer.Fetcher.Optimism.WithdrawalEvent and EthereumJSONRPC.Receipt (#10130) hide chain specific fields behind Map.get (#10131) --- (#10096) feat: indexer for cross level messages on Arbitrum (#9312) chore: Bump ecto_sql from 3.11.1 to 3.11.2 Indexer/API separated images for Redstone Update CHANGELOG Improve response of address API to return multiple implementations for Diamond proxy (#10113) Fix GA pre-release && release workflows Update CHANGELOG feat: implement fetch_first_trace for Geth (#10087) Remove custom release CI for Immutable Update CHANGELOG for 6.6.0 Fix certified flag in the search API v2 endpoint (#10094) fix: Update Vyper inner compilers list to support all compilers (#10091)

fedor-ivn commented Mar 26, 2024

View reviewed changes

apps/indexer/test/indexer/block/fetcher_test.exs Outdated Show resolved Hide resolved

fedor-ivn commented Mar 26, 2024

View reviewed changes

fedor-ivn force-pushed the fi-celo-transactions branch 2 times, most recently from 7c372ce to d16cca3 Compare March 26, 2024 19:25

fedor-ivn force-pushed the fi-celo-transactions branch from a06d98f to 2e8a756 Compare April 2, 2024 20:33

fedor-ivn requested a review from nikitosing April 2, 2024 20:34

fedor-ivn commented Apr 3, 2024

View reviewed changes

fedor-ivn added the ci:celo label Apr 5, 2024

fedor-ivn force-pushed the fi-celo-transactions branch 4 times, most recently from 40f044d to a0e8cc3 Compare April 10, 2024 09:22

fedor-ivn requested a review from vbaranov April 10, 2024 18:41

nikitosing reviewed Apr 11, 2024

View reviewed changes

fedor-ivn force-pushed the fi-celo-transactions branch from 2a4851c to 3b7624a Compare April 15, 2024 10:41

fedor-ivn added ci:all and removed ci:celo labels Apr 15, 2024

fedor-ivn commented Apr 15, 2024

View reviewed changes

apps/explorer/lib/explorer/chain/cache/celo_core_contracts.ex Outdated Show resolved Hide resolved

fedor-ivn added 12 commits April 16, 2024 13:48

feat(json-rpc): support celo-specific fields in transactions

ef6c53c

chore: add new chain type to matrix builder

e31566e

feat: add celo-specific fields to transactions schema

8cf18ff

chore: add "celo" to cspell.json

8ca776a

chore: fix formatting

7c04a5b

refactor: improve naming for celo fields and ignore ethCompatible f…

161e96d

…ield

chore: add block fetcher test for celo fields

e61a36b

feat: add celo core contract abis

9b3e45a

feat: add a cache for core celo contracts

3e90794

feat: treat celo native token transfers as token transfers

3d31c9f

fix: update cspell.json with celo words

8d338e4

fix(dialyzer): pattern can never match the type

1100f96

fedor-ivn added ci:celo and removed ci:all labels May 17, 2024

fedor-ivn force-pushed the fi-celo-transactions branch 11 times, most recently from 3615ae2 to b6f833b Compare May 18, 2024 12:58

fedor-ivn added 2 commits May 18, 2024 13:06

fix: swap arguments in put_if_present

f927d74

fedor-ivn force-pushed the fi-celo-transactions branch from 90e5394 to b095ec3 Compare May 18, 2024 15:42

fix: indexer tests

5cf8be3

fedor-ivn added ci:all and removed ci:celo labels May 18, 2024

fedor-ivn commented May 18, 2024

View reviewed changes

fedor-ivn added 2 commits May 22, 2024 21:22

refactor: always include gas_token to the response

decbd25

feat: add gas_token to all transaction-related endpoints

18d82d1

fedor-ivn force-pushed the fi-celo-transactions branch from b095ec3 to 18d82d1 Compare May 23, 2024 12:56

fedor-ivn added 4 commits May 30, 2024 12:42

feat: celo core contracts with historical data

42569b5

fix: uncles field is expected but not present in RPC node response

a782ea9

fix: credo & format

86b86f6

fedor-ivn force-pushed the fi-celo-transactions branch from cd6f72d to 86b86f6 Compare May 30, 2024 14:59

fedor-ivn added 2 commits May 30, 2024 18:02

fix: define CELO_CORE_CONTRACTS in tests

e1553c2

fix: put gas_token under celo section

780b1fb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: index celo transactions #9713

feat: index celo transactions #9713

fedor-ivn commented Mar 24, 2024 •

edited

fedor-ivn Mar 26, 2024

fedor-ivn Mar 26, 2024 •

edited

fedor-ivn Apr 3, 2024

nikitosing Apr 11, 2024

fedor-ivn Apr 12, 2024

nikitosing Apr 11, 2024

fedor-ivn Apr 12, 2024

fedor-ivn May 18, 2024

fedor-ivn May 18, 2024

		@@ -614,6 +607,39 @@ defmodule Indexer.Block.FetcherTest do
		assert fifth_address.fetched_coin_balance_block_number == block_number

		EthereumJSONRPC.Nethermind ->

	import Config

	alias EthereumJSONRPC.Variant

	config :indexer, Indexer.Fetcher.Beacon.Blob.Supervisor, disabled?: true
	config :indexer, Indexer.Fetcher.Beacon.Blob, start_block: 0

	variant = Variant.get()

	Code.require_file("#{variant}.exs", "#{__DIR__}/../../../explorer/config/test")
	Code.require_file("#{variant}.exs", "#{__DIR__}/../../../indexer/config/test")

	import Config

	config :indexer,
	json_rpc_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	]

	import Config

	config :explorer,
	json_rpc_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	],
	subscribe_named_arguments: [
	transport: EthereumJSONRPC.Mox,
	transport_options: [],
	variant: EthereumJSONRPC.Geth
	]

feat: index celo transactions #9713

Are you sure you want to change the base?

feat: index celo transactions #9713

Conversation

fedor-ivn commented Mar 24, 2024 • edited

Motivation

Changelog

Enhancements

Environment variables

Transactions

Token-duality feature

API updates

Checklist for your Pull Request (PR)

Choose a reason for hiding this comment

fedor-ivn Mar 26, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedor-ivn commented Mar 24, 2024 •

edited

fedor-ivn Mar 26, 2024 •

edited