MultipleResultsError when Saving Missing Block Ranges in Explorer Utility "GenServer Indexer.Block.Catchup.MissingRangesCollector terminating\n** (stop)"

[mpekhota]

Description

I launched blockscout for mainnet and I see that the old blocks have stopped being scanned and added to the database, new blocks are taken from the blockchain without any problems.
I used the chart v1.5.0 to run with your v6.4.0-beta docker image.

It seems like I'm encountering errors related to a GenServer called Explorer.Utility.MissingRangesManipulator. This GenServer seems to be involved in handling missing block ranges. The error message indicates that there's an issue with an Ecto query, specifically, it's expecting at most one result but getting 2. This is likely happening when trying to save a batch of missing block ranges.

Type of the installation

Helm charts (k8s)

Type of the JSON RPC archive node

Geth

Type of the chain

L1

Link to the page

No response

Steps to reproduce

I launched blockscout for mainnet and I see that the old blocks have stopped being scanned and added to the database, new blocks are taken from the blockchain without any problems.
I used the chart blockscout-stack-1.5.0 to run with your v6.4.0-beta docker image.

Backend version

v6.4.0-beta

Frontend version

v.1.28.1

Elixir & Erlang/OTP versions

1.14.5

Operating system

Linux

Additional information

{"time":"2024-04-20T11:48:06.931Z","severity":"error","message":"GenServer Indexer.Block.Catchup.MissingRangesCollector terminating\n** (stop) exited in: GenServer.call(Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000)\n    ** (EXIT) an exception was raised:\n        ** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n            (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n            (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n            (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n            (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n            (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n            (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n            (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n    (elixir 1.14.5) lib/gen_server.ex:1038: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message: :update_past\nState: %{max_fetched_block_number: 11352381, min_fetched_block_number: 11252381}","metadata":{"error":{"initial_call":null,"reason":"{{{%Ecto.MultipleResultsError{message: \"expected at most one result but got 2 in query:\\n\\nfrom m0 in Explorer.Utility.MissingBlockRange,\\n  where: m0.from_number >= ^11_152_381,\\n  where: m0.to_number <= ^11_152_381\\n\"}, [{Ecto.Repo.Queryable, :one, 3, [file: 'lib/ecto/repo/queryable.ex', line: 157]}, {Explorer.Utility.MissingBlockRange, :save_range, 1, [file: 'lib/explorer/utility/missing_block_range.ex', line: 45]}, {Enum, :\"-map/2-lists^map/1-0-\", 2, [file: 'lib/enum.ex', line: 1658]}, {Explorer.Utility.MissingRangesManipulator, :handle_call, 3, [file: 'lib/explorer/utility/missing_ranges_manipulator.ex', line: 46]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 721]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 750]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}, {GenServer, :call, [Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000]}}, [{GenServer, :call, 3, [file: 'lib/gen_server.ex', line: 1038]}, {Indexer.Block.Catchup.MissingRangesCollector, :handle_info, 2, [file: 'lib/indexer/block/catchup/missing_ranges_collector.ex', line: 144]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 695]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 771]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}"}}}
{"time":"2024-04-20T11:48:08.001Z","severity":"info","message":"Sent 200 in 67ms on GET /api/v1/health/readiness?","metadata":{"duration":67526,"endpoint":"/api/v1/health/readiness?","method":"GET","request_id":"F8f6ke-QwZPsFM8AC4BE","status":"200","unit":"microsecond"}}
{"time":"2024-04-20T11:48:09.668Z","severity":"info","message":"Start sanitizing of empty blocks. Batch size is 100","metadata":{"fetcher":"empty_blocks_to_refetch"}}
{"time":"2024-04-20T11:48:09.807Z","severity":"info","message":"Batch of empty blocks is sanitized","metadata":{"fetcher":"empty_blocks_to_refetch"}}
{"time":"2024-04-20T11:48:10.516Z","severity":"error","message":"GenServer Explorer.Utility.MissingRangesManipulator terminating\n** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message (from Indexer.Block.Catchup.MissingRangesCollector): {:save_batch, [11252380..11152381//-1]}\nState: %{}\nClient Indexer.Block.Catchup.MissingRangesCollector is alive\n\n    (stdlib 3.17) gen.erl:233: :gen.do_call/4\n    (elixir 1.14.5) lib/gen_server.ex:1035: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3","metadata":{"error":{"initial_call":null,"reason":"** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n"}}}
{"time":"2024-04-20T11:48:10.517Z","severity":"error","message":"GenServer Indexer.Block.Catchup.MissingRangesCollector terminating\n** (stop) exited in: GenServer.call(Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000)\n    ** (EXIT) an exception was raised:\n        ** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n            (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n            (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n            (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n            (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n            (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n            (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n            (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n    (elixir 1.14.5) lib/gen_server.ex:1038: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message: :update_past\nState: %{max_fetched_block_number: 11352381, min_fetched_block_number: 11252381}","metadata":{"error":{"initial_call":null,"reason":"{{{%Ecto.MultipleResultsError{message: \"expected at most one result but got 2 in query:\\n\\nfrom m0 in Explorer.Utility.MissingBlockRange,\\n  where: m0.from_number >= ^11_152_381,\\n  where: m0.to_number <= ^11_152_381\\n\"}, [{Ecto.Repo.Queryable, :one, 3, [file: 'lib/ecto/repo/queryable.ex', line: 157]}, {Explorer.Utility.MissingBlockRange, :save_range, 1, [file: 'lib/explorer/utility/missing_block_range.ex', line: 45]}, {Enum, :\"-map/2-lists^map/1-0-\", 2, [file: 'lib/enum.ex', line: 1658]}, {Explorer.Utility.MissingRangesManipulator, :handle_call, 3, [file: 'lib/explorer/utility/missing_ranges_manipulator.ex', line: 46]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 721]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 750]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}, {GenServer, :call, [Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000]}}, [{GenServer, :call, 3, [file: 'lib/gen_server.ex', line: 1038]}, {Indexer.Block.Catchup.MissingRangesCollector, :handle_info, 2, [file: 'lib/indexer/block/catchup/missing_ranges_collector.ex', line: 144]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 695]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 771]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}"}}}
{"time":"2024-04-20T11:48:14.104Z","severity":"error","message":"GenServer Explorer.Utility.MissingRangesManipulator terminating\n** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message (from Indexer.Block.Catchup.MissingRangesCollector): {:save_batch, [11252380..11152381//-1]}\nState: %{}\nClient Indexer.Block.Catchup.MissingRangesCollector is alive\n\n    (stdlib 3.17) gen.erl:233: :gen.do_call/4\n    (elixir 1.14.5) lib/gen_server.ex:1035: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3","metadata":{"error":{"initial_call":null,"reason":"** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n"}}}
{"time":"2024-04-20T11:48:14.106Z","severity":"error","message":"GenServer Indexer.Block.Catchup.MissingRangesCollector terminating\n** (stop) exited in: GenServer.call(Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000)\n    ** (EXIT) an exception was raised:\n        ** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n            (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n            (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n            (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n            (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n            (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n            (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n            (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n    (elixir 1.14.5) lib/gen_server.ex:1038: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message: :update_past\nState: %{max_fetched_block_number: 11352381, min_fetched_block_number: 11252381}","metadata":{"error":{"initial_call":null,"reason":"{{{%Ecto.MultipleResultsError{message: \"expected at most one result but got 2 in query:\\n\\nfrom m0 in Explorer.Utility.MissingBlockRange,\\n  where: m0.from_number >= ^11_152_381,\\n  where: m0.to_number <= ^11_152_381\\n\"}, [{Ecto.Repo.Queryable, :one, 3, [file: 'lib/ecto/repo/queryable.ex', line: 157]}, {Explorer.Utility.MissingBlockRange, :save_range, 1, [file: 'lib/explorer/utility/missing_block_range.ex', line: 45]}, {Enum, :\"-map/2-lists^map/1-0-\", 2, [file: 'lib/enum.ex', line: 1658]}, {Explorer.Utility.MissingRangesManipulator, :handle_call, 3, [file: 'lib/explorer/utility/missing_ranges_manipulator.ex', line: 46]}, {:gen_server, :try_handle_call, 4, [file: 'gen_server.erl', line: 721]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 750]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}, {GenServer, :call, [Explorer.Utility.MissingRangesManipulator, {:save_batch, [11252380..11152381//-1]}, 7000]}}, [{GenServer, :call, 3, [file: 'lib/gen_server.ex', line: 1038]}, {Indexer.Block.Catchup.MissingRangesCollector, :handle_info, 2, [file: 'lib/indexer/block/catchup/missing_ranges_collector.ex', line: 144]}, {:gen_server, :try_dispatch, 4, [file: 'gen_server.erl', line: 695]}, {:gen_server, :handle_msg, 6, [file: 'gen_server.erl', line: 771]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 226]}]}"}}}
{"time":"2024-04-20T11:48:17.694Z","severity":"error","message":"GenServer Explorer.Utility.MissingRangesManipulator terminating\n** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\nLast message (from Indexer.Block.Catchup.MissingRangesCollector): {:save_batch, [11252380..11152381//-1]}\nState: %{}\nClient Indexer.Block.Catchup.MissingRangesCollector is alive\n\n    (stdlib 3.17) gen.erl:233: :gen.do_call/4\n    (elixir 1.14.5) lib/gen_server.ex:1035: GenServer.call/3\n    (indexer 6.4.0) lib/indexer/block/catchup/missing_ranges_collector.ex:144: Indexer.Block.Catchup.MissingRangesCollector.handle_info/2\n    (stdlib 3.17) gen_server.erl:695: :gen_server.try_dispatch/4\n    (stdlib 3.17) gen_server.erl:771: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3","metadata":{"error":{"initial_call":null,"reason":"** (Ecto.MultipleResultsError) expected at most one result but got 2 in query:\n\nfrom m0 in Explorer.Utility.MissingBlockRange,\n  where: m0.from_number >= ^11_152_381,\n  where: m0.to_number <= ^11_152_381\n\n    (ecto 3.11.2) lib/ecto/repo/queryable.ex:157: Ecto.Repo.Queryable.one/3\n    (explorer 6.4.0) lib/explorer/utility/missing_block_range.ex:45: Explorer.Utility.MissingBlockRange.save_range/1\n    (elixir 1.14.5) lib/enum.ex:1658: Enum.\"-map/2-lists^map/1-0-\"/2\n    (explorer 6.4.0) lib/explorer/utility/missing_ranges_manipulator.ex:46: Explorer.Utility.MissingRangesManipulator.handle_call/3\n    (stdlib 3.17) gen_server.erl:721: :gen_server.try_handle_call/4\n    (stdlib 3.17) gen_server.erl:750: :gen_server.handle_msg/6\n    (stdlib 3.17) proc_lib.erl:226: :proc_lib.init_p_do_apply/3\n"}}}

[mpekhota]

@vbaranov any suggestions?

[Qwerty5Uiop]

Hi @mpekhota! Could you please show the result of this query?
select * from missing_block_ranges where from_number >= 11152381 and to_number <= 11152381;

[mpekhota]