Free/Unlimited/Fast RPC endpoints for indexing with ponder

[ozgrakkurt]

Hey! very cool project

I am especially impressed with how easy it is to setup and test some stuff.

I work at envio and we are building an rpc service which is implemented on top of our custom storage layer for historical evm data. We currently have free and unlimited endpoints for 20 evm chains.

I tested running ponder with our rpc and we seem to be over 200x as fast as blastapi/chainstack premium plans when indexing FairToken (I found this randomly on etherscan for testing).

The config I use is this:

import type { Config } from "@ponder/core";
import { http } from "viem";

export const config: Config = {
  networks: [
    {
      name: "mainnet",
      chainId: 1,
      transport: http("https://eth.rpc.hypersync.xyz/ethistanbul"),
      maxRpcRequestConcurrency: 10,
    },
  ],
  contracts: [
    {
      name: "FairToken",
      network: "mainnet",
      abi: "./abis/FairToken.json",
      address: "0x9b20dabcec77f6289113e61893f7beefaeb1990a",
      startBlock: 4805840,
      maxBlockRange: 100000,      
    },
  ],
};

and the indexer is generated using etherscan import method (very nice to use).

Might need to set maxBlockRange to a lower number for denser contracts like ens because we limit response sizes.

If it is ok with owners of the project, we would like to create free/unlimited endpoints for ponder users to use and list them in some documentation in ponder.

We already have endpoints we created for eth istanbul but they will be deprecated in next months so you can test with

https://eth.rpc.hypersync.xyz/ethistanbul
https://polygon.rpc.hypersync.xyz/ethistanbul
https://celo.rpc.hypersync.xyz/ethistanbul
etc..

https://envio.dev/hyper_rpc

[0xOlias]

Thanks for opening, moved to discussion. Could you share more details about the benchmark you ran that found your RPC to be 200x as fast as blastapi/chainstack premium plans? Most Ponder users today use Alchemy, including the Ponder dev team, so I'd also be curious to see a benchmark between HyperRPC and Alchemy.

[ozgrakkurt]

I just measured completion time of historical sync with the config I posted and with the same config using blast/chainstack enterprise plans. But they are using much smaller maxBlockRange because the providers don't allow such big ranges even for enterprise plan. Still when I run our rpc with similarly small batch size, our rpc is a lot faster.

In any case indexer finishes full historical sync with our rpc before it can reach 1% with other providers. For reference, I was using a beefy machine with nvme disk.

I don't have access to an alchemy account now but I would guess the results would be similar to blast/chainstack.

It should be easily replaceable by just running some indexers with alchemy and then our rpc. One note about this is that in order to use stateful methods (filter methods) with out rpc, you would need the stateful version of the endpoint as described in https://envio.dev/hyper_rpc

I admit 200x seems a little too much, also I admit when running ens our RPC and other RPCs have about the same performance. I am guessing this is due to sheer volume of data causing bottlenect in compression/decompression json serialisation/deserialisation etc. Our rpc excels at filtering so it will have a bigger edge when a contract doesn't have log so frequently like ens. I found FairToken to be a good average

[ozgrakkurt]

@0xOlias we also have an in-progress client for our hypersync api which is a much faster alternative to RPC. I am curious what you think about integrating this into ponder as an alternative to rpc.

This client has a highly improved binary transport and uses a more versatile api to get the data so it has natural pagination, better compression much faster encoding/decoding etc. For example it can index a contract that has 500k logs in about 5mins (this takes hours with RPC in my experience).

We are currently adding docs and implementing event decoding in this client so it will give already decoded events and remove the need for ethers or viem or similar library for decoding. It is written in rust and uses alloy for decoding and arrow columnar format for binary transport