Performance test (against BQ)

[danthegoodman1]

Github events has 232M rows and lots of example queries: https://ghe.clickhouse.tech/

https://clickhouse.com/docs/en/getting-started/example-datasets/nyc-taxi has 3B rows but is smaller in size can do this too and much less complex schema

Should run the same queries against. We can leave bigquery for 5+ hours (maybe even days) to merge, and fully merge up icedb. Record the performance of inserting and merging too, probably far faster than bigquery.

Then record the same quieries. The data scanned, the storage price, and query times, and the total query price.

For IceDB we will run it all on a single node, the largest ec2 instance we can get, connected to an s3 vpc endpoint with no auth.

[danthegoodman1]

With IceDB should test:

• ClickHouse just reading all current data files before merge
• ^^ through s3 proxy
• merge and read raw data files
• ^^ through s3 proxy