Existing Parquet data

[loicalleyne]

I have data already in Parquet format, is there a way to load this data into IceDB's manifest?

[danthegoodman1]

I would probably just advise re-inserting it, but in theory you could look at manually constructing the log: https://github.com/danthegoodman1/icedb/blob/main/icedb/log.py

[loicalleyne]

My project's use case is to keep a sort of hot cache of logs for ad hoc queries, expecting up to have several TB of logs in rotation which is why I'd prefer to not have to remerge the files.
I found IceDB while researching Iceberg, very interested in its potential to solve some challenges we have.

[ivan-tkatchev]

We have our own ingestion pipeline that already writes files to S3. Re-inserting is not a real option.

The ability to do Clickhouse-style merges seems like a huge killer feature, but if IceDB can't build on top of existing data pipelines, then might as well just use Clickhouse instead.

[danthegoodman1]

You could swap out the log class for something that understands how to parse your existing data structure