StarRocks Internal Table Format advantages over Apache Iceberg

[alberttwong]

The StarRocks internal format will be better than query on Iceberg. The internal format has more optimization, like late materialization, partition pruning, more predicates pushed down.

Using Shared Nothing Architecture, this StarRocks Table format is stored on disk. Using Shared Data architecture, this StarRocks Table format is stored in S3.

[mcgray]

Can you elaborate, please? What is this internal format specifically?

[thevijayshekhawat]

Also, can this storage format be used to store data in Object storage like S3 and GCS and query directly from Starrocks?

[atifiu]

Internal format means SR proprietary format like oracle or sql server.

[alberttwong]

I updated the answer to include object store.

[thevijayshekhawat]

Thanks, @alberttwong , Follow-up question, is this format available with StarRocks 3.1? Also, Can you share some links to documentation about how this internal table format can be used to create tables on Object Store? We are currently testing StarRocks with IceBerg and it would be great to compare the performance across different table formats.

[alberttwong]

This is the default when you create "data" in starrocks. It's listed as "internal catalog".

[alberttwong]

moved to https://forum.starrocks.io/t/starrocks-internal-table-format-advantages-over-apache-iceberg/121