You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DuckDb allows to read files from S3, that is pretty awesome.
In my setup the S3 table prefixes can change really often and variably. The single source of truth for where a specific table is located is contained in Glue Data Catalog, that allow to make the table querable via Athena/Trino/Spark/Redshift Spectrum.
It will be amazing to have duck db integration with a glue catalog (or catalog in general), that allow to pick automatically a specific s3 location for a specific table e.g.
select * from catalog('glue','my_db', 'my_table');
Under the hood I expect that based on what is contained in the catalog metadata information the right under the hood functions are invoked in duckdb e.g. read_parquet/ iceberg_scan
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
DuckDb allows to read files from S3, that is pretty awesome.
In my setup the S3 table prefixes can change really often and variably. The single source of truth for where a specific table is located is contained in Glue Data Catalog, that allow to make the table querable via Athena/Trino/Spark/Redshift Spectrum.
It will be amazing to have duck db integration with a glue catalog (or catalog in general), that allow to pick automatically a specific s3 location for a specific table e.g.
Under the hood I expect that based on what is contained in the catalog metadata information the right under the hood functions are invoked in duckdb e.g. read_parquet/ iceberg_scan
Beta Was this translation helpful? Give feedback.
All reactions