Feature Request: S3 metadata functions

[danthegoodman1]

I would be useful to be able to have functions to operate on the metadata of S3 files (HEAD requests). For example being able to aggregate the file sizes of an S3 glob pattern.

This is probably possible (probably omits parquet overhead) with parquet using the total_compressed_size function, but limited for other file formats.

Fwiw you can get the file name now with the filename = true option:

SELECT *
FROM read_parquet('s3://bucket/*.parquet', filename = true);

[Mause]

Are you looking for the results of read_blob/read_text maybe? They return the file contents, but also the stat data

[danthegoodman1]

Wouldn't that actually read the whole file though? That would be really wasteful

[danthegoodman1]

Or is there an optimization for if only size is selected?

[danthegoodman1]

LMAO yes https://duckdb.org/docs/guides/file_formats/read_file.html#support-for-projection-pushdown

[carlopi]

It's not super clear in the docs that no content implies only metadata is fetched, if you have ideas on how to make it more clear help is welcome!

[danthegoodman1]

I spent a while looking and searching. Maybe this could be referenced in the httpfs and s3 pages?

[danthegoodman1]

Closing as can extract from existing functions