Native Support for Deltalake

[Mageswaran1989]

What is the recomended way to read DeltaLake tables?

Can we have a way to read DeltaLake tables? something like read_deltalake('....') natively part of the SQL syntax?

• DeltaLake Python bindings : https://pypi.org/project/deltalake/
• Example code which uses Deltalake to load the table as PyArrow parquet files and uses DuckDB to run some SQL on top of it

[hannes]

My personal take on this is to let Iceberg and DeltaLake fight to the death and then support the eventual winner. There is no recommended way currently unless @Alex-Monahan has something on this

[hkchindeko]

The guys over Delta Lake think this war is useless and I kind of agree with them. Starting with the upcoming Delta Lake 3.0, delta lake table will support at least in the beginning the Iceberg API in read only, and later support for Hudi will be added.
I guess that a good start in the right direction as we already have too many of such ridiculous format war or war in general to deal with.

[paradiddle-luuk]

Indeed, Delta introduced read only support for Iceberg clients (Universal Format). Native Delta support would prevent users from corrupting their Delta tables by manipulating the underlying parquet files directly. Analogues to what Fivetran presented at DuckCon 4 where they used DuckDB to update customer Iceberg data with incremental parquet files. I assume they do some custom metadata synchronization not to corrupt the customer data. I do not expect everyday users to do this reliably.

Personally, I would love to see DuckDB applicable across cloud data platforms by supporting Delta for MS Fabric and Databricks amongst others.

[Alex-Monahan]

PyArrow is the recommended method! You don't have to load it all into memory as a table, you can load it as a dataset as well (which DuckDB can read!).

https://github.com/delta-io/delta-rs/blob/main/python/tests/test_table_read.py

One of Arrow's goals is to make this kind if integration easier. All you need is format to Arrow (Delta to Arrow), then Arrow to engine (Arrow to DuckDB). So it's a really good option here.

[kmatt]

This would not apply to using Deltalake tables from the DuckDB CLI, correct?

[hkchindeko]

That the thing, using the Arrow as an exchange format between Delta and DuckDB doesn't work if you’re using the CLI. And in terms of experience is just a pain to have to do that while you can natively for instance access a SQLite database and table from DuckDB!
How this native support will work? It can use Arrow, at least we want it to be boringly invisible for us. If one likes the freedom of writing multiple line of code to go from Delta to DuckDB then crac crac… just continue to do so. It is not asked to remove such API.

[OneCyrus]

looks like databricks just added a uniform layer to Delta Lake 3.0 to be compatible with Hudi and Iceberg.

Delta Lake support would be great.
Also Microsoft decided to use Delta Lake as the one format to support.

https://www.databricks.com/company/newsroom/press-releases/announcing-delta-lake-30-new-universal-format-offers-automatic

[hkchindeko]

One more reason to add the support for Delta Lake. It’s kind of absurd to have 3 open format to do the same thing where more or less the only thing to differentiate them is the community size! Seriously, this is just ridiculous.

[djouallah]

@hkchindeko the issue is not philosophical, someone needs to do the work, databricks can sponsor DuckDBLabs to add an extension , they have plenty of money to do that

[kmatt]

Wonder if this could be used as a basis for Delta? The APIs are likely very different, but some aspects of the implementation would be similar: https://github.com/duckdblabs/duckdb_iceberg

[edgBR]

Polars bears can quack and have integration with delta:

https://www.edgarbahilo.com/poors-man-data-lake-with-polars-deltalake/

Also polars is going to support merge operations over delta in next release so potentially:

1. Read delta with polars.
2. Operate with DuckDB
3. Convert again to polars and write to delta.

[geekodour]

Another related demo: https://github.com/milicevica23/dbt-duckdb-delta-plugin-demo

[arghaffari]

@hannes We need it not only for direct use by DuckDB itself natively, but mostly to benefit from reading and writing Delta tables using third parties to which we do not have access under the hood, like Rill Data. This includes Metabase DuckDB connector visualizations, which utilize DuckDB. Our use isn't limited to scenarios where DuckDB supports it anyhow living off other tools; we need a native support.

[djouallah]

What is the recomended way to read DeltaLake tables?

Can we have a way to read DeltaLake tables? something like read_deltalake('....') natively part of the SQL syntax?

• DeltaLake Python bindings : https://pypi.org/project/deltalake/
• Example code which uses Deltalake to load the table as PyArrow parquet files and uses DuckDB to run some SQL on top of it

it is done https://github.com/duckdb/duckdb_delta