DataFusion write to delta

[dhruvils414]

Hello

I would like to write into Delta lake via DataFusion just like spark.

Future support

Append, overwrite, Merge into

[MrPowers]

Let's make sure to get the docs page updated too when/if this gets completed: https://delta-io.github.io/delta-rs/integrations/delta-lake-datafusion/

[ion-elgreco]

We already use datafusion in delta-rs, so not sure what you mean?

[MrPowers]

@ion-elgreco - any chance we can expose the syntax to write to a Delta tables with DataFusion in the docs, so it's easy for me to learn how to do it?

[ion-elgreco]

@MrPowers what docs are we talking about here? Because we use datafusion on the rust side, but this is not documented well. All of the writing is just dispatched to rust from pyuthon

[dhruvils414]

I think it didn't support directly. Based Article 1, We need convert to pyarrow before we read from datafusion.

from datafusion import SessionContext
from deltalake import DeltaTable

ctx = SessionContext()
table = DeltaTable("G1_1e9_1e2_0_0")
ctx.register_dataset("my_delta_table", table.to_pyarrow_dataset())

Article 1 https://delta-io.github.io/delta-rs/integrations/delta-lake-datafusion/#delta-lake-performance-benefits-for-datafusion-users