DeltaTable.to_pyarrow_dataset() is much slower than DeltaTable.files()

[adriangb]

I'm seeing ~500ms for the former and ~1ms for the later (running locally, absolute numbers shouldn't matter)

@ion-elgreco is it possible the table state is not being properly re-used or similar?

[ion-elgreco]

@adriangb both do quiete different things

.files() returns the filepaths of the table in a list, while to_pyarrow_dataset also has to construct dataset fragments of each file while parsing the table stats into pyarrow.compute expressions

[ion-elgreco]

@wjones127 as far as I can see thats creating an iteration of the cached data in the state

[ion-elgreco]

@adriangb @wjones127 it's here:

https://github.com/delta-io/delta-rs/blob/main/python%2Fdeltalake%2Ftable.py#L1097-L1099

There we build a new table instead of reusing the state. I'll see if I can make a fix

[ion-elgreco]

@adriangb can you try this one out: #2485

[adriangb]

(1000, 2)
3
Time taken on : 0.07 ms
Time taken on : 2.83 ms
3
Time taken on gs://: 0.08 ms
Time taken on gs://: 0.99 ms

Great stuff!

[adriangb]

Would love a release with that fix 🙏🏻