does Delta reader project columns

[djouallah]

I am using this read a list of files from a delta table, but based on memory usage, it seems the reader is loading the whole table first ? is there a better way to read only a specific column?

existing_files = DeltaTable(Destination,storage_options=storage_options).to_pandas(columns=["file"])['file'].unique().tolist()

[ion-elgreco]

I am using this read a list of files from a delta table, but based on memory usage, it seems the reader is loading the whole table first ? is there a better way to read only a specific column?

existing_files = DeltaTable(Destination,storage_options=storage_options).to_pandas(columns=["file"])['file'].unique().tolist()

To_pandas materializes the whole table. You should project on the pyarrow_dataset and then collect.

[djouallah]

Thanks

[djouallah]

this looks a bit too verbose :)

DeltaTable(Destination,storage_options=storage_options).to_pyarrow_dataset().to_table(columns=['file']).to_pandas()['file'].unique().tolist()

[ion-elgreco]

pl.scan_delta(destination, storage_options=storage_options).select('file').collect().get_column('file')