Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unit and integration tests for the vacuum command #227

Closed
rtyler opened this issue Apr 29, 2021 · 2 comments · May be fixed by #1817
Closed

Unit and integration tests for the vacuum command #227

rtyler opened this issue Apr 29, 2021 · 2 comments · May be fixed by #1817
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers

Comments

@rtyler
Copy link
Member

rtyler commented Apr 29, 2021

Our preliminary support for vacuum would benefit from some more unit and integration tests to ensure that it's only deleting the right files whenever the vacuum is invoked.
@rtyler rtyler added enhancement New feature or request binding/rust Issues for the Rust crate labels Apr 29, 2021
@fvaleye fvaleye mentioned this issue May 17, 2021
Closed
@fvaleye fvaleye mentioned this issue May 24, 2021
Merged
@danmx
Copy link

danmx commented Sep 5, 2022

Referencing @mrk-its comment from #97 so it's not lost.

@rtyler @fvaleye It looks like there are still two serious issues with vacuum implementation:

* vacuum lists all files in dataset using `StorageBackend.list_objs`. The problem is this function returns all files (including these in subdirectories) on `s3` backend and `gcs` backend (althrough I'm not sure about gcs). On `file` and `azure` backends this function  lists only first-level files (without recursing to subdirectories).

* vacuum ignores files not referenced by delta log at all (so not included on DeltaTableState.files() and DeltaTableState.all_tombstones() lists).

@MrPowers MrPowers added the good first issue Good for newcomers label Nov 2, 2023
@Jan-Schweizer Jan-Schweizer mentioned this issue Nov 7, 2023
Open
@ion-elgreco
Copy link
Collaborator

This should be well tested at this point, also with introduction of VACUUM START, VACUUM END, I added a couple tests on the Python side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
binding/rust Issues for the Rust crate enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: add handling for unmanaged files to vacuum command Jan-Schweizer/delta-rs
4 participants
@rtyler @MrPowers @danmx @ion-elgreco