Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(python, rust): cleanup expired logs post-commit hook #2459

Open
wants to merge 15 commits into
base: main
Choose a base branch
from

Conversation

ion-elgreco
Copy link
Collaborator

@ion-elgreco ion-elgreco commented Apr 28, 2024

Description

This adds the cleanup expired logs post-commit hook. Tables by default have delta.enableExpiredLogCleanup = true, and delta.logRetentionDuration = interval 30 days. So moving forward those logs will be automatically cleaned up.

If users set delta.enableExpiredLogCleanup = false during table creation, the log clean up will never be executed in the post-commit. When cleanup_expired_logs in the post-commit hook properties is passed, then that take precedence over the table config.

Additionally to most operations you can now pass in Python the PostCommitHookProperties to control the post commit hook, only really needed for advanced use cases, where you want to disable creating checkpoints across the board or override the log clean up.

@Blajda this is a follow up PR of the hooks, will look into creating an auto_vacuum_hook afterwards

@ion-elgreco ion-elgreco requested a review from Blajda April 28, 2024 12:08
@github-actions github-actions bot added binding/python Issues for the Python package binding/rust Issues for the Rust crate labels Apr 28, 2024
@ion-elgreco ion-elgreco enabled auto-merge (squash) April 28, 2024 12:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/python Issues for the Python package binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants