Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to disable tombstones? (never delete data, infinite point-in-time history) #99

Open
danthegoodman1 opened this issue Aug 20, 2023 · 0 comments
Labels
question Further information is requested

Comments

@danthegoodman1
Copy link
Owner

danthegoodman1 commented Aug 20, 2023

If someone wants to never delete data (have full point-in-time access), then we can disable the creation of tombstones and never tombstone clean.

This means that merges create new files that omit what would normally be given a tombstone (log tombstones and file markers with tombstones), and log files are never cleaned up.

This might need to be paired with something like persistent reader instances that can just keep up with the state in the log from S3 changefeeds or something rather than having to read the log in every time.Or they can just list the log from the last file they know about (StartAfter key in list call) and materialize the log state locally or something. Maybe store materializations of the log back in S3 to start from as a checkpoint too, but that's outside the scope of what icedb would need to implement.

Or is this just entirely outside the scope of what icedb offers?

@danthegoodman1 danthegoodman1 added the question Further information is requested label Aug 20, 2023
@danthegoodman1 danthegoodman1 changed the title Option to disable tombstones? Option to disable tombstones? (never delete data, full point-in-time support) Aug 20, 2023
@danthegoodman1 danthegoodman1 changed the title Option to disable tombstones? (never delete data, full point-in-time support) Option to disable tombstones? (never delete data, infinite point-in-time history) Aug 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant