You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If someone wants to never delete data (have full point-in-time access), then we can disable the creation of tombstones and never tombstone clean.
This means that merges create new files that omit what would normally be given a tombstone (log tombstones and file markers with tombstones), and log files are never cleaned up.
This might need to be paired with something like persistent reader instances that can just keep up with the state in the log from S3 changefeeds or something rather than having to read the log in every time.Or they can just list the log from the last file they know about (StartAfter key in list call) and materialize the log state locally or something. Maybe store materializations of the log back in S3 to start from as a checkpoint too, but that's outside the scope of what icedb would need to implement.
Or is this just entirely outside the scope of what icedb offers?
The text was updated successfully, but these errors were encountered:
danthegoodman1
changed the title
Option to disable tombstones?
Option to disable tombstones? (never delete data, full point-in-time support)
Aug 20, 2023
danthegoodman1
changed the title
Option to disable tombstones? (never delete data, full point-in-time support)
Option to disable tombstones? (never delete data, infinite point-in-time history)
Aug 20, 2023
If someone wants to never delete data (have full point-in-time access), then we can disable the creation of tombstones and never tombstone clean.
This means that merges create new files that omit what would normally be given a tombstone (log tombstones and file markers with tombstones), and log files are never cleaned up.
This might need to be paired with something like persistent reader instances that can just keep up with the state in the log from S3 changefeeds or something rather than having to read the log in every time.Or they can just list the log from the last file they know about (StartAfter key in list call) and materialize the log state locally or something. Maybe store materializations of the log back in S3 to start from as a checkpoint too, but that's outside the scope of what icedb would need to implement.
Or is this just entirely outside the scope of what icedb offers?
The text was updated successfully, but these errors were encountered: