Add delete command #1
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, Litestream would avoid closing the SQLite3 connection in order to ensure that the WAL file was not cleaned up by the database if it was the last connection. This commit changes the behavior by introducing a file control call to perform the same action. This allows us to close the database file normally in all cases.
Workaround for benbjohnson#436
A failsafe to force WAL truncation if somehow between normal checkpoints the WAL grows to extreme sizes. Default is around 200 megabytes with a 4k page size.
I recently noticed that the cost for ListBucket calls was increasing for an application that was using Litestream. After investigating it seemed that the bucket had retained the entire history of data, while Litestream was continually logging that it was deleting the same data: ``` 2022-10-30T12:00:27Z (s3): wal segmented deleted before 0792d3393bf79ced/00000233: n=1428 <snip> 2022-10-30T13:00:24Z (s3): wal segmented deleted before 0792d3393bf79ced/00000233: n=1428 ``` This is occuring because the DeleteObjects call is a batch item, that returns the individual object deletion errors in the response[1]. The S3 replica client discards the response, and only handles errors in the original API call. I had a misconfigured IAM policy that meant all deletes were failing, but this never actually bubbled up as a real error. To fix this, I added a check for the response body to handle any errors the operation might have encountered. Because this may include a large number of errors (in this case 1428 of them), the output is summarized to avoid an overly large error message. When items are not found, they will not return an error[2] - they will still be marked as deleted, so this change should be in-line with the original intentions of this code. 1: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_Example_2 2: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
Allows removing a single generation or all files from replica path.
hifi
force-pushed
the
beeper
branch
3 times, most recently
from
7517879
to
9e2d067
Compare
hifi
force-pushed
the
beeper
branch
2 times, most recently
from
ae9c078
to
ed7121b
Compare
hifi
force-pushed
the
beeper
branch
2 times, most recently
from
5fb407c
to
222708d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP