Rework backend retries #4784
Merged
Rework backend retries #4784
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
This was referenced May 9, 2024
MichaelEischer
force-pushed
the
rework-backend-retries
branch
2 times, most recently
from
22fe701
to
a927b29
Compare
This simplifies finding the request in the log output that cause an operation to fail.
Depending on how long an operation takes to fail, the total retry duration can currently vary between 1.5 and 15 minutes. In particular for temporarily interrupted network connections, the former timeout is too short. Thus always use a limit of 15 minutes.
Previously, if an operation failed after 15 minutes, then it would never be retried. This means that large backend requests are more unreliable than smaller ones.
Retries in restic try to solve two main problems: - retry a temporarily failed operation - tolerate temporary network interruptions The first problem only requires a few retries, whereas the last one benefits primarily from spreading the requests over a longer duration. Increasing the default multiplier and the initial interval works for both cases. The first few retries only take a few seconds, while later retries quickly reach the maximum interval of one minute. This ensures that the total number of retries issued by restic will remain at around 21 retries for a 15 minute period. As the concurrency in restic is bounded, retries drastically reduce the number of requests sent to a backend. This helps to prevent overloading the backend.
The toplevel context in restic only canceled if the user interrupts a restic operation. If the network connection has failed this can require waiting the full retry duration of 15 minutes which is a bad user experience for interactive usage. Thus limit the delay to one minute in this case.
MichaelEischer
force-pushed
the
rework-backend-retries
branch
from
a927b29
to
e4a4808
Compare
This was referenced May 25, 2024
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR change? What problem does it solve?
Most restic commands are long running and would have to start over in case of a failed backend operation. Thus, retries are necessary to work reliably even if some requests fail.
From what I'm aware of, there are two main uses cases for retries in restic:
The PR makes several changes:
Was the change previously discussed in an issue or on the forum?
Replaces #2515.
Mentioned in #4627.
Checklist
changelog/unreleased/that describes the changes for our users (see template).gofmton the code in all commits.