Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework repository.StreamPacks & better restorer error handling #4605

Merged

Commits on Apr 21, 2024

  1. repository: Add blob loading fallback to LoadBlobsFromPack

    Try to retrieve individual blobs via LoadBlob if streaming did not work.
    MichaelEischer committed Apr 21, 2024
    Configuration menu
    Copy the full SHA
    621012d View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. repository: streamPack: replace streaming with chunked download

    Due to the interface of streamPack, we cannot guarantee that operations
    progress fast enough that the underlying connections remains open. This
    introduces partial failures which massively complicate the error
    handling.
    
    Switch to a simpler approach that retrieves the pack in chunks of 32MB.
    If a blob is larger than this limit, then it is downloaded separately.
    
    To avoid multiple copies in memory, an auxiliary interface
    `discardReader` is introduced that allows directly accessing the
    downloaded byte slices, while still supporting the streaming used by the
    `check` command.
    MichaelEischer committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    666a0b0 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cf700d8 View commit details
    Browse the repository at this point in the history
  3. repository: streamPack: separate requests for gap larger than 1MB

    With most cloud providers, traffic is much more expensive than API
    calls. Thus slightly bias streamPack towards a bit more API calls in
    exchange for slightly less traffic.
    MichaelEischer committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    20d8eed View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. add changelog

    MichaelEischer committed May 1, 2024
    Configuration menu
    Copy the full SHA
    676f0dc View commit details
    Browse the repository at this point in the history