"not enough cache capacity" when restoring from MinIO (S3)

[dev-rowbot]

Output of restic version

restic 0.9.4 compiled with go1.11.4 on linux/amd64

How did you run restic exactly?

I have run restic directly and using the docker image, the result is the same.

docker run --rm --name restic-restore-eventstore \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e RESTIC_REPOSITORY=${RESTIC_REPOSITORY} \
    -e RESTIC_PASSWORD=${RESTIC_PASSWORD} \
    -v $PWD/eventstore-data:/eventstore-data \
    restic/restic --no-cache \
    restore $1 \
    --target /eventstore-data

What backend/server/service did you use to store the repository?

Environment

• restic docker container with a mounted volume to restore in to
• Minio cluster (4 x nodes) behind an nginx reverse proxy / load balancer

I am restoring “eventstore” data from an S3 bucket which consists of

• 232 “chunk” files all around 257 MB in size
• A couple “chk” files which are tiny
• index files which are between 23 and 367 MB

Expected behavior

Restic should be able to restore this data with no issues

Actual behavior

The output always includes a few of these errors:

ignoring error for /data/backup/eventstore-volume/very_long_sha/_data/chunk-000000.000000: not enough cache capacity: requested 9976743, available 6836375

The files that have errors are incomplete and the data is unusable.

Steps to reproduce the behavior

Based on the comments from the forum: https://forum.restic.net/t/not-enough-cache-capacity-when-restoring-from-minio-s3/1588

I assume you would need a repository that contains pack files with an average size much large than the provided 5MB.

Do you have any idea what may have caused this?

The averagePackSize in filerestorer.go is too small.

Do you have an idea how to solve the issue?

To quote @cdhowie

I think it would make more sense for the pack cache to use either the maximum theoretically possible pack size, or look for the largest pack in the repository and use that value.

Alternatively there should be a retry built into the filerestorer to handle the situation where reserving space fails momentarily.

Did restic help you or made you happy in any way?

Yes :-)

[ifedorenko]

This was already reported in #2074 (among other things) and should be fixed as part of #2195. The simplified restorer logic does not need to reorder downloaded packs, so there is no packs cache. I am keeping this bug open just to make the error easier to find.

[nmartinez-nimeops]

Hi, Same issue with 0.9.5 :(

[schmitch]

the same issue happens on 0.9.6 on windows with a file store.

[rawtaz]

Closing this now that #2195 has been merged. Feel perfectly free to reopen it if the specifics that this issue is about has not been resolved. If there's still improvements to be made that are not of the type discussed in this issue, please open a new issue. Thanks!