restic is "impossible" to use with an anonymous s3://... server

[ramses0]

Output of restic version

Reproduced using: restic 0.16.4-dev (compiled manually) compiled with go1.21.5 on linux/amd64

First discovered: restic 0.11. compiled with go1.15.9 on linux/amd64 (apt install restic)

What backend/service did you use to store the repository?

SeaweedFS / S3

Problem description / Steps to reproduce

Detailed information: seaweedfs/seaweedfs#5300

The restic-relevant summary is as follows:

• restic supports --repo s3://...
• restic uses minio/v7 which supports NewCredentialChain
• restic uses the following credentials chain
  • &credentials.Static
  • &credentials.EnvAWS
  • &credentials.Static
  • &credentials.EnvMinio
  • &credentials.FileAWSCredentials
  • &credentials.FileMinioClient
  • &credentials.IAM
• ...surprisingly, there exists in the world more s3-compatible servers than EC2-instances-running-in-AWS

The following interaction between restic, minio-go, and seaweedfs is dramatically bad:

minio-go: https://github.com/minio/minio-go/blob/a0865af930cd1cb96688574b1d89085bc0f2e371/pkg/credentials/iam_aws.go#L339

// getCredentials - obtains the credentials from the IAM role name associated with
// the current EC2 service.
//
// If the credentials cannot be found, or there is an error
// reading the response an error will be returned.
func getCredentials(client *http.Client, endpoint string) (ec2RoleCredRespBody, error) {
	if endpoint == "" {
		endpoint = DefaultIAMRoleEndpoint // => const DefaultIAMRoleEndpoint = "http://169.254.169.254"
	}

restic: https://github.com/restic/restic/blob/v0.16.4/internal/backend/s3/s3.go#L60

	// Chains all credential types, in the following order:
...snip...
	//  - IAM profile based credentials. (performs an HTTP
	//    call to a pre-defined endpoint, only valid inside
	//    configured ec2 instances)
	creds := credentials.NewChainCredentials([]credentials.Provider{
...snip...
		&credentials.IAM{
			Client: &http.Client{
				Transport: http.DefaultTransport,
			},
		},
	})

...and results in a time differential of ~0.75s => 8m30s for backing up a single file as described in intimate detail here: seaweedfs/seaweedfs#5300 (comment)

time ./restic -r "s3:http://localhost:8333/resticbucket" backup /etc/motd --password-command "echo 'test'"

Removing the &credentials.IAM{...} block results in "expected" behaviour of ~1s to backup a single 1 kb file. WITH the IAM{...} block, the time balloons to 8m30s (for the same single 1 kb file), or additionally, using export AWS_... with appropriate secrets reduces the time to the expected ~1s.

Critically: The backup "task succeeds unsuccessfully" (instead of: "task failed successfully") in that it actually DOES appear to successfully backup the file, however it takes an impossibly long time to do so.

Expected behavior

I know the authors/maintainers are quite security-sensitive when it comes to (eg) requiring a backup password.

I've (hopefully obviously) done quite a bit of investigation as to the root cause, and had some time to think about some possible alternative behaviours.

No-op: Somehow inject a getCredentials() => &credentials.REFUSE_ANONYMOUS{...} && panic() type thing

• this would avoid falling into the "interminably slow, but eventually succeeds" tarpit of minio reaching out to 169... addresses

Option to disable IAM: --s3-iam/--no-s3-iam type setting.

• in order to avoid breaking changes, --s3-iam would have be be "enabled by default"
• there's a strong case to be made for having an option to disable: --no-s3-iam
• ...in actuality, there is more non-AWS compute in the world than there are EC2 instances in AWS
• ...in actuality, there are more backup sources outside of AWS/EC2 than inside AWS/EC2
• Because credentials.IAM{...} can effectively "break restic", it seems like there should be a way to disable it!

Explicitly opt-in to anonymous: --allow-s3-anonymous/--no-allow-s3-anonymous type setting.

• given the "security-sensitive" posture, maybe default to: &credentials.REFUSE_ANONYMOUS{...} && panic()
• ...and then --allow-s3-anonymous to explicitly enable passwordless writes to an S3 bucket
• ...there's a strong case to be made for disabling anonymous writes by default (which would have prevented the "slow tarpit that eventually succeeds" use case)
• The reasons for disallowing anonymous writes by default is:
  • backups may run unattended and extreme slowness may not be noticed
  • backups may accidentally written (even though encrypted) to a world-writeable (world-readable?) s3 bucket

restic detects --running-on-aws-ec2? and caches that somehow before forcefully attempting to invoke IAM endpoints?

• restic is in the wrong here, as evidenced by the comment:
• // performs an HTTP call to a pre-defined endpoint, only valid inside configured ec2 instances
• restic shouldn't do ec2-specific things unless explicitly requested to!
• if restic is going to do ec2-things by default, then it should do them correctly

minio-go better detects/caches 169... timeouts

• ...so that the entire golang + minio-go ecosystem hopefully avoids this type of slow behaviour
• This also seems "ideal", but they might have their own p.o.v.'s of why they're trying to request credentials for every access
• (eg: temporarily flaky 169... endpoint, etc...)

minio-go better supports &credentials.ExplicitAnonymous{...}

• https://github.com/minio/minio-go/blob/a0865af930cd1cb96688574b1d89085bc0f2e371/pkg/credentials/chain.go#L66
• // Always prioritize non-anonymous providers, if any.
• ...it explicitly de-prioritizes "anonymous access"
• ...this makes it "tricky" to try anonymous access before "falling down the credential chain" with &credentials.IAM{...}
• BECAUSE of this deprioritization and because restic added IAM to the chain, there's no way to get to anonymous without first passing IAM

restic does something different w.r.t. the &http.Client{ Transport: http.DefaultTransport } with Timeout: ...

• ...I tried messing around with Timeout: ... but didn't have much luck in reducing the "slow tarpit" behaviour

I'd love to hear author/maintainer feedback on the above.

IMHO the simplest answer is to provide a --disable-s3-iam-credentials which removes (or doesn't insert) the &credentials.IAM{...} provider. It's relatively minimal code changes and "punts" the question of anonymous access down to the minio-go layer, so restic isn't on the hook for supporting anonymous, but instead minio is attempting anonymous by default. PLUS, it's super-super-super-strange that restic is effectively invoking random network requests to random hosts without explicitly requesting that behaviour.

Actual behavior

restic is too slow when using s3://... and anonymous write access.

Do you have any idea what may have caused this?

Yes. :-) See: seaweedfs/seaweedfs#5300

Did restic help you today? Did it make you happy in any way?

Yes! First of all by mentioning Mr. Hess. I've had the pleasure of interacting with him a few times, and of course indirectly through his open-source work. Believe it or not, back in the ~2000's, I'd shared a compliment to one of the debian project maintainers privately (I believe it was regarding x.org packages with the general topic: "Thanks for building and maintaining these packages b/c I certainly couldn't figure it out, and it's a great time-saver!") which was then re-shared back out with the comment: "See, it's not all complaints!" For posterity: https://lists.debian.org/debian-x/2000/11/msg00416.html

Second of all, it's been "fun" (haha) to dig through and understand the issue w/ restic, seaweedfs, and minio. There's perverse pleasure in finding the exact spot(s) in the codebase that are causing trouble.

Thirdly, restic is currently "in the lead" as backup tool that I'm trying to get working. I've really appreciated the fuse-mount support for working with individual files. It's helpful and convenient to be able to dig in and inspect certain directories to make sure things are "behaving as expected" as well as to be able to quickly pop around and find something you might be looking for.

[vchirikov]

Ran into this issue too, it was very confusing because restic worked fine on windows and wasn't usable on linux, big thanks @ramses0 for the investigation

Unfortunately, there is no response from maintainers yet..