Support fetching packages from S3 buckets #215
Conversation
|
This is cool :) can we get this reviewed ? |
| @@ -375,7 +375,7 @@ func (a *APK) InitKeyring(ctx context.Context, keyFiles, extraKeyFiles []string) | |||
|
|
|||
| var asURL *url.URL | |||
| var err error | |||
| if strings.HasPrefix(element, "https://") { | |||
| if strings.HasPrefix(element, "https://") || strings.HasPrefix(element, "s3://") { | |||
There was a problem hiding this comment.
WDYT about https://gocloud.dev/howto/blob/ instead here? This could give us a standard interface for non-S3 APIs. AFAIK GCS's S3 compatibility is "best effort" which means not great.
There was a problem hiding this comment.
I think that would be fine to swap to if you'd prefer, though as I understand it pretty all blob storage implement the basic S3 APIs with full compatibility, it's as soon as you start using the more complex features like ACLs, CORS, etc that you start to run into issues. Given this is only ever calling GetObject I wouldn't expect to run into those issues.
| @@ -393,6 +393,15 @@ func (a *APK) InitKeyring(ctx context.Context, keyFiles, extraKeyFiles []string) | |||
| if err != nil { | |||
| return fmt.Errorf("failed to read apk key: %w", err) | |||
| } | |||
| case "s3": //nolint:goconst | |||
There was a problem hiding this comment.
This seems interesting, but I'm a bit worried about adding features that might move compatibility away from apk add and other non-go-apk tools.
It's not blocking, we're building/using a fork of The use-case is we build APK packages internally (using We experimented with using mountpoint-s3 and mounting the bucket as a |
| @@ -988,6 +997,8 @@ func packageAsURI(pkg InstallablePackage) (uri.URI, error) { | |||
|
|
|||
| if strings.HasPrefix(u, "https://") { | |||
| return uri.Parse(u) | |||
| } else if strings.HasPrefix(u, "s3://") { | |||
| return uri.URI(u), nil | |||
There was a problem hiding this comment.
It's worth calling out that this line has a bit of code-smell, uri.Parse will fail for the s3 schema: https://go.dev/play/p/PlTtp5oC7S3
This line currently just directly casts to the underlying uri.URI type to avoid this which isn't great
|
@imjasonh An alternative design that might be more palatable is to adopt the pattern used by Terraform via https://github.com/hashicorp/go-getter (or maybe even just swap to using this library) where a custom prefix of That could allow us to still specify the raw |
Adds experimental support for fetching packages directly from S3 bucket via a
s3://<bucket>/<optional prefix>/repository URL. It will load any credentials from the default AWS credentials chain. While this uses the AWS S3 SDK, it can be pointed at any S3 compatible API (ex: GCS, Cloudflare R2, etc) via the endpoint url configuration feature.