We will also need to create an S3 bucket along with creds, docs here: litestream.io/guides/s3

We should be able to use a k8s ServiceAccount associated with an IAM role to grant the pod access to the S3 bucket, instead of creating long-lived credentials and encoding them as a secret. I can set up that role/service account + the S3 bucket.

One thing that's missing is the secrets, is there a process for generating the sealed-secrets.yaml file?

I think using a service account instead of storing IAM credentials eliminates the need for this, but let us know if you still need this info and we can get you it.

We should be able to use a k8s ServiceAccount associated with an IAM role to grant the pod access to the S3 bucket, instead of creating long-lived credentials and encoding them as a secret. I can set up that role/service account + the S3 bucket.

Thanks for setting this up!

I think using a service account instead of storing IAM credentials eliminates the need for this, but let us know if you still need this info and we can get you it.

We need to somehow store the Gitlab API token besides the S3 creds, if secrets are the right way to do this

I updated some of the files in response to your comments, I appreciate the feedback.

One question, I'm seeing this line on some of the deployments:

nodeSelector:
  spack.io/node-pool: base

should I add that to this PR as well?

We need to somehow store the Gitlab API token besides the S3 creds, if secrets are the right way to do this

I added that here - 655968a. Let me know if you need a higher scope/more scopes than read_api.

One question, I'm seeing this line on some of the deployments:

nodeSelector:
  spack.io/node-pool: base

should I add that to this PR as well?

Yes, that should be added (that's needed for karpenter to schedule the pods correctly)

Thanks for adding the API token, that level of access is perfect. Once the webhook is set up, we'll need to also store the webhook secret

Hey @mvandenburgh, this is almost ready. Would it be possible to add a webhook for all job status changes? I am not sure what the FQDN will be inside the cluster, but it should be pointed to /v1/collect. Thanks

@cmelone I added that webhook to #827. Let me know once this PR is ready, and we can merge both this and #827 at the same time.

Thanks! is there a webhook secret that gets set? If not, I'll need to remove that as a required env variable in the app

Thanks! is there a webhook secret that gets set? If not, I'll need to remove that as a required env variable in the app

GitLab doesn't provide one, no. And since the service isn't exposed publicly, it is safe to allow unauthenticated requests.

Thanks! is there a webhook secret that gets set? If not, I'll need to remove that as a required env variable in the app

GitLab doesn't provide one, no. And since the service isn't exposed publicly, it is safe to allow unauthenticated requests.

GitLab does support/recommends using a secret token. Eventhough this won't be exposed publicly, I'd vote we should probably use one just in case someone were to get into the network and start feeding bad allocations back to gantry.

When creating a webhook on the GitLab side you provide the secret token as an argument as shown here. We can use any randomly generated string and then pass it in as a secret to the Kubernetes configuration.

Ah, thanks @alecbcs - I wasn't aware of that. I added the secret token to the webhook and also encoded it as a k8s secret here - 6d151b1

Sweet! Thanks @mvandenburgh :D

Thanks! I'll do some last minute checks and mark this as ready.

Once deployed, would it be possible to get CLI access to the pods via kubectl? I imagine that there will be some unexpected issues and it might be easier for me to debug this way. If this doesn't fit with the way you guys access the cluster, no worries

Should be ready for merging now

Once deployed, would it be possible to get CLI access to the pods via kubectl? I imagine that there will be some unexpected issues and it might be easier for me to debug this way. If this doesn't fit with the way you guys access the cluster, no worries

Do you have an IAM user on AWS?

Do you have an IAM user on AWS?

Not at the moment

I created @cmelone an IAM user and submitted #849 to add him to the cluster access list.