feat: have otel collect our docker stats #1516
Conversation
chesedo
changed the title
| @@ -68,17 +74,28 @@ exporters: | |||
| endpoint: "api.honeycomb.io:443" | |||
| headers: | |||
| "x-honeycomb-team": ${env:HONEYCOMB_API_KEY} | |||
| awss3/metrics: | |||
There was a problem hiding this comment.
How is the access to this bucket handled ?
There was a problem hiding this comment.
On AWS it will just use the instance metadata endpoint. The otel-s3-doc says it uses the default from the go-sdk which fallbacks to the "EC2 Instance Role Credentials" 1
There was a problem hiding this comment.
If it fallbacks to the EC2 role, then does it mean that the user application containers will inherit these permissions as well without having to do anything ?
There was a problem hiding this comment.
Oh dang, yes!! I recommend that we then rather find a way to install the collector directly on the VM 😄
| collection_interval: 1m | ||
| container_labels_to_metric_labels: | ||
| shuttle.idle_minutes: shuttle.idle_minutes | ||
| shuttle.project: shuttle.project_name |
There was a problem hiding this comment.
Should this be shuttle.project.name? And perhaps the same convention for the project id and idle minutes, but we haven't implemented that elsewhere in our code.
There was a problem hiding this comment.
Agreed, I'll make this change
|
By the way, do we know that this will not cause issues similar to what we saw in the past when we enabled Datadog docker stats collection? IIRC it was previously collecting every 10s, and this is once every minute. Are there other differences here? |
We don't know 😅 There are also no other differences. However, I just realized that the old collector of 10s might be active. Hopefully it is not active and the cause of our Docker issues 😅 |
|
Btw, this is a bit on ice while I'm working on the RDS subscriptions |
Description of change
Cause otel to push docker metrics to a S3 bucket
How has this been tested? (if applicable)
Will on staging