[EPIC] Metric Stats

[Dav1dde]

Description

We need outcomes for DDM / custom metrics.

Outcomes for Metrics are generally a tricky problem for several reasons. Metrics are aggregated in multiple stages, SDKs, customer Relays, pop Relays and processing Relays, which makes giving an accurate number of volume hard. On top of that, volume is just a small scaling factor we have to consider, a much bigger factor is the cardinality of a metric.

Ideally outcomes can capture the volume and the cardinality of a metric.

Outcomes should tell us the volume of a single metric (defined by its MRI) and its cardinality per hour.

Our current outcomes cannot capture this information, we need a new mechanism to collect metric outcomes.

Requirements

• Indefinite (?) retention
• Fast enough to query for billing purposes
• Volume and Cardinality needs to be represented
• Sentry UI needs access to show the user
• Billing needs access for billing
• Bizops needs access to process with their own pipelines (can they query snuba instead of reading the topic?)

Why not use Outcomes?

• Outcomes currently don't provide a way to group by metric name (incl. metric type and namespace).
• Cardinality needs to be max() aggregated not sum()'ed

Quantity / Volume

We want to determine the volume of metrics received by the first layer of our infrastructure (PoP-Relays). Client side aggregated metrics are counted with a quantity of 1.

For example: If Relay receives 500 statsd items for a single metric per hour, this metric would be considered to have a volume/quantity of 500 metrics per hour.

Cardinality

We are interested in the cardinality of a single metric (MRI) per hour.

Cardinality can be queried from storage or collected through the Relay cardinality limiter.

Metric Stats Namespaces

Volume: c:metric_stats/volume@none
Tags:

• mri: Metric Name/MRI: <type>:<namespace>/<name>[@<unit>]
• mri.type: Metric Type
• mri.namespace: Metric namespace (extracted from the MRI)
• outcome.id: Outcome ID, metric outcomes share the same numeric outcome ID with regular outcomes.
• outcome.reason: Optional machine readable, free-form reason code.

Cardinality: g:metric_stats/cardinality@none

• mri: Metric Name/MRI: <type>:<namespace>/<name>[@<unit>]
• mri.type: Metric Type
• mri.namespace: Metric namespace (extracted from the MRI)
• cardinality.limit: The cardinality limit id which generated this report.
• cardinality.window: Cardinality window size in seconds.
• cardinality.scope: Cardinality scope (name, project, organization).

If cardinality is tracked by a project or organization the mri* tags will not be present.

Examples

Payloads/Metrics emitted from Relay into the generic metrics topic.

Volume - Accepted

{
   "org_id":0,
   "project_id":42,
   "name":"c:metric_stats/volume@none",
   "type":"c",
   "value":2.0,
   "timestamp":1712223931,
   "tags":{
        "mri": "d:custom/foo@none",
        "mri.type": "d",
        "mri.namespace": "custom",
        "outcome.id": "0",
   },
   "retention_days":90
}

Cardinality by Name

{
   "org_id":0,
   "project_id":42,
   "name":"g:metric_stats/cardinality@none",
   "type":"g",
   "value":{
      "last":2.0,
      "min":2.0,
      "max":2.0,
      "sum":2.0,
      "count":1
   },
   "timestamp":1712223931,
   "tags":{
        "mri": "s:custom/bar@none",
        "mri.type": "s",
        "mri.namespace": "custom",
        "cardinality.limit": "custom-limit-with-some-id",
        "cardinality.scope": "name",
        "cardinality.window": "3600",
   },
   "retention_days":90
}

Cardinality by Type

{
   "org_id":0,
   "project_id":42,
   "name":"g:metric_stats/cardinality@none",
   "type":"g",
   "value":{
      "last":2.0,
      "min":2.0,
      "max":2.0,
      "sum":2.0,
      "count":1
   },
   "timestamp":1712223931,
   "tags":{
        "mri.type": "s",
        "mri.namespace": "custom",
        "cardinality.limit": "custom-limit-with-some-id",
        "cardinality.scope": "type",
        "cardinality.window": "3600",
   },
   "retention_days":90
}

Milestone 1 ✅ - Volume / Happy Path

Implement the happy path ("accepted") for volume metric stats: c:metric_stats/volume@none.

Tasks

Beta Give feedback

1. Create Kafka Topic -> OPS-5396
2. feat(metric-outcomes): Add initial topic and schema sentry-kafka-schemas#231
   +0
   
3. feat(metrics): Add metadata for buckets #3254
   +0
   
4. feat(metric-stats): Emit accepted volume #3281
   +0
   
5. ref(relay): Add metric stats rollout rate to global config sentry#67221
   Scope: Backend +1
   
6. https://github.com/getsentry/sentry-options-automator/pull/929
7. https://github.com/getsentry/ops/pull/9746
8. https://github.com/getsentry/sentry-options-automator/pull/931
9. fix(metric-stats): Prevent producing multiple metric stats for the same bucket #3290
   +0
   
10. ref(metrics): Add metric_stats usecase id to entity mapping sentry#67596
    Scope: Backend +1
    

Milestone 2 ✅ - Cardinality / Happy Path

Implement the happy path ("accepted") for cardinality metric stats: g:metric_stats/cardinality@none.

Tasks

Beta Give feedback

1. feat(cardinality): Implement name based cardinality limits #3313
   +0
   
2. ref(cardinality): Pipeline Redis script invocations #3321
   +0
   
3. feat(cardinality): Implement cardinality reporting #3342
   +0
   
4. feat(metric-stats): Report cardinality to metric stats #3360
   +0
   
5. ref(relay): Make cardinality limits more configurable sentry#68158
   Scope: Backend +1
   
6. https://github.com/getsentry/sentry-options-automator/pull/1033

Milestone 2.5 ✅ - Cardinality by Minute and Project

Tasks

Beta Give feedback

1. feat(metric-stats): Add additional tags to metric stats metrics #3438
   +0
   
2. feat(metric-stats): Add cardinality.{limit,scope} tags to strings indexer sentry#68891
   Scope: Backend +1
   
3. feat(metric-stats): Implement per type cardinality limits #3437
   +0
   
4. Enable per minute cardinality

Milestone 3 ✅ - Negative Outcomes

Tasks

Beta Give feedback

1. feat(ddm): Add negative outcomes to metric stats in pops #3452
   +0
2. feat(metric-stats): Add cardinality limited outcome id #3389
   +0
   
3. feat(metric-stats): Add cardinality limited outcome id sentry#68422
   Scope: Backend Scope: Frontend +2
   
4. Emit Negative Outcomes: Cardinality Limits #3433
   +0
   
5. Emit Negative Outcomes: Rate Limits #3434
   +0
   
6. Emit Negative Outcomes: Throughput Limiter #3435
   +0
   
7. Emit Negative Outcomes: Denied Metrics #3482
   +0
   
8. feat(metric-stats): Emit negative outcomes of metrics in metric stats #3436
   +0
   

Milestone 4 - Finishing Touches

Tasks

Beta Give feedback

1. Filter metric_namespaces when accepting metrics
2. Don't use bucket views, they use a shared metadata
3. Change Bucket Interval for metric_stats namespace in ST
4. https://github.com/getsentry/ops/pull/10345
5. Configurable/Longer Retention
6. Add Metric Type Tag
7. Check for missing integration tests (prime candidate: rate limits -> negative outcomes)
8. Come up with a better way to reject metrics than our current util function
9. Figure out BucketViews and volume
10. check_buckets drops buckets without reject_metrics
11. Cardinality Limits are overlapping, if a limit rejects a metric, all other limits are still increased