feat(metric-outcomes): Add initial topic and schema #231
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| { | ||
| "timestamp": 1709812800, | ||
| "org_id": 1, | ||
| "project_id": 1, | ||
| "mri": "d:custom/request_duration@seconds", | ||
| "outcome": 2, | ||
| "reason": "foobar", | ||
| "quantity": 42, | ||
| "cardinality": 1337 | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "timestamp": 1709812800, | ||
| "org_id": 1, | ||
| "project_id": 1, | ||
| "mri": "c:custom/clicks@none", | ||
| "outcome": 0, | ||
| "cardinality": 1337 | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| { | ||
| "timestamp": 1709812800, | ||
| "org_id": 1, | ||
| "project_id": 1, | ||
| "mri": "c:custom/clicks@none", | ||
| "outcome": 0, | ||
| "quantity": 1337 | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| { | ||
| "$schema": "http://json-schema.org/draft-07/schema#", | ||
| "title": "metrics-outcome", | ||
| "$ref": "#/definitions/MetricsOutcome", | ||
| "definitions": { | ||
| "MetricsOutcome": { | ||
| "properties": { | ||
| "timestamp": { | ||
| "description": "The UNIX timestamp in seconds when the outcome was created, optionally truncated to the last hour.", | ||
| "$ref": "#/definitions/U64" | ||
| }, | ||
| "org_id": { | ||
| "description": "The organization for which this outcome is being sent.", | ||
| "$ref": "#/definitions/U64" | ||
| }, | ||
| "project_id": { | ||
| "description": "The project for which this outcome is being sent.", | ||
| "$ref": "#/definitions/U64" | ||
| }, | ||
| "mri": { | ||
| "description": "Valid metric resource identifier `<type>:<namespace>/<name>[@<unit>]` for which the outcome is being sent.", | ||
| "type": "string" | ||
| }, | ||
| "outcome": { | ||
| "description": "Outcome ID, metric outcomes share the same numeric outcome ID with regular outcomes.", | ||
| "$ref": "#/definitions/U64" | ||
|
There was a problem hiding this comment. I wonder if we should limit this to existing outcome IDs, or deliberately leave it open. I tend towards leaving it open. There was a problem hiding this comment. I decided to follow the outcomes schema and leave it open. |
||
| }, | ||
| "reason": { | ||
| "description": "Optional machine readable, free-form reason code.", | ||
| "type": ["string", "null"] | ||
| }, | ||
| "quantity": { | ||
| "description": "Amount of metric buckets accepted by Relay (volume).", | ||
| "type": "integer" | ||
| }, | ||
| "cardinality": { | ||
| "description": "Maximum observed cardinality of the metric.", | ||
| "type": "integer" | ||
Dav1dde marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| }, | ||
| "required": ["timestamp", "org_id", "project_id", "mri", "outcome"] | ||
| }, | ||
| "U64": { | ||
| "type": "integer", | ||
| "minimum": 0, | ||
| "maximum": 18446744073709551615 | ||
| } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| topic: outcomes-metrics | ||
| pipeline: outcomes | ||
|
There was a problem hiding this comment. I think this is used primarily to identify a pipeline logically. @lynnagara and @untitaker can provide more context. There was a problem hiding this comment. I assume the metrics outcomes eventually makes it's way to the shared outcomes table in clickhouse with all the other outcomes? So this pipeline is probably fine There was a problem hiding this comment. It will be parallel to the current outcomes with their own schema and from the looks of it in a separate clickhouse cluster. There was a problem hiding this comment. If everything is fully separate, I think you can change this to |
||
| description: Outcomes for metrics sent to Sentry | ||
| services: | ||
| producers: | ||
| - getsentry/relay | ||
| consumers: | ||
| - getsentry/snuba | ||
| - getsentry/super-big-consumers | ||
|
Comment on lines
+5
to
+9
There was a problem hiding this comment. Is this pipeline meant to bypass the current metrics outcomes pipeline that takes data in after the indexer ? If yes can we get rid of the old one There was a problem hiding this comment. I am not familiar with any (other) metrics outcomes pipeline, so I assume the answer is yes. There was a problem hiding this comment. This is the billing metrics consumer https://github.com/getsentry/sentry/blob/0730d277c3ad16c4c8aaebd7b2edd51f08819dea/src/sentry/ingest/billing_metrics_consumer.py#L41 I cannot find the design docs anymore. |
||
| schemas: | ||
| - version: 1 | ||
| compatibility_mode: none | ||
| type: json | ||
| resource: outcomes-metrics.v1.schema.json | ||
| examples: | ||
| - outcomes-metrics/1 | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To what extent is the
outcomes-metricsschema actually different fromoutcomesor is it mostly the same?In Snuba, all of
outcomes,outcomes-billingandloadbalancer-outcomesall share the exact same consumer code, and we just re-use the same schema in those scenarios. In the past we generally focused on having schemas reflect the minimum set of requirements for a consumer to work correctly. If the consumer is the same one, it might be simpler to use the existing outcomes schema for the new topic rather than maintain a separate one. On the other hand if the consumer code is different, I think a different schema here makes sense.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The biggest differences to the existing outcomes are:
We considered trying to make this work with the current outcomes but came to the conclusion that the metric outcomes are too different and we'd rather have a separate concept just for metrics.