Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(spans): Add topic definition for buffered-segment topic #237

Merged

Conversation

shruthilayaj
Copy link
Member

@shruthilayaj shruthilayaj commented Mar 13, 2024

We're building a new pipeline to support performance issues with span streaming,
at a high level, it looks like this
Screenshot 2024-03-13 at 5 00 35 PM

the buffered-segments topic will expect to receive a list of spans. As of this
moment, these spans will have the exact same schema as ones we consume
from the snuba-spans kafka topic, the SpanEvent definition is just copied from
the snuba-span schema. we might deserialize the spans to drop fields before pushing to redis
as a memory optimization at which point the span schemas will diverge.

Copy link

github-actions bot commented Mar 13, 2024

versions in use:

The following repositories use one of the schemas you are editing. It is recommended to roll out schema changes in small PRs, meaning that if those used versions lag behind the latest, it is probably best to update those services before rolling out your change.

  • getsentry/sentry: pip:sentry-kafka-schemas==0.1.61

latest version: 0.1.61

benign changes

schemas/snuba-spans.v1.schema.json

  • {"path": "", "change": {"TypeAdd": {"added": "string"}}}
    
  • {"path": "", "change": {"TypeAdd": {"added": "number"}}}
    
  • {"path": "", "change": {"TypeAdd": {"added": "integer"}}}
    
  • {"path": "", "change": {"TypeAdd": {"added": "array"}}}
    
  • {"path": "", "change": {"TypeAdd": {"added": "boolean"}}}
    
  • {"path": "", "change": {"TypeAdd": {"added": "null"}}}
    
  • Removed a property _metrics_summary from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "_metrics_summary"}}}
    
  • Removed a property description from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "description"}}}
    
  • Removed a property duration_ms from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "duration_ms"}}}
    
  • Removed a property event_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "event_id"}}}
    
  • Removed a property exclusive_time_ms from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "exclusive_time_ms"}}}
    
  • Removed a property is_segment from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "is_segment"}}}
    
  • Removed a property measurements from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "measurements"}}}
    
  • Removed a property organization_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "organization_id"}}}
    
  • Removed a property parent_span_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "parent_span_id"}}}
    
  • Removed a property profile_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "profile_id"}}}
    
  • Removed a property project_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "project_id"}}}
    
  • Removed a property received from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "received"}}}
    
  • Removed a property retention_days from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "retention_days"}}}
    
  • Removed a property segment_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "segment_id"}}}
    
  • Removed a property sentry_tags from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "sentry_tags"}}}
    
  • Removed a property span_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "span_id"}}}
    
  • Removed a property start_timestamp_ms from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "start_timestamp_ms"}}}
    
  • Removed a property tags from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "tags"}}}
    
  • Removed a property trace_id from ``, but it is still accepted via additionalProperties=true

    {"path": "", "change": {"PropertyRemove": {"lhs_additional_properties": true, "removed": "trace_id"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "duration_ms"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "exclusive_time_ms"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "is_segment"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "project_id"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "received"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "retention_days"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "span_id"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "start_timestamp_ms"}}}
    
  • {"path": "", "change": {"RequiredRemove": {"property": "trace_id"}}}
    

✅ This PR should be safe to roll out to consumers first. Make sure to bump
the library in the following repos first:

getsentry/sentry

...then in the other repos:

getsentry/sentry

Take a look at the README for how to release a new version of sentry-kafka-schemas.

@shruthilayaj shruthilayaj merged commit eaaacca into main Mar 18, 2024
14 checks passed
@shruthilayaj shruthilayaj deleted the shruthi/feat/add-topic-definition-for-buffered-segment branch March 18, 2024 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants