feat(similarity-embedding): Create backfill script for inserting records #68466
+1,122
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Scope: Backend
Bundle ReportChanges will increase total bundle size by 9.79kB ⬆️
|
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
| and event.group.data["metadata"].get("has_embeddings_record_v1") | ||
| ) | ||
| ): | ||
| grouping_info = get_grouping_info( |
There was a problem hiding this comment.
follow up with any ramifications if grouping config changes
wedamija
reviewed
Apr 9, 2024
jangjodi
commented
Apr 9, 2024
jangjodi
changed the title
jangjodi
changed the title
JoshFerge
reviewed
May 9, 2024
|
|
||
| group_id_batch = list(group_id_message_batch.keys()) | ||
| time_now = datetime.now() | ||
| events_entity = Entity("events", alias="events") |
There was a problem hiding this comment.
follow up: should this be the errors entity?
JoshFerge
reviewed
May 9, 2024
|
|
||
| redis_client = redis.redis_clusters.get(settings.SENTRY_MONITORS_REDIS_CLUSTER) | ||
| if last_processed_id is None: | ||
| last_processed_id = int(redis_client.get(LAST_PROCESSED_REDIS_KEY) or 0) |
There was a problem hiding this comment.
follow up: use project_id in this redis key
There was a problem hiding this comment.
follow up: should we set a TTL on this redis key?
JoshFerge
approved these changes
May 9, 2024
JoshFerge
reviewed
May 14, 2024
src/sentry/features/temporary.py
Outdated
| @@ -276,6 +276,7 @@ def register_temporary_features(manager: FeatureManager): | |||
| manager.add("projects:similarity-embeddings", ProjectFeature, FeatureHandlerStrategy.INTERNAL) | |||
| manager.add("projects:similarity-embeddings-grouping", ProjectFeature, FeatureHandlerStrategy.INTERNAL) | |||
| manager.add("projects:similarity-embeddings-metadata", ProjectFeature, FeatureHandlerStrategy.INTERNAL) | |||
| manager.add("projects:similarity-embeddings-backfill", ProjectFeature, FeatureHandlerStrategy.INTERNAL) | |||
There was a problem hiding this comment.
do we want to use the FeatureHandlerStrategy.OPTIONS? or what kind of feature handler strategy were you thinking
There was a problem hiding this comment.
ah yes good catch, I changed it to options
JoshFerge
approved these changes
May 14, 2024
JoshFerge
added a commit
that referenced
this pull request
May 14, 2024
) Create endpoint to call the backfill script [here](#68466) --------- Co-authored-by: Josh Ferge <josh.ferge@sentry.io> Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com> Co-authored-by: Katie Byers <katie.byers@sentry.io>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Create script to backfill seer grouping records
Create function to call seer bulk insert endpoint (https://github.com/getsentry/seer/pull/480) in batches of 20 groups with a 10 second timeout
Add seer bulk insert function
Modify get_grouping_info function to take in an optional event
The script will be triggered using an endpoint, which will be in a follow-up PR. This is because the script will only be run for the s4s project for now, thus we only need to run it once.
closes https://github.com/getsentry/getsentry/issues/13420