Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specific anomaly tuning for colums #125

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Specific anomaly tuning for colums #125

wants to merge 9 commits into from

Conversation

suelai
Copy link
Contributor

@suelai suelai commented Dec 27, 2023 •
edited

What

Have flexibility at the column level to define anomalies. 3 new features:

  • define anomaly_detector on the column level, which overrides the model anomaly_detector
  • have absolute thresholds for the values
  • control the alerts with the change_percentage

Resolves re-data/re-data#413

How

I am bringing a new model re_data_selected_columns that has the config per each column. When a column config is present, re_data_anomalies will read from re_data_selected_columns instead of re_data_selected.
Example of column anomaly config:

      - name: test_data
        config:
          re_data_monitored: true
          re_data_time_filter: random_date
          re_data_anomaly_detector:
            name: z_score
            threshold: 3
          re_data_metrics_groups:
            - table_metrics
          re_data_metrics:
            column:
              price:
                - avg:
                    re_data_anomaly_detector:
                      name: z_score
                      threshold: 0.1
                      direction: both
                    absolute_threshold:
                      threshold: 5200
                      direction: up
                    change_percentage:
                      threshold: 5
                      direction: up

              customer_id:
                - distinct_values
                - avg

In the case above, for the avg(price) we will consider a strict z_score of 0.1 which overrides the values 3 on the model. Moreover, the values should be higher than 5200 and the percentage of change has to be higher than 5% for this to become an anomaly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] thresholds on alerts in the specific metric
2 participants
@suelai @davidzajac1