Skip to content
This repository has been archived by the owner on Apr 2, 2024. It is now read-only.

Adds support for Prometheus metric rollups #1799

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Adds support for Prometheus metric rollups #1799

wants to merge 13 commits into from

Commits on Jan 11, 2023

  1. Adds support to create/delete metric rollups via dataset-config. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

This commit adds support for creation or deletion of metric rollups by reading
the dataset-config file. The dataset config file contains a new field `downsample:`
which supports 2 options: `automatic` & `resolution`.

If `automatic` is false, then
connector updates in the database so that `_prom_catalog.scan_for_new_rollups` does not
create any new rollups (Caggs).

`resolution` is a comma separated list of resolutions for downsampling, in a way that
label:resolution:retention is the format that is needed.
Example: `short:5m:90d,long:1h:365d`
-> The resolution works in a strict manner. Its aim is to make sure that the database contains
exactly those resolutions that are mentioned in the dataset config. Like, if the database
already contains `short` and `long` downsampled data and the dataset is supplied with `short:5m:90d,very_long:1d:365d`
then `long` downsampled case is deleted and `very_long` is created.
-> `resolution` is applied if `automatic` is `true`. If automatic is not given in dataset config, it defaults to `true`
based on our plan to keep downsampling applied as a default setting.
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    297fbbd View commit details
    Browse the repository at this point in the history

  2. Update resolution config and add E2E tests for creation/deletion of r… 

    …ollups.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    a9fc048 View commit details
    Browse the repository at this point in the history

  3. Refactor DayDuration -> day.Duation 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    7125609 View commit details
    Browse the repository at this point in the history

  4. Update tests related to register_metric_view(). 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    f2bdb38 View commit details
    Browse the repository at this point in the history

  5. Adds support to instrument metric-rollups behaviour 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    e7e515c View commit details
    Browse the repository at this point in the history

  6. Update rollups creation/deletion/updation using dataset-config. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    432d5fe View commit details
    Browse the repository at this point in the history

  7. squashable: Resolve conflicts 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    6ca6007 View commit details
    Browse the repository at this point in the history

  8. Fix deepcopy-gen errors. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    de54527 View commit details
    Browse the repository at this point in the history

  9. squashable: fix tests. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c3fb475 View commit details
    Browse the repository at this point in the history

  10. Adds support for new downsampling UX. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>

This commit implements the update UX for creating downsampled data
for Prometheus metrics.

Now, downsampling can be done by
```
metrics:
  downsampling: 5m:30d,1h:365d
```

The above dataset-config setting will create 2 downsampling, ds_5m
(interval 5m & retention 30d) and ds_1h (interval 1h & retention 365d).

If an entry is removed, like
```
metrics:
  downsampling: 1h:365d
```

Then we disable the ds_5m downsampling from refreshing. Deletion can only
happen via SQL, `CALL _prom_catalog.delete_downsampling(<schema_name>)`. This
keeps the UX clean and simple.

If the removed dowmsapling is applied again later on, we simply enable the refreshing.

This also means that this commit adds ability to enable or disable downsampling as well.
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4c23c32 View commit details
    Browse the repository at this point in the history

  11. Adds support to query default downsampling views in absence of __colu… 

    …mn__.

Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4c4ca47 View commit details
    Browse the repository at this point in the history

  12. Support for apply_downsample_config() 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    e8c4cdd View commit details
    Browse the repository at this point in the history

  13. Add timeout for trying apply_downsamplg_config lock. 

    Signed-off-by: Harkishen-Singh <harkishensingh@hotmail.com>
    @Harkishen-Singh
    Harkishen-Singh committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c334d34 View commit details
    Browse the repository at this point in the history