Config.yaml ID relevancy of Unique ID in Dynatrace

[ZacBrumbaugh-KR]

Describe the bug
We are utilizing monaco V2 since last week and we have 2 clusters setup for high availability configured the same but in different regions. When utilizing Monaco v2, the ID in config.yaml seems to have more relevancy than the documentation suggests. The ID must be unique to create unique metric event alerts.

How to reproduce
Steps to reproduce the behavior:

1. Create a config.yaml with one alert

configs:
- id: foo
  type:
    api: anomaly-detection-metrics
  config:
    name: foo
    template: alert.json
    parameters:
      web_hook: <Injected_Environment_Value>
      region: <Injected_Environment_Value>

alert.json

"metricSelector": "some selector by cluster", 
"name": "Alert-{{ .region }} - {{ .web_hook }}", 
"description": "The {metricname} value was {alert_condition} normal behavior.", 
"severity": "CUSTOM_ALERT"

2. On each run of monaco (region 1 and region 2), make the name property unique in Anamoly detection metric event JSON

Expected behavior
When running monaco twice (one for each region) we should have two separate alerts, one for each region with corresponding alert names.

Actual behavior
The name of the alert is changed instead of a second alert being created.

[warber]

Hi @ZacBrumbaugh-KR

Unfortunately, the documentation does not reflect the fact that anomaly-detection-metrics is a "Non-unique name" type of API. This means that what you're trying to do cannot be supported by monaco as the name will not be recognized to be the unique identifier by the API. That's why monaco is using an ID that will resolve to the same value for your configuration resulting in an update of your existing configuration object. See here for further details. So this is not a "bug" of monaco but just an effect of how certain configuration APIs are working.

I suggest using multiple configurations to implement your use case.

configs:
  - id: foo
    type:
      api: anomaly-detection-metrics
    config:
      name: region-foo - webhook-foo
      template: alert.json
  - id: bar
    type:
      api: anomaly-detection-metrics
    config:
      name: region-bar - webhook-bar
      template: alert.json

You can still use environment variables if you make name a "compound" parameter if you want.
Further, if you want to control (via an environment variable) that not both configs will be deployed per run, you can do that via the skip field that can also read its value (true or false) from environment variables like an environment parameter.

[kbocock-krg]

Hi in the bug above and your suggestion to mitigate it could you please elaborate on the use of the config name property? When we get the json for anomaly detection it has a name field, but for some reason in Dynatrace UI the summary field has the json files name value. Very confusing. Another bug or as designed?

[warber]

This might be just the behavior of that endpoint. Whatever you have in the name field in JSON ends up as a summary in the UI. I can reproduce that on my DT environment. Note, that the "old" anomaly-detection-metrics API is replaced by the settings. So anomaly detection metrics events can also be created using the settings api (which I would strongly propose). The monaco CLI actually should also advise you to use the settings endpoint for it:

WARN  API for "anomaly-detection-metrics" is deprecated! Please consider migrating to "builtin:anomaly-detection.metric-events"!

you can take a look at how settings for anomaly detection metrics events look like just specifically downloading them from the environment e.g. using

monaco download --url  http://dt-url.com --token MONACO_TOKEN  --settings-schema builtin:anomaly-detection.metric-events

and use that instead.

[kbocock-krg]

Hi @warber when I run the command above with schema I get over 200 config downloads. This does not seem to be what I want. Currently we are downloading our configuration by entity using the docs here . Curious about the v1 in the api. Could you please share a clean example of using the settings-schema in monaco config.yaml with a corresponding .json for a metric event?

Ex. - We have no idea what we need for scope array. Type? etc.

type: settings: schema: builtin:anomaly-detection scope: [???????]

[warber]

Hi,

Ex. - We have no idea what we need for scope array. Type? etc.

This is why I've proposed to download a the corresponding settings objects via the monaco download ... command to have kind of an idea how the config for the corresponding settings schema looks like.

Could you please share a clean example of using the settings-schema in monaco config.yaml with a corresponding .json for a metric event?

Sure, the following is a minimal example of a metric event created with the settings 2.0 API:

config.yaml

configs:
- id: my-metric-event
  config:
    name: this-field-does-not-matter-for-settings
    template: metric-event.json
    skip: false
  type:
    settings:
      schema: builtin:anomaly-detection.metric-events
      schemaVersion: 1.0.14
      scope: environment

json template:

{
  "enabled": true,
  "summary": "My-metric-event",
  "queryDefinition": {
    "type": "METRIC_SELECTOR",
    "metricSelector": "builtin:apps.custom.reportedErrorCount"
  },
  "modelProperties": {
    "type": "STATIC_THRESHOLD",
    "threshold": 1.0,
    "alertOnNoData": false,
    "alertCondition": "ABOVE",
    "violatingSamples": 3,
    "samples": 5,
    "dealertingSamples": 5
  },
  "eventTemplate": {
    "title": "ad",
    "description": "The {metricname} value was {alert_condition} normal behavior.",
    "eventType": "CUSTOM_ALERT",
    "davisMerge": true,
    "metadata": []
  }
}

[kbocock-krg]

Hey thank for the example. We are getting some push back because we have about 20 teams using our 2 Dynatrace environments and seems that write access to settings is a security concern from a zero trust model. Is there a work around token or pat wise? Also is there a way to get this structure from a Dyantrace api or without creating it such that we can use it with monaco?

[kbocock-krg]

@warber any updates or suggestions?

[warber]

@UnseenWizzard

[UnseenWizzard]

Hi @kbocock-krg. You should be able to further limit access by applying IAM policies to settings schemas:
https://www.dynatrace.com/support/help/shortlink/settings20-landing#permissions-and-access

I don't know the details about this, but it should be possible to define specific policies like "read/write schema X,Y,Z only for management-zone A", and attach these policies to a user group. Which should carry to a users API access token.

[kbocock-krg]

@UnseenWizzard Is there a better way to not download all the metric events? Can we assign them to like a management zone and only get ours?

[UnseenWizzard]

Hi @kbocock-krg, that is currently not possible.

An improvement in version 2.2.0 will ensure that only modifiable Settings are downloaded (if the Dynatrace environment is new enough to provide that information). Maybe that helps a bit.

[kbocock-krg]

Thank you @UnseenWizzard. Wondering what is the correct way to ensure just for our team (given we share 2 environments) that we cleanup any metric events that are not in our monaco deployment? For such a cleanup assuming delete is used, how can we go about getting those events that are stale just in our space?

[UnseenWizzard]

Hi @kbocock-krg, there is currently no direct way with monaco to remove things that are not managed as config-as-code.
You would have to remove those on your own e.g. by creating a scrip that queries and removes unknown ones via the API.

There are currently no plans to introduce functionality like this in monaco - please feel free to share it in our community: https://community.dynatrace.com/ so other users can vote on ideas and product management can prioritize accordingly.