CloudWatch: Add support for dynamic labels

[sunker]

Currently, the cloudwatch data source supports alias formatting that allows the user to substitute the returned metric label with region, period, metric dimension name etc. This substitution is performed in the plugin backend and the code for this is complicated. Now the GetMetricData api supports dynamic labels, so instead of doing the substitution in Grafana, we should let the api handle this for us.

Even though it will be complicated, we should be able to migrate any existing alias patterns into the dynamic label pattern.

Current Pattern	New patterns
{{region}}	${PROP('Region')}
Namespace: {{namespace}}, Instance: {{InstanceId}}	Namespace: ${PROP('namespace')}, Instance: ${PROP('Dim.InstanceId')}

One problem though is that the LabelOptions which allows you to specify a time zone for the dynamic label calculations is set on request level and not on query level. Currently, there's no way for query editors to define panel level settings, but possibly the plugin api can be extended to support that. If not, even though it's not ideal we can make this a grafana configuration under the aws section.

[sarahzinger]

Just wanted to add that I think when we do this we may also want to revisit some odd behavior we have when switching between Code/Builder for Metric Search. Some context can be seen here in this closed draft: #45432

[vickyyyyyyy]

Converted this to an issue: #43905

[sunker]

A few more findings/thoughts:

One problem though is that the LabelOptions which allows you to specify a time zone for the dynamic label calculations is set on request level and not on query level. Currently, there's no way for query editors to define panel level settings, but possibly the plugin api can be extended to support that. If not, even though it's not ideal we can make this a grafana configuration under the aws section.

It's only a few dynamic label values that are time based. Using the default timezone of Grafana should be sufficient for the first version of dynamic labels in Grafana.

Migrations
Since alert queries may be executed before the dashboard has been updated and saved in the frontend, the alias field needs to migrated from the as-is pattern to the dynamic label pattern both in the frontend and in the backend.

Breaking changes
According to the dynamic labels documentation, you can use up to five dynamic values per label. Also, the ${LABEL} placeholder can only be used once per label. Using more than that in the Alias field in Grafana is currently possible. In the unlikelihood that more than five templates are being used in the Alias field, we need to decide how to handle that. We could 1) migrate all of them to dynamic labels, which will lead to an error being returned from the GMD API or 2) only migrate the first five values in the Alias field and truncate the rest of the values.

Rough estimate for implementing this feature: 40 hours

[sunker]

Implementation steps:

• Frontend: Call migration from inside query editor UI
• Frontend: Call the migration from inside datasource.query method so that it will work also
  the first time opening the dashboard
• Backend: Pass new alias pattern to the GMD API. label field in &cloudwatch.MetricDataQuery {}
• Backend: Migrate old alias pattern to new alias pattern (for old alert rules)
• Backend: Remove all legacy alias formatting code

[fridgepoet]

For information we aim to get this into Grafana 9.0 #48434