Aggregated index namespace matching main indices

[b2ronn]

Elasticsearch/Kibana/Fleet/Apm - 8.10.2.

Issue Description:
I am using the reroute processor to change namespaces according to the "labels.project" label. However, aggregated indices remain in the default namespace.

Feature Request:
I propose either considering this behavior as a potential bug or adding a feature that allows for the automatic relocation of aggregated indices to the same namespaces as their corresponding main indices when using the reroute processor. This enhancement would streamline the management of indices and improve overall data organization within the Elasticsearch ecosystem.

[axw]

@b2ronn is the "project" label configured as a global label? If so, then that field should also be available in aggregated metric documents, and you can add a reroute processor to the ingest pipeline for the metrics data streams too.

I agree that it would ideally be automatically synchronised. The problem is that the metrics are pre-aggregated by APM Server rather than Elasticsearch, and Elasticsearch handles the rerouting. We're thinking about how to generalise pre-aggregation, potentially moving it to Elasticsearch in the future, and will keep this in mind.

[b2ronn]

I forgot to clarify, "labels.project" it is global labels for APM RUM, and global labels are not present in aggregated metrics (the last version of the APM server, where global labels were in aggregated indexes, was 8.7.0)

[axw]

For a long term solution, we will need to have a way to configure which labels should be included as dimensions in aggregated metrics. Then you would configure APM Server to say the "project" label should be a dimension, and it would be available in the metrics documents for routing.

Until then, one workaround would be to encode the project name in the service name, and then remove it in an ingest pipeline for all traces, metrics, and logs data streams. For example, if you set the service name to service-project, then you could use a dissect ingest processor to update service.name to "service" and set labels.project to "project" like this:

{
    "dissect": {
      "field": "service.name",
      "pattern": "%{service.name}-%{labels.project}",
      "ignore_missing": true,
      "ignore_failure": true
    }
}