apmpackage: support phases (warm, cold) in ILM in apm package

[azhurbilo]

right now, built-in ILM policies with APM package don't use Warm/Cold nodes type

it will be good to support them

ideally APM package should support all customisation which ApmServer k8s CRD have

for example

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: xxxx
spec:
  ...
  elasticsearchRef:
    name: xxx
  kibanaRef:
    name: xxx
  config:
    logging.level: warning
    logging.json: true
    setup:
      dashboards.enabled: true
      ilm:
        enabled: true
        overwrite: true
        policy_file: "ilm-policy.json"
      template:
        settings:
          index:
            number_of_shards: 8
            number_of_replicas: 2

[axw]

In the past we had a warm phase as part of the bundled ILM policy. We remove it, because it turned out not to be a great default: #6170. The issue is that, when combined with autoscaling, it would cause additional (warm) data nodes to be scaled up after some time elapsed.

I agree that it should be possible to customise the policy declaratively. The Elasticsearch team is working on a new, simpler, means of specifying a data stream's retention: https://www.elastic.co/guide/en/elasticsearch/reference/8.10/data-stream-lifecycle.html#data-stream-lifecycle. We will be using this in the future. I'm not currently sure what that means for warm/cold, so I'll leave this open to track.