Bump Kibana memory to 1500Mi

[kvalliyurnatt]

We are seeing nightly e2e test failures with Kibana being killed with OOM. Bumping the default Kibana memory to 1500Mi and it seems like Kibana memory consumption as increased. We suspect that the increased memory consumption is from moving to CgroupsV2. There is a bug
kubernetes/kubernetes#118916

[kvalliyurnatt]

buildkite test this -f p=gke,s=8.1.3 -m t=TestFleetMode,t=TestAPMKibanaAssociation,t=TestAgentVersionUpgradeToLatest8x,t=TestFleetAgentWithoutTLS

[thbkrkr]

LGTM

[kvalliyurnatt]

buildkite test this -f p=gke,s=8.1.3 -m t=TestFleetMode,t=TestAPMKibanaAssociation,t=TestAgentVersionUpgradeToLatest8x,t=TestFleetAgentWithoutTLS

[thbkrkr]

buildkite test this -f p=gke,s=8.1.3 -m t=TestFleetMode,t=TestAPMKibanaAssociation,t=TestAgentVersionUpgradeToLatest8x,t=TestFleetAgentWithoutTLS

[kvalliyurnatt]

buildkite test this -f p=gke,s=8.1.3 -m t=TestFleetMode,t=TestAPMKibanaAssociation,t=TestAgentVersionUpgradeToLatest8x,t=TestFleetAgentWithoutTLS

This results in too much memory being increased by default.

[thbkrkr]

Maybe a temporary stopgap action would be to start by just increasing the memory in the tests for kibana 8.1.x, something along these lines:

diff --git a/test/e2e/test/kibana/builder.go b/test/e2e/test/kibana/builder.go
index f7439def7..dce565004 100644
--- a/test/e2e/test/kibana/builder.go
+++ b/test/e2e/test/kibana/builder.go
@@ -8,6 +8,7 @@ import (
        "fmt"

        corev1 "k8s.io/api/core/v1"
+       "k8s.io/apimachinery/pkg/api/resource"
        metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
        "k8s.io/apimachinery/pkg/types"
        "k8s.io/apimachinery/pkg/util/rand"
@@ -69,7 +70,7 @@ func newBuilder(name, randSuffix string) Builder {
                Name:      name,
                Namespace: test.Ctx().ManagedNamespace(0),
        }
-       return Builder{
+       b := Builder{
                Kibana: kbv1.Kibana{
                        ObjectMeta: meta,
                        Spec: kbv1.KibanaSpec{
@@ -85,6 +86,20 @@ func newBuilder(name, randSuffix string) Builder {
                WithSuffix(randSuffix).
                WithLabel(run.TestNameLabel, name).
                WithPodLabel(run.TestNameLabel, name)
+
+       // bump Kibana memory in 8.1.x as we see abnormal memory usage, probably due to the move to cgroups v2
+       // (https://github.com/kubernetes/kubernetes/issues/118916)
+       ver := version.MustParse(test.Ctx().ElasticStackVersion)
+       if ver.GTE(version.MinFor(8, 1, 0)) && ver.LT(version.MinFor(8, 2, 0)) {
+               b = b.WithResources(corev1.ResourceRequirements{
+                       Requests: map[corev1.ResourceName]resource.Quantity{
+                               corev1.ResourceMemory: resource.MustParse("1500Mi"),
+                       },
+                       Limits: map[corev1.ResourceName]resource.Quantity{
+                               corev1.ResourceMemory: resource.MustParse("1500Mi"),
+                       }})
+       }
+       return b
 }

[naemono]

Maybe a temporary stopgap action would be to start by just increasing the memory in the tests for kibana 8.1.x, something along these lines:

I've done exactly what @thbkrkr suggested, which seems to make sense.

[thbkrkr]

Superseded by #7836.