Flaky UT TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled

[Rajalakshmi-Girish]

Which jobs are flaking?

https://prow.ppc64le-cloud.cis.ibm.net/job-history/gs/ppc64le-kubernetes/logs/periodic-kubernetes-unit-test-ppc64le

Which tests are flaking?

TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled of pkg/controller/statefulset/stateful_set_control_test.go

Since when has it been flaking?

Have started observing more flakes recently since last week.

Testgrid link

https://testgrid.k8s.io/ibm-k8s-unit-tests-ppc64le#periodic-k8s-unit-tests-ppc64le

Reason for failure (if possible)

No response

Anything else we need to know?

Able to reproduce using stress tool.

[root@maxwells1 kubernetes]# /root/go/bin/stress ./statefulset.test -test.run TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled
5s: 0 runs so far, 0 failures
10s: 0 runs so far, 0 failures
--------
--------
--------
8m25s: 108 runs so far, 0 failures
8m30s: 110 runs so far, 0 failures

/tmp/go-stress-20240425T015624-1279903570
--- FAIL: TestStatefulSetControl (0.16s)
    --- FAIL: TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled (0.08s)
        stateful_set_control_test.go:526: StatefulSetControl did not return InternalError, found %!s(<nil>)
E0425 02:04:54.791294 2073044 stateful_set.go:374] "Unhandled Error" err="user error: more than one StatefulSet is selecting pods with labels: map[apps.kubernetes.io/pod-index:0 foo:bar statefulset.kubernetes.io/pod-name:foo-0]. Sets: [foo foo2]" logger="UnhandledError"
E0425 02:04:54.803356 2073044 stateful_set.go:374] "Unhandled Error" err="user error: more than one StatefulSet is selecting pods with labels: map[apps.kubernetes.io/pod-index:0 foo:bar statefulset.kubernetes.io/pod-name:foo-0]. Sets: [foo foo2]" logger="UnhandledError"
E0425 02:04:54.806908 2073044 stateful_set.go:374] "Unhandled Error" err="user error: more than one StatefulSet is selecting pods with labels: map[apps.kubernetes.io/pod-index:0 foo:bar statefulset.kubernetes.io/pod-name:foo-0]. Sets: [foo foo2]" logger="UnhandledError"
E0425 02:04:54.814283 2073044 stateful_set.go:374] "Unhandled Error" err="user error: more than one StatefulSet is selecting pods with labels: map[apps.kubernetes.io/pod-index:0 foo:bar statefulset.kubernetes.io/pod-name:foo-0]. Sets: [foo foo2]" logger="UnhandledError"
FAIL


ERROR: exit status 1

8m35s: 112 runs so far, 1 failures (0.89%)

Relevant SIG(s)

/sig testing
/sig apps

[atiratree]

/triage accepted
/assign

[atiratree]

opened a PR with a fix: #124624