fix: disable SchedulerQueueingHints feature flag by default

[sanposhiho]

What type of PR is this?

/kind bug
/priority critical-urgent
/triage accepted

What this PR does / why we need it:

disable the feature gate of SchedulerQueueingHints by default.

---

We discussed at slack:
https://kubernetes.slack.com/archives/C2C40FMNF/p1702438650646789

And decided we'll make the feature gate for QHint disable by default. (Actually, there was a discussion of disablement for this before because we observed a memory consumption issue: #120622)
#122284 reinforces that idea. We'll make it enable by default once all issues here + memory consumption issue are addressed.

Which issue(s) this PR fixes:

Temporal fix for #122284

Special notes for your reviewer:

/cc @kubernetes/sig-scheduling-approvers
/cc @kerthcet
/cc @liggitt

Does this PR introduce a user-facing change?

The SchedulerQueueingHints feature gate moved back to disabled by default, due to unexpected memory use increases and to mitigate regressions found in some plugins' QueueingHint.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Please note that we're already in Test Freeze for the release-1.29 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.29.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Tue Dec 12 22:21:49 UTC 2023.

[k8s-ci-robot]

@sanposhiho: GitHub didn't allow me to request PR reviews from the following users: kubernetes/sig-scheduling-approvers.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

What type of PR is this?

/kind bug
/priority critical-urgent
/triage accepted

What this PR does / why we need it:

disable the feature gate of SchedulerQueueingHints by default.

Which issue(s) this PR fixes:

Temporal fix for #122284

Special notes for your reviewer:

/cc @kubernetes/sig-scheduling-approvers
/cc @kerthcet
/cc @liggitt

Does this PR introduce a user-facing change?

SchedulerQueueingHints feature flag is disabled by default because we found a critical bug in some plugins' QueueingHint.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

/sig scheduling
/kind regression
/assign @alculquicondor
/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 2d338f8273d497d4ef6526c89610f1b6208a1bfa

[liggitt]

/milestone v1.29
/approve
/hold for release team coordination

[liggitt]

/hold

There are unit tests that assume the gate is enabled. The minimal change is to enable the gate specifically for those tests:

func TestPriorityQueue_MoveAllToActiveOrBackoffQueue(t *testing.T) {
	c := testingclock.NewFakeClock(time.Now())
	logger, ctx := ktesting.NewTestContext(t)
	ctx, cancel := context.WithCancel(ctx)
	defer cancel()
	m := makeEmptyQueueingHintMapPerProfile()
+	defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, true)()

func TestRequeueByPermitRejection(t *testing.T) {
+	defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.SchedulerQueueingHints, true)()

[Priyankasaggu11929]

LGTM from 1.29 Release Team, per the discussion in https://kubernetes.slack.com/archives/C2C40FMNF/p1702438650646789

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, sanposhiho

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• pkg/features/OWNERS [liggitt]
• pkg/scheduler/OWNERS [liggitt,sanposhiho]
• test/integration/scheduler/OWNERS [liggitt,sanposhiho]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[liggitt]

/lgtm
/hold cancel

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: f47771b4b06efe66b6bfae8b0265e0606ae77d72

[sanposhiho]

@liggitt Thanks. Test_buildQueueingHintMap checks the feature gate actually, and I confirmed it passes without a change.

Running tool: /home/gitpod/go/bin/go test -timeout 30s -run ^Test_buildQueueingHintMap$ k8s.io/kubernetes/pkg/scheduler

ok  	k8s.io/kubernetes/pkg/scheduler	0.310s

Added SetFeatureGateDuringTest for the other two.

[ahg-g]

/lgtm

[sanposhiho]

Talked with @liggitt, we'll cherry-pick this for v1.28.

[kerthcet]

/lgtm

LGTM for quick fix, but seems we need to do more work later like in TestPriorityQueue_MoveAllToActiveOrBackoffQueue, as feature gate is always enabled.

[pacoxu]

we need to do more work later like in TestPriorityQueue_MoveAllToActiveOrBackoffQueue, as feature gate is always enabled.

Better to open an issue to track that.

[dims]

thanks @sanposhiho