Implement the integration tests for requeueing scenarios

[sanposhiho]

/sig scheduling
/priority important-soon
/kind feature

Part of: #122284 (comment).

---

We have few tests to check the scheduler's requeueing scenario in the integration test.
We should have the one so that we can catch a bug like #122284, which only happens the scheduling queue is actually working.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sanposhiho]

/assign @carlory @sanposhiho

[sanposhiho]

@carlory

A basic thought that I have now is that we have to cover the most common scenario only in an integration test, we don't have to cover all scenarios per each plugin.
So, for example, for nodeaffinity, we can just have two scenarios in which Pod is supposed to be requeued to activeQ/backoffQ (= node satisfying nodeaffinity is created), vs is supposed not to be requeued (= node unrelated to nodeaffinity is created). No need to cover other minor scenarios (e.g., a scenario addedNodeSelector comes into play, etc)

Plus, it'd be great if we could have a scenario where preCheck scenario is covered.
For example:

1. Pod with nodeaffinity is created
2. (1) is rejected because no node satisfies nodeaffinity
3. we create a new node that satisfies nodeaffinity, but has taint.
4. nodeadd event from (3) should be filtered out by precheck.
5. we remove taint from node.
6. nodeupdate event from (5) requeue the pod.

[carlory]

For example:
...

@sanposhiho There's already an existing intergation test. it follows similar steps.

https://github.com/carlory/kubernetes/blob/d36a7089cd9fa2abedd62f0964a42c21f837c53a/test/integration/scheduler/scheduler_test.go#L529

[sanposhiho]

Then, we may want to refactor that test to be a more general one.
What I want to make sure (eventually) is that all plugins take preCheck into consideration in EventsToRegisrter (+ QHint). That TestNodeEvents makes sure only noderesourcefit currently.

[sanposhiho]

As part of #122292, I implemented the e2e test, which would be a good starting point of this.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[sanposhiho]

/remove-lifecycle stale

#122292 is implemented, we have to cultivate the test cases there for existing QHints.