queue is put back before a job's resource allocating

[lowang-bh]

What happened:
In allocate action, a queue is put back to queues priority queue before the job's resource allocated, which will cause the queue is sorted not as expected as drf/proportion plugin really want to do.

volcano/pkg/scheduler/actions/allocate/allocate.go

Lines 155 to 164 in b53d6d8

	queues.Push(queue)
	if tasks.Empty() {
	continue
	}
	klog.V(3).Infof("Try to allocate resource to %d tasks of Job <%v/%v>",
	tasks.Len(), job.Namespace, job.Name)
	alloc.allocateResourcesForTasks(tasks, job, jobs, queue, allNodes)

What you expected to happen:

Put back the queue to priority queue after job's resource allocating finished.

How to reproduce it (as minimally and precisely as possible):

1. create two queues with same weight, queue-1 has a running job-a, queue-2 has no running jobs
2. submit a job-b whose resource request is large than job-a, to queue-1 and queue-2, and then submit another job-c whose request is same as job-a to queue-2; As queue-2's share is 0 and job in queue'2 will be pop from queues and allocated first (now all is as expected)
3. next step, job-b in queue-1 should be allocated before job-c in queue-2 (which is expected), because queue-1's share value is smaller than queue-2's .

Anything else we need to know?:

Environment:

• Volcano Version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[lowang-bh]

/good-first-issue

[volcano-sh-bot]

@lowang-bh:
This request has been marked as suitable for new contributors.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[panoswoo]

/assign

[panoswoo]

2. submit a job-b whose resource request is large than job-a, to queue-1 and queue-2,

I have a query. Is it possible to submit a job to two queues simultaneously?

[lowang-bh]

2. submit a job-b whose resource request is large than job-a, to queue-1 and queue-2,

I have a query. Is it possible to submit a job to two queues simultaneously?

You can use a yaml to submit, and increase the scheduling cycle time in start up parameters.

[Monokaix]

What's the expected result？allocate job-c first then job-b？

[lowang-bh]

What's the expected result？allocate job-c first then job-b？

Please see the detail testcase in PR #3413