volcano controller repeatedly create pod after job had deleted

[Wang-Kai]

What happened:

I created a job which has just one task and one pod. After the job, podgroup and pod are running phase, i update pod and then delete job almost at the same time. The operation may cause the job to be deleted at etcd, but it is still present at volcano controller cache. So, the K8s garbage-collector deleted the pod due to its owner having deleted it, and the volcano controller created the pod due to job alway in cache.

As the user can see, pods are repeatedly created and deleted.

What you expected to happen:

volcano controller removes data from cache completely after job has deleted, and doesn't need to re-add to cache while updating pod.

How to reproduce it (as minimally and precisely as possible):

Just update pod and delete job as close to the time as possible.

Anything else we need to know?:

controller parameters:

    - --logtostderr=false
    - --leader-elect=true
    - --lock-object-namespace=volcano-system
    - --enable-healthz=true
    - -v=3
    - --log-dir=/home/
    - --log-file-max-size=0
    - --kube-api-qps=2000
    - --kube-api-burst=3000
    - --worker-threads=300
    - 2>&1

Environment:

• Volcano Version: v1.8.2
• Kubernetes version (use kubectl version): v1.20
• Cloud provider or hardware configuration: 128c 256G
• OS (e.g. from /etc/os-release):Debian GNU/Linux 9 (stretch)
• Kernel (e.g. uname -a):Linux 5.10.0-103-bili-colo
• Install tools: kubectl