title | date | weight | description |
---|---|---|---|
Pending Workloads on-demand |
2023-12-05 |
3 |
Obtain the pending Workloads via the on-demand visibility API
|
This page shows you how to monitor pending workloads with VisibilityOnDemand feature.
The intended audience for this page are batch administrators, and batch users for Local Queue Visibility section.
From version v0.6.0, Kueue provides the ability for a batch administrators to monitor the pipeline of pending jobs, and help users to estimate when their jobs will start.
Make sure the following conditions are met:
- A Kubernetes cluster is running.
- The kubectl command-line tool has communication with your cluster.
- Kueue is installed in version v0.6.0 or later.
VisibilityOnDemand is an Alpha
feature disabled by default, check the Change the feature gates configuration section of the Installation for details.
Available in Kueue v0.6.0 and later
To install a simple setup of ClusterQueue
{{< include "examples/admin/single-clusterqueue-setup.yaml" "yaml" >}}
run the following command:
kubectl apply -f https://kueue.sigs.k8s.io/examples/admin/single-clusterqueue-setup.yaml
Now, let's create 10 jobs
{{< include "examples/jobs/sample-job.yaml" "yaml" >}}
using a command:
for i in {1..6}; do kubectl create -f https://kueue.sigs.k8s.io/examples/jobs/sample-job.yaml; done
To view pending workloads in ClusterQueue cluster-queue
run the following command:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/clusterqueues/cluster-queue/pendingworkloads"
You should get results similar to:
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jrjfr-8d56e",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jrjfr",
"uid": "5863cf0e-b0e7-43bf-a445-f41fa1abedfa"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 0,
"positionInLocalQueue": 0
},
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
},
{
"metadata": {
"name": "job-sample-job-t9b8m-4e770",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-t9b8m",
"uid": "64c26c73-6334-4d13-a1a8-38d99196baa5"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 2,
"positionInLocalQueue": 2
}
]
}
You can pass optional query parameters:
- limit
<integer>
- 1000 on default. It indicates max number of pending workloads that should be fetched. - offset
<integer>
- 0 by default. It indicates position of the first pending workload that should be fetched, starting from 0.
To view only 1 pending workloads use, starting from position 1 in ClusterQueue run:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/clusterqueues/cluster-queue/pendingworkloads?limit=1&offset=1"
You should get results similar to
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
}
]
}
Similarly to ClusterQueue, to view pending workloads in LocalQueue user-queue
run the following command:
kubectl get --raw /apis/visibility.kueue.x-k8s.io/v1alpha1/namespaces/default/localqueues/user-queue/pendingworkloads
You should get results similar to:
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jrjfr-8d56e",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jrjfr",
"uid": "5863cf0e-b0e7-43bf-a445-f41fa1abedfa"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 0,
"positionInLocalQueue": 0
},
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
},
{
"metadata": {
"name": "job-sample-job-t9b8m-4e770",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-t9b8m",
"uid": "64c26c73-6334-4d13-a1a8-38d99196baa5"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 2,
"positionInLocalQueue": 2
}
]
}
You can pass optional query parameters:
- limit
<integer>
- 1000 on default. It indicates max number of pending workloads that should be fetched. - offset
<integer>
- 0 by default. It indicates position of the first pending workload that should be fetched, starting from 0.
To view only 1 pending workloads use, starting from position 1 in LocalQueue run:
kubectl get --raw "/apis/visibility.kueue.x-k8s.io/v1alpha1/localqueues/user-queue/pendingworkloads?limit=1&offset=1"
You should get results similar to
{
"kind": "PendingWorkloadsSummary",
"apiVersion": "visibility.kueue.x-k8s.io/v1alpha1",
"metadata": {
"creationTimestamp": null
},
"items": [
{
"metadata": {
"name": "job-sample-job-jg9dw-5f1a3",
"namespace": "default",
"creationTimestamp": "2023-12-05T15:42:03Z",
"ownerReferences": [
{
"apiVersion": "batch/v1",
"kind": "Job",
"name": "sample-job-jg9dw",
"uid": "fd5d1796-f61d-402f-a4c8-cbda646e2676"
}
]
},
"priority": 0,
"localQueueName": "user-queue",
"positionInClusterQueue": 1,
"positionInLocalQueue": 1
}
]
}