Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ProvisioningRequest documentation by adding a new way of passing Parameters to ProvisioningRequest #1918

Merged
merged 1 commit into from Mar 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
26 changes: 22 additions & 4 deletions site/content/en/docs/admission-check-controllers/provisioning.md
Expand Up @@ -6,15 +6,27 @@ description: >
An admission check controller providing kueue integration with cluster autoscaler.
---

The Provisioning Admission Check Controller is an Admission Check Controller designed to integrate Kueue with [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler). Its primary function is to create [ProvisioningRequests](https://github.com/kubernetes/autoscaler/blob/4872bddce2bcc5b4a5f6a3d569111c11b8a2baf4/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/types.go#L41) for the workloads holding [Quota Reservation](/docs/concepts/#quota-reservation) and keeping the [AdmissionCheckState](/docs/concepts/admission_check/#admissioncheckstate) in sync.
The Provisioning AdmissionCheck Controller is an AdmissionCheck Controller designed to integrate Kueue with [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler). Its primary function is to create [ProvisioningRequests](https://github.com/kubernetes/autoscaler/blob/4872bddce2bcc5b4a5f6a3d569111c11b8a2baf4/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/types.go#L41) for the workloads holding [Quota Reservation](/docs/concepts/#quota-reservation) and keeping the [AdmissionCheckState](/docs/concepts/admission_check/#admissioncheckstate) in sync.

The controller is part of kueue. You can enable it by setting the `ProvisioningACC` feature gate. Check the [Installation](/docs/installation/#change-the-feature-gates-configuration) guide for details on feature gate configuration.
The controller is part of Kueue. You can enable it by setting the `ProvisioningACC` feature gate. Check the [Installation](/docs/installation/#change-the-feature-gates-configuration) guide for details on feature gate configuration.

The Provisioning Admission Check Controller is supported on [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) versions 1.29 and later. However, some cloud-providers may not have an implementation for it.

PBundyra marked this conversation as resolved.
Show resolved Hide resolved
## Parameters
## Usage

This controller uses a `ProvisioningRequestConfig` as parameters, like:
To use the Provisioning AdmissionCheck, create an [AdmissionCheck](docs/concepts/admission_check)
with `kueue.x-k8s.io/provisioning-request` as a `.spec.controllerName` and create a ProvisioningRequest configuration usign a `ProvisioningRequestConfig` object. See an example below.

## ProvisioningRequest configuration

There are two ways to configure ProvisioningRequest sent for your Jobs.

- **ProvisioningRequestConfig:** This configuration in the AdmissionCheck applies to all the jobs that go through this check.
It enables you to set `provisioningClassName`, `managedResources`, and `parameters`
- **Job annotation**: This configuration enables you to set `parameters` to a specific job. If both the annotation and the ProvisioningRequestConfig refer to the same parameter, the annotation value takes precedence.

### ProvisioningRequestConfig
A `ProvisioningRequestConfig` looks like the following:

```yaml
apiVersion: kueue.x-k8s.io/v1beta1
Expand All @@ -33,6 +45,12 @@ Where:

Check the [API definition](https://github.com/kubernetes-sigs/kueue/blob/main/apis/kueue/v1beta1/provisioningrequestconfig_types.go) for more details.

### Job annotation

Another way to pass ProvisioningRequest's [parameters](https://github.com/kubernetes/autoscaler/blob/0130d33747bb329b790ccb6e8962eedb6ffdd0a8/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go#L115) is by using Job annotations. Every annotation with the ***provreq.kueue.x-k8s.io/*** prefix will be directly passed to created ProvisioningRequest. E.g. `provreq.kueue.x-k8s.io/ValidUntilSeconds: "60"` will pass `ValidUntilSeconds` parameter with the value of `60`. See more examples below.

Once Kueue creates a ProvisioningRequest for the job you submitted, modifying the value of annotations in the job will have no effect in the ProvisioningRequest.

## Example

### Setup
Expand Down
2 changes: 2 additions & 0 deletions site/static/examples/provisioning/sample-job.yaml
Expand Up @@ -5,6 +5,8 @@ metadata:
namespace: default
labels:
kueue.x-k8s.io/queue-name: user-queue
annotations:
provreq.kueue.x-k8s.io/maxRunDurationSeconds: "600"
spec:
parallelism: 3
completions: 3
Expand Down