Skip to content

Latest commit

 

History

History
62 lines (42 loc) · 3.68 KB

provisioning.md

File metadata and controls

62 lines (42 loc) · 3.68 KB
Raw
title date weight description
Provisioning Admission Check Controller
2023-10-23
1
An admission check controller providing kueue integration with cluster autoscaler.

The Provisioning AdmissionCheck Controller is an AdmissionCheck Controller designed to integrate Kueue with Kubernetes cluster-autoscaler. Its primary function is to create ProvisioningRequests for the workloads holding Quota Reservation and keeping the AdmissionCheckState in sync.

The controller is part of Kueue. You can enable it by setting the ProvisioningACC feature gate. Check the Installation guide for details on feature gate configuration.

The Provisioning Admission Check Controller is supported on Kubernetes cluster-autoscaler versions 1.29 and later. However, some cloud-providers may not have an implementation for it.

Usage

To use the Provisioning AdmissionCheck, create an AdmissionCheck with kueue.x-k8s.io/provisioning-request as a .spec.controllerName and create a ProvisioningRequest configuration usign a ProvisioningRequestConfig object. See an example below.

ProvisioningRequest configuration

There are two ways to configure ProvisioningRequest sent for your Jobs.

  • ProvisioningRequestConfig: This configuration in the AdmissionCheck applies to all the jobs that go through this check. It enables you to set provisioningClassName, managedResources, and parameters
  • Job annotation: This configuration enables you to set parameters to a specific job. If both the annotation and the ProvisioningRequestConfig refer to the same parameter, the annotation value takes precedence.

ProvisioningRequestConfig

A ProvisioningRequestConfig looks like the following:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
  name: prov-test-config
spec:
  provisioningClassName: queued-provisioning.gke.io
  managedResources:
  - nvidia.com/gpu

Where:

  • provisioningClassName - describes the different modes of provisioning the resources. Check autoscaling.x-k8s.io ProvisioningRequestSpec.provisioningClassName for details.
  • managedResources - contains the list of resources managed by the autoscaling.

Check the API definition for more details.

Job annotation

Another way to pass ProvisioningRequest's parameters is by using Job annotations. Every annotation with the provreq.kueue.x-k8s.io/ prefix will be directly passed to created ProvisioningRequest. E.g. provreq.kueue.x-k8s.io/ValidUntilSeconds: "60" will pass ValidUntilSeconds parameter with the value of 60. See more examples below.

Once Kueue creates a ProvisioningRequest for the job you submitted, modifying the value of annotations in the job will have no effect in the ProvisioningRequest.

Example

Setup

{{< include "/examples/provisioning/provisioning-setup.yaml" "yaml" >}}

Job using a ProvisioningRequest

{{< include "/examples/provisioning/sample-job.yaml" "yaml" >}}