Install Scheduler-plugins

Create a Kubernetes Cluster

Firstly you need to have a Kubernetes cluster, and a kubectl command-line tool must be configured to communicate with the cluster.

The Kubernetes version must equal to or greater than v1.19.0. To check the version, use kubectl version --short.

If you do not have a cluster yet, create one by using one of the following provision tools:

kind
kubeadm
minikube

Install release v0.19.8 and use Coscheduling

In this section, we will walk you through how to replace the default scheduler with the scheduler-plugins image. As the new image is built on top of the default scheduler, you won't lose any vanilla Kubernetes scheduling capability. Instead, a lot of extra out-of-box functionalities (implemented by the plugins in this repo) can be obtained, such as coscheduling.

The following steps are based on a Kubernetes cluster created by Kind.

Log into the control plane node

sudo docker exec -it $(sudo docker ps | grep control-plane | awk '{print $1}') bash

Backup kube-scheduler.yaml

cp /etc/kubernetes/manifests/kube-scheduler.yaml /etc/kubernetes/kube-scheduler.yaml

Create /etc/kubernetes/sched-cc.yaml

apiVersion: kubescheduler.config.k8s.io/v1beta1
kind: KubeSchedulerConfiguration
leaderElection:
  # (Optional) Change true to false if you are not running a HA control-plane.
  leaderElect: true
clientConnection:
  kubeconfig: /etc/kubernetes/scheduler.conf
profiles:
- schedulerName: default-scheduler
  plugins:
    queueSort:
      enabled:
      - name: Coscheduling
      disabled:
      - name: "*"
    preFilter:
      enabled:
      - name: Coscheduling
    permit:
      enabled:
      - name: Coscheduling
    reserve:
      enabled:
      - name: Coscheduling
    postBind:
      enabled:
      - name: Coscheduling
  # pluginConfig is needed for coscheduling plugin to manipulate PodGroup CR objects.
  pluginConfig:
  - name: Coscheduling
    args:
      kubeConfigPath: /etc/kubernetes/scheduler.conf

❗IMPORTANT❗ Starting with release v0.19, several plugins (e.g., coscheduling) introduced CRD to optimize their design and implementation. And hence we need an extra step to:
- apply extra RBAC privileges to user system:kube-scheduler so that the scheduler binary is able to manipulate the custom resource objects
- install a controller binary managing the custom resource objects
Next, we apply the compiled yaml located at manifests/install/all-in-one.yaml.
```
$ kubectl apply -f all-in-one.yaml
```
After this step, a deployment called scheduler-plugins-controller is expected to run in namespace scheduler-plugins:
```
$ kubectl get deploy -n scheduler-plugins
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
scheduler-plugins-controller   1/1     1            1           19h
```
❗IMPORTANT❗ Install the CRDs your workloads depend on.

You can refer to each folder under manifests to obtain the CRD yaml for each plugin. Here we install coscheduling CRD:
```
$ kubectl apply -f manifests/coscheduling/crd.yaml
```

Modify /etc/kubernetes/manifests/kube-scheduler.yaml to run scheduler-plugins with coscheduling

Generally, we need to make a couple of changes:

pass in the composed scheduler-config file via argument --config
(optional) remove duplicated CLI parameters (e.g., --leader-elect), as they may have been defined in the config file
replace vanilla Kubernetes scheduler image with scheduler-plugin image
mount the scheduler-config file to be readable when scheduler starting

Here is a diff:

16d15
<     - --config=/etc/kubernetes/sched-cc.yaml
17a17,18
>     - --kubeconfig=/etc/kubernetes/scheduler.conf
>     - --leader-elect=true
19,20c20
<     image: k8s.gcr.io/scheduler-plugins/kube-scheduler:v0.19.8
---
>     image: k8s.gcr.io/kube-scheduler:v1.19.8
50,52d49
<     - mountPath: /etc/kubernetes/sched-cc.yaml
<       name: sched-cc
<       readOnly: true
60,63d56
<   - hostPath:
<       path: /etc/kubernetes/sched-cc.yaml
<       type: FileOrCreate
<     name: sched-cc

Verify that kube-scheduler pod is running properly with a correct image: k8s.gcr.io/scheduler-plugins/kube-scheduler:v0.19.8

$ kubectl get pod -n kube-system | grep kube-scheduler
kube-scheduler-kind-control-plane            1/1     Running   0          3m27s

$ kubectl get pods -l component=kube-scheduler -n kube-system -o=jsonpath="{.items[0].spec.containers[0].image}{'\n'}"
k8s.gcr.io/scheduler-plugins/kube-scheduler:v0.19.8

Test Coscheduling

Now, we're able to verify how the coscheduling plugin works.

Create a PodGroup custom object called pg1:

# podgroup.yaml
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: PodGroup
metadata:
  name: pg1
spec:
  scheduleTimeoutSeconds: 10
  minMember: 3

$ kubectl apply -f podgroup.yaml

Create a deployment labelled pod-group.scheduling.sigs.k8s.io: pg1 to associated with PodGroup pg1 created in the previous step.

# deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pause
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pause
  template:
    metadata:
      labels:
        app: pause
        pod-group.scheduling.sigs.k8s.io: pg1
    spec:
      containers:
      - name: pause
        image: k8s.gcr.io/pause:3.2

As PodGroup pg1 requires at least 3 pods to be scheduled all-together, and there are only 2 Pods so far, so it's expected to observer they are pending:

All nginx pods are expected to be Pending as they cannot be co-scheduled altogether.
```
$ kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
pause-58f7d7db67-7sqgp   0/1     Pending   0          9s
pause-58f7d7db67-jbmfv   0/1     Pending   0          9s
```

Now let's delete the deployment to re-create it with replicas=3, so as to qualify for minMember (i.e., 3) of the associated PodGroup:

$ kubectl delete -f deploy.yaml && sed 's/replicas: 2/replicas: 3/' deploy.yaml | kubectl apply -f -
deployment.apps "pause" deleted
deployment.apps/pause created

And wait for a couple of seconds, it's expected to see all Pods get into running state:

$ kubectl get pod
NAME                     READY   STATUS    RESTARTS   AGE
pause-64f5c9ccf4-kprg7   1/1     Running   0          8s
pause-64f5c9ccf4-tc8lx   1/1     Running   0          8s
pause-64f5c9ccf4-xrgkw   1/1     Running   0          8s

You can also get the PodGroup's spec via:

$ kubectl get podgroup pg1 -o yaml
apiVersion: scheduling.sigs.k8s.io/v1alpha1
kind: PodGroup
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"scheduling.sigs.k8s.io/v1alpha1","kind":"PodGroup","metadata":{"annotations":{},"name":"pg1","namespace":"default"}, "spec":{"minMember":3,"scheduleTimeoutSeconds":10}}
  creationTimestamp: "2021-03-16T00:22:34Z"
  generation: 1
  managedFields:
  ...
  name: pg1
  namespace: default
  resourceVersion: "135603"
  selfLink: /apis/scheduling.sigs.k8s.io/v1alpha1/namespaces/default/podgroups/pg1
  uid: b4ac3562-54ab-4c1e-89bb-541a81c6acce
spec:
  minMember: 3
  scheduleTimeoutSeconds: 10
status:
  phase: pending

⚠ NOTE: There are some UX issues need to be addressed in controller side - #166.

Install old-version releases

If you're running at v0.18.9, which doesn't depend on PodGroup CRD, you should refer to the install doc in branch release-1.18 for detailed installation instructions.

Uninstall scheduler-plugins

Delete the deployment

$ kubectl delete deploy pause -n default

Recover kube-scheduler.yaml and delete sched-cc.yaml

If the cluster is created by kubeadm or minikube, log into Master node:

$ mv /etc/kubernetes/kube-scheduler.yaml /etc/kubernetes/manifests/
$ rm /etc/kubernetes/sched-cc.yaml

If the cluster is created by kind, enter the Master's container:

$ sudo docker exec -it $(sudo docker ps | grep control-plane | awk '{print $1}') bash
$ mv /etc/kubernetes/kube-scheduler.yaml /etc/kubernetes/manifests/
$ rm /etc/kubernetes/sched-cc.yaml
exit

Check state of default scheduler

$ kubectl get pod -n kube-system | grep kube-scheduler
kube-scheduler-kind-control-plane            1/1     Running   0          91s

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

install.md

install.md

Install Scheduler-plugins

Table of Contents

Create a Kubernetes Cluster

Install release v0.19.8 and use Coscheduling

Test Coscheduling

Install old-version releases

Uninstall scheduler-plugins

Files

install.md

Latest commit

History

install.md

File metadata and controls

Install Scheduler-plugins

Table of Contents

Create a Kubernetes Cluster

Install release v0.19.8 and use Coscheduling

Test Coscheduling

Install old-version releases

Uninstall scheduler-plugins