v1.4.0-Beta

What's New

1. Support multi-scheduler

In Kubernetes cluster with multiple schedulers, different kinds of workloads should be mapped to certain scheduler sometimes. For example, K8s native workloads such as deployment in namespace kube-system are mapped to default-sheduler while AI and Big data jobs are mapped to Volcano. This feature aims to implements that automaticallty. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/multi-scheduler.md. (#1576, #1521, @huone1 @william-wang )

2. Support proportion of resources for GPU node

In order to make full use of scarce resources such as GPU, one solution is to bind them with other resources as shares. For example, it is common to see a lot of CPU-intensive workloads are scheduled to GPU nodes. When GPU-intensive workloads come, they cannot be scheduled because of lack of CPU or Memory in GPU nodes. If workloads requiring both GPU, CPU, Memory at certatin range can be scheduled to GPU nodes first, it is possible to make full use of GPUs. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/proportional.md. (#1527, @king-jingxiang )

3. Support CPU NUMA-Aware scheduling

As to CPU-intensive workloads especially in AI, Big Data and HPC fileds, It will result in a significant performance improvement if CPU NUMA is enabled. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md. (#1493, @huone1 )

4. Provide framework of stress test

In this release, A kind of framework for Volcano stress test is provided. (#1516, @rudeigerc )

Other Notable Changes

• update roadmap for v1.5(#1708, @william-wang )
• refine scheduler framework(#1705, @wpeng102 )
• fix daily release job fails(#1701, @Thor-wl )
• optimization register controller log(#1683, @hwdef )
• lower ssh key access permission(#1682, @hwdef )
• automatically set GOMAXPROCS for scheduler & controller & webhook-manager(#1681, @SataQiu )
• add revive linter as drop-in replacement of archived golint(#1678, @gy95 )
• modify default scheduler cfg(#1673, @shinytang6 )
• expose detailed scheduling reason of pending tasks(#1672, @eggiter)
• format env key(#1660, @wpeng102 )
• not delete failed pods on last try(#1657, @wpeng102 )
• optimize function setOversubscription(#1653, @huone1 )
• upgrade controller tools to v0.6.0(#1649, @shinytang6 )
• enable configuring volcano webhooks(#1645, @hacker-qian)
• add user-provided rsa key-pair and mount rsa key-pair in InitContainers since we want to clone private git repository before run training task(#1644, @python279 )
• refactor the preempt function in plugin gang(#1643, @huone1 )
• delete bindingTasks from NodeInfo structure(#1636, @huone1 )
• scheduler plugin framework support SharedInformerFactory(#1635, @wpeng102 )
• add task-spec in pod label(#1626, @python279 )
• rename resource comparision functions(#1624, @Thor-wl )
• update resource comparision doc(#1622, @Thor-wl )
• add Equal function(#1621, @Thor-wl )
• add LessEqualPartly function(#1613, @Thor-wl )
• add LessEqualInAllDimension function and remove LessEqual/LessEqualStrict functions(#1611, @thor)
• doc: add resource quota plugin doc(#1583, @merryzhou )
• improve: clean unready status when job ready(#1582, @lowang-bh )
• improve: reduce calculation of total nodes' resource in stead of storing it when snapshot(#1578, @lowang-bh )
• add resource comparison doc(#1573, @Thor-wl )
• update Less function(#1569, @Thor-wl )
• support oversubscription in volcano framework(#1566, @wpeng102 )
• add multi-scheduling design doc(#1565, @huone1 )
• improve: preempt break out if intersection is null(#1563, @lowang-bh )
• add arch node selectors to the installer arm&amd64(#1556, @holdenk )
• optimizing openSession(#1541, @hacker-qian)
• improve: remove nodename from taskinfo when RemoveTask(#1517, @lowang-bh )
• the member Others should be copied in func (ni *NodeInfo) Clone() (#1512, @huone1 )
• use struct NodeScoreList better than HostPriorityList for score(#1510, @huone1 )
• add min success design(#1505, @zen-xu )
• change min resource to 0.1 in resource_info(#1489, @wpeng102 )

Bug Fixes

• fix diff function(#1712, @Thor-wl )

• make 'existing pods anti-affinity rules' work(#1668, @eggiter)

• add setting MinResources to pg for normal pod(#1666, @huone1 )

• fix OOM will occur if pod info is sync before node info(#1662, @huone1 )

• fix addmission parsing bug(#1656, @hacker-qian)

• fix overused judgement when deal with allocate and proportion(#1637, @Thor-wl )

• fix bugs about resource comparison(#1628, @Thor-wl )

• reset task.NodeName after call DeallocateFunc(#1618, @merryzhou )

• adds the missing field to the Webhook(#1615, @hwdef )

• fix a problem about equivalence ecache feature (#1593, @huone1 )

• func FeasibleNodesToFind to use list with a centain order(#1574, @lowang-bh )

• fix static check warning in pkg folder(#1552, @hwdef )

• fix bug in predicates plugin(#1547, @hacker-qian)

• fix(scheduler): reclaim action minus and comparison bug(#1540, @shinytang6 )

• fix resource comparasion bug in task topology(#1546, @Thor-wl )

• fix select wrong queue when proportion is disable(#1497, @zen-xu )

• fix the error for make verify(#1508, @huone1 )

v1.3.0

What's New

1. Support minAvailable at task level

Just as the minAvailable at job level, minAvailable at task level will regard replicases at the same task as group and decide whether to schedule pods at the task. Only when the minAvailable is meet will the pods will be scheduled together. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/task-minavailable.md. (#1459, @shinytang6 )

2. Support minSuccess for Job

Support to configure the least number of pods belonging to the job. It's useful to mark the status of job when minsuccess reaches or not and accelerates the job status judgement. (#1384, @zen-xu )

3. Support task-topology

In big data processing jobs like Tensorflow & Spark, tasks transmitted a large amount of data between each other, causing transmission delay took a large proportion in job execution time. So task topology plugin was proposed to modify scheduling strategy according to transmission topology inside a job, so as to cut the data amount to be transmitted between nodes, decrease transmission delay proportion in job execution time, and improve resource utilization. More details please refer to https://github.com/volcano-sh/volcano/blob/master/docs/design/task-topology-plugin.md. (#1353, @jiangkaihua )

4. Create new repository volcano.sh/apis

Separate apis from volcano.sh/volcanosh. Any downstream projects can introduce the CRD clientset/lister/informer with the K8s version it needs. (https://github.com/volcano-sh/apis, @Thor-wl )

Other Notable Changes

• fix the bug of CRD apiversion and installation volcano with kind(#1483, @Thor-wl )
• doc: add schedulerName in gpu sharing user guide(#1481, @ChAnYaNG97 )
• update scheduler default QPS and Burst(#1480, @Thor-wl )
• vcctl queue support get kubeconfig from env(#1477, @yahaa )
• add queue annotation in deployment example and add queue yaml(#1474, @nolimitkun )
• support K8s v1.19. (#1444, @Thor-wl )
• optimize yaml unmarshal logic(#1427, @sniperking1234 )
• doc: add multi schedulers design doc(#1403, @zen-xu )
• simplify unit-test(#1394, @zen-xu )
• add new target update-development-yaml in Makefile(#1386, @zen-xu )
• add additional printer columns to crd Job(#1385, @zen-xu )
• update ca.crt and server.csr validity period to 10 years(#1382, @zen-xu )
• Helm support crd v1(#1378, @zen-xu )
• feat(webhook): add podgroup admission(#1375, @shinytang6 )
• support auto updating crd manifests in helm templates when run make generate-yaml(#1374, @zen-xu )
• refactor(e2e): separate utils as a single package(#1362, @rudeigerc )
• support taint toleration preferNoScheduler in release-0.4(#1354, @huone1 )
• support taintToleration preferNoschdule(#1352, @huone1 )

Bug Fixes

• fix: lose preemptor when considering Preemption between Tasks within same Job (#1453, @lowang-bh )
• scheduler need configmap role to enable elect funtion(#1443, @wpeng102 )
• fix(scheduler): use nodeMap to fix anti-affinity problem(#1430. @shinytang6 )
• fix: use task.Name to make podName in admission(#1412, @merryzhou )
• add bindingTasks to judge whether adding node to the snapshot.(#1388, @zen-xu )
• fix reserving for deleted targetJob raise nil pointer(#1371, @zen-xu )
• fix sla jobOderFn when sla not set(#1365, @merryzhou )
• fix: it is possible to Occur OutOfCpu， when exist some pods including init container(#1364, @huone1 )
• fix wrong Pipeline in action allocate(#1360, @yzs981130 )
• fix: prevent SelectBestNode func arise panic(#1344, @yahaa )
• fix(scheduler): move JobInfo helper functions to method(#1343, @Thrimbda )

v1.2.0

What's New

1. Add TDM plugin

TDM(Time Division Multiplexing) plugin aims to provide a mechanism for nodes, which can be used for K8S and other cluster(such as Yarn) in separate time.(#1269, @yahaa )

2. Add SLA plugin

SLA(Service Level Agreement) plugin works for job resource reservation feature. Users can set SLA for jobs to ensure specified jobs to be scheduled in time. It provides an better design and implementation for job resource reservation. (#1303, @jiangkaihua )

Other Notable Changes

• improve addResourceList func in job_controller_util.go(#1332, @shinytang6 )
• update overcommit plugin(#1324, @jiangkaihua )
• add e2e for sla plugin(#1319, @jiangkaihua )
• make sure non-preemptable and revocable workload not preempt other tasks in tdm plugin(#1314, @wpeng102 )
• support only specify preemptable=true for revocable workload(#1313, @wpeng102 )
• support revocable-zone annotaion for workload(#1312, @wpeng102 )
• add fail event for annotation admission(#1308, @wpeng102 )
• support min pod alive for tdm plugin(#1300, @wpeng102 )
• update enqueue action, import overcommit plugin to limit pending jobs from inqueue.(#1298, @jiangkaihua )
• build cache for revocable nodes(#1293, @yahaa )
• separate JobPipelined into two semantics for preempt action(#1288, @wpeng102 )
• support minAlive and evictMaxNum for job(#1287, @wpeng102 )
• non preemptable deployment preempt resource(#1286, @wpeng102 )
• update job-resource-reservation-design doc(#1282, @Thor-wl )
• add tdm design doc(#1277, @wpeng102 )
• refine deployment.yaml example(#1274, @wpeng102 )
• tdm plugin add victimsFn(#1276, @wpeng102 )
• add Makefile flag SUPPORT_PLUGINS(#1266, @zen-xu )
• update ssh secret when job updated(#1263, @shinytang6 )
• add job plugin example(#1254, @shinytang6 )

Bug Fixes

• replace removed command of kind when getting kube config(#1315, @rudeigerc )
• fix log in job_controller_actions.go(#1305, @gaocegege )
• correct log info in cache.go(#1302, @juchaosong )
• optimize nodeorder plugin(#1292, @huone1 )
• enhance tdm max evict step(#1290, @yahaa )
• revert ssh subpath for ssh plugin(#1280, @shinytang6 )
• fix e2e helm install timeout(#1262, @huone1 )
• fix more pods are reclaimed than required(#1260, @huone1 )
• fix CI: add hacky retry mechanism(#1248, @shinytang6 )

v1.1.2

Changes since v1.1.1

• bug fix: Use musl-gcc build image, because vc-scheduler default image is alpine, which only has musl-libc(#1225, @zen-xu)

v1.1.1

What's New

1. support vc-scheduler loading custom plugins

Separate plugin implementation with scheduler. Support implement custom plugins and load to vc-scheduler dynamically.(#1218, @zen-xu)

2. add MaxRequeueNum as a controller-manager param

Support configure MaxRequeueNum in config file of vc-scheduler, default to 15 times.(#1087, @shinytang6)

3. add design documentation of CPU careful regulation

Give the design of CPU careful regulation in socket level.(#1051, @ProgramerGu)

Other Notable Changes

• add deployment example with volcano scheduling(#1222, @Thor-wl)
• add Queue & Namespace support for volcano monitoring(#1200, @alcorj-mizar)
• job event optimize (#1192, @mikechengwei)
• optimize local script(#1191, @mikechengwei)
• generate v1 crds. remove subresource status in pg crd(#1179, @stpabhi)
• mutate default of job spec(#1170, @shinytang6)
• CI switch to Github Action(#1160, @daixiang0)
• queue-resource design add(#1158, @hudson741)
• feature design for queue resource reservation(#1130, @Thor-wl)
• add podName and svcName length validate(#1127, @mikechengwei)
• update getting-started.md(#1126, @daixiang0)
• some improvements of scheduler(#1111, @shinytang6)

Bug Fixes

• fix allocate action: high priority queue should not block others(#1209, @yesterday)
• fix script: fix daily release err(#1208, @yesterday)
• fix prepare-for-development dead link(#1205, @naveensrinivasan)
• fix gox path err im Makefile(#1201, @shinytang6)
• fix proportion can not reclaim issue(#1194, @wpeng102)
• fix dose not delete network policy when job finished(#1186, @wpeng102)
• fix queue cannot use idle resource issue(#1176, @wpeng102)
• fix scheduling duration update after complete(#1167, @alcorj-mizar)
• fix queue unknown state bug(#1151, @wpeng102)
• fix terminate job and release resources when drop job out of queue(#1138, @merryzhou)
• fix helm upgrade bugs on namespace hardcode(#1132, @alcorj-mizar)

v1.1.0

What's New

1. Add monitor compontent

Monitor compontent added support display some base metrics about Volcano.(#1066, @alcorj-mizar)

2. Support resource reservation for big job automatically

Reserve resource for pending job which is at highest priority among pending jobs and waits for a long time. The big job is recognized by scheduler automatically.(#1044, @Thor-wl)

3. Support HDRF

Hierarchical dominant resource fairness is configured with a weighted tree, such that each node in the tree has a positive weight value.(#928, @ggaaooppeenngg)

Other Notable Changes

• add queue weight validation(#1092, @Thor-wl)
• add file watcher for scheduler config file(#1016, @hzxuzhonghu)
• add arm64 support(#1000, @Thor-wl)

Bug Fixes

• not skip lower prioprity job when job with high priority allocated failed(#1089, @merryzhou)
• remove creationTimestamp comparation in jobOrderFn of gang plugin(#1061, @zionwu)
• fix queue capability overuse when specify minAvailable less than task replicas(#1042, @zen-xu)
• fix duplicate preemptee in victims(#1023, @xiaoanyunfei)
• fix scheduler panic when volcano job use pvc(#1022, @wpeng102)
• fix scheduler panic if minResource not set(#1010, @hzxuzhonghu)
• remove controlled resource in OnJobDelete(#1005, @hzxuzhonghu)
• fix ssh authorize key and remove no-root flag(#996, @hzxuzhonghu)
• plugins/binpack: fix typo of BinpackMemory(#994, @aixeshunter)
• scheduler: avoid pushing back empty jobs in allocate action(#992, @lixiang233)
• fix unallocate(#984, @hzxuzhonghu)
• fix bug of queue capability lose efficacy(#974, @hzxuzhonghu)

v0.4.2

• Fix queue capability validation failed when some running jobs finished or deleted (#959, @Thor-wl)

v1.0.1

Changelog since v1.0.0

• Fix job's scheduler name does not take effect (#944, @hzxuzhonghu)
• Fix podgroup status and event (#951, @hzxuzhonghu)
• Support job scale down to zero (#945, @hzxuzhonghu)
• Fix queue capability validation failed when some running jobs finished or deleted (#959, @Thor-wl)

v0.4.1

v0.4.1 (2020-07-14)

• UpdateJob when Job annotations or labels changed (#919, @Thor-wl)
• Fix panic in controller (#901, @Thor-wl)
• Support scale up and down (#796, @hzxuzhonghu)
• Fix crd to support job patch ops (#794, @hzxuzhonghu)

v1.0.0

1.0 What's New

1. GPU Sharing

Volcano now supports gpu sharing between different pods (#852, @tizhou86, @hzxuzhonghu).

2. Preempt and reclaim enhancement

Volcano is now able to support preempt for batch job (#738, @carmark).

3. Dynamic scale up and down

Volcano job now supports dynamically scale up and down (#787, @hzxuzhonghu).

4. Support integrate with flink operator

Users are now able to run flink job with volcano. Follow the instructions here to make use of the feature. @hzxuzhonghu).

5. Support DAG job with argo

Users are now able to run DAG job with volcano. Follow the instructions here to make use of the feature. @alcorf-mizar).

Other Notable Changes

• Update go version to 1.14 (#886, @hzxuzhonghu)
• Bump to k8s 1.18 to keep up with kubernetes (#855, @hzxuzhonghu)
• Add mindspore example (#845, @lyd911)
• Add golangCi-lint check (#799, #821, #824, #825, #827, #829, #830, #833, #835, #841. @masihtehrani @daixiang0 @Thor-wl)
• Allow specifying admission webhook port (#832, @hzxuzhonghu)
• Add support for most requested priority (#831, @daixiang0)
• Set pod DNSPolicy to ClusterFirstWithHostNet when hostnetwork set (#779, @hzxuzhonghu)
• Add bestNodeFn for plugins to select best node of its own (#790, @jiangkaihua)
• Add e2e cases for drf (#905, @Thor-wl)
• Add e2e cases for reclaim (#898 #906, @alcorj-mizar)
• Add e2e cases for preempt (#892, @Thor-wl )
• Add e2e for queue (#872, @Thor-wl)
• Add test code for queue controller (#858, @alcorj-mizar)

Bug Fixes

• Fix panic in controller (#903, @Thor-wl)
• Fix panic in allocate (#843, @k82cn)
• Fix job phase transition time set (#789, @hzxuzhonghu)
• Fix crd to support job patch ops (#786, @hzxuzhonghu)