KEP #77 Dynamically Sized Jobs #1851
Conversation
k8s-ci-robot
added
kind/documentation
|
Hi @vicentefb. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
needs-ok-to-test
✅ Deploy Preview for kubernetes-sigs-kueue ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
|
|
||
| ## Phases for MVP (alpha) | ||
|
|
||
| ### Phase 1 - Scale Down |
There was a problem hiding this comment.
WIP implementation for Phase 1: #1852
vicentefb
force-pushed
the
ElasticJobsKep
branch
from
bc85dbb
to
e000b9c
Compare
|
/release-note-none |
k8s-ci-robot
added
release-note-none
|
@astefanutti, is this something your team is still interested in? |
| ## Design Details | ||
|
|
||
| ### Workload Slices | ||
| To support horizontal scaling of jobs, we will introduce the concept of a "Workload Slice”. A Workload Slice is a Workload object with an owner reference to the original Workload for a job. Workload Slices represent per-replica changes to a job that were not initially accounted for when the job was created. |
There was a problem hiding this comment.
Questions to confirm / update my understanding:
- Do the workload slices need to be submitted to the same LocalQueue, or just the same ClusterQueue is enough?
- Do the workload slices need to use identical PodTemplates?
If the two need to be true, what if there is a workload slice that violates the constraints?
Also,
3. Are the workload slices admitted, by Kueue scheduler. to the same resource flavors? (I suppose this is required so that we can aggregate them)
I think notes clarifying the above questions would be helpful.
There was a problem hiding this comment.
Do the workload slices need to be submitted to the same LocalQueue, or just the same ClusterQueue is enough?
Because LocalQueue is referenced by the top-level RayCluster, it will be submitted to the same LocalQueue
Do the workload slices need to use identical PodTemplates?
Yes, this is already enforced by KubeRay
If the two need to be true, what if there is a workload slice that violates the constraints?
From our initial prototyping so far, I think it will be hard to break these constraints. Both the LocalQueue and Pod template are referenced top-level in the RayCluster API.
- Are the workload slices admitted, by Kueue scheduler. to the same resource flavors? (I suppose this is required so that we can aggregate them)
Yes, I think from discussions with Aldo we agreed that all workload slices need to be belong to the same resource flavor
There was a problem hiding this comment.
Yes, this is already enforced by KubeRay
Are the Workload slices going to be created by KubeRay?
From our initial prototyping so far, I think it will be hard to break these constraints. Both the LocalQueue and Pod template are referenced top-level in the RayCluster API.
The constraints probably can be violated if a user creates the Workload slices by hand. Might be worth a note how we handle a mismatch.
There was a problem hiding this comment.
Are the Workload slices going to be created by KubeRay?
No I meant KubeRay enforces identical podtemplates across replicas, which translates to identical podtemplates across WorkloadSlices
The constraints probably can be violated if a user creates the Workload slices by hand. Might be worth a note how we handle a mismatch.
Good point, we should document this in the KEP
There was a problem hiding this comment.
The constraints probably can be violated if a user creates the Workload slices by hand. Might be worth a note how we handle a mismatch.
Good point, we should document this in the KEP
I'm especially interested in the behavior with the prebuild feature.
There was a problem hiding this comment.
prebuilt workload feature for context #1575
| ## Design Details | ||
|
|
||
| ### Workload Slices | ||
| To support horizontal scaling of jobs, we will introduce the concept of a "Workload Slice”. A Workload Slice is a Workload object with an owner reference to the original Workload for a job. Workload Slices represent per-replica changes to a job that were not initially accounted for when the job was created. |
There was a problem hiding this comment.
The constraints probably can be violated if a user creates the Workload slices by hand. Might be worth a note how we handle a mismatch.
Good point, we should document this in the KEP
I'm especially interested in the behavior with the prebuild feature.
applied toc
vicentefb
force-pushed
the
ElasticJobsKep
branch
from
c4708da
to
9af7511
Compare
k8s-ci-robot
added
the
approved
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: 0b6b913a553f2022fc45dc516e7044467714352c
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alculquicondor, tenzen-y, vicentefb The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
* added kep * kep updated applied toc * updated kep * toc updated * added info in unit tests and integration tests section * added details about workload slices * rephrase scale down section * updated and added details on slices, generalized design details and typos * update * added details about mutikueue and removed users from approvers
What type of PR is this?
/kind documentation
/kind feature
What this PR does / why we need it:
KEP for Dynamically Sized Jobs
Which issue(s) this PR fixes:
Fixes #77
Special notes for your reviewer:
A WIP for Phase 1 can be found here: #1852
Does this PR introduce a user-facing change?