Add pod importer #1825

trasc · 2024-03-12T15:08:06Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Add a CLI tool able to import existing pods into Kueue.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

k8s-ci-robot · 2024-03-12T15:08:08Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

netlify · 2024-03-12T15:08:23Z

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

Name	Link
🔨 Latest commit	`e6e41fb`
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-sigs-kueue/deploys/65fb0514966904000878b4ce

yaroslava-serdiuk · 2024-03-19T09:33:52Z

Could you add README file with short description how to use the script, please?

cmd/importer/main.go

cmd/importer/README.md

cmd/importer/main.go

cmd/importer/README.md

cmd/importer/pod/check.go

cmd/importer/main.go

cmd/importer/pod/import.go

cmd/importer/util/util.go

test/integration/importer/importer_test.go

cmd/importer/README.md

yaroslava-serdiuk · 2024-03-20T10:51:30Z

cmd/importer/README.md

+- The label key (`queuelabel`) providing the queue mapping is provided.
+- A mapping from ane of the encountered `queuelabel` values to an existing LocalQueue exists.
+- The LocalQueues involved in the import are using an existing ClusterQueue.
+- The ClusterQueues involved have at least one ResourceGroup using an existing ResourceFlavor. This ResourceFlavor is used when the importer creates the admission for the created workloads.


Why at least one? What will happen if it uses non-existing ResourceFlavors?
Probably we should at least log warning.

If the first resource flavor is not created, the check will fail.

yaroslava-serdiuk · 2024-03-20T10:56:41Z

cmd/importer/README.md

+### Example
+
+```bash
+./bin/importer import -n ns1,ns2 --queuelabel=src.lbl --queuemapping=src-val=user-queue,src-val=user-queue --dry-run=false


Why queuemapping has two identical parameters with src-val label, which was not specified in --queuelabel?

Why queuemapping has two identical parameters with src-val label

fixed

cmd/importer/README.md

cmd/importer/pod/check.go

cmd/importer/pod/import.go

alculquicondor · 2024-03-20T14:53:27Z

cmd/importer/pod/check.go

+	log := ctrl.LoggerFrom(ctx)
+	log.Info("Check done", "checked", summary.TotalPods, "failed", summary.FailedPods)
+	for e, pods := range summary.ErrorsForPods {
+		log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])


Suggested change

log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

log.Info("Check failed for Pod", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

alculquicondor · 2024-03-20T14:53:44Z

cmd/importer/pod/import.go

+	log := ctrl.LoggerFrom(ctx)
+	log.Info("Import done", "checked", summary.TotalPods, "failed", summary.FailedPods)
+	for e, pods := range summary.ErrorsForPods {
+		log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])


Suggested change

log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

log.Info("Could not import Pod", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

also, klog.KRef(pods[0])

pods is []string

alculquicondor · 2024-03-20T15:15:53Z

cmd/importer/pod/check.go

+			return fmt.Errorf("%q flavor %q: %w", cq.Name, rf, util.ErrCQInvalid)
+		}
+
+		// do some additional checks like:


Let's enhance the dry run:

Let's produce a list of the assignments to be done for each pod (local queue name, cluster queue name, flavor).

Someting like "log.Info("Check succeeded" , "pod", kref.., "lq", LocalQueue ....)"?

Maybe more like:
Dry run: Pod to be imported

alculquicondor

/lgtm
/approve

k8s-ci-robot · 2024-03-20T15:59:25Z

LGTM label has been added.

Git tree hash: 45d53443f4403e435ec314b656d77f7537bf3f00

k8s-ci-robot · 2024-03-20T15:59:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alculquicondor, trasc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alculquicondor]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* [pod] Export `FromObject()` * [cmd/importer] Initial implementation. * Review Remarks * Add extra labels * Review Remarks * Review Remarks * Review Remarks * Review Remarks

k8s-ci-robot requested review from denkensk and mimowo March 12, 2024 15:08

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Mar 12, 2024

trasc force-pushed the importer branch 5 times, most recently from f545a9f to 00aa28a Compare March 18, 2024 09:39

trasc force-pushed the importer branch from 00aa28a to c86a3de Compare March 19, 2024 09:36

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Mar 19, 2024

alculquicondor reviewed Mar 19, 2024

View reviewed changes

cmd/importer/main.go Show resolved Hide resolved

cmd/importer/main.go Outdated Show resolved Hide resolved

cmd/importer/main.go Outdated Show resolved Hide resolved

trasc force-pushed the importer branch from c86a3de to 3eb81c5 Compare March 19, 2024 15:19

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 19, 2024

trasc changed the title ~~[WIP]Importer~~ Add pod importer Mar 19, 2024

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Mar 19, 2024

trasc marked this pull request as ready for review March 19, 2024 15:25

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 19, 2024

k8s-ci-robot requested a review from alculquicondor March 19, 2024 15:26

alculquicondor reviewed Mar 19, 2024

View reviewed changes

cmd/importer/main.go Outdated Show resolved Hide resolved

cmd/importer/README.md Show resolved Hide resolved

cmd/importer/main.go Show resolved Hide resolved

trasc added 2 commits March 19, 2024 20:18

[pod] Export FromObject()

87a4306

[cmd/importer] Initial implementation.

4525213

trasc added 3 commits March 19, 2024 20:18

Review Remarks

0e0b365

Add extra labels

42e1363

Review Remarks

09d7cec

trasc force-pushed the importer branch from 066d98a to 09d7cec Compare March 19, 2024 18:22

alculquicondor reviewed Mar 19, 2024

View reviewed changes

Review Remarks

32b896b

yaroslava-serdiuk reviewed Mar 20, 2024

View reviewed changes

alculquicondor reviewed Mar 20, 2024

View reviewed changes

cmd/importer/pod/check.go Outdated Show resolved Hide resolved

cmd/importer/pod/import.go Outdated Show resolved Hide resolved

Review Remarks

37dd2ce

alculquicondor reviewed Mar 20, 2024

View reviewed changes

Review Remarks

e6e41fb

alculquicondor reviewed Mar 20, 2024

View reviewed changes

k8s-ci-robot assigned alculquicondor Mar 20, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 20, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 20, 2024

k8s-ci-robot merged commit 530fafc into kubernetes-sigs:main Mar 20, 2024
14 checks passed

k8s-ci-robot added this to the v0.7 milestone Mar 20, 2024

trasc deleted the importer branch March 22, 2024 06:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add pod importer #1825

Add pod importer #1825

trasc commented Mar 12, 2024 •

edited

k8s-ci-robot commented Mar 12, 2024

netlify bot commented Mar 12, 2024 •

edited

yaroslava-serdiuk commented Mar 19, 2024

yaroslava-serdiuk Mar 20, 2024

trasc Mar 20, 2024

yaroslava-serdiuk Mar 20, 2024

trasc Mar 20, 2024

alculquicondor Mar 20, 2024

trasc Mar 20, 2024

alculquicondor Mar 20, 2024

alculquicondor Mar 20, 2024

trasc Mar 20, 2024

trasc Mar 20, 2024

alculquicondor Mar 20, 2024

alculquicondor Mar 20, 2024

trasc Mar 20, 2024

alculquicondor Mar 20, 2024

trasc Mar 20, 2024

alculquicondor left a comment

k8s-ci-robot commented Mar 20, 2024

k8s-ci-robot commented Mar 20, 2024

	log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])
	log.Info("Check failed for Pod", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

	log.Info("Error", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])
	log.Info("Could not import Pod", "err", e, "occurrences", len(pods), "obsevedFirstIn", pods[0])

Add pod importer #1825

Add pod importer #1825

Conversation

trasc commented Mar 12, 2024 • edited

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

k8s-ci-robot commented Mar 12, 2024

netlify bot commented Mar 12, 2024 • edited

✅ Deploy Preview for kubernetes-sigs-kueue canceled.

yaroslava-serdiuk commented Mar 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alculquicondor left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Mar 20, 2024

k8s-ci-robot commented Mar 20, 2024

trasc commented Mar 12, 2024 •

edited

netlify bot commented Mar 12, 2024 •

edited