Wishlist of CEL libraries

[alculquicondor]

What would you like to be added?

This is just my wishlist. We need all of them, so no specific priority.

• Regexes for object names, label keys, values, container names, etc. I think this one is already in the works?
• validate the type metav1.Conditions
• simple functions to check if conditions are true/false.
• taints and tolerations
• labelSelectors (we don't need this one, but it will probably be useful for some projects)
• The ultimate validation: Pod templates, but worth starting with just containers :) Very useful for job CRDs.
• ... (waiting for feedback)

Why is this needed?

As we were working on transitioning Kueue validation webhooks to CEL, we ended up with a lot of repeated validation lines. In some cases, validating something is so involved that we decided to keep most of the webhooks, otherwise readability would be significantly impacted.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[alculquicondor]

/sig api-machinery
cc @cici37 @jpbetz

[alculquicondor]

@IrvingMg, @trasc did I miss anything important?

[sftim]

What do you mean by Pod templates @alculquicondor?

One pattern I'd like to see used more: when you create something that embeds a Pod template, the controller for that kind tries to dry-run make a PodTemplate. That way you get one place to put customer validation (eg a ValidatingAdmissionPolicy), and it can apply to lots of API kinds without repetition.

[alculquicondor]

One pattern I'd like to see used more: when you create something that embeds a Pod template, the controller for that kind tries to dry-run make a PodTemplate.

Interesting. I've never seen that. It sounds bullet proof from a validation perspective. Is there dry-run support in the apiserver? But then it would have to be called from the webhook?

[jpbetz]

Regexes for object names, label keys, values, container names, etc. I think this one is already in the works?

Yes, this one is progressing here: #123572 (cc @alexzielenski)

[jpbetz]

The ultimate validation: Pod templates, but worth starting with just containers :) Very useful for job CRDs.

We might do something special to validated embedded types like this that doesn't involved CEL. But yes, I agree there is a huge need here. Do you happen to have any references to specific use cases? I'm working on accumulating those.

[liggitt]

Is there dry-run support in the apiserver?

yes, since 1.12: https://github.com/kubernetes/kubernetes/blob/release-1.12/staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/types.go#L485-L491

[alculquicondor]

Re: dry-run support
I see. Still, that would imply that a webhook has to do an API call. Would you still recommend this?

Do you happen to have any references to specific use cases?

1. All of the kubeflow job CRDs, to start. They will eventually fail at Pod or k8s Job creation, if the template is wrong, as they are not doing any validation.
2. Kueue Workload objects https://kueue.sigs.k8s.io/docs/concepts/workload/. These are created out of existing Jobs, Pods, or arbitrary CRDs (like Kubeflow jobs). The only problematic case is the last one.

[kannon92]

3. JobSet validation (based on JobTemplates) Investigate ways to validate Job templates at JobSet validation time kubernetes-sigs/jobset#422

[kannon92]

@danielvegamyhre was looking into kubectl-validate as a way to validate templates as a library.

[IrvingMg]

@IrvingMg, @trasc did I miss anything important?

Not that I can think of. That's everything we need for now.

[alculquicondor]

WRT conditions, right now, we have to do things like this:

has(self.status) && has(self.status.conditions) && self.status.conditions.exists(c, c.type == 'QuotaReserved' && c.status == 'True')

It would be good to have a simplified experience, similar to meta.IsConditionTrue in golang.

I added a separate item for this.

[liggitt]

fwiw, optionals can make that more concise:

self.?status.?conditions.orValue([]).exists(c, c.type == 'QuotaReserved' && c.status == 'True')