Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(apimachinery): deprecate meta.UnsafeGuessKindToResource #119924

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

pmalek
Copy link
Contributor

@pmalek pmalek commented Aug 13, 2023

What type of PR is this?

/kind cleanup
/kind deprecation

What this PR does / why we need it:

This PR marks apimachinery's meta.UnsafeGuessKindToResource as deprecated given its unreliability and heuristic nature.

Which issue(s) this PR fixes:

Related to a discussion in #110053

Special notes for your reviewer:

Should I align all k8s.io/gengo versions to be the same as before this PR and not bump it or is bumping it from k8s.io/gengo v0.0.0-20220902162205-c0856e24416d to k8s.io/gengo v0.0.0-20230306165830-ab3349d207d4 OK? I gather that I'd need to run go mod vendor then as well because running tests in pkg/kubelet complains about this.

Does this PR introduce a user-facing change?

Mark apimachinery's `meta.UnsafeGuessKindToResource`. Users are suggested to use `k8s.io/gengo/namer`'s `NewAllLowercasePluralNamer()` instead.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. labels Aug 13, 2023
@k8s-ci-robot
Copy link
Contributor

Please note that we're already in Test Freeze for the release-1.28 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.28.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Sun Aug 13 10:37:24 UTC 2023.

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. area/test do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 13, 2023
@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 13, 2023
@k8s-ci-robot
Copy link
Contributor

Hi @pmalek. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 13, 2023
@shiva1333
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 14, 2023
@@ -62,6 +64,8 @@ type DefaultRESTMapper struct {
kindToScope map[schema.GroupVersionKind]RESTScope
singularToPlural map[schema.GroupVersionResource]schema.GroupVersionResource
pluralToSingular map[schema.GroupVersionResource]schema.GroupVersionResource

namer namer.Namer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may give fmt error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK. What's the fmt tool to check this and how can I trigger it? gofmt doesn't report anything

@k8s-ci-robot k8s-ci-robot added the area/dependency Issues or PRs related to dependency changes label Aug 14, 2023
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pmalek
Once this PR has been reviewed and has the lgtm label, please assign liggitt for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added area/kubectl sig/cli Categorizes an issue or PR as relevant to SIG CLI. labels Aug 14, 2023
@pmalek
Copy link
Contributor Author

pmalek commented Aug 14, 2023

/test pull-kubernetes-unit

@pmalek
Copy link
Contributor Author

pmalek commented Aug 14, 2023

/retest-required

@pmalek
Copy link
Contributor Author

pmalek commented Aug 14, 2023

I've checked locally an all UTs pass but there are some build related errors that I'm not sure what they are about. Can someone guide me how to approach this? https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/119924/pull-kubernetes-unit/1691019004237320192

@@ -108,7 +108,7 @@ func NewDiscoveryRESTMapper(groupResources []*APIGroupResources) meta.RESTMapper
singular := gv.WithResource(resource.SingularName)
// this is for legacy resources and servers which don't list singular forms. For those we must still guess.
if len(resource.SingularName) == 0 {
_, singular = meta.UnsafeGuessKindToResource(gv.WithKind(resource.Kind))
singular = gv.WithResource(strings.ToLower(resource.Kind))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change looks good

@@ -24,6 +24,8 @@ import (

"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/gengo/namer"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gengo is for code-generation prior to build, not something we want to rely on at runtime

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. What suggestion do you have on this one? namer ability to provide plural names was suggested in #110053 (comment) as a replacement for UnsafeGuessKindToResource. What other options do we have?

@@ -93,11 +97,15 @@ func NewDefaultRESTMapper(defaultGroupVersions []schema.GroupVersion) *DefaultRE
defaultGroupVersions: defaultGroupVersions,
singularToPlural: singularToPlural,
pluralToSingular: pluralToSingular,
namer: namer.NewAllLowercasePluralNamer(nil),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see how embedding a namer whose exceptions can't be overridden is an improvement on the current code

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending response on the alternative for namer one question pops in my head: what improvement would you possibly see here? Make NewDefaultRESTMapper a variadic function accepting exceptions or add second "constructor" which allows to specify the exceptions keeping this intact (as is now)?

@@ -123,6 +131,9 @@ var unpluralizedSuffixes = []string{
// UnsafeGuessKindToResource converts Kind to a resource name.
// Broken. This method only "sort of" works when used outside of this package. It assumes that Kinds and Resources match
// and they aren't guaranteed to do so.
//
// Deprecated: This is and may yield unwanted and unexpected results.
// Use k8s.io/gengo/namer's NewAllLowercasePluralNamer() instead.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment about not depending on gengo at runtime

@@ -228,6 +231,9 @@ func NewObjectTracker(scheme ObjectScheme, decoder runtime.Decoder) ObjectTracke
decoder: decoder,
objects: make(map[schema.GroupVersionResource]map[types.NamespacedName]runtime.Object),
watchers: make(map[schema.GroupVersionResource]map[string][]*watch.RaceFreeFakeWatcher),
namer: namer.NewAllLowercasePluralNamer(map[string]string{
"Endpoints": "endpoints",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really see how this is an improvement on the current state... this is taking the existing unsafe kind → resource code and replicating the exceptions in multiple other places.

To actually fix this, the kind → resource transformation has to be configurable in the fake clients / testing fixtures

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. I'd assume that you mean all the client "constructors" that utilize NewObjectTracker e.g.

func NewSimpleClientset(objects ...runtime.Object) *Clientset {
o := testing.NewObjectTracker(scheme, codecs.UniversalDecoder())
.

What would you suggest we change/add to support said configurability? We can't introduce a breaking change (even if we wanted NewSimpleClientset already has a variadic parameter set objects). Do we want to add something like NewSimpleClientsetWithExpections? 🤔

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I think NewObjectTracker and NewSimpleClientset should have variants that allow providing custom kind → resource mappings

Copy link
Member

@liggitt liggitt Aug 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can give ourselves WithOptions signatures to only have to replumb once, with something like

// NewSimpleClientset returns a clientset that will respond with the provided objects.
// It's backed by a very simple object tracker that processes creates, updates and deletions as-is,
// without applying any validations and/or defaults. It shouldn't be considered a replacement
// for a real clientset and is mostly useful in simple unit tests.
// Equivalent to NewSimpleClientsetWithOptions(testing.Options{}, objects...)
func NewSimpleClientset(objects ...runtime.Object) *Clientset {
	return NewSimpleClientsetWithOptions(testing.Options{}, objects...)
}

// NewSimpleClientset returns a clientset that will respond with the provided objects.
// It's backed by a very simple object tracker that processes creates, updates and deletions as-is,
// without applying any validations and/or defaults. It shouldn't be considered a replacement
// for a real clientset and is mostly useful in simple unit tests.
func NewSimpleClientsetWithOptions(options testing.Options, objects ...runtime.Object) *Clientset {
	o := testing.NewObjectTrackerWithOptions(options, scheme, codecs.UniversalDecoder())

and

diff --git a/staging/src/k8s.io/client-go/testing/fixture.go b/staging/src/k8s.io/client-go/testing/fixture.go
index 396840670fd..38ecf9745a6 100644
--- a/staging/src/k8s.io/client-go/testing/fixture.go
+++ b/staging/src/k8s.io/client-go/testing/fixture.go
@@ -216,18 +216,34 @@ type tracker struct {
 	// watchers' channel. Note that too many unhandled events (currently 100,
 	// see apimachinery/pkg/watch.DefaultChanSize) will cause a panic.
 	watchers map[schema.GroupVersionResource]map[string][]*watch.RaceFreeFakeWatcher
+
+	// ...
+	kindToResource map[schema.GroupVersionKind]schema.GroupVersionResource
 }
 
 var _ ObjectTracker = &tracker{}
 
+// ...
+type Options struct {
+	// KindToResource allows customizing the mapping for a given kind.
+	// If unset, UnsafeGuessKindToResource is used to attempt to derive the resource for a given kind.
+	KindToResource map[schema.GroupVersionKind]schema.GroupVersionResource
+}
+
 // NewObjectTracker returns an ObjectTracker that can be used to keep track
 // of objects for the fake clientset. Mostly useful for unit tests.
+// Equivalent to NewObjectTrackerWithOptions(Options{}, scheme, decoder)
 func NewObjectTracker(scheme ObjectScheme, decoder runtime.Decoder) ObjectTracker {
+	return NewObjectTrackerWithOptions(Options{}, scheme, decoder)
+}
+
+func NewObjectTrackerWithOptions(options Options, scheme ObjectScheme, decoder runtime.Decoder) ObjectTracker {
 	return &tracker{
 		scheme:   scheme,
 		decoder:  decoder,
 		objects:  make(map[schema.GroupVersionResource]map[types.NamespacedName]runtime.Object),
 		watchers: make(map[schema.GroupVersionResource]map[string][]*watch.RaceFreeFakeWatcher),
+
+		kindToResource: options.KindToResource,
 	}
 }
 
@@ -333,6 +349,8 @@ func (t *tracker) Add(obj runtime.Object) error {
 		return fmt.Errorf("no registered kinds for %v", obj)
 	}
 	for _, gvk := range gvks {
+               gvr, ok := t.kindToResource[gvk]
+               if !ok {
                        // NOTE: UnsafeGuessKindToResource is a heuristic and default match. The
                        // actual registration in apiserver can specify arbitrary route for a
                        // gvk. If a test uses such objects, it cannot preset the tracker with
                        // objects via Add(). Instead, it should trigger the Create() function
                        // of the tracker, where an arbitrary gvr can be specified.
-               gvr, _ := meta.UnsafeGuessKindToResource(gvk)
+                       gvr, _ = meta.UnsafeGuessKindToResource(gvk)
                        // Resource doesn't have the concept of "__internal" version, just set it to "".
                        if gvr.Version == runtime.APIVersionInternal {
                                gvr.Version = ""
                        }
+               }
 
 		err := t.add(gvr, obj, objMeta.GetNamespace(), false)
 		if err != nil {

@@ -278,12 +280,17 @@ func TestPatchWithMissingObject(t *testing.T) {
func TestGetWithExactMatch(t *testing.T) {
scheme := runtime.NewScheme()
codecs := serializer.NewCodecFactory(scheme)
namer := namer.NewAllLowercasePluralNamer(nil)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using a namer without configurable exceptions just bakes in the special cases at another layer... same comment applies everywhere we're using NewAllLowercasePluralNamer(nil)


plural, singular := UnsafeGuessKindToResource(version.WithKind(testCase.Kind))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert the changes to this test... this is exercising UnsafeGuessKindToResource which we don't want to lose coverage of

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, reverted this but I retained the change using the subtests and adding the Endpoints test case.

@alexzielenski
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 15, 2023
@pmalek pmalek requested a review from liggitt August 19, 2023 11:41
@k8s-ci-robot
Copy link
Contributor

@pmalek: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubernetes-integration 4d1cf44 link true /test pull-kubernetes-integration
pull-kubernetes-e2e-gce 4d1cf44 link true /test pull-kubernetes-e2e-gce
pull-kubernetes-conformance-kind-ipv6-parallel 4d1cf44 link false /test pull-kubernetes-conformance-kind-ipv6-parallel
pull-kubernetes-node-e2e-containerd 4d1cf44 link true /test pull-kubernetes-node-e2e-containerd
pull-kubernetes-dependencies 4d1cf44 link true /test pull-kubernetes-dependencies
pull-kubernetes-e2e-kind 4d1cf44 link true /test pull-kubernetes-e2e-kind
pull-kubernetes-conformance-kind-ga-only-parallel 4d1cf44 link true /test pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-e2e-kind-ipv6 4d1cf44 link true /test pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-verify 4d1cf44 link true /test pull-kubernetes-verify
pull-kubernetes-unit 4d1cf44 link true /test pull-kubernetes-unit

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 2, 2023
@k8s-ci-robot
Copy link
Contributor

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@dims dims added the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Oct 24, 2023
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

@k8s-ci-robot
Copy link
Contributor

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

  • Reopen this PR with /reopen
  • Mark this PR as fresh with /remove-lifecycle rotten
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@pmalek
Copy link
Contributor Author

pmalek commented Feb 4, 2024

/reopen
/remove-lifecycle rotten

@k8s-ci-robot k8s-ci-robot reopened this Feb 4, 2024
@k8s-ci-robot
Copy link
Contributor

@pmalek: Reopened this PR.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Feb 4, 2024
@pohly
Copy link
Contributor

pohly commented Mar 14, 2024

/cc

Needed for #123938

@k8s-ci-robot k8s-ci-robot requested a review from pohly March 14, 2024 17:21
@pmalek
Copy link
Contributor Author

pmalek commented Mar 14, 2024

Hi @pohly,

This one has gotten a little bit stale because I lacked time to work on it but I might find some time these days. It's actually good that other people will also benefit from this being fixed.

What's the time horizon for #123938 to get merged so that I roughly can estimate if I'd find some time to work on this

@pohly
Copy link
Contributor

pohly commented Mar 14, 2024

I'm using a workaround in #123938 because it should better get merged before test freeze next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/dependency Issues or PRs related to dependency changes area/kubectl area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. kind/deprecation Categorizes issue or PR as related to a feature/enhancement marked for deprecation. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/cli Categorizes an issue or PR as relevant to SIG CLI. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

None yet

8 participants