OCPBUGS-26498: Add test for UpgradeValidation contention #28710
Conversation
openshift-ci-robot
added
the
jira/severity-moderate
openshift-ci
bot
added
the
do-not-merge/work-in-progress
openshift-ci-robot
added
the
jira/valid-reference
|
@gcs278: This pull request references Jira Issue OCPBUGS-26498, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-bug
gcs278
force-pushed
the
OCPBUGS-26498-upgradeable-status
branch
3 times, most recently
from
743ed9b
to
d40ce53
Compare
|
Job Failure Risk Analysis for sha: d40ce53
|
gcs278
force-pushed
the
OCPBUGS-26498-upgradeable-status
branch
from
d40ce53
to
4fd7700
Compare
|
Job Failure Risk Analysis for sha: 4fd7700
|
|
openshift/router#575 has merged. /test all |
There was a problem hiding this comment.
Some Observations:
-
Redundant Check: Is there an existing bug related
to a redundant check (routeNameMatch != routeNameMatch), which
looks like it should be corrected tor.Name != routeNameMatch. There might be gaps in prior testing strategies. -
Error Handling Strategy: Current implementations within utility functions use Ginkgo assertions directly, which limits the flexibility in error handling and makes the utility functions tightly coupled with the test framework. This approach can be restrictive and obscures the utility functions’ usability in different contexts.
test/extended/router/stress.go
Outdated
| } | ||
|
|
||
| // createTestRoutes creates test routes with the name as the index number. | ||
| func createTestRoutes(client v1.RouteInterface, numOfRoutes int) { |
There was a problem hiding this comment.
Why not return an error and let the caller decide what should be done versus calling o.Expect(err).NotTo(o.HaveOccurred()). I see that other functions in this file adopt that pattern of do-stuff and call o.Expect() but you could separate the concerns:
| func createTestRoutes(client v1.RouteInterface, numOfRoutes int) { | |
| func createTestRoutes(client v1.RouteInterface, numOfRoutes int) error { | |
| var errs []error | |
| for i := 0; i < numOfRoutes; i++ { | |
| _, err := client.Create(context.Background(), &routev1.Route{ | |
| ObjectMeta: metav1.ObjectMeta{ | |
| Name: fmt.Sprintf("%d", i), | |
| }, | |
| Spec: routev1.RouteSpec{ | |
| To: routev1.RouteTargetReference{Name: "test"}, | |
| Port: &routev1.RoutePort{ | |
| TargetPort: intstr.FromInt(8080), | |
| }, | |
| }, | |
| }, metav1.CreateOptions{}) | |
| if err != nil { | |
| errs = append(errs, fmt.Errorf("failed to create route %d: %w", i, err)) | |
| } | |
| } | |
| if len(errs) > 0 { | |
| return fmt.Errorf("multiple errors occurred: %v", errs) | |
| } | |
| return nil | |
| } |
test/extended/router/stress.go
Outdated
| func findIngressCondition(ingress *routev1.RouteIngress, t routev1.RouteIngressConditionType) (_ *routev1.RouteIngressCondition) { | ||
| for i, existing := range ingress.Conditions { | ||
| if existing.Type != t { | ||
| continue | ||
| } | ||
| return &ingress.Conditions[i] | ||
| } | ||
| return nil | ||
| } |
There was a problem hiding this comment.
| func findIngressCondition(ingress *routev1.RouteIngress, t routev1.RouteIngressConditionType) (_ *routev1.RouteIngressCondition) { | |
| for i, existing := range ingress.Conditions { | |
| if existing.Type != t { | |
| continue | |
| } | |
| return &ingress.Conditions[i] | |
| } | |
| return nil | |
| } | |
| func findIngressCondition(ingress *routev1.RouteIngress, t routev1.RouteIngressConditionType) (_ *routev1.RouteIngressCondition) { | |
| for i := range ingress.Conditions { | |
| if ingress.Conditions[i].Type == t { | |
| return &ingress.Conditions[i] | |
| } | |
| } | |
| return nil | |
| } |
test/extended/router/stress.go
Outdated
| o.Expect(ok).To(o.BeTrue()) | ||
| if routeNameMatch != "" { | ||
| if r, ok := obj.Object.(*routev1.Route); ok { | ||
| if r == nil || routeNameMatch != routeNameMatch { |
There was a problem hiding this comment.
We're comparing the same variable. Typo? Should this be r.Name !=? (Later edit: looks like this code was always encoded this way.)
There was a problem hiding this comment.
Oh yikes that's a goof. Good catch.
test/extended/router/stress.go
Outdated
| func verifyConflictingWrites(client v1.RouteInterface, rv string, observeTime time.Duration, writeLimit int, routeNameMatch string) { | ||
| writes := 0 | ||
| w, err := client.Watch(context.Background(), metav1.ListOptions{Watch: true, ResourceVersion: rv}) | ||
| o.Expect(err).NotTo(o.HaveOccurred()) |
There was a problem hiding this comment.
Should a utility function assert like this? Why don't we return an error and let the caller decide? i.e., Shouldn't the assertions be triggered in g.It("converges when multiple routers are writing status", func() {, etc.
There was a problem hiding this comment.
Yea I it'd be more reusable if we don't do the assertions in the utility function. Done.
gcs278
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
gcs278
force-pushed
the
OCPBUGS-26498-upgradeable-status
branch
from
4fd7700
to
f4d848c
Compare
test/extended/router/stress.go
Outdated
| } | ||
|
|
||
| // createTestRoutes creates test routes with the name as the index number. | ||
| func createTestRoutes(client v1.RouteInterface, numOfRoutes int) { |
test/extended/router/stress.go
Outdated
| func verifyConflictingWrites(client v1.RouteInterface, rv string, observeTime time.Duration, writeLimit int, routeNameMatch string) { | ||
| writes := 0 | ||
| w, err := client.Watch(context.Background(), metav1.ListOptions{Watch: true, ResourceVersion: rv}) | ||
| o.Expect(err).NotTo(o.HaveOccurred()) |
There was a problem hiding this comment.
Yea I it'd be more reusable if we don't do the assertions in the utility function. Done.
test/extended/router/stress.go
Outdated
| o.Expect(ok).To(o.BeTrue()) | ||
| if routeNameMatch != "" { | ||
| if r, ok := obj.Object.(*routev1.Route); ok { | ||
| if r == nil || routeNameMatch != routeNameMatch { |
There was a problem hiding this comment.
Oh yikes that's a goof. Good catch.
test/extended/router/stress.go
Outdated
| func findIngressCondition(ingress *routev1.RouteIngress, t routev1.RouteIngressConditionType) (_ *routev1.RouteIngressCondition) { | ||
| for i, existing := range ingress.Conditions { | ||
| if existing.Type != t { | ||
| continue | ||
| } | ||
| return &ingress.Conditions[i] | ||
| } | ||
| return nil | ||
| } |
|
All other test failures weren't related to our new test The test is in e2e-gcp-ovn and it was succesfull. Spinning a couple more times. |
|
|
|
|
|
Image builds are failing. Appears like infrastructure issue. |
|
Job Failure Risk Analysis for sha: f4d848c
|
|
@frobware found a flake. holding until we figure it out. |
openshift-ci
bot
added
the
do-not-merge/hold
|
/assign @frobware |
gcs278
force-pushed
the
OCPBUGS-26498-upgradeable-status
branch
from
966f2ea
to
f4d32ea
Compare
|
/test e2e-gcp-ovn |
|
/test e2e-aws-ovn-fips |
|
/test e2e-aws-ovn-single-node |
test/extended/router/stress.go
Outdated
| func findMostRecentConditionTime(conditions []routev1.RouteIngressCondition) time.Time { | ||
| var recent time.Time | ||
| for j := range conditions { | ||
| condition := &conditions[j] | ||
| if condition.LastTransitionTime != nil && condition.LastTransitionTime.Time.After(recent) { | ||
| recent = condition.LastTransitionTime.Time | ||
| } | ||
| } | ||
| return recent | ||
| } |
There was a problem hiding this comment.
Do we need the temporary?
| func findMostRecentConditionTime(conditions []routev1.RouteIngressCondition) time.Time { | |
| var recent time.Time | |
| for j := range conditions { | |
| condition := &conditions[j] | |
| if condition.LastTransitionTime != nil && condition.LastTransitionTime.Time.After(recent) { | |
| recent = condition.LastTransitionTime.Time | |
| } | |
| } | |
| return recent | |
| } | |
| func findMostRecentConditionTime(conditions []routev1.RouteIngressCondition) time.Time { | |
| var recent time.Time | |
| for j := range conditions { | |
| if conditions[j].LastTransitionTime != nil && conditions[j].LastTransitionTime.Time.After(recent) { | |
| recent = conditions[j].LastTransitionTime.Time | |
| } | |
| } | |
| return recent | |
| } |
Add "The HAProxy router converges when multiple routers are writing conflicting upgrade validation status" test which validates router converge when writing conflicting status in a scenario that uses multiple conditions. Previously, we tested conflicting status fields (hostname), but don't have a test for conflicting status. This test add logic that exercises new logic in the router for the Upgrade Validation plugin.
gcs278
force-pushed
the
OCPBUGS-26498-upgradeable-status
branch
from
f4d32ea
to
3341e1c
Compare
|
/lgtm Great work! |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: frobware, gcs278, Miciah The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
Job Failure Risk Analysis for sha: 3341e1c
|
|
I feel pretty comfortable after 250+ successful runs on And Regardless, I think this test is resilient enough to go ahead and merge. |
openshift-ci
bot
removed
the
do-not-merge/hold
|
Tide is saying I need this job, but it's not showing required 🤷 |
|
/retest-required |
1 similar comment
|
/retest-required |
|
Flakes are unrelated:
/test e2e-gcp-ovn-builds |
|
@Miciah requesting override for e2e-gcp-ovn-builds (if you have the permissions...) The job history shows that it's almost nearly permafailing for multiple PRs, not just this one. I've followed up in slack in #forum-openshift-builds. I'll report a bug if pending the outcome of the slack conversation. |
|
@gcs278: The following tests failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
Another failure: |
|
/override e2e-gcp-ovn-builds |
|
@xueqzhan: /override requires failed status contexts, check run or a prowjob name to operate on.
Only the following failed contexts/checkruns were expected:
If you are trying to override a checkrun that has a space in it, you must put a double quote on the context. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/override ci/prow/e2e-gcp-ovn-builds |
|
@xueqzhan: Overrode contexts on behalf of xueqzhan: ci/prow/e2e-gcp-ovn-builds In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-merge-bot
bot
merged commit 3a0e98a
into
openshift:master
|
@gcs278: Jira Issue OCPBUGS-26498: Some pull requests linked via external trackers have merged: The following pull requests linked via external trackers have not merged: These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-26498 has not been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Add "The HAProxy router converges when multiple routers are writing conflicting upgrade validation status" test which validates router converge when writing conflicting status in a scenario that uses multiple conditions.
Previously, we tested conflicting status fields (hostname), but don't have a test for conflicting status. This test add logic that exercises new logic in the router for the Upgrade Validation plugin.
Also, this updates all of the router stress.go status tests write watch logic to use an informer, which allows for comparison of the old and new route objects along with the ability to start the informer in a separate go routine.