Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

e2e tests should fail fast #5672

Closed
Tracked by #5671
jbw976 opened this issue May 10, 2024 · 0 comments · Fixed by #5673
Closed
Tracked by #5671

e2e tests should fail fast #5672

jbw976 opened this issue May 10, 2024 · 0 comments · Fixed by #5673
Labels
e2e enhancement New feature or request good first issue Good for newcomers test
Milestone

Comments

@jbw976
Copy link
Member

jbw976 commented May 10, 2024

What problem are you facing?

In our e2e test framework, is that it looks like a failure won't stop the rest of the checks from being performed, so the test continues after what should be a fatal failure.

For example, In Assess("foo", funcs.AllOf(.... It seems reasonable to fail quickly if we've already detected a failed condition, instead of wasting more time on further checks that could take minute(s) each.

In the test output copied for convenience below, we fail an assessment (context deadline exceeded) and then proceed to perform more checks like Waiting 5m0s for... and Ensuring foo does not change within 1m0s.....

feature.go:741: XNopResource.nop.example.org propagate-fields-removals-l2dkr doesn't yet have desired value at field path spec.parameters.tags[newtag]:   any(
        - 	s"NotFound",
        + 	string("v2"),
          )
    env.go:428: XR did not have desired value "NotFound" at field path spec.parameters.tags[newtag]: context deadline exceeded:

<ommitted>

    env.go:428: Ensuring claim NopResource.nop.example.org default/propagate-fields-removals does not change within 1m0s
    env.go:428: Claim NopResource.nop.example.org default/propagate-fields-removals did not change within 1m0s
    env.go:428: Waiting 5m0s for NopResource.nop.example.org default/propagate-fields-removals to have value "I'm cool!" at field path status.coolerField...
    env.go:428: 1 resources have desired value "I'm cool!" at field path status.coolerField after 0.506s
    env.go:428: Ensuring composite resource XNopResource.nop.example.org propagate-fields-removals-l2dkr does not change within 1m0s
    env.go:428: Composite resource XNopResource.nop.example.org propagate-fields-removals-l2dkr did not change within 1m0s

How could Crossplane help solve your problem?

If an assessment fails in an e2e test then we should fail the entire test quickly. Performing further checks after a failure isn't ideal because:

  • it extends the running time of the test when we know the outcome isn't going to be successful
  • it adds additional output to the test artifacts that makes it harder to find the original/actual failure
@jbw976 jbw976 added enhancement New feature or request test e2e labels May 10, 2024
@jbw976 jbw976 added the good first issue Good for newcomers label May 10, 2024
@jbw976 jbw976 added this to the v1.17 milestone May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
e2e enhancement New feature or request good first issue Good for newcomers test
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant