Increase e2e test reliability

[jbw976]

What problem are you facing?

The e2e tests for Crossplane have not been as reliable as we'd like lately. For example, bumping the crossplane-runtime dependency in preparation for the v1.16 release was delayed due to e2e test failures that had been present in master for awhile (#5652 (comment)).

Ideally, we would be getting consistent green builds where all tests pass, but recently we've been getting more failures that are more likely to be test issues as opposed to genuine issues in the code we ship.

The less reliable our tests are, the less confidence we have in what we're shipping.

How could Crossplane help solve your problem?

This is a high level epic for improving our confidence and reliability in the e2e tests.

Framework improvements

Beta Give feedback

1. Set up buildpulse to start tracking test results (Fix unit test coverage reports, add E2E flake reports #5668)
2. e2e tests should fail fast #5672
   e2e enhancement good first issue test +4
3. test/e2e/funcs: add helpful error context to ApplyResources and others, vs. features.go:94 #5722
   +0

Flaky tests

Beta Give feedback

1. TestConfigurationWithDependency Bump crossplane-runtime to v1.17.0-rc.0 #5652 (comment)
2. TestPropagateFieldsRemovalToXRAfterUpgrade Bump crossplane-runtime dependency to v1.16.0-rc.2 #5653 (comment)
3. Realtime compositions E2E tests flaky #5228
   bug composition e2e flaky-test realtime-composition +5
   
4. TestCompositionInvalidComposed Increase e2e test reliability #5671 (comment)
5. more flaky tests from https://app.buildpulse.io/@crossplane/crossplane

[negz]

I've seen TestCompositionInvalidComposed fail a few times recently when running local tests. I've noticed that it does sometimes successfully create the XChild composed resource - I don't think it's supposed to. Perhaps making the patch that adds the invalid field to the child required would help?

[negz]

https://app.buildpulse.io/@crossplane/crossplane - For the record, this is now tracking our E2E flakes.