feat: channels use consumergroups

[Cali0707]

Fixes #3047

As part of the KEDA Kafka component scaling, we need to move channels to use consumergroups internally. This makes that change.

Proposed Changes

• Use the channelv2 reconciler implementation
• Use statefulsets instead of deployments for the channel

Release Note

Channels now use consumergroups internally. IMPORTANT: to downgrade from this release, you will need to delete any consumergroups owned by a channel, and delete the kafka-channel-receiver and kafka-channel-dispatcher statefulsets

Docs

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Cali0707

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [Cali0707]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Cali0707]

@pierDipi I'm planning on removing the channel v1 implementation in a follow up PR, so that it's easier to see any changes I make in this PR (rather than having a complicated diff)

[codecov]

Codecov Report

Attention: Patch coverage is 72.38806% with 74 lines in your changes are missing coverage. Please review.

Project coverage is 48.45%. Comparing base (aacc057) to head (b4f283e).
Report is 18 commits behind head on main.

Files	Patch %	Lines
control-plane/pkg/reconciler/channel/channel.go	73.47%	46 Missing and 15 partials ⚠️
...md/post-install/kafka_broker_deployment_deleter.go	0.00%	7 Missing ⚠️
control-plane/pkg/reconciler/channel/controller.go	88.00%	2 Missing and 1 partial ⚠️
control-plane/cmd/post-install/main.go	0.00%	2 Missing ⚠️
control-plane/cmd/kafka-controller/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #3816       +/-   ##
=============================================
- Coverage     73.70%   48.45%   -25.25%     
=============================================
  Files           100      246      +146     
  Lines          3419    14474    +11055     
  Branches        292        0      -292     
=============================================
+ Hits           2520     7014     +4494     
- Misses          723     6755     +6032     
- Partials        176      705      +529     

Flag	Coverage Δ
java-unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Cali0707]

Looks like everything is broken 😅
/hold

[Cali0707]

/cc @creydr @pierDipi

I made some changes around OIDC/TrustBundles for channelv2, where I am a lot less familiar with the code. Would you mind taking a look at the changes and seeing if they make sense?

[Cali0707]

/unhold

I think things should work now (the failing tests passed when I ran them locally, I think they are flaky in CI)

[Cali0707]

/retest-required

[Cali0707]

/cc @creydr

Would you mind taking another look now that the merge conflicts are fixed?

[creydr]

@Cali0707 thanks for updating. Can you check on @pierDipi comments? (e.g. #3816 (review))

[Cali0707]

/cc @creydr @pierDipi

Thanks for the reminder about the earlier comments @creydr!

[Cali0707]

/retest-required

[Cali0707]

/retest-required

[Cali0707]

@pierDipi @creydr could you take another look at this PR? Seems like everything is passing 🎉

[pierDipi]

same as the other one, received should be kept to Deployment

[pierDipi]

if we move to the top-level pkg and remove the old one, I think we don't need this update

[Cali0707]

/cc @pierDipi

Thanks for all the reviews so far!

[Cali0707]

     wait.go:200: test-yhaulcep/containersource-ndhkncsn condition is {"type":"Ready","status":"False","lastTransitionTime":"2024-05-14T16:28:18Z","reason":"TrustBundlePropagation","message":"failed to create ConfigMap test-yhaulcep/knative-eventing-bundlekne-bundle: configmaps \"knative-eventing-bundlekne-bundle\" already exists"} 

Looks like trust bundle issues on the container source

/retest-required

[Cali0707]

/test integration-tests

[knative-prow]

@Cali0707: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
reconciler-tests_eventing-kafka-broker_main	b4f283e	link	true	/test reconciler-tests

Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.