Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NE-1516: Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller #1826

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

miheer
Copy link
Contributor

@miheer miheer commented Mar 23, 2024

Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller

config/v1/types_ingress.go - Adds an API field eip-allocations in the cluster ingress config object whose value is set by the installer using install-config.yaml

operator/v1/types_ingress.go - Adds an API field eip-allocations in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller with service type balancer whose annotation

service.beta.kubernetes.io/aws-load-balancer-eip-allocations is set by the value of the field eip-allocations of the Ingress Controller.

Epic: https://issues.redhat.com/browse/NE-1274

Story: https://issues.redhat.com/browse/NE-1516

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 23, 2024

@miheer: This pull request references NE-1516 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller

config/v1/types_ingress.go - Adds an API field eip-allocations in the cluster ingress config object whose value is set by the installer using install-config.yaml operator/v1/types_ingress.go - Adds an API field eip-allocations in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller with service type balancer whose annotation service.beta.kubernetes.io/aws-load-balancer-eip-allocations is set by the value of the field eip-allocations of the Ingress Controller. Epic: https://issues.redhat.com/browse/NE-1274
Story: https://issues.redhat.com/browse/NE-1516

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 23, 2024
Copy link
Contributor

openshift-ci bot commented Mar 23, 2024

Hello @miheer! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

@openshift-ci openshift-ci bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Mar 23, 2024
@openshift-ci openshift-ci bot requested review from knobunc and mfojtik March 23, 2024 01:13
Copy link
Contributor

openshift-ci bot commented Mar 23, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: miheer
Once this PR has been reviewed and has the lgtm label, please assign knobunc for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 23, 2024

@miheer: This pull request references NE-1516 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set.

In response to this:

Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller

config/v1/types_ingress.go - Adds an API field eip-allocations in the cluster ingress config object whose value is set by the installer using install-config.yaml

operator/v1/types_ingress.go - Adds an API field eip-allocations in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller with service type balancer whose annotation

service.beta.kubernetes.io/aws-load-balancer-eip-allocations is set by the value of the field eip-allocations of the Ingress Controller.

Epic: https://issues.redhat.com/browse/NE-1274

Story: https://issues.redhat.com/browse/NE-1516

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@@ -151,8 +151,22 @@ type AWSIngressSpec struct {
// +kubebuilder:validation:Enum:=NLB;Classic
// +kubebuilder:validation:Required
Type AWSLBType `json:"type,omitempty"`

// networkLoadBalancerParameters holds configuration parameters for an AWS
// network load balancer. Present only if type is NLB.
Copy link
Contributor

@deads2k deads2k Mar 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add CEL to enforce this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer.
type AWSNetworkLoadBalancerParameters struct {
EIPAllocations EIPAllocations `json:"eip-allocations,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this do? Why is it plural, but only has a single value? Why would I want to set it?

Copy link
Contributor Author

@miheer miheer Mar 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations
service.beta.kubernetes.io/aws-load-balancer-eip-allocations specifies a list of elastic IP address configuration for an internet-facing NLB.
So, when the eip-allocations field is set in the Ingress Controller CR which is comma separated value of type string, we set the value of the service of load balancer type with service annotation mentioned above.
Do you think we should specify this as an array and then seperate the value in the ingress controller based on the delimiter comma ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added an array.

// network load balancer. Present only if type is NLB.
//
// +optional
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"networkLoadBalancer,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a featuregate to this field. It's done by

  1. add a featuregate like https://github.com/openshift/api/blob/master/config/v1/feature_gates.go#L182-L188
  2. add a // +openshift:enable:FeatureGate=Example comment to this type
  3. run hack/update-payload-featuregates.sh
  4. run make update

This will add the schema to TechPreviewNoUpgrade (files now auto-generated). Once your automated tests are passing or QE signs off, a PR enabling by default can flip the gate to true ref

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I need to try this. Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@candita
Copy link
Contributor

candita commented Mar 27, 2024

/assign @Miciah

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 30, 2024
@openshift-ci openshift-ci bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Mar 30, 2024
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Mar 30, 2024
@miheer miheer force-pushed the eip branch 2 times, most recently from a8e5e98 to f6536ae Compare April 1, 2024 03:24
reportProblemsToJiraComponent("Routing").
contactPerson("miheer").
productScope(ocpSpecific).
enableIn(TechPreviewNoUpgrade).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's now also a DevPreview you probably want to enable it on. DevPreviewNoUpgrade, TechPreviewNoUpgrade

}

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link to where the list parameters is? Can they set anything? Are there any formating requirements?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @deads2k ! sorry I did not get this point. Can you please explain more on this ?

// AWS Network load balancer.
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController
type AWSNetworkLoadBalancerParameters struct {
EIPAllocations []EIPAllocations `json:"eipAllocations,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no omitempty.

}

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer.
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this probably more appropriately belongs on either networkLoadBalancer above or eipAllocations below. I'd probably choose the eipAllocations

@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 22, 2024
@@ -634,7 +636,9 @@ type AWSClassicLoadBalancerParameters struct {

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer.
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate the field, not the type. Then this will generator correctly.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK done.

type AWSNetworkLoadBalancerParameters struct {
EIPAllocations []configv1.EIPAllocations `json:"eipAllocations,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need godoc on the field

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok added.

@openshift-merge-robot openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 23, 2024
@openshift-merge-robot openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 23, 2024
@miheer
Copy link
Contributor Author

miheer commented Apr 24, 2024

@deads2k I am still getting this issue after running make verify as mentioned in #1826 (comment) :

Found untracked file 0000_10_config-operator_01_ingresses.crd.yaml in payload CRD manifests. Please add the file to crd_globs in hack/update-payload-crds.sh. make: *** [verify-scripts] Error 1 misalunk@misalunk-mac api %

@miheer miheer force-pushed the eip branch 3 times, most recently from 41c7cea to 194ad66 Compare April 25, 2024 04:05
Comment on lines 3 to 5
crdName: ingresses.config.openshift.io
tests:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The integration job failed, and it looks like the reason is that it is running these tests for the "Default" featureset. Try adding the following to this file:

featureGate: SetEIPForNLBIngressController

…s Controller to set EIP via installer and Ingress Controller

`config/v1/types_ingress.go` - Adds an API field `eip-allocations` in the cluster ingress config object whose value is set by the installer using install-config.yaml
`operator/v1/types_ingress.go` - Adds an API field `eip-allocations` in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller
with service type balancer whose annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` is set by the value of the field `eip-allocations` of the Ingress Controller.
Epic: https://issues.redhat.com/browse/NE-1274
Story: https://issues.redhat.com/browse/NE-1516
Copy link
Contributor

openshift-ci bot commented Apr 25, 2024

@miheer: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-hypershift 24253ce link true /test e2e-aws-ovn-hypershift

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

}

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
type AWSNetworkLoadBalancerParameters struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@miheer we already have an identical type in https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L516. Can we choose one or the other?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Miciah let me know that we need both of these, but there may be a way to not declare them twice.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They are two different APIs. So, we can't.

type AWSNetworkLoadBalancerParameters struct {
// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation.
// The number of Allocation IDs must match the number of subnets that are used for the load balancer.
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy
Copy link
Contributor

@candita candita Apr 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must the EIP allocation ids have eipalloc- prefix? Any other restrictions on formatting, length, or uniqueness?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The annotation should not be mentioned at all.

// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController
EIPAllocations []EIPAllocations `json:"eipAllocations"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding a maximum slice length, and enforce with CEL.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need for CEL; just use +kubebuilder:validation:MaxItems.

// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation.
// The number of Allocation IDs must match the number of subnets that are used for the load balancer.
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would not point people to this AWS url unless we support all of the actions described there. Just summarize the parts our users need to know.

E.g., do we support "Associate an Elastic IP address with an instance or network interface"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providing a URL is fine, but only if the godoc explains why a URL is mentioned. If you tell me a URL but don't tell me why you told me that URL, I will ignore it. For example, you can say something like, "See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html for general information about configuration, characteristics, and limitations of Elastic IP addresses."

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, but if this AWS url ends up in OpenShift docs, I predict it will cause more questions than it answers.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine to reference the AWS documentation if it is relevant. We already reference it here:

// AWSLoadBalancerParameters provides configuration settings that are
// specific to AWS load balancers.
// +union
type AWSLoadBalancerParameters struct {
// type is the type of AWS load balancer to instantiate for an ingresscontroller.
//
// Valid values are:
//
// * "Classic": A Classic Load Balancer that makes routing decisions at either
// the transport layer (TCP/SSL) or the application layer (HTTP/HTTPS). See
// the following for additional details:
//
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html#clb
//
// * "NLB": A Network Load Balancer that makes routing decisions at the
// transport layer (TCP/SSL). See the following for additional details:
//
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html#nlb
//
// +unionDiscriminator
// +kubebuilder:validation:Required
// +required
Type AWSLoadBalancerType `json:"type"`

There are also references to documentation on amazon.com in the Machine API, Ingress config API, DNS config API, and Infrastructure config API, and there are references to alibabacloud.com, cloud.google.com, cloud.ibm.com, docker.com, kubernetes.io and k8s.io, microsoft.com, mozilla.com, and vmware.com documentation throughout OpenShift API definitions. It makes sense to be judicious about linking to non-Red Hat documentation, but as an integrator, we inevitably need to reference external documentation.

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
type AWSNetworkLoadBalancerParameters struct {
// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The whole point of this API is the the cluster-admin should not use the annotation.

// The number of Allocation IDs must match the number of subnets that are used for the load balancer.
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The aws-load-balancer-controller documentation is irrelevant; ELBs for ingresscontrollers are handled by cloud-controller-manager and cloud-provider-aws. Please avoid linking to the aws-load-balancer-controller documentation as it has been the source of much confusion in the past.

EIPAllocations []EIPAllocations `json:"eipAllocations"`
}

type EIPAllocations string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This definition is still missing godoc.

@@ -151,8 +152,28 @@ type AWSIngressSpec struct {
// +kubebuilder:validation:Enum:=NLB;Classic
// +kubebuilder:validation:Required
Type AWSLBType `json:"type,omitempty"`

// networkLoadBalancerParameters holds configuration parameters for an AWS
// network load balancer. Present only if type is NLB.
Copy link
Contributor

@Miciah Miciah Apr 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please capitalize "Network Load Balancer" correctly here and elsewhere.

Suggested change
// network load balancer. Present only if type is NLB.
// Network Load Balancer. This field is present only if the type field is set to NLB.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might be a general use of the phrase "network load balancer", which I saw alot in the EP. If it is an object, it should be networkLoadBalancer, all one word, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the phrase is used in the general sense, then it needs to be defined. I know what a "load balancer" is, but I don't know what a "network load balancer" is (or why the distinction would be relevant here), so I assumed that Miheer meant "Network Load Balancer" (AKA NLB).

@@ -544,7 +546,7 @@ type AWSLoadBalancerParameters struct {
// network load balancer. Present only if type is NLB.
//
// +optional
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"networkLoadBalancer,omitempty"`
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"nlbParameters,omitempty"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This API field is already in released versions of OpenShift. You cannot change the field name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, that's my fault. I asked for this change. Sorry @miheer.

// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController
EIPAllocations []EIPAllocations `json:"eipAllocations"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you are going to get pinged by API team on not having // +listType=<type> tag. See https://github.com/kubernetes/kube-openapi/blob/master/pkg/idl/doc.go. Atomic is generally safe bet.

@JoelSpeed can confirm.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a list of strings, I assume each item in the list should be unique right? Will the list have one actor or multiple? I think we probably want this to be atomic and to enforce uniqueness using CEL

Thinking about this which caused issues before with the set listType

@@ -633,8 +635,15 @@ type AWSClassicLoadBalancerParameters struct {
}

// AWSNetworkLoadBalancerParameters holds configuration parameters for an
// AWS Network load balancer.
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This "for example" and including a link seems quite odd to me for this type of GO doc. We don't list examples in AWSClassicLoadBalancerParameters. It seems a bit inconsistent.

EIPAllocations []EIPAllocations `json:"eipAllocations"`
}

type EIPAllocations string
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling this the same name as the field type is confusing to me. I don't think this should be plural, it is a singular EIPAllocation.

Suggested change
type EIPAllocations string
type EIPAllocation string

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants