New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
NE-1516: Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller #1826
base: master
Are you sure you want to change the base?
Conversation
@miheer: This pull request references NE-1516 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
Hello @miheer! Some important instructions when contributing to openshift/api: |
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: miheer The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
@miheer: This pull request references NE-1516 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.16.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
@@ -151,8 +151,22 @@ type AWSIngressSpec struct { | |||
// +kubebuilder:validation:Enum:=NLB;Classic | |||
// +kubebuilder:validation:Required | |||
Type AWSLBType `json:"type,omitempty"` | |||
|
|||
// networkLoadBalancerParameters holds configuration parameters for an AWS | |||
// network load balancer. Present only if type is NLB. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add CEL to enforce this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
config/v1/types_ingress.go
Outdated
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | ||
// AWS Network load balancer. | ||
type AWSNetworkLoadBalancerParameters struct { | ||
EIPAllocations EIPAllocations `json:"eip-allocations,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does this do? Why is it plural, but only has a single value? Why would I want to set it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations
service.beta.kubernetes.io/aws-load-balancer-eip-allocations specifies a list of elastic IP address configuration for an internet-facing NLB.
So, when the eip-allocations
field is set in the Ingress Controller CR which is comma separated value of type string, we set the value of the service of load balancer type with service annotation mentioned above.
Do you think we should specify this as an array and then seperate the value in the ingress controller based on the delimiter comma ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added an array.
config/v1/types_ingress.go
Outdated
// network load balancer. Present only if type is NLB. | ||
// | ||
// +optional | ||
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"networkLoadBalancer,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add a featuregate to this field. It's done by
- add a featuregate like https://github.com/openshift/api/blob/master/config/v1/feature_gates.go#L182-L188
- add a
// +openshift:enable:FeatureGate=Example
comment to this type - run
hack/update-payload-featuregates.sh
- run
make update
This will add the schema to TechPreviewNoUpgrade
(files now auto-generated). Once your automated tests are passing or QE signs off, a PR enabling by default can flip the gate to true ref
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK. I need to try this. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
/assign @Miciah |
a8e5e98
to
f6536ae
Compare
config/v1/feature_gates.go
Outdated
reportProblemsToJiraComponent("Routing"). | ||
contactPerson("miheer"). | ||
productScope(ocpSpecific). | ||
enableIn(TechPreviewNoUpgrade). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's now also a DevPreview you probably want to enable it on. DevPreviewNoUpgrade, TechPreviewNoUpgrade
} | ||
|
||
// AWSNetworkLoadBalancerParameters holds configuration parameters for an |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you link to where the list parameters is? Can they set anything? Are there any formating requirements?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hi @deads2k ! sorry I did not get this point. Can you please explain more on this ?
config/v1/types_ingress.go
Outdated
// AWS Network load balancer. | ||
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController | ||
type AWSNetworkLoadBalancerParameters struct { | ||
EIPAllocations []EIPAllocations `json:"eipAllocations,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no omitempty.
config/v1/types_ingress.go
Outdated
} | ||
|
||
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | ||
// AWS Network load balancer. | ||
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this probably more appropriately belongs on either networkLoadBalancer
above or eipAllocations
below. I'd probably choose the eipAllocations
operator/v1/types_ingress.go
Outdated
@@ -634,7 +636,9 @@ type AWSClassicLoadBalancerParameters struct { | |||
|
|||
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | |||
// AWS Network load balancer. | |||
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gate the field, not the type. Then this will generator correctly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK done.
operator/v1/types_ingress.go
Outdated
type AWSNetworkLoadBalancerParameters struct { | ||
EIPAllocations []configv1.EIPAllocations `json:"eipAllocations,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need godoc on the field
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok added.
@deads2k I am still getting this issue after running make verify as mentioned in #1826 (comment) :
|
41c7cea
to
194ad66
Compare
crdName: ingresses.config.openshift.io | ||
tests: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The integration job failed, and it looks like the reason is that it is running these tests for the "Default" featureset. Try adding the following to this file:
featureGate: SetEIPForNLBIngressController
…s Controller to set EIP via installer and Ingress Controller `config/v1/types_ingress.go` - Adds an API field `eip-allocations` in the cluster ingress config object whose value is set by the installer using install-config.yaml `operator/v1/types_ingress.go` - Adds an API field `eip-allocations` in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller with service type balancer whose annotation `service.beta.kubernetes.io/aws-load-balancer-eip-allocations` is set by the value of the field `eip-allocations` of the Ingress Controller. Epic: https://issues.redhat.com/browse/NE-1274 Story: https://issues.redhat.com/browse/NE-1516
@miheer: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
} | ||
|
||
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | ||
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html | ||
type AWSNetworkLoadBalancerParameters struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@miheer we already have an identical type in https://github.com/openshift/api/blob/master/operator/v1/types_ingress.go#L516. Can we choose one or the other?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Miciah let me know that we need both of these, but there may be a way to not declare them twice.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are two different APIs. So, we can't.
type AWSNetworkLoadBalancerParameters struct { | ||
// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. | ||
// The number of Allocation IDs must match the number of subnets that are used for the load balancer. | ||
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must the EIP allocation ids have eipalloc-
prefix? Any other restrictions on formatting, length, or uniqueness?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The annotation should not be mentioned at all.
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html | ||
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations | ||
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController | ||
EIPAllocations []EIPAllocations `json:"eipAllocations"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding a maximum slice length, and enforce with CEL.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need for CEL; just use +kubebuilder:validation:MaxItems
.
// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. | ||
// The number of Allocation IDs must match the number of subnets that are used for the load balancer. | ||
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy | ||
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would not point people to this AWS url unless we support all of the actions described there. Just summarize the parts our users need to know.
E.g., do we support "Associate an Elastic IP address with an instance or network interface"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Providing a URL is fine, but only if the godoc explains why a URL is mentioned. If you tell me a URL but don't tell me why you told me that URL, I will ignore it. For example, you can say something like, "See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html for general information about configuration, characteristics, and limitations of Elastic IP addresses."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, but if this AWS url ends up in OpenShift docs, I predict it will cause more questions than it answers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it is fine to reference the AWS documentation if it is relevant. We already reference it here:
api/operator/v1/types_ingress.go
Lines 513 to 535 in 8203151
// AWSLoadBalancerParameters provides configuration settings that are | |
// specific to AWS load balancers. | |
// +union | |
type AWSLoadBalancerParameters struct { | |
// type is the type of AWS load balancer to instantiate for an ingresscontroller. | |
// | |
// Valid values are: | |
// | |
// * "Classic": A Classic Load Balancer that makes routing decisions at either | |
// the transport layer (TCP/SSL) or the application layer (HTTP/HTTPS). See | |
// the following for additional details: | |
// | |
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html#clb | |
// | |
// * "NLB": A Network Load Balancer that makes routing decisions at the | |
// transport layer (TCP/SSL). See the following for additional details: | |
// | |
// https://docs.aws.amazon.com/AmazonECS/latest/developerguide/load-balancer-types.html#nlb | |
// | |
// +unionDiscriminator | |
// +kubebuilder:validation:Required | |
// +required | |
Type AWSLoadBalancerType `json:"type"` |
There are also references to documentation on amazon.com in the Machine API, Ingress config API, DNS config API, and Infrastructure config API, and there are references to alibabacloud.com, cloud.google.com, cloud.ibm.com, docker.com, kubernetes.io and k8s.io, microsoft.com, mozilla.com, and vmware.com documentation throughout OpenShift API definitions. It makes sense to be judicious about linking to non-Red Hat documentation, but as an integrator, we inevitably need to reference external documentation.
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | ||
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html | ||
type AWSNetworkLoadBalancerParameters struct { | ||
// You can assign Elastic IP addresses to the Network Load Balancer by adding the following annotation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The whole point of this API is the the cluster-admin should not use the annotation.
// The number of Allocation IDs must match the number of subnets that are used for the load balancer. | ||
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-xxxxxxxxxxxxxxxxx,eipalloc-yyyyyyyyyyyyyyyyy | ||
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html | ||
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The aws-load-balancer-controller documentation is irrelevant; ELBs for ingresscontrollers are handled by cloud-controller-manager and cloud-provider-aws. Please avoid linking to the aws-load-balancer-controller documentation as it has been the source of much confusion in the past.
EIPAllocations []EIPAllocations `json:"eipAllocations"` | ||
} | ||
|
||
type EIPAllocations string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This definition is still missing godoc.
@@ -151,8 +152,28 @@ type AWSIngressSpec struct { | |||
// +kubebuilder:validation:Enum:=NLB;Classic | |||
// +kubebuilder:validation:Required | |||
Type AWSLBType `json:"type,omitempty"` | |||
|
|||
// networkLoadBalancerParameters holds configuration parameters for an AWS | |||
// network load balancer. Present only if type is NLB. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please capitalize "Network Load Balancer" correctly here and elsewhere.
// network load balancer. Present only if type is NLB. | |
// Network Load Balancer. This field is present only if the type field is set to NLB. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That might be a general use of the phrase "network load balancer", which I saw alot in the EP. If it is an object, it should be networkLoadBalancer, all one word, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the phrase is used in the general sense, then it needs to be defined. I know what a "load balancer" is, but I don't know what a "network load balancer" is (or why the distinction would be relevant here), so I assumed that Miheer meant "Network Load Balancer" (AKA NLB).
@@ -544,7 +546,7 @@ type AWSLoadBalancerParameters struct { | |||
// network load balancer. Present only if type is NLB. | |||
// | |||
// +optional | |||
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"networkLoadBalancer,omitempty"` | |||
NetworkLoadBalancerParameters *AWSNetworkLoadBalancerParameters `json:"nlbParameters,omitempty"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This API field is already in released versions of OpenShift. You cannot change the field name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, that's my fault. I asked for this change. Sorry @miheer.
// https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html | ||
// https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.7/guide/service/annotations/#eip-allocations | ||
// +openshift:enable:FeatureGate=SetEIPForNLBIngressController | ||
EIPAllocations []EIPAllocations `json:"eipAllocations"` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you are going to get pinged by API team on not having // +listType=<type>
tag. See https://github.com/kubernetes/kube-openapi/blob/master/pkg/idl/doc.go. Atomic is generally safe bet.
@JoelSpeed can confirm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a list of strings, I assume each item in the list should be unique right? Will the list have one actor or multiple? I think we probably want this to be atomic and to enforce uniqueness using CEL
Thinking about this which caused issues before with the set
listType
@@ -633,8 +635,15 @@ type AWSClassicLoadBalancerParameters struct { | |||
} | |||
|
|||
// AWSNetworkLoadBalancerParameters holds configuration parameters for an | |||
// AWS Network load balancer. | |||
// AWS Network load balancer. For Example: Setting AWS EIPs https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This "for example" and including a link seems quite odd to me for this type of GO doc. We don't list examples in AWSClassicLoadBalancerParameters
. It seems a bit inconsistent.
EIPAllocations []EIPAllocations `json:"eipAllocations"` | ||
} | ||
|
||
type EIPAllocations string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling this the same name as the field type is confusing to me. I don't think this should be plural, it is a singular EIPAllocation.
type EIPAllocations string | |
type EIPAllocation string |
Adds a field in the cluster ingress config object and Ingress Controller to set EIP via installer and Ingress Controller
config/v1/types_ingress.go
- Adds an API fieldeip-allocations
in the cluster ingress config object whose value is set by the installer using install-config.yamloperator/v1/types_ingress.go
- Adds an API fieldeip-allocations
in the Ingress Controller CR object using which the Ingress Operator scales an Ingress Controller with service type balancer whose annotationservice.beta.kubernetes.io/aws-load-balancer-eip-allocations
is set by the value of the fieldeip-allocations
of the Ingress Controller.Epic: https://issues.redhat.com/browse/NE-1274
Story: https://issues.redhat.com/browse/NE-1516