Expose Strategy problem #11495

clickersmudge · 2022-12-01T13:44:22Z

clickersmudge
Dec 1, 2022

Hi,
To begin with, thank you for your help so far on other topics. Hope you haven't had enough of me yet :D.

We have a problem with Expose Strategy.

exposeStrategy: LoadBalancer

kubectl apply --filename seed.yaml
   ...
# Optional: ExposeStrategy explicitly sets the expose strategy for this seed cluster, if not set, the default provided by the master is used.
  exposeStrategy: LoadBalancer
  # NodeportProxy can be used to configure the NodePort proxy service that is
  # responsible for making user-cluster control planes accessible from the outside.
  nodeportProxy:
    # Envoy configures the Envoy application itself.
    envoy:
      loadBalancerService:
        # Annotations are used to further tweak the LoadBalancer integration with the
        # cloud provider.
        annotations:
          load-balancer.hetzner.cloud/type: lb11
          load-balancer.hetzner.cloud/location: nbg1
   ...

Scenario 1

Hetzner project "_kubeone_kkp".
Project where KKP is installed

Network: kubeone

Cluster created in the same project as KKP and plugged into the "kubeone" network, creates correctly with machinedeployment.

Scenario 2

Hetzner project "_kubeone_kkp".
Project where KKP is installed

Network: network

machinedeployment still pending.

Scenario 3

Hetzner project "_testowy".

Network: testowy

machinedeployment still pending.

Problem 1
It seems that creating a project and pinning to the same network works. In another scenario, it no longer does. Machines are created in Hetzner, but the process has a continuous pending.
Is this correct behaviour? Should the new clusters be plugged into the same project and the same network?

Problem 2
"kubeconfig-admin-lgfrd82j9z" not working.

kubeconfig-admin-lgfrd82j9z
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHVENDQWdHZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFlTVJ3d0dnWURWUVFERXhOeWIyOTAKTFdOaExqRTVNaTR4TmpndU1DNDNNQjRYRFRJeU1USXdNVEV5TlRFd09Gb1hEVE15TVRFeU9ERXlOVEV3T0ZvdwpIakVjTUJvR0ExVUVBeE1UY205dmRDMWpZUzR4T1RJdU1UWTRMakF1TnpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCCkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU01eG5mYUExT05YZUp4ZmRSdWE5L1loc0ZRRmtIOTJtc2h2N1FNWkl1c0cKVEtUd2VBc3o2OXhSQm5FZk5wWVBaMk5YSXd4amJMb2c2YlVqMWZ0a2NLbWRqRTBseUgxMkFzM0tTRHpoeWk3MwpFazVsQ2xZR2cxY3RhSzhjQ0pqT05SUTFaNzJRWGxJa1lhVG5Bc21VL2t5SFFyaVNxKy9QOWkxNVJVMUxyWlZMCjJjNHlGb3RSTTFKZmtMMDltaW9NTE02OXNIckFLZFFFVmdLaDRWU0NMUWE1STdhUjF0WGY3b3p5dnpCd0RqRHQKZzEwbFFsR3RjNFY1UFV5NnBwQW5SM1dNb1huU29GOE52QjQ5d2YxUU9UeWVBK3BvRDJQaFpQdTNVTmZwb05DNApXNnlYdEFKbHpDZmUzRi9nWEx3ZnFXY1l2R1VXcHduazVJNlp4Qm1HK1ZFQ0F3RUFBYU5pTUdBd0RnWURWUjBQCkFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZPWHA5UXBLa0NXcHdXMWYKSkFaTUM5eGZUL3hITUI0R0ExVWRFUVFYTUJXQ0UzSnZiM1F0WTJFdU1Ua3lMakUyT0M0d0xqY3dEUVlKS29aSQpodmNOQVFFTEJRQURnZ0VCQURORmNOanI0NHBTdFlLNmR3SVlUV2dnOC94dXYrNXMwait5d1ozdUpBRUVaWTVuCnZkbHJndEZSSDFveUh2NEJraHQwcTJDVzdjNTRDNlB1OTkvQUV4V0ozNExSOEJuWHRBR2YraHVCZEZPRGNLQlgKbFkwN0tRa2czZ1k4ajZkNFFqQlFlN3FCUFoyalUrZy9pYTM1Y2Y1Q3pxRG9nU01kQ2RjdXBvZDUyK0VUelRXNApxYSs3UVhkY0d2dkpITlpyMFFlYXVsc1FPK3ZTTklxOTRML1N6SzJ3eWhpWktrRXVsWUpHMmpCclA0U2Vvcyt5CjRpdkh4VHpCcnJ3L3gvRVhmUHU3NWtYZFBtUTE1SXVrZDJ2UmlVVEdYTkVnVGRyM3E3WTNWc1lnU285T0xDc1QKTjZtMGNhQ3V1WHM5UlRlblN1dzhrWUNSUnJNRXA3dmpJSTgvL2NnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    server: https://192.168.0.7:31312
  name: lgfrd82j9z
contexts:
- context:
    cluster: lgfrd82j9z
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    token: 9l8dkr.qg5t4rwtzzstg99c

The problem seems to be the local IP it generates.

exposeStrategy: NodePort

kubectl apply --filename seed.yaml
   ...
# Optional: ExposeStrategy explicitly sets the expose strategy for this seed cluster, if not set, the default provided by the master is used.
  exposeStrategy: NodePort
  # NodeportProxy can be used to configure the NodePort proxy service that is
  # responsible for making user-cluster control planes accessible from the outside.
  nodeportProxy:
    # Envoy configures the Envoy application itself.
    envoy:
      loadBalancerService:
        # Annotations are used to further tweak the LoadBalancer integration with the
        # cloud provider.
        annotations:
          load-balancer.hetzner.cloud/type: lb11
          load-balancer.hetzner.cloud/location: nbg1
   ...

All of the above scenarios " Scenario 1, Scenario 2, Scenario 3" and "kubeconfig" works.

Answered by embik

Dec 1, 2022

@clickersmudge I've checked with some colleagues and it appears that KKP cannot handle the situation in your setup properly for the "LoadBalancer" expose strategy. In short, KKP expects all IPs on a LoadBalancer Service to be "valid" and doesn't check if they're private or public. Since the Hetzner CCM publishes both public and private IPs, it's essentially bad luck that the private IPs take higher priority.

There isn't any workaround unless you convince the Hetzner CCM to not publish private IPs (which I'm not sure is possible, given that the option mentioned above was not effective). It's therefore possible that you cannot use this strategy in your environment right now. We'll however l…

View full answer

embik · 2022-12-01T13:55:16Z

embik
Dec 1, 2022
Collaborator

It appears that when using the LoadBalancer strategy, your load balancers only get assigned private IP addresses in your Hetzner network. It doesn't seem that you're configuring routing between those networks, so it's not surprising nodes fail to join since they cannot talk to the public endpoint of your user cluster.

Check out the Service object in user cluster namespaces to validate they only get assigned a private IP on your Hetzner network.

I'm not entirely sure if this will solve your problem, but going by the Hetzner CCM code and documentation, you might need to add this annotation as well:

load-balancer.hetzner.cloud/use-private-ip: "false"

0 replies

clickersmudge · 2022-12-01T14:30:38Z

clickersmudge
Dec 1, 2022
Author

@embik We will be checking. For now.

kubectl describe services -n cluster-iqzmnslpxp         front-loadbalancer
Name:                     front-loadbalancer
Namespace:                cluster-iqzmnslpxp
Labels:                   <none>
Annotations:              load-balancer.hetzner.cloud/location: nbg1
                          load-balancer.hetzner.cloud/type: lb11
Selector:                 app=nodeport-proxy-envoy
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.99.13.73
IPs:                      10.99.13.73
LoadBalancer Ingress:     167.235.111.8, 2a01:4f8:1c1f:64ba::1, 192.168.0.10
Port:                     cluster-iqzmnslpxp-apiserver-external-31189  31189/TCP
TargetPort:               31189/TCP
NodePort:                 cluster-iqzmnslpxp-apiserver-external-31189  31585/TCP
Endpoints:                10.244.1.148:31189,10.244.4.16:31189
Port:                     cluster-iqzmnslpxp-openvpn-server-32720  32720/TCP
TargetPort:               32720/TCP
NodePort:                 cluster-iqzmnslpxp-openvpn-server-32720  31844/TCP
Endpoints:                10.244.1.148:32720,10.244.4.16:32720
Port:                     healthz  8002/TCP
TargetPort:               8002/TCP
NodePort:                 healthz  30435/TCP
Endpoints:                10.244.1.148:8002,10.244.4.16:8002
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

1 reply

embik Dec 1, 2022
Collaborator

Hm, looks like the LoadBalancer does have an external IP ...

clickersmudge · 2022-12-01T14:59:15Z

clickersmudge
Dec 1, 2022
Author

@embik

Unfortunately, it did not work.

seed.yaml

# Optional: ExposeStrategy explicitly sets the expose strategy for this seed cluster, if not set, the default provided by the master is used.
  exposeStrategy: LoadBalancer
  # NodeportProxy can be used to configure the NodePort proxy service that is
  # responsible for making user-cluster control planes accessible from the outside.
  nodeportProxy:
    # Envoy configures the Envoy application itself.
    envoy:
      loadBalancerService:
        # Annotations are used to further tweak the LoadBalancer integration with the
        # cloud provider.
        annotations:
          load-balancer.hetzner.cloud/type: lb11
          load-balancer.hetzner.cloud/location: nbg1
          load-balancer.hetzner.cloud/use-private-ip: "false"

All load balancers have private and public ip.

0 replies

clickersmudge · 2022-12-01T15:14:04Z

clickersmudge
Dec 1, 2022
Author

@embik
If, for example, we download the kubeconfig cluster file.
And we will change the IP address in it to the IP address of the LoadBalancer.

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIRENDQWdTZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFmTVIwd0d3WURWUVFERXhSeWIyOTAKTFdOaExqRTVNaTR4TmpndU1DNHhNVEFlRncweU1qRXlNREV4TkRNMk5EZGFGdzB6TWpFeE1qZ3hORE0yTkRkYQpNQjh4SFRBYkJnTlZCQU1URkhKdmIzUXRZMkV1TVRreUxqRTJPQzR3TGpFeE1JSUJJakFOQmdrcWhraUc5dzBCCkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTV2NjlUWUZvMk9Dcy9lNTlleU96S0xULzJNYm9QZmNJWjhKUjA3SXcKbG5HNGNseHhza3NLYllVU09mUTVCM1lkTmwwT0VUOFkrM0hkWm5wbjFCdzZCWjgzQnk0Z0dMREFQdEFQbEhOKwpCcXYvTzdyT1IzMEMvSGNYZlQwZU8wOThDSUxSdDdneW9tbUo5bytMWEFUbGFXa3lHY0k2UnVNWVRXYVVycDk2CmZSa0lGaG1jK2NnQUNnWGdIK0UyTFJsM1VrUEdJVU13R1YzVjVRYW1KR1JYUzdZMlZ1SWhjUG9Ncm1MOTZxaWYKcWZ0T3h2SzNCcFRiZ0Q2V0VZYngyOXc4aEsrUWxmTS9KNUV5VHBjN25sSmNCVmtqYllNU2RhelVFenVRWGJrcAp2VDVrQ0N4cThSd3dHb3ZnOTN0QjJWdFdRZkdwTHJUWTJSWWI4WHN5YmlDeVlRSURBUUFCbzJNd1lUQU9CZ05WCkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVV5R2JYKyt0c2E3b1EKZFlQQzFvTTk5QlllVHpJd0h3WURWUjBSQkJnd0ZvSVVjbTl2ZEMxallTNHhPVEl1TVRZNExqQXVNVEV3RFFZSgpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFINXQyU1pVRlhFVGljeHVNU1B0TnZuUktxZkJBbDBYR2tUWi9FZEwvRWlGCi9yOVNGZ3VEVHp6MHE0cFVDOTh3eFBGRTVDanpiYmltT0czZlZoNUdTNE5GVTFWMGNMZ2ExUDlQSE55UFNZK3MKcHg3ZFlKemxlMW5jbU9rMVUydjJYeVVBeDVBNWtERVU1UWdhbUhUOCtzLzBHQ2FLbFRraExNUVgvMmdrbVVWaAoyVStxelJ3SHl3bVNoV0hCV0pIUHVZb2g3K0NpczBOKzQwSEtSTU53OWJiZ01aNlFHTlVybTZrV1FNTHV1YnFCCkxEUmtzSVVkSEV0YzdNVFFRREx4RVR4Y2g2c3IrWlhHTkN6dDI0NUhKVnB0cXZaa2k0ME1HSVo2cG1qZFExWDkKbm0va3lDWHZmbVdCYU5wZVJGZ1dOOU5GdkNnL1g4UlJoVC9iUndpNzF5dz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://167.235.105.153:32243
  name: q9q6phpcnp
contexts:
- context:
    cluster: q9q6phpcnp
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    token: 9vmd5z.qcghfgb4nkstg48t

kubectl get nodes
Unable to connect to the server: x509: certificate is valid for 10.240.16.1, 127.0.0.1, 192.168.0.11, not 167.235.105.153

0 replies

embik · 2022-12-01T15:27:52Z

embik
Dec 1, 2022
Collaborator

@clickersmudge I've checked with some colleagues and it appears that KKP cannot handle the situation in your setup properly for the "LoadBalancer" expose strategy. In short, KKP expects all IPs on a LoadBalancer Service to be "valid" and doesn't check if they're private or public. Since the Hetzner CCM publishes both public and private IPs, it's essentially bad luck that the private IPs take higher priority.

There isn't any workaround unless you convince the Hetzner CCM to not publish private IPs (which I'm not sure is possible, given that the option mentioned above was not effective). It's therefore possible that you cannot use this strategy in your environment right now. We'll however look into this and prioritise public IPs over private IPs in a future release.

1 reply

embik Dec 1, 2022
Collaborator

I've raised #11499 to track this.

xmudrii · 2022-12-01T15:55:59Z

xmudrii
Dec 1, 2022
Collaborator

Hey @clickersmudge,

We investigated this a little bit and found out that you can annotate your LoadBalancer Service with the following annotation to disable the private IP:

load-balancer.hetzner.cloud/disable-private-ingress: true

Similarly, you can use the following annotation to disable the IPv6 address if you don't need it:

load-balancer.hetzner.cloud/ipv6-disabled: true

Once annotations are applied, Hetzner CCM will automatically update the Load Balancer on Hetzner, usually after a few seconds.

1 reply

clickersmudge Dec 1, 2022
Author

@xmudrii, @embik Everything works as it should. Thank you for your time.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose Strategy problem #11495

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Expose Strategy problem #11495

clickersmudge Dec 1, 2022

exposeStrategy: LoadBalancer

Scenario 1

Scenario 2

Scenario 3

exposeStrategy: NodePort

Replies: 6 comments · 3 replies

embik Dec 1, 2022 Collaborator

clickersmudge Dec 1, 2022 Author

embik Dec 1, 2022 Collaborator

clickersmudge Dec 1, 2022 Author

clickersmudge Dec 1, 2022 Author

embik Dec 1, 2022 Collaborator

embik Dec 1, 2022 Collaborator

xmudrii Dec 1, 2022 Collaborator

clickersmudge Dec 1, 2022 Author

clickersmudge
Dec 1, 2022

Replies: 6 comments 3 replies

embik
Dec 1, 2022
Collaborator

clickersmudge
Dec 1, 2022
Author

embik Dec 1, 2022
Collaborator

clickersmudge
Dec 1, 2022
Author

clickersmudge
Dec 1, 2022
Author

embik
Dec 1, 2022
Collaborator

embik Dec 1, 2022
Collaborator

xmudrii
Dec 1, 2022
Collaborator

clickersmudge Dec 1, 2022
Author