Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

You cannot enable both slow start and weighted random algorithm on a target group #3686

Open
ibalat opened this issue May 9, 2024 · 3 comments

Comments

@ibalat
Copy link

ibalat commented May 9, 2024

Describe the bug
I am trying to update target group attributes but alb controller is logging below log. But, my attributes not include any slow start config. This error effect all target groups and it logging same error log for all target groups. It locked all alb

I use:
"alb.ingress.kubernetes.io/target-group-attributes" = "load_balancing.algorithm.type=weighted_random, load_balancing.algorithm.anomaly_mitigation=on, deregistration_delay.timeout_seconds=35"

I tried all combination like no attribute and slow_start.duration_seconds=0 but every time it logged same error

In UI:
image

{"level":"debug","ts":"2024-05-09T14:36:04Z","logger":"events","msg":"Failed deploy model due to InvalidConfigurationRequest: You cannot enable both slow start and weighted random algorithm on a  │
│ target group\n\tstatus code: 400, request id: xxx","type":"Warning","object":{"kind":"Ingress","namespace":"stage","name":"xxx","uid":"xxx","apiVersio │
│ n":"networking.k8s.io/v1","resourceVersion":"xxx"},"reason":"FailedDeployModel"}

Expected outcome
It have to update target group attributes

Environment

  • AWS Load Balancer controller version v2.7.2
  • Kubernetes version v1.29
  • Using EKS (yes/no), if so version? yes
@ibalat
Copy link
Author

ibalat commented May 10, 2024

we checked all ingress with kubectl get ingress -nprod -oyaml | grep -A 10 slow_start and just one ingress contains "slow_start" configuration (at the same time, this ingress set weigted algorithm on alb UI). This damaged ingress blocked all remain ingress updates. I think that this situation is a bug. I guess alb controller merge all configs and update bulky. If one ingress has a problem, all others logs same error message and update not completed. Can it be solved please

@aravindsagar
Copy link

Hi, target groups do not support slow start and weighted random algorithm at the same time. Also if any of the ingresses in an ingress group is broken, updates to the whole group will fail. This is by design to avoid further issues arising.

Can you let us know if the slow start or the load balancing algorithm attributes were changed in the UI or it got into this state in some other way?

@ibalat
Copy link
Author

ibalat commented May 22, 2024

thanks for info. Sure, example issue scenario is below:

  1. UI has weighted random algorithm and slow start is 0
  2. I set "alb.ingress.kubernetes.io/target-group-attributes" = "slow_start.duration_seconds=60, deregistration_delay.timeout_seconds=35" on ingress
  3. alb controller throw You cannot enable both slow start and weighted random algorithm on a target group
  4. I manually updated UI with round robin algorithm and slow start 60
  5. apply again same ingress attributes and alb controller works now (successfully reconciled)

My scenario explicitly show that there is a problem at switching algorithm and slow start values.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants