Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Picking up first running pod instead of first unknown status pod from list #199

Merged
merged 2 commits into from Feb 4, 2024

Conversation

rvarunrathod
Copy link
Contributor

It is sort of an edge case.
This will only happen when you have multiple instances of the same service.

  1. When you have low resources, one pod is in a running state, and the other is in a pending state. It tries to port forward to the pending state pod.
    e.g.
NAME        READY   STATUS    
myapp-adfa   1/1     Running  
myapp-rasf   0/1     Pending    <- connection to this
  1. I had a service with replications. When I was doing a rolling upgrade with new deployment, I had set wrong secret that pod could not find and got stuck in the ContainerCreating state. So there were older pods in a running state instead of connecting to them, it was trying to connect the stuck pod.
    e.g.
NAME        READY   STATUS    
myapp-adfa   1/1     Running  
myapp-rasf   0/1     ContainerCreating    <- connection to this
error upgrading connection: unable to upgrade connection: pod not found ("myapp_dev")
馃憮  Forwarder: lost port-forward connection trying to reconnect...
error upgrading connection: unable to upgrade connection: pod not found ("myapp_dev")
馃憮  Forwarder: lost port-forward connection trying to reconnect...

Copy link
Owner

@eko eko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this fix @rvarunrathod.

I think a good approach for later would be to allow using a port-forward on a service (rather than on a pod directly). This will avoid selecting non-ready pods.

@eko eko merged commit 42e987d into eko:master Feb 4, 2024
1 check failed
@rvarunrathod
Copy link
Contributor Author

rvarunrathod commented Feb 5, 2024

Hi @eko, Yes, port-forwarding to service is a good approach as it will make it fault-tolerant and increase availability, and we don't have to worry about which pod to pick. I will happily help you with implementation if you want me to.

@eko
Copy link
Owner

eko commented Feb 5, 2024

@rvarunrathod I do not have so much time for now so if you want to work on this, you're really welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants