-
Notifications
You must be signed in to change notification settings - Fork 802
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GRPC server graceful shutdown #1110
Comments
Have you tried this config: Line 111 in e52df51
|
Thanks |
Looks like it works. At least now I see that K8s can't kill it for configured period of time.
But after 5 minutes the app is getting killed anyway. I can't find a conf that is responsible for that.
Looks like it's Spring conf. I will try to experiment with it more |
You could add a log line/debug break point here: Line 154 in e52df51
To check if the waiting gets interrupted somehow. |
Thanks, I'm already looking into such a logic. It's not easy to debug everything with K8s. I will try to add more logs by DEBUG or use Telepresence or something to understand why the service is getting killed after 5 minutes. |
Depending on your setup debugging in K8s is easy. |
The connection is closed from the K8 side. When I run the server without K8s and send |
Thanks for the update |
When running a Java process inside a Docker container, sending a SIGTERM signal (kill -TERM 1) results in immediate termination rather than a graceful shutdown. This issue does not occur when running the same Java process locally and do the same kill.
Locally, it works fine but when server is inside Docker container graceful shutdown doesn't work and I can't understand why |
I'm running out of ideas. Do you have any ideas on further investigation or narrowing down the scope? |
Sorry, unfortunately not. |
I think |
Maybe also create an issue upstream in grpc-java and link it here. |
Yes, I expected it should already be handled downstream. Creating an issue on grpc-java is a good thing. |
I've created an issue: grpc/grpc-java#11229 |
The context
We deploy our service in K8s and provide a gRPC streaming API so the server can hold open connections for a period of time.
We need to have a CD to redeploy the new version of the service, but we want to prevent K8s from killing our service if there is an open GRPC stream.
The question
Do we have a support for graceful shutdown of the service only when we don't have open connections?
I see this: https://github.com/grpc-ecosystem/grpc-spring/blob/master/grpc-server-spring-boot-starter/src/main/java/net/devh/boot/grpc/server/serverfactory/GrpcServerLifecycle.java#L58
But I don't see we check the state of the service itself
The text was updated successfully, but these errors were encountered: