Add graceful shutdown #10701
Add graceful shutdown #10701
Conversation
This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated. The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed. There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new `micronaut.server.netty.listeners.*.support-graceful-shutdown` config option to disable graceful shutdown for a management port. Graceful shutdown is triggered by application code through a new GracefulShutdownManager API. Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed. Docs are still pending, I want Graeme to review the API before I write it up.
|
| * @author Jonas Konrad | ||
| * @since 4.5.0 | ||
| */ | ||
| public interface GracefulShutdownLifecycle { |
There was a problem hiding this comment.
not sure lifecycle is the right name here, since this is only graceful shutdown and not startup. Rename to GracefulShutdownCapable
context/src/main/java/io/micronaut/runtime/server/GracefulShutdownLifecycle.java
Outdated
Show resolved
Hide resolved
| * @param stages The input lifecycles | ||
| * @return A future that completes when all inputs have completed shutdown | ||
| */ | ||
| static CompletionStage<?> shutdownAll(Stream<? extends GracefulShutdownLifecycle> stages) { |
There was a problem hiding this comment.
what about timeouts? The idea is a window where graceful shutdown is allowed but then it still has to shutdown. I would expect some duration parameter here
There was a problem hiding this comment.
The idea is that the caller adds a timeout, and after the timeout calls normal Lifecycle.stop.
context/src/main/java/io/micronaut/runtime/server/GracefulShutdownManager.java
Show resolved
Hide resolved
...src/main/java/io/micronaut/http/server/netty/configuration/NettyHttpServerConfiguration.java
Show resolved
Hide resolved
|
|
||
| @Override | ||
| public Publisher<HealthResult> getResult() { | ||
| return Mono.just(HealthResult.builder("gracefulShutdown") |
There was a problem hiding this comment.
move to constant rename to graceful-shutdown
There was a problem hiding this comment.
Other health indicators are camel case, eg diskSpace and compositeDiscoveryClient()
| * down | ||
| */ | ||
| public CompletionStage<?> shutdownGracefully() { | ||
| return GracefulShutdownLifecycle.shutdownAll(delegates.stream()); |
There was a problem hiding this comment.
I think we need better error handling here. What happens when there are multiple implementations and one of them fails to shutdown? How do we know and log which one failed?
Say other modules like the JMS, Kafka etc. modules implement this interface I would want to know which one failed and why.
There was a problem hiding this comment.
The interface is not really designed to fail. If an implementation cannot gracefully shut down for whatever reason, it should just signal completion and Lifecycle.stop will shut it down hard.
| @Override | ||
| public Publisher<HealthResult> getResult() { | ||
| return Mono.just(HealthResult.builder("gracefulShutdown") | ||
| .status(shuttingDown ? HealthStatus.DOWN : HealthStatus.UP) |
There was a problem hiding this comment.
it would be nice to return some detail showing the state of what GracefulShutdownLifecycle is currently shutting down and hence we can know what is blocking the shutdown process
There was a problem hiding this comment.
ive added a shutdown state mechanism
| * @return A future that completes when all {@link GracefulShutdownLifecycle} beans have shut | ||
| * down | ||
| */ | ||
| public CompletionStage<?> shutdownGracefully() { |
There was a problem hiding this comment.
what calls GracefulShutdownManager.shutdownGracefully()?
There was a problem hiding this comment.
In this PR nothing. could make it a management endpoint, or just leave it to application code.
There was a problem hiding this comment.
there is already a shutdown endpoint so if enabled it should be called from that, but also we would likely need to integrate the shutdown hooks we add already to handle SIGTERM
We need to be able to specify a termination grace period as well, in Spring this spring.lifecycle.timeout-per-shutdown-phase=20s
|
Perhaps we should try and address #6493 in this PR as well |
This PR adds a new interface, GracefulShutdownLifecycle, that parts of the framework can implement to provide graceful shutdown functionality. It is designed to complement LifeCycle.stop(), though the implementations are unrelated.
The netty HTTP server, on graceful shutdown, will stop accepting new connections, and wind down existing ones. HTTP/1 connections are closed as soon as in-progress requests have been served, while HTTP/2 and /3 connections send a GOAWAY packet to indicate to the client that no further requests will be processed.
There is also a readiness health indicator that turns DOWN when a graceful shutdown begins. To keep the health endpoint accessible, there is a new
micronaut.server.netty.listeners.*.support-graceful-shutdownconfig option to disable graceful shutdown for a management port.Graceful shutdown is triggered by application code through a new GracefulShutdownManager API.
Graceful shutdown is best-effort. In-progress requests may take too long, or HTTP/2 clients may refuse to shut down their connection. For this reason, GracefulShutdownLifecycle returns a CompletionStage. The stage will complete when a graceful shutdown has been achieved, but this may be never, so a user should always add a timeout. After a graceful shutdown (or timeout), a normal LifeCycle.stop should be performed.
Docs are still pending, I want Graeme to review the API before I write it up.