-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
org.redisson.client.WriteRedisConnectionException: Unable to write command into connection! After restarting redis cluster. #5857
Comments
I tried version 3.27.2, but it didn't work either. |
Odd, our cases seem to be a bit different. The logs do complain about CPU, but the actual CPU used by the app is still the same. |
Does this error appear for CLUSTER INFO command only? Is 100.96.94.107:6379 offline? |
Can anyone try the attached version? |
I can't really comment on other commands, because when the instance cannot pass the above health check, it stops accepting other requests. As for 100.96.94.107:6379, in k8s platform rebooting an instance starts it up again with a different IP. Other redis nodes update their own ip for the node, but post 3.27 Redisson seems to not update the IP and ends up trying to ping a node that is no longer even present. I'll try to test with the version above. |
One thing to mention, we have Redis configured in a way that before termination, if the instance is a master node My first test with 3.30.1-SNAPSHOT has been with this enabled Second test is once again with this configuration turned off |
Thanks much for testing. Please try the new build below. |
@mrniko First test with the role switching before shutdown disabled Second with the special failover enabled |
Please try the new build. I added more info output regarding cluster nodes which reported exception. |
Spring Boot version: 3.2.5
Java version: 21 (also tested on 17) NOT using virtual threads
Redis version: 7.2.4
Redission version: 3.30.0 (tested on 3.29.0, 3.28.0, 3.27.2)
Redisson configuration:
We run a 6 node redis cluster in kubernetes. When performing a rolling restart of all nodes one by one, after restarting the final node, redisson fails to properly reconnect for the case of Redis health check in Spring Boot.
Expected behavior
Rollout finishes and redisson continues communicating with newly started nodes.
Actual behavior
After rollout finishes, redisson starts spamming
WriteRedisConnectionException
andCancellationException
until the application instance is restarted.Truncated log: app.log
Steps to reproduce or test case
Spring boot application using Redisson with enabled redis health check on application readiness probe.
Perform a rollout of all 6 redis cluster nodes.
As far as I can tell, this behavior started between version 3.27.2 and 3.28.0.
3.27.2 is OK, any version afterwards is NOK.
The text was updated successfully, but these errors were encountered: