You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Our Redis instances has been having some connectivity issues (which we are resolving as well), however I noticed something interesting in our Resque logs. It's possible for the heartbeat to die, while the worker remains running.
In our Resque logs I see this for the worker that is still running but no longer listed within Resque.workers
#<Thread:0x00007f974e3b8ef8 /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:527 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (Redis::CannotConnectError)
from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (RedisClient::CannotConnectError)
from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `connect_tcp': Connection refused (Errno::ECONNREFUSED)
from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `initialize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
If this thread dies, there is nothing that will restart it?
Our Redis instances has been having some connectivity issues (which we are resolving as well), however I noticed something interesting in our Resque logs. It's possible for the heartbeat to die, while the worker remains running.
In our Resque logs I see this for the worker that is still running but no longer listed within
Resque.workers
If this thread dies, there is nothing that will restart it?
resque/lib/resque/worker.rb
Lines 527 to 533 in 2f9d080
Maybe each time it runs a new job it should check if the heartbeat thread is still alive, and if not start a new heartbeat.
The text was updated successfully, but these errors were encountered: