start_heartbeat can die while worker is still running

[arianf]

Our Redis instances has been having some connectivity issues (which we are resolving as well), however I noticed something interesting in our Resque logs. It's possible for the heartbeat to die, while the worker remains running.

In our Resque logs I see this for the worker that is still running but no longer listed within Resque.workers

#<Thread:0x00007f974e3b8ef8 /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:527 run> terminated with exception (report_on_exception is true):
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (Redis::CannotConnectError)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:48:in `rescue in initialize': Failed to connect to helm-redis-primary:6379 (ECONNREFUSED) (RedisClient::CannotConnectError)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:44:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'
/usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `connect_tcp': Connection refused (Errno::ECONNREFUSED)
	from /usr/local/bundle/ruby/3.2.0/gems/hiredis-client-0.12.0/lib/redis_client/hiredis_connection.rb:45:in `initialize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `new'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:664:in `block in connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client/middlewares.rb:12:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:663:in `connect'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:657:in `raw_connection'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:624:in `ensure_connected'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-client-0.12.0/lib/redis_client.rb:223:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/client.rb:73:in `call_v'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:167:in `block in send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `synchronize'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis.rb:166:in `send_command'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-5.0.6/lib/redis/commands/server.rb:178:in `time'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:479:in `call_with_namespace'
	from /usr/local/bundle/ruby/3.2.0/gems/redis-namespace-1.9.0/lib/redis/namespace.rb:354:in `block (2 levels) in <class:Namespace>'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/data_store.rb:96:in `server_time'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:492:in `heartbeat!'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:529:in `block (2 levels) in start_heartbeat'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `loop'
	from /usr/local/bundle/ruby/3.2.0/gems/resque-2.4.0/lib/resque/worker.rb:528:in `block in start_heartbeat'

If this thread dies, there is nothing that will restart it?

resque/lib/resque/worker.rb

Lines 527 to 533 in 2f9d080

	@heartbeat_thread = Thread.new do
	loop do
	heartbeat!
	signaled = @heartbeat_thread_signal.wait_for_signal(Resque.heartbeat_interval)
	break if signaled
	end
	end

Maybe each time it runs a new job it should check if the heartbeat thread is still alive, and if not start a new heartbeat.

[ChrisCPO]

Is there any way this is related to? #1895