Issue running thin standalone server with redis

[njt1982]

I have spent the day debugging a very strange issue.

Locally, the Thin server runs fine against Redis 3.2.5 using all default settings.

On our testing server, I have tried Redis 2.8.x (OS repo version) and 3.2.5 (via docker); they both exhibit the same issue.

If I start the server with default settings, it seems to use the synchrony driver. No matter what I set the timeout to, the connection times out with this:

vendor/bundle/ruby/2.1.0/gems/redis-3.3.2/lib/redis/connection/synchrony.rb:124:in `read': Connection timed out (Redis::TimeoutError)

Redis is running fine (default port). I can telnet to it and MONITOR or KEYS and it responds instantly. The services are all running locally, so firewall should not be a problem here.

I have tried changing the driver to hiredis and ruby, both of these solve the timeout problem but replace it with one that doesn't even error. Thin starts but then freezes; WS & WSS connections just time out (Chrome reports "pending", whereas locally I immediately get a 101 response).

My colleague has tried my code on his laptop and it runs fine. Clearly, this is an environment issue, but I cannot figure out what, specifically, the problem is.

Any tips?

[njt1982]

In the read method in synchrony, I did an inspect on @connection...

#<Redis::Connection::RedisClient:0x0000000f16fd28 @signature=10, @req=#<EventMachine::DefaultDeferrable:0x0000000f16f288 @deferred_timeout=nil, @deferred_status=:succeeded, @callbacks=[], @errbacks=[#<Proc:0x0000000f16f238@/home/deploy/workspace/OBFUSCATED/vendor/bundle/ruby/2.1.0/gems/em-synchrony-1.0.5/lib/em-synchrony.rb:65>], @deferred_args=[[:reply, ["subscribe", "websocket_rails.events", 1]]]>, @connected=true, @reader=#<Hiredis::Ext::Reader:0x0000000f16fc88>, @deferred_status=:succeeded, @callbacks=[], @errbacks=[], @deferred_timeout=nil, @deferred_args=[], @timeout=0>

Interesting it mentioned Hiredis...

[njt1982]

Debugging the client options... interesting it switches to the Ruby driver just before timing out?!

>> Writing PID to /home/deploy/workspace/OBFUSCATED/tmp/pids/websocket_rails.pid
>> Thin web server (v1.5.1 codename Straight Razor)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:30793, CTRL+C to stop
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Ruby, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
>> Exiting!
/home/deploy/workspace/OBFUSCATED/vendor/bundle/ruby/2.1.0/gems/redis-3.3.2/lib/redis/connection/synchrony.rb:124:in `read': Connection timed out (Redis::TimeoutError)

[njt1982]

Plot thickens... not only is that last connection using Ruby, it looks like its switched to using a socket?!

{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Synchrony:0x0000000f410988 @connection=#<Redis::Connection::RedisClient:0x0000000f410c80 @signature=10, @req=nil, @connected=true, @reader=#<Hiredis::Ext::Reader:0x0000000f410be0>, @deferred_status=:succeeded, @callbacks=[], @errbacks=[#<Proc:0x0000000f410af0@/home/deploy/workspace/OBFUSCATED/vendor/bundle/ruby/2.1.0/gems/redis-3.3.2/lib/redis/connection/synchrony.rb:86>], @deferred_timeout=nil, @deferred_args=[], @timeout=0>>
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Synchrony, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Synchrony:0x0000000f410988 @connection=#<Redis::Connection::RedisClient:0x0000000f410c80 @signature=10, @req=#<EventMachine::DefaultDeferrable:0x0000000f45b6e0 @deferred_timeout=nil, @deferred_status=:succeeded, @callbacks=[], @errbacks=[#<Proc:0x0000000f45b690@/home/deploy/workspace/OBFUSCATED/vendor/bundle/ruby/2.1.0/gems/em-synchrony-1.0.5/lib/em-synchrony.rb:65>], @deferred_args=[[:reply, ["subscribe", "websocket_rails.events", 1]]]>, @connected=true, @reader=#<Hiredis::Ext::Reader:0x0000000f410be0>, @deferred_status=:succeeded, @callbacks=[], @errbacks=[], @deferred_timeout=nil, @deferred_args=[], @timeout=0>>
{:host=>"127.0.0.1", :port=>6379, :driver=>Redis::Connection::Ruby, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Ruby:0x0000000f458e90 @sock=#<Redis::Connection::TCPSocket:fd 26>>

[njt1982]

And another dump when using hiredis and localhost:

>> Writing PID to /home/deploy/workspace/OBFUSCATED/tmp/pids/websocket_rails.pid
>> Thin web server (v1.5.1 codename Straight Razor)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:30793, CTRL+C to stop
{:host=>"localhost", :port=>6379, :driver=>Redis::Connection::Hiredis, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Hiredis:0x0000000e44dbe0 @connection=#<Hiredis::Ext::Connection:0x0000000e44dc58>>
{:host=>"localhost", :port=>6379, :driver=>Redis::Connection::Hiredis, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Hiredis:0x0000000e44dbe0 @connection=#<Hiredis::Ext::Connection:0x0000000e44dc58>>
{:host=>"localhost", :port=>6379, :driver=>Redis::Connection::Hiredis, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Hiredis:0x0000000e4560b0 @connection=#<Hiredis::Ext::Connection:0x0000000e456128>>
{:host=>"localhost", :port=>6379, :driver=>Redis::Connection::Hiredis, :url=>nil, :scheme=>"redis", :path=>nil, :timeout=>5.0, :password=>nil, :db=>0, :id=>nil, :tcp_keepalive=>0, :reconnect_attempts=>1, :inherit_socket=>false, :connect_timeout=>5.0, :read_timeout=>5.0, :write_timeout=>5.0, :_parsed=>true}
#<Redis::Connection::Hiredis:0x0000000e4560b0 @connection=#<Hiredis::Ext::Connection:0x0000000e456128>>

[njt1982]

So it seems to be an issue with Rails Eager Loading.

It worked on mine and my colleagues local installs because config/environments/development.rb has config.eager_load = false - however our testing server has config.eager_load = true...

[donpapa26]

Hi

try to increase the timeout (default 5.0 ) in the config\initializers\websocket_rails.rb

config.redis_options = {:host => '127.0.0.1', :port => '6379', :timeout => 25.0}

[nninkking]

Hi, I am gonna use websocket-rails gem in rails project deployed on heroku.
Following is from config/initializers/websocket_rails.rb

config.standalone = true
config.synchronize = true
config.redis_options = { :host => ENV['REDISTOGO_HOST'], :port => ENV['REDISTOGO_PORT'], :user => ENV['REDISTOGO_USER'], :password => ENV['REDISTOGO_PASSWORD'] }

As you can see, I am gonna enable standalone server mode.
https://github.com/websocket-rails/websocket-rails/wiki/Standalone-Server-Mode
Following is the Procfile.
web: bundle exec passenger start -p $PORT -e ${RACK_ENV:-development} --max-pool-size 1 --min-instances 1 --nginx-config-template config/nginx.conf.erb
Heroku doesn't support multiple web process type, so I couldn't figure out where to put following commands.
rake websocket_rails:start_server
Please let me know how I can enable standalone mode on heroku.
Thank you in advance.

[MhdSyrwan]

This might help issue#313
According to the link, using this fork gem 'websocket-rails', github: 'moaa/websocket-rails', branch: 'sync_fixes' would solve the issue.

I've tried that, it works !