Are two connections per worker needed?

[ferchault]

We noticed that every worker creates two connections to the redis server. Does this have technical reasons? If we can reduce the total number of connections, this would improve scalability of rq with the same number of redis servers. We are happy to contribute code/time, since we run thousands of workers.

Steps to reproduce:

• Start a worker, ask redis for client list:

> CLIENT LIST
id=4615 addr=141.51.204.30:42454 laddr=141.51.205.42:42424 fd=10 name=a70e4d9125b944a08305123ccf781aed age=4 idle=4 flags=P db=0 sub=1 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1984 events=r cmd=subscribe user=default redir=-1 resp=2 lib-name=redis-py lib-ver=5.0.1
id=4616 addr=141.51.204.30:42456 laddr=141.51.205.42:42424 fd=11 name= age=4 idle=4 flags=b db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=87 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=2063 events=r cmd=blmove user=default redir=-1 resp=2 lib-name=redis-py lib-ver=5.0.1

I placed this at the head of redis-py's connection.AbstractConnection.__init__:

import traceback
traceback.print_stack()

which showed the when the connections were opened when running the command line worker from rq 1.15.1:

rq worker -c config  QUEUENAME
  File "PYENV/bin/rq", line 8, in <module>
    sys.exit(main())
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/rq/cli/helpers.py", line 422, in wrapper
    return ctx.invoke(func, cli_config, *args[1:], **kwargs)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/rq/cli/cli.py", line 259, in worker
    cleanup_ghosts(cli_config.connection)
  File "PYENV/lib/python3.11/site-packages/rq/contrib/legacy.py", line 19, in cleanup_ghosts
    for worker in Worker.all(connection=conn):
  File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 245, in all
    worker_keys = worker_registration.get_keys(queue=queue, connection=connection)
  File "PYENV/lib/python3.11/site-packages/rq/worker_registration.py", line 78, in get_keys
    return {as_text(key) for key in redis.smembers(redis_key)}
  File "PYENV/lib/python3.11/site-packages/redis/commands/core.py", line 3399, in smembers
    return self.execute_command("SMEMBERS", name)
  File "PYENV/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1084, in get_connection
    connection = self.make_connection()
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1124, in make_connection
    return self.connection_class(**self.connection_kwargs)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 584, in __init__
    super().__init__(**kwargs)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 168, in __init__
    traceback.print_stack()
10:08:04 Worker rq:worker:c1c2c26d2b434ec6b6ece70c0bf8160f started with PID 12784, version 1.15.1
10:08:04 Subscribing to channel rq:pubsub:c1c2c26d2b434ec6b6ece70c0bf8160f
  File "PYENV/bin/rq", line 8, in <module>
    sys.exit(main())
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/rq/cli/helpers.py", line 422, in wrapper
    return ctx.invoke(func, cli_config, *args[1:], **kwargs)
  File "PYENV/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "PYENV/lib/python3.11/site-packages/rq/cli/cli.py", line 301, in worker
    worker.work(
  File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 403, in work
    self.bootstrap(logging_level, date_format, log_format)
  File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 571, in bootstrap
    self.set_state(WorkerStatus.STARTED)
  File "PYENV/lib/python3.11/site-packages/rq/worker.py", line 861, in set_state
    connection.hset(self.key, 'state', state)
  File "PYENV/lib/python3.11/site-packages/redis/commands/core.py", line 5019, in hset
    return self.execute_command("HSET", name, *items)
  File "PYENV/lib/python3.11/site-packages/redis/client.py", line 533, in execute_command
    conn = self.connection or pool.get_connection(command_name, **options)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1084, in get_connection
    connection = self.make_connection()
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 1124, in make_connection
    return self.connection_class(**self.connection_kwargs)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 584, in __init__
    super().__init__(**kwargs)
  File "PYENV/lib/python3.11/site-packages/redis/connection.py", line 168, in __init__
    traceback.print_stack()
10:08:04 Worker rq:worker:c1c2c26d2b434ec6b6ece70c0bf8160f started with PID 12784, version 1.15.1
10:08:04 Subscribing to channel rq:pubsub:c1c2c26d2b434ec6b6ece70c0bf8160f```

Note that no job is running, so it is really just the bare worker waiting for work.

[selwin]

Thanks for raising this up, this is a very valuable insight.

I have never profiled the number of connections each worker makes but if I could guess the two connections come from the worker itself and a worker thread that opens a pubsub connection to Redis, listening to remote commands.

This is just a guess though, if you're willing to investigate further and want to have an option to reduce the number of open connections, I'd be happy to accept contributions in these areas. An option I can think of right away would be to disable the pubsub thread if it's not used.