Incorrect update of the redis-info to consul

[MatthiasWiesner]

It seems that the redis-info is incorrect updated to consul. The redis info, section 'Replication', on the redis-master server shows two connected slaves:

# Replication
role:master
connected_slaves:2
slave0:ip=10.224.199.37,port=6381,state=online,offset=942807,lag=2
slave1:ip=10.224.199.11,port=6381,state=online,offset=942807,lag=1
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6

While the the health check of the same redis-master server in consul shows:

# Replication
role:master
connected_slaves:0
master_replid:63e4319d7e22a674e2192e6d1d3fc90e548253fb
master_replid2:5b5e7340f5dd557915f84b8f8b5dfa9d877ca1f6

Resec was started with these environment vars:

REDIS_ADDR="10.224.199.21:6381"
CONSUL_SERVICE_PREFIX="redis-cache"
CONSUL_LOCK_SESSION_NAME="resec/cache"
CONSUL_LOCK_KEY="resec/cache.lock"

Edit: It seems that pretty much all stats information are affected as well.

Is this intentional, did I miss something or is this a bug?

[RedStalker]

Hi @MatthiasWiesner!
I'm also faced with such a problem during testing version 0.6. I suppose this is happening because of Consul service logic. When ReSec chooses Master it will immediately register it as a service with Redis info output. At that moment Slave servers are still not enabled for replication via ReSec and Master server doesn't know anything about them. After Master being elected Slaves also registering in Consul this time with proper output. But that doesn't trigger Master node to update its info in Consul service.
We might possibly ask devs to update service when Redis info output changes

[burdandrei]

@RedStalker actually it's not a resec bug, but consul config:
https://www.consul.io/docs/agent/options.html#check_update_interval
so by default what you see in UI is updated every 5 minutes, but you can configure it to whatever time you want!

[RedStalker]

Hello @burdandrei!
Thank you for your answer! I've created a demo using a docker-compose file in the repo and I was unable to fix the issue from the Consul configuration side. After waining more than 5 minutes, Service check output still showing
connected_slaves:0

Modifing Consul configuration with additional config also doesn't help
{ "check_update_interval": "0s" }
Maybe I'm missing something?

[jippi]

This has been fixed in the 0.6 branch I'm working on over in #30

[RedStalker]

Hello @jippi !
I'm using source code from branch https://github.com/seatgeek/resec/commits/resec-0.6 and after manual master failover new Master node shows service output when it used to be Slave

Resec: master replication status
ServiceName - redis-master
CheckID redis@redis-resec-3:6379:replication-status-check
Output

Server

redis_version:5.0.6

Replication

role:slave
master_host:redis-resec-1
master_port:6379
master_link_status:down

[jippi]

@RedStalker not sure I fully understand your answer, did it fix it for you, or what is your expectation ? :)

[RedStalker]

@jippi I mean that I'm using code with the latest commits and Service check output in Consul for Master still doesn't show proper information from Redis info.

Steps that I'm doing

1. Turn on Consul cluster with 3 nodes
2. Start 3 Redis containers and 3 ReSec containers
3. Master is being elected in the Consul services
4. Slaves are being configured as service in Consul
5. Master Service output in Consul shows connected_slaves:0 but the replication is working and Redis-cli inside container shows connected_slaves:2
6. After manually stopping the Master container Slave is being promoted to master and Service output for new Master shows

role:slave
master_host:redis-resec-1
master_port:6379
master_link_status:down

It is expected behavior? My expectation is that should the Service update it's output when something changes in Redis replication output?