[Feature] check the synchronization status of nodes and avoid those which are still in data synchronization

[Alex-Cheng]

Is your feature request related to a problem? Please describe.
I want to build-up a cluster with several nodes for load-balancing, however I need to frequently update database on one node and spread muations to others. I need to let chproxy avoide those nodes which are in progress of data synchronization.

Describe the solution you'd like
When data sync is running for a ReplicatedMergeTree tables, chproxy could check the data sync status of each node to make sure that only fully synched latest updates are in the service.

[gontarzpawel]

Do you have any idea about how this feature could be implemented? @Alex-Cheng

We're familiar with the scenario of out of sync replica (e.g. node outage). In which case, we apply following approach:

• disable clickhouse server from the read path (remove https port - for chproxy access, and tcp port - for intersever communication)
• restart server with updated config

This way clickhouse node will be considered as unreachable, unhealthy by chproxy and queries will omit it. When lag is caught up, we restart that server with fixed configuration.

However, it makes sense only when it's not a common behaviour but rather sporadical.

[Alex-Cheng]

"disabling clickhouse server from the read path and restarting server" helps, but it need manual operations, not automatical. I need an automatical way. My idea is to check the /replicas_status which implemented by ReplicasStatusHandler, and skip the node which /replicas_status returns false.