Traffic stability during rolling upgrades

[tnozicka]

This is a tracker for our epic to make sure rolling restarts are graceful and no connection gets handled ungracefully.

Unfortunately, at this point we use services that publish unready endpoints and the traffic gets errors, or gets stuck in contract tables. There are also graceful termination fixes needed for handling ScyllaDB shutdown.

Tasks

Beta Give feedback

1. Forward ClusterIP traffic to inter-node communication port before CQL is ready #1614
   approved kind/bug needs-rebase priority/important-soon size/XXL +5
2. Loss of availability below required consistency level during node shutdown scylladb#15899
   P2 area/correctness bug status/regression +4
   
3. During rollout restart operator doesn't wait until previously restarted pod becomes part of a Scylla cluster #1077
   kind/bug priority/important-soon +2