Behaviour when one of the pods remains unready after a failure

[christiaangoossens]

While having CNPG in production for a while, I have noticed the following behaviour:

1. Let's say we create a cluster with 3 nodes. test-cluster01 is now the primary.
2. Some failure occurs, let's say that node 1 failed, thus test-cluster01 is now offline. test-cluster02 and test-cluster03 are fine.
3. It takes a little bit of time for the cluster to realise and it correctly fails over to test-cluster02.

-- Some time passes --

4. Node 1 comes back online, maybe it restarted, maybe it did not. Anyway, test-cluster01 is behind the others in the cluster.
5. For whatever reason, there is some postgres error in test-cluster01, maybe it has disk corruption, maybe it is just behind and put itself out of the cluster. Postgres errors popup in the logs.
6. test-cluster01 will always remain unready because of Postgres not starting due to errors.

Now the question comes: When does CNPG just give up? For me it just seems to keep trying on cluster01, instead of just deleting the PVC and pod after a while and recreating it. Even after a week, I will still have test-cluster01 on unready, with two healthy pods (02 and 03).

For now, the only solution is deleting the PVC of 01 manually and deleting the pod after, upon which CNPG correctly creates 04 to replace it.

What do we think this should do? Should it auto-recover and make all instances healthy or is there a reason why it does not?