Polyaxon Operator HA

[boniek83]

Use case

After network outage of node where polyaxon-operator was working, polyaxon-operator did not came back working correctly. It was stuck in leader election, which made it stop working at all.
E1206 17:17:24.588551 1 leaderelection.go:357] Failed to update lock: Put "https://10.40.0.1:443/apis/coordination.k8s.io/v1/namespaces/polyaxon-websensa/leases/ops.core.polyaxon.com": context deadline exceeded I1206 17:18:42.637003 1 leaderelection.go:278] failed to renew lease polyaxon-websensa/ops.core.polyaxon.com: timed out waiting for the condition

Feature description

Polyaxon-operator should be highly available.

Polyaxon CE 1.12.3
Kubernetes 1.19.8

[polyaxon-team]

You can try Polyaxon v1.13.0, the leader election is not enabled by default, we decided to go with a simpler concurrency model than a replication model for the operator.

It will be hard to reproduce this error again, so we will keep this issue as a reference.