[Feature proposal] Consider switching to server side apply (SSA)

[grzesuav]

Description

Server side apply or SSA, was introduced in 1.16, reaching GA over time. It basically mimic kubectl apply behavior in more declarative way, managed by kubernetes API server. General benefit would be that metacontroller could drop its dynamic apply implementation. However we would need to make sure that they are equivalent. As far I can tell, the list/map handling is driven by markers on CRD schema so it would be actually better, as now metacontroller implementations tries to "guess" when to merge lists and in SSA user declares it in CRD.

TODO before implementation

1. check how use SSA from client-go / controller-runtime
2. verify compatibility of SSA versus metacontroller dynamic apply
3. document and check requirements for handling list merges
4. Check how it works with builtin types (i.e. does it considers all the lists from metacontroller "guessing" list also as candidates for merging)

Challenges

1. Explaining users how they should write CRD schema in order to have predictable list merges.
2. Conflicts handling etc

Related issues

• [Bug] Mounting configMap to multiple paths detects diff. #650
• StatefulSets with volumeClaimTemplates trigger constant updates #566
• environment var array order not being respected #562
• spec.containers.resources.requests.cpu value sometimes gets transformed leading to endless recreate loop #444