Flagger should support custom annotations on VirtualService/DestinationRule resources

[rolandkool]

Describe the feature

We have a requirement to add custom annotations to VirtualService and DestinationRule resources. Our use case is that we add an annotation to indicate the resource needs to be synced to other clusters as we run Istio in multi-primary mode.

If we add a custom annotation through a mutating webhook, Flagger ends up in a loop where it cannot progress because the mutation triggers a reconcile and the VirtualService is reset to a weight of 100/0.

Proposed solution

We'd like to have a possibility for Flagger to either ignore custom annotations (through a parameter that can be specified when start flagger) or a way to specify them through the canary custom resource, similar as to what can be done for the Service to support External DNS annotations.

We don't have a preference for either solution.