Drift detection and Continuous Reconciliation

[kutsyk]

I'm trying to figure out how Karmada is solving problems of drift detection and continuous reconciliation.

It is not clear for me how to do next things:

1. How to check if all propagation policies and override policies has been rolled out and applied correctly?
2. How to detect if one of the managed cluster failed to apply propagation/override policy?
3. What metrics to use to monitor is all propagated components work in all managed clusters?

I believe these 3 questions are not something new and most of people who are using orchestration tool should have already answered them, but I can't get my head around this with Karmada.

Thanks ahead for the help

[XiShanYongYe-Chang]

How to check if all propagation policies and override policies has been rolled out and applied correctly?

PP and OP need to be separated. For PP, it is available to view the FullyApplied condition status of ResourceBinding. For OP, it is necessary to actively check whether the differentiated configuration takes effect in the Work resource. In general, the current solution is still defective for automated checks, and status checks cannot be performed directly from the API.

How to detect if one of the managed cluster failed to apply propagation/override policy?

For PP and OP, the relationship between them and resources is only matched or not matched. When the matching is successful, they will be reflected in ResourceBinding and Work resources respectively.

Then, the resource template in the work is synchronized to the member cluster. You can view the synchronization result in the work status.

What metrics to use to monitor is all propagated components work in all managed clusters?

You can check the value of the health field in Work status:

karmada/pkg/apis/work/v1alpha1/work_types.go

Line 107 in 5e1191f

Health ResourceHealth `json:"health,omitempty"`

User can use the custom interpreter InterpretHealth to define this value.

[RainbowMango]

Just out of curiosity, are you evaluating Karmada? Do you have a schedule or something? We can see how to support you better.

[kutsyk]

Hi,
@XiShanYongYe-Chang , thanks for the clarification.

@RainbowMango , yes, I'm evaluating the tool around set of points and to better understand how it works and if we should use it. We are at the end of our schedule and the last points I have to understand are those described in my initial question.

Seems there is a way to monitor statuses, but only through objects and values in their fields. There is no data exposed as metrics, do I understand this correctly?

[RainbowMango]

We have some metrics exposed by /metrics endpoint, but not include the items you want.
I would say that metrics can be added at any time when there is a need.

[kutsyk]

@RainbowMango , @XiShanYongYe-Chang , do you have idea on how to identify if OverridePolicy failed and how to find reasons for it?

Also, what will happen if override policy that should be applied to 5 cluster fails on 2, how can I identify those clusters and reason for error?

Thanks

[XiShanYongYe-Chang]

Hi @kutsyk, oo you mean the op apply failure is caused by the op write problem or the failure of synchronization to the member cluster?

[kutsyk]

Hi, The failure of synchronisation to the member cluster Kutsyk Vasyl

…

On Wed, 8 May 2024 at 12:21, Chang ***@***.***> wrote: Hi @kutsyk <https://github.com/kutsyk>, oo you mean the op apply failure is caused by the op write problem or the failure of synchronization to the member cluster? — Reply to this email directly, view it on GitHub <#4895 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSIWHTGF6LMVDYIX23HZVLZBH4ELAVCNFSM6AAAAABHASLMACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBQGI2TCOJSGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[XiShanYongYe-Chang]

In my opinion, if the resource synchronization fails, it is not easy to determine whether the failure is caused by the OP. In other words, the two are decoupled. The OP is used for the work resource, and the resource synchronization failure is a further operation on the work resource. Can you give an example of a failure caused by the OP?