addon: Add support for logging agent health checking

[btaani]

With this PR, the addon agent will perform health checks on the spoke-cluster's ClusterLogForwarder and OTEL collector instances.

For the OTEL collector, if the .spec.replicas field is 0, it will report AVAILABLE = False to the addon:

$ oc get managedclusteraddons.addon.open-cluster-management.io -n spoke-cluster multicluster-observability-addon 
NAME                               AVAILABLE   DEGRADED   PROGRESSING
multicluster-observability-addon   False                  

For the ClusterLogForwarder resource, the health is based on the status fields .status.conditions.type and .status.conditions.status

[coveralls]

Pull Request Test Coverage Report for Build 9196016992

Details

• 53 of 67 (79.1%) changed or added relevant lines in 1 file are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+1.3%) to 51.888%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
internal/addon/addon.go	53	67	79.1%

Totals
Change from base Build 8877284128:	1.3%
Covered Lines:	371
Relevant Lines:	715

---

💛 - Coveralls

[JoaoBraveCoding]

General comment, currently MCOA supports having multiple signals enabled or disabled so in the ideal scenario the health prober should consider that all signals might not always be enabled (if possible).

[iblancasa]

We need to get this information from the ManagedClusterAddOn object. The same for Logging

[JoaoBraveCoding]

Good job! Looks promising 👀

[JoaoBraveCoding]

I feel this way it's easier to understand the code path

Suggested change

	if *value.Value.String == "Ready" {
	return nil
	}
	return fmt.Errorf("%w: status condition type is %s for %s/%s", ErrWrongType, *value.Value.String, identifier.Namespace, identifier.Name)
	if *value.Value.String != "Ready" {
	return fmt.Errorf("%w: status condition type is %s for %s/%s", ErrWrongType, *value.Value.String, identifier.Namespace, identifier.Name)
	}
	return nil

[JoaoBraveCoding]

Same here

Suggested change

	if *value.Value.Integer >= 1 {
	return nil
	}
	return fmt.Errorf("%w: replicas is %d for %s/%s", ErrWrongType, *value.Value.Integer, identifier.Namespace, identifier.Name)
	if *value.Value.Integer < 1 {
	return fmt.Errorf("%w: replicas is %d for %s/%s", ErrWrongType, *value.Value.Integer, identifier.Namespace, identifier.Name)
	}
	return nil

[JoaoBraveCoding]

I think I would change the if... else if... else to a switch statement, that might make it more easy read