Adding Prometheus metrics to AMQP Operator

[lieberlois]

This PR introduces prometheus metrics to the operator. The supported metrics are the following prometheus metrics:

PROMETHEUS_HANDLER_CALLS_TOTAL_COUNTER = Counter(
    'handler_calls_total', 
    'Amount of times a handler has been called', 
    ["handler", "action"]
)

PROMETHEUS_HANDLER_EXCEPTION_COUNTER = Counter(
    'handler_exceptions_total', 
    'Amount of exceptions thrown in kopf handlers', 
    ["handler", "action"]
)

PROMETHEUS_RESOURCES_CREATED_TOTAL_GAUGE = Gauge(
    'resources_created_total', 
    'Amount of resources created using the operator', 
    ["type"]
)

While handler calls and exceptions are maintained manually using calls like

PROMETHEUS_HANDLER_CALLS_TOTAL_COUNTER.labels(handler=_HANDLER_NAME, action=_ACTION_NAME).inc()

the created resources are monitored using Kopfs In Memory Index feature. This enables us to monitor resources of any kind without having to query the Kube API on a regular basis.

Note: right now, it seems like the in-memory indices are only populated when a new resource is applied , but not when the operator starts up. This is unexpected behavior, since the docs suggest otherwise here. Feel free to contribute or comment if you know more about this.

The metrics that are scraped by prometheus look like this in practice:

[swoehrl-mw]

In regards to the kopf index: As discussed offline, using a (dummy) resume handler to set the count for number of resources should work as according to the docs the index is complete before any resource handlers are called.

Please also add a paragraph to the README and mention that the prometheus operator is needed for metrics to work (because of the ServiceMonitor CRD).

Note: For the changes that are the same in all handlers I only commented on the first one, but they apply to all.

[swoehrl-mw]

Any reason why this is done in two loops (collecting resources first, then couting them)?

[swoehrl-mw]

I'd prefer to take the name from the type object (k8s.AMQPBroker.kind), that way the name is only written in one place.

[swoehrl-mw]

As this is hardcoded in the python source, it makes no sense to parametrize it here.

[swoehrl-mw]

I prefer not to have a separate Service object and just add a second port to the main service.

[swoehrl-mw]

Should use the same labels as the deployment ({{- include "operator.labels" . | nindent 4 }}).

[swoehrl-mw]

Default should be false, to not interfere with the quickstart installation that does not require prometheus.

[swoehrl-mw]

Would it be possible to do this with a (custom) decorator function? Would be nicer to read

[swoehrl-mw]

Nitpick. Number of times sounds better than amount. Also for the other descriptions below.

Suggested change

	'Amount of times a handler has been called',
	'Number of times a handler has been called',

[swoehrl-mw]

Would be nicer as an enum.

[swoehrl-mw]

Shouldn't the kind be the key with namespace+name as the value?
Something like return {body.get("kind"): f"{namespace}:{name}"}