Sloth Multicluster

[Etta9]

Hi there,
I have two different k8s clusters:

• SRE k8s cluster, in which I have deployed tools like Grafana, Prometheus
• App k8s cluster, in which I have deployed my application.

I want to use Sloth in order to define some SLOs for my app and I want to use Grafana to monitor them. Note that my app has been deployed with its prometheus server.

I would like to deploy Sloth in my SRE k8s cluster and not in the App k8s cluster, is it possible? If so, how can I tell sloth to use an external prometheus server ( such as my app's prometheus server)? If not, what approach I need to use for my case?

Thank you

[eddiebarth1]

I also am looking for information around this.
In the old service-level-operator, we were able to do it by defining the --def-sli-source-path and having a configmap with our Thanos querier address. So far I'm not yet seeing an equivalent for Sloth. Does that exist @slok ?

[mike-brown-angi]

We got over on this by using sloth to generate the PrometheusRules and then loaded them into our ruler of choice (mimir).

[jleloup]

In addition to @mike-brown-angi answer: you could use Grafana Agent mimir.rules.kubernetes to fetch the PrometheusRules created by a Sloth running there into Grafana Mimir.

A similar setup can be achieved with Thanos as long as you have deployed Thanos Ruler through the ThanosRuler CRD from the prometheus-operator (in order to get support from the very same operator to load PrometheusRules into Thanos Ruler).