Monitor Scylla with Prometheus and Grafana

Prometheus / Grafana is a popular monitoring stack these days. The following is a quick and dirty instruction on how to set a Prometheus / Grafana monitoring server, and how to connect Scylla to it.

Note this is NOT a production setup. For that, you probably need to run all program below as services etc

Setting Scylla

I'm using 100.100.100.100 as Scylla node IP example

collectd_exporter

Unlike other monitoring stacks, like Graphite, Prometheus pull metrics from servers. By default, Scylla pushes metrics in collectd format to the local server. To have Prometheus read the metrics, we will run a local collectd_exporter server on each Scylla node. collectd_exporter act as collectd server and expose the metrics as HTTP server for Prometheus.

first, we need the local collectd server to forward metrics to collectd_exporter. Update /etc/collectd.d/scylla.conf as proxy

LoadPlugin network
LoadPlugin disk
LoadPlugin interface
LoadPlugin unixsock
LoadPlugin df
LoadPlugin processes
<Plugin network>
        Listen "127.0.0.1" "25826"
        Server "127.0.0.1" "65534"
        Forward true
</Plugin>
<Plugin disk>
</Plugin>
<Plugin interface>
</Plugin>
<Plugin "df">
  FSType "xfs"
  IgnoreSelected false
</Plugin>
<Plugin unixsock>
	SocketFile "/var/run/collectd-unixsock"
	SocketPerms "0666"
</Plugin>
<Plugin processes>
    Process "scylla"
</Plugin>

and restart collectd

sudo service collectd restart

Second we need to start collectd_exporter

1. download collectd_exporter wget https://github.com/prometheus/collectd_exporter/releases/download/0.2.0/collectd_exporter-0.2.0.linux-amd64.tar.gz

2. extract

gunzip collectd_exporter-0.2.0.linux-amd64.tar.gz
tar -xvf collectd_exporter-0.2.0.linux-amd64.tar

2. run

./collectd_exporter -collectd.listen-address="0.0.0.0:65534"

(source: https://github.com/prometheus/collectd_exporter)

Done. To Validate, direct your browser to the server address, port 9103 path /metrics. For example: http://100.100.100.100:9103/metrics

Server side

Use scylla-grafana-monitoring to run a Prometheus/Grafana server with Scylla dashboard