Monitor Scylla with Prometheus and Grafana

Prometheus / Grafana is a popular monitoring stack these days. The following is a quick and dirty instruction on how to set a Prometheus / Grafana monitoring server, and how to connect Scylla to it.

Note this is NOT a production setup. For that, you probably need to run all program below as services etc

Setting Scylla

I'm using 100.100.100.100 as Scylla node IP example

collectd_exporter

Unlike other monitoring stacks, like Graphite, Prometheus pull metrics from servers. By default, Scylla pushes metrics in collectd format to the local server. To have Prometheus read the metrics, we will run a local collectd_exporter server on each Scylla node. collectd_exporter act as collectd server and expose the metrics as HTTP server for Prometheus.

first, we need the local collectd server to forward metrics to collectd_exporter. Update /etc/collectd.d/scylla.conf as proxy

LoadPlugin network
LoadPlugin disk
LoadPlugin interface
LoadPlugin unixsock
LoadPlugin df
LoadPlugin processes
<Plugin network>
        Listen "127.0.0.1" "25826"
        Server "127.0.0.1" "65534"
        Forward true
</Plugin>
<Plugin disk>
</Plugin>
<Plugin interface>
</Plugin>
<Plugin "df">
  FSType "xfs"
  IgnoreSelected false
</Plugin>
<Plugin unixsock>
	SocketFile "/var/run/collectd-unixsock"
	SocketPerms "0666"
</Plugin>
<Plugin processes>
    Process "scylla"
</Plugin>

and restart collectd

sudo service collectd restart

Second we need to start collectd_exporter

1. download collectd_exporter wget https://github.com/prometheus/collectd_exporter/releases/download/0.2.0/collectd_exporter-0.2.0.linux-amd64.tar.gz
2. extract

tar -xvf collectd_exporter-0.2.0.linux-amd64.tar.gz

3. run

./collectd_exporter -collectd.listen-address="0.0.0.0:65534"

(source: https://github.com/prometheus/collectd_exporter)

Done. To Validate, direct your browser to the server address, port 9103 path /metrics. For example: http://100.100.100.100:9103/metrics

Running collectd_exporter as a service

A better alternative to the above step 3, is to run collectd_exporter as a service to do that:

3. Install it

sudo mv collectd_exporter /usr/bin/

4. create a service file

cat> collectd-exporter.service << EOF
[Unit]
Description=Collectd Exporter

[Service]
Type=simple
User=scylla
Group=scylla
ExecStart=/usr/bin/collectd_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo mv collectd-exporter.service /etc/systemd/system/

Note that the above service file assumes that there is a scylla user, if there is none, use the same user you use to run scylla.

5. starting the service

systemctl start collectd-exporter.service

6. testing the service

systemctl status collectd-exporter.service

Should show the status as Active Following the step

Server side

Use scylla-grafana-monitoring to run a Prometheus/Grafana server with Scylla dashboard