Monitor Scylla with Prometheus and Grafana

Prometheus / Grafana is a popular monitoring stack these days. The following is a quick and dirty instruction on how to set a Prometheus / Grafana monitoring server, and how to connect Scylla to it.

Note this is NOT a production setup. For that, you probably need to run all program below as services etc

Setting Scylla

I'm using 100.100.100.100 as Scylla node IP example

collectd_exporter

Unlike other monitoring stacks, like Graphite, Prometheus pull metrics from servers. By default, Scylla pushes metrics in collectd format to the local server. To have Prometheus read the metrics, we will run a local collectd_exporter server on each Scylla node. collectd_exporter act as collectd server and expose the metrics as HTTP server for Prometheus.

first, we need the local collectd server to forward metrics to collectd_exporter. Update /etc/collectd.d/scylla.conf as proxy

LoadPlugin network
LoadPlugin disk
LoadPlugin interface
LoadPlugin unixsock
LoadPlugin df
LoadPlugin processes
<Plugin network>
        Listen "127.0.0.1" "25826"
        Server "127.0.0.1" "65534"
        Forward true
</Plugin>
<Plugin disk>
</Plugin>
<Plugin interface>
</Plugin>
<Plugin "df">
  FSType "xfs"
  IgnoreSelected false
</Plugin>
<Plugin unixsock>
	SocketFile "/var/run/collectd-unixsock"
	SocketPerms "0666"
</Plugin>
<Plugin processes>
    Process "scylla"
</Plugin>

and restart collectd

sudo service collectd restart

Second we need to start collectd_exporter

1. download collectd_exporter wget https://github.com/prometheus/collectd_exporter/releases/download/0.2.0/collectd_exporter-0.2.0.linux-amd64.tar.gz

2. extract ...``` tar -xvf collectd_exporter-0.2.0.linux-amd64.tar.gz

3. run 

./collectd_exporter -collectd.listen-address="0.0.0.0:65534"

(source: https://github.com/prometheus/collectd_exporter)

Done. To Validate, direct your browser to the server address, port 9103 path /metrics.
For example: `http://100.100.100.100:9103/metrics`

### Running collectd_exporter as a service
A better alternative to the above step 3, is to run collectd_exporter as a service to do that:

3. Install it

sudo mv collectd_exporter /usr/bin/

4. create a service file

cat> collectd-exporter.service << EOF [Unit] Description=Collectd Exporter

[Service] Type=simple User=scylla Group=scylla ExecStart=/usr/bin/collectd_exporter

[Install] WantedBy=multi-user.target EOF

sudo mv collectd-exporter.service /etc/systemd/system/

Note that the above service file assumes that there is a scylla user, if there is none, use the same user you use to run scylla.

5. starting the service

systemctl start collectd-exporter.service

6. testing the service

systemctl status collectd-exporter.service

Should show the status as Active
Following the step 
## Server side
Use [scylla-grafana-monitoring](https://github.com/scylladb/scylla-grafana-monitoring) to run a Prometheus/Grafana server with Scylla dashboard