-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Monitor Scylla with Prometheus and Grafana
Prometheus / Grafana is a popular monitoring stack these days. The following is a quick and dirty instruction on how to set a Prometheus / Grafana monitoring server, and how to connect Scylla to it.
Note this is NOT a production setup. For that, you probably need to run all program below as services etc
From 1.4 version and later, the Scylla Prometheus API is the preferred way to monitor the system. By default the Prometheus API is open, so no configuration is required.
For node information (disk, network etc') we use the node_exporter, with Scylla installation there is an installation script node_exporter_install that can install node_exporter. If you run scylla_setup, you will be asked if you want node_exporter to be installed. You can always run node_exporter_install at a later time.
I'm using 100.100.100.100 as Scylla node IP example
Unlike other monitoring stacks, like Graphite, Prometheus pull metrics from servers. By default, Scylla pushes metrics in collectd format to the local server. To have Prometheus read the metrics, we will run a local collectd_exporter server on each Scylla node. collectd_exporter act as collectd server and expose the metrics as HTTP server for Prometheus.
First, we need the local collectd server to forward metrics to collectd_exporter.
Update /etc/collectd.d/scylla.conf
as proxy
LoadPlugin network
LoadPlugin disk
LoadPlugin interface
LoadPlugin unixsock
LoadPlugin df
LoadPlugin processes
LoadPlugin cpu
<Plugin network>
Listen "127.0.0.1" "25826"
Server "127.0.0.1" "65534"
Forward true
</Plugin>
<Plugin disk>
</Plugin>
<Plugin interface>
</Plugin>
<Plugin "df">
FSType "xfs"
IgnoreSelected false
</Plugin>
<Plugin unixsock>
SocketFile "/var/run/collectd-unixsock"
SocketPerms "0666"
</Plugin>
<Plugin processes>
Process "scylla"
</Plugin>
<Plugin cpu>
ReportByState true
ReportByCpu true
ValuesPercentage true
</Plugin>
and restart collectd
sudo service collectd restart
Second we need to start collectd_exporter
- download collectd_exporter
wget https://github.com/prometheus/collectd_exporter/releases/download/0.2.0/collectd_exporter-0.2.0.linux-amd64.tar.gz
- extract
tar -xvf collectd_exporter-0.2.0.linux-amd64.tar.gz
- run
./collectd_exporter -collectd.listen-address=0.0.0.0:65534
(source: https://github.com/prometheus/collectd_exporter)
Done. To Validate, direct your browser to the server address, port 9103 path /metrics.
For example: http://100.100.100.100:9103/metrics
A better alternative to the above step 3, is to run collectd_exporter as a service to do that:
- Install it
sudo mv collectd_exporter /usr/bin/
- create a service file
cat> collectd-exporter.service << EOF
[Unit]
Description=Collectd Exporter
[Service]
Type=simple
User=scylla
Group=scylla
ExecStart=/usr/bin/collectd_exporter -collectd.listen-address=0.0.0.0:65534
[Install]
WantedBy=multi-user.target
EOF
sudo mv collectd-exporter.service /etc/systemd/system/
Note that the above service file assumes that there is a scylla user, if there is none, use the same user you use to run scylla.
- starting the service
sudo systemctl start collectd-exporter.service
- testing the service
sudo systemctl status collectd-exporter.service
Should show the status as Active
Use scylla-grafana-monitoring to run a Prometheus/Grafana server with Scylla dashboard
Note: that if you are using Scylla 1.3 or earlier you should checkout scylla-grafana-monitoring v0.1
tag.
Scylla 1.4 and later works with master