troubleshooting: Monitoring does not work

Using Grafana/Prometheus monitoring tool

Problem: "No data points" on all charts

• Prometheus targets might be wrong. Check your prometheus/scylla_servers.yml and prometheus/node_exporter_servers.yml to make sure prometheus is pulling the data from the right targets (scylla servers)

• Your dashboard and Scylla version might not align. If you are running Scylla 1.5.x for example, you need to start the monitoring server with ./start-all.sh -v 1.5. If you are monitoring more than one version of Scylla, you can use ./start-all.sh -v 1.5,1.6 More on start-all.sh options

Problem: All of Grafana chart shows error (!) sign

The problem is in the connection between Grafana and Prometheus. On the monitoring server:

1. check Prometheus is running with sudo docker ps. If its not running is probably a mistake in the prometheus.yml file
2. if it is running, go to "Data Source" in the Grafana GUI, choose Prometheus and click Test Connection.

Problem: Grafana shows server level metrics like disk usage, but not Scylla metrics.

Prometheus fail to fetch metrics from Scylla servers.

• use curl 172.17.0.2:9180/metrics (where 172.17.0.2 is a Scylla server IP) to fetch binary metric data from Scylla. If curl does not return data, the problem is the connectivity between the monitoring and Scylla server. Please check your IPs and firewalls.

Problem: Grafana dashboard shows Scylla metrics, like load, but not server metrics like disk usage

Prometheus fail to fetch metrics from node_exporter

1. Make sure node_exporter is running on each Scylla server. node_exporter is installed by scylla_setup. If it does not, make sure to install and run it.

2. If is running, use curl 172.17.0.2:9100/metrics (where 172.17.0.2 is a Scylla server IP) to fetch binary metric data from Scylla. If curl does not return data, the problem is the connectivity between the monitoring and Scylla server. Please check your IPs and firewalls.

Still not working? might be a bug

1. Check for known Scylla issues and Monitoring issues
2. Ask question in the mailing list or open a new issue

Using Scylla collectd metrics directly

Problem: Scylla metric are not shown on my monitoring tool

You might use scylla-monitoring or any other monitoring tool. You came to this page because its not working. Here is what you should do.

Do Scylla generate metrics?

• Login to scylla box, install wireshark sudo yum install wireshark
• Use tshark (the CLI version) to look for collect metrics: sudo tshark -i eth0 -f "dst port 25826"
• If you see metrics, Scylla generate metrics, and you can jump to the next step.
• If not, make sure scylla is configure to send metrics

Do the monitoring server get metrics?

• Login to monitoring box, install wireshark (see above) and run sudo tshark -i eth0 -f "dst port 25826"
• See metrics? the problem might be the monitoring system
  • Does the monitoring server listen for on the right port?
• Do not see metrics? might be one of the following:
  • Firewall between the servers block the metric port
  • Scylla send metric to the wrong IP:port
  • Monitoring IP is not accessible from scylla server