Using CAdvisor, Prometheus, and Grafana To Build A Valheim Dashboard

[bforbis]

I would like to share some information on how to setup a Grafana dashboard to track information about your running Valheim server. This Grafana dashboard keeps track of the following information:

• CPU Usage
• Memory Usage
• Server Online Status
• Player Online Count

You can download the Grafana dashboard at https://grafana.com/grafana/dashboards/14123

Docker Compose
The first step is to setup your Valheim docker compose to include CAdvisor, Prometheus, and Grafana

version: "2.4"
services:
  valheim:
    image: mbround18/valheim:latest
    container_name: valheim
    restart: unless-stopped
    ports:
      - 2456:2456/udp
      - 2457:2457/udp
      - 2458:2458/udp
      - 2459:2459/tcp
    environment:
      PORT: 2456
      HTTP_PORT: 2459
      PUBLIC: 1
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: always
    ports:
      - 9090:9090/tcp
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    ports:
      - 3000:3000/tcp
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: password
    volumes:
      - grafana-provisioning:/etc/grafana/provisioning:ro
  cadvisor:
    image: gcr.io/google-containers/cadvisor:latest
    container_name: cadvisor
    restart: always
    ports:
      - 8080:8080/tcp
    privileged: true
    command:
      # Defaults to 1s interval, 5s interval for reduced cpu
      - "-housekeeping_interval=5s"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      - /dev/disk:/dev/disk:ro

Prometheus Config

You should also have a prometheus.yml config file bind mounted to your prometheus container. This will be setup to scrape both your CAdvisor container and your Valheim HTTP Status server /metrics endpoint served at your HTTP_PORT

scrape_configs:
  - job_name: cadvisor
    static_configs:
    - targets: ['cadvisor:8080']
  - job_name: valheim
    static_configs:
    - targets: ['valheim:2459']

Grafana Datasource Provisioning
You should also setup Grafana to automatically setup your prometheus server as a datasource. You can do this by creating a datasource provisioning file

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    isDefault: true
    editable: false
    url: http://prometheus:9090

[mbround18]

Woah Excellent work!! :) It looks good!

[bforbis]

Looks like Grafana decided to feature this dashboard in a showcase for GrafanaCON, pretty neat!
https://grafana.com/blog/2021/06/22/grafana-dashboard-showcase-visualizations-for-prometheus-home-energy-usage-github-and-more/

[mbround18]

Thats awesome! congratz! :)

[ryanschmitty]

Hey @bforbis, I was trying to set this up on an Amazon Lightsail server, but it wasn’t working. I think it might have something to do with the Grafana datasource provisioning on the server. Could you provide a little more detail on what the dashboard is taking to? Is it the Grafana instance on your server at <public_ip>:9090? Or the Prometheus instance at that address? I don’t have write access to the Grafana-provisioning/ folder that was created, so I’m guessing I did something wrong there because I assume the Grafana datasource provisioning yaml is required.

[ryanschmitty]

It’s also just a little confusing as a Grafana and Prometheus newb why there’s a Grafana instance running on the server at all when the dashboard on Grafana.com wants you to connect to a Prometheus datasource during import. The system architecture here is just a little confusing.

[bforbis]

The dashboard is talking to a metrics database which is configured using the grafana provisioning and labeled as "Prometheus".

Grafana datasource provisioning is not required, it just makes it easy for you to have a hard coded configuration for the prometheus servers your grafana server will auto-connect to which can then be saved/backed up as config in your repo.

If you are having trouble setting up the datasource provisioning config in your grafana, you can always just manually setup a datasource by logging into Grafana as an admin user and setting up the datasource using the UI (Make sure to name the server "Prometheus" so that the dashboard can reference that location). Keep in mind that if your persistence is not backed up, you may lose these configurations and have to redo them in the case of data loss.

why there’s a Grafana instance running on the server at all when the dashboard on Grafana.com wants you to connect to a Prometheus datasource during import

I think you might just be confused on what Grafana and Prometheus are. A quick intro:
Grafana is the server that hosts your dashboards and makes metrics queries on your behalf which you configure into your dashboards. Prometheus is the metrics database that returns metrics based off those queries.

Hope this helps!

[ryanschmitty]

Wait a minute: is the dashboard being served up by the Grafana instance running on my server? I think that’s where I might be confused because I went to Grafana.com and tried to setup and account and dashboard there…

i.e. do I go to http://(my server ip):3000 to import/create and view my dashboard?

[ryanschmitty]

It's all working now, thanks.

There were a few snags I hit:

1. had to proactively create the grafana-provisioning/datasources/datasource.yml file before docker-compose up because the Grafana docker would create the grafana-provisioning/ folder and deny write access.
2. had to realize the Grafana server running on my host (at port 3000) is what serves the dashboard (as opposed to Grafana.com)
3. had to use the container_name: valheim config because the Grafana dashboard hardcodes that container name for the queries (mine was default named, which was something like "ubuntu_valheim_1").

[bforbis]

Glad you got it working. Enjoy!

[g0lg0-93]

Thanks for creating this, it's excellent! Spun up my first self-hosted & public valheim server, however, I'm using lloesche/valheim-server-docker instead of the one you're using. I've set the 'STATUS_HTTP' & 'STATUS_HTTP_PORT' to match (2549) as well as add the port on the environment portion of my compose file, yet I cannot retrieve any data from the valheim server. Is there something I'm missing to get this to work?

Ps. quite nooby when it comes to monitoring to this degree :)

[bforbis]

I am not very familiar with the lloesche/valheim-server project, but if it does not expose a /metrics HTTP endpoint that returns metrics in Prometheus format, you will not be able to get this dashboard to work. Just a brief look at that projects README, it seems that the STATUS API is just some custom JSON format, which is not the same thing as prometheus.

The prometheus endpoint this setup is hitting to retrieve player count is something that was custom to the mbround18 project, part of a feature request that I put in a while ago. If you want the same thing for the other project, you'll either need to create a feature request there or you could build a translation service that reads the STATUS JSON and re-exposes it in the needed prometheus format (seems like a pain, but shouldn't be too hard).

If you are interested in just CPU utilization, that stuff is generic enough that you could just use cAdvisor and a standard cAdvisor grafana graph.

[mbround18]

@g0lg0-93 unfortunately, we are not affiliated with that repository and you will have to direct all your issues or inquires over to lloesche/valheim-server-docker

However, if you want to hack it, you can download our bundle and start Huginn inside of the lloesche container. Thats how you can expose a metrics endpoint.We bundle odin and huginn seperately in an archive file. Which you are welcome to integrate into the lloesche container. Additional note, the odin binary can also be used to control the lloesche version of valhiem with some minor configuration.

[g0lg0-93]

We're not using the Valheim server atm (call it maintenance), so I'm going to move our world over to your repo and implement this amazing work!