Metrics being populated into Grafana only every 30 minutes

[Seitam]

Your question

Hello,

I am unable to display metrics in Grafana with a frequency higher than 30 minutes. The runs I do are always displayed at minutes "00" and "30". Ideally I would like to run tests every 10 minutes or less.

I am using the Performance Dashboard docker and crontab to execute the following command:

sudo docker run --rm -v "$(pwd):/sitespeed.io" --network host sitespeedio/sitespeed.io:33.4.0 --graphite.host localhost https://www.sitespeed.io/ --slug yourTestName --graphite.addSlugToKey true

I've read Sitespeed.io's documentation, looking for Graphite config parameters that would explain this behaviour (e.g. carbon.conf / storage-schemas), but I am failing to see what is probably obvious.

Any help will be greatly appreciated,

Thanks.

[soulgalore]

Hi @Seitam the magic happens in Graphite storage-schemas.conf: https://www.sitespeed.io/documentation/sitespeed.io/graphite/#configure-graphite

The retention decides how often your metric will be stored. Graphite allocates storage depending on your retention policy, so you want make sure it matches how often you want to test. I think https://github.com/statsd/statsd/blob/master/docs/graphite.md has a good explanation on how it works.

Then if you plan to use annotations (I use them to go to the HTML result file), you can configure them so they match your retention policy (see --graphite.annotationRetentionMinute). Like if you have 30 min for a metric, you also need to configure the annotation to be 30 min, that way the annotation will match the metrics timestamp in Graphite (so they appear on the right place).

Hope that helps!

[Seitam]

Hi @soulgalore ,

Thanks for the quick answer,

I already had taken a look at these config files and annotation retention policies, but failed to make it work:

The default storage-schemas.conf says:

[carbon]
pattern = ^carbon\.
retentions = 60:1d

[sitespeed_crux]
pattern = ^sitespeed_io\.crux\.
retentions = 60m:40d

[sitespeed]
pattern = ^sitespeed_io\.
retentions = 10m:40d

[catch_them_all]
pattern = .*
retentions = 5m:1d,15m:30d

Based on the config above, I believe I should not be worried about running tests every 10 minutes, as "carbon" is configured with a retention of 60s for a day, "catch_them_all" at 5m (for a day), and "sitespeed" at 10m (for 40 days).

However, when I execute:

sudo docker run --rm -v "$(pwd):/sitespeed.io" --network host sitespeedio/sitespeed.io:33.4.0 --graphite.host localhost https://www.sitespeed.io/ --slug yourTestName --graphite.addSlugToKey true --graphite.annotationRetentionMinute 10

I still see data points only every 30 minutes.
What I have noticed is that runs done between minutes "00" and "29" overwrite the value displayed for "00". The same happens for runs done between minutes "30" and "59" (they overwrite the result of "30"). I do not see this behaviour defined in any file.

What am I missing?

Thanks!

[soulgalore]

Hi @Seitam ok. Did you change the settings in storage conf or was that the settings the first time you did a test run? If you change settings, you need to either remove all metrics for the new settings to work or use a script that convert them. Else you can try to remove the metrics from disk and retry.

If it do not, I need to look into the code and test to verify that we send the correct timestamp to Graphite.

[Seitam]

Hi @soulgalore , I didn't modify the settings, these are the default ones that come with the docker.

Thanks!

[soulgalore]

Ok last question: What base OS are you using?

[Seitam]

@soulgalore Ubuntu 22.04.3 LTS

Thanks

[soulgalore]

Ok, there were people that had some problem before running it locally on a Mac. I verified that the timestamps are ok when they are sent from sitespeed.io (you can see the timestamps by starting at tcp server in the project sitespeed.io/tools/tcp-server.js and take that port it outputs and then use that port for graphite like sitespeed.io https://www.wikipedia.org --graphite.host 127.0.0.1 --graphite.port 52057 -n 1.

Looking at the default configurations that comes with our Graphite container it is set to 30 min:
https://github.com/sitespeedio/docker-graphite-statsd/blob/main/conf/graphite/storage-schemas.conf#L22-L24

Do you use that container or your own Graphite container? If you use it, maybe the ovverrides do not work when you mount your configuration files?

[Seitam]

Hi @soulgalore ,

I have been using the image git clone https://github.com/sitespeedio/sitespeed.io.git. The Graphite config files that come with it do not have 30 minutes of retention, but the aforementioned lines we discussed.

[soulgalore]

You mean that you are using the docker compose file in that repo? https://github.com/sitespeedio/sitespeed.io/blob/main/docker/docker-compose.yml ? That compose file uses the Docker container for Graphite in https://github.com/sitespeedio/docker-graphite-statsd see https://github.com/sitespeedio/sitespeed.io/blob/main/docker/docker-compose.yml#L28

In your docker-compose file, have you changed the line https://github.com/sitespeedio/sitespeed.io/blob/main/docker/docker-compose.yml#L44 to mount in a configuration file, else it will use the default one that comes in the container (with 30 min).

[Seitam]

Oh, that must be it then, thanks!