Grafana Register per Second Weirdness

[lukeescude]

Please review the attached screenshot - This is what our Register per Second graph looks like... Those spikes are when the Homer database becomes 100% full, so we clear out the data and restart the containers.

Is the register per second graph supposed to look like waves? I feel like anything measured "per second" shouldn't really have any patterns to it...

[lmangani]

Hi @lukeescude
This very much depends on the traffic itself and how quickly the devices will register and refresh, etc. so the waves might be normal, and you should use aggregation and averaging to get them into a logical line I suppose.

I would be more concerned about those spikes - this should not happen or impact statistics this bad....

[lukeescude]

Well, we are filtering out OPTIONS and NOTIFY packets so they aren't stored to the database (I am not certain if that function actually works, but I can double check). Whenever we stop all containers, clear the postgresdb, then start all the containers again, those spikes occur.

Here is a screenshot from a larger timeline. (past 2 days, instead of past 12 hours)

It definitely keeps things looking interesting!

[lukeescude]

When the hard drive fills to 99%, the following script is run:

docker kill $(docker ps -q)
cd /root/homer7-docker/heplify-server/hom7-prom-all/ && rm -rf postgres-data && docker-compose up -d

Line 1 kills all the docker containers. Line 2 erases postgres-data and brings the containers back up with docker-compose.

Interestingly, this actually isn't a good long-term solution... Slowly, over time, the hard drive fills to 99% more quickly each time. So eventually, we must run the following full maintenance code (every 2 weeks):

docker kill $(docker ps -q)
cd /root/homer7-docker/heplify-server/hom7-prom-all/ && rm -rf postgres-data && git pull && docker system prune -a && docker-compose up -d

First line kills all containers, second line erases postgres-data, pulls new git from homer7-docker, System Prune for docker, and brings containers back up.

[lukeescude]

Here is another example with less data, so it's easier to see - It's like Grafana is treating Register Per Second as a sin() wave or something, since our platform doesn't burst register requests like that.