Integrate Prometheus, cAdvisor and Grafana to monitor Containers for resource usage ?

[karuppiah7890]

https://github.com/prometheus/prometheus
https://github.com/google/cadvisor
https://github.com/grafana/grafana

This is some DevOps stuff. I think it will be useful for people to know the usage of system resources by the Docker containers and may be we can set some simple alerts too, to know that some resource usage limit has reached. Also, we can export some custom metrics for M4G application to know that the application is running and healthy. We need to decide on the metrics though. Also, currently we run the application using PM2 process manager, may be we can get some metrics from that too

[ishanjain28]

If anyone else hasn't starting working on this yet, I can integrate Prometheus and Grafana in this application. I'll have to look up what cadvisor does and how to integrate it.

Thanks for the suggestion, @karuppiah7890

[karuppiah7890]

@ishanjain28 cAdvisor is used to monitor Docker containers :)

And actually, this is still a suggestion. We have to discuss and make sure it's a necessary feature and then only work on it. We need to wait for @AndrewGHC and the other contributors to comment on this. I would suggest you to wait on this. And it's not a very tough implementation to plug and play Prometheus and Grafana, we just need to run the Docker containers. The important part is about creating a /metrics endpoint in Mail for Good application so that Prometheus can scrape the data from there. We need to discuss that too, and then we can start with this

[ishanjain28]

@karuppiah7890 Okay, I'll wait.

[AndrewWalsh]

@karuppiah7890 @ishanjain28 Previously we ran the app with PM2 as a process manager. Our logging in this regard could do with improvement.

My knowledge on devops is limited, particularly with concern to deploying dockerised containers on remote services. I'm open to suggestions in this regard.

[karuppiah7890]

@AndrewGHC Actually, Prometheus is only for monitoring, and it's quite different from logging, but if we can create application logs with specific format and scrape the logs in the form of metrics, then we can use them as metrics too. But this won't be very great for error detection and finding root cause of error and stuff. This is more like - monitoring if everything is working normal and you can set alerts to define what is abnormal and get notifications during abnormal situations, like high CPU or memory usage