Instrument data source requests consistently.

[bergquist]

Today we instrument Data source requests with two metrics grafana_datasource_request_total and grafana_plugin_request_total

• grafana_datasource_request_total instruments all data source requests. Ex grafana_datasource_request_duration_seconds_count{code="200", datasource="grafanacloudusage", method="post"}

• grafana_plugin_request_total instruments all plugins requests and contains the plugin_id as label. Ex grafana_plugin_request_total{endpoint="checkHealth", plugin_id="prometheus", status="ok"}

Ideally we want to instrument everything using the same metrics since that makes it easier for operator to slice and dice based on what they want instead of exposing two almost similar series.

I suggest we instrument both use-cases with grafana_datasource_request_total but extend it with plugin_id and endpoint. plugin_id should not impact the number of series exported and endpoint is bound to 4 different values shou which is within reasonable limits. grafana_plugin_request_total has status and grafana_datasource_request_total has code which I think we can merge into one value. Considering the vast majority of the data sources use http status code I think we should default to that.

With these new values we could aggregate how ever we see fit.

sum by (datasource) (irate(grafana_datasource_request_total[5m]))

sum by (plugin_id) (irate(grafana_datasource_request_total[5m]))

[marefr]

First just wanted to highlight some details:

• grafana_datasource_request_* metrics is for outgoing HTTP requests and currently includes "datasource", "code", "method" as labels
• grafana_plugin_request_* metrics is for instrumenting any backend plugin request and currently includes "plugin_id", "endpoint", "status" as labels. Status label is either ok or error.

These are different by nature:

• grafana_datasource_request_* metrics is captured if the datasource is using HTTP and requests comes in via
  • /api/datasources/proxy
  • /api/ds/query
  • /api/tsdb/query
  • backend, e.g. alerting or recorded queries
• grafana_plugin_request_* metrics is captured if a plugin has a backend and requests comes in via
  • /api/ds/query
  • /api/tsdb/query
  • /api/plugins/<plugin id>/health
  • /api/plugins/<plugin id>/resources
  • /api/plugins/<plugin id>/metrics
  • /api/datasources/id/health
  • /api/datasources/id/resources

Note: For a core backend datasource both grafana_datasource_request_* and grafana_plugin_request_* metrics will be captured.

Ideally we want to instrument everything using the same metrics since that makes it easier for operator to slice and dice based on what they want instead of exposing two almost similar series.

I understand that. My main question is what kind of answers do you need this metrics to answer and how are you planning to use the labels?

I suggest we instrument both use-cases with grafana_datasource_request_total but extend it with plugin_id and endpoint. plugin_id should not impact the number of series exported and endpoint is bound to 4 different values shou which is within reasonable limits. grafana_plugin_request_total has status and grafana_datasource_request_total has code which I think we can merge into one value. Considering the vast majority of the data sources use http status code I think we should default to that.

Not saying no to this, but feels a bit weird. To me it feels more like we're lacking a metric for querying data. Or I would rather include more labels on grafana_plugin_request_* to include datasource reference/name. Again, depends a bit on what answers you want from the metrics.

[bergquist]

grafana_datasource_request_* metrics is for outgoing HTTP requests and currently includes "datasource", "code", "method" as labels

Oh I assumed SQL datasources where also included but ofc they aren't... That explains some issues we had.. 🤦‍♂️

Not saying no to this, but feels a bit weird. To me it feels more like we're lacking a metric for querying data. Or I would rather include more labels on grafana_plugin_request_* to include datasource reference/name. Again, depends a bit on what answers you want from the metrics.

I find grafana_datasource_* easier to understand as an operator (datasources is all over the project. Its what we configure and specify in dashboards). querydata or grafana_plugin_* does make sense as an grafana developer but we are not the audience. I don't have a super strong opinion about the name thou. We should measure all Datasource requests the same way to make it easier for operators.
This does mean we need to replace http status codes with a smaller set of status codes but I think that's fine in this case. I think this would cover 99% of the datasource instrumentation needed by operators. Something like

grafana_<todo>_{endpoint="queryData", plugin_id="prometheus", status="ok", datasource="grafanacloud-prom"}

We should measure latency, response size with the same labels. But I'm not sure what to do with web.sockets..

We would still need a metric for the plugin specific endpoints but I don't think that many people care about those metrics.

/api/plugins/<plugin id>/health
/api/plugins/<plugin id>/resources
/api/plugins/<plugin id>/metrics

[gelicia]

Hello, as you may have heard, we are transitioning away from using discussions to discuss feature requests. Due to the age and number of responses to this discussion, we are deciding to close it. If this is something you would like to see in Grafana, feel free to open an issue so the discussion can continue. Thank you!