[Idea] Prometheus metrics

[rurus9]

In the world of kubernetes, exposes prometheus metrics has become a de facto standard.
Have you thought about adding such functionality to glewlwyd2 (ex. for used OAuth2 flows: number of issued tokens, number of refusals, number of refreshed tokens)?

Thanks for the great project, its very usefull for me.

[babelouest]

Hi,

I like the idea but I don't know how prometheus works.

Glewlwyd sends a log message for many types:

• You have the event logs when a normal event appears, such as token creation, token refresh, user logged in, etc.
• You have the security logs described in the fail2ban filter paragraph. These events are triggered when a security event appears, such as user authorization invalid, invalid code or token, etc.
• You also have other kind of messages like error messages or debug messages for other purposes. They are more useful to find a one-time problem than being crunched by prometheus, mostly because their format isn't normalized like the event logs or the security logs

I imagine the fastest way would be for you to configure your prometheus with me as helper on what log message to use. I can also improve the log messages if we find improvements to make.

Then, if we manage to make something useful, I can improve the documentation with our experience on it.

What do you think?

[rurus9]

Hi, @babelouest

Prometheus metrics

So, Prometheus metrics are different than logs, they are for different purposes.
It works like this: Prometheus scrapes (http pull) some endpoints to store metrics in time series. When you have metrics in Prometheus, you can make some graphs, alerts, or use them in Grafana.

It's http, so you can use curl to just get metrics, a real example with the mariadb metrics:

$ curl http://10.42.155.54:9104/metric
(...)
# HELP mysql_global_status_innodb_row_ops_total Total number of MySQL InnoDB row operations.
# TYPE mysql_global_status_innodb_row_ops_total counter
mysql_global_status_innodb_row_ops_total{operation="deleted"} 0
mysql_global_status_innodb_row_ops_total{operation="inserted"} 80
mysql_global_status_innodb_row_ops_total{operation="read"} 715
mysql_global_status_innodb_row_ops_total{operation="updated"} 2
# HELP mysql_global_status_innodb_secondary_index_triggered_cluster_reads Generic metric from SHOW GLOBAL STATUS.
# TYPE mysql_global_status_innodb_secondary_index_triggered_cluster_reads untyped
mysql_global_status_innodb_secondary_index_triggered_cluster_reads 207
# HELP mysql_global_status_innodb_secondary_index_triggered_cluster_reads_avoided Generic metric from SHOW GLOBAL STATUS.
# TYPE mysql_global_status_innodb_secondary_index_triggered_cluster_reads_avoided untyped
mysql_global_status_innodb_secondary_index_triggered_cluster_reads_avoided 0
# HELP mysql_global_status_innodb_system_rows_deleted Generic metric from SHOW GLOBAL STATUS.
# TYPE mysql_global_status_innodb_system_rows_deleted untyped
mysql_global_status_innodb_system_rows_deleted 7
# HELP mysql_global_status_innodb_system_rows_inserted Generic metric from SHOW GLOBAL STATUS.
# TYPE mysql_global_status_innodb_system_rows_inserted untyped
mysql_global_status_innodb_system_rows_inserted 7
(...)

In the world of kubernetes, prometheus metrics are widely used.

For glewlwyd e.g. it would look like this:

# HELP glewlwyd2_oauth2_client_credentials_access_tokens_generated total number of issued tokens
# TYPE glewlwyd2_oauth2_client_credentials_access_tokens_generated counter
glewlwyd2_oauth2_client_credentials_access_tokens_generated 12945
# HELP glewlwyd2_oauth2_client_credentials_access_tokens_forbidden total number of forbidded tokens
# TYPE glewlwyd2_oauth2_client_credentials_access_tokens_forbidden counter
glewlwyd2_oauth2_client_credentials_access_tokens_forbidded 804

You need special endpoint (exporter) in you application for Prometheus, usually IP:PORT/metrics, so that he can scrapes metrics, and you can count rate, build dashboard in Grafana, set alarms et cetera.
Some links
https://www.redhat.com/sysadmin/introduction-prometheus-metrics-and-performance-monitoring
https://prometheus.io/docs/concepts/metric_types/

Logs

There is no logs for client credentials flow in OAuth2 plugin events? For me would be very helpful to have client IP in logs, similar to DEBUG logs, something like:
Debug log (from working glewlwyd2 instance):

2021-03-26T14:05:09Z - Glewlwyd DEBUG: oauth2 check_auth_type_client_credentials_grant - Error invalid input parameters. client_id: '(null)', scope: 'SCOPE', origin: 192.168.1.1

Proposed logs (client credentials flow, issued token and forbidden token):

<date_timestamp> - Glewlwyd INFO: Event oauth2 - Plugin '<plugin_name>' - Flow: <flow_name> - Access token generated for client '<client_id>' with scope list '<scope_list>' origin '<IP Address>'
<date_timestamp> - Glewlwyd INFO: Event oauth2 - Plugin '<plugin_name>' - Flow: <flow_name> - Access token forbidden for client '<client_id>' with scope list '<scope_list>' origin '<IP Address>'

It would be ideal to be able to set json as the log format, so you can put logs to json log parser (e.g. fluentd or logstash have plugins for it), with the example above it would be:

{"level":"info","plugin":"oauth2","flow":"client_credentials","action":"token_issued","scope_list":"SCOPES","origin":"IP_ADDRESS"}

What do you think?

[babelouest]

I think we have enough data to move this conversation to the issue page.

I have several questions to properly understand the objective, which can be answered in a new issue thread.

Also, the missing logs in the OAuth2 plugin should be answered in its own issue, to split a big problem into simple ones.

Can you open 2 separate issues for that? (or more if you see more issues like that)

The one about the metrics should list the metrics you think are useful and what they represent.
For example, does the metric glewlwyd2_oauth2_client_credentials_access_tokens_forbidden represent when a security event is triggered by an unauthorized client or when an client access_token is wrongly used in an endpoint?

[rurus9]

Sure, I'm on it already, thanks for the response.