Massive logs from Hasura v2.13.0

[stevenyongezypay]

We just upgraded our Hasura from older v1.3.2 to v2.13.0.

The upgrades went well however we discover some massive logs that appear almost every 1 to 5 seconds that says something like this:

{
    "detail": {
        "http_info": {
            "content_encoding": null,
            "http_version": "HTTP/1.1",
            "ip": "10.50.30.89",
            "method": "GET",
            "status": 200,
            "url": "/healthz"
        },
        "operation": {
            "request_id": "d655fae4-7360-471a-bd43-a55313c6312d",
            "request_mode": "non-graphql",
            "response_size": 2,
            "uncompressed_response_size": 2
        },
        "request_id": "d655fae4-7360-471a-bd43-a55313c6312d"
    },
    "level": "info",
    "timestamp": "2022-10-19T03:04:00.733+0000",
    "type": "http-log"
}

We couldn't figure out if these are health check logs as they seem to be different to what has been documented here:

https://hasura.io/docs/latest/deployment/logging/#health-check-log-structure

We would like to know what logs are these and how do we adjust the interval or even disable them totally as they are killing out logs quota!

[meetzaveri]

Hey @stevenyongezypay

Are you using self-hosted version of Hasura or Hasura Cloud ?

Seeing those records, those logs are health check logs. We can see that pattern of logs pointing tohealthz endpoint, so those are health checks made internally from graphql-engine.

[stevenyongezypay]

Hello @meetzaveri

We are hosting the Hasura in our AWS ESC Fargate container by pulling Hasura Docker image:

"image": "hasura/graphql-engine:v2.13.0"

Seeing those records, those logs are health check logs. We can see that pattern of logs pointing tohealthz endpoint, so those are health checks made internally from graphql-engine.

We suspect they are health checks too but what we do not understand is that it's making health check to 4 different IPs. One of the IPs is the Hasura container itself and we couldn't figure out the what are the other 3 IPs from.

We thought one of the other 3 might be the database server as Hasura can do Source Health Check but that does not seem to be case as that is available for Cloud edition if not mistaken.

Why would Hasura checking health checks for different IPs other than itself?

In fact we are fine that it's doing health check, but how do we configure the interval? Right now it's doing like every 1 to 5 seconds! We are unable to figure out what types of health check are these and where to configure them.

[meetzaveri]

Hey @stevenyongezypay

Unfortunately, we don't have configuration to set frequency of logs generated by engine. Moreover, I've created a GH issue as a feature request as we received request from other users regarding this type of feature. Link - #9833

Please comment or add reaction there so that it can gain some traction. This would help other users to raise their opinion and eventually based on the reception it receives, it will prove as a validation for product team to include this as a feature request.

[akhil-naidu]

how do we adjust the interval or even disable them totally

The healthz endpoint is an internal log, with the log level being info for healthy instance and warn for unhealthy instances.

By default, Hasura sets the logging level to info and to reduce the number of logs being captured, one can always set the log-level = warn to display only the logs of levels from and above warn, which also includes error.

When the log-level = info, all the logs related to info, warn and error will be displayed

To change the log-level, use the ENV variable HASURA_GRAPHQL_LOG_LEVEL = warn. For more info on the log level, refer this documentation

One can also configure the types of logs that are to be displayed using the HASURA_GRAPHQL_ENABLED_LOG_TYPES ENV variable; for more info on log types, go through this documentation.

---

Quick Summary

1. Hasura provides 4 different different levels of logging, debug, info, warn, and error. And by using the HASURA_GRAPHQL_LOG_LEVEL you can configure the level.
2. Hasura also provides 5 different types of logs, startup, query-log, http-log, websocket-log, webhook-log. You can also choose what type of logs to be displayed using the ENV variable HASURA_GRAPHQL_ENABLED_LOG_TYPES