How to group logs by request like python 2.7 standard?

[lucasvanhalst]

I've searched around in this repo's issues and and on other places, but couldn't find a definitive answer/solution.
#110 is basically the same but I don't see a solution there that still works.

On python 2.7 standard, logs are grouped by request, as shown below (same code running on py27 standard and py39 standard, both use flask as web framework)

python 2: there is 1 request log that you can expand to view all logs made in that request (the request log also has the highest severity of all logs):

python 3 with the following setup, the logs aren't grouped (the only thing relating them to each other is the traceId):

(in main.py / application entrypoint)

from google.cloud.logging import Client as LogClient

log_client = LogClient()
excluded_loggers = ('urllib3.connectionpool',)
log_client.setup_logging(log_level=logging.DEBUG, excluded_loggers=excluded_loggers)

I've also tried the deprecated AppEngineHandler but that gave the same results.

I know I can manually group the logs by filtering on trace id, but that's not nearly as convenient... Is there a solution available so we can have the log grouping like it was on python 2.7?

[minherz]

hi @lucasvanhalst ,
the log grouping (a.k.a. aggregation) is a feature of Log Explorer and does not depend on the programming language or the version of the logging client library. It works by grouping application (a.k.a. child) logs under the load balancer (a.k.a. parent) log entry following the matching set of fields in the log entries.

The child log entries have to have the same resource metadata AND the different logName AND the same httpRequest.URL AND trace fields.

To implement it you have to use the logging client library in Python that automatically populates the httpRequest and trace fields and to make sure that all log entries ingested in your backend (behind LB) propagate the trace information within nested calls.

[lucasvanhalst]

@minherz thanks, but that was exactly my problem, I am using that library as documented (that library is this repository).
The problem is that in python 2.7 (which does not use this library), all logs made during a request are contained inside the parent/request log under protoPayload.line (this is an array containing all logs made during that request). So the log explorer simply shows all the request logs, and the app logs are contained in it, no extra work from the explorer is needed to show all logs made inside that request.
With the new logging library, those logs are not contained in the parent log, but are simply correlated to it by the traceId. The 'line' property of the request logs is only set when logs made by GAE itself are made, for example when an instance is terminated because of excessive memory use.
This means that for every log you do, a new log entry is made, separate from the request log, which makes it harder to view all logs for a specific request.

Recently though (3 weeks?), it seems like the log explorer added a function to automatically load the logs related to a request when you click on a request log, and there's also a new (or old, and I didn't notice it..) option 'correlate by' added which seems to achieve exactly what I want. It only shows the request logs (when you choose 'request log' as the property to correlate logs by), and loads the app logs for that request when you open the request log.
This makes it basically the same as python 2.7, although a lot slower as it has to load the app logs separately, which usually takes at least 3 seconds, and sometimes as long as 20.
The only thing missing now to be on par with 2.7 functionallity is the log level from the request log being elevated to the highest level from the app logs for that request, but that's a minor thing.

Unfortunatlly, when you activate the correlate by function, you cannot save the query you made which is a bit annoying

[daniel-sanche]

After checking in with the team, I've found that this is due to a difference in how logs are structured on the logging backend between App Engine Python 2.7 and App Engine Python 3+. Unfortunately, this isn't something that can be changed at the client library level, so I'm going to close this issue.

The Log Correlation feature you mentioned above is the closest equivalent to what you're looking for with the new backend. We have raised the issues you ran into while using the feature with the team

Thanks for reporting this

[pnico]

Please revise the documentation that directs appengine customers to use this library in the case when all they want is a way to group logs per request. If this is all they need, and they do not need to call any functions in the logging API itself, they do not need this library to do it, and importing it has a large memory cost.

[shreejad]

@pnico The "Correlate By" feature will not give associated Application logs if the Cloud Logging client's 'setup_logging' method is not called - https://cloud.google.com/logging/docs/setup/python#write_logs_with_the_standard_python_logging_handler