New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: use standard output logs on serverless environments #228
Changes from all commits
dc2e4a8
0e37217
3a32c3d
e454c38
761eccd
4495cf5
6a3a595
3c1af26
cb8b57b
f25da46
b686173
dc85dd3
133c584
637f95c
570bc2d
3aa5baa
d1014f2
5bec0ea
7392bd2
cf20b69
12d2d37
4ae78d4
614d1bf
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,10 +16,10 @@ | |
|
||
import logging | ||
|
||
|
||
from google.cloud.logging_v2.logger import _GLOBAL_RESOURCE | ||
from google.cloud.logging_v2.handlers.transports import BackgroundThreadTransport | ||
from google.cloud.logging_v2.handlers._monitored_resources import detect_resource | ||
from google.cloud.logging_v2.handlers._helpers import get_request_data | ||
|
||
DEFAULT_LOGGER_NAME = "python" | ||
|
||
|
@@ -28,6 +28,38 @@ | |
_CLEAR_HANDLER_RESOURCE_TYPES = ("gae_app", "cloud_function") | ||
|
||
|
||
class CloudLoggingFilter(logging.Filter): | ||
"""Python standard ``logging`` Filter class to add Cloud Logging | ||
information to each LogRecord. | ||
|
||
When attached to a LogHandler, each incoming log will receive trace and | ||
http_request related to the request. This data can be overwritten using | ||
the `extras` argument when writing logs. | ||
""" | ||
|
||
def __init__(self, project=None): | ||
self.project = project | ||
|
||
def filter(self, record): | ||
# ensure record has all required fields set | ||
record.lineno = 0 if record.lineno is None else record.lineno | ||
record.msg = "" if record.msg is None else record.msg | ||
record.funcName = "" if record.funcName is None else record.funcName | ||
record.pathname = "" if record.pathname is None else record.pathname | ||
# find http request data | ||
inferred_http, inferred_trace = get_request_data() | ||
if inferred_trace is not None and self.project is not None: | ||
inferred_trace = f"projects/{self.project}/traces/{inferred_trace}" | ||
|
||
record.trace = getattr(record, "trace", inferred_trace) or "" | ||
record.http_request = getattr(record, "http_request", inferred_http) or {} | ||
record.request_method = record.http_request.get("requestMethod", "") | ||
record.request_url = record.http_request.get("requestUrl", "") | ||
record.user_agent = record.http_request.get("userAgent", "") | ||
record.protocol = record.http_request.get("protocol", "") | ||
Comment on lines
+54
to
+59
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. are you planning to do anything about log severity too? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This "Filter" class is to add new fields to the LogRecord object, that we can then use later when exporting logs in the Handler classes. Common fields like "severity" already exist in the LogRecord by default, so no need to add them here There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. are you planning to add other httprequest fields from here as well. Nonblocking comment. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. eventually, but some of them are more complicated, and I want to make sure they work consistently across environments. I want to put in the basics first |
||
return True | ||
|
||
|
||
class CloudLoggingHandler(logging.StreamHandler): | ||
"""Handler that directly makes Cloud Logging API calls. | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
# Copyright 2021 Google LLC All Rights Reserved. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
"""Logging handler for printing formatted structured logs to standard output. | ||
""" | ||
|
||
import logging.handlers | ||
|
||
from google.cloud.logging_v2.handlers.handlers import CloudLoggingFilter | ||
|
||
GCP_FORMAT = '{"message": "%(message)s", "severity": "%(levelname)s", "logging.googleapis.com/trace": "%(trace)s", "logging.googleapis.com/sourceLocation": { "file": "%(pathname)s", "line": "%(lineno)d", "function": "%(funcName)s"}, "httpRequest": {"requestMethod": "%(request_method)s", "requestUrl": "%(request_url)s", "userAgent": "%(user_agent)s", "protocol": "%(protocol)s"} }' | ||
|
||
|
||
class StructuredLogHandler(logging.StreamHandler): | ||
"""Handler to format logs into the Cloud Logging structured log format, | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe add a link to https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogSeverity or another appropriate link, so they know "structured log format" is a formally defined hting |
||
and write them to standard output | ||
""" | ||
|
||
def __init__(self, *, name=None, stream=None, project=None): | ||
""" | ||
Args: | ||
name (Optional[str]): The name of the custom log in Cloud Logging. | ||
stream (Optional[IO]): Stream to be used by the handler. | ||
""" | ||
super(StructuredLogHandler, self).__init__(stream=stream) | ||
self.name = name | ||
self.project_id = project | ||
|
||
# add extra keys to log record | ||
self.addFilter(CloudLoggingFilter(project)) | ||
|
||
# make logs appear in GCP structured logging format | ||
self.formatter = logging.Formatter(GCP_FORMAT) | ||
|
||
def format(self, record): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. don't you need to check that record is not null? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That shouldn't be necessary. We're just delegating the formatting work to another standard Python formatter object. If there were any error cases to worry about, the function we're calling can handle it. |
||
"""Format the message into structured log JSON. | ||
Args: | ||
record (logging.LogRecord): The log record. | ||
Returns: | ||
str: A JSON string formatted for GKE fluentd. | ||
""" | ||
|
||
payload = self.formatter.format(record) | ||
return payload |
+1 −0 | deployable/go/.dockerignore | |
+2 −2 | deployable/go/Dockerfile | |
+4 −0 | deployable/nodejs/.dockerignore | |
+4 −0 | deployable/nodejs/.gitignore | |
+35 −0 | deployable/nodejs/Dockerfile | |
+68 −29 | deployable/nodejs/app.js | |
+2 −2 | deployable/nodejs/package.json | |
+31 −0 | deployable/nodejs/tests.js | |
+105 −0 | envctl/env_scripts/nodejs/cloudrun.sh | |
+1 −0 | envctl/env_scripts/nodejs/functions.sh | |
+38 −0 | tests/nodejs/test_cloudrun.py |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
how is ContainerEngineHandler different from StructuredLogHandler?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They are essentially the same, but I'm going to leave ContainerEngineHandler in for now for API consistency. I plan on removing ContainerEngineHandler and AppEngineHandler in v3.0.0 in favor of more general classes