structlog-extensions-nralbers: Extension processors for structlog

Structlog extensions are a set of structlog processors and utility functions to add new logging options to structlog. The primary purpose is to supply tools to convert existing structlog logging output into Elastic Common Schema json output so users can easily plug their application output into centralised logging solutions such as ELK stack.

At present the extensions consist of a CombinedLogParser (which for example can be used to convert gunicorn access log output into ECS fields), and the NestedDictJSONRenderer, which can be used to convert the output of the CombinedLogParser into ECS json format output.

Usage

CombinedLogParser

This processor will parse events formatted in Apache Combined log format into Elastic common schema fields.

Example

This is an example for configuring gunicorn to emit json logs.

gunicorn.conf.py

import structlog
import structlog_extensions

# --- Structlog logging initialisation code

pre_chain = [
    # Add the log level and a timestamp to the event_dict if the log entry
    # is not from structlog.
    structlog.stdlib.add_log_level,
    structlog.stdlib.add_logger_name,
    structlog_extensions.processors.CombinedLogParser("gunicorn.access")
]

logconfig_dict = {
            "version": 1,
            "disable_existing_loggers": False,
            "formatters": {
                "json_formatter": {
                    "()": structlog.stdlib.ProcessorFormatter,
                    "processor": structlog.processors.JSONRenderer(),
                    "foreign_pre_chain": pre_chain,
                }
            },
            "handlers": {
                "error_console": {
                    "class": "logging.StreamHandler",
                    "formatter": "json_formatter",
                },
                "console": {
                    "class": "logging.StreamHandler",
                    "formatter": "json_formatter",
                }
            },
        }

NestedDictJSONRenderer

This processor will convert key names using a specified separator into nested dictionaries prior to rendering the output as JSON using the structlog JSONRenderer. This processor can for example convert Elastic Common Schema namespaced keynames produced by the CombinedLogParser into the nested JSON format that ECS specifies. This processor must be the final processor in a chain because it renders the output as a string instead of passing along an event dictionary.

Example

When using this logging initialisation:

# --- std logging initialisation code using structlog rendering
import structlog
import structlog_extensions

pre_chain = [
    # Add the log level and a timestamp to the event_dict if the log entry
    # is not from structlog.
    structlog.stdlib.add_log_level,
    structlog.stdlib.add_logger_name,
    structlog_extensions.processors.CombinedLogParser("gunicorn.access")
]

logging.dict_config( {
            "version": 1,
            "disable_existing_loggers": False,
            "formatters": {
                "json_formatter": {
                    "()": structlog.stdlib.ProcessorFormatter,
                    "processor": structlog_extensions.processors.NestedDictJSONRenderer('.'),
                    "foreign_pre_chain": pre_chain,
                }
            },
            "handlers": {
                "error_console": {
                    "class": "logging.StreamHandler",
                    "formatter": "json_formatter",
                },
                "console": {
                    "class": "logging.StreamHandler",
                    "formatter": "json_formatter",
                }
            },
        })

These entries (produced by structlog_extensions.processors.CombinedLogParser):

{ 'http.request.method': 'get', 'http:.request.referrer': 'http://www.example.com', 'http.version': '1.0'}`

will be transformed into the following nested json structure:

{ 'http': { 'version': '1.0',
            'request': { 'method': 'get',
                         'referrer': 'http://www.example.com'}
            }
}