IMPORTANT: PROJECT SUNSET AND ARCHIVED

No new maintainers found in #38

Logback Redis Batch Appender

Enables Java applications to log directly to redis via the jedis client as part of centralized logging with the ELK stack.

More specifically, it uses async appenders and JSON encoding of the logstash-logback-encoder project. Messages are sent to redis in batches for performance reasons. The redis sentinel functionality is supported.

Maven dependencies

<dependency>
   <groupId>de.idealo.logback</groupId>
   <artifactId>logback-redis</artifactId>
   <version>1.5.0</version>
</dependency>

used best in conjunction with

<dependency>
   <groupId>net.logstash.logback</groupId>
   <artifactId>logstash-logback-encoder</artifactId>
   <version>6.2</version>
</dependency>      

Configuration

Simple Configuration

<configuration>
    ...
    <appender name="REDIS" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
        <appender class="de.idealo.logback.appender.RedisBatchAppender">
            <connectionConfig>
                <scheme>SENTINEL</scheme>
                <sentinelMasterName>${sentinel.master.name}</sentinelMasterName>
                <sentinels>${sentinel.host.list}</sentinels>
                <key>${sentinel.key}</key>
                <ssl>${redis.ssl}</ssl> <!-- false (default) or true -->
            </connectionConfig>
            <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <arguments/>
                    <mdc/>
                    <pattern>
                        <pattern>
                            {
                            "@stage":"${STAGE}",
                            "app": "${projectName}",
                            "host": "${HOSTNAME}",
                            "key" : "userservices",
                            "level": "%level",
                            "logger": "%logger",
                            "message": "%message",
                            "thread": "%thread",
                            "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss.SSSZZ}"
                            }
                        </pattern>
                    </pattern>
                    <stackTrace/>
                </providers>
            </encoder>
        </appender>
    </appender>
    ...
</configuration>

Parameters

• connectionConfig:
  • key: key under which messages are stored in redis
  • scheme (NODE | SENTINEL): defines whether redis is accessed via a single node or via sentinel
  • for scheme=SENTINEL:
    • sentinelMasterName: name of the sentinel master
    • sentinels: comma separated list of sentinels with the following structure: host1:port1,host2:port2
  • for scheme=NODE:
    • host: redis host
    • port: redis port
  • method (RPUSH | PUBLISH): defines the method to that should be used to send values to redis. with method PUBLISH the value defined as key is used as channel name. if method is omitted, then RPUSH is used.
  • ssl: Whether to use SSL to communicate with redis (false or true, default is false). Your client and server certificates must be set up correctly.
• maxBatchMessages: number of messages which are sent as batch size to redis
• maxBatchSeconds: time interval in seconds after a batch of messages is sent to redis if the batch size is not reached
• encoder: encoder for JSON formatting of the messages
• ringBuffer and waitStrategyType determine how the logstash-logback-encoder asynchronously processes the messages. Note that messages may be lost if the ring buffer size is too small ("If the RingBuffer is full (e.g. due to slow network, etc), then events will be dropped.").

Extended Configuration

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
    <appender name="REDIS_APPENDER" class="net.logstash.logback.appender.LoggingEventAsyncDisruptorAppender">
        <ringBufferSize>131072</ringBufferSize>
        <appender class="de.idealo.logback.appender.RedisBatchAppender">
            <connectionConfig>
                <!-- redis sentinel: -->
                <scheme>SENTINEL</scheme>
                <sentinelMasterName>mymaster</sentinelMasterName>
                <sentinels>server:26379</sentinels>
                <!-- redis node: -->
                <!--<scheme>NODE</scheme>-->
                <!--<host>server</host>-->
                <!--<port>6379</port>-->
                <key>keyForRedis</key>
            </connectionConfig>
            <maxBatchMessages>1000</maxBatchMessages>
            <maxBatchSeconds>10</maxBatchSeconds>
            <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
                <providers>
                    <mdc/>
                    <pattern>
                        <pattern>
                            {
                            "timestamp": "%d{yyyy-MM-dd'T'HH:mm:ss.SSSZZ}",
                            "message": "%message",
                            "logger": "%logger",
                            "thread": "%thread",
                            "level": "%level",
                            "host": "${HOSTNAME}",
                            "file": "%file",
                            "line": "%line",
                            "app": "${projectName}"
                            }
                        </pattern>
                    </pattern>
                    <stackTrace>
                        <throwableConverter class="net.logstash.logback.stacktrace.ShortenedThrowableConverter">
                            <maxDepthPerThrowable>30</maxDepthPerThrowable>
                            <maxLength>4096</maxLength>
                            <shortenedClassNameLength>20</shortenedClassNameLength>
                            <rootCauseFirst>true</rootCauseFirst>
                        </throwableConverter>
                   </stackTrace>
                </providers>
            </encoder>
        </appender>
    </appender>
    ...
</configuration>

This appender configuration can either be included in a logback.xml file (via "included" tag) or be directly contained in a logback.xml (without "included" tag).

JSON Format Created by the Appender (Example) (= Input for Logstash)

{
    "_index": "myIndex-2015.09.15.12",
    "_type": "logs",
    "_id": "AU_RAKiIuARjD1TqcEFe",
    "_score": null,
    "_source": {
        "mdcKey1": "value1",
        "mdcKey2": "value2",
        "seq": "198",
        "timestamp": "2015-09-15T14:35:19.256+0200",
        "message": "logback-1:198",
        "logger": "LoggingTest",
        "thread": "main",
        "level": "INFO",
        "host": "myHost",
        "file": "?",
        "line": "?",
        "@version": "1",
        "@timestamp": "2015-09-15T12:35:25.251Z",
        "type": "logs"
},
    "fields": {
    "@timestamp": [
        1442320525251
    ]
},
    "sort": [
    1442320525251
]
}

Logging Appender Errors to a File

The following logger configuration in the logback.xml is recommended in order to write error messages of the appender directly to a file in error situations (especially if redis is not available):

<logger name="de.idealo.logback.appender" level="error" additivity="false">
    <appender-ref ref="FILE_FOR_REDISBATCHAPPENDER"/>
</logger>

Shutdown

Shutdown Hook

The redis batch appender must be shut down on application shutdown in order to ensure that cleans up background threads and pools and ensures that remaining messages are sent to Redis before shutting down the app. This is performed by the stop method of the redis batch appender that is automatically called when putting a shutdown hook in logback.xml:

<shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>

Spring Boot Apps

The shutdown hook above doesn't work in Spring Boot apps when they are shut down via the actuator shutdown URL (POST {baseUrl}/shutdown). Instead, this can be done by the following Spring component:

@Component
public class LogbackStopListener implements ApplicationListener<ContextClosedEvent> {
    @Override
    public void onApplicationEvent(final ContextClosedEvent event) {
        LogbackUtils.stopLogback();
    }
}