Splunk Logging for Java

Version 1.6.3

This project provides utilities to easily log data using Splunk's recommended best practices to any supported logger, using any of the three major Java logging frameworks (Logback, Log4J 2, and java.util.logging), to the HTTP Event Collector and to Splunk TCP inputs.

In particular, it provides:

Appenders for HTTP Event Collector for pushing from Logback, Log4J 2 and java.util.logging.
The SplunkCimLogEvent class, which encapsulates the CIM (Common Information Model) in Splunk Enterprise and best practices for semantic logging.
A TCP appender for Logback, which is the only one of the three frameworks listed above that doesn't provide native support for writing to TCP ports.
Example configuration files for all three frameworks, showing how to configure them to write to Splunk TCP ports.

Advice

Splunk Universal Forwarder vs Splunk TCP Inputs

If you can, it is better to log to files and monitor them with a Splunk Universal Forwarder. This provides you with the features of the Universal Forwarder, and added robustness from having persistent files. However, there are situations where using a Universal Forwarder is not a possibility. In these cases, writing directly to a TCP input is a reasonable approach.

In either scenario, we recommend using the SplunkCimLogEvent class provided by this library to construct your log events according to Splunk's recommended best practices.

Resilience

All of the TCP appenders we show config files for (SocketHandler for java.util.logging, SocketAppender for Log4J 2, and the TCPAppender provided with this library for Logback) will attempt to reconnect in case of dropped connections.

Data Cloning

You can use data cloning by providing multiple instances of your TCP handler in your logging configuration, each instance pointing to different indexers.

Load Balancing

Rather than trying to reinvent load balancing across your indexers in your log configuration, set up a Splunk Universal Forwarder with a TCP input. Have all your logging sources write to that TCP input, and use the Universal Forwarder's load balancing features to distribute the data from there to a set of indexers.

Thread Safety

Log4j and Logback are thread-safe.

Sending events to HTTP Event Collector

HTTP Event Collector requires Splunk 6.3+. Splunk Java library supports sending events through java.util.logging, log4j and logback standard loggers. In order to use HTTP Event Collector it has to be enabled on the server and an application token should be created.

Splunk Logging for Java includes several examples of configuration files in src/test/resources folder. For instance java.util.logging configuration looks like:

handlers=com.splunk.logging.HttpEventCollectorLoggingHandler
com.splunk.logging.HttpEventCollectorLoggingHandler.url=https://splunk-server:8088
com.splunk.logging.HttpEventCollectorLoggingHandler.token=<token-guid>

Sending events is simple:

Logger LOGGER = java.util.logging.Logger.getLogger("splunk.java.util");
LOGGER.info("hello world");

Message Format

An event message format could be configured for HTTP event appender in logging framework configuration. It could have one of the two possible values - text, json. It is an optional property with default value as 'text'. Message format 'json' is used where the event message could be in json format.

It is also possible to use a custom event body serializer for the HTTP event adapter, to format the logging event however you please. Simply create a class implementing com.splunk.logging.EventBodySerializer, and add the full class name as a property (eventBodySerializer) to the adapter. Default will be a JSON event body containing message, severity, and other properties.

For more information, see http://dev.splunk.com/view/SP-CAAAE2K.

License

Splunk logging for Java is licensed under the Apache License 2.0.

Details can be found in the LICENSE file.

Using Splunk Logging for Java

To use the Splunk Logging for Java library, you will need to add it and the logging library you have chosen to use to your project, open a TCP input on a Splunk instance to write your log events to, configure your logging system, and then use the SplunkCimLogEvent class to generate well formed log entries.

Add the Splunk Logging for Java library to your project. If you are using Maven, add the following to your dependencies section:
```
<dependency>
    <groupId>com.splunk.logging</groupId>
    <artifactId>splunk-library-javalogging</artifactId>
    <version>1.6.3</version>
</dependency>
```
You might also want to add the following repository to your repositories section:
```
<repository>
    <id>splunk</id>
    <name>splunk-releases</name>
    <url>http://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>
```
If you are using Ant, download the corresponding JAR file from http://dev.splunk.com/goto/sdk-slj.

Add the logging framework you plan to use. The three big ones in use today are Logback, Log4J 2.x, and java.util.logging (which comes with your JDK). If you are using Maven, add the corresponding dependencies below to your pom.xml:

Logback:

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.5</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-classic</artifactId>
    <version>1.0.13</version>
</dependency>
<dependency>
    <groupId>ch.qos.logback</groupId>
    <artifactId>logback-core</artifactId>
    <version>1.0.13</version>
</dependency>

Log4J 2.x:

<dependency>
    <groupId>org.apache.logging.log4j</groupId>
    <artifactId>log4j-core</artifactId>
    <version>2.0-beta9</version>
</dependency>

java.util.logging ships with the JDK.

Create a TCP input in Splunk that you will write to. To learn how, see Get data from TCP and UDP ports.

Configure your logging system. Here are simple example configurations for each of the three systems. The log4j2.xml and logback.xml files should be put somewhere in the classpath of your program. jdklogging.properties should be specified to your program by passing the following to the Java executable:

-Djava.util.logging.config.file=/path/to/jdklogging.properties

Logback (to be put in logback.xml on the classpath)

<configuration>
    <!--
    You should send data to Splunk using TCP inputs. You can find the 
    documentation on how to open TCP inputs on Splunk at http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports.

    Logback does not ship with a usable appender for TCP sockets (its 
    SocketAppender serializes Java objects for deserialization by a 
    server elsewhere). Instead, use the TcpAppender provided with this 
    library.

    This example assumes that you have Splunk running on your local 
    machine (127.0.0.1) with a TCP input configured on port 15000. 
    Note that TCP inputs are *not* the same as Splunk's management 
    port.

    You can control the format of what is logged by changing the 
    encoder (see http://logback.qos.ch/manual/layouts.html#ClassicPatternLayout 
    for details), but the pattern below produces a simple timestamp, 
    followed by the full message and a newline, like the following:

        2012-04-26 14:54:38,461 [%thread] %level text of my event
    -->
    
    <appender name="socket" class="com.splunk.logging.TcpAppender">
        <RemoteHost>127.0.0.1</RemoteHost>
        <Port>15000</Port>
        <layout class="ch.qos.logback.classic.PatternLayout">
        <pattern>%date{ISO8601} [%thread] %level: %msg%n</pattern>
        </layout>
    </appender>

    <logger name="splunk.logger" additivity="false" level="INFO">
        <appender-ref ref="socket"/>
    </logger>

    <root level="INFO">
        <appender-ref ref="socket"/>
    </root>
</configuration>

Log4j 2.x (to be put in log4j2.xml on the classpath)

<Configuration status="info" name="example" packages="">
    <!-- Define an appender that writes to a TCP socket. We use Log4J's 
    SocketAppender, which is documented at https://logging.apache.org/log4j/2.x/manual/appenders.html#SocketAppender.

    You can find the documentation on how to open TCP inputs on Splunk 
    at http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports. 
    Note that TCP inputs are *not* the same as Splunk's management port.
    -->
    
    <Appenders>
        <Socket name="socket" host="127.0.0.1" port="15000">
        <PatternLayout pattern="%p: %m%n" charset="UTF-8"/>
        </Socket>
    </Appenders>
    <!-- Define a logger named 'splunk.logger' which writes to the socket appender we defined above. -->
    <Loggers>
        <Root level="INFO">
        </Root>
        <Logger name="splunk.logger" level="info">
        <AppenderRef ref="socket"/>
        </Logger>
    </Loggers>
</Configuration>

java.util.logging

# We will write to a Splunk TCP input using java.util.logging's 
# SocketHandler. This line sets it to be the default handler for 
# all loggers.
handlers = java.util.logging.SocketHandler
config =

# Set the default logging level for the root logger
.level = INFO

# Implicitly create a logger called 'splunk.logger', set its 
# level to INFO, and make it log using the SocketHandler.
splunk.logger.level = INFO
splunk.logger.handlers = java.util.logging.SocketHandler

# Configure the SocketHandler to write to TCP port localhost:15000. 
# Note that TCP inputs are *not* the same as Splunk's management 
# port. You can find the documentation about how to open TCP 
# inputs in Splunk at http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports.
#
# You can find the documentation on using a SocketHandler at http://docs.oracle.com/javase/7/docs/api/java/util/logging/SocketHandler.html.

java.util.logging.SocketHandler.level = INFO
java.util.logging.SocketHandler.host = localhost
java.util.logging.SocketHandler.port = 15000

# With Java 7, you can set the format of SimpleFormatter. On Java 6, 
# you cannot and you will probably want to write a custom formatter 
# for your system. The syntax of the format string is given at 
# http://docs.oracle.com/javase/7/docs/api/java/util/Formatter.html#syntax 
# and http://docs.oracle.com/javase/7/docs/api/java/util/logging/SimpleFormatter.html
# for logging specific behavior.
java.util.logging.SocketHandler.formatter = SimpleFormatter
java.util.logging.SimpleFormatter.format = "%1$F %1$r %4$s: %6$s%n"

Import com.splunk.logging.SplunkCimLogEvent and use it to create events. This example code uses Logback as the logger, but the part relevant to SplunkCimLogEvent will be unchanged for other frameworks:

 logger.info(new SplunkCimLogEvent("Event name", "event-id") {{
     // You can add an arbitrary key=value pair with addField.
     addField("name", "value");

     // If you are logging exceptions, use addThrowable, which
     // does nice formatting. If ex is an exception you have caught
     // you would log it with
     addThrowableWithStacktrace(ex);

     // SplunkCimLogEvent provides lots of convenience methods for
     // fields defined by Splunk's Common Information Model. See
     // the SplunkCimLogEvent JavaDoc for a complete list.
     setAuthAction("deny");
 }});

Using logback-access with HEC appender

logback-access logs different type of events (ch.qos.logback.access.spi.IAccessEvent) as logback classic, which logs ch.qos.logback.classic.spi.ILoggingEvent.

To use this library with logback-access, you can try the following configuration:

logback access (to be put in logback-access.xml on the classpath)

 <configuration>
     <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
         <encoder>
             <pattern>combined</pattern>
         </encoder>
     </appender>
 
     <appender name="hec_access_appender" class="com.splunk.logging.HttpEventCollectorLogbackAppender">
         <url>https://localhost:8088</url>
         <token>00000000-0000-0000-0000-000000000000</token>
         <host>devhost</host>
         <index>main</index>
         <source>logback-client</source>
         <sourcetype>logback</sourcetype>
         <disableCertificateValidation>true</disableCertificateValidation>
 
         <layout class="ch.qos.logback.access.PatternLayout">
             <pattern>%h %l %u %t %r %s %b</pattern>
         </layout>
     </appender>
 
     <appender-ref ref="CONSOLE" />
     <appender-ref ref="hec_access_appender" />
 </configuration>

If you run into any issue, try add a debug=true attribute to the configuration for debugging.

Splunk Enterprise

If you haven't already installed Splunk Enterprise, download it here: http://www.splunk.com/download. For more about installing and running Splunk Enterprise and system requirements, see Splunk Enterprise Installation Manual.

Resources

Documentation for this library

http://dev.splunk.com/goto/sdk-slj

Splunk Common Information Model

http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/UnderstandandusetheCommonInformationModel

Splunk Best Practice Logging Semantics

http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6

Introduction to the Splunk product and some of its capabilities

http://docs.splunk.com/Documentation/Splunk/latest/User/SplunkOverview

Contributions

Get the Splunk Java Logging Framework from GitHub and clone the resources to your computer. For example, use the following command:

git clone https://github.com/splunk/splunk-library-javalogging.git

If you want to make a code contribution, go to the Open Source page for more information.

Support

The Splunk logging library for Java is community-supported.

You can find help through our community on Splunk Answers (use the logging-library-java tag to identify your questions).
File issues on GitHub.

Contact us

You can reach the Dev Platform team at devinfo@splunk.com.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Splunk Logging for Java

Version 1.6.3

Advice

Splunk Universal Forwarder vs Splunk TCP Inputs

Resilience

Data Cloning

Load Balancing

Thread Safety

Sending events to HTTP Event Collector

Message Format

License

Using Splunk Logging for Java

Using logback-access with HEC appender

Splunk Enterprise

Resources

Contributions

Support

Contact us

Files

README.md

Latest commit

History

README.md

File metadata and controls

Splunk Logging for Java

Version 1.6.3

Advice

Splunk Universal Forwarder vs Splunk TCP Inputs

Resilience

Data Cloning

Load Balancing

Thread Safety

Sending events to HTTP Event Collector

Message Format

License

Using Splunk Logging for Java

Using logback-access with HEC appender

Splunk Enterprise

Resources

Contributions

Support

Contact us