logs2kafka

Simple tool to listen for incoming log messages and send them further to Kafka (version 0.9 or greater is required).

The tool is designed to produce JSON based log messages which have at least the following fields:

{
  "ts" : "2016-06-03T11:55:55.391Z",
  "service" : "my-unique-service-name",
  "host" : "hostname.company.com"
}

In addition if a SERVER_IP env variable is provided a "server_ip" field is added with this value.

Each message is sent to a kafka cluster (KAFKA_CONNECTION_STRING env variable, eg. "localhost:9042") into a topic (TOPIC env variable, default is "logs"), into a random partition inside this topic. If the partition is not available then another partition is picked up by random.

A single message can have any additional fields and the relay simply passes these thru without modification.

Other suggested fields:

• "level" being ["DEBUG", "INFO", "WARN", "ERROR"]
• "msg" being a human readable log message

Sending logs to this relay

There are a few different ways to send log messages to logs2kafka:

• graylog over udp (json based format)
• json or plain-old-log-lines over tcp (to-be-done)
• syslog

Graylog format is the preferred way to send messages. Graylog is a json based format, which is automatically converted to match the logs2kafka format: there are a few properties which are renamed and coverted from graylog format. This format also supports long messages where more than one udp packet is required for the transmission.

Docker daemon supports natively the Graylog format, so info such as docker image name, container id is handled correctly.

Also Docker labels are transferred correctly, so labels defined in Kubernetes pod manifests can be transferred to the logging system.

Following convertions are done to the logging format:

• Logs2kafka will add an ISO-8601 formatted timestamp to "ts" field
• If the message log "level" property is an integer then it is converted to appropriate string ["DEBUG", "INFO", "WARN", "ERROR", "UNKNOWN"] (integer is assumed to have the syslog level numberingm)
• Graylof field "_image_name" is converted into "docker_image"
• Graylog field "short_message" is renamed to "msg"
• Graylog field "timestamp" is dropped in favor of the autogenerated "ts" field.
• Graylog field "_container_name" is renamed to "container_name"
• Machine hostname is added into the "host" field if either of them are missing.

The service will refuse to forward a message if it doesn't have "service" field set.

Statsd metrics

Service will also send some statsd metrics into a statsd server (STATSD_HOST and STATSD_PORT env variables):

• If a message contains "level" attribute, then a counter with name service + ".app.log." + level is emitted.

• logs2kafka.sendFailed is incremented in all Kafka errors/warnings.

• logs2kafka.retrying is incremented each time a message is being tried into another partition (so first try does not count).

• logs2kafka.couldNotSend is incremented if message failed completely and was lost.

Local machine logs and tailing

As logs2kafka can store local copy of the logs into the machine with log rotation, these logs can be viewed and tailed with the logs2kafka command. Type "logs2kafka tail " to start tailing.