Log Courier is a tool created to ship log files speedily and securely to remote Logstash instances for processing whilst using small amounts of local resources. The project is an enhanced fork of Logstash Forwarder 0.3.1 with many fixes and behavioural improvements.
Table of Contents generated with DocToc
Log Courier implements the following features:
- Follow active log files
- Follow rotations
- Follow standard input stream
- Suspend tailing after periods of inactivity
- Set extra fields, supporting hashes and arrays
(
tags: ['one','two']) - Reload configuration without restarting
- Secure TLS shipping transport with server certificate verification
- TLS client certificate verification
- Secure CurveZMQ shipping transport to load balance across multiple Logstash instances (optional, requires ZeroMQ 4+)
- Plaintext TCP shipping transport for configuration simplicity in local networks
- Plaintext ZMQ shipping transport
- Administration utility to monitor the shipping speed and status
- Multiline codec
- Filter codec
- Logstash Integration with an input and output plugin
*nix: Most requirements can usually be installed by your favourite package
manager.
OS X: Git and GNU make are provided automatically by XCode.
Windows: GNU make for Windows can be found
here.
To build, simply run make as follows.
git clone https://github.com/driskell/log-courier
cd log-courier
make
The log-courier program can then be found in the 'bin' folder.
Note: If you receive errors whilst running make, try gmake instead.
Log Courier does not utilise the lumberjack Logstash plugin and instead uses its own custom plugin. This allows significant enhancements to the integration far beyond the lumberjack protocol allows.
Install using the Logstash 1.5+ Plugin manager.
cd /path/to/logstash
bin/logstash plugin install logstash-input-log-courier
Detailed instructions, including integration with Logstash 1.4.x, can be found on the Logstash Integration page.
To use the 'plainzmq' and 'zmq' transports, you will need to install ZeroMQ (>=3.2 for cleartext 'plainzmq', >=4.0 for encrypted 'zmq').
*nix: ZeroMQ >=3.2 is usually available via the package manager. ZeroMQ >=4.0
may need to be built and installed manually.
OS X: ZeroMQ can be installed via Homebrew.
Windows: ZeroMQ will need to be built and installed manually.
Once the required version of ZeroMQ is installed, run the corresponding make
command to build Log Courier with the ZMQ transports.
# ZeroMQ >=3.2 - cleartext 'plainzmq' transport
make with=zmq3
# ZeroMQ >=4.0 - both cleartext 'plainzmq' and encrypted 'zmq' transport
make with=zmq4
Note: If you receive errors whilst running make, try gmake instead.
Please ensure that the versions of ZeroMQ installed on the Logstash hosts and the Log Courier hosts are of the same major version. A Log Courier host that has ZeroMQ 4.0.5 will not work with a Logstash host using ZeroMQ 3.2.4 (but will work with a Logstash host using ZeroMQ 4.0.4.)
Running make selfsigned will automatically build and run the lc-tlscert
utility that can quickly and easily generate a self-signed certificate for the
TLS shipping transport.
Likewise, running make curvekey will automatically build and run the
lc-curvekey utility that can quickly and easily generate CurveZMQ key pairs
for the CurveZMQ shipping transport. This tool is only available when Log
Courier is built with ZeroMQ >=4.0.
Both tools also generate the required configuration file snippets.