Configuring Log Rotation in the DataSift Connector

Depending on your usage of the Connector, you may find a significant amount of space taken up by various log files. This guide explains where some of them are located, and how to reduce your log retention times to save disk space.

Most of the logs used by the Connector use LOG4J; take a look at the Apache LOG4J docs for the default configuration documentation.

The most significant log configuration files can be found in the following locations;

• /etc/datasift/*/log4j.properties
• /opt/kafka/config/log4j.properties
• /etc/grafana/grafana-datasift.ini

In general, you can edit your log4j.appender.file.MaxFileSize and log4j.appender.file.MaxBackupIndex config to change the max size of each log file, and the number of old copies we keep (right now the default is up to 5 250MB log files for each service).

If you push a large amount of data through the Connector, you might see your Kafka logs getting very large. By default we store interactions in Kafka for 7 days; this can be reduced to save disk space by editing your Kafka data log retention. For more information on this, take a look at our Limiting or extending Kafka messaging Wiki page.