feature: add logrotate for librenms log files #339

charlyforot · 2023-03-03T14:54:26Z

LibreNMS log files can become very large: #310

The goal of this PR is to configure and activate logrotate with the environment variables provided in the README.

Working great from my side with LOGROTATE_ENABLED=true after have built the docker image.

The logrotate configuration file named librenms is correctly created.

librenms-dispatcher:/opt/librenms# cat /etc/logrotate.d/librenms 
/opt/librenms/logs/*.log {
	su librenms librenms
	create 664 librenms librenms
	weekly
	rotate 6
	compress
	delaycompress
	missingok
	notifempty
}

We can see that the crons runs.

librenms-dispatcher:/opt/librenms/logs# crontab -l
# do daily/weekly/monthly maintenance
# min	hour	day	month	weekday	command
*/15	*	*	*	*	run-parts /etc/periodic/15min
0	*	*	*	*	run-parts /etc/periodic/hourly
0	2	*	*	*	run-parts /etc/periodic/daily
0	3	*	*	6	run-parts /etc/periodic/weekly
0	5	1	*	*	run-parts /etc/periodic/monthly

logrotate cron will be called every day.

librenms-dispatcher:/opt/librenms/logs# find /etc/periodic/ -name logrotate
/etc/periodic/daily/logrotate

We can try to force the logrotate to test this feature :

librenms-dispatcher:/opt/librenms/logs# logrotate /etc/logrotate.conf --force

librenms-dispatcher:/opt/librenms/logs# ls -lah
total 12K
drwxrwxr-x  2 librenms librenms 4.0K Mar  3 15:40 .
drwxr-xr-x 10 librenms librenms 4.0K Mar  3 14:25 ..
-rw-rw-r--  1 librenms librenms    0 Mar  3 15:40 librenms.log
-rw-rw-r--  1 librenms librenms    5 Mar  3 15:16 librenms.log-20230303

All done.

murrant · 2023-04-07T01:09:43Z

FYI, I don't think this makes any sense to be user configurable. Just enable it with sensible defaults.

… into addLogrotate

charlyforot · 2023-05-01T08:22:43Z

@murrant Following your advice, I have removed config variables. I just kept the env variable to enable/disable logrotate.

README.md

rootfs/etc/cont-init.d/09-config-logrotate.sh

crazy-max · 2023-05-10T02:20:50Z

rootfs/etc/cont-init.d/09-config-logrotate.sh

+# shellcheck shell=bash
+set -e
+
+if [ ${LOGROTATE_ENABLED:-false} = true ]


We should only set logrotate for the main service and not sidecar containers:

Suggested change

if [ ${LOGROTATE_ENABLED:-false} = true ]

if [ "$SIDECAR_DISPATCHER" = "1" ] || [ "$SIDECAR_SYSLOGNG" = "1" ] || [ "$SIDECAR_SNMPTRAPD" = "1" ]; then

exit 0

fi

if [ ${LOGROTATE_ENABLED:-false} = true ]

I'm not sure for this because if we consider a main service on a host using a sidecar dispatcher on a remote host, the dispatcher will generate a lot of logs contrary to the main service.

I have tested this case, I have installed only a sidecar dispatcher on a remote host and it generates a lot of logs locally.

Do you agree that we have to keep the logrotate configuration on the sidecar_dispatcher ?

rootfs/etc/cont-init.d/09-config-logrotate.sh

JamesMenetrey · 2023-10-23T09:36:46Z

Hello @charlyforot,

Thanks for this PR; it seems much needed. Do you still plan to fix the remaining elements, as highlighted by @crazy-max?

Cheers,
Jämes

charlyforot · 2023-10-30T13:33:53Z

Hello @charlyforot,

Thanks for this PR; it seems much needed. Do you still plan to fix the remaining elements, as highlighted by @crazy-max?

Cheers, Jämes

Hello @JamesMenetrey

Sorry for the response time, I'm going to fix this ASAP

Cheers

Co-authored-by: CrazyMax <github@crazymax.dev>

murrant · 2023-10-30T14:26:05Z

This makes me wonder. Isn't it common to log to stdout for docker containers? Shouldn't we be logging there instead of to a file?

JamesMenetrey · 2024-01-09T09:19:16Z

@murrant Admittedly, logging into stdout is more canonical for logging using Docker, but I have to say I would be happy to have a log rotate option rather than nothing :)

charlyforot · 2024-03-14T12:50:18Z

This makes me wonder. Isn't it common to log to stdout for docker containers? Shouldn't we be logging there instead of to a file?

Yes, it's right !

I tried to redirect some logs from librenms.log to stdout, however it's not easy because there are differents ways to log into LibreNMS code :

In python files with logger
In PHP files with d-echo(), echo(), Log::channel()->alerts, logfile()

I can't redirect some PHP logs to stdout because of code design, if someone has an idea ?

Otherwise, we could disable some logs which are duplicated between librenms.log and stdout container output by adding these lines into rootfs/etc/cont-init.d/03-config.sh :

# Config : Disable logs to file already present in stdout
sed -i 's/logfile(/\/\/logfile(/g' ${LIBRENMS_PATH}/poll-billing.php
sed -i 's/logfile(/\/\/logfile(/g' ${LIBRENMS_PATH}/discovery.php
sed -i "s/Log::channel('single')/Log::channel('stdout')/g" ${LIBRENMS_PATH}/LibreNMS/Poller.php

This would remove the large part of the annoying logs from librenms.log.

Other logs in librenms.log come from poll-billing.php and discovery.php

charlyforot added 4 commits March 3, 2023 15:24

add logrotate for librenms log files

63545dc

add empty line at the end of the bash scrit for configure logrotate

921820c

update README.md

fe1f04f

update README.md

62e99c1

charlyforot requested a review from crazy-max as a code owner March 3, 2023 14:54

Merge branch 'master' into addLogrotate

1e622c2

charlyforot added 2 commits May 1, 2023 10:19

remove config variable for logrotate

a24b3c6

Merge branch 'addLogrotate' of github.com:charlyforot/librenms-docker…

311eff5

… into addLogrotate

crazy-max requested changes May 10, 2023

View reviewed changes

README.md Outdated Show resolved Hide resolved

rootfs/etc/cont-init.d/09-config-logrotate.sh Outdated Show resolved Hide resolved

crazy-max reviewed May 10, 2023

View reviewed changes

rootfs/etc/cont-init.d/09-config-logrotate.sh Outdated Show resolved Hide resolved

crazy-max linked an issue Jul 22, 2023 that may be closed by this pull request

Enable LOGROTATE #310

Open

Update README.md

3bfc1ee

Co-authored-by: CrazyMax <github@crazymax.dev>

charlyforot added 2 commits October 31, 2023 10:19

include logrotate config into 03-config.sh

ec23e8b

add empty end line into 03-config.sh

6534c6b

charlyforot requested a review from crazy-max October 31, 2023 09:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: add logrotate for librenms log files #339

feature: add logrotate for librenms log files #339

charlyforot commented Mar 3, 2023 •

edited

murrant commented Apr 7, 2023

charlyforot commented May 1, 2023 •

edited

crazy-max May 10, 2023

charlyforot Oct 30, 2023

JamesMenetrey commented Oct 23, 2023

charlyforot commented Oct 30, 2023

murrant commented Oct 30, 2023

JamesMenetrey commented Jan 9, 2024

charlyforot commented Mar 14, 2024

feature: add logrotate for librenms log files #339

Are you sure you want to change the base?

feature: add logrotate for librenms log files #339

Conversation

charlyforot commented Mar 3, 2023 • edited

murrant commented Apr 7, 2023

charlyforot commented May 1, 2023 • edited

crazy-max May 10, 2023

Choose a reason for hiding this comment

charlyforot Oct 30, 2023

Choose a reason for hiding this comment

JamesMenetrey commented Oct 23, 2023

charlyforot commented Oct 30, 2023

murrant commented Oct 30, 2023

JamesMenetrey commented Jan 9, 2024

charlyforot commented Mar 14, 2024

charlyforot commented Mar 3, 2023 •

edited

charlyforot commented May 1, 2023 •

edited