NAS-127333 / 24.10 / Add /var/log/syslog.1 and /var/log/messages.1 to ixdiagnose debug

[bmeagherix]

Experimented with adding Pattern('syslog.+') and Pattern('messages.+') instead, but just the Pattern('syslog.+') caused a HA debug to grow from ~11MB to ~30MB - and there'd be no reason to suppose it couldn't grow even more in time.

Just adding the .1 (as per this PR) meant ~11MB grew to ~14MB.

[bugclerk]

Jira URL: https://ixsystems.atlassian.net/browse/NAS-127333

[yocalebo]

What filled up syslog? This actually points out a bug that we should be using a pattern for the middleware files. These are text files so I expect a rather large compression ratio.

[bmeagherix]

What filled up syslog?

I was testing ALUA for weeks - large number of targets and repeated configuration changes & reboots. Probably worst case scenario wrt logs.

root@m30-137-c1[~]# ls -l /var/log/{syslog*,messages*,kern.log*,middlewared.log*}
-rw-r----- 1 root adm    382931 Feb 12 11:46 /var/log/kern.log
-rw-r----- 1 root adm  50950289 Feb 11 22:57 /var/log/kern.log.1
-rw-r----- 1 root adm   6524363 Feb  9 22:33 /var/log/kern.log.2.gz
-rw-r----- 1 root adm   1828637 Feb  6 17:59 /var/log/kern.log.3.gz
-rw-r----- 1 root adm    371846 Feb 12 11:46 /var/log/messages
-rw-r----- 1 root adm  50030108 Feb 12 00:00 /var/log/messages.1
-rw-r----- 1 root adm   6458696 Feb 10 00:00 /var/log/messages.2.gz
-rw-r----- 1 root adm   1831973 Feb  7 00:00 /var/log/messages.3.gz
-rw-r----- 1 root root  2562721 Feb 12 15:09 /var/log/middlewared.log
-rw-r----- 1 root root 10485685 Feb 11 18:58 /var/log/middlewared.log.1
-rw-r----- 1 root root 10485731 Feb  7 18:54 /var/log/middlewared.log.2
-rw-r----- 1 root root 10473978 Feb  5 09:23 /var/log/middlewared.log.3
-rw-r----- 1 root root 10484055 Feb  2 16:01 /var/log/middlewared.log.4
-rw-r----- 1 root adm    519422 Feb 12 15:20 /var/log/syslog
-rw-r----- 1 root adm  64547408 Feb 12 00:00 /var/log/syslog.1
-rw-r----- 1 root adm   9126204 Feb 10 00:00 /var/log/syslog.2.gz
-rw-r----- 1 root adm   1967502 Feb  7 00:00 /var/log/syslog.3.gz

and

root@m30-137-c2[~]# ls -l /var/log/{syslog*,messages*,kern.log*,middlewared.log*}
-rw-r----- 1 root adm    577361 Feb 12 14:38 /var/log/kern.log
-rw-r----- 1 root adm  28484689 Feb 11 22:57 /var/log/kern.log.1
-rw-r----- 1 root adm   4697225 Feb  9 22:33 /var/log/kern.log.2.gz
-rw-r----- 1 root adm   1434673 Feb  7 19:17 /var/log/kern.log.3.gz
-rw-r----- 1 root adm    565703 Feb 12 14:38 /var/log/messages
-rw-r----- 1 root adm  28005790 Feb 12 00:00 /var/log/messages.1
-rw-r----- 1 root adm   4676514 Feb 10 00:00 /var/log/messages.2.gz
-rw-r----- 1 root adm   1443621 Feb  8 00:00 /var/log/messages.3.gz
-rw-r----- 1 root root  8914222 Feb 12 14:39 /var/log/middlewared.log
-rw-r----- 1 root root 10485731 Feb  9 10:23 /var/log/middlewared.log.1
-rw-r----- 1 root root 10485633 Feb  6 09:08 /var/log/middlewared.log.2
-rw-r--r-- 1 root root 10484960 Feb  5 13:35 /var/log/middlewared.log.3
-rw-r----- 1 root root 10468177 Feb  5 13:32 /var/log/middlewared.log.4
-rw-r----- 1 root root 10472241 Feb  5 13:29 /var/log/middlewared.log.5
-rw-r----- 1 root adm    794091 Feb 12 15:20 /var/log/syslog
-rw-r----- 1 root adm  35913202 Feb 12 00:00 /var/log/syslog.1
-rw-r----- 1 root adm   6601909 Feb 10 00:00 /var/log/syslog.2.gz
-rw-r----- 1 root adm   2311122 Feb  8 00:00 /var/log/syslog.3.gz

Both scst and dlm are quite noisy, so we could choose to make them less so, but IMO it'd good to get a sense of how large these files get in the wild.

OTOH, not fetching at least the .1 could mean that if unlucky you get a debug like the one I received from Max today, with 3 lines of history in kern.log and 9 lines in messages - not enough to work from!

This actually points out a bug that we should be using a pattern for the middleware files. These are text files so I expect a rather large compression ratio.

We are already using a pattern wrt middlewared (and therefore getting all the middlewared log files in the debug):

        Pattern('middlewared.+'),

[yocalebo]

Thanks for clarification. I think we need to add a syslog filter for both of these subsystems and log them to their own files respectively and add them to the debug. We already do this for k3s and a couple others.