rsyslogd 8.2310.0-4.el9 throws imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-4.el9 try https://www.rsyslog.com/e/2433 ]

[captainfalcon23]

Expected behavior

No errors

Actual behavior

Upon upgrading rsyslog from a previous working version (rsyslog-8.2102.0-117.el9.x86_64.rpm) to the latest offered within Oracle 9 repos (just recently added) rsyslog-8.2310.0-4.el9.x86_64.rpm results in:

imjournal: open() failed for path: '/var/lib/rsyslog/imjournal.state.tmp': Operation not permitted [v8.2310.0-4.el9 try https://www.rsyslog.com/e/2433 ]

No idea what the real impact of this is either.

Steps to reproduce the behavior

Upgrade to rsyslog-8.2310.0 using yum/dnf.

Environment

• rsyslog version:
  BROKEN:

rsyslogd  8.2310.0-4.el9 (aka 2023.10) compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        Yes
        Config file:                            /etc/rsyslog.conf
        PID file:                               /var/run/rsyslogd.pid
        Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

WORKING:

rsyslogd  8.2102.0-117.el9 (aka 2021.02) compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        Yes
        Config file:                            /etc/rsyslog.conf
        PID file:                               /var/run/rsyslogd.pid
        Number of Bits in RainerScript integers: 64

See https://www.rsyslog.com for more information.

• platform: Oracle Linux 9
• for configuration questions/issues, include rsyslog.conf:

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# create mode of files - CIS Item 4.2.1.3
$FileCreateMode 0640

#Disablerate-limitingof log entries
$SystemLogRateLimitInterval 60
$SystemLogRateLimitBurst 1500000

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state

# changes to journal rate limits - see packer build
$imjournalRatelimitInterval 60
$imjournalRatelimitBurst 1500000

#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

[dmitrydonskih]

The same for me.
Linux machine 5.15.0-200.131.27.1.el9uek.x86_64 Oracle Linux 9.4.
Updated to rsyslog-8.2310.0-4.el9.x86_64

This is not due to missing /var/lib/rsyslog itself:

# ls -la /var/lib/rsyslog/
total 8
drwx------.  2 root root   28 apr 8 15:53 .
drwxr-xr-x. 47 root root 4096 may 13 06:58 ..
-rw-------   1 root root  127 may 13 07:09 imjournal.state

[trafford-tbe]

But it's complaining about .tmp file and only for first start after update. Subsequent starts don't display this.

[captainfalcon23]

https://aws.amazon.com/about-aws/whats-new/2024/05/application-load-balancer-ipv6-internet-clients/

Yes they do. Restarting the service, or even rebooting the host, causes the same issue.

[cleonn]

Same for me on a RHEL 9.4. Persist after restart of service and reboot of server.

[traylenator]

I think since this patch

• imjournal: Add FileCreateMode module parameter #5109

was backported

• https://issues.redhat.com/browse/RHEL-949
• https://bugzilla.redhat.com/show_bug.cgi?id=2181552

I am still confused exactly to what is going but I believe rsyslogd no longer supports the legacy configuration

$ModLoad imjournal
$FileCreateMode 0600

in particular the cratemode is ignored and default Umask from the unit is tried. 0640 and this is then denied due to:

RestrictSUIDSGID=yes

in the unit.

Fixes seem to be overriding RestrictSUIDSGID to no in the unit file which is probably a bad thing to do or
better switching to the non-legacy format from $ModLoad. i.e

#$ModLoad imjournal
module(load="imjournal"            
       UsePid="system" # PID nummbe
       FileCreateMode=0600
       StateFile="imjournal.state")

Note this is the default format of EL9 to use this method configuration.

Need to check this again some time.

[rgerhards]

this now sounds like a packaging bug. It would be good to report this to RH

@Cropi any idea?

[bdombrow]

Had the same problem on RH 9.4. Removed 8.2310 and installed 8.2102. Lost my configuration in the process and redid it, this time I left the default config and put overrides in conf.d. Forgot to pin the version and auto update moved it back to 8.2310 the next day and it's still working. Haven't had time to restore an image with the old configuration to see what the differences are.

[traylenator]

this now sounds like a packaging bug. It would be good to report this to RH

Well its a back port that introduces unexpected features. .. Given that $ModLoad stuff is all super legacy for a number of years
its probably just best to migrate away from it as I will. The EL9 default configuration does not use $ModLoad at all, its kind of lucky that it worked up to now.

[rgerhards]

Well, $ModLoad works, but the mask is a breaking change (as was discussed in the mentioned issue tracker). So depending on RH's default config, they need to change packaging. Just trying to get it straight.

In any case, it's a good idea to move forward to the advanced format. It's much clearer.

[davidelang]

On Sat, 18 May 2024, Steve Traylen wrote: I am still confused exactly to what is going but I believe rsyslogd no longer supports the legacy configuration

no, support has not been dropped.

``` $ModLoad imjournal $FileCreateMode 0600 ``` in particular the cratemode is ignored and default Umask from the unit is tried. 0640 and this is then denied due to:

but if you ar expecting the filecreatemode to affect the loading of imjournal, then it needs to be set before you do the modload. David Lang

…

``` RestrictSUIDSGID=yes ``` in the unit. Fixes seem to be overriding `RestrictSUIDSGID` to `no` in the unit file which is probably a bad thing to do or better switching to the non-legacy format from $ModLoad. i.e ``` #$ModLoad imjournal module(load="imjournal" UsePid="system" # PID nummbe FileCreateMode=0600 StateFile="imjournal.state") ``` Note this is the default format of EL9 to use this method configuration. Need to check this again some time.

[traylenator]

@davidelang just confirming that changing the order of:

$ModLoad imjournal
 $FileCreateMode 0600

to

$FileCreateMode 0600
$ModLoad imjournal

does indeed fix things after this patch 👍

[traylenator]

For info if anyone else has the misfortune of running EL7 still then the order must be kept as it is.

[rgerhards]

For info if anyone else has the misfortune of running EL7 still then the order must be kept as it is.

That sounds like a bug in the old version. I overlooked it, but in obsolete legacy style you always need to set params before they are used (one important reason we switched to the new style). Just FYI.