When containers are sending their logs to journald, how to set journalmatch to a string instead of a service?

[paulvirtuel]

I have several containers sending logs to journald with a prefix so I can differentiate them:
"container/postfix", "container/dovecot", "container/apache", ...

Here are examples of the log:

Mar 25 00:26:27 host.example.com container/postfix/smtp/smtpd[138665]: disconnect from unknown[IP] ehlo=1 auth=0/1 commands=1/2
Mar 25 17:34:06 host.example.com 29f2dee492d0[124699]: Mar 25 17:34:06 container/dovecot lmtp(postmaster@example.com)<1374><4KVNHY61AWZeBQAANZ3TdQ>: Info: msgid=<20240325013211.AA8CB3EEF@mail.example.com>: saved mail to INBOX

I am trying to get each jail to look only at their related logs by using its prefix string.

So far, I have tried to set journalmatch in my jail.local like this:

[apache-auth]
chain = DOCKER-USER
enabled = true
journalmatch = MESSAGE=container/apache
...
[postfix]
chain = DOCKER-USER
enabled = true
journalmatch = MESSAGE=container/postfix
...
[dovecot]
chain = DOCKER-USER
enabled = true
journalmatch = MESSAGE=container/dovecot

But I am not sure if it works.

I tried doing a manual test and the result is no lines are found from the log. Perhaps my command line is wrong as I could not find details about this in the documents I read. I tried those, also replaced the " with an asterisk *:

fail2ban-regex --journalmatch=MESSAGE="container/dovecot" systemd-journal[journalflags=1] dovecot
fail2ban-regex --journalmatch=MESSAGE=container/dovecot systemd-journal[journalflags=1] dovecot

Running tests
=============

Use   failregex filter file : dovecot, basedir: /etc/fail2ban
Use      datepattern : {^LN-BEG}TAI64N
{^LN-BEG} : Default Detectors
Use         systemd journal
Use         encoding : UTF-8
Use    journal match : MESSAGE=container/dovecot


Results
=======

Prefregex: 0 total
|  ^\s*(?:\S+\s+)?(?:(?:dovecot(?:-auth)?|auth)(?:\[\d+\])?:?\s+)?(?:kernel:\s?\[ *\d+\.\d+\]:?\s+)?(?:(?:dovecot: )?auth(?:-worker)?(?:\([^\)]+\))?: )?(?:pam_unix(?:\(dovecot:auth\))?: |(?:pop3|imap|managesieve|submission)-login: )?(?:Info: )?(?:conn \w+:auth(?:-worker)? \([^\)]+\): auth(?:-worker)?<\d+>: )?(?P<content>.+)$
`-

Failregex: 0 total

Ignoreregex: 0 total

Date template hits:

Lines: 0 lines, 0 ignored, 0 matched, 0 missed
[processed in 0.03 sec]

1- Any help on how to properly set journalmatch or something that would work with the journald logs?

2- Are there any tools or tips to test that all jails are working since there are so many.

[sebres]

1. This doesn't have something with the MESSAGE - what you're seeing with container/postfix/smtp/smtpd[...] is called UNIT in journal fields. To check what exactly you can use, you need to inspect the output of journalctl -o json-pretty and then use it with journalmatch.
2. Your approach can produce another issue with some filters, e. g. if the filter expect smtp/smtpd[...] but got container/postfix/smtp/smtpd[...] it would not match.
   So you have to filter by filter they still work, or to rewrite parts of them or overwrite certain parameters like _daemon, like here:
   

   fail2ban/config/filter.d/postfix.conf

   Line 13 in b59fd2e

   _daemon = postfix(-\w+)?/[^/\[:\s]+(?:/smtp[ds])?

[paulvirtuel]

I was able to fix the journalmatch based on your guidance.

I found the fields for the journalmatch after some investigation using:

   journalctl -o json-pretty

Another search that helped is the grep option in journalctl:

   journalctl --since "1day ago" -g container*

Then I updated my jail.local:

[postfix]
chain = DOCKER-USER
enabled = true
journalmatch = SYSLOG_IDENTIFIER="container/postfix/smtp/smtpd" SYSLOG_IDENTIFIER="container/postfix/smtp"
mode = aggressive

[postfix-rbl]
chain = DOCKER-USER
enabled = true
journalmatch = SYSLOG_IDENTIFIER="container/postfix/smtpd"
filter = postfix[mode=rbl]

[postfix-sasl]
chain = DOCKER-USER
enabled = true
journalmatch = SYSLOG_IDENTIFIER="container/postfix/submission/smtpd"

[dovecot]
chain = DOCKER-USER
enabled = true
journalmatch = CONTAINER_NAME="container_dovecot"
filter = dovecot[mode=aggressive]

I also added container/ in front of the _daemon and it seems to work:

   # _daemon = postfix...
   _daemon = container\/postfix...

After all this, the command fail2ban-regex is really useful to debug!

   fail2ban-regex --journalmatch='SYSLOG_IDENTIFIER="container/postfix/smtp/smtpd" SYSLOG_IDENTIFIER="container/postfix/smtp"' systemd-journal[journalflags=1] --print-all-matched postfix

I was not able to use the default postfix-sasl.conf, I had to change the failregex to find matches:

     #failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed(:[ A-Za-z0-9+/:]*={0,2})?\s*$
      failregex = ^%(__prefix_line)swarning: [-._\w]+\[<HOST>\]: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed:.*$

After the change, it worked for that kind of log:

fail2ban-regex --journalmatch='SYSLOG_IDENTIFIER="container/postfix/submission/smtpd"' systemd-journal[journalflags=1] --print-all-matched postfix-sasl
   |- Matched line(s):
   |  2024-03-26T16:23:02.038706+00:00 host.example.com container/postfix/submission/smtpd[115]: warning: unknown[<IP>]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=invalid

Thanks for your help!

[sebres]

You're welcome.
Just by the way: you don't need .*$ (just remove it) - the RE is the same as without this, and means simply there is no anchor at end.
Also newest version of filter isn't anchored here anymore:

fail2ban/config/filter.d/postfix.conf

Lines 27 to 28 in b59fd2e

	mdre-auth = ^[^[]*\[<HOST>\]%(_port)s: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed:(?! Connection lost to authentication server| Invalid authentication mechanism)
	mdre-auth2= ^[^[]*\[<HOST>\]%(_port)s: SASL ((?i)LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed:(?! Connection lost to authentication server)

(but it'd additionally ignore messages with suffix like "Connection lost to authentication server", as no auth-failures)

[paulvirtuel]

Thanks for the suggestions.

After a couple days of testing my config, I realized that the nftables rule for DOCKER-USER were not preventing banned IPs from reaching my containers. I had warnings already banned messages all the time.

When looking at the nftables rule that docker creates, I saw that the DOCKER-USER chain uses the forward hook instead of the input hook. I think because docker uses nat, it is ignoring the chains that are on the input hook.

So, I created this file in /etc/fail2ban/action.d/nftables-docker.conf to specify the forward hook:

[INCLUDES]
before = nftables.conf

[Init]
chain_hook = forward

I also updated my /etc/fail2ban/jail.local to use it:

[postfix]
...
chain = DOCKER-USER
banaction = nftables-docker

And now the IP ban works for the docker containers.

Not sure if it would be useful on the wiki, but I found this command that shows the status of all jails:
fail2ban-client status | sed -n 's/,//g;s/.*Jail list://p' | xargs -n1 fail2ban-client status

[sebres]

You don't need a new action for this, it is possible to specify all init-parameters directly in banaction definition:

banaction = nftables[chain_hook=forward]

Exactly how one does with all others:

fail2ban/config/jail.conf

Line 212 in b59fd2e

action_ = %(banaction)s[port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]