Option not exposed for systemd namespaces

[mvastola]

Environment:

• Fail2Ban version (including any possible distribution suffixes): 0.11.1
  • Consulted code in HEAD and confirmed issue not resolved.
  • Attempted with both the OS default python3-systemd package and the HEAD from its' git repo.
• OS, including release name/version: Ubuntu 20.04.1 LTS
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

This is an extension of #2793, but I am unable to use systemd namespaces as suggested in that issue. Namely, adding _NAMESPACE=whatever to the filter does not work.

Steps to reproduce

Best I can tell, this is an issue with systemd-python. In narrowing down the issue after appending to journalmatch didn't work, I ran the following:

from systemd import journal
j = journal.Reader()
j.this_boot()
j.add_match(_SYSTEMD_UNIT="ssh.service", _NAMESPACE='auth')
for entry in j:
    print(str(j))

It produced no output, whereas I definitely have hundreds of messages.

Apparently, this feature has only been available for a few months in a recent change to the systemd-python package.

Installing that latest code and then running:

from systemd import journal
j = journal.Reader(namespace='auth')
j.this_boot()
j.add_match(_SYSTEMD_UNIT="ssh.service")
for entry in j:
    print(str(j))

I can see it works perfectly.

Looking into this further, it seems the solution should be as simple as to add backend = systemd[namespace="auth"] to the config.

The only thing preventing that, however, seems to be that the current FilterSystemd._getJournalArgs method does not recognize/extract this argument. If it did so, in theory, everything would work.

Expected behavior

fail2ban would construct journal.Reader here with the namespace parameter passed on in the jrnlargs dict.

Observed behavior

namespace parameter was not extracted by FilterSystemd._getJournalArgs and thus not passed to the constructor for journal.Reader. Instead it was passed onto the superclass' (JournalFilter) constructor, generating the error:

2021-01-14 23:53:17,049 fail2ban                [3600053]: ERROR   NOK: ("__init__() got an unexpected keyword argument 'namespace'",)

Configuration, dump and another helpful excerpts

Any customizations done to /etc/fail2ban/ configuration

In filters.d/sshd-custom.conf:

[INCLUDES]
before = sshd.conf

[Definition]
logtype = journal
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

In jails.d/ssh.conf:

[sshd]
enabled = true
filter = sshd-custom
backend = systemd[namespace="auth"]

Relevant parts of /var/log/fail2ban.log file:

See above

Relevant lines from monitored log files in question:

N/A

[sebres]

incorrect (missed the point), saved for the record...

backend = systemd[namespace="auth"]

Well, this is not a parameter of systemd backend (see #1523 for possible options), but rather of its filter.
Shortly: arguments of backends are the options which is supplied to the journal constructor, parameter journalmatch of filter contains the expressions for match.

To add it in filter you have to do it like described in #2793 (comment) or even try this (without to overwrite it completely, but rather to extend current parameter journalmatch specified in sshd-filter), either in jail.local:

[sshd]
filter = sshd[journalmatch="<known/journalmatch> + _NAMESPACE=auth"]

or you could rewrite/extend it in filter.d/sshd.local:

[Definition]
journalmatch = <known/journalmatch> + _NAMESPACE=auth

so you don't really need your filters.d/sshd-custom.conf, in *.local file you can overwrite every value you may need.

[mvastola]

Well, this is not a parameter of systemd backend (see #1523 for possible options), but rather of its filter.
Shortly: arguments of backends are the options which is supplied to the journal constructor, parameter journalmatch of filter contains the expressions for match.

I know it's not a parameter to the backend, my point is rather that it needs to be.

The underlying python-systemd library doesn't work with a filter containing + _NAMESPACE=whatever is used. (I haven't looked into why this is, but I demonstrated it up above.) Instead, specifying the namespace requires namespace to be given to the journal.Reader constructor.

The obvious (and extremely simple) way to get this done is to add a couple of lines to the FilterSystemd._getJournalArgs method to extract the option for the journal constructor args.

[sebres]

Sorry, misunderstood you.

The obvious (and extremely simple) way to get this done is to add a couple of lines to the FilterSystemd._getJournalArgs method to extract the option for the journal constructor args.

Sure, and since the implementation will be backwards compatible, it is simple enough...

[sebres]

Should be fixed in 164105f now for 0.10 branch.
I'll merge it later in 0.11 and 1.0.
Thx!
Don't hesitate to ping me if I missed something yet.

[mvastola]

That was super-fast. Thanks! (Will let you know if there are issues, but it's such a simple fix I can't imagine how.)

[mvastola]

@sebres , actually just thought of things you might want to add (but I leave it totally up to you):

• An informative error message if the version of python-systemd is too old to support the namespace argument and it is given.
• A warning if _NAMESPACE is present in the journal match, because in looking at the python-systemd, it looks like the underlying issue is -- if a namespace is to be filtered -- the journal must be opened with the namespace given, meaning that _NAMESPACE can't ever work.
  • Possible alternative: Apparently, the call to sd_journal_open_namespace (which is made by pythond-systemd) could be given the flag SD_JOURNAL_ALL_NAMESPACES (via the flags argument) , which would then read journal entries namespaces, which would then theoretically make a _NAMESPACE filter possible. So you might choose to enable the SD_JOURNAL_ALL_NAMESPACES flag by default (or at least expose it as a clear option). This also would then make the filtermatch more intuitive (because namespace wouldn't have to be set differently) as well as allow for more complex filters that don't look in just a single namespace.

In any case, could you correct the label on this issue? It might make it easier to locate. (I know I would personally be more likely to skip over any issues tagged closed-as-incorrect if I was searching to see if anyone else had my problem.)

Thanks again!

[sebres]

So you might choose to enable the SD_JOURNAL_ALL_NAMESPACES flag by default (or at least expose it as a clear option).

Well, I would be careful with flags like that, especially by default (too often seen issues like "too many open files" errors, so for instance #2444 did it even more strict (I'm still not in mode to loosen that).

As for the docu, it must be extended yet, sure...