sshd.conf empty username in "Invalid user" regexp leads to ipdns warning

[wienfuchs]

sshd logging with empty username leads the regexp to catch the port instead of the ip address and finally results in ipdns warning.

Environment:

• Fail2Ban version: 0.11.1 (fail2ban/focal 0.11.1-1 all)
• OS: Ubuntu 20.04 LTS
• Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)
  (none, exept jail.local config below)

The issue:

sshd logging with empty username leads the regexp to catch the port instead of the ip address and finally results in ipdns warning.

Steps to reproduce

syslog example line:

Jun  6 04:17:04 myshot sshd[1189074]: Invalid user  from 65.49.20.68 port 34916
Jun  6 04:17:09 myshot sshd[1189074]: Connection closed by invalid user  65.49.20.68 port 34916 [preauth]

fail2ban result:

2020-06-06 04:17:09,123 fail2ban.ipdns          [11148]: WARNING Determined IP using DNS Lookup: 34916 = {'0.0.136.100'}

Expected behavior

determining 65.49.20.68 as IP instead of 34916

Observed behavior

port 34916 is catched as determined IP

Any additional information

Configuration, dump and another helpful excerpts

Any customizations done to /etc/fail2ban/ configuration

[sshd]
enabled = true
mode = aggressive
port = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Relevant parts of /var/log/fail2ban.log file:

2020-06-06 04:17:09,123 fail2ban.ipdns          [11148]: WARNING Determined IP using DNS Lookup: 34916 = {'0.0.136.100'}

Relevant lines from monitored log files in question:

Jun  6 04:17:04 myshot sshd[1189074]: Invalid user  from 65.49.20.68 port 34916
Jun  6 04:17:09 myshot sshd[1189074]: Connection closed by invalid user  65.49.20.68 port 34916 [preauth]

[sebres]

I did not follow something here - such RE matching Invalid user at begin of log-line after prefix does not exists in our sshd-filter.
So the example line will not found at all by our default filter. PoC:

$ fail2ban-regex -rvv 'Jun 6 04:17:04 myshot sshd[1189074]: Invalid user from 65.49.20.68 port 34916' 'sshd[mode=aggressive]'
...
Lines: 1 lines, 0 ignored, 0 matched, 1 missed
|- Missed line(s):
|  Jun 6 04:17:04 myshot sshd[1189074]: Invalid user from 65.49.20.68 port 34916
`-

If it is caused by this line, you should provide the regexp matching that.
And if it is your own RE you must fix it (make it more precise, with anchors and no catch-alls, use <ADDR> instead <HOST>.
If it's rather some other line - can you provide whole log excerpt (all messages for sshd[1189074])?

As an interim solution (since I didn't see ever that sshd logging a hostname normally), you can disable this with usedns = no which switches (this will additionally make the filter a bit faster).
But better would be to fix the regex.

[wienfuchs]

The match comes from attached filter.d sshd.conf supplied with Ubuntu 20 fail2ban package in original, not modified at all. I don't really know, which of the lines match this, but this is the file where the match relates to.

What I can see - somehow github made "user from" with two blanks to show as "user from" with only one blank in original post above.

That's why the regex does not match, excerpt original comment and lines with proper blanks is attached.

sshd.conf.txt
lines_with_blanks.txt

[sebres]

The match comes from attached filter.d sshd.conf supplied with Ubuntu 20 fail2ban package in original, not modified at all.

You are right (sorry overlooked that):

^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>%(__suff)s$

What I can see - somehow github made "user from" with two blanks to show as "user from" with only one blank in original post above.

You have to use markdown formatting to enclose the logs (I modified the original issue).

And now it finds the match, but it works as expected:

$ fail2ban-regex -rvv 'Jun  6 04:17:04 myshot sshd[1189074]: Invalid user  from 65.49.20.68 port 34916' 'sshd[mode=aggressive]'

Running tests
=============

Use   failregex filter file : sshd, basedir: /etc/fail2ban/
Use   filter options : {'mode': 'aggressive'}
Real  filter options : {'logtype': 'file', 'mode': 'aggressive', 'datepattern': '{^LN-BEG}'}
Use         maxlines : 1
Use      datepattern : Default Detectors
Use      single line : Jun  6 04:17:04 myshot sshd[1189074]: Invalid user...


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
...
|   6) [1] ^[iI](?:llegal|nvalid) user <F-USER>.*?</F-USER> from <HOST>(?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*$
|      65.49.20.68  Sat Jun 06 04:17:04 2020
...
`-
...
Lines: 1 lines, 0 ignored, 1 matched, 0 missed

$ fail2ban-regex -o row 'Jun  6 04:17:04 myshot sshd[1189074]: Invalid user  from 65.49.20.68 port 34916' 'sshd[mode=aggressive]'
['65.49.20.68', 1591409824.0,   {'user': '', 'ip6': None, 'mlfid': ' myshot sshd[1189074]: ', 'dns': None, 'ip4': '65.49.20.68'}],

Note that this regex is anchored from left and right, so it does no matter what is the part of user before from is exactly and the non-greedy catch-all would match empty here.
And I don't see in whole RE what exactly can cause that 34916 can be matched as dns:

^[iI](?:llegal|nvalid) user (?P<user>.*?) from (?:\[?(?:(?:::f{4,6}:)?(?P<ip4>(?:\d{1,3}\.){3}\d{1,3})|(?P<ip6>(?:[0-9a-fA-F]{1,4}::?|::){1,7}(?:[0-9a-fA-F]{1,4}|(?<=:):)))\]?|(?P<dns>[\w\-.^_]*\w))(?: (?:port \d+|on \S+|\[preauth\])){0,3}\s*$

I still guess it is the other log-line, so I repeat - please provide "whole log excerpt (all messages for sshd[1189074])". Or even grep it by 34916 (or some timestamp).
Or try fail2ban-regex like I provided above unless you'll see dns match, for instance:

fail2ban-regex -o row systemd-journal 'sshd[mode=aggressive]' | grep "'dns': '"

* replace systemd-journal with log file name (if it is a log either).

Also to be sure the filter has really unmodified state (no locals), provide output of fail2ban-client -d | grep sshd.

[wienfuchs]

Thank you for the formatting hint - attached are the grep'ed lines from the two logs (auth.log and fail2ban.log)

auth.log.grepped.txt
fail2ban.log.grepped.txt

fail2ban-regex -o row auth.log.grepped.txt 'sshd[mode=aggressive]' | grep "'dns': '"
gives
['0.0.136.100', 1591409829.0, {'mlfid': ' myshot sshd[1189074]: ', 'user': ' 65.49.20.68 port', 'ip4': None, 'ip6': None, 'dns': '34916', 'users': {' 65.49.20.68 port'}, 'mlfforget': 'Connection closed'}],

So yes, you're right, the "closed" line triggers this, I guessed it wrong. Edited initial post accordingly.

[sebres]

OK, thx!
The fix is simple (.+? -> .*?):

-__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.+?</F-USER>)?
+__authng_user = (?: (?:invalid|authenticating) user <F-USER>\S+|.*?</F-USER>)?

I'll also fix current sshd-filter soon.