Date pattern addition in filter file

[jjoshi-space]

Environment:

Fill out and check ([x]) the boxes which apply. If your Fail2Ban version is outdated,
and you can't verify that the issue persists in the recent release, better seek support
from the distribution you obtained Fail2Ban from

• Fail2Ban version (including any possible distribution suffixes): 0.9-6
• OS, including release name/version: Centos 06
• [x ] Fail2Ban installed via OS/distribution mechanisms
• You have not applied any additional foreign patches to the codebase
• Some customizations were done to the configuration (provide details below is so)

The issue:

datepattern for a tomcat log is not according to the standard of any dates
Hence, the filter catches the required string from the log but unable to look for the date time
Summary here
The date pattern mentioned in the tomcat logs is: (/var/log/tomcat2/catalina.out)

**[06-Jun-2018_19:37:41,631]** [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : IMAP from ip : 192.168.0.124 finished with status :  Failure. unknown status. Return Code : 108. Execution finished in : 58 ms>

Steps to reproduce

filter created to capture the above log:

failregex = (?:Authentication for user :) .*(?:from ip : (?P<host>\S*)).*(?:finished with status :  Failure).*(?: Return Code : ).*
ignoreregex = (?:Authentication for user :) .*(?:from ip : (?P<host>\S*)).*(?:finished with status :  Failure).*(?:Return Code : 114).*

Expected behavior

1. The above filter should capture failed logins
2. The filter should skip the failed login if the Return code value is 114

Observed behavior

1. The filter created is not able to parse the date from the logs of /var/log/tomcat2/catalina.out

Any additional information

Configuration, dump and another helpful excerpts

1. No changes to jail.config
2. Created a jail in jail.local for capturing the failed logins (name = trialJail)

Any customizations done to /etc/fail2ban/ configuration

Added jail in jail.local

[trialJail]

enabled = true
filter = multiService
action = iptables-multiport[name=trialJail, port="smtp,imap,pop3", protocol=tcp]
logpath = /var/log/tomcat2/catalina.out
findtime = 1200
bantime = 30
maxretry = 2
ignoreip =

Relevant parts of /var/log/fail2ban.log file:

preferably obtained while running fail2ban with loglevel = 4

1. The error thrown by fail2ban for not being able to parse the date is as follows

Jun  6 19:37:41 Master fail2ban.filter[7747]: WARNING Found a match for u'[06-Jun-2018_19:37:41,631] [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : IMAP from ip : 192.168.0.124 finished with status :  Failure. unknown status. Return Code : 108. Execution finished in : 58 ms>' but no valid date/time found for u'[06-Jun-2018_19:37:41,631] [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : IMAP from ip : 192.168.0.124 finished with status :  Failure. unknown status. Return Code : 108. Execution finished in : 58 ms>'. Please try setting a custom date pattern (see man page jail.conf(5)). If format is complex, please file a detailed issue on https://github.com/fail2ban/fail2ban/issues in order to get support for this format.

Relevant lines from monitored log files in question:

The log lines that I am working on are: (lines that needs to be captured)

[06-Jun-2018_17:07:42,500] [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : **IMAP** from ip : 192.168.0.124 finished with status :  Failure. unknown status. Return Code : 108. Execution finished in : 280 ms>
[06-Jun-2018_17:09:57,596] [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : null from ip : null finished with status :  Failure. invalid parameters are provided to function. Return Code : 106. Execution finished in : 1 ms>
[06-Jun-2018_17:25:20,653] [INFO] [mithi.mcs.auth.MithiAuthService(doAuth)]  - <Authentication for user : trial@trial.int for service : **POP** from ip : 192.168.0.124 finished with status :  Failure. unknown status. Return Code : 108. Execution finished in : 84 ms>

The main focus of this question is how do I parse the date pattern ?
Bonus question would be how do I capture all the above logs in one single filter (something that I am currently working on (ports : smtp,pop3,imap))

[sebres]

how do I parse the date pattern

Your pattern could be for example '^[%d-%b-%Y_%H:%M:%S,%f]'.

Note, in the config-files, you should specify it with dual percent-character, like:

datepattern = ^\[%%d-%%b-%%Y_%%H:%%M:%%S,%%f\]

PoC:

$ fail2ban-regex -v -d '^\[%d-%b-%Y_%H:%M:%S,%f\]' "$str" 'from ip : <HOST>'

Running tests
=============

Use      datepattern : ^\[Day-MON-Year_24hour:Minute:Second,Microseconds\]
Use   failregex line : from ip : <HOST>
Use      single line : [06-Jun-2018_17:07:42,500] [INFO] [mithi.mcs.auth....


Results
=======

Failregex: 1 total
|-  #) [# of hits] regular expression
|   1) [1] from ip : <HOST>
|      192.168.0.124  Wed Jun 06 17:07:42 2018
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1] ^\[Day-MON-Year_24hour:Minute:Second,Microseconds\]
`-

Lines: 1 lines, 0 ignored, 1 matched, 0 missed
[processed in 0.00 sec]

Also note that fail2ban cuts out the date-time matching datepattern from the string before search the match for failregex.

Bonus question

Bonus answer - read the manual, for example start with Regular Expression Syntax :: Python documentation.
For example your logged ports: smtp,pop3,imap may be something like
for service : (?:SMTP|POP3|IMAP|null) from ip,
or just as not-space \S+ regex:
for service : \S+ from ip...

[jjoshi-space]

That worked well, thank you Sir
Another question, I have a ignore regex pattern to the above logs which escapes Return Code : 114.
ignoreregex = .*(?:Return Code : 114.).*
The above ignoreregex command works perfectly fine to escape the Return Code : 114.
So the question is , how do I mention the above ignoreregex in failregex itself to capture all Codes but not 114 so as to avoid the computation of ignoreregex ?
The failregex that works for all Codes is - (working)
failregex = (?:for service :).* (?:smtp|POP|IMAP|null).*(?:from ip : <HOST>).*(?: finished with status : Failure.).*(?:Return Code : ).*
The failregex which shall get all Codes but not 114 is - (not working)
So far I've tried,

> failregex = (?:for service :).* (?:smtp|POP|IMAP|null).*(?:from ip : <HOST>).*(?: finished with status : Failure.).*(?!.*Return Code : 114.).*

[sebres]

Something like .*(?!.*Return Code : 114.).* will never work (let alone it's vulnerable), because catch all .* will always find something, e. g. by something. Return Code : 114. it can pretty catch something. R and the rest eturn Code : 114. does not match your negative lookahead.

So you should just use more precise pattern:

>>> import re
>>> cr = re.compile(r"\S+\. Return Code : (?!114)\d+\.")
>>> print(bool(cr.match("something. Return Code : 108.")))
True
>>> print(bool(cr.match("something. Return Code : 114.")))
False

And last but not least, don't use catch-alls and try to anchor regex (at least from one side, e. g. from begin ^... or from both ^...$, note .*$ or ^.* are not an anchors and make no sense).
So be more precise in regex, especially if some foreign (user) input can occur in the log-line. Otherwise you makes it vulnerable for intruder (e. g. by manipulating of the line, he could avoid detection or even could force to ban other IP instead of his real IP.

[jjoshi-space]

How do I implement it in fail regex? I have provided a working regex for all numbers in my previous comment, I want to implement one which ignores 114 in the failregex itself. Thanks for your time and help :)

[sebres]

Sorry, no time ATM to make your regex fully correct. And I do not make fast and dirty solutions on principle.
Try stackoverflow-similar sites.

[sebres]

Here you are:

^\s*\[\S+\]\s+\[[^\]]+\]\s+-\s+<Authentication for user : \S+[^:]* for service : \S+ from ip : <HOST> finished with status :  Failure\.\s+(?:[^.]+\.)* Return Code : (?!114)\d+\.

But, a bit more precise possible filter will look like:

[Definition]

# services (smtp,imap,etc,...):
_services = (?:\S+)

# all errors (not-precise):
#_errors = (?:[^.]+\.)*
# only expected errors (more precise), example:
#_errors = (?:unknown status|authenication error|something else)\.
_errors = (?:unknown status)\.

# return-codes (any excepting 114):
_ret_codes = (?!114)\d+

failregex = ^\s*\[\S+\]\s+\[[^\]]+\]\s+-\s+<Authentication for user : \S+[^:]* for service : %(_services)s from ip : <HOST> finished with status :  Failure\.\s+%(_errors)s Return Code : %(_ret_codes)s\.

[Init]

datepattern = ^\[%%d-%%b-%%Y_%%H:%%M:%%S,%%f\]

You've 3 additional interpolation-variables (starting with underscore) to control what exactly should be banned.
You can also use first (commented) _errors substitution for all possible errors, but as the comment says - it is not-precise... and I don't like such loosening without looking for the format of all possible log-messages formats there.
But I believe you can self extend this list as in example above.

[jjoshi-space]

The failregex is now able to do the work of ignoreregex within itself !!
I understand that ignoreregex function is provided for that to make things clear with the available functionalities.
The above example made it clear how flexible the filter can be upon different scenarios.
Thanks for your time and help.
Good Day, Sir :)