Recently, one of the web servers I’m responsible for got hammered with a distributed high load of HTTP requests that got the server down for 20 minutes. Once I got the server up again the distributed attack was still running and the case was clear from the log file, and we was able to stop it by temporarily banning the offending IPs (which by the way appears to be from Russia).
I had to find a quick solution since the attacker can start the distributed attack from new IPs other than the one we blocked. Few of the ways I thought about was to limit number of requests from the web server by denying further requests, or probably Intrusion Prevention System, or to use Fail2Ban. I decided to give Fail2Ban a try.
Why Fail2Ban?
Back when I used OpenBSD they introduced if I remember correctly a new SMTP server to only respond to spam’ing IPs, the idea was to load a list of IPs that are known to be spammers from SPAM list databases, when they arrive to mail server they are redirected to a so called stuttering SMTP. An SMTP service that consume spammer resource by slowing down the connection and keeping it open as long as possible.
I wanted to consume resource of our attackers as well, when Fail2Ban discover their behavior it will add offending IPs to be dropped by the firewall till their connection times out so we will stale them for some time. It’s not very effective, but at least that what i had in my mind.
What is Fail2Ban?
As per Wikipedia: Fail2ban is an intrusion prevention software framework that protects computer servers from brute-force attacks. Written in the Python programming language, it is able to run on POSIX systems that have an interface to a packet-control system or firewall installed locally, for example, iptables or TCP Wrapper.
It does so by monitoring log file for predefined regular expressions that contains IP of attackers with a set of criteria, like time window of the attack and number of tries, and when a match is found it takes action of preventing access of the offending IP, either add it to the firewall, hosts.deny and probably other actions.
Enough with blah blah and now with the technical stuff:
Software stack:
- Varnish caching server (port 80)
- Nginx (port 8080)
- PHP-FPM (port 9000)
- MySQL (port 3306)
Sample log entry in Nginx:
127.0.0.1 - - [15/May/2016:11:55:14 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" "XXX.XXX.XXX.XXX"
Note that the offending URL appears as last entry not in the beginning as usually happens, because of Varnish.
On CentOS 7, I installed Fail2Ban using:
# yum -y install fail2ban |
It will install Fail2Ban servers, client (which will connect to the server to control it or display information), and other utilities for testing and so (like fail2ban-regex).
No I defined the filter to match IP from the above log entry as:
# /etc/fail2ban/filter.d/nginx-wp-xmlrpc.conf: # fail2ban filter configuration for nginx behind varnish running wordpress website. # it will prevent brute force attacks for login via xmlrpc. [Definition] # Match the following: # 127.0.0.1 - - [15/May/2016:22:40:37 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)" "XXX.XXX.XXX.XXX" failregex = POST /xmlrpc.php HTTP/.*""$ ignoreregex = |
Now to test it:
# fail2ban-regex /var/log/nginx/access.log /etc/fail2ban/filter.d/nginx-wp-xmlrpc.conf Running tests ============= Use failregex filter file : nginx-wp-xmlrpc, basedir: /etc/fail2ban Use log file : /var/log/nginx/access.log Use encoding : UTF-8 Results ======= Failregex: 11774 total |- #) [# of hits] regular expression | 1) [11774] POST /xmlrpc.php HTTP/.*"<HOST>"$ `- Ignoreregex: 0 total Date template hits: |- [# of hits] date format | [16427] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)? `- Lines: 16427 lines, 0 ignored, 11774 matched, 4653 missed [processed in 2.17 sec] Missed line(s): too many to print. Use --print-all-missed to print all 4653 lines |
11774 lines! 11774 attack attempts. And to be sure:
# grep "POST /xmlrpc.php" /var/log/nginx/access.log | wc -l 11774 # |
No with the Jail for the offending IPs:
# /etc/fail2ban/jail.d/01-nginx-wp-xmlrpc.conf: # For now if we got 20 occurrences of those in 2 minutes we will ban the offender # ban for 12 hours</code> [nginx-wp-xmlrpc] enabled = true logpath = /var/log/nginx/access.log maxretry = 20 findtime = 120 bantime = 43200 # In secs. Or negative for permanent. port = http,https |
Now start the the server:
# systemctl start fail2ban # systemctl enable fail2ban Created symlink from /etc/systemd/system/multi-user.target.wants/fail2ban.service to /usr/lib/systemd/system/fail2ban.service. # |
Check the status of Fail2Ban server:
# fail2ban-client status Status |- Number of jail: 1 `- Jail list: nginx-wp-xmlrpc # fail2ban-client status nginx-wp-xmlrpc Status for the jail: nginx-wp-xmlrpc |- Filter | |- Currently failed: 0 | |- Total failed: 0 | `- File list: /var/log/nginx/access.log `- Actions |- Currently banned: 0 |- Total banned: 0 `- Banned IP list: # |
Make sure all is fine in: /var/log/fail2ban.log