A brief description of the set up I run here, for no other reason than to provide some background and to pad out this article so it’s more than one paragraph long 🙂
I have various http servers. Apache does all the work building pages & serving them up while Nginx sits in front of it on a different machine and caches the data. This gives me a performance boost because, more often than not, a page requested by a visitor is already loaded in the cache and Nginx can just send it to them quickly. The Nginx Cache is actually located in RAM rather than on disk which makes reading it even faster. See my previous article on that.
Fail2ban
I love fail2ban! It monitors logs & bans IP addresses attempting underhanded and malicious activities. If they are up to no good, I kick them out.
Fail2ban was running on the Apache machine and has been working very well for a long time, but I concluded I also needed fail2ban on the Nginx box because it was serving pages from the cache and those requests might never get to Apache, therefore scumbags could play silly buggers and get a response until they asked for something not cached. Even if they tried to hit the Apache box and we blocked, Nginx would return a reply of some description.
Here is my trouble
My troubles started soon after installing fail2ban on the Nginx machine. I figured it would just work because the OS is exactly the same. Both machines are running Centos 7. I copied my config files, filters, actions, etc. from the Apache box and made some changes to the log path to reflect Nginx. I disabled all filters except the WordPress related because I wanted that one working before I mucked around with others.
I tested the configuration like this;
fail2ban-regex /var/log/nginx/mylogfile /etc/fail2ban/filter.d/wordpress.conf
The results said there were 10432 matches. Of course some of those matches would be repeated from the same IP address but even so, a few IP addresses should have been blocked.
The fail2ban action, which works 100% of the time on the Apache machine is like this;
firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address= drop"
The fail2ban log (/var/log/fail2ban.log) shows the following;
INFO [wordpress-login] Found 68.65.122.200 - 2020-04-22 05:20:39
What is should show is;
INFO [wordpress-login] Found 68.65.122.200
The difference is that Fail2ban was attempting to ban the IP address and the date time stamp. I don’t know why and I never figured it out. I tried everything I could think of, including creating a custom log format so the Nginx log was identical to the Apache logs. Nothing!
I still don’t understand what was going wrong. Both machines have the same OS, same version of fail2ban, the same config files etc. It just didn’t work. I even compared logs using a Hex Editor hoping there were stray characters which were misleading fail2ban. Nothing made sense.
Workaround
After two days of screwing around and going nowhere I came up with a workaround. Not being an expert in Linux I asked myself “can the Apache machine execute a command on the Nginx box?” It turns out it can using ssh.
The following command allowed fail2ban to execute the same banning operation on the Nginx box, therefore temporarily solving my problem. I say “temporarily” because it is a workaround, not a solution and the workaround creates additional security issues I intend to solve.
First on the Nginx box I created to scripts, one called ban.sh the other unban.sh and made them executable.
echo \ firewall-cmd \ --zone=public --add-rich-rule="rule family=ipv4 source address=$1 drop" > ban.sh chmod +x ban.sh
Use of the backslash, for those not in the know, allows you to execute large commands over multiple lines. It wont execute until you reach the end.
Fail2ban, when unloaded reverses all the bans so we need an unban script too;
echo \ firewall-cmd \ --zone=public --remove-rich-rule="rule family=ipv4 source address=$1 drop" > unban.sh chmod +x unban.sh
Update the fail2ban action taken when blocking and unblocking;
# First the blocking firewall-cmd --zone=public --add-rich-rule="rule family=ipv4 source address= drop" ssh -t root@192.168.0.100 ./ban.sh <ip> # Now unblocking firewall-cmd --zone=public --remove-rich-rule="rule family=ipv4 source address= drop" ssh -t root@192.168.0.100 ./unban.sh <ip>
You may notice the ban.sh and unban.sh scripts are passed the IP address for the scumbag you are booting out. The two scripts created earlier have $1 as a variable, when the scripts are called they are passed the IP address.
Downside
In order to make this work I had to enable ssh login without a password. I have also had to enable root login via ssh which I hate doing. I have denied all addresses except the specific address for the Apache box. As I said previously, this is a temporary workaround and not a solution. I also do not use the standard port 22 or 2222 nor do I port forward from any routers. All ssh access must come from inside the network otherwise it is ignored.