As the title suggests, my Nginx performance dropped dramatically and my life became hell for over a week as a result. Allow me to explain further.
I have five domains all of which previously ran on a single server with Apache, Postfix, Dovecot, MySQL and a few other minor services. About 18 months I decided I would install Nginx as a Reverse Proxy with Apache as the back-end. Choosing that combination allowed me to build the Nginx config without affecting my production Apache configuration. Once I had perfected the new config I would direct port 80 towards Nginx and allow it to start caching pages.
All my domains are WordPress and it took a lot of trial and error getting it right but eventually I succeeded and Nginx became my front-end http processor. You can download my entire config at the bottom of the page.
Using a small script I was able to use wget to mirror each site on my laptop and time how long it took. With an empty Nginx cache all five domains were mirrored in about 5 – 6 minutes; once the cache was loaded with pages Nginx was able to server all sites in about 1 minute 20 seconds. By using wget I was effectively pre-loading the cache which meant visitors were served faster right away.
Sudden drop in performance
After running successfully for 18 months I noticed the timing for my pre-load script had increased from 5 – 6 minutes to 25 to 30 minutes. Once the cache had been pre-loaded it was about 8 or 9 minutes to mirror all five domains. Something had gone wrong and I had no idea what it was.
I’m a fiddler – That means I’m always trying to improve things. A tweak here and a tap with a hammer there so I can squeeze a little more out of it. The results I look for are resource usage and end performance.
My starting configuration was based on config examples from https://www.ryadel.com/en/nginx-reverse-proxy-cache-wordpress-apache-iis/ which over time I have modified to suit my needs. Specifically I needed Nginx working for multiple domains, all of which run with https.
Some of my domains allow other users to create accounts and blogs of their own and the original scripts didn’t take that into account. When a new blog was posted it wouldn’t appear until the cache had expired and that wasn’t ideal.
What had changed in my Nginx configuration
My rational when tweaking is something along the lines of ; make small changes then test everything including logging in to the site. This approach often saves me time when something goes wrong because backtracking isn’t difficult. More often than not my methods start out well and then I get a bit excited and make dozens of small changes in between test runs. It is this new found excitement which brings me unstuck.
Reading my logs I discovered that every scumbag hacker around the universe was attempting to access and use a common WordPress file; specifically xmlrpc.php. After a little bit of reading I found that most hosts block access to this file and I concluded I should do the same. The recommended method for denying access was to add the following code to my server block.
location location ~* "/xmlrpc.php$" {
return 444;
access_log off;
}
Return code 444 effectively denies the connection without sending an acknowledgement. From the remotes point of view it is as if you don’t exist. Returning 403 or some other response tells the remote you are here but denying them access. Obviously if they try to access other files they will know you exist, whereas if they are running a script attempting to exploit xmlrpc.php they will give up and move on to another site.
Unbeknown to me it was this location block causing the performance degradation. Before I worked that out I would make changes to other bits of code looking for the culprit while digging myself a bigger hole with every change.
Solution – for me at least
Because I had made so many small changes along the way I decided a complete rebuild was my best option. So I started again and edited, trimmed and tweaked my config. You will find my new config at the bottom of this article.
When the performance of a brand new configuration suddenly dropped again I started commenting out large bits and pieces until the performance went back up. The issue went away when many of my location blocks were removed. I did what any admin would do and I poured myself a beer and began re-adding blocks and testing in between each change.
When I enabled the xmlrpc block everything turned to custard again. I was confused because I was also returning a 444 code for other situations such as bad bots or countries I was filtering.
location ~* "/xmlrpc.php$" {
deny all;
access_log off;
}
The simple change above cured my issues and now with a little more tweaking all five sites are mirror in about four minutes. Once the cache has been pre-loaded the same mirroring function completes in around one minute averaging 16 seconds per site.
I do not understand why this small change cured my problems and don’t really care. I’m still denying the remote scumbag and my fail2ban configuration stops them after three attempts.
My new Nginx configuration
My completely rebuilt Nginx config below was based on the example from ryadel.com with some major changes allowing for SSL and multiple domains.
Since my last post on the topic of Nginx (Nginx Reverse Proxy in RAM = Performance +) I have introduced new hardware. Nginx is running on a dedicated bare metal server while each of the domains are Linux Containers (LXC) handled through Proxmox. Each Container is running with Apache, MySQL etc. You don’t have to do things as I do but it works well for me. I mention this because there is a config file called backends.conf which you can modify to suit your needs. Also note that I cache stuff onto a RAM Drive rather than a physical disk. I don’t have SSD but if you do you should use it.
I am using nginx-more because it includes GepIP2 and nginx-more-headers.
Download sample-nginx.tar
Extract the archive into a folder of your choice and make modifications as required. I am not claiming this is the greatest Nginx configuration but it will give you a starting point. Once your modifications are complete make a backup of your existing Nginx folder and copy the new files into your Nginx folder.