Ultimate Referrer Blacklist | Automatic Nginx Converter

README.md

In an effort to help stop spam referrals from even entering your website, might as well just catch them and send them to a 403 or something.

There is a blacklist file available from perishablepress.com found here ultimate-referrer-blacklist.txt.

This file is useful, but it presents exactly one problem: It's in pseudo .htaccess. It's meant to be copied from and pasted into an .htaccess file. Because there also exists a huge list of url patterns that comprises of about 90% of the txt that isn't .htaccess. This is also not useful at all in any shape nor form as is to servers using Nginx who wish to make use of this list (like me).

What I have done is written a generic PHP script that downloads the file, runs through each line, and converts it into an nginx directive. The intent here is to run this file, and dump the output somewhere else on the server, in this case a new file called blacklist.conf located in the core /etc/nginx directory.

Within your server block in your sites-available configurations, you can simply use an include {file_path}; directive to bring it all in, and into as many server blocks as you want.

I am open to forks to improve this system, and additional scripting languages to accomplish the same goal.

blacklist_converter.php

<?php
// /var/www/scripts/blacklist_converter.php

$content = file_get_contents('https://perishablepress.com/blacklist/ultimate-referrer-blacklist.txt');

$lines   = explode(PHP_EOL, $content);
$results = [];

foreach($lines as $line)
{
    if(trim($line)) {
        if(strpos($line, 'RewriteCond') !== FALSE && strpos($line, 'RewriteCond %{HTTP_REFERER} !') === FALSE) {
            $lineItem = trim(str_replace(["RewriteCond %{HTTP_REFERER}", "[OR]"], "", $line));
            $results[] = sprintf('if ($http_referer ~ "%s"){ return 403; }', $lineItem);
        } else if(trim($line)[0] == '#') {
            // comment
            $results[] = $line;
        } else {
            if(strpos($line, 'https?') === 0) {
                $results[] = sprintf('if ($http_referer ~ "%s"){ return 403; }', $line);
            }
        }
    } else {
        $results[] = "";
    }
}

print implode(PHP_EOL, $results);

nginx_server_config

# /etc/nginx/sites-available/default 
server {
  listen 80;
  # .. other configuration here
  include /etc/nginx/blacklist.conf;
  # .. other configuration here
}

weekly_cron_blacklist_updater

#!/bin/sh
# /etc/cron.weekly/blacklist_updater
# blacklist updater cron weekly

php /var/www/scripts/blacklist_generator.php > /etc/nginx/blacklist.conf
nginx -t && service nginx reload