Block AI crawlers in nginx — fetch published bot IP ranges (OpenAI, Perplexity), render an nginx geo map, validate, install atomically, reload only on change

README.md

Block AI crawlers in nginx

update-ai-blocklist.sh fetches the officially published IP ranges of AI crawlers (OpenAI GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User) and renders them into an nginx geo map that sets $ai_blocked_ip to 1 for matching client IPs.

It validates every prefix as CIDR, dedupes across vendors, refuses suspiciously short lists, installs atomically, checks the result with nginx -t (rolls back to the previous list on failure) and reloads nginx only when the list actually changed — so it is safe to run from cron.

Requirements

bash, curl, jq, nginx with the (default) geo module.

Usage

sudo ./update-ai-blocklist.sh

This writes /etc/nginx/conf.d/ai-blocklist.conf. Then act on the variable wherever you want — block your whole site:

server {
    if ($ai_blocked_ip) {
        return 403;
    }
    # ...
}

…or only specific URLs. Pass the locations as arguments and the script prints ready-to-paste snippets:

./update-ai-blocklist.sh /observation '~ ^/api/reports'

location /observation {
    if ($ai_blocked_ip) {
        return 403;
    }

    # ... your existing proxy_pass/fastcgi/root config for this location
}

Tip: if you'd rather slow the bots down than lock them out, key a limit_req_zone on the variable instead of returning 403:

map $ai_blocked_ip $ai_limit_key {
    0 "";
    1 $binary_remote_addr;
}
limit_req_zone $ai_limit_key zone=aibots:10m rate=6r/m;

Behind a load balancer / CDN

When nginx is not the edge, the client IP arrives in X-Forwarded-For. Set the trusted proxy ranges and the geo map will use that header for those senders:

GEO_PROXY="10.0.0.0/8 173.245.48.0/20" ./update-ai-blocklist.sh

Configuration

Variable	Default	Purpose
OUT	/etc/nginx/conf.d/ai-blocklist.conf	Target file
MIN_PREFIXES	20	Abort if fewer prefixes were parsed
RELOAD	auto	auto/yes/no — test + reload nginx
GEO_PROXY	(empty)	Space-separated CIDRs of trusted proxies

Cron

17 4 * * * /usr/local/bin/update-ai-blocklist.sh >/dev/null

Status messages go to stdout, errors to stderr — discarding stdout keeps cron mail limited to real problems.

Adding sources

Add a 'name url' line to the SOURCES array. The endpoint must serve JSON in the common {"prefixes": [{"ipv4Prefix": "..."}, {"ipv6Prefix": "..."}]} format. Anthropic (ClaudeBot) does not publish a stable JSON endpoint yet — the commented line in SOURCES is the placeholder for when it ships.

update-ai-blocklist.sh

#!/usr/bin/env bash
set -euo pipefail

# Fetches the published crawler IP ranges of AI bots and renders them into an
# nginx geo map ($ai_blocked_ip). Validates every prefix, installs atomically,
# checks with `nginx -t`, rolls back on failure and only reloads on change.
#
# Usage: update-ai-blocklist.sh [location ...]
#
# Optional location arguments (e.g. "/observation" or "~ ^/api/reports") do not
# change the generated blocklist — they print ready-to-paste nginx snippets
# that apply the block to exactly those URLs in your server {} block.
#
# Environment overrides (useful for testing):
#   OUT=...             target conf path (default /etc/nginx/conf.d/ai-blocklist.conf)
#   MIN_PREFIXES=...    safety floor, refuse a suspiciously short list (default 20)
#   RELOAD=auto|yes|no  nginx test + reload (default auto: only when nginx is in PATH)
#   GEO_PROXY=...       space-separated CIDRs of trusted proxies; when set, nginx
#                       takes the client IP from X-Forwarded-For for these senders

OUT=${OUT:-/etc/nginx/conf.d/ai-blocklist.conf}
MIN_PREFIXES=${MIN_PREFIXES:-20}
RELOAD=${RELOAD:-auto}

# ordered list -> deterministic output, also works with macOS bash 3.2
SOURCES=(
    'gptbot https://openai.com/gptbot.json'
    'oai-searchbot https://openai.com/searchbot.json'
    'chatgpt-user https://openai.com/chatgpt-user.json'
    'perplexitybot https://www.perplexity.ai/perplexitybot.json'
    'perplexity-user https://www.perplexity.ai/perplexity-user.json'
    # 'claudebot <anthropic-json-url>'  # add once Anthropic ships a stable endpoint
)

IPV4_CIDR='^([0-9]{1,3}\.){3}[0-9]{1,3}/([0-9]|[12][0-9]|3[0-2])$'
IPV6_CIDR='^[0-9A-Fa-f:]+/([0-9]|[1-9][0-9]|1[01][0-9]|12[0-8])$'

die() {
    printf 'ERROR: %s\n' "$*" >&2
    exit 1
}

if [[ ${1:-} == -h || ${1:-} == --help ]]; then
    sed -n '3,/^$/s/^# \{0,1\}//p' "$0"
    exit 0
fi

PROTECTED_LOCATIONS=("$@")

print_paste_snippets() {
    (( ${#PROTECTED_LOCATIONS[@]} > 0 )) || return 0
    local location
    printf '\nPaste into every server {} block that should deny AI bots:\n\n'
    for location in "${PROTECTED_LOCATIONS[@]}"; do
        # $ai_blocked_ip is an nginx variable, not shell
        # shellcheck disable=SC2016
        printf 'location %s {\n    if ($ai_blocked_ip) {\n        return 403;\n    }\n\n    # ... your existing proxy_pass/fastcgi/root config for this location\n}\n\n' "$location"
    done
}

for dependency in curl jq; do
    command -v "$dependency" >/dev/null || die "required tool '$dependency' is not installed"
done

workdir=$(mktemp -d)
staged=''
trap 'rm -rf "$workdir" ${staged:+"$staged"}' EXIT

fetch_prefixes() {
    local name=$1 url=$2 json prefixes prefix
    json=$(curl -fsS --max-time 20 --retry 2 "$url") \
        || die "fetch failed for $name ($url) — previous list stays in place"
    prefixes=$(jq -r '.prefixes[]? | (.ipv4Prefix // .ipv6Prefix) | strings' <<<"$json") \
        || die "unparseable JSON from $name"
    [[ -n $prefixes ]] || die "no prefixes in response of $name (upstream format change?)"
    while IFS= read -r prefix; do
        [[ $prefix =~ $IPV4_CIDR || $prefix =~ $IPV6_CIDR ]] \
            || die "refusing invalid prefix '$prefix' from $name"
        printf '%s\n' "$prefix"
    done <<<"$prefixes"
}

body=$workdir/body
for entry in "${SOURCES[@]}"; do
    read -r name url <<<"$entry"
    printf '    # %s\n' "$name"
    fetch_prefixes "$name" "$url" | sed 's/^/    /; s/$/ 1;/'
done >"$body"

# duplicate networks across vendors would trigger nginx geo warnings
deduped=$workdir/deduped
awk '/^[[:space:]]*#/ { print; next } !seen[$1]++' "$body" >"$deduped"

count=$(grep -c ' 1;$' "$deduped" || true)
(( count >= MIN_PREFIXES )) || die "only $count prefixes parsed (< $MIN_PREFIXES), aborting"

new_conf=$workdir/new.conf
{
    printf '# managed by update-ai-blocklist.sh — do not edit\n'
    # $ai_blocked_ip is an nginx variable, not shell
    # shellcheck disable=SC2016
    printf 'geo $ai_blocked_ip {\n    default 0;\n'
    read -ra trusted_proxies <<<"${GEO_PROXY:-}"
    for proxy_cidr in ${trusted_proxies[@]+"${trusted_proxies[@]}"}; do
        printf '    proxy %s;\n' "$proxy_cidr"
    done
    cat "$deduped"
    printf '}\n'
} >"$new_conf"

if cmp -s "$new_conf" "$OUT" 2>/dev/null; then
    printf '%s unchanged (%d prefixes), nothing to do\n' "$OUT" "$count"
    print_paste_snippets
    exit 0
fi

outdir=$(dirname "$OUT")
[[ -d $outdir ]] || die "target directory $outdir does not exist"

backup=''
if [[ -f $OUT ]]; then
    backup=$workdir/previous.conf
    cp -p "$OUT" "$backup"
fi

staged=$(mktemp "$outdir/.ai-blocklist.XXXXXX")
cp "$new_conf" "$staged"
chmod 0644 "$staged"
mv -f "$staged" "$OUT"
staged=''

if [[ $RELOAD == no ]] || { [[ $RELOAD == auto ]] && ! command -v nginx >/dev/null; }; then
    printf '%s updated: %d prefixes (nginx reload skipped)\n' "$OUT" "$count"
    print_paste_snippets
    exit 0
fi

if ! nginx -t 2>"$workdir/nginx-test.out"; then
    cat "$workdir/nginx-test.out" >&2
    cp "$new_conf" "${OUT}.rejected"
    if [[ -n $backup ]]; then
        mv -f "$backup" "$OUT"
        die "nginx -t rejected the new config; previous config restored, rejected version kept at ${OUT}.rejected"
    fi
    rm -f "$OUT"
    die "nginx -t rejected the new config; removed it (no previous version existed), rejected version kept at ${OUT}.rejected"
fi

if command -v systemctl >/dev/null; then
    systemctl reload nginx
else
    nginx -s reload
fi
printf '%s updated: %d prefixes, nginx reloaded\n' "$OUT" "$count"
print_paste_snippets