Check WHOIS server DNS status #shell

README.md

ZoneDB Scripts

This directory contains utility scripts for maintaining and validating the ZoneDB database.

Scripts

check_domain.sh

Purpose: Quick DNS resolution checker for individual domains.

What it does:

• Tests DNS resolution for one or more domain names
• Shows A and AAAA records (IPv4 and IPv6 addresses)
• Distinguishes between different DNS states:
  • Domain exists and resolves to IP addresses ✅
  • Domain exists but has no A/AAAA records ⚠️
  • Domain doesn't exist (NXDOMAIN) ❌
  • DNS query failed for other reasons ⚠️

Why it exists: Provides a quick way to test DNS resolution for domains, especially useful when investigating issues found by other scripts or when adding new domains to the database.

Usage:

# Check a single domain
./scripts/check_domain.sh google.com

# Check multiple domains
./scripts/check_domain.sh google.com github.com nonexistent.domain

Example Output:

✅ google.com exists and resolves to:
142.251.29.139
142.251.29.101
2607:f8b0:4004:c1b::64
2607:f8b0:4004:c1b::71

❌ nonexistent.domain does not exist (NXDOMAIN)

whois.sh

Purpose: Validates and cleans up whoisServer entries in metadata JSON files.

What it does:

• Scans all JSON files in the metadata/ directory for whoisServer entries
• Tests DNS resolution for each whois server hostname using dig
• Reports which whois servers are no longer resolvable (likely defunct)
• Optionally removes broken whoisServer entries from the JSON files

Why it exists: Over time, whois servers can become defunct or move to different hostnames. This script helps maintain data quality by identifying and optionally removing outdated whois server references that no longer resolve via DNS.

DNS Logic: The script considers a whois server valid unless dig returns NXDOMAIN status. This approach correctly handles:

• Servers with only IPv6 addresses (AAAA records)
• Servers behind CNAMEs
• Servers with no A records but that exist in DNS
• Temporary DNS server issues (SERVFAIL, REFUSED, etc.)

Only hostnames that return NXDOMAIN (domain doesn't exist) are considered broken and flagged for removal.

Performance: Uses concurrent processing (up to 10 parallel DNS checks) to handle the large number of whois servers efficiently.

Usage:

# Show only failed DNS resolutions (default)
./scripts/whois.sh

# Show both successful and failed DNS resolutions
./scripts/whois.sh --show-all

# Remove broken whoisServer entries from JSON files
./scripts/whois.sh --remove-broken

# Remove broken entries and show all results
./scripts/whois.sh --remove-broken --show-all

Example Output:

simple_check.sh

#!/bin/bash

check_domain() {
  local domain="$1"
  local addrs cname rcode

  # First, get response code for A query
  rcode=$(dig "$domain" A +noall +comments 2>/dev/null | grep 'status:' | awk '{print $6}' | tr -d ',')

  if [[ "$rcode" == "NXDOMAIN" ]]; then
    echo "❌ $domain does not exist (NXDOMAIN)"
    return 1
  elif [[ "$rcode" != "NOERROR" ]]; then
    echo "⚠️ $domain query failed with status: $rcode"
    return 3
  fi

  # Look for A/AAAA records
  addrs=$(dig "$domain" A AAAA +short 2>/dev/null)
  if [[ -n "$addrs" ]]; then
    echo "✅ $domain exists and resolves to:"
    echo "$addrs"
    return 0
  fi

  # If no A/AAAA, check for CNAME
  cname=$(dig "$domain" CNAME +short 2>/dev/null)
  if [[ -n "$cname" ]]; then
    echo "ℹ️ $domain is a CNAME alias to $cname"
    # Try to resolve the target too
    check_domain "$cname"
    return $?
  fi

  echo "⚠️ $domain exists in DNS but has no A/AAAA records"
  return 2
}

# Main script execution
if [[ $# -eq 0 ]]; then
  echo "Usage: $0 <domain1> [domain2] [domain3] ..."
  echo "Check DNS resolution for one or more domains"
  exit 1
fi

echo
# Check each domain provided as arguments
for domain in "$@"; do
  check_domain "$domain"
	echo
done

whois.sh

#!/bin/bash

# whois.sh - Check whoisServer hostnames in metadata JSON files and optionally remove unresolvable ones
#
# This script scans all JSON files in the metadata/ directory, extracts whoisServer values,
# and tests their DNS resolution. It can optionally remove entries that fail DNS checks.
#
# DNS Logic: A whoisServer is considered valid unless dig returns NXDOMAIN status.
# This handles cases where servers exist but have no A records, CNAME-only records, etc.
#
# Concurrency: Uses xargs -P 10 to run up to 10 DNS checks in parallel for speed.
#
# Usage:
#   ./scripts/whois.sh                          # Show only failed DNS resolutions
#   ./scripts/whois.sh --show-all              # Show both successful and failed resolutions
#   ./scripts/whois.sh --remove-broken         # Remove failed whoisServer entries from JSON files
#   ./scripts/whois.sh --remove-broken --show-all  # Remove broken entries and show all results

set -euo pipefail  # Exit on error, undefined vars, or pipe failures

# ANSI color codes for colored output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color

# Parse command line arguments
REMOVE_BROKEN=false  # Whether to actually remove broken whoisServer entries from JSON files
SHOW_ALL=false       # Whether to show successful DNS resolutions (default: only show failures)

for arg in "$@"; do
    case $arg in
        --remove-broken)
            REMOVE_BROKEN=true
            ;;
        --show-all)
            SHOW_ALL=true
            ;;
        *)
            echo "Usage: $0 [--remove-broken] [--show-all]"
            echo "  --remove-broken: Remove whoisServer entries that fail DNS resolution"
            echo "  --show-all: Show both successful and failed DNS resolutions (default: only failed)"
            exit 1
            ;;
    esac
done

# Counters for final summary statistics
total_files=0      # Total JSON files processed (not used currently)
files_with_whois=0 # Total files containing whoisServer entries (not used currently)
removed_servers=0  # Number of whoisServer entries actually removed from JSON files
failed_dns=0       # Number of whoisServer hostnames that failed DNS resolution
successful_dns=0   # Number of whoisServer hostnames that passed DNS resolution

# Display startup banner and configuration
echo "Checking whoisServer DNS resolution in metadata files..."
if [[ "$REMOVE_BROKEN" == "true" ]]; then
    echo -e "${YELLOW}--remove-broken flag detected: will remove failed whoisServer entries${NC}"
fi
if [[ "$SHOW_ALL" == "true" ]]; then
    echo -e "${BLUE}--show-all flag detected: will show both successful and failed resolutions${NC}"
else
    echo -e "${BLUE}Showing only failed DNS resolutions (use --show-all to see successful ones)${NC}"
fi
echo

# DNS resolution test function - uses same logic as check_domain.sh
# Returns true (0) if the hostname exists in DNS, false (1) if NXDOMAIN
check_dns() {
    local server="$1"
    local rcode

    # Get the DNS response code (NXDOMAIN, NOERROR, etc.)
    rcode=$(timeout 5 dig "$server" A +noall +comments 2>/dev/null | grep 'status:' | awk '{print $6}' | tr -d ',')

    # Only consider NXDOMAIN as a failure (domain doesn't exist)
    # NOERROR means domain exists, even if it has no A records
    [[ "$rcode" != "NXDOMAIN" ]]
}

# Create temporary file to store file:whoisServer pairs for batch processing
temp_file=$(mktemp)

echo -e "${BLUE}Scanning JSON files...${NC}"

# STEP 1: Find all JSON files that contain "whoisServer" key
# Using grep -l for speed (only returns filenames, doesn't read file contents)
# Then extract the actual whoisServer value using jq for each matching file
grep -l '"whoisServer"' metadata/*.json 2>/dev/null | while read -r file; do
    # Extract whoisServer value, handling cases where it might be null or missing
    whois_server=$(jq -r '.whoisServer // empty' "$file" 2>/dev/null)
    if [[ -n "$whois_server" && "$whois_server" != "null" ]]; then
        # Store as "filename:hostname" for later processing
        echo "$file:$whois_server" >> "$temp_file"
    fi
done

# Count how many whoisServer entries we found
file_count=$(wc -l < "$temp_file" 2>/dev/null || echo "0")

if [[ $file_count -eq 0 ]]; then
    echo "No files with whoisServer entries found."
    exit 0
fi

echo -e "${BLUE}Found $file_count files with whoisServer entries. Checking DNS resolution...${NC}"
echo

# STEP 2: Define function for concurrent DNS checking
# This function will be exported and run in parallel subprocesses
check_single_server() {
    local line="$1"
    # Parse the "filename:hostname" format
    IFS=':' read -r file whois_server <<< "$line"

    if [[ -z "$file" || -z "$whois_server" ]]; then return; fi

    # Use same DNS logic as check_domain.sh - only fail on NXDOMAIN
    local rcode
    rcode=$(timeout 5 dig "$whois_server" A +noall +comments 2>/dev/null | grep 'status:' | awk '{print $6}' | tr -d ',')

    # Only consider NXDOMAIN as a failure (domain doesn't exist)
    # NOERROR, SERVFAIL, REFUSED, etc. all mean the domain exists in some form
    if [[ "$rcode" != "NXDOMAIN" ]]; then
        echo "SUCCESS:$whois_server:$file"
    else
        echo "FAILED:$whois_server:$file"
    fi
}

# Export function so it's available to parallel subprocesses created by xargs
export -f check_single_server

# Create temporary file to store DNS check results
results_file=$(mktemp)
# Ensure both temp files are cleaned up when script exits
trap "rm -f $temp_file $results_file" EXIT

# STEP 3: Run DNS checks concurrently
# Use xargs -P 10 to run up to 10 DNS checks in parallel for speed
# Each line from temp_file becomes an argument to check_single_server function
# Results are captured in results_file for processing
cat "$temp_file" | xargs -I {} -P 10 bash -c 'check_single_server "{}"' > "$results_file"

# STEP 4: Process DNS check results and optionally modify JSON files
# Read results in format "STATUS:hostname:filename" and parse each line
while IFS=':' read -r status whois_server file; do
    if [[ -z "$status" ]]; then continue; fi  # Skip empty lines

    if [[ "$status" == "SUCCESS" ]]; then
        # DNS resolution succeeded - hostname has valid DNS records
        successful_dns=$((successful_dns + 1))
        # Only show successful results if --show-all flag is set
        if [[ "$SHOW_ALL" == "true" ]]; then
            echo -e "${GREEN}✓${NC} $whois_server ($(basename "$file"))"
        fi
    elif [[ "$status" == "FAILED" ]]; then
        # DNS resolution failed - hostname has no DNS records (likely doesn't exist)
        failed_dns=$((failed_dns + 1))
        # Always show failed results (this is what users care about)
        echo -e "${RED}✗${NC} $whois_server ($(basename "$file")) - DNS failed"

        # If --remove-broken flag is set, actually remove the whoisServer key from JSON
        if [[ "$REMOVE_BROKEN" == "true" ]]; then
            removed_servers=$((removed_servers + 1))

            # Use jq to create a new JSON file without the whoisServer key
            # This preserves all other data and maintains proper JSON formatting
            json_temp=$(mktemp)
            jq 'del(.whoisServer)' "$file" > "$json_temp"

            # Atomically replace the original file with the modified version
            mv "$json_temp" "$file"

            echo -e "${YELLOW}  Removed $whois_server from $(basename "$file")${NC}"
        fi
    fi
done < "$results_file"

# STEP 5: Display final summary statistics
echo
echo "Summary:"
echo "========="
echo "Files with whoisServer: $file_count"
echo -e "${GREEN}DNS resolution successful: $successful_dns${NC}"
echo -e "${RED}DNS resolution failed: $failed_dns${NC}"

# Show additional information based on flags used
if [[ "$REMOVE_BROKEN" == "true" ]]; then
    echo -e "${YELLOW}whoisServer entries removed: $removed_servers${NC}"
    if [[ $removed_servers -gt 0 ]]; then
        echo
        echo -e "${YELLOW}Modified files should be reviewed and committed.${NC}"
    fi
else
    # If user didn't use --remove-broken, remind them how to actually fix the issues
    if [[ $failed_dns -gt 0 ]]; then
        echo
        echo -e "${YELLOW}To remove failed whoisServer entries, run with --remove-broken flag${NC}"
    fi
fi