Skip to content

Instantly share code, notes, and snippets.

@pj8912
Created June 23, 2025 09:55
Show Gist options
  • Save pj8912/1a9c3f4a6c003f2fa3b412cfd3a4607c to your computer and use it in GitHub Desktop.
Save pj8912/1a9c3f4a6c003f2fa3b412cfd3a4607c to your computer and use it in GitHub Desktop.
robots.txt - prevent ai crawlers from crawling your site
User-agent: Googlebot-Image
Disallow: /
# AI scrapers
User-agent: CCBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: omgili
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: AdsBot-Google
Disallow: /
User-agent: AwarioRssBot
Disallow: /
User-agent: AwarioSmartBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: Meltwater
Disallow: /
User-agent: peer39_crawler
Disallow: /
User-agent: PiplBot
Disallow: /
User-agent: Seekr
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: SummalyBot
Disallow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment