Skip to content

Instantly share code, notes, and snippets.

@timelf123

timelf123/robots.txt

Forked from underdown/robots.txt
Created October 8, 2012 04:18
Show Gist options
  • Save timelf123/3850693 to your computer and use it in GitHub Desktop.
Save timelf123/3850693 to your computer and use it in GitHub Desktop.
Download ZIP
default robots.txt
Raw
robots.txt
User-agent: ia_archiver
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: *
Allow: /
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment