Skip to content

Instantly share code, notes, and snippets.

@meskarune

meskarune/robots.txt

Created September 5, 2018 18:20
Show Gist options
  • Save meskarune/b32f340b3a5a7d7243fc5fae2b56607f to your computer and use it in GitHub Desktop.
Save meskarune/b32f340b3a5a7d7243fc5fae2b56607f to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt
# http://www.robotstxt.org/
# https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
User-agent: *
Disallow: /~* # block user web directories
Disallow: /private/*
Crawl-delay: 60 # slow bots down
Disallow: /*.pdf$ # block PDFs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment