Skip to content

Instantly share code, notes, and snippets.

@denzildoyle

denzildoyle/robots.txt

Last active August 29, 2015 14:07
Show Gist options
  • Save denzildoyle/2d336a6cb68e204cddf6 to your computer and use it in GitHub Desktop.
Save denzildoyle/2d336a6cb68e204cddf6 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
robots.txt
Allow indexing of everything
User-agent: *
Disallow:
or
User-agent: *
Allow: /
Disallow indexing of everything
User-agent: *
Disallow: /
Disawllow indexing of a specific folder
User-agent: *
Disallow: /folder/
Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder
User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment