Skip to content

Instantly share code, notes, and snippets.

@bencooling

bencooling/Server: robots.txt

Created May 14, 2013 03:04
Show Gist options
  • Save bencooling/5573364 to your computer and use it in GitHub Desktop.
Save bencooling/5573364 to your computer and use it in GitHub Desktop.
Download ZIP
Server: robots.txt
Raw
Server: robots.txt
// Create robots file
$ touch robots.txt
// Block all robots
User-agent: *
Disallow: /
// Block a single page
Disallow: /private_file.html
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment