Skip to content

Instantly share code, notes, and snippets.

@juliojsb
Created August 23, 2015 12:21
Show Gist options
  • Save juliojsb/f1197661f673bfa49b90 to your computer and use it in GitHub Desktop.
Save juliojsb/f1197661f673bfa49b90 to your computer and use it in GitHub Desktop.
Robots.txt file for my website (http://www.juliojosesanz.com/robots.txt) useful for other Wordpress sites
# ---------------------------------------------------
# CRAWLERS/BOTS NOT ALLOWED
# ---------------------------------------------------
# Digg Mirror (Blocks Digg crawler)
User-agent: duggmirror
Disallow: /
# Alexa archiver
User-agent: ia_archiver
Disallow: /
# ---------------------------------------------------
# CRAWLERS/BOTS ALLOWED
# ---------------------------------------------------
# Twitterbot
User-agent: Twitterbot
Disallow:
# Google Image
User-agent: Googlebot-Image
Disallow:
# Google AdSense
User-agent: Mediapartners-Google
Disallow:
# Rules for the rest of crawlers/bots
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-content/languages/
Allow: /wp-content/uploads/
# ---------------------------------------------------
# SITEMAP
# ---------------------------------------------------
Sitemap: http://www.juliojosesanz.com/sitemap.xml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment