Skip to content

Instantly share code, notes, and snippets.

@mistergraphx
Created July 15, 2020 10:47
Show Gist options
  • Save mistergraphx/0c2303e16be5c3ecb9fdd320aff8c3d9 to your computer and use it in GitHub Desktop.
Save mistergraphx/0c2303e16be5c3ecb9fdd320aff8c3d9 to your computer and use it in GitHub Desktop.
optimized robots.txt for wp
# Sample robots.txt
# replace domain.com with your website url
User-agent: *
Sitemap: https://domain.com/page-sitemap.xml
# Sitemap: https://domain.com/program-sitemap.xml
# Sitemap: https://domain.com/campus-sitemap.xml
# Sitemap: https://domain.com/event-sitemap.xml
# Sitemap: https://domain.com/campus-sitemap.xml
# Sitemap: https://domain.com/post-sitemap.xml
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-admin
Disallow: /wp-includes
Allow: /wp-content/plugins/
Allow: /wp-content/themes/
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /category/
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?s=
Disallow: /*?p=
Disallow: /*.php$
Allow: /*.js$
Allow: /*.css$
Disallow: /*.inc$
Disallow: /*.gz$
Disallow: /*.swf$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment