Skip to content

Instantly share code, notes, and snippets.

@thegulshankumar

thegulshankumar/robots.txt

Last active November 18, 2019 23:32
Show Gist options
  • Save thegulshankumar/ba7db849ade688a7161ad5c93ddd5ad1 to your computer and use it in GitHub Desktop.
Save thegulshankumar/ba7db849ade688a7161ad5c93ddd5ad1 to your computer and use it in GitHub Desktop.
Download ZIP
Idle robots.txt for WordPress
Raw
robots.txt
Sitemap: https://www.example.com/sitemap_index.xml
User-agent: *
Disallow: /?s=*
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Allow: /wp-admin/images/
Allow: /wp-admin/css/
Allow: /wp-admin/js/
@thegulshankumar
Copy link
Author

thegulshankumar commented Apr 5, 2017 •
edited
Loading

Line by line Explanation

  1. Sitemap Declared
  2. For all user-agents
  3. Blocked Search Results
  4. Blocked administrative path
  5. Allowed ajax theme elements
  6. Allowed wp-admin images path for Embed content
  7. Allowed wp-admin CSS
  8. Allowed wp-admin JS

[The robots.txt rule is written as per Google Mobile Friendly guidelines which says, do not block CSS and JS path.]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment