Created
June 4, 2018 17:50
-
-
Save willboudle/9efcd1908c7aae3c669de8f0c8f8e06d to your computer and use it in GitHub Desktop.
Magento Robots.txt 2 - Blue Acorn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| User-agent: * | |
| Disallow: /index.php/ | |
| Disallow: /*? | |
| Disallow: /*.js$ | |
| Disallow: /*.css$ | |
| Disallow: /customer/ | |
| Disallow: /checkout/ | |
| Disallow: /js/ | |
| Disallow: /lib/ | |
| Disallow: /media/ | |
| Allow: /media/catalog/product/ | |
| Disallow: /*.php$ | |
| Disallow: /skin/ | |
| Disallow: /catalog/product/view/ | |
| User-agent: Googlebot-Image | |
| Disallow: / | |
| Allow: /media/catalog/product/ | |
| Sitemap: http://example.com/sitemap/sitemap.xml |
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Crawlers Setup
User-agent: *
Crawl-delay: 10
Allowable Index
Allow: /*?p=
Allow: /media/
Directories
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /includes/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Paths (clean URLs)
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Files
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt
Paths (no clean URLs)
Disallow: /.js$
Disallow: /.css$
Disallow: /.php$
Disallow: /?p=&
Disallow: /?SID=