Skip to content

Instantly share code, notes, and snippets.

@peterdalle

peterdalle/robots.txt

Created December 3, 2016 15:18
Show Gist options
  • Save peterdalle/302303fb67c2bb73a9a09df78c59ba1d to your computer and use it in GitHub Desktop.
Save peterdalle/302303fb67c2bb73a9a09df78c59ba1d to your computer and use it in GitHub Desktop.
Download ZIP
Robots.txt that makes sure Facebook and Twitter can crawl images on your site.
Raw
robots.txt
# Disallow everything.
User-agent: *
Disallow: /
# Certain social media sites are whitelisted to allow crawlers to access page markup when links to /images are shared.
User-agent: Twitterbot
Allow: /images
User-agent: facebookexternalhit
Allow: /images
@MetaMmodern
Copy link

Thank you, it workes

@meagencycloud
Copy link

nice thanks

@inspire22
Copy link

Facebook's facebookexternalhit doesn't follow robots.txt anyways I don't think

@sector07g
Copy link

@inspire22 yes, it work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment