Skip to content

Instantly share code, notes, and snippets.

@girvan

girvan/gist:6017328

Last active May 3, 2017 08:57
Show Gist options
  • Save girvan/6017328 to your computer and use it in GitHub Desktop.
Save girvan/6017328 to your computer and use it in GitHub Desktop.
Download ZIP
bots request header
Raw
gistfile1.md

facebook

  'HTTPS: 
  USER_AGENT: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
  HOST: tw.piliapp.com
  ACCEPT: */*
  ACCEPT_ENCODING: deflate, gzip
  RANGE: bytes=0-524287
  CONNECTION: keep-alive

google plus one

  HOST: tw.piliapp.com
  CONNECTION: Keep-alive
  ACCEPT: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  USER_AGENT: Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google (+https://developers.google.com/+/web/snippet/)
  ACCEPT_ENCODING: gzip,deflate
  X_FORWARDED_FOR: 1.162.83.11

curl

  USER_AGENT: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
  HOST: tw.piliapp.com
  ACCEPT: */*

wget

  USER_AGENT: Wget/1.13.4 (linux-gnu)
  ACCEPT: */*
  HOST: tw.piliapp.com
  CONNECTION: Keep-Alive

google page speed

  HOST: tw.piliapp.com
  CACHE_CONTROL: no-store, no-cache
  X_FORWARDED_FOR: 1.162.83.11
  CONNECTION: Keep-alive
  ACCEPT: */*
  USER_AGENT: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, like Gecko; Google Page Speed Insights) Chrome/22.0.1229 Safari/537.4
  ACCEPT_ENCODING: gzip,deflate

google structure data testing tool

  HOST: tw.piliapp.com
  CONNECTION: Keep-alive
  USER_AGENT: Mozilla/5.0 (compatible; X11; Linux x86_64; Google-StructuredDataTestingTool; +http://www.google.com/webmasters/tools/richsnippets)
  ACCEPT_ENCODING: gzip,deflate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment