JoeyBurzynski · October 22, 2018 16:25
diff --git a/crawler-user-agent-strings-with-descriptions.txt b/crawler-user-agent-strings-with-descriptions.txt
 Googlebot/2.1 (+http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview Analytics) Chrome/27.0.1453 Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot\/ / URL: http://www.google.com/bot.html
 DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot-Mobile
 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot-Mobile
 Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot-Mobile
 Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot-Mobile
 SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)  # Pattern: Googlebot-Mobile
 Googlebot-Image/1.0  # Pattern: Googlebot-Image
 Googlebot-News  # Pattern: Googlebot-News
 Googlebot-Video/1.0  # Pattern: Googlebot-Video
 AdsBot-Google (+http://www.google.com/adsbot.html)  # Pattern: AdsBot-Google([^-]|$) / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 AdsBot-Google-Mobile-Apps  # Pattern: AdsBot-Google-Mobile / URL: https://support.google.com/adwords/answer/2404197
 Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)  # Pattern: AdsBot-Google-Mobile / URL: https://support.google.com/adwords/answer/2404197
 Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)  # Pattern: AdsBot-Google-Mobile / URL: https://support.google.com/adwords/answer/2404197
 Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=728742641706423)  # Pattern: Feedfetcher-Google / URL: https://support.google.com/webmasters/answer/178852
 Mediapartners-Google  # Pattern: Mediapartners-Google / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)  # Pattern: Mediapartners-Google / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)  # Pattern: Mediapartners-Google / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)  # Pattern: Mediapartners-Google / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)  # Pattern: APIs-Google / URL: https://support.google.com/webmasters/answer/1061943?hl=en
 Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0;  http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0;  http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)  # Pattern: bingbot / URL: http://www.bing.com/bingbot.htm
 Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)  # Pattern: Slurp / URL: http://help.yahoo.com/help/us/ysearch/slurp
 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)  # Pattern: Slurp / URL: http://help.yahoo.com/help/us/ysearch/slurp
 Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)  # Pattern: Slurp / URL: http://help.yahoo.com/help/us/ysearch/slurp
 WGETbot/1.0 (+http://wget.alanreed.org)  # Pattern: [wW]get
 Wget/1.14 (linux-gnu)  # Pattern: [wW]get
 eCairn-Grabber/1.0 (+http://ecairn.com/grabber) curl/7.15  # Pattern: curl
 LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)  # Pattern: LinkedInBot
 LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)  # Pattern: LinkedInBot
 Python-urllib/2.5  # Pattern: Python-urllib
 Python-urllib/2.5  # Pattern: Python-urllib
 Python-urllib/2.6  # Pattern: Python-urllib
 Python-urllib/2.7  # Pattern: Python-urllib
 Python-urllib/3.1  # Pattern: Python-urllib
 Python-urllib/3.2  # Pattern: Python-urllib
 Python-urllib/3.3  # Pattern: Python-urllib
 Python-urllib/3.4  # Pattern: Python-urllib
 Python-urllib/3.5  # Pattern: Python-urllib
 Python-urllib/3.6  # Pattern: Python-urllib
 python-requests/2.18.4  # Pattern: python-requests
 2Bone_LinkChecker/1.0 libwww-perl/6.03  # Pattern: libwww
 2Bone_LinkChkr/1.0 libwww-perl/6.03  # Pattern: libwww
 W3C-checklink/2.90 libwww-perl/5.64  # Pattern: libwww
 W3C-checklink/3.6.2.3 libwww-perl/5.64  # Pattern: libwww
 W3C-checklink/4.2 [4.20] libwww-perl/5.803  # Pattern: libwww
 W3C-checklink/4.2.1 [4.21] libwww-perl/5.803  # Pattern: libwww
 W3C-checklink/4.3 [4.42] libwww-perl/5.805  # Pattern: libwww
 W3C-checklink/4.3 [4.42] libwww-perl/5.808  # Pattern: libwww
 W3C-checklink/4.3 [4.42] libwww-perl/5.820  # Pattern: libwww
 W3C-checklink/4.5 [4.154] libwww-perl/5.823  # Pattern: libwww
 W3C-checklink/4.5 [4.160] libwww-perl/5.823  # Pattern: libwww
 amibot - http://www.amidalla.de - [email protected] libwww-perl/5.831  # Pattern: libwww
 httpunit/1.x  # Pattern: httpunit
 NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; [email protected])  # Pattern: nutch
 istellabot-nutch/Nutch-1.10  # Pattern: nutch
 Go-http-client/1.1  # Pattern: Go-http-client / URL: https://golang.org/pkg/net/http/
 phpcrawl  # Pattern: phpcrawl / URL: http://phpcrawl.cuab.de/
 adidxbot/1.1 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 adidxbot/2.0 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 librabot/1.0 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 librabot/2.0 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot/1.0 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot/1.1 (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot/2.0b (+http://search.msn.com/msnbot.htm)  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot/2.0b (+http://search.msn.com/msnbot.htm).  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 msnbot/2.0b (+http://search.msn.com/msnbot.htm)._  # Pattern: msnbot / URL: http://search.msn.com/msnbot.htm
 FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)  # Pattern: FAST-WebCrawler
 FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)  # Pattern: FAST-WebCrawler
 FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)  # Pattern: FAST-WebCrawler
 FAST-WebCrawler/3.8  # Pattern: FAST-WebCrawler
 FAST Enterprise Crawler 6 / Scirus [email protected]; http://www.scirus.com/srsapp/contactus/  # Pattern: FAST Enterprise Crawler
 FAST Enterprise Crawler 6 used by Schibsted ([email protected])  # Pattern: FAST Enterprise Crawler
 BIGLOTRON (Beta 2;GNU/Linux)  # Pattern: BIGLOTRON
 Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)  # Pattern: Teoma / URL: http://about.ask.com/en/docs/about/webmasters.shtml
 Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)  # Pattern: Teoma / URL: http://about.ask.com/en/docs/about/webmasters.shtml
 ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)  # Pattern: convera / URL: http://ews.converasearch.com/crawl.htm
 Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2  # Pattern: seekbot / URL: http://www.seekbot.net/bot.html
 Gigabot/1.0  # Pattern: Gigabot / URL: http://www.gigablast.com/spider.html
 Gigabot/2.0 (http://www.gigablast.com/spider.html)  # Pattern: Gigabot / URL: http://www.gigablast.com/spider.html
 GigablastOpenSource/1.0  # Pattern: Gigablast / URL: https://github.com/gigablast/open-source-search-engine
 Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; [email protected])  # Pattern: exabot
 Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)  # Pattern: exabot
 Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)  # Pattern: exabot
 Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)  # Pattern: exabot
 Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)  # Pattern: exabot
 ia_archiver (+http://www.alexa.com/site/help/webmasters; [email protected])  # Pattern: ia_archiver
 ia_archiver-web.archive.org  # Pattern: ia_archiver
 GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)  # Pattern: GingerCrawler
 Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)  # Pattern: HTTrack
 Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)  # Pattern: grub.org
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)  # Pattern: speedy
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)  # Pattern: speedy
 Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)  # Pattern: speedy
 Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)  # Pattern: speedy
 Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)  # Pattern: speedy
 Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)  # Pattern: bibnum.bnf
 findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)  # Pattern: findlink
 findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)  # Pattern: findlink
 panscient.com  # Pattern: panscient
 yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html  # Pattern: yacybot
 BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent  # Pattern: ips-agent
 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7  # Pattern: ips-agent
 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4  Firefox/3.5.3  # Pattern: ips-agent
 Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24  # Pattern: ips-agent
 Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1  # Pattern: ips-agent
 MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://www.majestic12.co.uk/bot.php?+)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)  # Pattern: MJ12bot
 Mozilla/5.0 (compatible; woriobot +http://worio.com)  # Pattern: woriobot
 Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)  # Pattern: woriobot
 Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)  # Pattern: yanga
 Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; [email protected])  # Pattern: buzzbot
 MLBot (www.metadatalabs.com/mlbot)  # Pattern: mlbot
 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)  # Pattern: YandexBot / URL: http://yandex.com/bots
 Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)  # Pattern: yandex.com\/bots / URL: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml#robot-in-logs
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)  # Pattern: yandex.com\/bots / URL: https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.xml#robot-in-logs
 Linguee Bot (http://www.linguee.com/bot)  # Pattern: Linguee Bot / URL: http://www.linguee.com/bot
 Linguee Bot (http://www.linguee.com/bot; [email protected])  # Pattern: Linguee Bot / URL: http://www.linguee.com/bot
 CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)  # Pattern: CyberPatrol / URL: http://www.cyberpatrol.com/cyberpatrolcrawler.asp
 Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])  # Pattern: voilabot
 Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 ([email protected])  # Pattern: voilabot
 Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected])  # Pattern: voilabot
 Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)  # Pattern: Baiduspider / URL: http://www.baidu.jp/spider/
 Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )  # Pattern: spbot / URL: http://www.seoprofiler.com/bot
 PostRank/2.0 (postrank.com)  # Pattern: postrank / URL: http://www.postrank.com
 PostRank/2.0 (postrank.com; 1 subscribers)  # Pattern: postrank / URL: http://www.postrank.com
 Mozilla/5.0 (compatible;  Page2RSS/0.7; +http://page2rss.com/)  # Pattern: page2rss / URL: http://www.page2rss.com
 Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)  # Pattern: sitebot / URL: http://www.sitebot.org
 Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)  # Pattern: linkdex / URL: http://www.linkdex.com
 Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)  # Pattern: linkdex / URL: http://www.linkdex.com
 Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)  # Pattern: linkdex / URL: http://www.linkdex.com
 Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)  # Pattern: linkdex / URL: http://www.linkdex.com
 Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)  # Pattern: linkdex / URL: http://www.linkdex.com
 linkdex.com/v2.0  # Pattern: linkdex / URL: http://www.linkdex.com
 linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)  # Pattern: linkdex / URL: http://www.linkdex.com
 Mozilla/5.0 (compatible; Blekkobot; ScoutJet; +http://blekko.com/about/blekkobot)  # Pattern: blekkobot / URL: http://blekko.com/about/blekkobot
 Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])  # Pattern: ezooms / URL: http://www.phpbb.com/community/viewtopic.php?f=64&t=935605&start=450#p12948289
 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])  # Pattern: dotbot
 dotbot  # Pattern: dotbot
 Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/  # Pattern: Mail.RU_Bot
 Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/  # Pattern: Mail.RU_Bot
 Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)  # Pattern: discobot / URL: http://discoveryengine.com/discobot.html
 Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)  # Pattern: discobot / URL: http://discoveryengine.com/discobot.html
 mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)  # Pattern: discobot / URL: http://discoveryengine.com/discobot.html
 Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)  # Pattern: heritrix / URL: http://crawler.archive.org/
 Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)  # Pattern: europarchive.org / URL: 
 Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)  # Pattern: NerdByNature.Bot / URL: http://www.nerdbynature.net/bot
 Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)  # Pattern: Ahrefs(Bot|SiteAudit)
 Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/)  # Pattern: Ahrefs(Bot|SiteAudit)
 fuelbot  # Pattern: fuelbot
 CrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot)  # Pattern: CrunchBot
 Mozilla/5.0 (compatible; Go-http-client/1.1; [email protected])  # Pattern: centurybot9
 Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)  # Pattern: IndeedBot
 Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)  # Pattern: mappydata
 woobot  # Pattern: woobot
 ZoominfoBot (zoominfobot at zoominfo dot com)  # Pattern: ZoominfoBot
 Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)  # Pattern: PrivacyAwareBot
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot  # Pattern: Multiviewbot
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot  # Pattern: SWIMGBot
 Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)  # Pattern: Grobbot
 Mozilla/5.0 (compatible; eright/1.0; [email protected])  # Pattern: eright
 Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)  # Pattern: Apercite
 semanticbot  # Pattern: semanticbot
 semanticbot ([email protected])  # Pattern: semanticbot
 Aboundex/0.2 (http://www.aboundex.com/crawler/)  # Pattern: Aboundex / URL: http://www.aboundex.com/crawler/
 Aboundex/0.3 (http://www.aboundex.com/crawler/)  # Pattern: Aboundex / URL: http://www.aboundex.com/crawler/
 CipaCrawler/3.0 ([email protected]; http://www.domaincrawler.com/www.example.com)  # Pattern: domaincrawler
 Summify (Summify/1.0.1; +http://summify.com)  # Pattern: summify / URL: http://summify.com
 CCBot/2.0 (http://commoncrawl.org/faq/)  # Pattern: CCBot / URL: http://www.commoncrawl.org/bot.html
 Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)  # Pattern: seznambot
 Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)  # Pattern: seznambot
 Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)  # Pattern: seznambot
 Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)  # Pattern: seznambot
 Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)  # Pattern: seznambot
 ec2linkfinder  # Pattern: ec2linkfinder
 Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)  # Pattern: aiHitBot
 facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)  # Pattern: facebookexternalhit
 facebookexternalhit/1.1  # Pattern: facebookexternalhit
 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)  # Pattern: facebookexternalhit
 Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)  # Pattern: Yeti / URL: http://naver.me/bot
 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)  # Pattern: RetrevoPageAnalyzer
 Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)  # Pattern: Sogou / URL: http://www.sogou.com/docs/help/webmasters.htm#07
 Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)  # Pattern: Sogou / URL: http://www.sogou.com/docs/help/webmasters.htm#07
 Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)  # Pattern: Sogou / URL: http://www.sogou.com/docs/help/webmasters.htm#07
 Wotbox/2.0 ([email protected]; http://www.wotbox.com)  # Pattern: wotbox / URL: http://www.wotbox.com
 Wotbox/2.01 (+http://www.wotbox.com/bot/)  # Pattern: wotbox / URL: http://www.wotbox.com
 DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/2.0 ([email protected])  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)  # Pattern: ichiro / URL: http://help.goo.ne.jp/help/article/1142
 DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)  # Pattern: DuckDuckBot / URL: http://duckduckgo.com/duckduckbot.html
 DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)  # Pattern: DuckDuckBot / URL: http://duckduckgo.com/duckduckbot.html
 drupact/0.7; http://www.arocom.de/drupact  # Pattern: drupact / URL: http://www.arocom.de/drupact
 Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 coccoc  # Pattern: coccoc / URL: http://help.coccoc.vn/
 coccoc/1.0 ()  # Pattern: coccoc / URL: http://help.coccoc.vn/
 coccoc/1.0 (http://help.coccoc.com/)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 coccoc/1.0 (http://help.coccoc.vn/)  # Pattern: coccoc / URL: http://help.coccoc.vn/
 www.integromedb.org/Crawler  # Pattern: integromedb / URL: http://www.integromedb.org/Crawler
 it2media-domain-crawler/1.0 on crawler-prod.it2media.de  # Pattern: it2media-domain-crawler
 it2media-domain-crawler/2.0  # Pattern: it2media-domain-crawler
 Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)  # Pattern: siteexplorer.info
 Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)  # Pattern: siteexplorer.info
 Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)  # Pattern: proximic / URL: http://www.proximic.com/info/spider.php
 Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)  # Pattern: proximic / URL: http://www.proximic.com/info/spider.php
 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1;  http://www.changedetection.com/bot.html )  # Pattern: changedetection / URL: http://www.changedetection.com/bot.html
 WeSEE:Search  # Pattern: WeSEE:Search
 WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)  # Pattern: WeSEE:Search
 Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected]  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://www.moz.com/dp/rogerbot, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, [email protected])  # Pattern: rogerbot / URL: http://moz.com/help/pro/what-is-rogerbot-
 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; )  Firefox/1.5.0.11; 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11)  Firefox/1.5.0.11; 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)  # Pattern: 360Spider / URL: http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html
 psbot-image (+http://www.picsearch.com/bot.html)  # Pattern: psbot / URL: http://www.picsearch.com/bot.html
 psbot-page (+http://www.picsearch.com/bot.html)  # Pattern: psbot / URL: http://www.picsearch.com/bot.html
 psbot/0.1 (+http://www.picsearch.com/bot.html)  # Pattern: psbot / URL: http://www.picsearch.com/bot.html
 CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper  # Pattern: CC Metadata Scaper / URL: http://wiki.creativecommons.org/Metadata_Scraper
 Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)  # Pattern: GrapeshotCrawler / URL: http://www.grapeshot.co.uk/crawler.php
 Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)  # Pattern: urlappendbot / URL: http://www.profound.net/urlappendbot.html
 Mozilla/5.0 (compatible; fr-crawler/1.1)  # Pattern: fr-crawler
 binlar_2.6.3 [email protected]  # Pattern: binlar
 binlar_2.6.3 [email protected]  # Pattern: binlar
 binlar_2.6.3 [email protected]  # Pattern: binlar
 binlar_2.6.3 [email protected]  # Pattern: binlar
 binlar_2.6.3 [email protected]  # Pattern: binlar
 SimpleCrawler/0.1  # Pattern: SimpleCrawler
 Twitterbot/0.1  # Pattern: Twitterbot / URL: https://dev.twitter.com/cards/getting-started
 Twitterbot/1.0  # Pattern: Twitterbot / URL: https://dev.twitter.com/cards/getting-started
 cXensebot/1.1a  # Pattern: cXensebot / URL: http://www.cxense.com/bot.html
 Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)  # Pattern: smtbot / URL: http://www.similartech.com/smtbot
 SMTBot (similartech.com/smtbot)  # Pattern: smtbot / URL: http://www.similartech.com/smtbot
 Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)  # Pattern: bnf.fr_bot / URL: http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html
 A6-Indexer  # Pattern: A6-Indexer / URL: http://www.a6corp.com/a6-web-scraping-policy/
 ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected]  # Pattern: ADmantX / URL: http://www.admantx.com
 Facebot/1.0  # Pattern: Facebot / URL: https://developers.facebook.com/docs/sharing/best-practices#crawl
 Mozilla/5.0 (compatible; OrangeBot/2.0; [email protected]  # Pattern: OrangeBot\/
 Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)  # Pattern: memorybot / URL: http://mignify.com/bot.htm
 Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)  # Pattern: AdvBot / URL: http://advbot.net/bot.html
 Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)  # Pattern: MegaIndex / URL: https://www.megaindex.ru/?tab=linkAnalyze
 SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)  # Pattern: SemanticScholarBot / URL: http://s2.allenai.org/bot.html
 ltx71 - (http://ltx71.com/)  # Pattern: ltx71 / URL: http://ltx71.com/
 nerdybot  # Pattern: nerdybot / URL: http://nerdybot.com/
 Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)  # Pattern: xovibot / URL: http://www.xovibot.net/
 BUbiNG (+http://law.di.unimi.it/BUbiNG.html)  # Pattern: BUbiNG / URL: http://law.di.unimi.it/BUbiNG.html
 Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*  # Pattern: Qwantify / URL: https://www.qwant.com/
 Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)  # Pattern: archive.org_bot / URL: http://www.archive.org/details/archive.org_bot
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)  # Pattern: Applebot / URL: http://www.apple.com/go/applebot
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)  # Pattern: Applebot / URL: http://www.apple.com/go/applebot
 Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)  # Pattern: Applebot / URL: http://www.apple.com/go/applebot
 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)  # Pattern: Applebot / URL: http://www.apple.com/go/applebot
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)  # Pattern: Applebot / URL: http://www.apple.com/go/applebot
 Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0  # Pattern: TweetmemeBot / URL: http://datasift.com/bot.html
 crawler4j (http://code.google.com/p/crawler4j/)  # Pattern: crawler4j / URL: https://github.com/yasserg/crawler4j
 Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)  # Pattern: findxbot / URL: http://www.findxbot.com
 Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)  # Pattern: S[eE][mM]rushBot / URL: http://www.semrush.com/bot.html
 SEMrushBot  # Pattern: S[eE][mM]rushBot / URL: http://www.semrush.com/bot.html
 Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; [email protected])  # Pattern: yoozBot / URL: http://yooz.ir
 Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)  # Pattern: lipperhey / URL: http://www.lipperhey.com/
 Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)  # Pattern: lipperhey / URL: http://www.lipperhey.com/
 Mozilla/5.0 (compatible; Lipperhey Site Explorer; http://www.lipperhey.com/)  # Pattern: lipperhey / URL: http://www.lipperhey.com/
 Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)  # Pattern: lipperhey / URL: http://www.lipperhey.com/
 Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Y!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html))  # Pattern: Y!J / URL: https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6
 Domain Re-Animator Bot (http://domainreanimator.com) - [email protected]  # Pattern: Domain Re-Animator Bot / URL: http://domainreanimator.com
 AddThis.com robot [email protected]  # Pattern: AddThis / URL: https://www.addthis.com
 Screaming Frog SEO Spider/5.1  # Pattern: Screaming Frog SEO Spider / URL: http://www.screamingfrog.co.uk/seo-spider
 MetaURI API/2.0 +metauri.com  # Pattern: MetaURI / URL: http://www.useragentstring.com/MetaURI_id_17683.php
 Scrapy/1.0.3 (+http://scrapy.org)  # Pattern: Scrapy / URL: http://scrapy.org/
 LivelapBot/0.2 (http://site.livelap.com/crawler)  # Pattern: Livelap[bB]ot / URL: http://site.livelap.com/crawler
 Livelapbot/0.1  # Pattern: Livelap[bB]ot / URL: http://site.livelap.com/crawler
 Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)  # Pattern: OpenHoseBot / URL: http://www.openhose.org/bot.html
 CapsuleChecker (http://www.capsulink.com/)  # Pattern: CapsuleChecker / URL: http://www.capsulink.com/about
 Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 [email protected]  # Pattern: [email protected] / URL: http://infegy.com/
 Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)  # Pattern: IstellaBot / URL: http://www.tiscali.it/
 Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)  # Pattern: DeuSu\/ / URL: https://deusu.de/robot.html
 Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)  # Pattern: DeuSu\/ / URL: https://deusu.de/robot.html
 Cliqzbot/0.1 (+http://cliqz.com [email protected])  # Pattern: Cliqzbot\/ / URL: http://cliqz.com/company/cliqzbot
 Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)  # Pattern: Cliqzbot\/ / URL: http://cliqz.com/company/cliqzbot
 Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)  # Pattern: Cliqzbot\/ / URL: http://cliqz.com/company/cliqzbot
 Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)  # Pattern: Cliqzbot\/ / URL: http://cliqz.com/company/cliqzbot
 MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)  # Pattern: MojeekBot\/ / URL: https://www.mojeek.com/bot.html
 netEstate NE Crawler (+http://www.sengine.info/)  # Pattern: netEstate NE Crawler / URL: +http://www.website-datenbank.de/
 netEstate NE Crawler (+http://www.website-datenbank.de/)  # Pattern: netEstate NE Crawler / URL: +http://www.website-datenbank.de/
 SafeSearch microdata crawler (https://safesearch.avira.com, [email protected])  # Pattern: SafeSearch microdata crawler / URL: https://safesearch.avira.com
 Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)  # Pattern: Gluten Free Crawler\/ / URL: http://glutenfreepleasure.com/
 Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)  # Pattern: Sonic / URL: http://www.yama.info.waseda.ac.jp/~crawler/info.html
 Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)  # Pattern: Sonic / URL: http://www.yama.info.waseda.ac.jp/~crawler/info.html
 Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)  # Pattern: Sonic / URL: http://www.yama.info.waseda.ac.jp/~crawler/info.html
 Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)  # Pattern: Sysomos / URL: http://www.sysomos.com
 www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36  # Pattern: deadlinkchecker / URL: http://www.deadlinkchecker.com
 www.deadlinkchecker.com XMLHTTP/1.0  # Pattern: deadlinkchecker / URL: http://www.deadlinkchecker.com
 www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36  # Pattern: deadlinkchecker / URL: http://www.deadlinkchecker.com
 Slack-ImgProxy (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Slack-ImgProxy 0.59 (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Slack-ImgProxy 0.66 (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Slack-ImgProxy 1.106 (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Slack-ImgProxy 1.138 (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Slack-ImgProxy 149 (+https://api.slack.com/robots)  # Pattern: Slack-ImgProxy / URL: https://api.slack.com/robots
 Embedly [email protected]  # Pattern: Embedly / URL: http://support.embed.ly
 Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)  # Pattern: Embedly / URL: http://support.embed.ly
 Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)  # Pattern: Embedly / URL: http://support.embed.ly
 Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)  # Pattern: RankActiveLinkBot / URL: https://rankactive.com/resources/rankactive-linkbot
 iskanie (+http://www.iskanie.com)  # Pattern: iskanie / URL: http://www.iskanie.com
 SafeDNSBot (https://www.safedns.com/searchbot)  # Pattern: SafeDNSBot / URL: https://www.safedns.com/searchbot
 Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5  # Pattern: SkypeUriPreview
 Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)  # Pattern: Veoozbot / URL: http://www.veooz.com/veoozbot.html
 Slackbot-LinkExpanding (+https://api.slack.com/robots)  # Pattern: Slackbot / URL: https://api.slack.com/robots
 Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)  # Pattern: Slackbot / URL: https://api.slack.com/robots
 Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)  # Pattern: redditbot / URL: http://www.reddit.com/feedback
 datagnionbot (+http://www.datagnion.com/bot.html)  # Pattern: datagnionbot / URL: http://www.datagnion.com/bot.html
 Google-Adwords-Instant (+http://www.google.com/adsbot.html)  # Pattern: Google-Adwords-Instant / URL: http://www.google.com/adsbot.html
 Mozilla/5.0 (compatible; adbeat_bot; [email protected]; [email protected])  # Pattern: adbeat_bot
 adbeat_bot  # Pattern: adbeat_bot
 WhatsApp  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.15/i  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.16/i  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.17/i  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.449 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.453 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.510 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.540 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.548 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.555 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.12.556 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.16.1/i  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.16.13 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.16.2/i  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.16.42 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 WhatsApp/2.16.57 A  # Pattern: WhatsApp / URL: https://www.whatsapp.com/
 Mozilla/5.0 (compatible;contxbot/1.0)  # Pattern: contxbot
 Pinterest/0.2 (+http://www.pinterest.com/bot.html)  # Pattern: pinterest / URL: http://www.pinterest.com/bot.html
 Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)  # Pattern: electricmonk / URL: https://www.duedil.com/our-crawler/
 GarlikCrawler/1.2 (http://garlik.com/, [email protected])  # Pattern: GarlikCrawler / URL: http://garlik.com/
 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b  # Pattern: BingPreview\/ / URL: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
 Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko  # Pattern: BingPreview\/ / URL: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
 Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0;  WOW64;  Trident/6.0;  BingPreview/1.0b)  # Pattern: BingPreview\/ / URL: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;  WOW64;  Trident/5.0;  BingPreview/1.0b)  # Pattern: BingPreview\/ / URL: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
 Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b  # Pattern: BingPreview\/ / URL: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
 Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/  # Pattern: vebidoobot / URL: https://blog.vebidoo.de/vebidoobot/
 Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)  # Pattern: FemtosearchBot / URL: http://femtosearch.com
 Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)  # Pattern: Yahoo Link Preview / URL: https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html
 Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)  # Pattern: MetaJobBot / URL: http://www.metajob.de/the/crawler
 DomainStatsBot/1.0 (http://domainstats.io/our-bot)  # Pattern: DomainStatsBot / URL: http://domainstats.io/our-bot
 mindUpBot (datenbutler.de)  # Pattern: mindUpBot / URL: http://www.datenbutler.de/
 Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)  # Pattern: Daum\/ / URL: http://cs.daum.net/faq/15/4118.html?faqId=28966
 Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de  # Pattern: Jugendschutzprogramm-Crawler / URL: http://www.jugendschutzprogramm.de
 Xenu Link Sleuth/1.3.8  # Pattern: Xenu Link Sleuth / URL: http://home.snafu.de/tilman/xenulink.html
 Pcore-HTTP/v0.40.3  # Pattern: Pcore-HTTP / URL: https://bitbucket.org/softvisio/pcore/overview
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot  # Pattern: moatbot / URL: https://moat.com
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/600.1.3 (KHTML, like Gecko) Version/8.0 Mobile/12A4345d Safari/600.1.4 moatbot  # Pattern: moatbot / URL: https://moat.com
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +http://kosm.io/bot.html)  # Pattern: KosmioBot / URL: http://kosm.io/bot.html
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)  # Pattern: pingdom / URL: http://www.pingdom.com
 Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)  # Pattern: pingdom / URL: http://www.pingdom.com
 Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)  # Pattern: PhantomJS / URL: http://phantomjs.org/
 Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)  # Pattern: Gowikibot / URL: http://www.gowikibot.com
 Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)  # Pattern: PiplBot / URL: http://www.pipl.com/bot/
 Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)  # Pattern: Discordbot / URL: https://discordapp.com
 TelegramBot (like TwitterBot)  # Pattern: TelegramBot
 Mozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)  # Pattern: Jetslide / URL: http://jetsli.de/crawler
 Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)  # Pattern: newsharecounts / URL: http://newsharecounts.com/crawler
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html  # Pattern: James BOT / URL: http://cognitiveseo.com/bot.html
 Barkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) http://www.exensa.com/crawl - [email protected] -- based on BuBiNG  # Pattern: Barkrowler / URL: http://www.exensa.com/crawl
 Barkrowler/0.7 (+http://www.exensa.com/crawl)  # Pattern: Barkrowler / URL: http://www.exensa.com/crawl
 Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)  # Pattern: TinEye / URL: http://www.tineye.com/crawler.html
 TinEye/1.1 (http://tineye.com/crawler.html)  # Pattern: TinEye / URL: http://www.tineye.com/crawler.html
 SocialRankIOBot; http://socialrank.io/about  # Pattern: SocialRankIOBot / URL: http://socialrank.io/about
 Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11  # Pattern: trendictionbot / URL: http://www.trendiction.de/bot
 Ocarinabot  # Pattern: Ocarinabot
 Mozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot)  # Pattern: epicbot / URL: http://www.epictions.com/epicbot
 Mozilla/5.0 (compatible; Primalbot; +https://www.primal.com;)  # Pattern: Primalbot / URL: https://www.primal.com
 Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)  # Pattern: DuckDuckGo-Favicons-Bot / URL: http://duckduckgo.com
 Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com  # Pattern: GnowitNewsbot / URL: http://www.gnowit.com
 Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)  # Pattern: Leikibot / URL: http://www.leiki.com
 @LinkArchiver twitter bot  # Pattern: LinkArchiver
 Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])  # Pattern: YaK\/ / URL: http://linkfluence.com
 Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)  # Pattern: PaperLiBot / URL: http://support.paper.li/entries/20023257-what-is-paper-li
 Digg Deeper/v1 (http://digg.com/about)  # Pattern: Digg Deeper / URL: http://digg.com/about
 dcrawl/1.0  # Pattern: dcrawl
 Mozilla/5.0 (compatible; Snacktory; +https://github.com/karussell/snacktory)  # Pattern: Snacktory / URL: https://github.com/karussell/snacktory
 Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)  # Pattern: AndersPinkBot / URL: http://anderspink.com/bot.html
 Fyrebot/1.0  # Pattern: Fyrebot
 Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; [email protected] http://everyonesocial.com/)  # Pattern: EveryoneSocialBot / URL: http://everyonesocial.com
 Mediatoolkitbot ([email protected])  # Pattern: Mediatoolkitbot / URL: http://mediatoolkit.com
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0  # Pattern: Luminator-robots
 Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)  # Pattern: ExtLinksBot / URL: https://extlinks.com/Bot.html
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)  # Pattern: SurveyBot
 NING/1.0  # Pattern: NING\/
 okhttp/2.5.0  # Pattern: okhttp
 okhttp/2.7.5  # Pattern: okhttp
 okhttp/3.2.0  # Pattern: okhttp
 okhttp/3.5.0  # Pattern: okhttp
 Nuzzel  # Pattern: Nuzzel
 omgili/0.5 +http://omgili.com  # Pattern: omgili / URL: http://omgili.com
 PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)  # Pattern: PocketParser / URL: https://getpocket.com/pocketparser_ua
 YisouSpider  # Pattern: YisouSpider
 Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected])  # Pattern: um-LN
 Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)  # Pattern: ToutiaoSpider / URL: http://web.toutiao.com/media_cooperation/
 Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)  # Pattern: MuckRack / URL: http://muckrack.com
 Jamie's Spider (http://jamiembrown.com/)  # Pattern: Jamie's Spider / URL: http://jamiembrown.com/
 AHC/2.0  # Pattern: AHC\/
 Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])  # Pattern: NetcraftSurveyAgent
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)  # Pattern: Laserlikebot
 Apache-HttpClient/4.2.3 (java 1.5)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.2.5 (java 1.5)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.3.1 (java 1.5)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.3.3 (java 1.5)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.3.5 (java 1.5)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.4.1 (Java/1.8.0_65)  # Pattern: Apache-HttpClient
 Apache-HttpClient/4.5.3 (Java/1.8.0_121)  # Pattern: Apache-HttpClient
 AppEngine-Google; (+http://code.google.com/appengine; appid: example)  # Pattern: AppEngine-Google
 Jetty/9.3.z-SNAPSHOT  # Pattern: Jetty
 Upflow/1.0  # Pattern: Upflow
 Thinklab (thinklab.com)  # Pattern: Thinklab / URL: thinklab.com
 Traackr.com  # Pattern: Traackr.com / URL: Traackr.com
 Ruby, Twurly v1.1 (http://twurly.org)  # Pattern: Twurly / URL: http://twurly.org
 http.rb/2.2.2 (Mastodon/1.5.1; +https://example-masto-instance.org/)  # Pattern: Mastodon
 http_get  # Pattern: http_get
 Mozilla/5.0 (compatible; DnyzBot/1.0)  # Pattern: DnyzBot
 Mozilla/5.0 (compatible; DnyzBot/1.0) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/64.0.3282.167 Safari/537.36  # Pattern: DnyzBot
 Mozilla/5.0 (compatible; DnyzBot/1.0) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/64.0.3264.0 Safari/537.36  # Pattern: DnyzBot
 Mozilla/5.0 (compatible; botify; http://botify.com)  # Pattern: botify
 Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)  # Pattern: 007ac9 Crawler
 Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)  # Pattern: BehloolBot
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)  # Pattern: BrandVerity
 check_http/v2.2.1 (nagios-plugins 2.2.1)  # Pattern: check_http
 Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36  # Pattern: BDCbot
 Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)  # Pattern: ZumBot
 EZID (EZID link checker; https://ezid.cdlib.org/)  # Pattern: EZID
 ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)  # Pattern: ICC-Crawler / URL: http://ucri.nict.go.jp/en/icccrawler.html
 ArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)  # Pattern: ArchiveBot / URL: https://github.com/ArchiveTeam/ArchiveBot
 LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)  # Pattern: ^LCC  / URL: http://corpora.informatik.uni-leipzig.de/crawler_faq.html
 Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)  # Pattern: filterdb.iss.net\/crawler / URL: http://filterdb.iss.net/crawler/
 BLP_bbot/0.1  # Pattern: BLP_bbot
 Mozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot)  # Pattern: BomboraBot / URL: http://www.bombora.com/bot
 Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)  # Pattern: Buck\/ / URL: https://app.hypefactors.com/media-monitoring/about.html
 Companybook-Crawler (+https://www.companybooknetworking.com/)  # Pattern: Companybook-Crawler / URL: https://www.companybooknetworking.com/
 Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)  # Pattern: Genieo / URL: http://www.genieo.com/webfilter.html
 magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)  # Pattern: magpie-crawler / URL: http://www.brandwatch.net
 MeltwaterNews www.meltwater.com  # Pattern: MeltwaterNews / URL: http://www.meltwater.com
 Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com)  # Pattern: Moreover / URL: http://www.moreover.com
 newspaper/0.2.5  # Pattern: newspaper\/
 newspaper/0.2.6  # Pattern: newspaper\/
 newspaper/0.1.0.7  # Pattern: newspaper\/
 Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)  # Pattern: ScoutJet / URL: http://www.scoutjet.com/
 sentry/8.22.0 (https://sentry.io)  # Pattern: (^| )sentry\/ / URL: https://sentry.io
 Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)  # Pattern: StorygizeBot / URL: http://www.storygize.com
 Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)  # Pattern: UptimeRobot / URL: http://www.uptimerobot.com/
 OutclicksBot/2 +https://www.outclicks.net/agent/VjzDygCuk4ubNmg40ZMbFqT0sIh7UfOKk8s8ZMiupUR  # Pattern: OutclicksBot / URL: https://www.outclicks.net
 OutclicksBot/2 +https://www.outclicks.net/agent/gIYbZ38dfAuhZkrFVl7sJBFOUhOVct6J1SvxgmBZgCe  # Pattern: OutclicksBot / URL: https://www.outclicks.net
 OutclicksBot/2 +https://www.outclicks.net/agent/PryJzTl8POCRHfvEUlRN5FKtZoWDQOBEvFJ2wh6KH5J  # Pattern: OutclicksBot / URL: https://www.outclicks.net
 OutclicksBot/2 +https://www.outclicks.net/agent/p2i4sNUh7eylJF1S6SGgRs5mP40ExlYvsr9GBxVQG6h  # Pattern: OutclicksBot / URL: https://www.outclicks.net
 Mozilla/5.0 (compatible; seoscanners.net/1; [email protected])  # Pattern: seoscanners / URL: http://www.seoscanners.net/
 Hatena Antenna/0.3  # Pattern: Hatena
 Hatena::Russia::Crawler/0.01  # Pattern: Hatena
 Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36  # Pattern: Google Web Preview
 MauiBot ([email protected])  # Pattern: MauiBot
 Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)  # Pattern: AlphaBot / URL: http://alphaseobot.com/bot.html
 SBL-BOT (http://sbl.net)  # Pattern: SBL-BOT / URL: http://sbl.net
 IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)  # Pattern: IAS crawler / URL: http://integralads.com/site-indexing-policy/
 Mozilla/5.0 (compatible; adscanner/)  # Pattern: adscanner
 Netvibes (crawler/bot; http://www.netvibes.com  # Pattern: Netvibes / URL: http://www.netvibes.com
 Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)  # Pattern: acapbot
 Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)  # Pattern: acapbot
 Baidu-YunGuanCe-Bot(ce.baidu.com)  # Pattern: Baidu-YunGuanCe / URL: https://ce.baidu.com/topic/topic20150908
 Baidu-YunGuanCe-SLABot(ce.baidu.com)  # Pattern: Baidu-YunGuanCe / URL: https://ce.baidu.com/topic/topic20150908
 Baidu-YunGuanCe-ScanBot(ce.baidu.com)  # Pattern: Baidu-YunGuanCe / URL: https://ce.baidu.com/topic/topic20150908
 Baidu-YunGuanCe-PerfBot(ce.baidu.com)  # Pattern: Baidu-YunGuanCe / URL: https://ce.baidu.com/topic/topic20150908
 Baidu-YunGuanCe-VSBot(ce.baidu.com)  # Pattern: Baidu-YunGuanCe / URL: https://ce.baidu.com/topic/topic20150908
 bitlybot/3.0 (+http://bit.ly/)  # Pattern: bitlybot / URL: http://bit.ly/
 bitlybot/2.0  # Pattern: bitlybot / URL: http://bit.ly/
 bitlybot  # Pattern: bitlybot / URL: http://bit.ly/
 blogmuraBot (+http://www.blogmura.com)  # Pattern: blogmuraBot / URL: http://www.blogmura.com
 Bot.AraTurka.com/0.0.1  # Pattern: Bot.AraTurka.com / URL: http://www.araturka.com
 bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)  # Pattern: bot-pge.chlooe.com
 Mozilla/5.0 (compatible; BoxcarBot/1.1; [email protected])  # Pattern: BoxcarBot / URL: https://boxcar.io/
 BTWebClient/180B(9704)  # Pattern: BTWebClient / URL: http://www.utorrent.com/
 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)  # Pattern: ContextAd Bot
 ContextAd Bot 1.0  # Pattern: ContextAd Bot
 Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.)  # Pattern: Digincore bot / URL: http://www.digincore.com/crawler.html
 Disqus/1.0  # Pattern: Disqus / URL: https://disqus.com/
 Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)  # Pattern: Feedly / URL: https://www.feedly.com/fetcher.html
 FeedlyBot/1.0 (http://feedly.com)  # Pattern: Feedly / URL: https://www.feedly.com/fetcher.html
 Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)  # Pattern: Fetch\/
 Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)  # Pattern: Fever / URL: http://feedafever.com
 Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)  # Pattern: Flamingo_SearchEngine
 Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)  # Pattern: FlipboardProxy / URL: https://about.flipboard.com/browserproxy/
 Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)  # Pattern: FlipboardProxy / URL: https://about.flipboard.com/browserproxy/
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)  # Pattern: FlipboardProxy / URL: https://about.flipboard.com/browserproxy/
 g2reader-bot/1.0 (+http://www.g2reader.com/)  # Pattern: g2reader-bot / URL: http://www.g2reader.com/
 Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)  # Pattern: imrbot / URL: http://www.mignify.com
 K7MLWCBot/1.0 (+http://www.k7computing.com)  # Pattern: K7MLWCBot / URL: http://www.k7computing.com
 Kemvibot/1.0 (http://kemvi.com, [email protected])  # Pattern: Kemvibot / URL: http://kemvi.com
 Landau-Media-Spider/1.0(http://bots.landaumedia.de/bot.html)  # Pattern: Landau-Media-Spider / URL: http://bots.landaumedia.de/bot.html
 linkapediabot (+http://www.linkapedia.com)  # Pattern: linkapediabot / URL: http://www.linkapedia.com
 Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)  # Pattern: vkShare / URL: http://vk.com/dev/Share
 Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com  # Pattern: Siteimprove.com
 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com  # Pattern: Siteimprove.com
 Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com  # Pattern: Siteimprove.com
 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com  # Pattern: Siteimprove.com
 Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)  # Pattern: BLEXBot\/ / URL: http://webmeup-crawler.com
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost  # Pattern: DareBoost / URL: https://www.dareboost.com/
 Mozilla/5.0 (compatible; ZuperlistBot/1.0)  # Pattern: ZuperlistBot\/
 Mozilla/5.0 (compatible; Miniflux/2.0.7; +https://miniflux.net)  # Pattern: Miniflux\/ / URL: https://miniflux.net
 Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)  # Pattern: Feedspotbot\/ / URL: http://www.feedspot.com/fs/bot
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)  # Pattern: Diffbot\/ / URL: http://www.diffbot.com
 Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)  # Pattern: SEOkicks / URL: https://www.seokicks.de/robot.html
 Mozilla/5.0 (compatible; tracemyfile/1.0; [email protected])  # Pattern: tracemyfile
 Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)  # Pattern: Nimbostratus-Bot
 Mozilla/5.0 zgrab/0.x  # Pattern: zgrab / URL: https://zmap.io/
 Mozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)  # Pattern: PR-CY.RU / URL: https://a.pr-cy.ru/
 AdsTxtCrawler/1.0  # Pattern: AdsTxtCrawler
 Datafeedwatch/2.1.x  # Pattern: Datafeedwatch / URL: https://www.datafeedwatch.com/
 Zabbix  # Pattern: Zabbix / URL: https://www.zabbix.com/documentation/3.4/manual/web_monitoring
 TangibleeBot/1.0.0.0 (http://tangiblee.com/bot)  # Pattern: TangibleeBot / URL: http://tangiblee.com/bot
 google-xrawler  # Pattern: google-xrawler / URL: https://webmasters.stackexchange.com/questions/105560/what-is-the-google-xrawler-user-agent-used-for
 axios/0.18.0  # Pattern: axios / URL: https://github.com/axios/axios
 Amazon CloudFront  # Pattern: Amazon CloudFront / URL: https://aws.amazon.com/cloudfront/
 Pulsepoint XT3 web scraper  # Pattern: Pulsepoint
 Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34  # Pattern: CloudFlare-AlwaysOnline / URL: https://www.cloudflare.com/always-online/
 Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +https://www.cloudflare.com/always-online) AppleWebKit/534.34  # Pattern: CloudFlare-AlwaysOnline / URL: https://www.cloudflare.com/always-online/
 Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)  # Pattern: Google-Structured-Data-Testing-Tool / URL: https://search.google.com/structured-data/testing-tool
 Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)  # Pattern: Google-Structured-Data-Testing-Tool / URL: https://search.google.com/structured-data/testing-tool
 WordupInfoSearch/1.0  # Pattern: WordupInfoSearch
 Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)  # Pattern: WebDataStats / URL: https://webdatastats.com/
 Jersey/2.25.1 (HttpUrlConnection 1.8.0_141)  # Pattern: HttpUrlConnection
 Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)  # Pattern: Seekport Crawler / URL: http://seekport.com/
 ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)  # Pattern: ZoomBot / URL: http://suite.seozoom.it/bot.html
 VelenPublicWebCrawler (velen.io)  # Pattern: VelenPublicWebCrawler
 MoodleBot/1.0  # Pattern: MoodleBot
 jpg-newsbot/2.0; (+https://vipnytt.no/bots/)  # Pattern: jpg-newsbot / URL: https://vipnytt.no/bots/
 Mozilla/5.0 (Java) outbrain  # Pattern: outbrain / URL: https://www.outbrain.com/help/advertisers/invalid-url/
 W3C_Validator/1.3  # Pattern: W3C_Validator / URL: https://validator.w3.org/services
 Validator.nu/LV  # Pattern: Validator\.nu / URL: https://validator.w3.org/services
 W3C-checklink  # Pattern: W3C-checklink / URL: https://validator.w3.org/services
 W3C-mobileOK/DDC-1.0  # Pattern: W3C-mobileOK / URL: https://validator.w3.org/services
 W3C_I18n-Checker/1.0  # Pattern: W3C_I18n-Checker / URL: https://validator.w3.org/services
 FeedValidator/1.3  # Pattern: FeedValidator / URL: https://validator.w3.org/services
 Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0  # Pattern: W3C_CSS_Validator / URL: https://validator.w3.org/services
 W3C_Unicorn/1.0  # Pattern: W3C_Unicorn / URL: https://validator.w3.org/services
 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  Googlebot 2.1
 Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) BingBot 2.0
 Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) Baidu Spider 2.0 
 Pinterest/0.2 (+http://www.pinterest.com/)  Pinterest Bot 0.2
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon Google Favicon Crawler 
 Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com) DuckDuckGo Favicons Bot 1.0
 Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+) Majestic-12 Distributed Search Bot 1.4 
 Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/) Majestic-12 Distributed Search Bot 1.4 
 Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)  Pinterest Bot
 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) Yahoo! Slurp Web Crawler Bot 
 Mediapartners-Google  Google's Media Partners system (AdSense) 
 Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)  Proximic Search
 Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  Googlebot 2.1
 Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) Sogou "Search Dog" 4 
 Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)  Slackbot Link Checker 1.0
 Pinterest/0.2 (+http://www.pinterest.com/bot.html)  Pinterest Bot 0.2
 Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)  Yandex Search Bot 3
 Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)  Ahrefs Backlink Research Bot 5.2 
 Twitterbot/1.0  TwitterBot 1.0 
 Mozilla/5.0 (Windows NT 6.1; rv:6.0) Gecko/20110814 Firefox/6.0 Google Favicon  Google Favicon Crawler 
 Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler) Proximic Search
 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider  360Spider
 Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0 TweetMeme Bot 4
 ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected] ADmantX Platform Semantic Analyzer 
 Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, [email protected])  OpenSiteExplorer Crawler 
 Google favicon  Google Favicon Crawler 
 Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) Grapeshot Bot 2.0
 Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/) Majestic-12 Distributed Search Bot 1.4 
 Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/) Google+ Snippet Fetcher
 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b BingPreview 1.0b 
 Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) Genieo Bot 1.0 
 Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) ExaLead Crawler 3
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)  360Spider
 Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools) DomainTools SurveyBot 2.3
 admantx-usastn/2.4 (+http://www.admantx.com/service-fetcher.html) ADmantX Platform Semantic Analyzer 
 ADmantX Platform Sync US Semantic Analyzer - ADmantX Inc. - www.admantx.com - [email protected] ADmantX Platform Semantic Analyzer 
 ADmantX Platform Semantic Analyzer US - Turn - ADmantX Inc. - www.admantx.com - [email protected] ADmantX Platform Semantic Analyzer 
 Mozilla/5.0 (compatible; meanpathbot/1.0; +http://www.meanpath.com/meanpathbot.html)  MeanPath Bot 1.0 
 Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 evaliant Evaliant Impressions Bot 41 Windows Blink Common
 Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_2 like Mac OS X) AppleWebKit/601.1 (KHTML, like Gecko) CriOS/51.0.2704.104 Mobile/13F69 Safari/601.1.46 evaliant Evaliant Impressions Bot 51 iOS Webkit  Common
 ADmantX Platform Semantic Analyzer Appnexus - ADmantX Inc. - www.admantx.com - [email protected]  ADmantX Platform Semantic Analyzer 
 Mozilla/5.0 (compatible; Googlebot/2.1; startmebot/1.0; +https://start.me/bot)  Googlebot 2.1
 Mozilla/5.0 (compatible; YodaoBot/1.0; http://www.yodao.com/help/webmaster/spider/; ) YodaoBot Search Bot 1.0
 admantx-sap/2.4 (+http://www.admantx.com/service-fetcher.html)  ADmantX Platform Semantic Analyzer 
 facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) Facebook Bot 1.1 
 msnbot/2.0b (+http://search.msn.com/msnbot.htm) MSN Bot 2.0b 
 Sosospider+(+http://help.soso.com/webspider.htm)  Sosospider Search Bot
 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider  360Spider
 Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)  Googlebot 2.1
 Googlebot/2.1 (+http://www.googlebot.com/bot.html)