Created
January 7, 2024 04:56
-
-
Save josuamarcelc/c2c9caef79cde38865bb36f4fad11483 to your computer and use it in GitHub Desktop.
Template of User Agents - Robots.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| User-Agent: Googlebot | |
| Allow: / | |
| User-Agent: Googlebot-Mobile | |
| Allow: / | |
| User-Agent: Googlebot-Image | |
| Allow: / | |
| User-Agent: Googlebot-News | |
| Allow: / | |
| User-Agent: Googlebot-Video | |
| Allow: / | |
| User-Agent: AdsBot-Google | |
| Allow: / | |
| User-Agent: AdsBot-Google-Mobile | |
| Allow: / | |
| User-Agent: Feedfetcher-Google | |
| Allow: / | |
| User-Agent: Mediapartners-Google | |
| Allow: / | |
| User-Agent: Mediapartners Googlebot | |
| Allow: / | |
| User-Agent: APIs-Google | |
| Allow: / | |
| User-Agent: Google-InspectionTool | |
| Allow: / | |
| User-Agent: Storebot-Google | |
| Allow: / | |
| User-Agent: GoogleOther | |
| Allow: / | |
| User-Agent: bingbot | |
| Allow: / | |
| User-Agent: Slurp | |
| Allow: / | |
| User-Agent: wWget | |
| Allow: / | |
| User-Agent: LinkedInBot | |
| Allow: / | |
| User-Agent: Python-urllib | |
| Allow: / | |
| User-Agent: python-requests | |
| Allow: / | |
| User-Agent: aiohttp | |
| Allow: / | |
| User-Agent: httpx | |
| Allow: / | |
| User-Agent: libwww-perl | |
| Allow: / | |
| User-Agent: httpunit | |
| Allow: / | |
| User-Agent: nutch | |
| Allow: / | |
| User-Agent: Go-http-client | |
| Allow: / | |
| User-Agent: phpcrawl | |
| Allow: / | |
| User-Agent: msnbot | |
| Allow: / | |
| User-Agent: jyxobot | |
| Allow: / | |
| User-Agent: FAST-WebCrawler | |
| Allow: / | |
| User-Agent: FAST Enterprise Crawler | |
| Allow: / | |
| User-Agent: BIGLOTRON | |
| Allow: / | |
| User-Agent: Teoma | |
| Allow: / | |
| User-Agent: convera | |
| Allow: / | |
| User-Agent: seekbot | |
| Allow: / | |
| User-Agent: Gigabot | |
| Allow: / | |
| User-Agent: Gigablast | |
| Allow: / | |
| User-Agent: exabot | |
| Allow: / | |
| User-Agent: ia_archiver | |
| Allow: / | |
| User-Agent: GingerCrawler | |
| Allow: / | |
| User-Agent: webmon | |
| Allow: / | |
| User-Agent: HTTrack | |
| Allow: / | |
| User-Agent: gruborg | |
| Allow: / | |
| User-Agent: UsineNouvelleCrawler | |
| Allow: / | |
| User-Agent: antibot | |
| Allow: / | |
| User-Agent: netresearchserver | |
| Allow: / | |
| User-Agent: speedy | |
| Allow: / | |
| User-Agent: fluffy | |
| Allow: / | |
| User-Agent: findlink | |
| Allow: / | |
| User-Agent: msrbot | |
| Allow: / | |
| User-Agent: panscient | |
| Allow: / | |
| User-Agent: yacybot | |
| Allow: / | |
| User-Agent: AISearchBot | |
| Allow: / | |
| User-Agent: ips-agent | |
| Allow: / | |
| User-Agent: tagoobot | |
| Allow: / | |
| User-Agent: MJ12bot | |
| Allow: / | |
| User-Agent: woriobot | |
| Allow: / | |
| User-Agent: yanga | |
| Allow: / | |
| User-Agent: buzzbot | |
| Allow: / | |
| User-Agent: mlbot | |
| Allow: / | |
| User-Agent: yandexcombots | |
| Allow: / | |
| User-Agent: purebot | |
| Allow: / | |
| User-Agent: Linguee Bot | |
| Allow: / | |
| User-Agent: CyberPatrol | |
| Allow: / | |
| User-Agent: voilabot | |
| Allow: / | |
| User-Agent: Baiduspider | |
| Allow: / | |
| User-Agent: citeseerxbot | |
| Allow: / | |
| User-Agent: spbot | |
| Allow: / | |
| User-Agent: twengabot | |
| Allow: / | |
| User-Agent: postrank | |
| Allow: / | |
| User-Agent: Turnitin | |
| Allow: / | |
| User-Agent: scribdbot | |
| Allow: / | |
| User-Agent: page2rss | |
| Allow: / | |
| User-Agent: sitebot | |
| Allow: / | |
| User-Agent: linkdex | |
| Allow: / | |
| User-Agent: Adidxbot | |
| Allow: / | |
| User-Agent: ezooms | |
| Allow: / | |
| User-Agent: dotbot | |
| Allow: / | |
| User-Agent: MailRU_Bot | |
| Allow: / | |
| User-Agent: discobot | |
| Allow: / | |
| User-Agent: heritrix | |
| Allow: / | |
| User-Agent: findthatfile | |
| Allow: / | |
| User-Agent: europarchiveorg | |
| Allow: / | |
| User-Agent: NerdByNatureBot | |
| Allow: / | |
| User-Agent: sistrix crawler | |
| Allow: / | |
| User-Agent: AhrefsBotSiteAudit | |
| Allow: / | |
| User-Agent: fuelbot | |
| Allow: / | |
| User-Agent: CrunchBot | |
| Allow: / | |
| User-Agent: IndeedBot | |
| Allow: / | |
| User-Agent: mappydata | |
| Allow: / | |
| User-Agent: woobot | |
| Allow: / | |
| User-Agent: ZoominfoBot | |
| Allow: / | |
| User-Agent: PrivacyAwareBot | |
| Allow: / | |
| User-Agent: Multiviewbot | |
| Allow: / | |
| User-Agent: SWIMGBot | |
| Allow: / | |
| User-Agent: Grobbot | |
| Allow: / | |
| User-Agent: eright | |
| Allow: / | |
| User-Agent: Apercite | |
| Allow: / | |
| User-Agent: semanticbot | |
| Allow: / | |
| User-Agent: Aboundex | |
| Allow: / | |
| User-Agent: domaincrawler | |
| Allow: / | |
| User-Agent: wbsearchbot | |
| Allow: / | |
| User-Agent: summify | |
| Allow: / | |
| User-Agent: CCBot | |
| Allow: / | |
| User-Agent: edisterbot | |
| Allow: / | |
| User-Agent: SeznamBot | |
| Allow: / | |
| User-Agent: ec2linkfinder | |
| Allow: / | |
| User-Agent: gslfbot | |
| Allow: / | |
| User-Agent: aiHitBot | |
| Allow: / | |
| User-Agent: intelium_bot | |
| Allow: / | |
| User-Agent: facebookexternalhit | |
| Allow: / | |
| User-Agent: Yeti | |
| Allow: / | |
| User-Agent: RetrevoPageAnalyzer | |
| Allow: / | |
| User-Agent: lb-spider | |
| Allow: / | |
| User-Agent: Sogou | |
| Allow: / | |
| User-Agent: lssbot | |
| Allow: / | |
| User-Agent: careerbot | |
| Allow: / | |
| User-Agent: wotbox | |
| Allow: / | |
| User-Agent: wocbot | |
| Allow: / | |
| User-Agent: ichiro | |
| Allow: / | |
| User-Agent: DuckDuckBot | |
| Allow: / | |
| User-Agent: lssrocketcrawler | |
| Allow: / | |
| User-Agent: drupact | |
| Allow: / | |
| User-Agent: webcompanycrawler | |
| Allow: / | |
| User-Agent: acoonbot | |
| Allow: / | |
| User-Agent: openindexspider | |
| Allow: / | |
| User-Agent: gnam gnam spider | |
| Allow: / | |
| User-Agent: web-archive-netcombot | |
| Allow: / | |
| User-Agent: backlinkcrawler | |
| Allow: / | |
| User-Agent: coccoc | |
| Allow: / | |
| User-Agent: integromedb | |
| Allow: / | |
| User-Agent: content crawler spider | |
| Allow: / | |
| User-Agent: toplistbot | |
| Allow: / | |
| User-Agent: it2media-domain-crawler | |
| Allow: / | |
| User-Agent: ip-web-crawlercom | |
| Allow: / | |
| User-Agent: siteexplorerinfo | |
| Allow: / | |
| User-Agent: elisabot | |
| Allow: / | |
| User-Agent: proximic | |
| Allow: / | |
| User-Agent: changedetection | |
| Allow: / | |
| User-Agent: arabot | |
| Allow: / | |
| User-Agent: WeSEESearch | |
| Allow: / | |
| User-Agent: niki-bot | |
| Allow: / | |
| User-Agent: CrystalSemanticsBot | |
| Allow: / | |
| User-Agent: rogerbot | |
| Allow: / | |
| User-Agent: 360Spider | |
| Allow: / | |
| User-Agent: psbot | |
| Allow: / | |
| User-Agent: InterfaxScanBot | |
| Allow: / | |
| User-Agent: CC Metadata Scaper | |
| Allow: / | |
| User-Agent: g00g1enet | |
| Allow: / | |
| User-Agent: GrapeshotCrawler | |
| Allow: / | |
| User-Agent: urlappendbot | |
| Allow: / | |
| User-Agent: brainobot | |
| Allow: / | |
| User-Agent: fr-crawler | |
| Allow: / | |
| User-Agent: binlar | |
| Allow: / | |
| User-Agent: SimpleCrawler | |
| Allow: / | |
| User-Agent: Twitterbot | |
| Allow: / | |
| User-Agent: cXensebot | |
| Allow: / | |
| User-Agent: smtbot | |
| Allow: / | |
| User-Agent: bnffr_bot | |
| Allow: / | |
| User-Agent: A6-Indexer | |
| Allow: / | |
| User-Agent: ADmantX | |
| Allow: / | |
| User-Agent: Facebot | |
| Allow: / | |
| User-Agent: OrangeBot | |
| Allow: / | |
| User-Agent: memorybot | |
| Allow: / | |
| User-Agent: AdvBot | |
| Allow: / | |
| User-Agent: MegaIndex | |
| Allow: / | |
| User-Agent: SemanticScholarBot | |
| Allow: / | |
| User-Agent: ltx71 | |
| Allow: / | |
| User-Agent: nerdybot | |
| Allow: / | |
| User-Agent: xovibot | |
| Allow: / | |
| User-Agent: BUbiNG | |
| Allow: / | |
| User-Agent: Qwantify | |
| Allow: / | |
| User-Agent: archiveorg_bot | |
| Allow: / | |
| User-Agent: Applebot | |
| Allow: / | |
| User-Agent: TweetmemeBot | |
| Allow: / | |
| User-Agent: crawler4j | |
| Allow: / | |
| User-Agent: findxbot | |
| Allow: / | |
| User-Agent: SeEmMrushBot | |
| Allow: / | |
| User-Agent: yoozBot | |
| Allow: / | |
| User-Agent: lipperhey | |
| Allow: / | |
| User-Agent: YJ | |
| Allow: / | |
| User-Agent: Domain Re-Animator Bot | |
| Allow: / | |
| User-Agent: AddThis | |
| Allow: / | |
| User-Agent: Screaming Frog SEO Spider | |
| Allow: / | |
| User-Agent: MetaURI | |
| Allow: / | |
| User-Agent: Scrapy | |
| Allow: / | |
| User-Agent: LivelapbBot | |
| Allow: / | |
| User-Agent: OpenHoseBot | |
| Allow: / | |
| User-Agent: CapsuleChecker | |
| Allow: / | |
| User-Agent: collectioninfegycom | |
| Allow: / | |
| User-Agent: IstellaBot | |
| Allow: / | |
| User-Agent: DeuSu | |
| Allow: / | |
| User-Agent: betaBot | |
| Allow: / | |
| User-Agent: Cliqzbot | |
| Allow: / | |
| User-Agent: MojeekBot | |
| Allow: / | |
| User-Agent: netEstate NE Crawler | |
| Allow: / | |
| User-Agent: SafeSearch microdata crawler | |
| Allow: / | |
| User-Agent: Gluten Free Crawler | |
| Allow: / | |
| User-Agent: Sonic | |
| Allow: / | |
| User-Agent: Sysomos | |
| Allow: / | |
| User-Agent: Trove | |
| Allow: / | |
| User-Agent: deadlinkchecker | |
| Allow: / | |
| User-Agent: Slack-ImgProxy | |
| Allow: / | |
| User-Agent: Embedly | |
| Allow: / | |
| User-Agent: RankActiveLinkBot | |
| Allow: / | |
| User-Agent: iskanie | |
| Allow: / | |
| User-Agent: SafeDNSBot | |
| Allow: / | |
| User-Agent: SkypeUriPreview | |
| Allow: / | |
| User-Agent: Veoozbot | |
| Allow: / | |
| User-Agent: Slackbot | |
| Allow: / | |
| User-Agent: redditbot | |
| Allow: / | |
| User-Agent: datagnionbot | |
| Allow: / | |
| User-Agent: Google-Adwords-Instant | |
| Allow: / | |
| User-Agent: adbeat_bot | |
| Allow: / | |
| User-Agent: WhatsApp | |
| Allow: / | |
| User-Agent: contxbot | |
| Allow: / | |
| User-Agent: pinterestcombot | |
| Allow: / | |
| User-Agent: electricmonk | |
| Allow: / | |
| User-Agent: GarlikCrawler | |
| Allow: / | |
| User-Agent: BingPreview | |
| Allow: / | |
| User-Agent: vebidoobot | |
| Allow: / | |
| User-Agent: FemtosearchBot | |
| Allow: / | |
| User-Agent: Yahoo Link Preview | |
| Allow: / | |
| User-Agent: MetaJobBot | |
| Allow: / | |
| User-Agent: DomainStatsBot | |
| Allow: / | |
| User-Agent: mindUpBot | |
| Allow: / | |
| User-Agent: Daum | |
| Allow: / | |
| User-Agent: Jugendschutzprogramm-Crawler | |
| Allow: / | |
| User-Agent: Xenu Link Sleuth | |
| Allow: / | |
| User-Agent: Pcore-HTTP | |
| Allow: / | |
| User-Agent: moatbot | |
| Allow: / | |
| User-Agent: KosmioBot | |
| Allow: / | |
| User-Agent: pPingdom | |
| Allow: / | |
| User-Agent: AppInsights | |
| Allow: / | |
| User-Agent: PhantomJS | |
| Allow: / | |
| User-Agent: Gowikibot | |
| Allow: / | |
| User-Agent: PiplBot | |
| Allow: / | |
| User-Agent: Discordbot | |
| Allow: / | |
| User-Agent: TelegramBot | |
| Allow: / | |
| User-Agent: Jetslide | |
| Allow: / | |
| User-Agent: newsharecounts | |
| Allow: / | |
| User-Agent: James BOT | |
| Allow: / | |
| User-Agent: BarkrRowler | |
| Allow: / | |
| User-Agent: TinEye | |
| Allow: / | |
| User-Agent: SocialRankIOBot | |
| Allow: / | |
| User-Agent: trendictionbot | |
| Allow: / | |
| User-Agent: Ocarinabot | |
| Allow: / | |
| User-Agent: epicbot | |
| Allow: / | |
| User-Agent: Primalbot | |
| Allow: / | |
| User-Agent: DuckDuckGo-Favicons-Bot | |
| Allow: / | |
| User-Agent: GnowitNewsbot | |
| Allow: / | |
| User-Agent: Leikibot | |
| Allow: / | |
| User-Agent: LinkArchiver | |
| Allow: / | |
| User-Agent: YaK | |
| Allow: / | |
| User-Agent: PaperLiBot | |
| Allow: / | |
| User-Agent: Digg Deeper | |
| Allow: / | |
| User-Agent: dcrawl | |
| Allow: / | |
| User-Agent: Snacktory | |
| Allow: / | |
| User-Agent: AndersPinkBot | |
| Allow: / | |
| User-Agent: Fyrebot | |
| Allow: / | |
| User-Agent: EveryoneSocialBot | |
| Allow: / | |
| User-Agent: Mediatoolkitbot | |
| Allow: / | |
| User-Agent: Luminator-robots | |
| Allow: / | |
| User-Agent: ExtLinksBot | |
| Allow: / | |
| User-Agent: SurveyBot | |
| Allow: / | |
| User-Agent: NING | |
| Allow: / | |
| User-Agent: okhttp | |
| Allow: / | |
| User-Agent: Nuzzel | |
| Allow: / | |
| User-Agent: omgili | |
| Allow: / | |
| User-Agent: PocketParser | |
| Allow: / | |
| User-Agent: YisouSpider | |
| Allow: / | |
| User-Agent: um-LN | |
| Allow: / | |
| User-Agent: ToutiaoSpider | |
| Allow: / | |
| User-Agent: MuckRack | |
| Allow: / | |
| User-Agent: Jamies Spider | |
| Allow: / | |
| User-Agent: AHC | |
| Allow: / | |
| User-Agent: NetcraftSurveyAgent | |
| Allow: / | |
| User-Agent: Laserlikebot | |
| Allow: / | |
| User-Agent: Apache-HttpClient | |
| Allow: / | |
| User-Agent: AppEngine-Google | |
| Allow: / | |
| User-Agent: Jetty | |
| Allow: / | |
| User-Agent: Upflow | |
| Allow: / | |
| User-Agent: Thinklab | |
| Allow: / | |
| User-Agent: Traackrcom | |
| Allow: / | |
| User-Agent: Twurly | |
| Allow: / | |
| User-Agent: Mastodon | |
| Allow: / | |
| User-Agent: http_get | |
| Allow: / | |
| User-Agent: DnyzBot | |
| Allow: / | |
| User-Agent: botify | |
| Allow: / | |
| User-Agent: 007ac9 Crawler | |
| Allow: / | |
| User-Agent: BehloolBot | |
| Allow: / | |
| User-Agent: BrandVerity | |
| Allow: / | |
| User-Agent: check_http | |
| Allow: / | |
| User-Agent: BDCbot | |
| Allow: / | |
| User-Agent: ZumBot | |
| Allow: / | |
| User-Agent: EZID | |
| Allow: / | |
| User-Agent: ICC-Crawler | |
| Allow: / | |
| User-Agent: ArchiveBot | |
| Allow: / | |
| User-Agent: LCC | |
| Allow: / | |
| User-Agent: filterdbissnetcrawler | |
| Allow: / | |
| User-Agent: BLP_bbot | |
| Allow: / | |
| User-Agent: BomboraBot | |
| Allow: / | |
| User-Agent: Buck | |
| Allow: / | |
| User-Agent: Companybook-Crawler | |
| Allow: / | |
| User-Agent: Genieo | |
| Allow: / | |
| User-Agent: magpie-crawler | |
| Allow: / | |
| User-Agent: MeltwaterNews | |
| Allow: / | |
| User-Agent: Moreover | |
| Allow: / | |
| User-Agent: newspaper | |
| Allow: / | |
| User-Agent: ScoutJet | |
| Allow: / | |
| User-Agent: sentry | |
| Allow: / | |
| User-Agent: StorygizeBot | |
| Allow: / | |
| User-Agent: UptimeRobot | |
| Allow: / | |
| User-Agent: OutclicksBot | |
| Allow: / | |
| User-Agent: seoscanners | |
| Allow: / | |
| User-Agent: Hatena | |
| Allow: / | |
| User-Agent: Google Web Preview | |
| Allow: / | |
| User-Agent: MauiBot | |
| Allow: / | |
| User-Agent: AlphaBot | |
| Allow: / | |
| User-Agent: SBL-BOT | |
| Allow: / | |
| User-Agent: IAS crawler | |
| Allow: / | |
| User-Agent: adscanner | |
| Allow: / | |
| User-Agent: Netvibes | |
| Allow: / | |
| User-Agent: acapbot | |
| Allow: / | |
| User-Agent: Baidu-YunGuanCe | |
| Allow: / | |
| User-Agent: bitlybot | |
| Allow: / | |
| User-Agent: blogmuraBot | |
| Allow: / | |
| User-Agent: BotAraTurkacom | |
| Allow: / | |
| User-Agent: bot-pgechlooecom | |
| Allow: / | |
| User-Agent: BoxcarBot | |
| Allow: / | |
| User-Agent: BTWebClient | |
| Allow: / | |
| User-Agent: ContextAd Bot | |
| Allow: / | |
| User-Agent: Digincore bot | |
| Allow: / | |
| User-Agent: Disqus | |
| Allow: / | |
| User-Agent: Feedly | |
| Allow: / | |
| User-Agent: Fetch | |
| Allow: / | |
| User-Agent: Fever | |
| Allow: / | |
| User-Agent: Flamingo_SearchEngine | |
| Allow: / | |
| User-Agent: FlipboardProxy | |
| Allow: / | |
| User-Agent: g2reader-bot | |
| Allow: / | |
| User-Agent: G2 Web Services | |
| Allow: / | |
| User-Agent: imrbot | |
| Allow: / | |
| User-Agent: K7MLWCBot | |
| Allow: / | |
| User-Agent: Kemvibot | |
| Allow: / | |
| User-Agent: Landau-Media-Spider | |
| Allow: / | |
| User-Agent: linkapediabot | |
| Allow: / | |
| User-Agent: vkShare | |
| Allow: / | |
| User-Agent: Siteimprovecom | |
| Allow: / | |
| User-Agent: BLEXBot | |
| Allow: / | |
| User-Agent: DareBoost | |
| Allow: / | |
| User-Agent: ZuperlistBot | |
| Allow: / | |
| User-Agent: Miniflux | |
| Allow: / | |
| User-Agent: Feedspot | |
| Allow: / | |
| User-Agent: Diffbot | |
| Allow: / | |
| User-Agent: SEOkicks | |
| Allow: / | |
| User-Agent: tracemyfile | |
| Allow: / | |
| User-Agent: Nimbostratus-Bot | |
| Allow: / | |
| User-Agent: zgrab | |
| Allow: / | |
| User-Agent: PR-CYRU | |
| Allow: / | |
| User-Agent: AdsTxtCrawler | |
| Allow: / | |
| User-Agent: Datafeedwatch | |
| Allow: / | |
| User-Agent: Zabbix | |
| Allow: / | |
| User-Agent: TangibleeBot | |
| Allow: / | |
| User-Agent: google-xrawler | |
| Allow: / | |
| User-Agent: axios | |
| Allow: / | |
| User-Agent: Amazon CloudFront | |
| Allow: / | |
| User-Agent: Pulsepoint | |
| Allow: / | |
| User-Agent: CloudFlare-AlwaysOnline | |
| Allow: / | |
| User-Agent: Google-Structured-Data-Testing-Tool | |
| Allow: / | |
| User-Agent: WordupInfoSearch | |
| Allow: / | |
| User-Agent: WebDataStats | |
| Allow: / | |
| User-Agent: HttpUrlConnection | |
| Allow: / | |
| User-Agent: Seekport Crawler | |
| Allow: / | |
| User-Agent: ZoomBot | |
| Allow: / | |
| User-Agent: VelenPublicWebCrawler | |
| Allow: / | |
| User-Agent: MoodleBot | |
| Allow: / | |
| User-Agent: jpg-newsbot | |
| Allow: / | |
| User-Agent: outbrain | |
| Allow: / | |
| User-Agent: W3C_Validator | |
| Allow: / | |
| User-Agent: Validatornu | |
| Allow: / | |
| User-Agent: W3C-checklink | |
| Allow: / | |
| User-Agent: W3C-mobileOK | |
| Allow: / | |
| User-Agent: W3C_I18n-Checker | |
| Allow: / | |
| User-Agent: FeedValidator | |
| Allow: / | |
| User-Agent: W3C_CSS_Validator | |
| Allow: / | |
| User-Agent: W3C_Unicorn | |
| Allow: / | |
| User-Agent: Google-PhysicalWeb | |
| Allow: / | |
| User-Agent: Blackboard | |
| Allow: / | |
| User-Agent: ICBot | |
| Allow: / | |
| User-Agent: BazQux | |
| Allow: / | |
| User-Agent: Twingly | |
| Allow: / | |
| User-Agent: Rivva | |
| Allow: / | |
| User-Agent: Experibot | |
| Allow: / | |
| User-Agent: awesomecrawler | |
| Allow: / | |
| User-Agent: Dataprovidercom | |
| Allow: / | |
| User-Agent: GroupHigh | |
| Allow: / | |
| User-Agent: theoldreadercom | |
| Allow: / | |
| User-Agent: AnyEvent | |
| Allow: / | |
| User-Agent: Uptimebotorg | |
| Allow: / | |
| User-Agent: Nmap Scripting Engine | |
| Allow: / | |
| User-Agent: 2ipru | |
| Allow: / | |
| User-Agent: Clickagy | |
| Allow: / | |
| User-Agent: Caliperbot | |
| Allow: / | |
| User-Agent: MBCrawler | |
| Allow: / | |
| User-Agent: online-webceo-bot | |
| Allow: / | |
| User-Agent: B2B Bot | |
| Allow: / | |
| User-Agent: AddSearchBot | |
| Allow: / | |
| User-Agent: Google Favicon | |
| Allow: / | |
| User-Agent: HubSpot | |
| Allow: / | |
| User-Agent: Chrome-Lighthouse | |
| Allow: / | |
| User-Agent: HeadlessChrome | |
| Allow: / | |
| User-Agent: CheckMarkNetwork | |
| Allow: / | |
| User-Agent: wwwuptimecom | |
| Allow: / | |
| User-Agent: Streamline3Bot | |
| Allow: / | |
| User-Agent: serpstatbot | |
| Allow: / | |
| User-Agent: MixnodeCache | |
| Allow: / | |
| User-Agent: curl | |
| Allow: / | |
| User-Agent: SimpleScraper | |
| Allow: / | |
| User-Agent: RSSingBot | |
| Allow: / | |
| User-Agent: Jooblebot | |
| Allow: / | |
| User-Agent: fedoraplanet | |
| Allow: / | |
| User-Agent: Friendica | |
| Allow: / | |
| User-Agent: NextCloud | |
| Allow: / | |
| User-Agent: Tiny Tiny RSS | |
| Allow: / | |
| User-Agent: RegionStuttgartBot | |
| Allow: / | |
| User-Agent: Bytespider | |
| Allow: / | |
| User-Agent: Datanyze | |
| Allow: / | |
| User-Agent: Google-Site-Verification | |
| Allow: / | |
| User-Agent: TrendsmapResolver | |
| Allow: / | |
| User-Agent: tweetedtimes | |
| Allow: / | |
| User-Agent: NTENTbot | |
| Allow: / | |
| User-Agent: Gwene | |
| Allow: / | |
| User-Agent: SimplePie | |
| Allow: / | |
| User-Agent: SearchAtlas | |
| Allow: / | |
| User-Agent: Superfeedr | |
| Allow: / | |
| User-Agent: feedbot | |
| Allow: / | |
| User-Agent: UT-Dorkbot | |
| Allow: / | |
| User-Agent: Amazonbot | |
| Allow: / | |
| User-Agent: SerendeputyBot | |
| Allow: / | |
| User-Agent: Eyeotabot | |
| Allow: / | |
| User-Agent: officestorebot | |
| Allow: / | |
| User-Agent: Neticle Crawler | |
| Allow: / | |
| User-Agent: SurdotlyBot | |
| Allow: / | |
| User-Agent: LinkisBot | |
| Allow: / | |
| User-Agent: AwarioSmartBot | |
| Allow: / | |
| User-Agent: AwarioRssBot | |
| Allow: / | |
| User-Agent: RyteBot | |
| Allow: / | |
| User-Agent: FreeWebMonitoring SiteChecker | |
| Allow: / | |
| User-Agent: AspiegelBot | |
| Allow: / | |
| User-Agent: NAVER Blog Rssbot | |
| Allow: / | |
| User-Agent: zenback bot | |
| Allow: / | |
| User-Agent: SentiBot | |
| Allow: / | |
| User-Agent: Domains Project | |
| Allow: / | |
| User-Agent: Pandalytics | |
| Allow: / | |
| User-Agent: VKRobot | |
| Allow: / | |
| User-Agent: bidswitchbot | |
| Allow: / | |
| User-Agent: tigerbot | |
| Allow: / | |
| User-Agent: NIXStatsbot | |
| Allow: / | |
| User-Agent: Atom Feed Robot | |
| Allow: / | |
| User-Agent: Ccurebot | |
| Allow: / | |
| User-Agent: PagePeeker | |
| Allow: / | |
| User-Agent: Vigil | |
| Allow: / | |
| User-Agent: rssbot | |
| Allow: / | |
| User-Agent: startmebot | |
| Allow: / | |
| User-Agent: JobboerseBot | |
| Allow: / | |
| User-Agent: seewithkids | |
| Allow: / | |
| User-Agent: NINJA bot | |
| Allow: / | |
| User-Agent: Cutbot | |
| Allow: / | |
| User-Agent: BublupBot | |
| Allow: / | |
| User-Agent: BrandONbot | |
| Allow: / | |
| User-Agent: RidderBot | |
| Allow: / | |
| User-Agent: Taboolabot | |
| Allow: / | |
| User-Agent: Dubbotbot | |
| Allow: / | |
| User-Agent: FindITAnswersbot | |
| Allow: / | |
| User-Agent: infoobot | |
| Allow: / | |
| User-Agent: Refindbot | |
| Allow: / | |
| User-Agent: BlogTrafficdd Feed-Fetcher | |
| Allow: / | |
| User-Agent: SeobilityBot | |
| Allow: / | |
| User-Agent: Cincraw | |
| Allow: / | |
| User-Agent: Dragonbot | |
| Allow: / | |
| User-Agent: VoluumDSP-content-bot | |
| Allow: / | |
| User-Agent: FreshRSS | |
| Allow: / | |
| User-Agent: BitBot | |
| Allow: / | |
| User-Agent: PHP-Curl-Class | |
| Allow: / | |
| User-Agent: Google-Certificates-Bridge | |
| Allow: / | |
| User-Agent: centurybot | |
| Allow: / | |
| User-Agent: Viber | |
| Allow: / | |
| User-Agent: eventures Investment Crawler | |
| Allow: / | |
| User-Agent: evc-batch | |
| Allow: / | |
| User-Agent: PetalBot | |
| Allow: / | |
| User-Agent: virustotal | |
| Allow: / | |
| User-Agent: PTST | |
| Allow: / | |
| User-Agent: minicrawler | |
| Allow: / | |
| User-Agent: Cookiebot | |
| Allow: / | |
| User-Agent: trovitBot | |
| Allow: / | |
| User-Agent: seostarco | |
| Allow: / | |
| User-Agent: IonCrawl | |
| Allow: / | |
| User-Agent: Uptime-Kuma | |
| Allow: / | |
| User-Agent: SeekportBot | |
| Allow: / | |
| User-Agent: FreshpingBot | |
| Allow: / | |
| User-Agent: Feedbin | |
| Allow: / | |
| User-Agent: CriteoBot | |
| Allow: / | |
| User-Agent: Snap URL Preview Service | |
| Allow: / | |
| User-Agent: Better Uptime Bot | |
| Allow: / | |
| User-Agent: RuxitSynthetic | |
| Allow: / | |
| User-Agent: Google-Read-Aloud | |
| Allow: / | |
| User-Agent: ValveSteam | |
| Allow: / | |
| User-Agent: OdklBot | |
| Allow: / | |
| User-Agent: GPTBot | |
| Allow: / | |
| User-Agent: YandexRenderResourcesBot | |
| Allow: / | |
| User-Agent: LightspeedSystemsCrawler | |
| Allow: / | |
| User-Agent: ev-crawler | |
| Allow: / | |
| User-Agent: BitSightBot | |
| Allow: / | |
| User-Agent: woorankreview | |
| Allow: / | |
| User-Agent: Google-Safety | |
| Allow: / | |
| User-Agent: AwarioBot | |
| Allow: / | |
| User-Agent: DataForSeoBot | |
| Allow: / | |
| User-Agent: Linespider | |
| Allow: / | |
| User-Agent: WellKnownBot | |
| Allow: / | |
| User-Agent: A Patent Crawler | |
| Allow: / | |
| User-Agent: StractBot | |
| Allow: / | |
| User-Agent: searchmarginalianu | |
| Allow: / | |
| User-Agent: YouBot | |
| Allow: / | |
| User-Agent: Nicecrawler | |
| Allow: / | |
| User-Agent: Neevabot | |
| Allow: / | |
| User-Agent: BrightEdge Crawler | |
| Allow: / | |
| User-Agent: SiteCheckerBotCrawler | |
| Allow: / | |
| User-Agent: TombaPublicWebCrawler | |
| Allow: / | |
| User-Agent: CrawlyProjectCrawler | |
| Allow: / | |
| User-Agent: KomodiaBot | |
| Allow: / | |
| User-Agent: KStandBot | |
| Allow: / | |
| User-Agent: CISPA Webcrawler | |
| Allow: / | |
| User-Agent: MTRobot | |
| Allow: / | |
| User-Agent: hyscoreio | |
| Allow: / | |
| User-Agent: AlexandriaOrgBot | |
| Allow: / | |
| User-Agent: 2ip bot | |
| Allow: / | |
| User-Agent: Yellowbrandprotectionbot | |
| Allow: / | |
| User-Agent: SEOlizer | |
| Allow: / | |
| User-Agent: vuhuvBot | |
| Allow: / | |
| User-Agent: INETDEX-BOT | |
| Allow: / | |
| User-Agent: Synapse | |
| Allow: / | |
| User-Agent: t3versionsBot | |
| Allow: / | |
| User-Agent: deepnoc | |
| Allow: / | |
| User-Agent: Cocolyzebot | |
| Allow: / | |
| User-Agent: hypestat | |
| Allow: / | |
| User-Agent: ReverseEngineeringBot | |
| Allow: / | |
| User-Agent: sempitech | |
| Allow: / | |
| User-Agent: Iframely | |
| Allow: / | |
| User-Agent: MetaInspector | |
| Allow: / | |
| User-Agent: node-fetch | |
| Allow: / | |
| User-Agent: lkxscan | |
| Allow: / | |
| User-Agent: python-opengraph | |
| Allow: / | |
| User-Agent: OpenGraphCheck | |
| Allow: / | |
| User-Agent: developersgooglecomwebsnippet | |
| Allow: / | |
| User-Agent: SenutoBot | |
| Allow: / | |
| User-Agent: MaCoCu | |
| Allow: / | |
| User-Agent: NewsBlur | |
| Allow: / | |
| User-Agent: inoreader | |
| Allow: / | |
| User-Agent: NetSystemsResearch | |
| Allow: / | |
| User-Agent: PageThing | |
| Allow: / | |
| User-Agent: WordPress | |
| Allow: / |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Pattern of User Agent in crawler / robots
https://gist.github.com/josuamarcelc/6bfbdc14c6292e195844032bea7211d1