ResolveWang
/ gist:37f7d31d569a53f0b7dbf23c444eecbf
Created
January 20, 2018 13:16
scrapy download speed checker
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| class GoogleNewsSpider(RedisSpider, SpiderMixin): | |
| """spider for google news""" | |
| name = 'google_news' | |
| redis_key = 'search_engine:google_news' | |
| req_num = 10 | |
| proxy_mode = 2 | |
| custom_settings = { | |
| "DOWNLOAD_DELAY": 20, | |
| "CONCURRENT_REQUESTS_PER_DOMAIN": 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import time | |
| import requests | |
| from weibospider.db.dao import WbDataOper | |
| # 借助百度地图API实现地理位置的经纬度获取 | |
| class LocationHelper: | |
| # ak 是在 http://lbsyun.baidu.com/apiconsole/key 申请的 |
OlderNewer