Skip to content

Instantly share code, notes, and snippets.

@metaperl
Created May 9, 2020 13:40
Show Gist options
  • Save metaperl/53842aaa69b632cfff1e8bdfd656037f to your computer and use it in GitHub Desktop.
Save metaperl/53842aaa69b632cfff1e8bdfd656037f to your computer and use it in GitHub Desktop.
Extracting text from a Selector element in Scrapy
import scrapy
from behold import Behold
class SignalStartSpider(scrapy.Spider):
name = 'signalstart'
start_urls = [
'https://www.signalstart.com/search-signals',
]
def parse(self, response):
cols = "rank name gain pips drawdown trades type monthly chart price age added action"
skip = [9,13]
td = dict()
for i, col in enumerate(cols.split()):
td[i] = col
Behold().show('td')
for provider in response.xpath("//div[@class='row']//tr"):
data_row = dict()
for i, datum in enumerate(provider.xpath('td')):
if i in skip:
continue
data_row[td[i]] = datum
# Behold().show('datum')
yield data_row
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment