Skip to content

Instantly share code, notes, and snippets.

@fernandez14

fernandez14/prepare-datasets.sh

Last active January 2, 2022 00:32
Show Gist options
  • Save fernandez14/32a91bfa98a863d77da91a7673aa3045 to your computer and use it in GitHub Desktop.
Save fernandez14/32a91bfa98a863d77da91a7673aa3045 to your computer and use it in GitHub Desktop.
Download ZIP
Spider-srv API calls examples.
Raw
prepare-datasets.sh
micro query sg.micro.srv.spider Spider.PrepareDatasets '{
"selector_id": 1,
"group": "motorcycle"
}'
Raw
selector-example.json
{
"name": {"use": "text", "value": "h1 span", "filters": ["trim-space"]},
"screen": {"use": "text", "value": "#phone div[data-gallery] .col-xs-16 div.bor-t-1", "filters": ["trim-space"]},
"os": {"use": "text", "value": "#phone div[data-gallery] .col-xs-16 div.bor-t-1", "i": 1, "filters": ["trim-space"]},
"camera": {"use": "text", "value": "#phone div[data-gallery] .col-xs-16 div.bor-t-1", "i": 2, "filters": ["trim-space"]},
"storage": {"use": "text", "value": "#phone div[data-gallery] .col-xs-16 div.bor-t-1", "i": 3, "filters": ["trim-space"]},
}
Raw
track-sitemap.sh
micro query sg.micro.srv.spider Spider.TrackSitemap '{
"from": "http://www.data.com/specs/",
"name": "Data specs",
"depth": 4,
"strict": true,
"patterns": [
{"name": "motorcycle", "matches": "^{manufacturer:slug}/{type:slug}/{year:num}/{series:slug}/{model:slug}.html"},
{"name": "filtered-list", "matches": "^{list:slug}/{filter:slug}.html"},
{"name": "list", "matches": "^{list:slug}.html"},
{"name": "landing", "matches": "^$"}
]
}
Raw
track-url.sh
micro query sg.micro.srv.spider Spider.TrackURL '{
"url": "https://comparateca.com/iphone-6",
"group": "comparateca"
}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment