Skip to content

Instantly share code, notes, and snippets.

@rsarai

rsarai/scrapy_mercado_pipeline.py

Created September 2, 2017 21:02
Show Gist options
  • Save rsarai/e7470df8cce3745d539b2f27f6552a46 to your computer and use it in GitHub Desktop.
Save rsarai/e7470df8cce3745d539b2f27f6552a46 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
scrapy_mercado_pipeline.py
import scrapy
from scrapy.contrib.pipeline.images import ImagesPipeline
from scrapymercado.constants import URLS_AND_NAMES
class ScrapymercadoPipeline(ImagesPipeline):
def get_media_requests(self, item, info):
for image_url in item.get('image_urls', []):
yield scrapy.Request(image_url, meta={'filepatch': URLS_AND_NAMES.get(image_url)})
def file_path(self, request, response=None, info=None):
return 'full/%s.jpg' % request.meta.get('filepatch')
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment