Ahmed Rafik Djerah rafikahmed

rafikahmed / index.html

Created July 17, 2018 14:02

	<!DOCTYPE html>
	<html lang="en">

	<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<meta http-equiv="X-UA-Compatible" content="ie=edge">
	<title>Xpath Syntax</title>
	</head>

rafikahmed / index.html

Created September 27, 2018 17:54

HTML Document

rafikahmed / index.html

Created September 27, 2018 18:22

XPath

	<p class='someClass'>Paragraph 1</p>
	<p id='someId'>Paragraph 2</p>

rafikahmed / jokes.py

Last active September 27, 2018 19:00

	import scrapy

	class JokesSpider(scrapy.Spider):
	name= 'jokes'
	allowed_domains = ['www.laughfactory.com']
	start_urls = [
	'http://www.laughfactory.com/jokes/family-jokes'
	]

	def parse(self, response):

rafikahmed / jokes.py

Last active September 29, 2018 10:01

	def parse(self, response):
	for joke in response.xpath("//div[@class='jokes']"):
	yield {
	'joke_text': joke.xpath(".//div[@class='joke-text']/p").extract_first()
	}

rafikahmed / jokes.py

Created September 29, 2018 10:21

	next_page= response.xpath("//li[@class='next']/a/@href").extract_first()
	if next_page is not None:
	next_page_link= response.urljoin(next_page)
	yield scrapy.Request(url=next_page_link, callback=self.parse)

rafikahmed / items.py

Created September 29, 2018 11:28

	class JokeItem(scrapy.Item):
	joke_text= scrapy.Field()

rafikahmed / jokes.py

Created September 29, 2018 11:31

	import scrapy
	from demo_project.items import JokeItem
	from scrapy.loader import ItemLoader

	class JokesSpider(scrapy.Spider):
	name= 'jokes'

	allowed_domais = ['www.laughfactory.com']

	start_urls = [

rafikahmed / items.py

Created September 29, 2018 11:37

	import scrapy
	from scrapy.loader.processors import MapCompose, TakeFirst
	from w3lib.html import remove_tags

	def remove_whitespace(value):
	return value.strip()

	class JokeItem(scrapy.Item):
	joke_text= scrapy.Field(
	input_processor= MapCompose(remove_tags, remove_whitespace),

rafikahmed / .gitignore

Created October 30, 2018 08:10