bcambel’s gists

bcambel / gist:2622001

Created May 6, 2012 12:21 — forked from osiloke/gist:1138798

Generating URLs to crawl from outside a Scrapy spider

	from scrapy import log
	from scrapy.item import Item
	from scrapy.http import Request
	from scrapy.contrib.spiders import XMLFeedSpider


	def NextURL():
	"""
	Generate a list of URLs to crawl. You can query a database or come up with some other means
	Note that if you generate URLs to crawl from a scraped URL then you're better of using a

bcambel / sql_mongodb_dump_middleware.py

Created May 21, 2012 09:12 — forked from kesor/sql_mongodb_dump_middleware.py

Django MongoDB + SQL dump middleware

	from django.core.exceptions import MiddlewareNotUsed
	from django.conf import settings
	from django.db import connection
	from pymongo.connection import Connection
	from time import time
	import struct
	import bson
	from bson.errors import InvalidBSON

	class SqldumpMiddleware(object):

bcambel / latency.txt

Created June 3, 2012 22:37 — forked from jboner/latency.txt

Latency numbers every programmer should know

	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns
	Send 1K bytes over 1 Gbps network 10,000 ns 0.01 ms
	Read 4K randomly from SSD 150,000 ns 0.15 ms
	Read 1 MB sequentially from memory 250,000 ns 0.25 ms
	Round trip within same datacenter 500,000 ns 0.5 ms

bcambel / index.html

Created June 8, 2012 23:42 — forked from ralphbean/index.html

d3 viz for narcissus

	<!DOCTYPE html>
	<html>
	<head>
	<title>Proof of concept for d3 viz + narcissus</title>
	<link rel="stylesheet" type="text/css" href="spider.css"></style>
	<script type="text/javascript" src="http://mbostock.github.com/d3/d3.min.js"></script>
	<script type="text/javascript" src="http://mbostock.github.com/d3/d3.geom.min.js"></script>
	<script type="text/javascript" src="http://mbostock.github.com/d3/d3.layout.min.js"></script>
	<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>
	<script type="text/javascript" src="spider.js"></script>

bcambel / sample.config

Created July 31, 2012 21:28 — forked from sidupadhyay/sample.config

Sample HA Proxy Config for Tornado/Socket.io Backends

	global
	maxconn 10000 # Total Max Connections. This is dependent on ulimit
	nbproc 2

	defaults
	mode http
	option redispatch
	maxconn 2000
	contimeout 5000
	clitimeout 50000

bcambel / supervisor.conf

Created December 6, 2012 17:07 — forked from tsabat/supervisor.conf

Sample supervisor config file

	; Sample supervisor config file.

	[unix_http_server]
	file=/tmp/supervisor.sock ; (the path to the socket file)
	;chmod=0700 ; sockef file mode (default 0700)
	;chown=nobody:nogroup ; socket file uid:gid owner
	;username=user ; (default is no username (open server))
	;password=123 ; (default is no password (open server))

	;[inet_http_server] ; inet (TCP) server disabled by default

bcambel / supervisord-example.conf

Created December 7, 2012 02:35 — forked from didip/supervisord-example.conf

Example for supervisord conf file

	; Sample supervisor config file.

	[unix_http_server]
	file=/tmp/supervisor.sock ; (the path to the socket file)
	;chmod=0700 ; sockef file mode (default 0700)
	;chown=nobody:nogroup ; socket file uid:gid owner
	;username=user ; (default is no username (open server))
	;password=123 ; (default is no password (open server))

	;[inet_http_server] ; inet (TCP) server disabled by default

bcambel / gist:4365127

Created December 23, 2012 18:37 — forked from anonymous/gist:4365124

	# Many times when crawling we run into problems where content that is rendered on the page is generated with Javascript and therefore scrapy is unable to crawl for it (eg. ajax requests, jQuery craziness). However, if you use Scrapy along with the web testing framework Selenium then we are able to crawl anything displayed in a normal web browser.
	#
	# Some things to note:
	# You must have the Python version of Selenium RC installed for this to work, and you must have set up Selenium properly. Also this is just a template crawler. You could get much crazier and more advanced with things but I just wanted to show the basic idea. As the code stands now you will be doing two requests for any given url. One request is made by Scrapy and the other is made by Selenium. I am sure there are ways around this so that you could possibly just make Selenium do the one and only request but I did not bother to implement that and by doing two requests you get to crawl the page with Scrapy too.
	#
	# This is quite powerful

bcambel / message_queue_pipeline.py

Created December 23, 2012 18:41

	from scrapy.xlib.pydispatch import dispatcher
	from scrapy import signals
	from scrapy.exceptions import DropItem
	from scrapy.utils.serialize import ScrapyJSONEncoder

	from carrot.connection import BrokerConnection
	from carrot.messaging import Publisher

	from twisted.internet.threads import deferToThread

bcambel / INSTALL.txt

Created June 24, 2013 21:37 — forked from mrjoes/INSTALL.txt

Bahadir Cambel bcambel