Skip to content

Instantly share code, notes, and snippets.

@href
href / dict_namedtuple.py
Created October 27, 2011 12:00
Convert any dictionary to a named tuple
from collections import namedtuple
def convert(dictionary):
return namedtuple('GenericDict', dictionary.keys())(**dictionary)
"""
>>> d = dictionary(a=1, b='b', c=[3])
>>> named = convert(d)
>>> named.a == d.a
True
>>> named.b == d.b
@saidimu
saidimu / gist:1024207
Created June 14, 2011 02:36
Generating URLs to crawl from outside a Scrapy spider
from scrapy import log
from scrapy.item import Item
from scrapy.http import Request
from scrapy.contrib.spiders import XMLFeedSpider
def NextURL():
"""
Generate a list of URLs to crawl. You can query a database or come up with some other means
Note that if you generate URLs to crawl from a scraped URL then you're better of using a
@kennethreitz
kennethreitz / 0_urllib2.py
Created May 16, 2011 00:17
urllib2 vs requests
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import urllib2
gh_url = 'https://api.github.com'
req = urllib2.Request(gh_url)
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
@c4milo
c4milo / yourservice.conf
Created April 25, 2011 18:05
upstart example script
# Ubuntu upstart file at /etc/init/yourservice.conf
pre-start script
mkdir -p /var/log/yourcompany/
end script
respawn
respawn limit 15 5
start on runlevel [2345]