This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from collections import namedtuple | |
def convert(dictionary): | |
return namedtuple('GenericDict', dictionary.keys())(**dictionary) | |
""" | |
>>> d = dictionary(a=1, b='b', c=[3]) | |
>>> named = convert(d) | |
>>> named.a == d.a | |
True | |
>>> named.b == d.b |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from scrapy import log | |
from scrapy.item import Item | |
from scrapy.http import Request | |
from scrapy.contrib.spiders import XMLFeedSpider | |
def NextURL(): | |
""" | |
Generate a list of URLs to crawl. You can query a database or come up with some other means | |
Note that if you generate URLs to crawl from a scraped URL then you're better of using a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env python | |
# -*- coding: utf-8 -*- | |
import urllib2 | |
gh_url = 'https://api.github.com' | |
req = urllib2.Request(gh_url) | |
password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm() |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Ubuntu upstart file at /etc/init/yourservice.conf | |
pre-start script | |
mkdir -p /var/log/yourcompany/ | |
end script | |
respawn | |
respawn limit 15 5 | |
start on runlevel [2345] |
NewerOlder