Kent Kawashima kentwait

Crawling Anonymously with Tor in Python

adapted from the article "Crawling anonymously with Tor in Python" by S. Acharya, Nov 2, 2013.

The most common use-case is to be able to hide one's identity using TOR or being able to change identities programmatically, for example when you are crawling a website like Google and you don’t want to be rate-limited or blocked via IP address.

Tor

Install Tor.

This gist lets you keep IPython notebooks in git repositories. It tells git to ignore prompt numbers and program outputs when checking that a file has changed.

To use the script, follow the instructions given in the script's docstring.

For further details, read this blogpost.

The procedure outlined here is inspired by this answer on Stack Overflow.

	#!/usr/bin/python
	# coding=utf-8

	# Python version of Zach Holman's "spark"
	# https://github.com/holman/spark
	# by Stefan van der Walt <[email protected]>

	"""
	USAGE:

	BOOL isDarkImage(UIImage* inputImage){

	BOOL isDark = FALSE;

	CFDataRef imageData = CGDataProviderCopyData(CGImageGetDataProvider(inputImage.CGImage));
	const UInt8 *pixels = CFDataGetBytePtr(imageData);

	int darkPixels = 0;

	int length = CFDataGetLength(imageData);

	#!/usr/bin/env python
	"""strip outputs from an IPython Notebook

	Opens a notebook, strips its output, and writes the outputless version to the original file.

	Useful mainly as a git filter or pre-commit hook for users who don't want to track output in VCS.

	This does mostly the same thing as the `Clear All Output` command in the notebook UI.

	LICENSE: Public Domain

	import pyaudio
	import wave
	import audioop
	from collections import deque
	import os
	import urllib2
	import urllib
	import time
	import math

	package main

	import "fmt"

	// 料理を作るI/F
	type Cooker interface {
	Cook() string
	}

	// 塗るI/F

	import math
	import multiprocessing
	import random
	import sys
	import time


	def merge(*args):
	# Support explicit left/right args, as well as a two-item
	# tuple which works more cleanly with multiprocessing.

	// Playbook - http://play.golang.org/p/3wFl4lacjX

	package main

	import (
	"bytes"
	"crypto/aes"
	"crypto/cipher"
	"crypto/rand"
	"encoding/base64"

	1. Check Python3 root
	>>> import sys
	>>> import os
	>>> sys.executable
	'/usr/local/bin/python3'

	OR

	$ which python3
	/usr/local/bin/python3