Jakub Janarek jxub

Web Scraping Workshop

Using Requests and Beautiful Soup, with the most recent Beautiful Soup 4 docs.

Getting Started

Install our tools (preferably in a new virtualenv):

pip install beautifulsoup4

tmux cheatsheet

As configured in my dotfiles.

start new:

tmux

start new with session name:

Locate the section for your github remote in the .git/config file. It looks like this:

[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = [email protected]:joyent/node.git

Now add the line fetch = +refs/pull/*/head:refs/remotes/origin/pr/* to this section. Obviously, change the github url to match your project's URL. It ends up looking like this:

Easy parallel python with concurrent.futures

As of version 3.3, python includes the very promising concurrent.futures module, with elegant context managers for running tasks concurrently. Thanks to the simple and consistent interface you can use both threads and processes with minimal effort.

For most CPU bound tasks - anything that is heavy number crunching - you want your program to use all the CPUs in your PC. The simplest way to get a CPU bound task to run in parallel is to use the ProcessPoolExecutor, which will create enough sub-processes to keep all your CPUs busy.

We use the context manager thusly:

with concurrent.futures.ProcessPoolExecutor() as executor:

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

-E: will describe the underlaying queries of the \ commands (cool for learning!)
-l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

	<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>

	<groupId>org.example</groupId>
	<artifactId>jpademo</artifactId>
	<version>1.0</version>
	<packaging>jar</packaging>

	<scm>

	#!/usr/bin/env python2.7

	import time

	_URL = 'http://localhost/tmp/derp.html'
	_NUMBER = 1000


	def test_urllib2():
	import urllib2

	/**
	* More info?
	* [email protected]
	* http://aspyct.org
	*
	* Hope it helps :)
	*/

	#include <stdio.h>
	#include <stdlib.h>

	13:15 <xQuasar> \| HASKELL IS FOR FUCKIN FAGGOTS. YOU'RE ALL A BUNCH OF
	\| FUCKIN PUSSIES
	13:15 <xQuasar> \| JAVASCRIPT FOR LIFE FAGS
	13:16 <luite> \| hello
	13:16 <ChongLi> \| somebody has a mental illness!
	13:16 <merijn> \| Wow...I suddenly see the error of my ways and feel
	\| compelled to write Node.js!
	13:16 <genisage> \| hi
	13:16 <luite> \| you might be pleased to learn that you can compile
	\| haskell to javascript now