Kuz Le techkuz

Web Scraping Workshop

Using Requests and Beautiful Soup, with the most recent Beautiful Soup 4 docs.

Getting Started

Install our tools (preferably in a new virtualenv):

pip install beautifulsoup4

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname

This document has now been incorporated into the uWSGI documentation:

http://uwsgi-docs.readthedocs.org/en/latest/tutorials/Django_and_nginx.html

Set up Django, nginx and uwsgi

Steps with explanations to set up a server using:

Branching

Quick Legend

Description, Instructions, Notes

Instance	Branch

-- Firstly, remove PRIMARY KEY attribute of former PRIMARY KEY
ALTER TABLE <table_name> DROP CONSTRAINT <table_name>_pkey;

-- Then change column name of  your PRIMARY KEY and PRIMARY KEY candidates properly.
ALTER TABLE <table_name> RENAME COLUMN <primary_key_candidate> TO id;

	-- show running queries (pre 9.2)
	SELECT procpid, age(clock_timestamp(), query_start), usename, current_query
	FROM pg_stat_activity
	WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%'
	ORDER BY query_start desc;

	-- show running queries (9.2)
	SELECT pid, age(clock_timestamp(), query_start), usename, query
	FROM pg_stat_activity
	WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'

	/*
	Notes:

	Despite Scala's appearances, it is, in fact, a Statically Typed language.
	It has just eliminated a great deal of the "type vomit" people are used
	to seeing in Statically Typed languages (e.g. C, C++, Java). It often
	can infer the type on its own. It also combines functional and
	object-oriented programming paradigms in a fashion that feels similar
	to Python.
	*/

	from celery import Task
	from celery.task import task
	from my_app.models import FailedTask
	from django.db import models

	@task(base=LogErrorsTask)
	def some task():
	return result

	class LogErrorsTask(Task):

	git branch -m old_branch new_branch # Rename branch locally
	git push origin :old_branch # Delete the old branch
	git push --set-upstream origin new_branch # Push the new branch, set local branch to track the new remote

	git fetch upstream
	git reset --hard upstream/master