yzdann

Web Scraping Workshop

Using Requests and Beautiful Soup, with the most recent Beautiful Soup 4 docs.

Getting Started

Install our tools (preferably in a new virtualenv):

pip install beautifulsoup4

yzdann

def clear_data(session):
    meta = db.metadata
    for table in reversed(meta.sorted_tables):
        print 'Clear table %s' % table
        session.execute(table.delete())
    session.commit()

yzdann

"""
Logical deletion for Django models. Uses is_void flag
to hide discarded items from queries. Overrides delete
methods to set flag and soft-delete instead of removing
rows from the database.
"""
from django.apps import apps
from django.contrib.admin.utils import NestedObjects
from django.db import models
from django.db.models import signals

yzdann

Pipenv Cheat Sheet

Install pipenv

pip3 install pipenv

Activate

pipenv shell

yzdann

o.......Open files, directories and bookmarks....................|NERDTree-o|
go......Open selected file, but leave cursor in the NERDTree.....|NERDTree-go|
t.......Open selected node/bookmark in a new tab.................|NERDTree-t|
T.......Same as 't' but keep the focus on the current tab........|NERDTree-T|
i.......Open selected file in a split window.....................|NERDTree-i|
gi......Same as i, but leave the cursor on the NERDTree..........|NERDTree-gi|
s.......Open selected file in a new vsplit.......................|NERDTree-s|
gs......Same as s, but leave the cursor on the NERDTree..........|NERDTree-gs|
O.......Recursively open the selected directory..................|NERDTree-O|
x.......Close the current nodes parent...........................|NERDTree-x|

yzdann

Operational PGP

This is a guide on how to email securely.

There are many guides on how to install and use PGP to encrypt email. This is not one of them. This is a guide on secure communication using email with PGP encryption. If you are not familiar with PGP, please read another guide first. If you are comfortable using PGP to encrypt and decrypt emails, this guide will raise your security to the next level.

yzdann

#!/bin/bash
# Generates ten data files, each 300 MB in size and filled with random data.

[[ ! -d "data/" ]] && mkdir "data/"

for i in {1..10}; do
    dd if="/dev/random" of="data/file${i}" bs=1m count=300
done

yzdann

FILE SPACING:

 # double space a file
 sed G

 # double space a file which already has blank lines in it. Output file
 # should contain no more than one blank line between lines of text.
 sed '/^$/d;G'

yzdann

jq cheatsheet

Extract field from list of objects

jq 'map(.foo)'

[ { foo: 1 }, { foo: 2 } ]
[1, 2]

yzdann

Project Title

One Paragraph of project description goes here

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Prerequisites