sany2k8

# best practice: linux
nano ~/.pgpass
*:5432:*:username:password
chmod 0600 ~/.pgpass

# best practice: windows
edit %APPDATA%\postgresql\pgpass.conf
*:5432:*:username:password

# linux

sany2k8

# Require PIL (Python Imaging Library)
import traceback, Image

def resize():
	filePath = 'example.jpg'
	ratio = 0.5
	
	image = Image.open(filePath)
	
	width = image.size[0]

sany2k8

Adding an existing project to GitHub using the command line

Simple steps to add existing project to Github.

1. Create a new repository on GitHub.

In Terminal, change the current working directory to your local project.

2. Initialize the local directory as a Git repository.

git init

sany2k8

API workthough

1. Open a browser

   # start an instance of firefox with selenium-webdriver
   driver = Selenium::WebDriver.for :firefox
   # :chrome -> chrome
   # :ie     -> iexplore
   

• Go to a specified URL

sany2k8

# 10_basic.py
# 15_make_soup.py
# 20_search.py
# 25_navigation.py
# 30_edit.py
# 40_encoding.py
# 50_parse_only_part.py

sany2k8

Scrapy Cheatsheet

For test

scrapy shell https://example.com/

Run

scrapy crawl spider_name

Output

scrapy crawl dapps -o data/07-07-dapps.csv
scrapy crawl dapps -t csv -o - >"data/dapp/$DATE-dapp.csv"

sany2k8

def create_link(file_name, file_id):
    return f'http://localhost:port/{file_name}/{file_id}'
  
def make_clickable(val):
    # target _blank to open new window
    return '<a target="_blank" href="{}">{}</a>'.format(val, val)

data['link'] = data.apply(lambda x: create_link(str(x['file_path']), x['file_id']), axis=1)
clickable_table = data.style.format({'link': make_clickable})

sany2k8

1) see re: increasing shmmax http://stackoverflow.com/a/10629164/1283020

2) add to postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'   # (change requires restart)
136 pg_stat_statements.max = 1000
137 pg_stat_statements.track = all

3) restart postgres

4) check it out in psql

sany2k8

Presto Specific

• Don’t SELECT *, Specify explicit column names (columnar store)
• Avoid large JOINs (filter each table first)
  • In PRESTO tables are joined in the order they are listed!!
  • Join small tables earlier in the plan and leave larger fact tables to the end
  • Avoid cross joins or 1 to many joins as these can degrade performance
• Order by and group by take time
  • only use order by in subqueries if it is really necessary
• When using GROUP BY, order the columns by the highest cardinality (that is, most number of unique values) to the lowest.

sany2k8

MySQL Cheat Sheet

Help with SQL commands to interact with a MySQL database

MySQL Locations

• Mac /usr/local/mysql/bin
• Windows /Program Files/MySQL/MySQL version/bin
• Xampp /xampp/mysql/bin

Add mysql to your PATH