sany2k8

import asyncio
import aiohttp
import time


async def gather_with_concurrency(n, *tasks):
    semaphore = asyncio.Semaphore(n)

    async def sem_task(task):
        async with semaphore:

sany2k8

Upload files to AWS

Make sure you already have S3 bucket, access key and secret key before go through this notes.

How to connect to AWS?

Boto3 allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.

Step 1: Install boto3 with pip

pip install boto3

sany2k8

# best practice: linux
nano ~/.pgpass
*:5432:*:username:password
chmod 0600 ~/.pgpass

# best practice: windows
edit %APPDATA%\postgresql\pgpass.conf
*:5432:*:username:password

# linux

sany2k8

# Require PIL (Python Imaging Library)
import traceback, Image

def resize():
	filePath = 'example.jpg'
	ratio = 0.5
	
	image = Image.open(filePath)
	
	width = image.size[0]

sany2k8

Adding an existing project to GitHub using the command line

Simple steps to add existing project to Github.

1. Create a new repository on GitHub.

In Terminal, change the current working directory to your local project.

2. Initialize the local directory as a Git repository.

git init

sany2k8

API workthough

1. Open a browser

   # start an instance of firefox with selenium-webdriver
   driver = Selenium::WebDriver.for :firefox
   # :chrome -> chrome
   # :ie     -> iexplore
   

• Go to a specified URL

sany2k8

# 10_basic.py
# 15_make_soup.py
# 20_search.py
# 25_navigation.py
# 30_edit.py
# 40_encoding.py
# 50_parse_only_part.py

sany2k8

Scrapy Cheatsheet

For test

scrapy shell https://example.com/

Run

scrapy crawl spider_name

Output

scrapy crawl dapps -o data/07-07-dapps.csv
scrapy crawl dapps -t csv -o - >"data/dapp/$DATE-dapp.csv"

sany2k8

def create_link(file_name, file_id):
    return f'http://localhost:port/{file_name}/{file_id}'
  
def make_clickable(val):
    # target _blank to open new window
    return '<a target="_blank" href="{}">{}</a>'.format(val, val)

data['link'] = data.apply(lambda x: create_link(str(x['file_path']), x['file_id']), axis=1)
clickable_table = data.style.format({'link': make_clickable})

sany2k8

1) see re: increasing shmmax http://stackoverflow.com/a/10629164/1283020

2) add to postgresql.conf:
shared_preload_libraries = 'pg_stat_statements'   # (change requires restart)
136 pg_stat_statements.max = 1000
137 pg_stat_statements.track = all

3) restart postgres

4) check it out in psql