Skip to content

Instantly share code, notes, and snippets.

import asyncio
import aiohttp
import time
async def gather_with_concurrency(n, *tasks):
semaphore = asyncio.Semaphore(n)
async def sem_task(task):
async with semaphore:
@sany2k8
sany2k8 / upload-to-aws-flask.md
Created December 16, 2021 13:02 — forked from leongjinqwen/upload-to-aws-flask.md
upload files to aws s3 bucket with flask

Upload files to AWS

Make sure you already have S3 bucket, access key and secret key before go through this notes.

How to connect to AWS?

Boto3 allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2.

Step 1: Install boto3 with pip

pip install boto3
@sany2k8
sany2k8 / script.sh
Created October 25, 2021 08:49 — forked from vielhuber/script.sh
PostgreSQL: Backup and restore export import pg_dump with password on command line #sql
# best practice: linux
nano ~/.pgpass
*:5432:*:username:password
chmod 0600 ~/.pgpass
# best practice: windows
edit %APPDATA%\postgresql\pgpass.conf
*:5432:*:username:password
# linux
@sany2k8
sany2k8 / resize.py
Created July 18, 2021 07:55 — forked from franzwong/resize.py
Resize image with Python
# Require PIL (Python Imaging Library)
import traceback, Image
def resize():
filePath = 'example.jpg'
ratio = 0.5
image = Image.open(filePath)
width = image.size[0]
@sany2k8
sany2k8 / Add_Existing_Project_To_Git.md
Created March 17, 2021 14:06 — forked from alexpchin/Add_Existing_Project_To_Git.md
Add Existing Project To Git Repo

Adding an existing project to GitHub using the command line

Simple steps to add existing project to Github.

1. Create a new repository on GitHub.

In Terminal, change the current working directory to your local project.

2. Initialize the local directory as a Git repository.

git init

API workthough

  1. Open a browser

    # start an instance of firefox with selenium-webdriver
    driver = Selenium::WebDriver.for :firefox
    # :chrome -> chrome
    # :ie     -> iexplore
    
  • Go to a specified URL
# 10_basic.py
# 15_make_soup.py
# 20_search.py
# 25_navigation.py
# 30_edit.py
# 40_encoding.py
# 50_parse_only_part.py
@sany2k8
sany2k8 / scrapy_cheatsheet.md
Created February 24, 2021 06:40 — forked from zlin888/scrapy_cheatsheet.md
scrapy cheatsheet

Scrapy Cheatsheet

For test

scrapy shell https://example.com/

Run

scrapy crawl spider_name

Output

scrapy crawl dapps -o data/07-07-dapps.csv
scrapy crawl dapps -t csv -o - >"data/dapp/$DATE-dapp.csv"

@sany2k8
sany2k8 / get_url_link.py
Created February 22, 2021 09:20 — forked from elena-roff/get_url_link.py
Creates a clickable URL from two fields of the pandas DataFrame
@sany2k8
sany2k8 / pg_stat_statements
Created January 6, 2021 08:00 — forked from troyk/pg_stat_statements
enable postgres pg_stat_statements
1) see re: increasing shmmax http://stackoverflow.com/a/10629164/1283020
2) add to postgresql.conf:
shared_preload_libraries = 'pg_stat_statements' # (change requires restart)
136 pg_stat_statements.max = 1000
137 pg_stat_statements.track = all
3) restart postgres
4) check it out in psql