Skip to content

Instantly share code, notes, and snippets.

View billyfung's full-sized avatar

Billy billyfung

View GitHub Profile
@evan-burke
evan-burke / insert_paginator.py
Last active April 15, 2020 22:02
psycopg2 execute_values wrapper for accurate row counts
# One of the fastest ways to insert bulk data into Postgres (at least, aside from COPY) is using the psycopg2 extras function execute_values.
# However, this doesn't return an accurate row count value - instead, it just returns the row count for the last page inserted.
# This wraps the execute_values function with its own pagination to return an accurate count of rows inserted.
# Performance is approximately equivalent to underlying execute_values function - within 5-10% or so in my brief tests.
import psycopg2
import psycopg2.extras
import math
db_connection_string = "dbname=EDITME host=EDITME"
@mcescalante
mcescalante / instructions.md
Last active July 25, 2023 14:07
Run a Flask application with gunicorn and nginx

This guide assumes:

  • You have a VPS (something like DigitalOcean, Linode, etc.)
  • You have a Flask application and a basic understanding of command line instructions
  • You've got nginx installed and know where your configuration files are

Instructions

  1. First, you should install gunicorn on your box or virtualenv with pip:
@conormm
conormm / r-to-python-data-wrangling-basics.md
Last active March 25, 2025 00:24
R to Python: Data wrangling with dplyr and pandas

R to python data wrangling snippets

The dplyr package in R makes data wrangling significantly easier. The beauty of dplyr is that, by design, the options available are limited. Specifically, a set of key verbs form the core of the package. Using these verbs you can solve a wide range of data problems effectively in a shorter timeframe. Whilse transitioning to Python I have greatly missed the ease with which I can think through and solve problems using dplyr in R. The purpose of this document is to demonstrate how to execute the key dplyr verbs when manipulating data using Python (with the pandas package).

dplyr is organised around six key verbs:

@lynsei
lynsei / gist:ff0e6e62e1842220a876
Last active October 3, 2017 17:13
instant-rancher-server-on-automated-ssl-in-10-lines-of-bash-code.sh
INSTANT RANCHER-SERVER ON AUTOMATED SSL in 10 Lines of Bash Code
# make a rancher-server:443 for ca-validated A+ SSL using Ubuntu-Trusty/jWilder-Nginx-Reverse-Proxy/LetsEncrypt+proxy-companion on AWS EC2 (or basically any cloud host using Ubuntu Trusty)
# by Dr. C.Hogan
# install docker fresh on ubuntu-trusty & set things up proper
sudo apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" >> /etc/apt/sources.list.d/docker.list; sudo apt-get-update
@vasanthk
vasanthk / System Design.md
Last active April 28, 2025 19:50
System Design Cheatsheet

System Design Cheatsheet

Picking the right architecture = Picking the right battles + Managing trade-offs

Basic Steps

  1. Clarify and agree on the scope of the system
  • User cases (description of sequences of events that, taken together, lead to a system doing something useful)
    • Who is going to use it?
    • How are they going to use it?

Give me back my sanity

One of the many things I do for my group at work is to take care of automating as many things as possible. It usually brings me a lot of satisfaction, mostly because I get a kick out of making people's lives easier.

But sometimes, maybe too often, I end up in drawn-out struggles with machines and programs. And sometimes, these struggles bring me to the edge of despair, so much so that I regularly consider living on a computer-less island growing vegetables for a living.

This is the story of how I had to install Pandoc in a CentOS 6 Docker container. But more generally, this is the story of how I think computing is inherently broken, how programmers (myself included) tend to think that their way is the way, how we're ultimately replicating what most of us think is wrong with society, building upon layers and layers of (best-case scenario) obscure and/or weak foundations.

*I would like to extend my gratitude to Google, StackOverflow, GitHub issues but mostly, the people who make the

@lost-theory
lost-theory / gist:3005268
Created June 27, 2012 16:35
consume JSON REST API blueprint inside flask app
from flask import Flask, jsonify, Blueprint, current_app
import json
## REST api ###################################################################
api = Blueprint('api', __name__)
@api.route("/users")
def users():
return jsonify(users=[