Skip to content

Instantly share code, notes, and snippets.

View dcragusa's full-sized avatar

David Ragusa dcragusa

  • L4 Software Engineer at Google
  • San Mateo
View GitHub Profile
@dcragusa
dcragusa / stuck.sql
Last active May 1, 2018 07:00
View and deal with stuck queries on Postgres
-- View activity
select datname,usename,pid,client_addr,waiting,query_start,query from pg_stat_activity;
-- View locks
select * from pg_locks where relation=(select oid from pg_class where relname='tablename_str');
select pg_cancel_backend(pid_int); -- ask nicely
select pg_terminate_backend(pid_int); -- rude
@dcragusa
dcragusa / delete.sh
Created April 12, 2018 23:24
Delete MANY files in a directory
screen ionice -c3 rm -rv dir_to_delete/ | pv -l > /dev/null
@dcragusa
dcragusa / clear.sh
Created August 14, 2018 14:05
Put this in bashrc, r now actually clears terminal instead of scrolling down
function r() {
clear && printf '\033[3J'
}
@dcragusa
dcragusa / multiindex_to_nested_dict.py
Last active September 19, 2024 04:48
Converts a MultiIndexed Dataframe to a nested dict - this was used in a Django template
import pandas as pd
from collections import OrderedDict
from pandas.core.indexes.multi import MultiIndex
def multiindex_to_nested_dict(df: pd.DataFrame, value_only = False) -> OrderedDict:
if isinstance(df.index, MultiIndex):
return OrderedDict((k, multiindex_to_nested_dict(df.loc[k])) for k in df.index.remove_unused_levels().levels[0])
else:
if value_only:
return OrderedDict((k, df.loc[k].values[0]) for k in df.index)
@dcragusa
dcragusa / error_decorator.py
Created September 3, 2018 15:48
A general error catching decorator to be applied to functions: includes an sql rollback and email notification
def error_handling(rollback_toggle=False):
def wrap(func: C) -> C:
@wraps(func)
def func_wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except Exception as e:
tb = sys.exc_info()[2]
exception_locals = tb.tb_next.tb_frame.f_locals
@dcragusa
dcragusa / dictextensions.py
Created September 6, 2018 12:24
A couple of helpful extensions to dict objects
class AttrDict(dict):
# access data by dot notation e.g. {'a': 1} -> d.a = 1
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.__dict__ = self
def __getattr__(self, name):
return self.get(name, None)
@dcragusa
dcragusa / kill_pid_group.sh
Created September 7, 2018 12:45
Kill a process (and any spawned subprocesses) by PID
echo killing
# Get progress group id from pid
PGID=$(ps opgid `cat ${PID_PATH}` | grep -o '[0-9]*')
# Kill the process group
kill -- -$PGID
echo killed
# Now that it's killed, don't forget to remove the PID file
rm ${PID_PATH}
@dcragusa
dcragusa / debug.py
Created September 10, 2018 10:38
Convenience module that when imported automatically drops you into pdb when an exception is hit
# From https://stackoverflow.com/a/242531/2864129
# Put in pythonpath and 'import debug'
import sys
def info(type_, value, tb):
if hasattr(sys, 'ps1') or not sys.stderr.isatty() or type_ is KeyboardInterrupt:
# we are in interactive mode / we don't have a tty-like device / user-triggered, so we call the default hook
sys.__excepthook__(type_, value, tb)
else:
@dcragusa
dcragusa / proxy.py
Created September 19, 2018 11:13
Get a requests session with proxy and appropriate user agent
import requests
from requests.adapters import HTTPAdapter
PROXIES = {
'http': 'http://proxy.example.com:3128',
'https': 'https://proxy.example.com:3128'
}
class ProxyUAAdapter(HTTPAdapter):
@dcragusa
dcragusa / sftp.py
Created September 19, 2018 11:45
An SFTP client based on Paramiko, with utility function to download a whole folder tree
import os
import stat
from paramiko import SSHClient, SFTPClient, AutoAddPolicy
from tools.file_tools import make_775_dir, file_exists
join = os.path.join