Skip to content

Instantly share code, notes, and snippets.

@jtratner
jtratner / mode-perf-comparison.md
Last active December 26, 2015 22:49
Series.mode() perf tests.

Results from doing all of value_counts() first

---- Basics - int64 ----
100 length
10000 loops, best of 3: 100 µs per loop
1000 length
10000 loops, best of 3: 166 µs per loop
10000 length
1000 loops, best of 3: 367 µs per loop
100000 length
@jtratner
jtratner / generate_typed_dicts.py
Created April 10, 2018 00:32
Example of auto generating type stubs
from datetime import datetime
STRING_DEFAULT = 'string'
DATETIME_DEFAULT = datetime(2015, 1, 1)
INT_DEFAULT = 123
def generate_typed_dicts(name, nested_dict):
full_context = []
final_dict = [f"class {name}(TypedDict):"]
@jtratner
jtratner / athena_client.py
Created April 10, 2018 00:35
athena client type stubs
from mypy_extensions import TypedDict
QueryExecutionContext = TypedDict('QueryExecutionContext', {
'Database': str,
})
EncryptionConfiguration = TypedDict('EncryptionConfiguration', {
'EncryptionOption': str,
'KmsKey': str,
}, total=False)
ResultConfiguration = TypedDict('ResultConfiguration', {
@jtratner
jtratner / compare_pipfiles.py
Created June 24, 2019 00:04
Compare Pipfiles
import json
import sys
import csv
def pipfile_to_pkg_version(path):
with open(path, "r") as fp:
new_pip = json.load(fp)
by_env = {}
@jtratner
jtratner / follow_workflow.sh
Created July 11, 2019 16:13
Follow DNAnexus workflow
#!/bin/bash
set -o pipefail
ANALYSIS_ID=$1
dx describe $ANALYSIS_ID
project=$(dx describe $ANALYSIS_ID --json | jq -r .project)
function get_jobs_for_state {
dx find jobs --root $ANALYSIS_ID --state $1 --brief --project ${project} --all-jobs
}
until ! (cat <(get_jobs_for_state runnable) <(get_jobs_for_state running) <(get_jobs_for_state waiting_on_input) <(get_jobs_for_state idle) | grep job | head -n 1); do
FIRST_JOB=$(get_jobs_for_state running | head -n 1 | grep job)
Using virtualenv: /private/tmp/example-deleteme/.venv
Updating dependencies
Resolving dependencies...
1: fact: example-deleteme is 0.1.0
1: derived: example-deleteme
1: fact: example-deleteme depends on dxpy (^0.306.0)
1: fact: example-deleteme depends on Sphinx (^3.5.4)
1: selecting example-deleteme (0.1.0)
1: derived: Sphinx (>=3.5.4,<4.0.0)
1: derived: dxpy (>=0.306.0,<0.307.0)