Skip to content

Instantly share code, notes, and snippets.

View tomquisel's full-sized avatar

Tom Quisel tomquisel

30 followers · 1 following
View GitHub Profile
@tomquisel
tomquisel / findagent
Created February 7, 2015 00:00
find ssh agent socket
function findagent {
if ssh-add -l; then
echo "Your SSH Agent is already working."
return 0
fi
for sock in `ls /tmp/ssh-*/agent.*`; do
export SSH_AUTH_SOCK=$sock
if ssh-add -l; then
echo "Your SSH Agent is fixed. New socket=$sock."
return 0
@tomquisel
tomquisel / gist:9ea45a3c420ba13b3dab
Created February 13, 2015 01:25
Base python template
#!/usr/bin/env python
import sys, os, re, random
import time, datetime
from collections import defaultdict
import pandas as pd
import numpy as np
from argparse import ArgumentParser
def main():
args = handle_args()
@tomquisel
tomquisel / r_lm.py
Created October 13, 2015 23:25
R linear regression in ipython notebook
%reload_ext rpy2.ipython
%R -n library(dplyr)
%%R
do_regression = function(mpr, features) {
df = read.csv("activity_quantity_df.csv", header=T)
df$mem_gender = as.factor(df$mem_gender)
df$mem_age = as.factor(df$mem_age)
feature_str = paste(features, collapse=" + ")
print(paste(mpr, "~", feature_str, " + mem_gender*mem_age"))
@tomquisel
tomquisel / rsyslogd_tag.md
Last active November 17, 2015 20:31
How to direct logs to a file based on tag in rsyslogd

How to direct logs to a file using a logger tag

ls /etc/rsyslog.d/

20-ufw.conf  21-cloudinit.conf  50-default.conf  60-etl.conf  postfix.conf

cat /etc/rsyslog.d/60-etl.conf

:syslogtag, isequal, "claims_etl:" /var/log/etl/claims.log
@tomquisel
tomquisel / es_read.py
Created November 18, 2015 21:52
Elasticsearch -> Spark dataframe
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
df = sqlContext.read.format("org.elasticsearch.spark.sql").load("index/type")
df.printSchema()
@tomquisel
tomquisel / compressed_hdf5.py
Created December 3, 2015 18:15
df.to_hdf('/home/tom/data/behaviorXoutcomes/user_features.h5', 'data', complevel=1, complib='zlib')
@tomquisel
tomquisel / keybase.md
Created December 12, 2015 03:30

Keybase proof

I hereby claim:

  • I am tomquisel on github.
  • I am tom (https://keybase.io/tom) on keybase.
  • I have a public key whose fingerprint is 5772 C3AE 84D2 175D 3C2C DE71 1BA1 B957 7D34 77A2

To claim this, I am signing this object:

@tomquisel
tomquisel / r_wilcoxon.py
Created April 6, 2016 20:54
def do_R_wilcoxon(before, after, paired=True, two_tailed=False):
b_mat = before.as_matrix()
a_mat = after.as_matrix()
alternative = 'g'
if two_tailed:
alternative = 'two.sided'
%R -i b_mat,a_mat,paired,alternative -o wilcox_res wilcox_res=wilcox.test(b_mat, a_mat, paired=paired, exact=TRUE, conf.int=TRUE, alternative=alternative)
#print list(enumerate(wilcox_res))
# this returns just the p-value and the conf interval
return {
@tomquisel
tomquisel / prefit_voting_classifier.py
Last active August 29, 2024 17:40
Version of scikit-learn's VotingClassifier that uses prefit models rather than requiring a refit.
class VotingClassifier(object):
"""Stripped-down version of VotingClassifier that uses prefit estimators"""
def __init__(self, estimators, voting='hard', weights=None):
self.estimators = [e[1] for e in estimators]
self.named_estimators = dict(estimators)
self.voting = voting
self.weights = weights
def fit(self, X, y, sample_weight=None):
raise NotImplementedError
@tomquisel
tomquisel / parse_xdr.ts
Created February 15, 2019 21:54
convert Stellar XDR to minimal JSON object
// adapted from: https://github.com/stellar/laboratory/blob/master/src/utilities/extrapolateFromXdr.js
import * as _ from "lodash";
import { xdr, StrKey, Keypair, Operation } from "stellar-sdk";
/**
* Convert XDR string to minimal JSON object
* @param input input xdr string
* @param type type of XDR (TransactionResult, TransactionMeta, etc...)
*/
export function extrapolateFromXdr(input: string, type: string) {