Skip to content

Instantly share code, notes, and snippets.

View benman1's full-sized avatar

Ben Auffarth benman1

218 followers · 227 following
View GitHub Profile
@benman1
benman1 / Dockerfile
Created November 12, 2019 15:47
anaconda dockerfile
FROM continuumio/miniconda3
RUN apt update -qq && apt install -y libgl1-mesa-glx libegl1-mesa libxrandr2 libxrandr2 libxss1 libxcursor1 libxcomposite1 libasound2 libxi6 libxtst6 build-essential vim curl wget libhdf5-dev
RUN useradd -m docker -s /bin/bash -p '*' && chown -R 1000:1000 /opt/conda
USER docker
RUN conda create -n env python=3.8 && RUN echo "source activate env" > ~/.bashrc
ENV PATH /opt/conda/envs/env/bin:$PATH
@benman1
benman1 / pearson_metric_tensorflow.py
Created November 5, 2019 13:24
for regression - calculate the correlation coefficient between predictions and true values
def pearson(x, y):
'''Metric for correlation coefficient.
Use as follows:
model.compile(loss='mse', optimizer='adam', metrics=[pearson])
'''
mx = K.mean(x)
my = K.mean(y)
xm, ym = x-mx, y-my
r_num = K.sum(tf.multiply(xm,ym))
r_den = K.sqrt(tf.multiply(K.sum(K.square(xm)), K.sum(K.square(ym))))
@benman1
benman1 / ArgMaxLayer.py
Created October 22, 2019 08:14
From one-hot encoding to Argmax using a keras layer (should work for tf>=1.13)
import numpy as np
from tensorflow.keras import backend as K
from tensorflow.python.keras.layers import InputSpec, Layer
class Argmax(Layer):
"""
Based on https://github.com/YerevaNN/R-NET-in-Keras/blob/master/layers/Argmax.py
"""
def __init__(self, axis=-1, **kwargs):
super(Argmax, self).__init__(**kwargs)
@benman1
benman1 / list_creation_pattern.kt
Last active September 30, 2019 13:36
create a list asynchronously in Kotlin
import kotlinx.coroutines.*
fun run() = runBlocking {
val l: List<Deferred<Int>> = List(100_000) {
async {
delay(1000)
it
}
@benman1
benman1 / largest.sh
Created August 31, 2019 19:00
largest files in macos using find
find $DIR -type f -exec stat -f "%z %N" {} \; | sort -nr | head -100
@benman1
benman1 / set_java_home.sh
Created July 30, 2019 13:22
automatically determine JAVA_HOME
java_exec=$(readlink -f `which javac`)
export JAVA_HOME="${java_exec::-10}"
@benman1
benman1 / cluster_test.py
Last active July 24, 2019 14:59
distributed tensorflow demo; after https://databricks.com/tensorflow/distributed-computing-with-tensorflow
import tensorflow as tf
# run this script from the master node
cluster = tf.train.ClusterSpec({"distributed": ["192.168.102.90:1111", "192.168.102.90:1111"]})
x = tf.constant(2)
with tf.device("/job:local/replica:0/task:0/device:GPU:0"): #"/job:distributed/task:1"):
@benman1
benman1 / concurrent_for_example.cpp
Created May 8, 2019 14:16
concurrent for loop with fibers in c++ using the fiberize library
#include <fiberize/fiberize.hpp>
#include <iostream>
// after https://github.com/fiberize/fiberize/blob/master/fiberize/examples/helloworld/main.cpp
using namespace fiberize;
#define VECTOR_SIZE 100
int vector[VECTOR_SIZE]
@benman1
benman1 / mini_bert.py
Last active May 1, 2019 13:53
mini bert transformation
from tensorflow.python.keras.layers import ZeroPadding1D, Concatenate, Reshape, TimeDistributed, LSTM, Dropout, Masking, Cropping1D
from tensorflow.python.keras.regularizers import l1
reg = l1(10e-5)
def mini_bert_block(
input_layer, embedding_dim=300,
drop_out=0.5, sequence_length=800, context_width=3,
):
if (context_width % 2) == 0:
@benman1
benman1 / tensorflow_jupyter.sh
Last active April 26, 2019 16:31
run tensorflow jupyter
docker pull tensorflow/tensorflow:1.13.1-gpu-py3-jupyter # or find one here: https://hub.docker.com/r/tensorflow/tensorflow/tags
docker run --runtime=nvidia --name jupyter_ben -d -v /bigusers/ben_again:/home/ben -v /catwalk:/catwalk -v /bigusers/ben:/ben -v /vX:/vX -v /u02/bigdata:/u02 -p 5051:8888 tensorflow/tensorflow:1.13.1-gpu-py3-jupyter
docker logs jupyter_ben # to see the token for identification