Skip to content

Instantly share code, notes, and snippets.

View alexklibisz's full-sized avatar
:octocat:

Alex Klibisz alexklibisz

:octocat:
View GitHub Profile
@alexklibisz
alexklibisz / lsh.py
Created May 24, 2018 05:02
ElastiK-Nearest-Neighbors LSH Example
import numpy as np
def make_lsh_model(nb_tables, nb_bits, nb_dimensions, vector_sample):
# vector_sample: np arr w/ shape (2 * nb_tables * nb_tables, nb_dimensions).
# normals, midpoints: np arrs w/ shape (nb_bits, nb_dimensions)
# thresholds: np arrs w/ shape (nb_bits)
# all_normals, all_thresholds: lists w/ one normal, one threshold per table.
all_normals, all_thresholds = [], []
for i in range(0, len(vector_sample), 2 * nb_bits):
vector_sample_a = vector_sample[i:i + nb_bits]
@alexklibisz
alexklibisz / index.md
Created May 13, 2018 23:18
Elasticsearch-Aknn Project Description

Test

@alexklibisz
alexklibisz / es_index.py
Created May 3, 2018 17:01
elasticsearch index JSON documents
"""Read cleaned twitter statuses from disk and insert them
to local elasticsearch instance.
Json downloads taken from here: http://jmcauley.ucsd.edu/data/amazon/
"""
from tqdm import tqdm
from elasticsearch import Elasticsearch, helpers
from pprint import pprint
from time import time
import json
import pdb
@alexklibisz
alexklibisz / code.py
Created July 5, 2017 00:59
17softmax
inputs = Input((200,200,3))
res = resnet50.ResNet50(include_top=False, weights=None, input_tensor=inputs)
# Add a classifier for each class instead of a single classifier.
res_out = Flatten()(res.output)
res_out = Dropout(0.1)(res_out)
classifiers = []
for n in range(17):
classifiers.append(Dense(2, activation='softmax')(res_out))
@alexklibisz
alexklibisz / fancylogloss.py
Last active June 26, 2017 20:42
fancy weighted log loss for keras
def wlogloss(yt, yp):
'''Weighted log loss for each example in batch.'''
# Weight false negative errors. This should decrease as recall increases.
# Get the mean (softmax) activation of outputs that should be positive.
meanpos = K.sum(yp * yt) / (K.sum(yt) + K.epsilon())
# This is the maximum up-weighting value for false negative errors.
wfnmax = 20.
# Compute the false negative multiplier between 0 and wfnmax. Maybe clipping this at (1, wfnmax) would be smarter.
wfnmult = (1. - meanpos) * wfnmax
@alexklibisz
alexklibisz / unet3d.prototxt
Created May 2, 2017 13:42
UNet 3D protoxt
name: 'kidney_seg_no_BN'
force_backward: true
layer { top: 'data' top: 'label' top: 'weights' name: 'loaddata' type: 'HDF5Data' hdf5_data_param { source: 'trainfileList.txt' batch_size: 1 } include: { phase: TRAIN }}
layer { type: 'CreateDeformation'
top: 'def'
name: 'create_deformation'
create_deformation_param {
random_offset_range_from_ignore_label: 0
batch_size: 1 nz: 116 ny: 132 nx: 132 ncomponents: 3
@alexklibisz
alexklibisz / unet.prototxt
Created April 14, 2017 17:40
unet prototxt
name: 'phseg_v5'
force_backward: true
layers { top: 'data' top: 'label' name: 'loaddata' type: HDF5_DATA hdf5_data_param { source: 'aug_deformed_phseg_v5.txt' batch_size: 1 } include: { phase: TRAIN }}
layers { bottom: 'data' top: 'd0b' name: 'conv_d0a-b' type: CONVOLUTION blobs_lr: 1 blobs_lr: 2 weight_decay: 1 weight_decay: 0 convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layers { bottom: 'd0b' top: 'd0b' name: 'relu_d0b' type: RELU }
layers { bottom: 'd0b' top: 'd0c' name: 'conv_d0b-c' type: CONVOLUTION blobs_lr: 1 blobs_lr: 2 weight_decay: 1 weight_decay: 0 convolution_param { num_output: 64 pad: 0 kernel_size: 3 engine: CAFFE weight_filler { type: 'xavier' }} }
layers { bottom: 'd0c' top: 'd0c' name: 'relu_d0c' type: RELU }
layers { bottom: 'd0c' top: 'd1a' name: 'pool_d0c-1a' type: POOLING pooling_param { pool: MA
@alexklibisz
alexklibisz / weighted_log_loss.py
Created April 11, 2017 20:30
Keras weighted log loss
def weighted_log_loss(yt, yp):
'''Log loss that weights false positives or false negatives more.
Punish the false negatives if you care about making sure all the neurons
are found and don't mind some false positives. Vice versa for punishing
the false positives. Concept taken from the UNet paper where they
weighted boundary errors to get cleaner boundaries.'''
emphasis = 'fn'
assert emphasis in ['fn', 'fp']
m = 2
@alexklibisz
alexklibisz / vnet prototxt
Created April 8, 2017 13:53
vnet prototxt
input: "data"
input_shape{dim: 1 dim: 1 dim: 128 dim: 128 dim: 64}
layer {
name: "conv_in128_chan16"
type: "Convolution"
bottom: "data"
top: "conv_in128_chan16"
param {
lr_mult: 1.0
@alexklibisz
alexklibisz / demo.js
Created March 16, 2016 06:02
quick-and-dirty-multi-property-search-02
function matchFilter(allItems, query, threshold) {
// Create an array of properties that are defined in the query.
// For the example, it will be [food_type, neighborhood]
const properties = Object.keys(query)
.filter(key => query[key].trim().length > 0);
// Create a comparison string for the query item.
// For the example, it will be "Mxicanmarketsquare"
const queryComp = properties.map(p => query[p]).join();