mjwillson’s gists

mjwillson / gist:675cc0259e4291d97104

Created September 7, 2015 17:55 — forked from benanne/gist:1759022

Theano AdvancedSubtensor memory leak

	import theano.tensor as T
	import theano
	import numpy as np
	import gc

	def freemem():
	gc.collect()
	gc.collect()
	gc.collect()
	return theano.sandbox.cuda.cuda_ndarray.cuda_ndarray.mem_info()[0] / 1024**2

mjwillson / iterable.py

Last active August 29, 2015 14:27

Decorate a generator function (or other iterator-returning function) as a multi-shot iterable. A fix for many Python gotchas relating to use of one-shot iterators

	class iterable(object):
	"""Decorates a generator function (or any other iterator-returning
	function) as something which implements the iterable protocol and
	can be safely passed to other code which may iterate over it
	multiple times.

	Usage:

	@iterable
	def foo():

mjwillson / ngrams_via_striding.py

Last active August 1, 2017 04:19

Matrix of sliding window ngrams without any copying via numpy striding tricks

	from numpy.lib.stride_tricks import as_strided
	def ngrams_via_striding(array, order):
	itemsize = array.itemsize
	assert array.strides == (itemsize,)
	return as_strided(array, (max(array.size + 1 - order, 0), order), (itemsize, itemsize))

	In [71]: a = numpy.arange(10)

	In [72]: ngrams_via_striding(a, 4)
	Out[72]:

mjwillson / ann.rb

Last active December 29, 2015 16:09

ann -- ultra-basic console-based multiclass text annotation tool

	#!/usr/bin/env ruby

	require 'optparse'

	OPTIONS = {}
	PARSER = OptionParser.new do \|opts\|
	opts.banner = "Usage: #{$0} [OPTIONS] INPUT_FILE [HOTKEY OUTPUT_FILE]..."
	opts.separator(<<END

	#{$0} -- ultra-basic console-based multiclass text annotation tool

mjwillson / scoped_streams.clj

Last active December 17, 2015 14:59

scoped streams in clojure

	(ns streams.core
	(:require [clojure.java.io :as io]))

	(def END (Object.))

	(defprotocol Stream
	(with-generator [_ callback]
	"Should call callback with a generator function, finally closing any
	resources associated with the stream after the callback returns.

mjwillson / sexpr-html-templating.clj

Last active December 17, 2015 12:49

Proof of concept: S-expressions for HTML templating in clojure

	(defprotocol ToHtml
	(to-html [x]))

	(extend-protocol ToHtml
	String
	(to-html [s]
	(clojure.string/escape s {\< "<" \> ">" \" """ \& "&"}))

	clojure.lang.IPersistentMap
	(to-html [attrs]

mjwillson / Encoding.java

Created April 30, 2013 16:33

convert between float arrays and byte arrays

	package randomindexing;

	/* Helpers for encoding various primitive arrays as byte arrays.
	(Why this isn't in the stdlib I have no idea! feel free to replace
	with some existing library implementation...)

	All encodings use little-endian byte order.
	I benchmarked using
	ByteBuffer.wrap(b).order(LITTLE_ENDIAN).asFloatBuffer().get(f),
	but it's around 3 times slower even on big input.

mjwillson / mapreduce.clj

Last active December 16, 2015 01:19

Clojure-like psuedo-code for something closer to the actual structure of a Hadoop MapReduce computation

	;; I'm sure I'm wrong here -- please correct me!
	;; (Although note I'm not trying to capture the exact algorithms and dataflow complexities of hadoop here, just the logical structure of MapReduce computations and a rough sketch of how they're distributed.)

	;; not just

	(->> data
	(pmap mapper)
	(reduce reducer))

	;; but something more like this.

mjwillson / multimethod-matrix-multiplication.clj

Last active December 10, 2015 17:58

An example of how multi-method-based dispatch might work for a binary operation like matrix multiplication. Illustrates how a variety of coercion-based defaults can be specified to make life easy on the implementer, while still easily allowing for dispatch to optimal implementation-specific routines whenever it's desired.

	;; First some dummy matrix types and operations for us to play with later:
	(deftype GenericDense [])
	(deftype FooMatrix [])
	(deftype BarMatrix [])
	(defn generic-multiply [x y] "generic-multiply")
	(defn foo-multiply [x y] "foo-multiply")
	(defn bar-multiply [x y] "bar-multiply")
	(defn foo-generic-multiply [x y] "foo-generic-multiply")
	(defn generic-foo-multiply [x y] "generic-foo-multiply")
	(defn bar-generic-multiply [x y] "bar-generic-multiply")

mjwillson / tax_bands.rb

Created October 29, 2009 22:41

Same thing in ruby

	Infinity = 1.0/0
	SAVINGS_INCOME_TAX_200910 = [[0, 6475], [0.1, 8915], [0.2, 43875], [0.4, Infinity]]
	INCOME_TAX_200910 = [[0, 6475], [0.2, 43875], [0.4, Infinity]]

	def tax(income, bands=INCOME_TAX_200910)
	rate, threshold = bands.first
	if income <= threshold
	rate * income
	else
	lowered_remaining_bands = bands[1..-1].map {\|r, t\| [r, t - threshold]}

Matthew mjwillson