Skip to content

Instantly share code, notes, and snippets.

View alexland's full-sized avatar

doug ybarbo alexland

View GitHub Profile
@alexland
alexland / compress.py
Last active August 29, 2015 14:03
compressed file IO in python
df = '~/path/to/some/text/file.csv'
df1 = 'path/to/some/text/file.gz'
import zlib
import gzip
with open(df, 'r', encoding='utf-8') as fh:
tx = fh.read()
# compress a text file
@alexland
alexland / convert-datetime-to-unix.py
Last active March 8, 2018 01:55
various date/time conversions in python
from datetime import datetime as DT
#------------------ epoch time -----------------#
>>> import time
>>> time.time()
1411060580.205373
@alexland
alexland / NumPy-persist-in-HDF5.ipynb
Last active August 29, 2015 14:02
shows how to persist a NumPy array using HDF5 and how to use that (HDF5) on-disk array for out-of-core matrix computation; this is the JSON (w/ the code embedded) for an ipython notebook; to view it, go to http://nbviewer.ipython.orgin; in the textbox, type in the id for this gist
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
>>> from scipy import misc as MSC
>>> import os
>>> df = os.path.expanduser("~/any-image.png")
>>> # now read the image in as a multi-dimensional array (matrix)
>>> # 'imread' is a wrapper over a PIL method
>>> mg1 = MSC.imread(df)
>>> mg1.shape
@alexland
alexland / serialize-numpy-array.py
Last active November 28, 2023 07:12
serialize, persist, retrieve, and de-serialize a NumPy array as a binary string (any dimension, any dtype); exemplary use case: a web app calculates some result--eg, from a Machine Learning algorithm, using NumPy and the result is a NumPy array; it is efficient to just return that result to rather than persist the array then retrieve it via query
import time
import numpy as NP
from redis import StrictRedis as redis
# a 2D array to serialize
A = 10 * NP.random.randn(10000).reshape(1000, 10)
# flatten the 2D NumPy array and save it as a binary string
array_dtype = str(A.dtype)
@alexland
alexland / softmax-mlp.py
Last active August 29, 2015 14:02
for a multi-layer perceptron using softmax for output layer, transform output-layer vector to probability vector, then calculate cross-entropy error
'''
these 2 functions assume this sort of MLP architecture:
(i) a classification problem modeled w/ softmax activation function
(ii) encode the output layer w/ 1-of-N
> so for raw data like this:
.3, .2, .6, .1, 'class I'
.6, .1, .8, .4, 'class II'
.5, .2, .7, .3, 'class III'
recode it for intput to a softmax MLP like so:
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
#Newbie programmer
def factorial(x):
if x == 0:
return 1
else:
return x * factorial(x - 1)
print factorial(6)
#First year programmer, studied Pascal
@alexland
alexland / column-print.py
Last active March 26, 2019 13:21
printing formatted tables using python 3 "advanced string formatting"
# let's say you have four sequences you want to print columnwise, like so:
19 59 97 44
92 57 63 68
66 21 69 90
75 66 12 19
# mock some data
import random as RND
gen_row = lambda: [ RND.randint(10, 99) for c in range(3) ]
@alexland
alexland / python idioms
Last active August 29, 2015 13:57
useful (but not common known) python idioms
# useful eg, for recursive functions such as merge sort or binary tree building/traversal in which sometimes the fn must return one of two portions of some container (a sequence, tree node, etc.)
def fnx(a):
if a % 2 == 0:
x = 45
y = None
else:
x = None
y = 3
return (x or y) + 2