Deron Eriksson deroneriksson

Python Stuff

NumPy array - print without cutoffs

from scipy.ndimage import measurements
im = array(...)
labels, num_features = measurements.label(im)
set_printoptions(threshold=nan, linewidth=nan)
print(str(labels[0:100,0:100]))

MNIST LeNet

import org.apache.sysml.api.mlcontext._
val ml = new MLContext(spark)
val clf = ml.nn.examples.Mnist_lenet
val dummy = clf.generate_dummy_data
val dummyVal = clf.generate_dummy_data
val params = clf.train(dummy.X, dummy.Y, dummyVal.X, dummyVal.Y, dummy.C, dummy.Hin, dummy.Win, 1)
val probs = clf.predict(dummy.X, dummy.C, dummy.Hin, dummy.Win, params.W1, params.b1, params.W2, params.b2, params.W3, params.b3, params.W4, params.b4)

SystemML Stuff

Run SystemML via MLContext in Spark Shell with SystemML jar file

spark-shell --executor-memory 4G --driver-memory 4G --driver-class-path=target/SystemML.jar

Run SystemML via MLContext in Spark Shell without SystemML jar file

spark-shell --executor-memory 4G --driver-memory 4G --driver-class-path="target/classes:target/lib/antlr4-runtime-4.5.3.jar:target/lib/wink-json4j-1.4.jar"

Handy Regular Expressions

Javadoc match: /\*\*(?s:(?!\*/).)*\*/

Javadoc match public methods: /\*\*(?s:(?!\*/).)*\*/(\s*)public

Trailing whitespace: [ \t]+$

Four spaces at beginning of line: ^[ ]{4}

Apache Committer Notes

Deploying to Apache Snapshot Repository

To deploy project artifacts to the Apache Snapshot Repository takes a little set-up. Here are the steps that I followed.

Maven Password Encryption

Follow the instructions here: https://maven.apache.org/guides/mini/guide-encryption.html

SystemML Python Notes

Running SystemML Python Tests

pip3 install -U -e src/main/python
mvn clean package
PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path target/SystemML.jar src/main/python/tests/test_mllearn.py
PYSPARK_PYTHON=python3 spark-submit --master local[*] --driver-class-path target/SystemML.jar src/main/python/tests/test_mlcontext.py

Jupyter Notes for SystemML Cancer Project

Install Python 3

brew update
brew install python3

Install Python Packages including Jupyter

SystemML with Jupyter Notes

Install Jupyter

brew install -U python (update my already installed python)
brew install python3
pip3 install jupyter
pip3 install numpy matplotlib scipy scikit-learn pandas

Release Candidate Version: 0.10.0-incubating-rc1

Release Candidate Checklist

Item		Status	Notes


	Recursively remove .DS_Store files
	find . -name '.DS_Store' -type f -delete