Phillip Rhodes mindcrime

🌎

Plotting world domination...

mindcrime / gist:f30b6641a58ecf6f059db9380a865f86

Created February 26, 2017 18:59

Complete example where values seem to not get scaled.

	package org.fogbeam.dl4j.spark;

	import java.io.File;
	import java.util.Arrays;
	import java.util.List;
	import java.util.concurrent.TimeUnit;

	import org.apache.spark.SparkConf;
	import org.apache.spark.api.java.JavaPairRDD;
	import org.apache.spark.api.java.JavaRDD;

mindcrime / gist:6c149827205184617d4edad6f00342f3

Created February 26, 2017 18:53

Attempt to manually use scaler

	JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/cutdown/0/**");

	ImageRecordReader irr = new ImageRecordReader(28, 28, 1, new ParentPathLabelGenerator() );
	List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
	irr.setLabels(labelsList);
	RecordReaderFunction rrf = new RecordReaderFunction(irr);
	JavaRDD<List<Writable>> rdd = origData.map(rrf);


	System.out.println( "DataSet RDD created");

mindcrime / gist:3ff5b382e64e519889b005699b926f53

Created February 26, 2017 11:02

dl4j scaling isssue

	JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/cutdown/0/**");

	ImageRecordReader irr = new ImageRecordReader(28, 28, 1, new ParentPathLabelGenerator() );
	List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
	irr.setLabels(labelsList);
	RecordReaderFunction rrf = new RecordReaderFunction(irr);
	JavaRDD<List<Writable>> rdd = origData.map(rrf);


	System.out.println( "DataSet RDD created");

mindcrime / gist:b1b11121755df79d1ad8e7f3711c763d

Created February 25, 2017 22:31

Apparently working code for loading images with Spark/DL4J

	JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/training/**");

	ImageRecordReader irr = new ImageRecordReader(28,28,1,new ParentPathLabelGenerator());
	List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
	irr.setLabels(labelsList);
	RecordReaderFunction rrf = new RecordReaderFunction(irr);
	JavaRDD<List<Writable>> rdd = origData.map(rrf);


	System.out.println( "DataSet RDD created");

mindcrime / gist:eac54feaeb6181d8ed1732ae7ca4ba1f

Created February 25, 2017 19:46

Loading MNIST in Spark (again)

	package org.fogbeam.dl4j.spark;

	import java.util.Arrays;
	import java.util.List;

	import org.apache.spark.SparkConf;
	import org.apache.spark.api.java.JavaPairRDD;
	import org.apache.spark.api.java.JavaRDD;
	import org.apache.spark.api.java.JavaSparkContext;
	import org.apache.spark.input.PortableDataStream;

mindcrime / gist:043270fa30ebfe9ba557642585c1609b

Created February 25, 2017 17:43

DL4J code for loading MNIST with Spark

	package org.fogbeam.dl4j.spark;

	import java.util.Arrays;
	import java.util.List;

	import org.apache.spark.SparkConf;
	import org.apache.spark.api.java.JavaPairRDD;
	import org.apache.spark.api.java.JavaRDD;
	import org.apache.spark.api.java.JavaSparkContext;
	import org.apache.spark.input.PortableDataStream;

mindcrime / gist:27acd60af805a3b05072606b9a7ab83d

Created February 25, 2017 17:38

DL4J error loading data

	17/02/25 12:32:41 INFO deprecation: map.input.start is deprecated. Instead, use mapreduce.map.input.start
	17/02/25 12:32:41 INFO deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
	17/02/25 12:32:41 INFO deprecation: map.input.length is deprecated. Instead, use mapreduce.map.input.length
	17/02/25 12:32:42 INFO deprecation: map.input.file is deprecated. Instead, use mapreduce.map.input.file
	17/02/25 12:32:42 INFO Nd4jBackend: Loaded [CpuBackend] backend
	17/02/25 12:32:42 INFO NativeOpsHolder: Number of threads used for NativeOps: 4
	17/02/25 12:32:43 INFO Reflections: Reflections took 242 ms to scan 12 urls, producing 29 keys and 176 values
	17/02/25 12:32:44 INFO Reflections: Reflections took 595 ms to scan 12 urls, producing 368 keys and 1424 values
	17/02/25 12:32:44 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
	java.lang.NumberFormatException: For input string: "[[[[0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00,

mindcrime / gist:b697acc97e68b721669353aa8c6b19b2

Created March 30, 2016 00:53

How to add a subdomain with AWS Route 53 using the Java SDK

	AWSCredentials credentials = null;
	try
	{
	credentials = new PropertiesFileCredentialsProvider( "credentials.properties" ).getCredentials();
	}
	catch (Exception e)
	{
	throw new AmazonClientException(
	"Cannot load the credentials from the credential profiles file. " +
	"Please make sure that your credentials file is at the correct " +