Skip to content

Instantly share code, notes, and snippets.

@mindcrime
Created February 26, 2017 11:02
Show Gist options
  • Save mindcrime/3ff5b382e64e519889b005699b926f53 to your computer and use it in GitHub Desktop.
Save mindcrime/3ff5b382e64e519889b005699b926f53 to your computer and use it in GitHub Desktop.
dl4j scaling isssue
JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/cutdown/0/**");
ImageRecordReader irr = new ImageRecordReader(28, 28, 1, new ParentPathLabelGenerator() );
List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
irr.setLabels(labelsList);
RecordReaderFunction rrf = new RecordReaderFunction(irr);
JavaRDD<List<Writable>> rdd = origData.map(rrf);
System.out.println( "DataSet RDD created");
DataNormalization scaler = new ImagePreProcessingScaler(0,1);
JavaRDD<DataSet> trainingData = rdd.map(new DataVecDataSetFunction(0,10, false, scaler, null ));
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment