Skip to content

Instantly share code, notes, and snippets.

@mindcrime
Created February 25, 2017 22:31
Show Gist options
  • Save mindcrime/b1b11121755df79d1ad8e7f3711c763d to your computer and use it in GitHub Desktop.
Save mindcrime/b1b11121755df79d1ad8e7f3711c763d to your computer and use it in GitHub Desktop.
Apparently working code for loading images with Spark/DL4J
JavaPairRDD<String, PortableDataStream> origData = sc.binaryFiles("/home/prhodes/development/experimental/ai_exp/NeuralNetworkSandbox/mnist_png/training/**");
ImageRecordReader irr = new ImageRecordReader(28,28,1,new ParentPathLabelGenerator());
List<String> labelsList = Arrays.asList( "0", "1", "2", "3", "4", "5", "6", "7", "8", "9" );
irr.setLabels(labelsList);
RecordReaderFunction rrf = new RecordReaderFunction(irr);
JavaRDD<List<Writable>> rdd = origData.map(rrf);
System.out.println( "DataSet RDD created");
JavaRDD<DataSet> trainingData = rdd.map(new DataVecDataSetFunction(1,10, false, null, null ));
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment