Created
July 22, 2022 08:43
-
-
Save AayushSameerShah/0064e3a6b4137c5617a13a1c9c11f926 to your computer and use it in GitHub Desktop.
When you are struggling for making the dataframe from List<List<Object>> get here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// suppose we have the list like this (not to run but just idea) | |
List<List<Integer>> data = [ | |
[1, 2, 3], | |
[2, 3, 4], | |
[3, 4, 5] | |
]; | |
// Now to convert each List<Integer> to Row so that can be used to make DF | |
List<Row> rows = new ArrayList<>(); | |
for (List<Integer> that_line : data){ | |
Row row = RowFactory.create(that_line.toArray()); | |
rows.add(row); | |
} | |
// Then just make the dataframe! (no instead of using RDD, use the List<Row> | |
Dataset<Row> r2DF = sparkSession.createDataFrame(rows, schema); // supposing you have schema already. | |
r2DF.show(); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From this post: https://stackoverflow.com/questions/47262422/how-to-convert-java-arraylist-to-apache-spark-dataset