Arnon Rotem-Gal-Oz Arnonrgo
Arnonrgo
/ load-parquet.clj
Created
August 9, 2015 12:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (-> ^SQLContext sqtx | |
| (.read) | |
| (.format "parquet") | |
| (.options (java.util.HashMap. {"mergeSchema" "false" "path" path})) | |
| (.load)) |
Arnonrgo
/ loadParquet.scala
Created
August 9, 2015 12:55
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val file = sqx.read.option("mergeSchema", "false").parquet(path) |
Arnonrgo
/ DirectOutputCommiter.clj
Created
August 9, 2015 13:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (spark-conf/set "spark.hadoop.mapred.output.committer.class" "com.appsflyer.spark.DirectOutputCommitter") |
Arnonrgo
/ DirectParquetOutputCommitter.clj
Created
August 9, 2015 13:37
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| (let [ctx (spark/spark-context conf) | |
| hadoop-conf (.hadoopConfiguration ^JavaSparkContext ctx)] | |
| (.set hadoop-conf "spark.sql.parquet.output.committer.class" "org.apache.spark.sql.parquet.DirectParquetOutputCommitter")) |
OlderNewer