Last active
September 1, 2017 19:45
-
-
Save hadoopsters/d07e813f6907b0aa35391ccceee7351c to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| // import this guy | |
| import org.apache.spark.sql.hive.HiveContext | |
| // this should look familiar | |
| val conf = new SparkConf() | |
| val sc = new SparkContext(conf) | |
| // setup this fella | |
| val hiveContext = new HiveContext(sc) | |
| val rdd = ... // this is where the data goessssss | |
| // boom shocka-locka, turn that RDD into a DF | |
| hiveContext.createDataFrame(rdd) | |
| .option("orc.compress", "snappy") | |
| .mode(SaveMode.Append) | |
| .orc("/this/is/yet/another/hdfs/directory/") |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment