Skip to content

Instantly share code, notes, and snippets.

@berngp
Last active December 19, 2015 05:39
Show Gist options
  • Save berngp/5905417 to your computer and use it in GitHub Desktop.
Save berngp/5905417 to your computer and use it in GitHub Desktop.
Spark-Project Build with version qualifier. The gists bellow represent changes to one single **SparkBuild.scala** but to avoid a verbose explanation I am just including the pertinent parts that should be modified.
// Hadoop version to build against. For example, "0.20.2", "0.20.205.0", or
// "1.0.4" for Apache releases, or "0.20.2-cdh3u5" for Cloudera Hadoop.
//val HADOOP_VERSION = "1.0.4"
//val HADOOP_MAJOR_VERSION = "1"
//val HADOOP_YARN = false
// For Hadoop 2 versions such as "2.0.0-mr1-cdh4.2.2", set the HADOOP_MAJOR_VERSION to "2"
val HADOOP_VERSION = "2.0.0-mr1-cdh4.2.2"
val HADOOP_MAJOR_VERSION = "2"
val HADOOP_YARN = false
// For Hadoop 2 YARN support
//val HADOOP_VERSION = "2.0.2-alpha"
//val HADOOP_MAJOR_VERSION = "2"
//val HADOOP_YARN = true
version := "0.8.0-h" + HADOOP_VERSION + "-SNAPSHOT",

$find . -name target -type d | xargs -I % find % -name \*.jar | grep cdh4.2.2

  • spark-bagel_2.9.3-0.8.0-h2.0.0-mr1-cdh4.2.2-SNAPSHOT.jar
  • spark-core_2.9.3-0.8.0-h2.0.0-mr1-cdh4.2.2-SNAPSHOT.jar
  • spark-examples_2.9.3-0.8.0-h2.0.0-mr1-cdh4.2.2-SNAPSHOT.jar
  • spark-repl_2.9.3-0.8.0-h2.0.0-mr1-cdh4.2.2-SNAPSHOT.jar
  • spark-streaming_2.9.3-0.8.0-h2.0.0-mr1-cdh4.2.2-SNAPSHOT.jar
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment