Skip to content

Instantly share code, notes, and snippets.

@pulsation

pulsation/spark2-addfile-test.scala

Created May 14, 2019 12:31
Show Gist options
  • Save pulsation/d70d923fc72bc04af844eaf5a7271010 to your computer and use it in GitHub Desktop.
Save pulsation/d70d923fc72bc04af844eaf5a7271010 to your computer and use it in GitHub Desktop.
Download ZIP
addFile test in spark2 shell
Raw
spark2-addfile-test.scala
import org.apache.spark.SparkFiles
import org.apache.spark.SparkContext
val context = spark.sparkContext
val fileName = "file.csv"
val fetchedFile = context.addFile(s"ftp://my-ftp-server.net/$fileName")
val readFile = context.textFile(SparkFiles.get(fileName)).collect()
readFile.foreach(println)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment