Skip to content

Instantly share code, notes, and snippets.

@j-thepac
Created April 5, 2022 03:45
Show Gist options
  • Save j-thepac/8129c68a22d9a441be160c6521c4ad9d to your computer and use it in GitHub Desktop.
Save j-thepac/8129c68a22d9a441be160c6521c4ad9d to your computer and use it in GitHub Desktop.
Send Simple text Files as Dstream
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.dstream.DStream
object DStreamSimpleText extends App {
val conf = new SparkConf().setMaster("local[*]").setAppName("App name")
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc, Seconds(5))
val filestream: DStream[String] = ssc.textFileStream(
"/Users/deepakjayaprakash/Downloads/testing"
) // read new file
val word: DStream[(String, Int)] =
filestream.flatMap(f => f.split(" ")).map(word => (word, 1)).reduceByKey(_ + _)
word.print() //Shd have - foreachRDD / saveAsTextFiles / print
ssc.start()
ssc.awaitTermination()
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment