Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save rssanders3/3d7631f29565e3c2bbd71f1d8596d467 to your computer and use it in GitHub Desktop.

Select an option

Save rssanders3/3d7631f29565e3c2bbd71f1d8596d467 to your computer and use it in GitHub Desktop.
val checkpointDirectory = "hdfs://..." // define checkpoint directory
// Function to create and setup a new StreamingContext
def functionToCreateContext(): StreamingContext = {
val ssc = new StreamingContext(...) // new context
val lines = ssc.socketTextStream(...) // create DStreams
...
ssc.checkpoint(checkpointDirectory) // set checkpoint directory
ssc // Return the StreamingContext
}
// Get StreamingContext from checkpoint data or create a new one
val ssc = StreamingContext.getOrCreate(checkpointDirectory, functionToCreateContext)
ssc.start() // Start the context
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment