Sean McNamara mcnamaras
mcnamaras
/ gist:040a362ca8100347e1a6
Created
October 13, 2014 05:08
spark streaming broadcast variable wrapper
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import java.io.{ObjectInputStream, ObjectOutputStream} | |
| import org.apache.spark.broadcast.Broadcast | |
| import org.apache.spark.streaming.StreamingContext | |
| import scala.reflect.ClassTag | |
| // This wrapper lets us update brodcast variables within DStreams' foreachRDD | |
| // without running into serialization issues | |
| case class BroadcastWrapper[T: ClassTag]( | |
| @transient private val ssc: StreamingContext, | |
| @transient private val _v: T |