TeraWilliam

Steps to verify our Kafka Mirror App works:

# Start Kafka
docker run -p 2181:2181 -p 9092:9092 \
      --name kafka --rm \
      --env ADVERTISED_HOST=172.17.0.2 \
      --env ADVERTISED_PORT=9092 \
      spotify/kafka
      
# Start producer

TeraWilliam

Williams-iMac:~ william$ python
Python 2.7.13 (default, Dec 18 2016, 07:03:39) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> def stuff():
...   yield 1
...   yield 2
...   yield 3
... 
>>> list(stuff())

TeraWilliam

https://news.ycombinator.com/item?id=6845585

TeraWilliam

    sealed trait DedupState[T]

    object DedupState {
      def apply[T] = NoElement[T]()

      case class EmitElement[T](value: T) extends DedupState[T]

      case class EmmittedElementAlready[T](value: T) extends DedupState[T]

TeraWilliam

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.2"
// http://mvnrepository.com/artifact/com.twitter/chill_2.10
libraryDependencies += "com.twitter" % "chill_2.10" % "0.5.0"