davideicardi

Avro serialization

There are 4 possible serialization format when using avro:

• Avro Json encoding
• Avro Data Serialization (https://avro.apache.org/docs/current/spec.html#Data+Serialization) Binary format with an header that contains the full schema, this is the format usually used when writing Avro files
• Avro Single Object Encoding (https://avro.apache.org/docs/current/spec.html#single_object_encoding) Binary format with an header with only the fingerprint/id of the schema, this it the format used by Kafka (see this
• Avro Binary Encoding (https://avro.apache.org/docs/current/spec.html#binary_encoding)

Mageswaran1989

nsivabalan

import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.sql.SaveMode._

val sc = spark.sparkContext
val ssc = new StreamingContext(sc, Seconds(1))

val inputPath = "/tmp/inputDir/"