blog post ideas

implicit_conversions.md

Using Scala implicit conversions to extend existing types and simplify code flow

When working with various frameworks, you'll find yourself dealing with particular datatypes over and over. In Apache Spark, one of those that you're frequently encounter is the Dataframe. Dataframes hold tabular data and allow you to perform parallelized queries and aggregations over practically limitless amounts of data. They're everywhere, and they're super useful, so you'll probably have lots of them. If you seek to avoid repetition in your code, you'll probably define a bunch of functions that take Dataframes as arguments and return Dataframes as well:

  def transformSomehow(df : DataFrame) : DataFrame = {
    //magic in here
  }

  def transformSomeOtherWay(df : DataFrame) : DataFrame = {
    //more magic
  }

  def readCSV(csvpath:String) : Dataframe =
    sqlContext.read
      .format("com.databricks.spark.csv")
      .option("header", "true") // Use first line of all files as header
      .option("inferSchema", "true") // Automatically infer data types
      .load(csvpath)

This is as it should be - any repeated logic is best coded as a standalone function so that it can be easily tested that way too. But, when you go to use that function, you'll end with a lot of code that looks like this:

val df : Dataframe = readCSV("path/to/csv")
val something = transformSomehow(df)
val anotherThing = transformSomeOtherWay(something)
anotherThing.saveAsTextFile("path/to/df")

This is ok, but we can do better! We've got a couple intermediate variables that we had to introduce, but whose names don't contribute anything to our understanding of the code beyond what the function names we already invoked. What we'd like is to use fluent-style syntax so that the flow of the code is clear, and there are no useless intermediate variables:

readCSV("path/to/csv")
  .transformSomehow    //won't compile: method transformSomehow not found for class Dataframe
  .transformSomeOtherWay
  .saveAsTextFile("path/to/txt")

Essentially, we'd like to add methods to the Dataframe class. In the usual object-oriented manner, we would probably extend it as MyDataframe. This is totally do-able in Scala. However, there are a number of annoying things about using inheritance:

• its invasive. Our code must now use our MyDataframe instead of Dataframe. This is potentially incompatible with other generic Spark code. At the very least its inconvienient. Depending on which transformations we want, we'll have to remember what kind of dataframe we want to construct.
• we'll probably be required to implement various methods we don't care about
• inheritance hierarchies are a pain to manage. multiple inheritance is notoriously tricky.
• I just don't like OOP! So I try to avoid it whenever possible.

We can get around all this and still add methods that feel like they were built into to the class we want to extend. Scala's implicit conversions allow us to take our generic doSomething(df : DataFrame) : DataFrame function and make it work in the manner shown above.

object DataFramePlus {
  object implicits {
    implicit def dFWithExtraOperations(df: DataFrame) = DataFramePlus(df)
  }
}

case class DataFramePlus(df : DataFrame) {
 def transformSomehow(df : DataFrame) : DataFrame = {
    //magic in here
  }
 def transformSomeOtherWay(df : DataFrame) : DataFrame = {
    //more magic
  }
}

To make this code operative, we'll need to import the implicits:

 import DataFramePlus.implicits._
 readCSV("path/to/csv")
  .transformSomehow    //now totally kosher
  .transformSomeOtherWay
  .saveAsTextFile("path/to/text")

##Warnings about implicits

Implicits are a powerful tool for injecting new behavior into old classes. Used judiciously, your code will be a lot simpler. However, they can be confusing if you can't tell where a method is coming from. I import my implicits into the smallest possible enclosing scope to mitigate this possibility. There is also a compile-time cost for method resolution. Using the smallest possible scope helps here too.