Skip to content

Instantly share code, notes, and snippets.

@marcovivero
Created July 28, 2015 01:08
Show Gist options
  • Save marcovivero/be469561e40ac42644b1 to your computer and use it in GitHub Desktop.
Save marcovivero/be469561e40ac42644b1 to your computer and use it in GitHub Desktop.
def timePartition (
data: DataFrame,
timeCol : String,
startTime : Long = 1404198000000L
) : Seq[DataFrame] = {
val addTimeMilli : Long = 604800000L // Discretize time into weeks.
val rightNowMilli : Long = new Date().getTime
(1 to (rightNowMilli / addTimeMilli).toInt).map(k => new Timestamp(startTime + k.toLong * addTimeMilli))
.map(time => data.filter(data(timeCol) < time))
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment