Skip to content

Instantly share code, notes, and snippets.

@myui
Last active October 29, 2016 00:59
Show Gist options
  • Save myui/b3ebbdd0604eed2b7dfec69682dc15f2 to your computer and use it in GitHub Desktop.
Save myui/b3ebbdd0604eed2b7dfec69682dc15f2 to your computer and use it in GitHub Desktop.
val testData =
ssc.textFileStream(...).map(LabeledPoint.parse) // Infinite stream
testData.predict { case testDf =>
// Explode features in input streams
val testDf_exploded = ...
testDf_exploded
.join(model, testDf_exploded("feature") === model("feature"), "LEFT_OUTER")
.select($"rowid", ($"weight" * $"value").as("value"))
.groupby("rowid").sum("value")
.select($"rowid", sigmoid($"SUM(value)"))
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment