Skip to content

Instantly share code, notes, and snippets.

@Yengas

Yengas/schema-optimization-explain.scala

Created November 15, 2018 16:32
Show Gist options
  • Save Yengas/a7f4c1fc8a53254768b17e4e27275352 to your computer and use it in GitHub Desktop.
Save Yengas/a7f4c1fc8a53254768b17e4e27275352 to your computer and use it in GitHub Desktop.
Download ZIP
Spark at Getir - 01
Raw
schema-optimization-explain.scala
// Previous approach to conversion
// definition
def rowToDocument(row: Row): BsonDocument
// usage
rowToDocument(row) // for each line
// The (micro) optimized approach
// definition
def rowToDocumentMapper(schema: StructType): (Row) => BsonDocument
// usage
val mapper = rowToDocumentMapper(dataSet.schema) // one time, before starting the row writing
mapper(row) // for each line
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment