Skip to content

Instantly share code, notes, and snippets.

@josep2
Created October 24, 2016 18:56
Show Gist options
  • Save josep2/07c181ea6ee150e122856802675546f8 to your computer and use it in GitHub Desktop.
Save josep2/07c181ea6ee150e122856802675546f8 to your computer and use it in GitHub Desktop.
import java.sql._
dataframe.coalesce("NUMBER OF WORKERS").mapPartitions((d) => Iterator(d)).foreach { batch =>
val dbc: Connection = DriverManager.getConnection("JDBCURL")
val st: PreparedStatement = dbc.prepareStatement("YOUR PREPARED STATEMENT")
batch.grouped("# Of Rows you want per batch").foreach { session =>
session.foreach { x =>
st.setDouble(1, x.getDouble(1))
st.addBatch()
}
st.executeBatch()
}
dbc.close()
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment