Skip to content

Instantly share code, notes, and snippets.

@zhu3pang
Last active June 19, 2021 08:12
Show Gist options
  • Save zhu3pang/10ccb611df0a3061de86903ab9a1b0d4 to your computer and use it in GitHub Desktop.
Save zhu3pang/10ccb611df0a3061de86903ab9a1b0d4 to your computer and use it in GitHub Desktop.
create empty dataframe in scala and java
import org.apache.spark.sql.types.{
StructType, StructField, StringType, IntegerType}
import org.apache.spark.sql.Row
//in scala
val schema = StructType(
StructField("k", StringType, true) ::
StructField("v", IntegerType, false) :: Nil)
// Spark < 2.0
// sqlContext.createDataFrame(sc.emptyRDD[Row], schema)
spark.createDataFrame(sc.emptyRDD[Row], schema)
//in java
sparkSession.createDataFrame(
sparkSession.sparkContext().emptyRDD(scala.reflect.ClassTag$.MODULE$.apply(Row.class)),
new StructType()
.add(new StructField("uid", DataTypes.LongType, true, Metadata.empty()))
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment