Skip to content

Instantly share code, notes, and snippets.

@isogram
Created June 2, 2020 07:33
Show Gist options
  • Save isogram/92fd9ee13fc322c6911d9b10ef401cba to your computer and use it in GitHub Desktop.
Save isogram/92fd9ee13fc322c6911d9b10ef401cba to your computer and use it in GitHub Desktop.
Pyspark Create Empty Dataframe
from pyspark.sql.types import StringType, FloatType, StructField, StructType
from pyspark.sql import SparkSession, SQLContext, Row
import pyspark
# spark initialization
spark_context = pyspark.SparkContext.getOrCreate()
spark_session = SparkSession(spark_context) \
.builder \
.enableHiveSupport() \
.getOrCreate()
sqlContext = SQLContext(spark_context)
field = [
StructField("FIELDNAME_1",StringType(), True),
StructField("FIELDNAME_2", FloatType(), True),
StructField("FIELDNAME_3", StringType(), True)
]
schema = StructType(field)
df = sqlContext.createDataFrame(spark_context.emptyRDD(), schema)
df.printSchema()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment