Script for converting Pandas DF to Spark's DF

pandas_to_spark.py

from pyspark.sql.types import *

# Auxiliar functions
# Pandas Types -> Sparks Types
def equivalent_type(f):
  if f == 'datetime64[ns]': return DateType()
  elif f == 'int64': return LongType()
  elif f == 'int32': return IntegerType()
  elif f == 'float64': return FloatType()
  else: return StringType()

def define_structure(string, format_type):
  try: typo = equivalent_type(format_type)
  except: typo = StringType()
  return StructField(string, typo)

#Given pandas dataframe, it will return a spark's dataframe
def pandas_to_spark(df_pandas):
  columns = list(df_pandas.columns)
  types = list(df_pandas.dtypes)
  struct_list = []
  for column, typo in zip(columns, types): 
    struct_list.append(define_structure(column, typo))
  p_schema = StructType(struct_list)
  return sqlContext.createDataFrame(df_pandas, p_schema)
  

from pyspark import SparkContext, SQLContext
sqlContext = SQLContext(SparkContext.getOrCreate())

Try adding this, then it would work...

Thanks for this one, it's been very useful on my end!

I am not that experienced in pyspark, but is there a reason why equivalent_type returns a new instance of XxxxType() at each call, rather than using a dictionary like below?

equivalent_type_dict = {
    'datetime64[ns]': DateTime(),
    'int64': LongType(),
    'int32': IntegerType(),
    'float64': DoubleType(),
    'float32': FloatType(),
}

def define_structure(string, format_type):
  try: typo = equivalent_type_dict[format_type]
  except: typo = StringType()
  return StructField(string, typo)

...

I can suggest updating this:

equivalent_type_dict = {
    ...
    'float64': DoubleType(),
    'float32': FloatType(),
    ...
}

Because, using FloatType will remove some decimals, like this:

Pandas: 9544.145833333334 | Float64
SparkDF: 9544.1455 | float
SparkDF: 9544.145833333334 | double

I can suggest updating this:

equivalent_type_dict = {
    ...
    'float64': DoubleType(),
    'float32': FloatType(),
    ...
}

Because, using FloatType will remove some decimals, like this:

Pandas: 9544.145833333334 | Float64
SparkDF: 9544.1455 | float
SparkDF: 9544.145833333334 | double

Thanks for the suggestion, I updated this part in my code.