Merelda mereldawu

Engineer. Explorer. Entrepreneur

mereldawu / pandas_to_spark.py

Created January 27, 2021 13:57 — forked from zaloogarcia/pandas_to_spark.py

Script for converting Pandas DF to Spark's DF

	from pyspark.sql.types import *

	# Auxiliar functions
	# Pandas Types -> Sparks Types
	def equivalent_type(f):
	if f == 'datetime64[ns]': return DateType()
	elif f == 'int64': return LongType()
	elif f == 'int32': return IntegerType()
	elif f == 'float64': return FloatType()
	else: return StringType()

mereldawu / Dockerfile

Last active October 16, 2018 22:29

scraper dockerfile with jupyter pre- and post-save hooks

	FROM jupyter/scipy-notebook:latest
	MAINTAINER MERELDA WU

	USER root

	RUN apt-get update && apt-get install -y vim

	RUN pip install --upgrade pip notebook
	RUN conda install -c conda-forge -y jupyter_contrib_nbextensions
	RUN jupyter contrib nbextension install --user && \