Skip to content

Instantly share code, notes, and snippets.

@abronte

abronte/spark_application.py

Created November 7, 2019 00:32
Show Gist options
  • Save abronte/f4a70db294a8a173a47321859479171b to your computer and use it in GitHub Desktop.
Save abronte/f4a70db294a8a173a47321859479171b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_application.py
from pyspark_gateway import PysparkGateway
pg = PysparkGateway()
from pyspark import SparkContext, SparkConf
conf = conf.set('spark.io.encryption.enabled', 'true')
sc = SparkContext(gateway=pg.gateway, conf=conf)
spark = SparkSession.builder.getOrCreate()
df = spark.read.parquet('hdfs://data/my_table.parquet')
df.show()
cnt = df.filter('foo = "bar"').count()
print(cnt)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment