Skip to content

Instantly share code, notes, and snippets.

@vsouza

vsouza/spark_context.py

Created October 5, 2016 02:34
Show Gist options
  • Save vsouza/77b2aaea67904177ae167fa5edb482c1 to your computer and use it in GitHub Desktop.
Save vsouza/77b2aaea67904177ae167fa5edb482c1 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spark_context.py
spark_context = SparkContext(appName=kinesis_app_name)
spark_streaming_context = StreamingContext(spark_context, spark_batch_interval)
sql_context = SQLContext(spark_context)
kinesis_stream = KinesisUtils.createStream(
spark_streaming_context, kinesis_app_name, kinesis_stream, kinesis_endpoint,
aws_region, kinesis_initial_position, kinesis_checkpoint_interval)
py_rdd = kinesis_stream.map(lambda x: json.loads(x))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment