Mohamed Nabil Hafez nabilm
royrusso
/ gist:51bc89427e1575d4d777
Last active
June 12, 2016 23:49
Load datasource in to dataframe, using Spark DataSource API
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| os.environ['SPARK_CLASSPATH'] = "/path/to/driver/postgresql-9.3-1103.jdbc41.jar" | |
| from pyspark import SparkContext | |
| from pyspark.sql import SQLContext, Row | |
| sc = SparkContext("local[*]", '<JOBNAME>') | |
| sqlctx = SQLContext(sc) | |
| df = sqlctx.load( | |
| source="jdbc", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # | |
| # Installation script for scrapyd on Debian/Ubuntu | |
| # http://scrapyd.readthedocs.org/ | |
| # | |
| # - Latest scrapyd from Scrapy package repository | |
| # - Password protection with http basic auth | |
| # - HTTPS with self-signed certificate | |
| # - Works on Amazon EC2 | |
| # | |
| FQDN=scrapy.example.com |