Skip to content

Instantly share code, notes, and snippets.

@zgulde
Last active August 22, 2022 17:47
Show Gist options
  • Save zgulde/bb6c92357fc29d5a2903bc431a1e87ce to your computer and use it in GitHub Desktop.
Save zgulde/bb6c92357fc29d5a2903bc431a1e87ce to your computer and use it in GitHub Desktop.
spark jdbc mysql driver connection setup
import pyspark
import env
MYSQL_CONNECTOR_VERSION = '5.1.38'
DATABASE = 'a_database'
TABLE = 'some_db_table'
spark = (pyspark.sql.SparkSession.builder
.config(f'spark.jars.packages', 'mysql:mysql-connector-java:{MYSQL_CONNECTOR_VERSION}')
.getOrCreate())
df = (spark.read
.option('driver', 'com.mysql.jdbc.Driver')
.option('url', f'jdbc:mysql://{env.host}/{DATABASE}')
.option('user', env.user)
.option('password', env.password)
.option('dbtable', TABLE)
.format('jdbc')
.load())
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment