john huang jhuangtw
jhuangtw
/ databricks_connection_to_gcloud_sql.py
Last active
April 5, 2022 09:38
connecting from Databricks spark to Google cloud-sql via proxy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Overview: | |
| # Install cloud_sql_proxy binary with appropriate credentials on all driver and worker nodes. | |
| # References: | |
| # https://cloud.google.com/sql/docs/postgres/connect-external-app#proxy | |
| # https://cloud.google.com/sql/docs/postgres/sql-proxy#authentication-options | |
| # Step 1. | |
| # Follow https://cloud.google.com/sql/docs/postgres/connect-external-app#proxy to set up: | |
| # - enable Cloud SQL API |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| pipenv install ipykernel jupyter | |
| pipenv shell | |
| python -m ipykernel install --user --name=$(basename $(pwd)) | |
| jupyter notebook | |
| # then in UI, switch kernel to the one matching your pipenv shell name |