Skip to content

Instantly share code, notes, and snippets.

@welly87

welly87/pyspark-kudu.sh

Created October 1, 2020 06:10
Show Gist options
  • Save welly87/08bf38934df8c2e385f6b4fa8f751b0f to your computer and use it in GitHub Desktop.
Save welly87/08bf38934df8c2e385f6b4fa8f751b0f to your computer and use it in GitHub Desktop.
Download ZIP
@welly87
Copy link
Author

welly87 commented Oct 1, 2020 •
edited
Loading 

table = "transactions_1k"
kudu_master = "178.128.112.105:7051,178.128.112.105:7151,178.128.112.105:7251"
sfmta_kudu = spark.read.option("kudu.master", kudu_master).option("kudu.table", table).option("kudu.scanLocality", "leader_only").format("kudu").load()

sfmta_kudu.createOrReplaceTempView(table)

sdf = spark.sql("SELECT count(*) FROM " + table)

@welly87
Copy link
Author

welly87 commented Oct 1, 2020

http://178.128.112.105:8050/tablets

@welly87
Copy link
Author

welly87 commented Oct 1, 2020 

pdf = sdf.select("*").toPandas()
pdf.head()

@welly87
Copy link
Author

welly87 commented Oct 1, 2020

https://www.cioreview.com/hover_img/matiltonbig1.jpg

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment