Skip to content

Instantly share code, notes, and snippets.

@hakanilter

hakanilter/json_hive_definition.py

Last active November 28, 2018 16:41
Show Gist options
  • Save hakanilter/a5781b4fb00469dcfe62f270d412b781 to your computer and use it in GitHub Desktop.
Save hakanilter/a5781b4fb00469dcfe62f270d412b781 to your computer and use it in GitHub Desktop.
Download ZIP
Fastest way to get Hive definition for a given Json file
Raw
json_hive_definition.py
def json_hive_def(path):
spark.read.json(path).createOrReplaceTempView("temp_view")
spark.sql("CREATE TABLE temp_table AS SELECT * FROM temp_view LIMIT 0")
script = spark.sql("SHOW CREATE TABLE temp_table").take(1)[0].createtab_stmt.replace('\n', '')
spark.sql("DROP TABLE temp_table")
return script
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment