Skip to content

Instantly share code, notes, and snippets.

@j-thepac
Created March 1, 2022 09:29
Show Gist options
  • Select an option

  • Save j-thepac/ebb467f91b9c080ae68e1b60e4891549 to your computer and use it in GitHub Desktop.

Select an option

Save j-thepac/ebb467f91b9c080ae68e1b60e4891549 to your computer and use it in GitHub Desktop.

Create xml file .../spark-3.0.0-bin-hadoop2.7/conf/hive-site.xml

<configuration> 
    <property> 
      <name>hive.metastore.warehouse.dir</name> 
      <value>/Users/deepakjayaprakash/Downloads/spark_database/</value> 
    </property> 
  </configuration>

Save

$spark-shell
(1 to 5).map(i =>"row"+i).toDF().coalesce(1).write.option("header","true").saveAsTable("test")
spark.catalog.listTables.show() 
:q

$spark-sql

notice path - "Warehouse path is 'file:/Users/deepakjayaprakash/Downloads/spark_database/spark-warehouse/'." in logs

spark-sql> select * from test;
spark-sql> quit;
$cd .../spark-3.0.0-bin-hadoop2.7/sbin/
$sh start-thriftserver.sh

The Server will start in silent mode

$cd .../spark-3.0.0-bin-hadoop2.7/bin/
$beeline
beeline> !connect jdbc:hive2://localhost:10000
<enter system username>
<password , just enter>
jdbc:hive2://localhost:10000> show databases; 
jdbc:hive2://localhost:10000> show tables; 

Exit

ctrl+c

Kill Server

$cd .../spark-3.0.0-bin-hadoop2.7/sbin/
$sh stop-thriftserver.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment