Skip to content

Instantly share code, notes, and snippets.

@thanoojgithub
Created November 20, 2020 17:07
Show Gist options
  • Save thanoojgithub/528cbcc042998068add7261dfec93124 to your computer and use it in GitHub Desktop.
Save thanoojgithub/528cbcc042998068add7261dfec93124 to your computer and use it in GitHub Desktop.
How to run HiveServer2 (Hive 2.3.3) on ubuntu 20.04
Pre-requisites:
1. Expecting Hadoop and Hive is well configured
2. hive CLI is working as expected
then, we can try running HiveServer2
hduser@thanoojubuntu-Inspiron-3521:~/softwares/apache-hive-2.3.3-bin/conf$ hive --service hiveserver2 --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.root.logger=INFO,console
2020-11-20 21:25:32: Starting HiveServer2
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hduser/softwares/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2020-11-20T21:25:38,276 INFO [main] session.SessionState: Created HDFS directory: /tmp/hduser/hduser/85cedd8b-b126-4f71-8d0c-2bcfb83fd759
2020-11-20T21:25:38,307 INFO [main] session.SessionState: Created local directory: /tmp/hduser/85cedd8b-b126-4f71-8d0c-2bcfb83fd759
2020-11-20T21:25:38,315 INFO [main] session.SessionState: Created HDFS directory: /tmp/hduser/hduser/85cedd8b-b126-4f71-8d0c-2bcfb83fd759/_tmp_space.db
2020-11-20T21:25:38,352 INFO [main] sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=85cedd8b-b126-4f71-8d0c-2bcfb83fd759, clientType=HIVESERVER2]
2020-11-20T21:25:38,357 WARN [main] session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
2020-11-20T21:25:39,136 INFO [main] metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
2020-11-20T21:25:39,181 INFO [main] metastore.ObjectStore: ObjectStore, initialize called
2020-11-20T21:25:41,323 INFO [main] metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
2020-11-20T21:25:43,541 INFO [main] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL
2020-11-20T21:25:43,547 INFO [main] metastore.ObjectStore: Initialized ObjectStore
2020-11-20T21:25:43,708 INFO [main] metastore.HiveMetaStore: Added admin role in metastore
2020-11-20T21:25:43,723 INFO [main] metastore.HiveMetaStore: Added public role in metastore
2020-11-20T21:25:43,756 INFO [main] metastore.HiveMetaStore: No user is added in admin role, since config is empty
2020-11-20T21:25:43,884 INFO [main] metastore.HiveMetaStore: 0: get_all_functions
2020-11-20T21:25:43,884 INFO [main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_all_functions
2020-11-20T21:25:43,923 INFO [main] service.CompositeService: Operation log root directory is created: /home/hduser/softwares/apache-hive-2.3.3-bin/conf/${system:java.io.tmpdir}/${system:user.name}/operation_logs
2020-11-20T21:25:43,930 INFO [main] service.CompositeService: HiveServer2: Background operation thread pool size: 100
2020-11-20T21:25:43,930 INFO [main] service.CompositeService: HiveServer2: Background operation thread wait queue size: 100
2020-11-20T21:25:43,931 INFO [main] service.CompositeService: HiveServer2: Background operation thread keepalive time: 10 seconds
2020-11-20T21:25:43,962 INFO [main] metastore.HiveMetaStore: 0: get_all_databases
2020-11-20T21:25:43,963 INFO [main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_all_databases
2020-11-20T21:25:43,976 INFO [main] metastore.HiveMetaStore: 0: get_all_tables: db=default
2020-11-20T21:25:43,976 INFO [main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_all_tables: db=default
2020-11-20T21:25:44,002 INFO [main] metastore.HiveMetaStore: 0: get_multi_table : db=default tbls=
2020-11-20T21:25:44,003 INFO [main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_multi_table : db=default tbls=
2020-11-20T21:25:44,220 INFO [main] server.Server: jetty-7.6.0.v20120127
2020-11-20T21:25:44,258 INFO [Thread-9] thrift.ThriftCLIService: Starting ThriftBinaryCLIService on port 10000 with 5...500 worker threads
2020-11-20T21:25:44,301 INFO [main] webapp.WebInfConfiguration: Extract jar:file:/home/hduser/softwares/apache-hive-2.3.3-bin/lib/hive-service-2.3.3.jar!/hive-webapps/hiveserver2/ to /tmp/jetty-0.0.0.0-10002-hiveserver2-_-any-/webapp
2020-11-20T21:25:44,534 INFO [main] handler.ContextHandler: started o.e.j.w.WebAppContext{/,file:/tmp/jetty-0.0.0.0-10002-hiveserver2-_-any-/webapp/},jar:file:/home/hduser/softwares/apache-hive-2.3.3-bin/lib/hive-service-2.3.3.jar!/hive-webapps/hiveserver2
2020-11-20T21:25:44,602 INFO [main] handler.ContextHandler: started o.e.j.s.ServletContextHandler{/static,jar:file:/home/hduser/softwares/apache-hive-2.3.3-bin/lib/hive-service-2.3.3.jar!/hive-webapps/static}
2020-11-20T21:25:44,625 INFO [main] server.AbstractConnector: Started [email protected]:10002
2020-11-20T21:25:44,630 INFO [main] http.HttpServer: Started HttpServer[hiveserver2] on port 10002
cd $HIVE_HOME/conf
gedit hive-site.xml
--------------------
<property>
<name>hive.server2.enable.doAs</name>
<value>false</value>
<description>
Setting this property to true will have HiveServer2 execute
Hive operations as the user making the calls to it.
</description>
</property>
cd $HADOOP_HOME/etc/hadoop/
gedit core-site.xml
---------------------------
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>*</value>
</property>
-------------------------------------------------------------------------------------------
hduser@thanoojubuntu-Inspiron-3521:~/softwares/apache-hive-2.3.3-bin/conf$ diff hive-default.xml.template hive-site.xml
39c39
< <value>/tmp/hive</value>
---
> <value>/tmp/${user.name}</value>
75c75
< <value>${system:java.io.tmpdir}/${system:user.name}</value>
---
> <value>/tmp/${user.name}</value>
80c80
< <value>${system:java.io.tmpdir}/${hive.session.id}_resources</value>
---
> <value>/tmp/${user.name}_resources</value>
530c530
< <value>mine</value>
---
> <value>Password!23</value>
545c545
< <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
---
> <value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
1020c1020
< <value>org.apache.derby.jdbc.EmbeddedDriver</value>
---
> <value>com.mysql.cj.jdbc.Driver</value>
1045c1045
< <value>APP</value>
---
> <value>hiveusr</value>
3976c3976
< <value>${system:java.io.tmpdir}/${system:user.name}/operation_logs</value>
---
> <value>/tmp/${user.name}/operation_logs</value>
4359c4359
< <value>true</value>
---
> <value>false</value>
hduser@thanoojubuntu-Inspiron-3521:~/softwares/apache-hive-2.3.3-bin/conf$
@thanoojgithub
Copy link
Author

thanoojgithub commented Nov 20, 2020

hduser - user running HiveServer2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment