rajkrrsingh

Golang App

mkdir golang-web-app
cd golang-web-app
vim main.go
package main

import (
	"fmt"
	"log"

rajkrrsingh

for more information on golang hive driver, please refer https://github.com/beltran/gohive

Sample Table

create table test (id int);
insert into table test values (1),(2),(3),(4),(5);
select * from test;
1
2
3
4

rajkrrsingh

package listenablefuture;

import com.google.common.util.concurrent.FutureCallback;
import com.google.common.util.concurrent.Futures;
import com.google.common.util.concurrent.ListenableFuture;
import com.google.common.util.concurrent.ListeningExecutorService;
import com.google.common.util.concurrent.MoreExecutors;
import com.google.common.util.concurrent.SettableFuture;
import com.google.common.util.concurrent.ThreadFactoryBuilder;

rajkrrsingh

HiveServer2 Dockerfile

watch out the JAVA_TOOL_OPTIONS which are having remote debugging options.

FROM centos

# Basic hygene
RUN yum upgrade -y && \
    yum update -y && \

rajkrrsingh

Create and run hiveserver2 docker container

mkdir hive-3-image
cd hive-3-image/
wget https://raw.githubusercontent.com/alanfgates/sqltest/master/dbs/hive/v3_1/Dockerfile
docker build . -t hive3-image
// Run it
docker run -it --net=myNetwork -p 10000:10000 hive3-image

You will see following logs on stdout if everything goes well

rajkrrsingh

Prerequisite hive settings:

set hive.server2.logging.operation.level=execution;
set hive.metastore.transactional.event.listeners=org.apache.hive.hcatalog.listener.DbNotificationListener;
set hive.metastore.dml.events=true;

Setup database and tables

rajkrrsingh

import kafka.admin.AdminUtils;
import kafka.admin.RackAwareMode;
import kafka.utils.ZKStringSerializer$;
import kafka.utils.ZkUtils;
import org.I0Itec.zkclient.ZkClient;
import org.I0Itec.zkclient.ZkConnection;

import java.io.BufferedReader;

rajkrrsingh

Create RESOURCE PLAN testrp;

create RESOURCE PLAN testrp;
CREATE POOL testrp.default.c1 WITH ALLOC_FRACTION=0.3, QUERY_PARALLELISM=3, SCHEDULING_POLICY='fair';
CREATE POOL testrp.default.c2 WITH ALLOC_FRACTION=0.5, QUERY_PARALLELISM=1, SCHEDULING_POLICY='fair';
SELECT * FROM SYS.WM_POOLS;
ALTER RESOURCE PLAN testrp VALIDATE;
ALTER RESOURCE PLAN testrp ENABLE ACTIVATE;

rajkrrsingh

#### ENV: HDP-3.1
#### Data setup:

```
cat sample-data.json 
{"name": "Raj","address": {"a": "b","c": "d","e": "f"}}
{"name": "Raj1","address": {"a": "bb","c": "dd","e": "ff"}}
```

#### Create topic in Kafka and Ingest data into it.

rajkrrsingh

Docker Commands, Help & Tips

Show commands & management commands

$ docker

Docker version info