vinodkc

Hi there 👋

vinodkc

Spark Listener Demo

This demonstrates Spark Job, Stage and Tasks Listeners

1) Start spark-shell

Welcome to
      ____              __
 / __/__ ___ _____/ /__

vinodkc

Spark Structured Streaming HWC integration

1) Setup Kafka topic

cd /usr/hdp/current/kafka-broker/bin/

./kafka-topics.sh --create --zookeeper c420-node2.coelab.cloudera.com:2181 --replication-factor 2 --partitions 3 --topic ss_input

vinodkc

Storm-Hive-Integration

Download storm-hive-examples < version >.jar from maven central or build it from https://github.com/apache/storm/tree/v1.2.1/examples/storm-hive-examples

Note: Ensure to use matching jar version of cluster version

We will try to save records with following fields into Hive table

{"id","name","phone","street","city","state"} 

vinodkc

Kafka Hive integration

Prerequisites

1. Hive3

2. Download Hive Kafka Storage Handler jar from https://mvnrepository.com/artifact/org.apache.hive/kafka-handler and add in hive lib location

3. Restart hive

vinodkc

1) Mark Down Cheat sheet  :- https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

vinodkc

Login to LLAP host node

A) Test with Spark-shell

step 1:

cd /tmp
wget https://raw.githubusercontent.com/dbompart/hive_warehouse_connector/master/hwc_info_collect.sh
chmod +x  hwc_info_collect.sh

vinodkc

Spark on Docker - HDP3 YARN

1. Kerberize the cluster

2. Enable CGroup from yarn and restart

To enable cgroups on an Ambari cluster, select YARN > Configs on the Ambari dashboard, then click CPU Isolation under CPU. Click Save, then restart all cluster components that require a restart

I got mount failure error: /sys/fs/cgroup/cpu/yarn Solution , run below command on all node manager hosts:

vinodkc

A) Spark structured streaming Kafka integration - SASL_PLAINTEXT 

1) Prerequisites

[consumer-user@c220-node1 sslss]$ ll
-rw------- 1 consumer-user root           144 Apr 21 08:56 consumer-user.keytab
-rw-rw-r-- 1 consumer-user consumer-user  229 Apr 21 09:40 kafka_client_jaas.conf

[consumer-user@c220-node1 sslss]$ cat kafka_client_jaas.conf
KafkaClient {

vinodkc

Kafka Topics

List existing topics

bin/kafka-topics.sh --zookeeper localhost:2181 --list

Describe a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic mytopic

Purge a topic

bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic mytopic --config retention.ms=1000

... wait a minute ...