vinodkc

./kafka-console-consumer.sh --bootstrap-server c320-node2.squadron-labs.com:6667  --topic __consumer_offsets --from-beginning --security-protocol SASL_PLAINTEXT --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --max-messages 10000 | cut -d, -f1 | cut -d[ -f2 | sort | uniq 

Note: 
1)Tune value of --max-messages to get more output,if any
2)Remove --security-protocol SASL_PLAINTEXT if not a secure cluster.

vinodkc

Spark HWC integration - HDP 3 Secure cluster

Prerequisites :

• Kerberized Cluster

• Enable hive interactive server in hive

• Get following details from hive for spark or try this HWC Quick Test Script

vinodkc

cd /usr/hdp/current/kafka-broker/bin/
[kafka@c220-node2 bin]$ ./kafka-topics.sh --create --zookeeper c220-node2.squadron-labs.com:2181 --replication-factor 2 --partitions 3 --topic source1
Created topic "source1".
[kafka@c220-node2 bin]$ ./kafka-topics.sh --create --zookeeper c220-node2.squadron-labs.com:2181 --replication-factor 2 --partitions 3 --topic dest1
Created topic "dest1".
[kafka@c220-node2 bin]$ ./kafka-console-producer.sh --broker-list c220-node2.squadron-labs.com:6667 --topic source1

[kafka@c220-node4 bin]$ ./kafka-console-consumer.sh --bootstrap-server c220-node4.squadron-labs.com:6667 --topic dest1

vinodkc

1) PACKAGE RUNTIME DEPENDENCIES IN LIB FOLDER INSIDE PACKAGED JAR

<build>
    <plugins>
        <plugin> 
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-dependency-plugin</artifactId>
            <version>2.8</version>
            <executions>
                <execution>

vinodkc

Project Dirs


 bin : Scripts to manage kafka and Zookeeper
 checkstyle : Checkstyle configurations
 clients : Kafka client code [Producer,Consumer, Admin tools classes]
 config : Default Config files
 connect : Kafka Connect module
 core : Kafka server code
 docs : Documentations

vinodkc

KafkaConnect

vinodkc

Steps to Enable Secure SSL Kafka brokers

1) Kerberize Cluster

1.1) Login to a Kafka client node and export KAFKA_CLIENT_KERBEROS_PARAMS

export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf" 

1.2) Lets test kafka producer and consumer without keytab

Use this jaas conf file

vinodkc

## Consumer Throughput: Single consumer thread, no compression
## Consumer Throughput: 3 consumer thread, no compression

bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1