kumar-de

Setup Kerberos client on your machine

Install the Kerberos client packages:

apt-get install krb5-user

Paste into /etc/krb5.conf and adjust the following to match your server-setup

[libdefaults]
 default_realm = ABC.TEST

kumar-de

Dockerized IBM MQ with TLS enabled

Setting up the Server-keyStore and certificate

Run a container using latest IBM-MQ image

docker volume create qmdata
docker pull ibmcom/mq:latest
docker run -ti --entrypoint=/bin/bash --volume qmdata:/mnt/mqm ibmcom/mq:latest

kumar-de

Build and push a docker image into hub.docker.com

---

Docker login already, if you haven't

docker login

Inside the directory containing the Dockerfile

docker build -t achintyakumar/cloud_connector:1.0 .

Tag the image

docker tag 888dd8r8f8 achintyakumar/cloud_connector:1.0

kumar-de

Java - OpenJDK 8

yum -y update
yum install java-1.8.0-openjdk

Docker CE

sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

kumar-de

Configure Python SDK in IntelliJ IDEA

Follow everything on this link:

https://www.jetbrains.com/help/idea/creating-virtual-environment.html

Open Project Structure CTRL+ALT+SHIFT+S

• Project settings -> Facets -> expand Python click on child -> Python Interpreter
• Project settings -> Modules -> Expand module -> Python -> Dependencies -> select Python module SDK

kumar-de

Clean up all firewall rules

echo "Clearing all iptables rules"
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

kumar-de

Running Spark job in Batch vs Streaming mode against Kerberos

Variables

BROKER="broker1:9092"
SPARK_URL="yarn"
MODE="cluster"
APP_NAME="ApplicationName"
SECURITYPROTOCOL="SASL_PLAINTEXT"
COMMON_JAR="common-jar.jar"

kumar-de

# The following example is for a job that produces daily, parquets partitioned by 'day' column
DB="database"
BASE_DIR="/output/parquets"
TBL="${DB}.table"       # name of the impala table
PQT="${BASE_DIR}/day"   # name of the parent directory containing output-subdirectories (in the format 'day=ddMMyyy')

# create/mount impala table
impala-shell -q "drop table if exists $TBL;
create external table $TBL (
account_id string,

kumar-de

Kafka console consumer with Kerberos

1. Create a jaas.conf file with the following contents:

KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="keytabFile.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"

kumar-de

Create a fat jar using Maven

<?xml version="1.0" encoding="UTF-8"?>
<build>
	<plugins>
        <!-- Set a compiler level -->
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>2.3.2</version>