j-thepac

# ----------------JSON VS YAML/YML-----------------
# 2 Spaces and No TABS !!!

# Comment

# Array
[1,2]
- 1
- 2

j-thepac

# docker build -t somename:v1 .
# docker run -it nicearray:v1
# kubectl create deployment somename --image=somename:v1

FROM broadinstitute/scala-baseimage
COPY $PWD /home/apps/
WORKDIR /home/apps
CMD scala File.scala

j-thepac

HttpClient in spark-shell REPL

• open link https://mvnrepository.com/artifact/com.softwaremill.sttp.client3/core
• open compatible scalalink
• click on Maven tab
• copy groupId:artifactId:version ie., com.softwaremill.sttp.client3:core_2.12:3.7.4
• create packagename as below
• Open terminal and run below commands :

$spark-shell --packages com.softwaremill.sttp.client3:core_2.12:3.7.4

j-thepac

#Image to only DOwnload kafka 
# docker-compse file to start zookeeper and broker is not ready !!!

FROM ibmjava
RUN apt-get update && \
    apt-get install -y curl

RUN mkdir /tmp/kafka
RUN curl https://downloads.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz  -o /tmp/kafka/kafka_2.12-3.2.0.tgz
RUN cd /tmp/kafka && \

j-thepac

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

j-thepac

Kafka Stream / Kafka based stream processors

Kafka Streams will consume topics , transformed and sank in some other topics ie., input and output of Kafka Stream is Topics

Adv:

• Lightweight – no external dependencies other than Kafka
• Scale out horizontally
• Use Scala
• code is very readable.
• perform both stateless transformations / stateful aggregations and joins
• Supports common stream processing such as dwindowing

j-thepac

Kafka

kafkaSample

Docker YML

## docker-compose.yml (start kafka Broker and Zookeeper)

    #docker-compose up -d #detach mode
	version: '3'
	services:
 zookeeper:

j-thepac

for i in range(1,10):
  print("Progress: "+i*"#" , end="\r")
  time.sleep(1)

j-thepac

from http.client import IncompleteRead
import requests
path="/Documents/test.zip"

with requests.get(url, headers=headers, stream=True) as r:
    with open(path, 'wb') as f:
        while(True):
            try:
                f.write(r.content)
                break

j-thepac

import scala.concurrent.{Await, Future}
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global
import java.util.Calendar

object FutureCleint extends App{
  
  val start =Calendar.getInstance().getTime.getSeconds
  val fx :String => String = (name:String)=>{