Deepak j-thepac

ETL

BigData:

OLTP (OnLine Transaction Processing) : is used for managing current day to day data information.
- ACID
Atomicity(entire transaction happens at once or nothing happens)

Issues in Spark :

Adv of Delta Lake

open link https://mvnrepository.com/artifact/com.softwaremill.sttp.client3/core
open compatible scalalink
click on Maven tab
copy groupId:artifactId:version ie., com.softwaremill.sttp.client3:core_2.12:3.7.4
create packagename as below
Open terminal and run below commands :

$spark-shell --packages com.softwaremill.sttp.client3:core_2.12:3.7.4

Kafka Streams will consume topics , transformed and sank in some other topics ie., input and output of Kafka Stream is Topics

	from pyspark.sql.window import Window

	"""
	Aggregate: min, max, avg, count, and sum.
	Ranking: rank, dense_rank, percent_rank, row_num, and ntile
	Analytical: cume_dist, lag, and lead
	Custom boundary: rangeBetween and rowsBetween
	"""

	# docker build -t somename:v1 .
	# docker run -it nicearray:v1
	# kubectl create deployment somename --image=somename:v1

	FROM broadinstitute/scala-baseimage
	COPY $PWD /home/apps/
	WORKDIR /home/apps
	CMD scala File.scala

	#Image to only DOwnload kafka
	# docker-compse file to start zookeeper and broker is not ready !!!

	FROM ibmjava
	RUN apt-get update && \
	apt-get install -y curl

	RUN mkdir /tmp/kafka
	RUN curl https://downloads.apache.org/kafka/3.2.0/kafka_2.12-3.2.0.tgz -o /tmp/kafka/kafka_2.12-3.2.0.tgz
	RUN cd /tmp/kafka && \

	version: '3'
	services:
	zookeeper:
	image: confluentinc/cp-zookeeper:latest
	environment:
	ZOOKEEPER_CLIENT_PORT: 2181
	ZOOKEEPER_TICK_TIME: 2000
	ports:
	- 22181:2181