Anjaiah Methuku anjijava16

Spark

	1. Hive Joins
	2. Functions SQL,Window Functions should write one example in notepad
	3. Top 3 Records, or Top n Records
	4. Best File formats hive : Ans Should be Parquet Why
	5. map side vs reduce side join
	6. Spark Connectors, Spark with Hive connectors
	6. Reduce By vs group by (Good ANs : group by having more shuffle but in reduce by less shuffle
	7. cache vs perist
	8. repartition vs colasec
	9. RDD vs Dataframe

	1. https://www.youtube.com/watch?v=AK7_m-aThfw
	1. https://www.youtube.com/watch?v=Oo2FoYgRBvE&list=PLaIYQ9kvDKjroSixUUJJEdlvh-jr8X3ER
	2. https://www.youtube.com/watch?v=-WEpWH1NHGU
	3. https://www.mygreatlearning.com/blog/sql-interview-questions/


	# HIve
	1. https://www.youtube.com/watch?v=8tFyr02GYzc&list=PLIeFQEzpZM9gDh5UWCPeI11M-Ykm4scq1

	Incoming/Sources/Inputs: S3 Files (CSV,Parquet) and RDBMS (Oracle,MySQL ) Data

	Languages: Python,Shell Script

	Cluster : AWS EMR (It is Like Hortnworks)

	Linux : Linux server (It is EC2)

	Processing :
	1. Hive SQL

	# Setting PATH for Python 3.8
	# The original version is saved in .zprofile.pysave
	PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"


	# Hadoop
	export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
	export HADOOP_INSTALL=$HADOOP_HOME
	export HADOOP_MAPRED_HOME=$HADOOP_HOME
	export HADOOP_COMMON_HOME=$HADOOP_HOME

	# Virtual Environmnet
	1. There are other great third-party tools for creating virtual environments, such as

	1. virtualenv
	2. conda
	3. Pipenv
	4. Poetry


	# Step1

	Jump to beginning of a line – Command+Left Arrow
	Jump to end of a line – Command+Right Arrow
	Jump to beginning of current word – Option+Right Arrow
	Jump to end of current word – Option+Right Arrow
	To backspace on a Mac, press the fn and Delete keys, as shown below
	command +Space ==> QUick search
	Control + LeftArrow ==> Left Open one
	Control+ RightArrow ==> Right Opne one
	command + c==> COPY
	command + p ==> Past

	vi .zprofile

	# Setting PATH for Python 3.8
	# The original version is saved in .zprofile.pysave
	PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"


	# Hadoop
	export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
	export HADOOP_INSTALL=$HADOOP_HOME

	<?xml version="1.0" encoding="UTF-8" standalone="no"?>
	<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
	Licensed to the Apache Software Foundation (ASF) under one or more
	contributor license agreements. See the NOTICE file distributed with
	this work for additional information regarding copyright ownership.
	The ASF licenses this file to You under the Apache License, Version 2.0
	(the "License"); you may not use this file except in compliance with
	the License. You may obtain a copy of the License at

	http://www.apache.org/licenses/LICENSE-2.0

	# Hadoop setup here
	1. https://blog.contactsunny.com/data-science/installing-hadoop-on-the-new-m1-pro-and-m1-max-macbook-pro

	# Hive Setup Here
	1. https://dbmstutorials.com/hive/hive-setup-on-mac.html