anjijava16

Incoming/Sources/Inputs:  S3 Files (CSV,Parquet) and RDBMS (Oracle,MySQL ) Data

Languages: Python,Shell Script

Cluster : AWS EMR (It is Like Hortnworks)

Linux : Linux server (It is EC2)

Processing :
     1. Hive SQL

anjijava16

# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"


# Hadoop
export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME

anjijava16

# Virtual Environmnet
1. There are other great third-party tools for creating virtual environments, such as 

1. virtualenv
2. conda
3. Pipenv
4. Poetry


# Step1

anjijava16

Jump to beginning of a line – Command+Left Arrow
Jump to end of a line – Command+Right Arrow
Jump to beginning of current word – Option+Right Arrow
Jump to end of current word – Option+Right Arrow
To backspace on a Mac, press the fn and Delete keys, as shown below
command +Space ==> QUick search
Control + LeftArrow ==> Left Open one
Control+ RightArrow ==> Right Opne one 
command + c==> COPY
command + p ==> Past

anjijava16

vi .zprofile

# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"


# Hadoop
export HADOOP_HOME=/Users/welcome/Desktop/hadoop/hadoop-3.2.1/
export HADOOP_INSTALL=$HADOOP_HOME

anjijava16

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?><!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

anjijava16

# Hadoop setup here
1. https://blog.contactsunny.com/data-science/installing-hadoop-on-the-new-m1-pro-and-m1-max-macbook-pro

# Hive Setup Here
1. https://dbmstutorials.com/hive/hive-setup-on-mac.html

anjijava16

welcome@Anjaiahs-MacBook-Pro ~ % more .zprofile

# Setting PATH for Python 3.8
# The original version is saved in .zprofile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:${PATH}"
export PATH
welcome@Anjaiahs-MacBook-Pro ~ % 

anjijava16

import csv
from datetime import datetime

INPUT_FILE_PATH = 'C:/Tech_Learn_welcome/Python_Utils/FastAPI/sai_workspace/fast_api_welcome/files/input.csv'
OUTPUT_FILE_PATH = 'C:/data/python_output/input.csv'
HEADER_SKIP = True
OUTPUT_HEADER = 'id,first,last,ssn,address,firstname_lastname,process_date'


def read_file():

anjijava16

mkdir delab_f

cd delab_f

python3 -m venv delab-venv

source delab-venv/bin/activate 

pip install jupyterlab