tecmaverick

GIT ARCH
1. Git takes a snapshot of the entire file when its changed, instead of delta changes. Every time you commit, or save the state of your project, Git basically takes a picture of what all your files look like at that moment and stores a reference to that snapshot. To be efficient, if files have not changed, Git doesn’t store the file again, just a link to the previous identical file it has already stored. Git thinks about its data more like a stream of snapshots.

2. In Git is checksummed before it is stored and is then referred to by that checksum. Using SHA-1 hash. Git stores everything in its database not by file name but by the hash value of its contents.

3. When you do actions in Git, nearly all of them only add data to the Git database. The Three States of file changes
3.1 Modified -  Changed the file but have not committed it to your database yet.
3.2 Staged - have marked a modified file in its current version to go into your next commit snapshot.
3.3 Committed - Data is safely stored in your 

tecmaverick

# Retrieves the list of unique file extensions, including those with multiple extensions like gz.txt
aws s3 ls s3://the_bucket_name/prefix_name --recursive | rev | cut -d "/" -f 1 | rev | cut -d "." -f 2- | sort | uniq

# Retrieves the count of file extensions in a given s3 prefix
aws s3 ls s3://the_bucket_name/prefix_name --recursive | rev | cut -d "/" -f 1 | rev | cut -d "." -f 2- | uniq -c

tecmaverick

#List all network connection
lsof -i

#List all open IPv4 connections
lsof -i 4

#List all open IPv6 connections
lsof -i 6

#List process running on specific port

tecmaverick

# Change the process priority to the lowest -20 lowest and 20 is the highest. 0 is the default priority for all process
renice 20 -p $(pgrep "ProcessName")


# List all open files (Disk and Network) by a specific process
lsof | awk '{ if ($1=="ProcessName") { print}}'

tecmaverick

# abc.csv contents
# Alpha, USA, 12
# Beta, USA, 13

# abc.csv contents after adding header
# Name, Country, Age
# Alpha, USA, 12
# Beta, USA, 13

tecmaverick

const detectIncognito = function() {
    return new Promise(function(resolve, reject) {
        var browserName = "Unknown";
        function __callback(isPrivate) {
            resolve({
                isPrivate: isPrivate,
                browserName: browserName,
            });
        }
        function identifyChromium() {

tecmaverick

#To execute script
# ./script.sh MyHeader MyCellValue input.csv output.csv

HEADER_NAME=$1
FIELD_VAL=$2
INPUT_FILENAME=$3
OUPUT_FILENAME=$4
awk -v HEADER_NAME=$1 -v FIELD_VAL=$2 -F"," '{if(NR==1){$0=HEADER_NAME","$0;};if (NR>1) { OFS = ",";{$0=FIELD_VAL","$0; }};if(NR>=0){FS=OFS=",";{f1=$1;f2=$2;f3=$3;f4=$4;f5=$5;$1=f2;$2=f1;$3=f3;$4=f4;$5=f5;print}}}' $INPUT_FILENAME > $OUPUT_FILENAME

tecmaverick

//Set local checkpoint directory. This is unreliable incase of driver restarts
sc.setCheckpointDir("file:///tmp/sparkcheckpoints")

//View the checkpoint dir
sc.getCheckpointDir.get


val rdd = sc.parallelize(Seq.range(0,100))
val filteredRdd = rdd.filter(x=> x>50).map(x=> x * 2)

tecmaverick

//Input Data
// studentid,coursename_with_attendance
// 01,CHEM:12|PHY:33|MATH:22
// 02,CHEM:34|PHY:3
// 03,MATH:12|COMP:45|CHEM:12
// 04,MATH:67|PHY:76
// 05,HIST:88|MARKT:33|BIOL:55
// 06,BIOL:88|PHY:77
// 07,BOTONY:34|ZOOL:77
// 08,BOTONY:34|COMP:99

tecmaverick

// Input
// studentid,coursename
// 01,CHEM|PHY|MATH
// 02,CHEM|PHY
// 03,MATH|COMP|CHEM
// 04,MATH|PHY
// 05,HIST|MARKT|BIOL
// 06,BIOL|PHY
// 07,BOTONY|ZOOL
// 08,BOTONY|COMP