thangarajan8

multi_date_format
07/01/2020 13:01
03/01/2020
02/01/2020 13:01
01/01/2020 13:01
05/01/2020 13:01
04-Jan-20
06/01/2020 13:01

thangarajan8

select date_parse('2021-12-31 00:00:00','%Y-%m-%d %H:%i:%s')

thangarajan8

import pandas as pd
import time
import numpy as np
#http://eforexcel.com/wp/wp-content/uploads/2020/09/5m-Sales-Records.zip
df = pd.read_csv("5m Sales Records.csv")

def filter1(df):
    start_time = time.time()
    
    for i in df.Country.unique():

thangarajan8

#https://www.hackerrank.com/challenges/missing-numbers/problem?isFullScreen=false 
a = "11 4 11 7 13 4 12 11 10 14".split(" ")
b = "11 4 11 7 3 7 10 13 4 8 12 11 10 14 12".split(" ")

result = []
arr = list(map(int,a))

brr = list(map(int,b))
a_dict = {}
b_dict = {}

thangarajan8

Repatition
  1. create even number of records in resultant partitions so the resources are consumed equally
  2. Go for full shuffle so it will cost effective
  3. used to increase or decerase number of partitions

Coalesce:
  1. Create un-even number of records in resultant partitions due to this load will be un-balanced
  2. won't go for full shuffle so it will be fast
  3. used to decrease number of partitions
  

thangarajan8

lines = sc.textFile('data.txt') #reading a text file

lines_filtered = lines.filter(lambda line : ('word1' in line)) #filtering line contain the word "word1"

lines_filtered.first() #took 1s to run
lines_filtered.collect() #took 100s to run

thangarajan8

import sys
from timeit import timeit
n = int(sys.argv[1])

test1 = f"""
a_list= []
for i in range({n}):
    a_list.append(i)
"""
print(timeit(test1))

thangarajan8

{"key_0": "key_0", "key_1": "key_1", "key_2": "key_2", "key_3": "key_3", "key_4": "key_4", "key_5": "key_5", "key_6": "key_6", "key_7": "key_7", "key_8": "key_8", "key_9": "key_9", "key_10": "key_10", "key_11": "key_11", "key_12": "key_12", "key_13": "key_13", "key_14": "key_14", "key_15": "key_15", "key_16": "key_16", "key_17": "key_17", "key_18": "key_18", "key_19": "key_19", "key_20": "key_20", "key_21": "key_21", "key_22": "key_22", "key_23": "key_23", "key_24": "key_24", "key_25": "key_25", "key_26": "key_26", "key_27": "key_27", "key_28": "key_28", "key_29": "key_29", "key_30": "key_30", "key_31": "key_31", "key_32": "key_32", "key_33": "key_33", "key_34": "key_34", "key_35": "key_35", "key_36": "key_36", "key_37": "key_37", "key_38": "key_38", "key_39": "key_39", "key_40": "key_40", "key_41": "key_41", "key_42": "key_42", "key_43": "key_43", "key_44": "key_44", "key_45": "key_45", "key_46": "key_46", "key_47": "key_47", "key_48": "key_48", "key_49": "key_49", "key_50": "key_50", "key_51": "key_51", "ke

thangarajan8

Latency Comparison Numbers (~2012)
----------------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

thangarajan8

var users = [{name: 'rajan', age: 35, favfood: 'icecream'},
             {name: 'king', age: 25, favfood: 'bear'},
             {name:"thanga", age:27, favfood :"briyani"}];
console.log(users);




console.log(users[0].name);
function getAllValuesByKey(users,keyName){