Sneha Mehrin snehamehrin

I am a data analytics specialist committed to improving people's lives by building data-driven products backed by scalable data strategies.

snehamehrin / dashboard.html

Created March 20, 2021 12:02

	<!DOCTYPE html>
	<html>
	<body>

	<div class="headerContainer">
	<div style="background-color:white;margin: 1rem;padding: 0.5rem;width:10%" id="heroku">
	<div style="float:left;width: 24%;">
	<svg class="slds-align_absolute-center" width="27" style="fill: #9E7CC1;padding-top: 4px;margin-left: 0px;" height="30" viewBox="0 0 27 30" xmlns="http://www.w3.org/2000/svg">
	<title>heroku-logo</title>
	<path d="M3 0C1.13 0 0 1.11 0 2.903v24.194C0 28.883 1.13 30 3 30h21c1.863 0 3-1.11 3-2.903V2.903C26.994 1.11 25.863 0 24 0H3zm21.042 2c.508.006.958.448.958.929V27.07c0 .487-.45.929-.958.929H2.958C2.45 28 2 27.558 2 27.071V2.93c0-.488.45-.93.958-.93h21.084zM20 25h-2.781v-8.506c0-.774-.237-1.048-.468-1.208-1.396-.959-5.414-.042-7.834.916L7 17.012 7.006 5h2.816v7.917a20.99 20.99 0 0 1 1.882-.482c2.988-.643 5.184-.47 6.616.505.787.536 1.68 1.59 1.68 3.554V25zm-6-15h3.293A16.109 16.109 0 0 0 20 5h-3.287c-.49 1.188-1.385 3.188-2.713 5zM7 25v-7l3 3.

snehamehrin / Sql.ipynb

Created December 1, 2020 21:38

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

snehamehrin / einsteint

Created August 8, 2020 21:52

	q = load "stack_overflow_dataset";
	q = filter q by date('creation_date_Year', 'creation_date_Month', 'creation_date_Day') in ["current month".."current month"];
	result = group q by 'creation_date_Day';
	result = foreach result generate q.'creation_date_Day' as 'creation_date_Day', unique(q.'questionid') as 'A';
	u = load "stack_overflow_dataset";
	u = filter u by date('creation_date_Year', 'creation_date_Month', 'creation_date_Day') in ["1 year ago".."current month - 1 year"];
	u = foreach u generate u.'creation_date_Day' as 'creation_date_Day', unique(u.'questionid') as 'B';
	final_result =cogroup result by 'creation_date_Day' left,u by 'creation_date_Day';
	final_result=foreach final_result generate result.'creation_date_Day' as 'Day',sum(result.'A') as 'Current_Year',coalesce(sum(u.'B'),0) as 'Previous_Year';

snehamehrin / Execute_Stack_Processing_Script.py

Last active August 7, 2020 12:31

	import boto3
	import setup

	def run_jobs():
	client = boto3.client('emr', region_name='us-east-1')
	response = client.list_clusters(

	ClusterStates=['WAITING'
	],

snehamehrin / copy files

Created August 7, 2020 11:55

	scp -i bigdata.pem RedshiftJDBC42-no-awssdk-1.2.20.1043.jar hadoop@ec2–3–88–110–90.compute-1.amazonaws.com:/home/hadoop/

	scp -i bigdata.pem Execute.sh hadoop@ec2–3–88–110–90.compute-1.amazonaws.com:/home/hadoop/

	scp -i bigdata.pem stack-processing.py hadoop@ec2–3–88–110–90.compute-1.amazonaws.com:/home/hadoop/

snehamehrin / Execute.sh

Last active August 7, 2020 11:49

	spark-submit --jars RedshiftJDBC42-no-awssdk-1.2.20.1043.jar
	--packages org.apache.spark:spark-avro_2.11:2.4.3,com.databricks:spark-redshift_2.11:2.0.1 stack-processing.py

snehamehrin / stack-processing.py

Last active August 23, 2020 12:18

	#Import All Functions
	from pyspark.sql import SQLContext
	from pyspark.sql import functions as F
	from pyspark.sql import SparkSession
	from pyspark.sql.functions import unix_timestamp, to_date, date_format, month, year, dayofyear, dayofweek, col
	from pyspark.sql.types import TimestampType
	from pyspark.sql import functions as F
	from pyspark.sql import SparkSession
	from pyspark.sql.functions import unix_timestamp, to_date, date_format, month, year, dayofyear, dayofweek, col
	from pyspark.sql.types import TimestampType

snehamehrin / s3access.py

Created August 6, 2020 23:08

	sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", "")
	sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", "")

snehamehrin / redshift.py

Created August 6, 2020 22:54

	df_duplicates.write.format("com.databricks.spark.redshift")\
	.option("url", "jdbc:redshift://redshift-cluster-1.c9lgtyzxfycf.us-east-1.redshift.amazonaws.com:5439/dev?user=awsuser&password=")\
	.option("dbtable", "stackoverflow")\
	.option("forward_spark_s3_credentials","true")\
	.option("tempdir", "s3n://stack-overflow-bucket")\
	.mode("append")\
	.save()

snehamehrin / Spark-Processing.ipynb

Created August 6, 2020 12:38

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.