Venkata Gowri Sai Rakesh Kumar Varanasi vvgsrk

Data Platform Architect at Finnair

vvgsrk / snowflake_stream_for_booking_table.sql

Last active June 25, 2020 15:09

	CREATE OR REPLACE STREAM db_dev.revenue.od_forecast_booking_stream
	ON TABLE db_dev.revenue.od_forecast_booking
	APPEND_ONLY = TRUE;

vvgsrk / Parse_Fixed_Width_Text_File_Using_Pandas.ipynb

Created April 12, 2020 09:52

Parse fixed width text file using Pandas library read_fwf method

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

vvgsrk / one_hour_interval_average_load_with_credits_and_query_count.sql

Created April 10, 2020 08:08

1 Hour Interval Average Load with Credits and Query count

	WITH wlh as (
	SELECT DATE_TRUNC('hour', wl.start_time) start_time_trunced_at_hour, HOUR(wl.start_time) start_time_hour,
	AVG(avg_running) avg_running, AVG(avg_queued_load) avg_queued_load,
	AVG(avg_queued_provisioning) avg_queued_provisioning, AVG(avg_blocked) avg_blocked
	FROM snowflake.account_usage.warehouse_load_history wl
	WHERE DATE_TRUNC('DAY', wl.start_time) = '2020-03-26'
	AND wl.warehouse_name = 'PUT_YOUR_WAREHOUSE_NAME'
	GROUP BY start_time_trunced_at_hour, start_time_hour
	ORDER BY start_time_trunced_at_hour asc
	),

vvgsrk / one_hour_interval_average_load_with_credits.sql

Created April 10, 2020 06:32

1 Hour Interval Average Load with Credits

	WITH wlh
	AS (SELECT DATE_TRUNC('hour', wl.start_time) start_time_trunced_at_hour,
	HOUR(wl.start_time) start_time_hour,
	AVG(avg_running) avg_running,
	AVG(avg_queued_load) avg_queued_load,
	AVG(avg_queued_provisioning) avg_queued_provisioning,
	AVG(avg_blocked) avg_blocked
	FROM snowflake.account_usage.warehouse_load_history wl
	WHERE DATE_TRUNC('day', wl.start_time) = '2020-03-26'
	AND wl.warehouse_name = 'PUT_YOUR_WAREHOUSE_NAME'

vvgsrk / five_minute_interval_average_load.sql

Created April 10, 2020 06:27

5 Minute Interval Average Load

	SELECT DATE_TRUNC('hour', start_time) start_time_trunced_at_hour,
	HOUR(start_time) start_time_hour,
	MINUTE(start_time) start_time_min,
	AVG(avg_running) avg_running,
	AVG(avg_queued_load) avg_queued_load,
	AVG(avg_queued_provisioning) avg_queued_provisioning,
	AVG(avg_blocked) avg_blocked
	FROM snowflake.account_usage.warehouse_load_history
	WHERE DATE_TRUNC('day', start_time) = '2020-03-26'
	AND warehouse_name = 'PUT_YOUR_WAREHOUSE_NAME'

vvgsrk / alter_warehouse_max_cluster.sql

Created April 10, 2020 05:36

Scale out snowflak warehouse

ALTER WAREHOUSE put_your_warehouse_name SET max_cluster_count = 10;

vvgsrk / One_Hour_Interval_Average_Load.sql

Created April 10, 2020 04:59

1 Hour Interval Average Load

	SELECT DATE_TRUNC('hour', start_time) start_time_trunced_at_hour,
	HOUR(start_time) start_time_hour,
	AVG(avg_running) avg_running,
	AVG(avg_queued_load) avg_queued_load,
	AVG(avg_queued_provisioning) avg_queued_provisioning,
	AVG(avg_blocked) avg_blocked
	FROM snowflake.account_usage.warehouse_load_history
	WHERE DATE_TRUNC('DAY', start_time) = '2020-03-17'
	AND warehouse_name = 'PUT_YOUR_WAREHOUSE_NAME'
	GROUP BY start_time_trunced_at_hour, start_time_hour

vvgsrk / TestSnowflakeConnection.scala

Last active March 14, 2019 17:29

Read a table from "Snowflake on AWS" using Spark in Windows PC

	package com.vvgsrk.data

	import org.apache.spark.sql.SparkSession
	import net.snowflake.spark.snowflake.Utils.SNOWFLAKE_SOURCE_NAME

	/** This object test "snowflake on AWS" connection using spark
	* from Eclipse, Windows PC.
	*
	* It uses Hadoop 2.7, Spark 2.3.2
	*

vvgsrk / gluepyspark-python-commands.py

Created January 30, 2019 21:14

AWS Glue gluepyspark python commands.

	# Invoke Spark Shell
	$ gluepyspark -v --properties-file /home/glue/glue_spark_shell.properties --packages com.databricks:spark-avro_2.11:4.0.0

	# Import required classes
	import sys
	from awsglue.transforms import *
	from awsglue.utils import getResolvedOptions
	from pyspark.context import SparkContext
	from awsglue.context import GlueContext
	from awsglue.dynamicframe import DynamicFrame, DynamicFrameReader, DynamicFrameWriter, DynamicFrameCollection

vvgsrk / glue-spark-shell-scala-commands.scala

Last active August 23, 2019 02:33

AWS Glue spark-shell scala commands

	// Invoke Spark Shell
	$ glue-spark-shell -v --properties-file /home/glue/glue_spark_shell.properties --packages com.databricks:spark-avro_2.11:4.0.0

	// Import Required Classes
	import org.apache.spark.SparkContext
	import com.amazonaws.services.glue.GlueContext
	import com.amazonaws.services.glue.DynamicFrame
	import com.amazonaws.services.glue.DynamicRecord
	import com.amazonaws.services.glue.MappingSpec
	import com.amazonaws.services.glue.errors.CallSite