Anjaiah Methuku anjijava16

💭

Awesome

Real-time Streaming ■ BigData ■ Machine Learning ■ CLOUD ■ JAVA ■ PYTHON ■ Blog ■ Remain Curious And Keep Learning .....

anjijava16 / any_sql_scripts_run.java

Last active November 25, 2021 06:11


	package com.iwinner.runanysql;
	import java.io.BufferedReader;
	import java.io.FileReader;
	import java.io.Reader;
	import java.sql.Connection;
	import java.sql.DriverManager;
	import java.sql.ResultSet;
	import java.sql.SQLException;
	import java.sql.Statement;

anjijava16 / Logging.json

Created July 23, 2021 13:23

	{
	insertId: "wr3zn3e373bw"
	logName: "projects/iwinner-data-318822/logs/cloudaudit.googleapis.com%2Factivity"
	operation: {
	first: true
	id: "operation-1627046240424-5c7ca39319163-07a56cfa-f16ec9a5"
	producer: "compute.googleapis.com"
	}
	protoPayload: {
	@type: "type.googleapis.com/google.cloud.audit.AuditLog"

anjijava16 / Algorthims.sh

Created July 21, 2021 03:07

https://morioh.com/p/4d9c2c2cb9ae?f=5c224490c513a556c9042463&fbclid=IwAR0EA40-1MS3YNSJRkdnGIjAe6of7e3kIRGVRXq1t14aT44eIx4N6ykJ4L8

anjijava16 / Spark_Best.scala

Last active July 17, 2021 23:14

	https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1
	https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a
	https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1
	https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a

	Small files spark
	https://github.com/capt-blackdron/pyspark_custom_repartitioner/blob/main/custom_repartitioner.py
	Scd type 2
	https://github.com/capt-blackdron/pyspark-examples/blob/main/SCD_Type_2_Implementation_in_PySpark.ipynb
	Spark program

anjijava16 / Binary_dataTypes.sh

Created July 13, 2021 04:33

	https://stackoverflow.com/questions/59617081/cast-a-column-to-binary-datatype-in-spark

	https://stackoverflow.com/questions/53075020/why-does-spark-infer-a-binary-instead-of-an-arraybyte-when-creating-a-datafram

	https://stackoverflow.com/questions/41149522/extract-byte-from-spark-binarytype

	https://stackoverflow.com/questions/47902691/filtering-by-binary-type-in-spark-sql

	https://stackoverflow.com/questions/60627680/spark-binarytype-to-scala-java

anjijava16 / slot_utilization.sql

Created July 11, 2021 15:30

	SELECT
	COUNT(*) TOTAL_QUERIES,
	SUM(total_slot_ms/TIMESTAMP_DIFF(end_time,creation_time,MILLISECOND)) AVG_SLOT_USAGE,
	SUM(TIMESTAMP_DIFF(end_time,creation_time,SECOND)) TOTAL_DURATION_IN_SECONDS,
	AVG(TIMESTAMP_DIFF(end_time,creation_time,SECOND)) AVG_DURATION_IN_SECONDS,
	SUM(total_bytes_processed*10e-12) TOTAL_PROCESSED_TB,
	EXTRACT (DATE FROM creation_time) AS EXECUTION_DATE,
	user_email as USER
	FROM `iwinner-data-318822.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT`
	WHERE state='DONE'

anjijava16 / Replicas_In_Cloud.scala

Created July 11, 2021 04:13

	Cloud Spanner has three types of replicas:
	i. read-write replicas,
	ii. read-only replicas,
	iii. witness replicas.

anjijava16 / AWS_AZUE_GCP.sh

Created July 10, 2021 21:29

	Azure:
	ADF/Databricks with Spark- Ingestion framework
	ADLS - Data Storage
	ADB - Transformations
	Data Flows/ Polybase - To load data to Warehouse
	Synapse - Datawarehouse
	Azure SQL - Metadata Storage
	ADF - Orchestration
	Logic Apps : Alerts/Email
	Azure Devops : Code Deployment

anjijava16 / Free_Rest_API.txt

Last active June 23, 2021 12:56

	https://www.boredapi.com/api/activity
	https://www.mockaroo.com/help/terms_of_use
	https://randomuser.me/api/

anjijava16 / boundary_query_vs_split_by_sqoop.sh

Created June 21, 2021 03:51

	https://stackoverflow.com/questions/40838036/what-is-the-difference-between-split-by-and-boundary-query-in-sqoop


	https://discuss.itversity.com/t/using-boundary-query/18673



	https://stackoverflow.com/questions/37206232/sqoop-import-composite-primary-key-and-textual-primary-key