Anjaiah Methuku anjijava16
anjijava16
/ cosmosdb_azure.scala
Created
January 18, 2022 00:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Read data | |
| dfCustomer = spark.read\ | |
| .format("cosmos.oltp")\ | |
| .option("spark.synapse.linkedService", "AdventureWorksSQL")\ | |
| .option("spark.cosmos.container", "Customer")\ | |
| .load() | |
| display(dfCustomer.limit(10)) | |
| # Write data |
anjijava16
/ Spark_Cosmosdb_cassandra.scala
Last active
December 27, 2021 17:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Spark Streaming with Cassandra (Azure Cosmos DB) | |
| 1. https://github.com/mspnp/azure-databricks-streaming-analytics |
anjijava16
/ HASHROW, HASHBUCKET, HASHAMP in Teradata
Last active
December 21, 2021 04:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| 1.http://www.teradatahelp.com/2010/09/how-can-you-view-data-distribution-in.html | |
| 1. http://rajgopal-gurrapushala.blogspot.com/2012/11/types-of-hash-functions-used-in-teradata_734.html | |
| Create Table emp | |
| ( | |
| ID BIGINT NOT NULL GENERATED BY DEFAULT AS IDENTITY | |
| (START WITH 1 | |
| INCREMENT BY 1 | |
| MINVALUE -999999999999999999 |
anjijava16
/ azure_synpase_cosmos_db.json
Created
December 20, 2021 04:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "_id" : "F0FF0BA8-CEF4-454C-9173-FA7C546F7179", | |
| "customerId" : "A852CB99-DAA1-4348-A86F-622D4478A7D0", | |
| "orderDate" : "2013-07-11T00:00:00", | |
| "shipDate" : "2013-07-18T00:00:00", | |
| "details" : [ | |
| { | |
| "sku" : "LJ-0192-M", | |
| "name" : "Long-Sleeve Logo Jersey, M", | |
| "price" : 49.99, |
anjijava16
/ any_sql_scripts_run.java
Last active
November 25, 2021 06:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| package com.iwinner.runanysql; | |
| import java.io.BufferedReader; | |
| import java.io.FileReader; | |
| import java.io.Reader; | |
| import java.sql.Connection; | |
| import java.sql.DriverManager; | |
| import java.sql.ResultSet; | |
| import java.sql.SQLException; | |
| import java.sql.Statement; |
anjijava16
/ Logging.json
Created
July 23, 2021 13:23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| insertId: "wr3zn3e373bw" | |
| logName: "projects/iwinner-data-318822/logs/cloudaudit.googleapis.com%2Factivity" | |
| operation: { | |
| first: true | |
| id: "operation-1627046240424-5c7ca39319163-07a56cfa-f16ec9a5" | |
| producer: "compute.googleapis.com" | |
| } | |
| protoPayload: { | |
| @type: "type.googleapis.com/google.cloud.audit.AuditLog" |
anjijava16
/ Algorthims.sh
Created
July 21, 2021 03:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| https://morioh.com/p/4d9c2c2cb9ae?f=5c224490c513a556c9042463&fbclid=IwAR0EA40-1MS3YNSJRkdnGIjAe6of7e3kIRGVRXq1t14aT44eIx4N6ykJ4L8 |
anjijava16
/ Spark_Best.scala
Last active
July 17, 2021 23:14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1 | |
| https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a | |
| https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1 | |
| https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a | |
| Small files spark | |
| https://github.com/capt-blackdron/pyspark_custom_repartitioner/blob/main/custom_repartitioner.py | |
| Scd type 2 | |
| https://github.com/capt-blackdron/pyspark-examples/blob/main/SCD_Type_2_Implementation_in_PySpark.ipynb | |
| Spark program |
anjijava16
/ Binary_dataTypes.sh
Created
July 13, 2021 04:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| https://stackoverflow.com/questions/59617081/cast-a-column-to-binary-datatype-in-spark | |
| https://stackoverflow.com/questions/53075020/why-does-spark-infer-a-binary-instead-of-an-arraybyte-when-creating-a-datafram | |
| https://stackoverflow.com/questions/41149522/extract-byte-from-spark-binarytype | |
| https://stackoverflow.com/questions/47902691/filtering-by-binary-type-in-spark-sql | |
| https://stackoverflow.com/questions/60627680/spark-binarytype-to-scala-java |
anjijava16
/ slot_utilization.sql
Created
July 11, 2021 15:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| SELECT | |
| COUNT(*) TOTAL_QUERIES, | |
| SUM(total_slot_ms/TIMESTAMP_DIFF(end_time,creation_time,MILLISECOND)) AVG_SLOT_USAGE, | |
| SUM(TIMESTAMP_DIFF(end_time,creation_time,SECOND)) TOTAL_DURATION_IN_SECONDS, | |
| AVG(TIMESTAMP_DIFF(end_time,creation_time,SECOND)) AVG_DURATION_IN_SECONDS, | |
| SUM(total_bytes_processed*10e-12) TOTAL_PROCESSED_TB, | |
| EXTRACT (DATE FROM creation_time) AS EXECUTION_DATE, | |
| user_email as USER | |
| FROM `iwinner-data-318822.region-us.INFORMATION_SCHEMA.JOBS_BY_PROJECT` | |
| WHERE state='DONE' |