Skip to content

Instantly share code, notes, and snippets.

View lordlinus's full-sized avatar

Sunil Sattiraju lordlinus

View GitHub Profile
colordiff <(jq -S '.configurations | .[].\"$1\".properties | select (.!=null)' $2) <(jq -S '.configurations | .[].\"$1\".properties | select (.!=null)' $3)
{
"attrFields": {
"attribute_1": "International aid/development worker",
"attribute_2": 3762,
"attribute_3": "B-",
"attribute_4": "Scott-Mendoza",
"attribute_5": "M",
"attribute_6": "2007-01-09",
"attribute_7": 8.302579812
},
#!/usr/bin/env python
# csv2hashed.py
#
# Usage:
# $ csv2hashed.py --salt SALT --input original.csv --output hashedfile.csv
#
# Assumptions:
# - This script performs basic normalization and is configured to read first_name, last_name and birthdate columns only
# - This script assumes the first column is always PersonID which
@lordlinus
lordlinus / databricks_cluster_deployment.yml
Created April 5, 2021 04:59
Azure devops yaml pipeline to deploy Azure Databricks cluster ( end to end )
# Azure DevOps pipeline to build Databricks cluster
# This pipeline depends on "variable-group-01" to provide the below variables
# ARM_CLIENT_ID
# ARM_SUBSCRIPTION_ID
# ARM_TENANT_ID
# mySecret NOTE: This variable is mapped to ARM_CLIENT_SECRET below
trigger:
- starter
@lordlinus
lordlinus / databricks_cluster_deployment.sh
Created April 5, 2021 05:02
Bash script to deploy Databricks Cluster and other dependencies
#! /usr/bin/bash
set -o errexit
set -o nounset
set -o pipefail
export ARM_SUBSCRIPTION_ID= XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_TENANT_ID= XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_CLIENT_ID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_CLIENT_SECRET=XXXXXXXXXXXXX
export MANAGEMENT_RESOURCE_ENDPOINT="https://management.core.windows.net/" # This is Fixed value (DO NOT CHANGE)
#! /usr/bin/bash
set -o errexit
set -o nounset
set -o pipefail
export ARM_SUBSCRIPTION_ID= XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_TENANT_ID= XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_CLIENT_ID=XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
export ARM_CLIENT_SECRET=XXXXXXXXXXXXX
# Login using service principle
# Get 2 values from the output
command | jq -r '[.Key1.key2,.Key1.key2]'
# Get last element from list
command | jq .key[-1].key2
@lordlinus
lordlinus / synapse_spark_job.py
Created June 10, 2021 08:42
synapse spark job submit via REST api
#!/usr/bin/env python3
import os
import logging
from subprocess import run, PIPE
import requests
import json
import time
from datetime import datetime
now_file = f"{datetime.utcnow().strftime('%X').replace(':','-')}.log"
@lordlinus
lordlinus / generate_fake_data.py
Created June 18, 2021 04:16
Generate fake data
import cProfile
import io
import pstats
import random
import pandas as pd
from mimesis import Address, Datetime, Person
from mimesis.enums import Gender
person = Person('en')
addess = Address()
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.