Sunil Sattiraju lordlinus
lordlinus
/ databricks_cluster_deployment.yml
Created
April 5, 2021 04:59
Azure devops yaml pipeline to deploy Azure Databricks cluster ( end to end )
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Azure DevOps pipeline to build Databricks cluster | |
| # This pipeline depends on "variable-group-01" to provide the below variables | |
| # ARM_CLIENT_ID | |
| # ARM_SUBSCRIPTION_ID | |
| # ARM_TENANT_ID | |
| # mySecret NOTE: This variable is mapped to ARM_CLIENT_SECRET below | |
| trigger: | |
| - starter |
lordlinus
/ csv2hashed.py
Last active
November 8, 2021 03:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env python | |
| # csv2hashed.py | |
| # | |
| # Usage: | |
| # $ csv2hashed.py --salt SALT --input original.csv --output hashedfile.csv | |
| # | |
| # Assumptions: | |
| # - This script performs basic normalization and is configured to read first_name, last_name and birthdate columns only | |
| # - This script assumes the first column is always PersonID which |
lordlinus
/ jolt transform combine
Created
June 25, 2020 17:00
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "attrFields": { | |
| "attribute_1": "International aid/development worker", | |
| "attribute_2": 3762, | |
| "attribute_3": "B-", | |
| "attribute_4": "Scott-Mendoza", | |
| "attribute_5": "M", | |
| "attribute_6": "2007-01-09", | |
| "attribute_7": 8.302579812 | |
| }, |
lordlinus
/ ambari_compare_config
Created
April 5, 2020 23:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| colordiff <(jq -S '.configurations | .[].\"$1\".properties | select (.!=null)' $2) <(jq -S '.configurations | .[].\"$1\".properties | select (.!=null)' $3) |
lordlinus
/ jq_flatten_json
Created
April 2, 2020 10:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| cat some.json| jq '. as $in | |
| | reduce leaf_paths as $path ({}; | |
| . + { ($path | map(tostring) | join(".")): $in | getpath($path) })' | sort |
lordlinus
/ jolt_add_new_fields_to_json_array
Created
March 2, 2020 09:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| [{ | |
| "operation": "default", | |
| "spec": { | |
| "temp": { | |
| "name": "Token", | |
| "type": "string", | |
| "doc": "DR generate Hex token" | |
| } | |
| } | |
| }, |
lordlinus
/ cbd_start.sh
Created
January 30, 2020 10:03
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| apt-get install \ | |
| apt-transport-https \ | |
| ca-certificates \ | |
| curl \ | |
| gnupg-agent \ | |
| software-properties-common -y | |
| curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - | |
| sudo add-apt-repository \ |
lordlinus
/ jq_csv_to_json
Last active
November 28, 2019 18:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # converting csv into json and fix the value with spaces | |
| jq split("\n") | .[] | split(",") | {k:.[0], v:[.[1] | split(" ") | .[]]} |
lordlinus
/ package-install-check.py
Last active
October 20, 2019 03:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import numpy as np,pandas as pd,dash,dash_daq as daq,fastai,fbprophet,joblib,lightgbm as lgb,matplotlib,matplotlib.pyplot as plt,plotly,psycopg2,pyqtgraph as pg,spacy,tensorflow as tf,torch,torch.nn as nn,torch.optim as optim,xgboost as xgb | |
| from tensorflow import keras | |
| package_version=f"""---Package and Version Installed --- | |
| Numpy: {np.__version__} | |
| Pandas: {pd.__version__} | |
| Dash: {dash.__version__} | |
| Dash_daq: {daq.__version__} | |
| Fastai: {fastai.__version__} | |
| Fbprophet: {fbprophet.__version__} | |
| JobLib: {joblib.__version__} |
lordlinus
/ final_conda_ps1
Created
October 8, 2019 05:51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| dir "c:\aws\download" # Check the dir and list the files | |
| Write-Host 'Conda packages Install' | |
| Set-Location "c:\aws\download" | |
| # Ensure anaconda is installed | |
| Start-Process -FilePath 'conda' -ArgumentList 'install anaconda -y' -wait | |
| # Install packages | |
| Start-Process -FilePath 'conda' -ArgumentList 'install tensorflow=1.14.0 keras=2.2.4 -y' -wait | |
| Start-Process -FilePath 'conda' -ArgumentList 'install -c conda-forge dash=1.3.0 orange3==3.23.1 lightgbm==2.3.0 -y' -wait |