dvu4 / download_blob_files_from_blob_container_in_azure_storage_account.md

Last active February 6, 2023 23:12

import typer
from azure.storage.blob import BlobServiceClient

STORAGEACCOUNTNAME = "qadseus2prxinvsa01"
STORAGEACCOUNTURL= "https://qadseus2prxinvsa01.blob.core.windows.net" #<storage_account_url>
STORAGEACCOUNTKEY= "<storage_account_key>"

dvu4 / search_for_a_pattern_in_files.md

Last active February 10, 2023 00:17

search for "service" using the `grep`

alias grep="grep --color=auto" 
grep -rn . -e "service"

Here's what each of the options and arguments mean:

-r: Recursively search all files in the specified directory and its subdirectories.

dvu4 / get_lastest_file_in_Azure_Data_Lake_Storage.md

Last active April 18, 2023 15:27

!pip install pyyaml

import yaml
from pyspark.dbutils import DBUtils
import pyspark.sql.functions as F
import time

dvu4 / get_lastest_file_in_Azure_Data_Lake_Storage_2.md

Last active April 18, 2023 15:34

import yaml
from pyspark.dbutils import DBUtils
import pyspark.sql.functions as F
import time



def get_dir_content(path):

dvu4 / display_secret_from_dbutils.secrets.get.md

Last active May 24, 2023 18:10

How to list the secret scope

from pyspark.dbutils import DBUtils
dbutils.secrets.listScopes()

the ouput :

Out[16]: [SecretScope(name='mleng-secrets')]

dvu4 / filter_file_by_date_with_dbutils.md

Last active June 22, 2023 20:59

List all files in container ds-tmartch-output in storage account prodfixdseus2tmartchsa01

path = "abfss://[email protected]/acc-activation"
dbutils.fs.ls(path)

The result looks like this

dvu4 / save_multiple_dataframes_into_text_file_in_Pyspark.md

Last active July 21, 2023 18:59

This function save multiple dataframes with different headers into one pipe delimited file (.txt file)

from pyspark.dbutils import DBUtils
import pyspark.sql.functions as F
import pyspark
import pandas as pd
from itertools import chain
from functools import reduce

dvu4 / Replacing value in a column by searching a dictionary in Pyspark.md

Last active July 17, 2024 22:09

from pyspark.dbutils import DBUtils
import pyspark.sql.functions as F
import pyspark
import pandas as pd
from itertools import chain
from functools import reduce


# mapping Walgreens status to RX-Lighting status and substatus

dvu4 / run_commands_in_loop.md

Created September 20, 2023 18:46

for i in  {"04eb34f2-3778-46e7-a1f5-391291a2bc6c","6af03cc1-8262-4e81-9fc0-bd6131b23996","7eecf3f5-67f0-4792-a9c2-c9f60ff5a96c","c5a71e40-4636-41bb-8232-b8b00fe470fb","026520ce-c449-43e1-ac4f-215db4827af0"}; do az ad sp show --id api://${i} --verbose ; done

dvu4 / create_databricks_hive_metastore.md

Last active September 25, 2023 21:32

This script will create Hive metastore table in Databricks Catalog

Create Hive metastore table

# Create database schema
spark.sql("CREATE DATABASE IF NOT EXISTS csgci")


# Create Hive Internal table
df.write.mode("overwrite").option("overwriteSchema", "true").saveAsTable("csgci.drug_ic")

Duc Vu dvu4

search for "service" using the grep

How to list the secret scope

Create Hive metastore table

search for "service" using the `grep`