MarkPryceMaherMSFT’s gists

MarkPryceMaherMSFT / create-list-delete-warehouses.ipynb

Created November 30, 2024 12:17

Script showing how to create, list and delete Fabric warehouses using the REST API

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

MarkPryceMaherMSFT / workspace size.ipynb

Last active December 6, 2024 16:02

This script can be used to find the size of a workspace, lakehouse , warehouse or table

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

MarkPryceMaherMSFT / create_many_tables.py

Created October 31, 2024 17:40

use with the create table scruot

	create_new_table("control", False, False, False, False, False, False, False, False, False )
	create_new_table("control", True, False, False, False, False, False, False, False, False )
	create_new_table("control", False, True, False, False, False, False, False, False, False )
	create_new_table("control", True, True, False, False, False, False, False, False, False )
	create_new_table("control", False, False, True, False, False, False, False, False, False )
	create_new_table("control", True, False, True, False, False, False, False, False, False )
	create_new_table("control", False, True, True, False, False, False, False, False, False )
	create_new_table("control", True, True, True, False, False, False, False, False, False )
	create_new_table("control", False, False, False, True, False, False, False, False, False )
	create_new_table("control", True, False, False, True, False, False, False, False, False )

MarkPryceMaherMSFT / generate_different_types_of_table.py

Created October 31, 2024 17:39

script to generate different delta tables

	def create_new_table(tablename, columnmapping,delete_vectors,partitions,columnmappingbyid,liquidclustering,optimize,vacuum,writeStatsAsStruct,writeStatsAsJson):
	# What the function does goes here
	# use the default sample table, publicholidays
	tablename = "PARQUET_2_0"

	if columnmapping==True and columnmappingbyid==False:
	tablename=tablename+"_columnmapping_by_name"
	if delete_vectors==True:
	tablename=tablename+"_delete_vectors"
	if partitions==True:

MarkPryceMaherMSFT / Create a case-insensitive collation warehouse.ipynb

Created October 17, 2024 11:09

Fabric Notebook to create a case-insensitive collation warehouse

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

MarkPryceMaherMSFT / createwarehouse.py

Last active November 30, 2024 10:53

Create Fabric warehouse with case insensitive collation

	# details from Article : https://learn.microsoft.com/en-us/fabric/data-warehouse/collation
	# default collation is Latin1_General_100_BIN2_UTF8
	# new collation is Latin1_General_100_CI_AS_KS_WS_SC_UTF8

	#sempy version 0.4.0 or higher
	!pip install semantic-link --q
	import json
	import sempy.fabric as fabric
	from sempy.fabric.exceptions import FabricHTTPException, WorkspaceNotFoundException

MarkPryceMaherMSFT / test_the_sync.py

Created September 27, 2024 21:11

test the performance of the sync

	import pandas as pd
	import struct
	import sqlalchemy
	import pyodbc
	import notebookutils
	import sempy.fabric as fabric
	import time
	from pyspark.sql import functions as fn
	from datetime import datetime
	def create_engine(connection_string : str):

MarkPryceMaherMSFT / wait_for_table_to_be_updated.py

Created September 27, 2024 20:51

Test how long it tasks to refresh data in the SQL Endpoint

	import pandas as pd
	import struct
	import sqlalchemy
	import pyodbc
	import notebookutils
	import sempy.fabric as fabric
	import time
	from pyspark.sql import functions as fn
	from datetime import datetime
	def create_engine(connection_string : str):

MarkPryceMaherMSFT / Query SQL Endpoint.py

Last active September 28, 2024 09:23

Script that gets the sql endpoint for a workspace and lakehouse

	import pandas as pd
	import struct
	import sqlalchemy
	import pyodbc
	import notebookutils
	import sempy.fabric as fabric
	def create_engine(connection_string : str):
	token = notebookutils.credentials.getToken('https://analysis.windows.net/powerbi/api').encode("UTF-16-LE")
	token_struct = struct.pack(f'<I{len(token)}s', len(token), token)
	SQL_COPT_SS_ACCESS_TOKEN = 1256

MarkPryceMaherMSFT / gist:2af9c203fd17157a4b68509bf055e0d9

Last active August 28, 2024 13:50

	# Purpose: Print out details of partitions, files per partitions, and size per partition in GB.
	from notebookutils import mssparkutils

	# Define ABFSS path for your delta table. You can get ABFSS path of a delta table by simply right-clicking on table name and selecting COPY PATH from the list of options.
	# Remove the path and the lakehouse name.
	delta_table_path = "abfss://[email protected]/"
	HowManyLogsIsTooManyLogs = 50 ## 50 feels like a biggest number
	HowLargeALogCanBeBeforeItsAIssue= 1 ## Value in MB

	# List all partitions for given delta table

Mark Pryce-Maher MarkPryceMaherMSFT