Skip to content

Instantly share code, notes, and snippets.

View thehappycheese's full-sized avatar

thehappycheese

  • Australia
View GitHub Profile
@thehappycheese
thehappycheese / find_nearest.py
Last active January 19, 2023 02:50
Fill blank row with nearest populated row with data using a chainage from/to location
for group_index, group in df.groupby(["road_number","cway"]):
blank_rows = group[group["cluster"].isna()]
filled_rows = group[group["cluster"].notna()]
for blank_row_index, blank_row in blank_rows.iterrows():
# find distance by looing for minimum "signed overlap"
overlap_min = np.maximum(filled_rows["slk_from"], blank_row["slk_from"])
overlap_max = np.minimum(filled_rows["slk_to"], blank_row["slk_to"])
@thehappycheese
thehappycheese / blocking_to_async_wrapper.py
Created January 24, 2023 08:11
A hack to turn a legacy blocking funciton into an async function
def run_in_executor(f):
"""
This is a hack to turn a legacy blocking funciton into an async function.
Thanks to balki https://stackoverflow.com/a/53719009/1782370
Example:
The following example shows how to use use a blocking
`azure.identity` credential type with `pandas.read_parquet()`.
Pandas normally requires that you use one of the limited
@thehappycheese
thehappycheese / fuzzy_column_name_match.py
Created January 24, 2023 08:13
Helper function to generate dictionary map to rename columns in one pandas dataframe to match those in another dataframe
def fuzzy_column_name_match(list1, list2):
from fuzzywuzzy import process
threshold = 60
response = {}
for name_to_find in list1:
resp_match = process.extractOne(name_to_find ,list2)
if resp_match[1] > threshold:
response[name_to_find] = resp_match[0]
return response
@thehappycheese
thehappycheese / calculate_nickmapbi_offset.py
Created March 28, 2023 05:04
Function to calculate approximate lane offset from carriageway and xsp
def calculate_nickmapbi_offset(carriageway, xsp):
return {
"L":{
f"L{num+1}": +3.5/2 - 3.5*num for num in range(0,7)
},
"R":{
f"R{num+1}": -3.5/2 + 3.5*num for num in range(0,7)
},
"S":{
f"L{num+1}": -3.5/2 - 3.5*num for num in range(0,7)
@thehappycheese
thehappycheese / get_arcgis_layer_as_geopandas.py
Created March 30, 2023 03:05
Fetch arcgis rest open data as geopandas dataframe
#pip install pandas geopandas arcgis arcgis2geojson
import json
import geopandas as gpd
from geopandas import GeoDataFrame
from arcgis.features import FeatureLayer
from arcgis2geojson import arcgis2geojson
def get_arcgis_layer_as_geopandas(url:str) -> GeoDataFrame:
layer = FeatureLayer(url)
@thehappycheese
thehappycheese / serve_once.py
Created October 10, 2023 05:51
Starts a thread and serves the given data just once before closing the thread. Use with Selenium.
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
def serve_once(data:str, port=8000, host="127.0.0.1"):
"""
Serves the given data on the given port and host, then returns after
responding to one request.
This is a helpful way to send HTML to a browser being controled using Selenium.
"""
@thehappycheese
thehappycheese / prompt.ps1
Created October 23, 2023 10:21
Shorten powershell prompt for tutorials
function prompt {
$currentPath = (Get-Location).Path
$driveLetter = $currentPath.Substring(0,1)
$folderName = Split-Path $currentPath -Leaf
"$driveLetter:/.../$folderName> "
}
from pathlib import Path
import keyring
import pandas as pd
import os
import sys
# You need access to the compiled .NET AdomdClient .dll
# You can get it using nuget
# ```powershell
# nuget.exe install Microsoft.AnalysisServices.AdomdClient.retail.amd64
# nuget.exe install Microsoft.AnalysisServices.Tabular
@thehappycheese
thehappycheese / split_dataframe_by_weighted_sequence.py
Created November 17, 2023 03:29
Split DataFrame by Weighted Sequence
import pandas as pd
def split_dataframe_by_weighted_sequence(
df:pd.DataFrame,
split_column:str,
weight_column:str,
skew:float=0.5
):
"""
Split a dataframe into two parts by a sequence of values and a sequence of weights.
@thehappycheese
thehappycheese / Mendelbressenuiuabrott.ua
Created December 26, 2023 19:02
Mendelbressenuiuabrott
# pairwise coefficient of interpolation
Interp ← ÷:⊃(-≡⊢|≡/-)◫2
MarchingSquares ← (
:
⊃(
≡(◫2)Interp
| ◫2⍜⍉Interp
| ≡≡(°⋯♭)◫2_2> # types
| ≡°⍉⍉⊞⊟∩(⇡-1)⊃(⊢⇌|⊢)△