Skip to content

Instantly share code, notes, and snippets.

View do-me's full-sized avatar

Dominik Weckmüller do-me

View GitHub Profile
@do-me
do-me / osrm_docker_windows.md
Created October 11, 2021 07:58 — forked from AlexandraKapp/osrm_docker_windows.md
How to set up your own OSRM backend with Docker on Windows

How to set up your own OSRM backend with Docker on Windows

The OSRM docker quick start provides a great explanation on how to set up the container: https://hub.docker.com/r/osrm/osrm-backend/

Yet - for Docker on Windows minor changes were necessary for me (otherwise I'd get "File not found" or "Permission" errors).

This is how it worked for me:

1. Pull the image

@do-me
do-me / ci.yml
Created February 28, 2022 17:21
Compress and deploy mkdocs (material) wiki with github pages. Uses gzipper for brotli, zopfli and gzip compression. Builds on standard CI by @squidfunk (https://squidfunk.github.io/mkdocs-material/publishing-your-site/#github-pages) from
# Mkdocs routine for continuous deployment
# This file must be on main branch under .github/workflows/ci.yml
name: ci
on:
push:
branches:
- main
jobs:
deploy:
@do-me
do-me / many_points_to_map.py
Created March 17, 2022 16:21
Display many points from pandas to datashader image with bounding box filter
# Reads hdf file, filters by bounding box, plots image with datashader
# Best in Jupyter. Script executed with 66.000.000 points in 7 sec on a laptop.
import datashader as ds, pandas as pd, colorcet
df = pd.read_hdf('output.hdf')
# Filter USA Contiguous 48
df = df[df['latitude'].between(22.547351, 51.327674)]
df = df[df['longitude'].between(-129.9599164,-53.274208)]
@do-me
do-me / Leaflet_Map_Generator.py
Created March 13, 2023 15:05
Small script for generating a standalone leaflet html with a custom tile server with pandas and folium
import pandas as pd
import json
import folium
from folium.plugins import MarkerCluster
# Load the JSON data, e.g. from https://cordis.europa.eu/datalab
with open("lat_long_input.json") as f:
data = json.load(f)
# Extract the fields of interest from the 'content' section of the JSON
@do-me
do-me / load_gzipped_csv_with_papa_parse.js
Last active April 26, 2023 15:42
Load a gzipped csv and parse with papa parse in javascript
function papa_load_default_csv(){
fetch('data/test.csv.gz')
.then(response => response.arrayBuffer())
.then(buffer => pako.inflate(new Uint8Array(buffer)))
.then(decompressed => {
// convert binary data to string
const decoder = new TextDecoder('utf-8');
const csv = decoder.decode(decompressed);
// parse csv with Papa Parse
Papa.parse(csv, {
@do-me
do-me / Pandas_groupby_sum.py
Created November 8, 2023 10:07
Simple pandas groupby with sum
import pandas as pd
df = pd.read_csv("your_file.csv")
df = df.groupby(["all","columns","you","need_for_grouping"])['value_for_sum'].sum().reset_index()
df
@do-me
do-me / Pandas_groupby_sum_original_index_num.py
Created November 8, 2023 10:32
Pandas groupby with sum and original min index number
import pandas as pd
df = pd.read_excel("adressen.xlsx")
grouped = df.groupby(["Straße", "Hausnummer", "PLZ", "Ort"]).agg({'Servicezeit in Sekunden': 'sum'}).reset_index()
### originale Reihenfolge ableiten
grouped['Reihenfolge'] = df.groupby(["Straße", "Hausnummer", "PLZ", "Ort"]).apply(lambda x: list(x.index)).reset_index(drop=True)
grouped['Reihenfolge'] = grouped['Reihenfolge'] .apply(lambda x: min([int(i) for i in x]))
grouped = grouped.sort_values("Reihenfolge").reset_index(drop=True)
@do-me
do-me / pydeck.py
Created November 16, 2023 10:56
Geopandas scatterplot jenks colors to pydeck deck.gl (ScatterplotLayer & GeoJsonLayer)
import geopandas as gpd
import pydeck as pdk
gdf = gpd.read_parquet("points.parquet") # lat lon columns in EPSG:4326 included
polys = gpd.read_parquet("polys.parquet") # load polygons/multipolygons
polys['geometry'] = polys['geometry'].buffer(0) # a trick in geopandas to fix invalid geometries, holes etc.
polys = polys.to_crs("4326") # IMPORTANT: Must be in 4326, else map won't display
def hex_to_rgb(hex_color):
hex_color = hex_color.lstrip('#')
@do-me
do-me / pandas_multiprocessing.py
Created November 17, 2023 09:41
Multiprocessing pandas with tqdm process_map
from tqdm.contrib.concurrent import process_map
# ONLY WORKING ON UBUNTU NOT ON WINDOWS - tested on Python 3.9 - 3.11
# import your pandas df here
# test_array = df["your_colum"].to_list() # to_list is crucial as it must be picklable
def test_function(x):
return x*x
@do-me
do-me / mount.bash
Created November 17, 2023 10:45
Mount a hard disk connected to a Windows PC on Ubuntu for Windows
sudo mount -t drvfs D: /mnt/d