kmatt / duckssh.py

Created August 29, 2023 01:36

DuckDB over SSH

	"""
	Run DuckDB query on over SSH to avoid scanning full file set on a remote server,
	and make results availble to local DuckDB process
	"""

	import io, paramiko, duckdb

	sql = "SELECT * FROM read_json_auto('/path/to/data.json')"
	cmd = f'duckdb -csv -s "{sql}"'

kmatt / pyvendor.sh

Created August 18, 2023 22:31

Vendor Python modules from Pip

pip download --no-deps --dest vendor -r requirements.txt

kmatt / build-spark-pip.sh

Created July 12, 2023 23:44

PySpark build

	# https://spark.apache.org/docs/3.4.1/building-spark.html

	export MAVEN_OPTS="-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g"

	#./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver
	./dev/make-distribution.sh --name custom-spark --pip --tgz

	#python setup.py sdist

	pip install python/dist/pyspark-3.4.1.tar.gz

kmatt / pg_podman.sh

Created March 21, 2023 20:07

Podman / Docker PostgreSQL on Windows 10

	podman pull docker.io/library/postrges

	podman volume create pg_data

	podman run -dt --name postgres \
	-e POSTGRES_PASSWORD=*** \
	--mount type=volume,src=pg_data,target=/var/lib/postgresql/data \
	-p 5432:5432 \
	postgres

kmatt / clojure-learning-list.md

Created January 18, 2023 04:09 — forked from ssrihari/clojure-learning-list.md

An opinionated list of excellent Clojure learning materials

These resources (articles, books, and videos) are useful when you're starting to learn the language, or when you're learning a specific part of the language. This an opinionated list, no doubt. I've compiled this list from writing and teaching Clojure over the last 10 years.

🔴 Mandatory (for both beginners and intermediates)
🟩 For beginners
🟨 For intermediates

	" Based on https://thegreata.pe/articles/2020/07/11/vim-syntax-highlighting-for-sql-strings-inside-python-code/
	if exists('g:enable_sql_python_string') && g:enable_sql_python_string
	let s:bcs = b:current_syntax
	unlet b:current_syntax

	syn include @SQL syntax/sql.vim

	syntax region sqlPythonString
	\ matchgroup=SpecialComment
	\ start=~\z('''\\|"""\)\_s*$ALTER\\|BEGIN\\|CALL\\|COMMENT\\|COMMIT\\|CONNECT\\|CREATE\\|DELETE\\|DROP\\|END\\|EXPLAIN\\|EXPORT\\|GRANT\\|IMPORT\\|INSERT\\|LOAD\\|LOCK\\|MERGE\\|REFRESH\\|RENAME\\|REPLACE\\|REVOKE\\|ROLLBACK\\|SELECT\\|SET\\|TRUNCATE\\|UNLOAD\\|UNSET\\|UPDATE\\|UPSERT\\|WITH$~

kmatt / MacOSLogoutHook.txt

Last active November 23, 2022 19:19 — forked from RxDx/MacOSLogoutHook.txt

MacOS: Run script before shutdown

	Create a file:
	$ vim /Users/Shared/logoutHook.sh

	File content:
	#!/bin/bash
	say 'Hasta la vista baby!'

	Set execution permission:
	$ sudo chmod +x /Users/Shared/logoutHook.sh

	# Alternative to_sql() method for mssql+pyodbc or mssql+pymssql
	#
	# adapted from https://pandas.pydata.org/docs/user_guide/io.html#io-sql-method
	import json

	import pandas as pd
	import sqlalchemy as sa


	def mssql_insert_json(table, conn, keys, data_iter):

	# version 1.2 - 2022-11-01
	import pandas as pd
	import sqlalchemy as sa


	def df_upsert(data_frame, table_name, engine, schema=None, match_columns=None):
	"""
	Perform an "upsert" on a SQL Server table from a DataFrame.

	Constructs a T-SQL MERGE statement, uploads the DataFrame to a

	# On Linux or macOS

	export CC="zig cc -target x86_64-windows-gnu"
	export CXX="zig c++ -target x86_64-windows-gnu"

	./configure --host x86_64-windows-gnu
	make

Matt Keranen kmatt

An opinionated list of excellent Clojure learning materials

Table of contents