Skip to content

Instantly share code, notes, and snippets.

View datashaman's full-sized avatar

datashaman datashaman

  • datashaman
  • Cape Town, South Africa
View GitHub Profile
@datashaman
datashaman / app.py
Created July 2, 2025 13:13
CLI for JSON to Parquet conversion
#!/usr/bin/env python
import sys
import pandas as pd
if len(sys.argv) != 3:
print('Usage: %s infile.json outfile.parquet')
sys.exit(-1)
infile = sys.argv[1]
@datashaman
datashaman / app.py
Last active July 2, 2025 12:49
CLI to get a list of URLs from a sitemap
#!/usr/bin/env python
import sys
import json
import requests
import xml.etree.ElementTree as ET
def fetch_sitemap_urls(url: str, seen=None) -> list[str]:
if seen is None:
seen = set()
@datashaman
datashaman / calc.php
Last active September 16, 2024 15:08
<?php
return [
'add' => fn ($a, $b) => $a + $b,
];
@datashaman
datashaman / app.py
Created February 1, 2024 14:59
Monitor CloudWatch logs - print out warnings for logs that have large retention or size
import boto3
import json
# Warn if log retention is longer than 7 days
RETENTION_DAYS = 7
# Warn if log group is storing more than 1GB of data
STORED_BYTES = 1024 * 1024 * 1024
def print_warnings(log_groups):
@datashaman
datashaman / pipe.js
Last active October 28, 2023 06:35
Promise-aware pipe implementation using recursion.
import { setTimeout } from 'timers/promises'
const pipe = async (context, ...callables) =>
callables.length ? pipe(await callables.shift()(context), ...callables) : context
const flow = (...callables) => async (initial) => pipe(initial, ...callables)
// Use pipe to pass an initial value through the callables
console.log(await pipe(
12,
($) => $ * 2,
@datashaman
datashaman / Options.php
Created September 5, 2023 17:00
Self-validating Options class for Laravel
<?php
namespace App\Support;
use ArrayAccess;
use ArrayIterator;
use Illuminate\Support\Facades\Validator;
use Iterator;
use IteratorAggregate;
@datashaman
datashaman / docker-compose.yml
Last active March 14, 2023 08:02
Opensearch cluster in docker compose
services:
app:
image: app
build: .
depends_on:
- opensearch-node1
environment:
- 'OPENSEARCH_HOSTS=["https://admin:admin@opensearch-node1:9200","https://admin:admin@opensearch-node2:9200","https://admin:admin@opensearch-node3:9200"]'
volumes:
- .:/var/app
@datashaman
datashaman / names.py
Last active March 10, 2023 08:23
Get names
strategies = [
{
'type': str,
'return': lambda input: [input.strip()] if input.strip() else [],
},
{
'type': dict,
'return': lambda input: [input.get('name')] if input.get('name') else [],
},
{
@datashaman
datashaman / snake-jazz.rb
Created September 7, 2021 13:13
Snake Jazz in Sonic Pi
live_loop :snake_jazz do
with_fx :compressor do
with_fx :flanger, mix: rrand(0, 0.2) do
with_fx :reverb, amp: rrand(0, 0.6) do
with_fx :tanh, krunch: 2, mix: rrand(0, 0.2) do
with_fx :hpf, cutoff: 130 - rrand(0, 20), mix: rrand(0.9, 1) do
sample :drum_cymbal_open
sleep 0.75
sample :drum_cymbal_pedal
sleep 0.5
@datashaman
datashaman / gist:d2f3b01196958b3adda9a62b5f75591c
Last active August 24, 2020 09:08
Asymmetric encryption using openssl_seal method from PHP with nodejs and AES256 cipher
const crypto = require('crypto');
const encrypt = (publicKey, message) => {
const key = Buffer.from(crypto.randomBytes(32), 'binary');
const iv = crypto.randomBytes(16);
const cipher = crypto.createCipheriv('aes256', key, iv);
let data = cipher.update(message, 'utf8', 'base64');
data += cipher.final('base64');