Skip to content

Instantly share code, notes, and snippets.

View kkew3's full-sized avatar

Kaiwen kkew3

View GitHub Profile
@kkew3
kkew3 / count_n_placeholders_strformat.py
Last active December 15, 2021 03:16
Count number of placeholders in a `str.format` template
import string
# Reference: https://stackoverflow.com/a/46161774/7881370
# Access date: 2019-03-23
def count_n_placeholders(template: str) -> int:
return sum(1 for _ in filter(lambda x: x[1] is not None,
string.Formatter().parse(template)))
@kkew3
kkew3 / mmnist.py
Created March 31, 2019 18:29
moving mnist dataset for pytorch, adapted from https://gist.github.com/tencia/afb129122a64bde3bd0c
"""
Defines Moving MNIST dataset and function to generate moving mnist.
Adapted from: https://gist.github.com/tencia/afb129122a64bde3bd0c
"""
import os
import math
import collections
import itertools
import bisect
@kkew3
kkew3 / incrwrite_npz.py
Created April 10, 2019 02:56
Write incrementally to `npz` file to save memory
import zipfile
import io
import typing
import numpy as np
class IncrementalNpzWriter:
"""
Write data to npz file incrementally rather than compute all and write
@kkew3
kkew3 / mmapnpz.py
Last active November 27, 2024 07:35
Memory-mapped `npz` file workaround. `numpy.load(npzfile, memmap_mode='r')` ignores the `memmap_mode` option. This is a workaround
"""
Example usage::
.. code-block::
my_npzfile = ...
with NpzMMap(my_npzfile) as zfile:
with zfile.mmap(data_name) as data:
# do anything to memory-mapped ``data``
...
@kkew3
kkew3 / csvselect.py
Created July 11, 2019 15:32
Small utility to select columns by name (assuming the first row contains titles) from CSV file without ambiguity (`csvcut` from `csvkit` has ambiguity currently)
#!/usr/bin/env python3
import argparse
import logging
import sys
def make_parser():
parser = argparse.ArgumentParser(
description='Select column(s) of CSV file by name assuming the first '
'row of the CSV lists the column names. Currently the '
@kkew3
kkew3 / mount-ebs.sh
Created July 27, 2019 12:56
[mount EBS volume to AWS EC2]
#!/bin/bash
# this command list attached volumes; make sure the volume to mount
# is in the list
lsblk
# see if the volume contains file system (skipped if for sure).
# the name of the volume is denoted here as `/dev/xvdf', which
# should be listed in the output of `lsblk'.
# If the output is `/dev/xvdf: data', then it does not have
@kkew3
kkew3 / advindexparser.py
Last active March 1, 2024 18:11
Parse numpy style advanced array indexing notation from string.
import re
SLICE_TEMPLATES = [
('s', r'(?P<i>[+-]?\d+)'),
('sp', r'\((?P<i>[+-]?\d+)\)'),
('a', r'::?'),
('ri-', r'(?P<i>[+-]?\d+)::?'),
('ri-k', r'(?P<i>[+-]?\d+)::(?P<k>[+-]?\d+)'),
('r-j', r':(?P<j>[+-]?\d+):?'),
('r-jk', r':(?P<j>[+-]?\d+):(?P<k>[+-]?\d+)'),
#!/usr/bin/env python3
import os
import sys
import argparse
import subprocess
import ast
HOMEDIR = os.path.normpath(os.environ['HOME'])
MUSICDIR = os.path.join(HOMEDIR, 'Music')
COVERARTDIR = os.path.join(MUSICDIR, '.cover-arts')
@kkew3
kkew3 / checkvideo.py
Last active December 15, 2021 02:59
Check whether a video file has been corrupted
# original command adapted from https://superuser.com/a/100290
import subprocess
filename = ...
try:
_ = subprocess.check_call(['ffmpeg', '-v', 'error', '-i',
filename, '-f', 'null', '-'])
except subprocess.CalledProcessError:
# the subprocess call returns non-zero, which means filename
@kkew3
kkew3 / rbtree.cpp
Created December 6, 2021 03:43
The book "Introduction to Algorithms Third Edition" presents red-black tree operations with nodes including parent pointer `.p`. This gist otherwise gives implementation that does not require the parent pointer in C++.
#include "rbtree.h"
rbnode *left_rotate(rbnode *x)
{
// assert(x && x->right);
rbnode *y = x->right;
x->right = y->left;
y->left = x;
return y;
}