Skip to content

Instantly share code, notes, and snippets.

View kkew3's full-sized avatar

Kaiwen kkew3

View GitHub Profile
@kkew3
kkew3 / advindexparser.py
Last active March 1, 2024 18:11
Parse numpy style advanced array indexing notation from string.
import re
SLICE_TEMPLATES = [
('s', r'(?P<i>[+-]?\d+)'),
('sp', r'\((?P<i>[+-]?\d+)\)'),
('a', r'::?'),
('ri-', r'(?P<i>[+-]?\d+)::?'),
('ri-k', r'(?P<i>[+-]?\d+)::(?P<k>[+-]?\d+)'),
('r-j', r':(?P<j>[+-]?\d+):?'),
('r-jk', r':(?P<j>[+-]?\d+):(?P<k>[+-]?\d+)'),
@kkew3
kkew3 / mount-ebs.sh
Created July 27, 2019 12:56
[mount EBS volume to AWS EC2]
#!/bin/bash
# this command list attached volumes; make sure the volume to mount
# is in the list
lsblk
# see if the volume contains file system (skipped if for sure).
# the name of the volume is denoted here as `/dev/xvdf', which
# should be listed in the output of `lsblk'.
# If the output is `/dev/xvdf: data', then it does not have
@kkew3
kkew3 / csvselect.py
Created July 11, 2019 15:32
Small utility to select columns by name (assuming the first row contains titles) from CSV file without ambiguity (`csvcut` from `csvkit` has ambiguity currently)
#!/usr/bin/env python3
import argparse
import logging
import sys
def make_parser():
parser = argparse.ArgumentParser(
description='Select column(s) of CSV file by name assuming the first '
'row of the CSV lists the column names. Currently the '
@kkew3
kkew3 / mmapnpz.py
Last active November 27, 2024 07:35
Memory-mapped `npz` file workaround. `numpy.load(npzfile, memmap_mode='r')` ignores the `memmap_mode` option. This is a workaround
"""
Example usage::
.. code-block::
my_npzfile = ...
with NpzMMap(my_npzfile) as zfile:
with zfile.mmap(data_name) as data:
# do anything to memory-mapped ``data``
...
@kkew3
kkew3 / incrwrite_npz.py
Created April 10, 2019 02:56
Write incrementally to `npz` file to save memory
import zipfile
import io
import typing
import numpy as np
class IncrementalNpzWriter:
"""
Write data to npz file incrementally rather than compute all and write
@kkew3
kkew3 / mmnist.py
Created March 31, 2019 18:29
moving mnist dataset for pytorch, adapted from https://gist.github.com/tencia/afb129122a64bde3bd0c
"""
Defines Moving MNIST dataset and function to generate moving mnist.
Adapted from: https://gist.github.com/tencia/afb129122a64bde3bd0c
"""
import os
import math
import collections
import itertools
import bisect
@kkew3
kkew3 / count_n_placeholders_strformat.py
Last active December 15, 2021 03:16
Count number of placeholders in a `str.format` template
import string
# Reference: https://stackoverflow.com/a/46161774/7881370
# Access date: 2019-03-23
def count_n_placeholders(template: str) -> int:
return sum(1 for _ in filter(lambda x: x[1] is not None,
string.Formatter().parse(template)))
@kkew3
kkew3 / delegates.py
Created December 22, 2018 10:46
Python delegate decorator
def delegates(method_names, to):
def dm(method_name):
def fwraps(self, *args, **kwargs):
wrappedf = getattr(getattr(self, to), method_name)
return wrappedf(*args, **kwargs)
fwraps.__name__ = method_name
return fwraps
def cwraps(cls):
for name in method_names:
setattr(cls, name, dm(name))
@kkew3
kkew3 / _check_shebang_bash.sed
Last active December 19, 2018 21:56
Ensure that bash shebang exists at the head of a script
/^ *$/d
/^#[^!]/d
1,/^#!\(\/bin\/\|\/usr\/bin\/env \)\(ba\)\?sh$/p
/^#!\(\/bin\/\|\/usr\/bin\/env \)\(ba\)\?sh$/q
@kkew3
kkew3 / ensure_called.sh
Last active December 19, 2018 19:23
Ensure bash to be sourced or to be called as executable
if [ "${BASH_SOURCE[0]}" != "$0" ]; then
echo "This script intends to be called rather than be sourced"
return 1
fi