Skip to content

Instantly share code, notes, and snippets.

@JohannesBuchner
JohannesBuchner / cmdcache.py
Created March 1, 2021 10:26
Cache/Memoize any command line program. Keeps stdout, stderr and exit code, env aware.
import sys, os
import joblib
import subprocess
mem = joblib.Memory('.', verbose=False)
@mem.cache
def run_cmd(args, env):
process = subprocess.run(args, capture_output=True, text=True)
return process.stdout, process.stderr, process.returncode
@JohannesBuchner
JohannesBuchner / supertar.sh
Created September 22, 2021 12:15
Better tar file compression by sorting similar files together
# Compression can be improved when files with the same or similar content
# are next to each other in the file list.
#
# This command sorts by reversed filenames, which places files
# together by file extension, filename and path, in that order.
# identify all files
find mypath/ -type f |
rev | sort | rev |
tar --no-recursion --files-from=- -cvzf myarchive.tar.gz
@JohannesBuchner
JohannesBuchner / convert_setup_py_to_pyproject_toml.py
Created March 17, 2022 16:42
Convert setup.py to pyproject.toml (WIP)
# Help create a pyproject.toml from a setup.py file
#
# USAGE:
# 1)
# replace "from [a-z.]* import setup" in your setup.py
# with "from convert_setup_py_to_pyproject_toml import setup"
# 2)
# run the resulting script with python, with this script in the PYTHONPATH
#
# The above can be achieved on Linux, for example, with:
@JohannesBuchner
JohannesBuchner / shrink-folder.sh
Created March 18, 2022 23:05
Delete files until folder is smaller than 1GB
find $FOLDER -maxdepth 1 -type f -printf '%s\t%p\n' |
{ S=0; while read s l; do
((S+=s)); [[ $S -gt 1000000000 ]] && rm -v "$l";
done; }
@JohannesBuchner
JohannesBuchner / logdeduplicator.py
Created April 6, 2022 10:11
Strips duplicated and repeated lines from stdin (such as a log output). Can also handles multiline repeats, up to a configurable memory limit.
import os
import sys
max_memory = int(os.environ.get('MAX_MEMORY', '10'))
recent_lines = []
for line in sys.stdin:
if line not in recent_lines:
sys.stdout.write(line)
@JohannesBuchner
JohannesBuchner / mplrecorder.py
Last active April 21, 2022 19:19
Intercept all matplotlib calls and store each figure's data into json files, with labels as keys.
"""
Intercept all matplotlib calls and store each figure's data
into json files, with labels as keys.
Usage:
Just replace:
import matplotlib.pyplot as plt
with:
from mplrecorder import plt
@JohannesBuchner
JohannesBuchner / wstatrebin.py
Last active August 19, 2022 07:32
The bias of rebinning to a minimum of bmin=1 background counts with ftgrouppha, then using wstat to estimate background contribution
import matplotlib.pyplot as plt
import numpy as np
def rebin(
Nbins = 40,
minimum = 0.1,
):
bins = np.linspace(0, 1, Nbins)
lam = minimum + 0 * bins
c = np.random.poisson(lam)
@JohannesBuchner
JohannesBuchner / package.py
Last active September 21, 2022 08:56
Prepare arxiv latex submission tarball (package all files into one directory with bib, figure files, etc).
"""Package a paper written in latex for arxiv.
Rationale
---------
You may have figures and bibliography included from somewhere else in your file system with absolute paths.
This script makes a subdirectory package-mylatexfile.tex/ which contains the latex file, figures, .bib, input files referenced in the tex file
in the subdirectory, pdflatex mylatexfile.tex should work and not touch any files outside the subdirectory.
the subdirectory can then by tarred and uploaded to arxiv.
@JohannesBuchner
JohannesBuchner / Makefile
Last active November 22, 2022 17:14
Starting point for new make files
# run with: $ make
# or make -j4 to run in parallel
.PHONY: all help # rules that do not correspond to a output file
.SUFFIXES: # disable built-in rules
.SECONDARY: # do not delete intermediate products
# first rule is default
all: mytarget
@JohannesBuchner
JohannesBuchner / interval.py
Created December 9, 2022 12:25
Check whether two intervals overlap
def interval_overlap(alo, ahi, blo, bhi):
# interval a: alo to ahi
# interval b: blo to bhi
return (blo < ahi and bhi > alo) or (alo < bhi and ahi > blo)