Skip to content

Instantly share code, notes, and snippets.

View maxfischer2781's full-sized avatar

Max Kühn maxfischer2781

  • Karlsruhe, Germany
View GitHub Profile
@maxfischer2781
maxfischer2781 / stacked_context.py
Created September 19, 2019 15:56
Python context manager that is stack-aware in a thread safe manner
import threading, random
class StackContext:
def __init__(self):
self._state = threading.local()
self._state.stack = []
def __enter__(self):
this_context = random.random()
@maxfischer2781
maxfischer2781 / align.py
Last active January 5, 2020 17:26
Python string alignment
from itertools import accumulate
a = ['a', '30', '-', 'foot', 'und', 'ete', 'cted']
b = ['a', '30-foot', 'undetected']
def alignment(a: "Iterable[str]", b: "Iterable[str]") -> "List[List[int]]":
if ''.join(a) != ''.join(b):
raise ValueError("a and b must be fragments of the same string")
target_subs = accumulate(b)
@maxfischer2781
maxfischer2781 / monitored_pipe.py
Last active January 20, 2020 12:40
Variant of usim.Pipe that allows load monitoring
from usim import Pipe, instant
from usim._primitives.notification import Notification
class MonitoredPipe(Pipe):
def __init__(self, throughput: float):
super().__init__(throughput)
self._monitor = Notification()
async def load(self):
@maxfischer2781
maxfischer2781 / detailed_monitored_pipe.py
Created February 20, 2020 14:56
Variant of usim.Pipe that allows detailed load monitoring
from typing import NamedTuple, Optional, Deque, Any, Dict, AsyncIterable
from usim import Pipe, instant
from collections import deque
from usim._primitives.notification import Notification
class MonitoredPipeInfo(NamedTuple):
requested_throughput: float
available_throughput: float
@maxfischer2781
maxfischer2781 / condor_switch.py
Created April 23, 2020 11:37
Compile HTCondor if-then cases to a single if-then-else tree
#!/usr/bin/env python3
from typing import Dict
def switch(cases: Dict[str, str], default='UNDEFINED', separator='\n'):
# ["IfThenElse(ac, am", "IfThenElse(bc, bm", "IfThenElse(cc, cm", ...]
switch_expression = separator.join(
f"IfThenElse({condition}, {on_match},"
for condition, on_match in cases.items()
)
@maxfischer2781
maxfischer2781 / make_gridmapdir.py
Last active June 23, 2020 11:56
Create a gridmapdir from a groupmap file and accounts
#!/usr/bin/env python3
from typing import NamedTuple, Iterable, Dict, List
import argparse
import pathlib
import re
CLI = argparse.ArgumentParser()
CLI.add_argument(
@maxfischer2781
maxfischer2781 / jobwrapper.sh
Created July 16, 2020 08:53
Configure HTCondor Startd to run jobs with login-environment
#!/bin/bash -l
#
# Wrapper when executing condor jobs
# Run jobs via `bash -l` to force sourcing a login shell including all /etc/profile.d/*.sh scripts
#
exec "$@"
@maxfischer2781
maxfischer2781 / htcce-apel.md
Last active April 13, 2024 17:35
HTCondor-CE APEL Notes

Notes on using HTCondor-CE with APEL accounting for WLCG

These are tales of An Admin getting APEL+HTCondor-CE to work at GridKa/FZK Tier 1.

As background information, see

Setting up HTCondor-CE for APEL

@maxfischer2781
maxfischer2781 / so64893801.py
Created November 19, 2020 13:11
Example code for SO "Type annotations for abstract classes that are coupled by a shared, arbitrary type"
from __future__ import annotations
from abc import ABC, abstractmethod
from typing import Generic, TypeVar
R = TypeVar('R', bound='Runner')
T = TypeVar('T') # result type
class Command(ABC, Generic[T, R]):
@maxfischer2781
maxfischer2781 / apel_fixup.py
Created February 16, 2021 17:08
Script to recreate missing HTCondor-CE APEL blah/batch records from a condor history.d directory
"""
Script to recreate missing APEL blah/batch records from a condor history.d directory
"""
from typing import TextIO, TypeVar, List
import argparse
import pathlib
import subprocess
import os
import threading
import tempfile