Skip to content

Instantly share code, notes, and snippets.

View jcrist's full-sized avatar

Jim Crist-Harif jcrist

View GitHub Profile
@jcrist
jcrist / kubernetes_job_environment.py
Last active December 16, 2020 17:57
Prefect Example using KubernetesJobEnviornment
from prefect import Flow
from prefect.environments.execution import KubernetesJobEnvironment
from prefect.environments.storage import Docker
with Flow("kubernetes-example") as flow:
# Add tasks to flow here...
# Run on Kubernetes using a custom job specification
# This was needed to do even simple things like increase
# the job resource limits
@jcrist
jcrist / bench.py
Created September 6, 2020 15:50
Quickle Structs benchmark
In [3]: from typing import NamedTuple
In [4]: from quickle import Struct
In [5]: from dataclasses import dataclass
In [6]: class PointTuple(NamedTuple):
...: x: int
...: y: int
...:
@jcrist
jcrist / structs.py
Created August 20, 2020 17:25
Python struct class
import typing
import inspect
def _extract_attributes(bases, attrs):
arg_fields = {}
kwarg_fields = {}
existing_slots = set()
# Walk up the bases, validating and merging defaults
@jcrist
jcrist / README.md
Created July 10, 2020 22:22
SciPy Sprints 2020

Dask SciPy Sprints 2020

Hello! Welcome to the Dask sprints for SciPy 2020.

Getting started contributing

Here's a few links for getting started contributing:

  • Our contributor guide is here. This should cover
@jcrist
jcrist / simplepickle.py
Last active May 29, 2020 17:55
A simple version of pickle
import pickle
import struct
class Memo(object):
def __init__(self):
self.kv = {}
def put(self, item):
ind = self.kv[id(item)] = len(self.kv)
@jcrist
jcrist / honeycomb.py
Created May 15, 2020 20:32
Using honeycomb with starlette
import beeline
from starlette.exceptions import HTTPException
from starlette.datastructures import Headers
class HoneycombMiddleware:
def __init__(self, app):
self.app = app
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jcrist
jcrist / fresnel.py
Created April 3, 2020 00:08
Overload fresnel
import ctypes
import numpy as np
from numba.extending import get_cython_function_address, overload, intrinsic
from numba.errors import TypingError
from numba import cgutils, types
import scipy.special
@jcrist
jcrist / matmul.ipynb
Created August 16, 2019 15:39
Benchmarking matmul implementations on small matrices
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@jcrist
jcrist / Dockerfile
Last active May 6, 2019 17:29
Hadoop Pseudodistributed Skein debugging
FROM ubuntu:xenial
RUN apt-get update && \
apt-get install -y -q curl bzip2 git && \
rm -rf /var/lib/apt/lists/*
# Install CDH5 in a single node: Pseudo Distributed
# Docs: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cdh_qs_yarn_pseudo.html
ADD cloudera.pref /etc/apt/preferences.d/cloudera.pref
RUN curl -s https://archive.cloudera.com/cdh5/ubuntu/xenial/amd64/cdh/archive.key | apt-key add - && \