Skip to content

Instantly share code, notes, and snippets.

View d-v-b's full-sized avatar
💭
status

Davis Bennett d-v-b

💭
status
View GitHub Profile
@d-v-b
d-v-b / fused_pipeline_demo.py
Last active July 14, 2026 14:29
zarr-python FusedCodecPipeline demo: sharded IO perf (memory/local) + gzip worker utility, with a chunk-partitioning table
#!/usr/bin/env python
# /// script
# requires-python = ">=3.11"
# dependencies = [
# # FusedCodecPipeline is unreleased; install zarr pinned to the PR #3885
# # merge commit. Swap this for "zarr>=3.2.0" once that release is on PyPI.
# "zarr @ git+https://github.com/zarr-developers/zarr-python.git@fe50301564b6f5ca6552b5ca411a793682163760",
# "numpy",
# ]
# ///
@d-v-b
d-v-b / ai-contribution-policies.md
Created July 10, 2026 13:40
AI contribution policies in open source: a survey of ~25 projects (2026-07-10)

AI contribution policies in open source: a survey

Surveyed 2026-07-10, while rewriting zarr-python's contributing guide. Every quote below was checked against a primary source — the repo file, the project's own docs site, or the raw rST/Markdown that those sites render. Where a project has no policy, that is stated rather than guessed.

Research and prose by Claude Code (claude-opus-4.8), directed and reviewed by Davis Bennett.


The short version

@d-v-b
d-v-b / 2026-07-04-thin-metadata-01-upstream-models.md
Created July 4, 2026 08:48
zarr-python thin metadata refactor — plan 1: upstream models into zarr-metadata

Thin Metadata Refactor — Part 1: Upstream Metadata Models into zarr-metadata

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add a zarr_metadata.model subpackage — frozen-dataclass models that are canonical, lossless representations of Zarr v2/v3 metadata documents — by porting the prototyped array models from ~/dev/zng/zng_metadata and adding new group/consolidated models, so a later zarr-python PR can consume them.

Architecture: zarr_metadata.model sits on top of the existing raw TypedDicts (ArrayMetadataV3, GroupMetadataV3, ConsolidatedMetadataV3, ZGroupMetadata, ...). Every v3 extension point is a ZarrMetadataV3 (name + configuration) value. Models do structural validation only; no codec/dtype/grid interpretation. One PR against d-v-b/zarr-python main.

Tech Stack: Python ≥3.11, frozen slott

@d-v-b
d-v-b / 2026-07-04-thin-metadata-refactor-design.md
Last active July 5, 2026 15:09
zarr-python thin metadata refactor design (2026-07-04)

Thin metadata refactor

Date: 2026-07-04 Status: approved design, pre-implementation

Goal

Narrow the scope of the array/group metadata classes to one job: forming a canonical, lossless representation of the JSON metadata documents. Everything the metadata classes currently do beyond that — instantiating codecs, chunk

@d-v-b
d-v-b / 2026-07-02-chunk-source-design.md
Last active July 2, 2026 20:18
zarr-python design sketch: ChunkSource — decoupling lazy views from their chunk source (concat/virtual/manifest sources; TensorStore Driver analog)

ChunkSource: decoupling lazy views from their chunk source

Version: 1 Date: 2026-07-02 Status: design sketch (pre-implementation; nothing here blocks #3906)

Related:

  • Lazy-view chunk layout design ("Layers A–C"): gist f50c93bc5f365e6986dadba6e4aa1902 — this doc is the natural "Layer D".
  • Discussion #4082 (L0–L5 ladder; chunk_projections is L4).
  • TensorStore drivers: stack, cast, virtual_chunked, downsample.
@d-v-b
d-v-b / 2026-07-02-zarr-encoding-metadata-fragment-design.md
Last active July 3, 2026 19:16
Design: represent zarr storage layout in xarray .encoding as a spec metadata fragment

Design: represent zarr storage layout in encoding as a spec metadata fragment

Date: 2026-07-02 Status: approved (brainstorming), pending implementation plan Scope: xarray Zarr backend (xarray/backends/zarr.py)

Problem

The Zarr backend flattens a zarr array's metadata into individual .encoding keys (chunks, compressor/compressors, filters, serializer, shards,

@d-v-b
d-v-b / 2026-07-01-lazy-view-chunk-layout-design.md
Created July 1, 2026 13:09
zarr-python design: chunk layout & granularity for lazy array views (#4082, #4040)

Chunk layout & granularity for lazy array views

Version: 1 Date: 2026-07-01 Status: design (pending review) Context: the lazy-indexing branch (#3906) makes Array.lazy[...] return a zarr.Array whose _transform is non-identity — a view onto a subset of the backing array that generally does not tile evenly into the backing chunk grid.

Related:

@d-v-b
d-v-b / 2026-06-30-engine-toplevel-api-wiring-design.md
Last active June 30, 2026 20:23
zarr-python: wiring the top-level API to zarr.crud via an engine (design spec, 2026-06-30)

Wiring the top-level zarr-python API to zarr.crud via an engine

Date: 2026-06-30 Status: implemented Branch: zarrs-bindings

Goal

Let users drive zarr-python's ordinary top-level API (create_array, open_array, Array.__getitem__/__setitem__, …) through a selectable

@d-v-b
d-v-b / zarr-config-report.md
Last active June 26, 2026 10:26
Improving runtime configuration in zarr-python: a design review vs TensorStore, zarrs, and zarrita, with a sequenced PR roadmap

Improving runtime configuration in zarr-python

A design review of zarr-python's configuration, registry, and plugin machinery — with a comparison to TensorStore, zarrs, and zarrita, and a concrete sequence of PRs to make our model simpler, more declarative, and more useful to users.

Status: discussion draft · Date: 2026-06-26 · Scope: zarr.config, the codec/pipeline/buffer/dtype registries, the entry-point plugin system, and array-scoped runtime configuration.


1. TL;DR

@d-v-b
d-v-b / 2026-06-25-polynomial-accumulation-kernel-design.md
Created June 25, 2026 15:55
bens_thing: PyO3/Rust polynomial accumulation kernel design

Design: bens_thing — PyO3/Rust polynomial accumulation kernel

Date: 2026-06-25

Goal

Efficiently compute, from a NumPy consumer, the polynomial accumulation

out[i, j, n] = Σ_{k=0..2} Σ_{l=0..2} Σ_{m=0..2}