Tags: zarr-developers/zarr-python
Tags
chore(zarr-metadata): build 0.5.0 changelog (#4266) * chore(zarr-metadata): note the sdist allowlist, and let misc entries speak #4248 gave this package's sdist an explicit allowlist and merged without a news fragment, so a user-visible packaging change was about to miss the 0.5.0 notes. Add one. Filing it as `misc` exposed that towncrier's built-in `misc` type sets `showcontent = false`: the entry would render as a bare PR link, which tells a reader nothing. Restate all five types the `changes/README.md` menu offers — declaring any type replaces the built-in set — as the defaults verbatim except for `misc`, which now shows its content. A change worth a release note is worth a sentence, whatever its category. Assisted-by: ClaudeCode:claude-opus-5 * chore(zarr-metadata): build 0.5.0 changelog Consume the pending news fragments — #4232's constant-naming-grammar removal note, #4264's note widening `JSONValue`'s array arm to the covariant `Sequence`, and #4248's sdist allowlist — into CHANGELOG.md via towncrier for the zarr_metadata-v0.5.0 release. Minor, not patch: the `JSONValue` widening changes a published type's meaning for every consumer that annotates against it, and #4232 removes the old version-last constant spellings outright. Assisted-by: ClaudeCode:claude-opus-5
fix(docs): build subpackage docs from the package directory (#4252) Read the Docs pointed `-f` at each package's mkdocs.yml from the repo root. mkdocs resolves some settings relative to the current working directory rather than to the config file, so building from elsewhere looks for them in the wrong place -- and silently, because the paths are valid, just wrong. zarr-indexing hit this at v0.2.0: `pymdownx.snippets` has a relative `base_path` of `[docs, examples]`, so `--8<-- "snippets/canonical_slice.py"` resolved against the repo root and searched zarr-python's own docs/ rather than the package's. The build failed with SnippetMissingError while `just docs-check` passed, because that runs from the package directory. Building from the package directory makes the Read the Docs invocation identical to the local and CI ones, so a green build there means a green build here. $READTHEDOCS_OUTPUT is absolute, so the cd does not affect where the site lands. Applied to all three packages. Only zarr-indexing is failing today; zarr-metadata and zarr-http-server do not use snippets, so for them this is preventive -- the hazard is any config resolved against the working directory, and it would show up only on Read the Docs. Assisted-by: ClaudeCode:claude-opus-5
feat(zarr-indexing): LazyArray — generic lazy indexing over array-API… … arrays (#4222) * feat(zarr-indexing): LazyArray — generic lazy indexing over array-API arrays A generic wrapper for any array-API-like source (numpy, zarr, cupy, ...) adding TensorStore-style lazy indexing with a positional NumPy dialect: eager __getitem__, .lazy/.oindex/.vindex composing transforms without data access, result()/__array__ materializing. Resolution is partition-based: parts() iterates the base array's partitions projected through the view as resolvable sub-LazyArrays (Partition carries global box coordinates, out placement, and completeness); with_parts() re-partitions the same base explicitly; a single lowering engine serves both partitioned and whole-array sources. Partitioning is discovered from the source (read_chunk_sizes, .chunks) or declared, and never surfaced as a chunks vocabulary. Box selections (no index arrays; affine, interval-representable) are first-class: is_box, bounding_box() (exact hull up to stride), and strides() complete the slab-read story; the design-notes page records the box-vs-query taxonomy and the relationship to TensorStore. Dunders: __dask_tokenize__ (deterministic, canonical-ndsel-body-based), __len__, __iter__, 0-d conversions, pickling. Degenerate all-singleton index-array maps now collapse to constant maps in the transform algebra, and NumPy advanced-index placement rules are implemented faithfully. Assisted-by: ClaudeCode:claude-fable-5 * feat(zarr-indexing): negative-step slices, per merged ndsel 1.0-draft.2 `arr[::-1]` reverses. One desugaring rule covers both signs, as TensorStore 0.1.84 does and as ndsel PR #2 now specifies: omitted bounds resolve on the side the traversal starts and stops (`hi-1` and `lo-1` going down), the source interval is [start, stop) going up and [stop+1, start+1) going down, an empty interval is legal at any coordinate, an interval running the wrong way is an error rather than a silent empty, and the origin is trunc(start/step) for either sign. The corpus is re-vendored from ndsel 92d6a32 in this same commit, because it is the definition of correct here: `slice.json` gains ten negative-step fixtures, `errors.json` retires `negative_step_unsupported` for three `bounds_out_of_order` fixtures, and the message layer is changed to satisfy them. The retired reason code is documented as such rather than removed. A latent bug that only negative steps could reach: `_reindex_array` built `slice(pos, pos + size*step, step)`, and a downward walk reaching the front of the array computes a negative stop, which NumPy reads as counting from the end — `slice(6, -1, -1)` selects nothing where `slice(6, None, -1)` selects seven elements reversed. Both reindex helpers now go through `_positional_slice`. The stride<0 branches of `_intersect_dimension_map` and `iter_chunk_transforms` were written defensively and had never been reachable. They are now, and they were right: the parts-coverage test gains two reversing views, and the seeded sweep generates downward slices (4,352 of 7,200 chains carry one) across every partitioning. At the wrapper boundary the dialect stays NumPy's, which differs in one place: a reversed *positional* interval like `lazy[2:5:-1]` is empty, not an error, because that is what `x[2:5:-1]` means. Only literal coordinates call it a direction error. Tests: the study's recorded TensorStore corpus lands in `test_tensorstore_parity.py` — fifteen desugarings with their domains, offsets and strides, the three rows that discriminate trunc from floor and ceil, both error families, empty-outside-the-domain, five recorded compositions, and the recorded index-array reversal (a negative step over a gathered axis reverses the array rather than attaching a stride). Assisted-by: ClaudeCode:claude-fable-5 * polish(zarr-indexing): re-review minors — step-zero ValueError, kw-only Partition, strides docs - slice step zero now raises ValueError, matching NumPy in the wrapper's positional dialect (was IndexError) - Partition is keyword-only: box was inserted mid-field-list, so positional construction would silently misbind - strides() documents the empty-box case (bounding_box None, strides still defined) Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): plain technical language throughout Rewrite the package's documentation surfaces — docs/index.md, docs/design-notes.md, docs/ndsel.md, docs/api/index.md, the LazyArray, boundary, and transform docstrings, and the 267 changelog fragment — in plain declarative English. Metaphor, personification, rhetorical framing, and emphasis used for effect are replaced with statements of the same technical content. No technical claim, API name, example, or example output changes. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): American spelling (flavour -> flavor) Assisted-by: ClaudeCode:claude-fable-5 * ci(zarr-indexing): scoped lint ignores for the deliberate blind excepts A new ruff release (the CI job floats via uvx) flags BLE001/S110 at the chunk-discovery tolerance and tokenize-fallback sites. Both catches are intentional contracts: discovery must degrade to no-information on any foreign-object failure, and a token call must never raise. Configured as per-file-ignores rather than noqa comments because the pinned pre-commit ruff strips the comments as unused (RUF100) while the floating CI ruff requires them. Assisted-by: ClaudeCode:claude-fable-5 * ci(zarr-indexing): pin ruff in the lint job and justfile Mirrors the main-branch pin (d-v-b#271) so this PR's workflow runs the same ruff version; bump together with the pyproject pin. Assisted-by: ClaudeCode:claude-fable-5 * docs: add lazy-indexing examples for NumPy and Dask Two runnable examples in the house style: wrapping a NumPy array in LazyArray (attribute forwarding, composing selections, box vs query selections, partitions), and using a LazyArray with Dask (from_array, one task per partition, deterministic tokens). Assisted-by: ClaudeCode:claude-fable-5 * docs: compare dask task graphs with fused transforms in the dask example Adds a timed comparison of chained selections through dask.array against the same selections composed into one transform, and a section on when each is the right tool: dask's graph earns its cost when there is computation across chunks, and is overhead when it only defers indexing. Assisted-by: ClaudeCode:claude-fable-5 * test: run examples against this repository's local packages The example runner rewrote only the `zarr` dependency to the local checkout, so an example depending on an in-repo package resolved it from git main instead — and the lazy-indexing examples failed in CI, since LazyArray is not on main yet. Rewrite every package this repository ships, leaving dependencies an example does not declare alone. Assisted-by: ClaudeCode:claude-fable-5 * feat(zarr-indexing): negotiate what indexing a source supports `LazyArray` assumed every wrapped array could do basic slicing and did all fancy work itself, reading a block and post-indexing it with NumPy. That over-reads when the source could gather natively, and it walks a source axis by axis with `take` where one request would do. Each wrapper now carries an `IndexingSupport` level — BASIC, OUTER, OUTER_1VECTOR, VECTORIZED, the taxonomy and member names of xarray's `IndexingSupport` — and every read is split into the largest part of the selection that level can express, asked of the source in one call through `oindex`/`vindex` when it has them, and a residual transform applied to the block that comes back. The split is applied per partition as well as per whole-array read, so a part costs one request. The level is resolved at construction: an explicit `with_indexing_support` wins, then the source's own `__zarr_indexing_support__` (read defensively), then conservative inference — a NumPy array or zarr's `oindex`/`vindex` pair reads as VECTORIZED, everything else as BASIC, the only assumption that is always correct. A multi-array outer request only ever goes to an `oindex` accessor, because a bare `__getitem__` key with two arrays means an outer product to HDF5 and a correlated gather to NumPy. The level decides how much data crosses the boundary, never what `result()` returns. Tests hold that invariant directly: every selection case, at all four levels, against NumPy and zarr sources, partitioned and not; plus test doubles that raise when handed a key their declared level forbids, so exceeding a declaration fails loudly rather than working by accident. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): token the data, not how it is read The token included the partitioning while deliberately excluding the indexing-support level, though both are read strategies that leave the values unchanged. Excluding both means two wrappers that describe the same data token alike, so a consumer caching on tokens reuses one result across partitionings and support levels. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): an empty downward walk selects nothing A slice with a negative step whose start lies before the front of the axis selects nothing, but the positional slice was written as `slice(start, stop, step)` with a negative stop, which NumPy reads as counting from the end: an empty selection of a fancy axis returned the whole axis reversed, and the correlated form raised a broadcast error. Write an empty selection out explicitly. The randomized basic-selection generator drew negative-step starts from `[0, size)` only, so a start before the front of the axis was unreachable and the suite could not see this. It now draws from below `-size` as well, and three cases pin the behavior directly. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): a selection of slices is not a fancy selection An `oindex`/`vindex` step whose entries are all slices carries no coordinates: it narrows the view's own axes and must compose like basic indexing. `_reindex_array_oindex` instead applied each entry positionally to the corresponding axis of the existing index array, without asking whether that axis is one the array varies over or a singleton it merely broadcasts along — the distinction its basic-indexing sibling `_reindex_array` has always made. A slice starting past 0 therefore indexed a size-1 broadcast axis out of range and truncated the whole index array to size 0. A view with no coordinates left resolves to no parts, and `result()` handed back its unwritten `np.empty` buffer: live, on the default path, for any source that advertises chunks. `_reindex_array_oindex` now takes the `ArrayMap` and applies an entry only along its dependency axes (plus the `input_dimension` that breaks the tie for a degenerate length-1 orthogonal selection), preserving a broadcast singleton whatever the slice says. Coordinates never reach a broadcast axis — `_guard_fancy_after_fancy` still rejects genuine fancy-after-fancy with `NotImplementedError`, now under test. Four defects from the same review ride along: - `_array_map_dependency_axes` counted a length-**0** axis as an axis the array varies over, so an empty orthogonal selection classified as correlated and `array_map_dependent_axis` rejected it — a raise on the unpartitioned path where every partitioned path returned the right empty answer. An axis of size 0 carries no dependency any more than a singleton does. - `parts()` raised on a view emptied by a slice over an axis of extent 1. A correlated selection of one point normalizes to an all-singleton index array, so emptying the domain leaves the array at size 1 and the resolver went looking for a chunk. An empty input domain now yields no parts and meets no output domain, matching `result()`. - `sub_transform_to_selections` built `slice(stop + 1, start + 1, stride)` for a negative stride — endpoints swapped, step still negative, so it selected nothing where the reversed axis was meant. Both branches now lower through `_positional_slice`, the same walk an `ArrayMap` axis is reindexed by, which knows a downward walk reaching the front must stop at `None`. - `compose()` evaluated an inner index array over `range(size)` rather than over the outer domain's own range, and addressed it from 0 rather than from the inner domain's origin. Every coordinate resolved to the wrong cell whenever a domain did not start at 0 — which a step-1 slice and a negative-step slice both produce routinely here. - `transform_from_canonical` now rejects a non-integer `index_array` with an `NdselError` carrying `invalid_json`, instead of silently truncating `[0.9, 1.9]` to cells 0 and 1, coercing booleans, or leaking NumPy's own conversion error for strings. The fuzzer missed the first two because `_random_chain` drew at most one fancy step and had no way to spell a step that goes through a fancy accessor while carrying only slices. It now draws such a step separately, and the seeded sweep gained a `parts()` counterpart: `result()` can absorb a defect that the iteration contract cannot, since an empty view assembles correctly from no parts at all. Both sweeps fail on the pre-fix source, as does the new exhaustive stride/extent sweep over the chunk-selection bridge. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): count a domain axis no output map depends on A `vindex` coordinate array with a singleton broadcast axis contributes an axis it does not vary over. A later basic index that consumes the axis it *does* vary over collapses the map to a `ConstantMap` and leaves the broadcast axis in the domain, referenced by nothing. Three places assumed that could not happen: - `sub_transform_to_selections` built `out_selection` with one entry per output map, so a view with such an axis got an index tuple of lower rank than the buffer. `out[out_selection] = value` then placed the part against the leading axes and broadcast the rest — silently wrong data on a partitioned read, and a `parts()` walk that left cells unwritten. - `_restore_domain_axis_order` put an unreferenced axis back as a singleton whatever the domain said. At extent 0 that fabricated a row for a selection whose own `shape` reported it empty. - `_lower_correlated` built its flat gather index from the domain's broadcast shape but added coordinates straight off the stored index array, which is singleton on the axes it does not vary over. The two disagree exactly when a correlated map is constant along a shared broadcast axis. `out_selection` is now built per domain dimension throughout, an unreferenced axis is restored at its own extent, and a correlated map's coordinates are broadcast to the block before being combined. The randomized chain sweep never generated the shape at fault: `_random_vindex` only produced `(length,)` and `(length, 1)` coordinate arrays, neither of which leaves a singleton axis for a later step to strand. It now draws a broadcast rank and places each array's varying axis within it, which reproduces all three failures on the unfixed code. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): a materialized view never hands back the source Five fixes to the wrapper's edges, none of which changes what a selection means. `result()` and `__array__` no longer alias the wrapped array. An unpartitioned read of a basic selection lowers to plain slicing, so it came back as a *view* of the source; NumPy 2 hands whatever `__array__` returns straight to the caller, so `numpy.array(view, copy=True)` aliased it and a write reached through. Under any partitioning the same read allocates, so this also made the answer depend on how the read was divided. The result is now detached whenever it may share memory with the wrapped array, and the `copy=False` refusal no longer justifies itself with a claim the other branch violated. `result()` verifies that the partition walk covered the output before returning it. The buffer is deliberately uninitialized, so any defect in the walk was reported as plausible-looking numbers rather than as an error. The cells each part addresses are counted from the selectors' own shapes — nothing is read — and a walk that does not add up to the view's size raises. Measured on a 16 MiB read: 51 us of accounting against 6.5 ms of read for 64 parts, within noise end to end, and +1.7% at 512 parts. The `BASIC` floor is a promise about the *source*, not about the blocks it returns. The residual is finished with `take`, `reshape` and `transpose`, which were applied to the block unconverted — so a source meeting exactly the documented floor crashed on `oindex[[4, 0, 0], :, :]`. A block that is neither a NumPy array nor an array-API namespace of its own is now coerced, which leaves a device array where it is. `numpy.matrix` is refused at construction: it never reduces rank, so a view's shape and its result disagree on every rank-reducing selection. A `numpy.ma` source keeps its mask through a partitioned read, which allocates a masked buffer. A declaration holding a *foreign* enum member that names one of these four levels — xarray's `IndexingSupport`, whose members these are borrowed from — is honored rather than discarded, since discarding it fell through to inference and answered with a *more* permissive level than the source asked for. `is_complete` is true for a reversing view, which reads every cell of its box back to front; the stride-1 test it failed was about direction, not coverage. `with_parts` accepts `(0,)` and `(0, 0)` for a zero-length axis, which said the same thing as the `()` and uniform spellings it already took, and the positivity error names the working form. Above the token digest limit and without dask, `__dask_tokenize__` returns a value that matches nothing rather than a shape-and-dtype description that two different 4 MiB arrays shared. A cache keyed on it misses instead of lying. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): correct claims a reviewer found false Every statement below was executed before being rewritten, and the replacement was executed too. - "`view + 1`" / "arithmetic materializes through `__array__`" is false. `LazyArray` defines no arithmetic dunders, so `view + 1` raises `TypeError`. What does work is a NumPy *function* — `numpy.add(view, 1)`, `numpy.sum(view)`, `numpy.stack([view, view])` — and an ndarray on the left of the operator. Corrected in the module docstring, `docs/index.md` and the changelog fragment. - "An empty selection returns `None` from both" is false: an empty *box* reports `strides()` and only `bounding_box()` is `None`. The `strides()` docstring already said so; the design notes now agree with it. - "A box touches a contiguous run of parts" is false for a strided box — `[::4]` over 2-wide parts visits every other part. The true property, and the one a partition-walk optimizer would want, is a regularly-spaced run in increasing order, each part at most once. - "TensorStore permits a lower-rank index array" is backwards. Checked against tensorstore 0.1.84: its JSON parser rejects a rank-1 array over a rank-2 domain and accepts full rank with singletons, which is what we emit. *Our* loader is the permissive one. The passage now says both models want full rank, keeps the real rationale (the singletons are what makes the orthogonal/vectorized distinction derivable), and describes our lower-rank acceptance as the compatibility affordance it is. - "Two limits remain" omitted fancy-after-fancy, which is a live `NotImplementedError` reachable from the documented surface, while `index.md` invited chaining fancy steps "anywhere in the chain". Current scope now lists five limits, including the diagonal-view and mixed correlated/orthogonal ones, and both prose pages point at it. - "A single whole-array part stays in the wrapped array's namespace" is only true with *no* partitioning: `result()` branches on whether a partitioning is in force, not on how many boxes it has, so `with_parts((4, 6))` on a 4x6 array returns a plain ndarray. - The changelog stated the support-detection precedence backwards (declaration wins, not inference); `index.md` had a sentence missing its noun; the module docstring's one-line `bounding_box()` summary dropped the stride caveat the three other locations keep; and the package README, the PyPI long description, never mentioned `LazyArray`. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): hold the full-rank invariant inside the engine The index-array rank was checked only from above, so a lower-rank array could exist inside the engine and be read for dependency axes it did not have. Nothing produced one: the tolerance was there for a test asserting compatibility with a body TensorStore itself rejects (verified against 0.1.84 — a rank-1 array over a rank-3 domain is an error in its JSON parser). Require the full input rank in the type, widen a lower-rank array at the JSON boundary where external input arrives, and give the test the shape TensorStore accepts. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): an index array spans the domain it is read over An index array axis must be the domain's extent or a singleton it broadcasts over. Any other size leaves input coordinates with no entry, which read as a smaller selection rather than as the error it is: the truncated array behind one of this review's silent-corruption bugs was a (3, 0) array over a (3, 2) domain, which this rejects at construction. Two fixtures carried the inconsistency they were meant to exercise — an empty array over a domain with room for two coordinates, and a widening case whose array covered three of four positions — and now describe domains their arrays span. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-indexing): a state machine for chained indexing, and the rank-0 part it found Adds `zarr_indexing.testing`, behind a `testing` extra: a Hypothesis state machine that composes indexing steps onto a LazyArray and checks each step's shape, `result()`, and `parts()` assembly against NumPy, plus the selection strategies on their own. A project can point it at its own array by overriding one method. The machine asserts the documented assembly literally — a part's values must arrive at the shape its out_selection addresses — which is how it found the defect it also fixes: intersecting a correlated transform with a part's bounds collapsed the surviving broadcast block into one axis even when the block was already rank 0, so a view narrowed to a single point produced parts of rank 1. A rank-0 block now stays rank 0, and `result()` drops the reshape that was absorbing the mismatch. Merged from the branch that produced it, which predates the coverage guard in `result()`; the guard stays and the reshape it compensated with goes. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): the defects an adversarial review found at the boundaries Six reviewers went at the package, each proving findings by execution. The algebra held: ~85k chained selections against NumPy, ~24k part assemblies with poisoned buffers, 3.8k transform round-trips evaluated as coordinate maps, all clean. Everything below was at a boundary. `result()` and `__array__(copy=True)` could hand back a live view of the source. `_detach` asked whether the *source* was an ndarray, but a duck array that merely stores its data in NumPy returns NumPy views, and those went straight to the caller — while three docstrings promised the opposite unconditionally. It now asks whether the *result* owns its buffer, so memory is released only when sharing is disproved rather than when it cannot be established. The wire format could not reload its own output. `tolist()` renders every empty array as `[]` once the leading axis is the zero-length one, so an ordinary empty selection lost the axis it varied over, and the loader put it back on a different one by prepending singletons. Nested lists cannot express the shape either, so the body carries it. `index_domain_from_json` was a second undefended way into the same objects: a bare `int()` that truncated 3.9, coerced "3" and True, and let a non-string label into a tuple[str, ...]. It goes through the message layer now, as a transform body always did. With it: a rank ceiling, i64 checks on desugared bounds so normalization stays idempotent, ordered index_array_bounds, and typed errors where raw ones leaked. `sub_transform_to_selections` transposed its blocks two ways. A ConstantMap was emitted as a bare integer, which NumPy counts among the *advanced* indices whenever an index array is present, moving the broadcast axis to the front when a slice separates them — while `out_selection` is built positionally. And the correlated branch documented a points-major block but assembled the chunk selection in output order, so a residual slice before the coordinates arrived slice-major. Constants are length-one slices now, named in drop_axes, and the correlated scatter is permuted to the block NumPy actually returns. Nothing caught either one because `LazyArray` resolves a part through its own lowering and never reads `chunk_selection`; one of the two was even asserted as correct in a passing test's comment. Three further failures were one stale field: `_apply_vindex` carried `input_dimension` onto a map the vindex had just made correlated. The value outlived the shape that justified it and was believed later by a scatter that filed positions under the wrong axis — which is why one view's answer depended on how it was partitioned. The dependency is read back off the array now, and `__post_init__` checks the field it was wrong about: ArrayMap was the one map whose `input_dimension` nothing validated. Also: `oindex` over a correlated view applied its index tuple positionally, NumPy's vectorized rule, collapsing two arrays into one axis; `compose()` indexed an inner array's broadcast singletons by the raw coordinate and sized its one-dimensional shortcut by the input rank while gating on the output rank; and neither checked that the outer transform's output lands inside the inner domain, where a negative would have wrapped. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-indexing): generate the selections that were never generated A mutation audit ran 31 mutations and 10 survived. They were not scattered: the invariants check thoroughly what a view *returns* and never what it *claims about itself*, and the strategies could not draw whole classes of selection. The generators now draw them. Orthogonal slices carry a step and may stop early, so a strided or reversed slice reaches `oindex` at all. Coordinate lists and masks can be empty, so a fancy selection that selects nothing exists — the shape that lost its axis on the way through JSON. Vectorized coordinate arrays can be multi-dimensional, so a rank-raising `vindex` is generated. Measured over 4000 draws each, every one of those counts was previously 0. The first run of the widened generators found a live defect: an empty Python list carries no element type and NumPy defaults it to float64, so `oindex[[]]` was refused as a non-integer index array. `json.py` already had that case; the boundary did not. NumPy takes `a[np.ix_([])]`, and so does this now. `Partition.is_complete` gets an invariant. It is what a consumer reads to decide it may take a whole-box read, so a wrongly-`True` one is silent corruption — and three separate mutations to `_covers_whole_part` survived the entire suite, including ones reporting `is_complete` for a part carrying two of its three cells. Asserted one way only: the flag is documented as conservative and only the claim to cover everything has to be earned. Two more state machines. The sorted one-dimensional fancy path needs both ranks to be 1, and the output rank is the *source's*, so the rank-3 default source walled it off entirely — 0 hits from the machine against 105 from the unit tests, in the path where reordering and duplicate coordinates are partitioned. A rank-1 source now reaches it 322 times per run. A source with an extent-1 axis covers the other side of a distinction the code draws from the domain rather than the array. Finally the two remaining mutants, both checked by mutating and confirming the failure: the index-array bound checks are probed one past the boundary rather than comfortably outside it, and `_out_selection_cell_count`'s guard is tested directly, since a partition walk only ever produces the forward in-bounds intervals that never reach it. Assisted-by: ClaudeCode:claude-fable-5 * refactor(zarr-indexing)!: settle the API decisions that get dearer after 1.0 `with_parts` decided what it had been given by inspecting the type of it: a sequence of integers meant uniform boxes, a sequence of sequences meant per-axis sizes, and `None` meant no partitioning at all — which also sent `result()` down an entirely different code path. Three semantics behind one parameter, and no way to ask for one of them and be told when you had spelled it wrong. They are now `with_parts`, `with_parts_per_axis` and `unpartitioned`. A harness that draws from a list of mixed partitionings still needs the dispatch, so it exists once, as `zarr_indexing.testing.repartition`. The sizes those methods take are relative to the array being read, not to the view reading it, and a narrowed view partitions the base extents — which could only be discovered from an error message. `base_shape` says it. `ArrayMap`, `IndexTransform` and `Partition` are `frozen=True`, which reads as a promise that a value can be compared and hashed. Both raised: `==` on the index arrays returned an array and then `ValueError: the truth value of an array is ambiguous`, and `hash()` refused an ndarray outright. So no transform could enter a set or key a cache, and the package had already grown an internal `_is_identity_transform` because of it. An `ArrayMap` was frozen but the array inside it was not, so reaching through a view's transform to `index_array[0] = 9` silently changed what the view returned. It is held through a read-only view now — a view rather than a flag on the caller's array, since constructing a map should not take away the right to write to an array you still own. `IndexDomain.narrow` clamped a slice bound to the domain, in a package whose stated invariant is no clamping and no negative wrapping. `narrow(slice(-3, None))` on `[0, 10)` therefore returned the whole axis — reading as the NumPy spelling of "the last three" and answering with something else — and a stop past the end returned a domain its own parent did not contain. Both raise `BoundsCheckError` now, and a stride raises `ValueError` like every other unimplemented request rather than `IndexError`. `Partition.array` was a view of the array while `LazyArray.array` was the raw source: adjacent types, one name, inverted meanings. The part's is `view`. Smaller: `BoundsCheckError` and `VindexInvalidSelectionError` are exported at the top level, being what exported functions raise; `errors.py` no longer claims `zarr.errors` re-exports them by identity, which is false and would have been believed; `parts()` says it is single-use; and `sub_transform_to_selections` says it is provisional rather than implying the rest of the API's stability. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): correct the claims a reviewer could check, and two dialect gaps `boundary.py` justified applying scalars before the advanced indices by asserting NumPy does the same, citing `a[0, [1, 2], :]`. NumPy groups a scalar *with* the advanced indices for placement, so the two disagree the moment a slice separates them: `a[0, ..., [1, 2]]` is `(2, 3)` where `a[0][..., [1, 2]]` is `(3, 2)`. Scalar-first is this package's documented dialect and stays; the reasoning was wrong, and a wrong reason invites someone to "fix" the correct end later. Two places where the dialect really did diverge, both now matching NumPy. A zero-dimensional integer array is a scalar — `a[np.array(2), :]` drops its axis — but only Python and NumPy integers counted, so a 0-d array was widened into a length-1 index array and kept an axis; that was a third answer, agreeing with neither NumPy nor eager zarr. And a multi-dimensional array in an orthogonal selection is refused where the rule lives, instead of surfacing two layers down as a rank complaint about an `index_array` the caller never wrote. `iter_chunk_transforms` documented two shapes for `out_indices` and returns three: the `dict[int, ndarray]` an orthogonal selection with several index arrays produces was missing, from the function downstream integrators use most. Packaging: the README is the PyPI long description, so the monorepo-only development section moved to CONTRIBUTING.md and the dead relative link to the justfile went with it. The examples pinned `zarr-indexing` to a moving `main` rather than to a release they document. Three of the six docs pins claimed to match the repo root and did not, and the justfile's ruff comment contradicted the package's own pin. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): collapse an empty index array instead of extending the format The previous commit fixed the empty-selection round trip by carrying an `index_array_shape` field, on the reasoning that JSON nested lists cannot express the shape of an array with a leading zero axis — `[]` is the only spelling of every empty shape, and `[[]]` is (1, 0) with nothing for (0, 1). That much is true, but the conclusion was wrong: it invented a field ndsel does not define, so the documents this package wrote stopped being ndsel documents. The reference implementation does not have the problem, because it never emits an empty index array. `t[ts.d[0][[]]]` in TensorStore is `out[0] = 0`, emitted as `{}` — a constant map. An empty index array names no cell, and it can only be empty because an input dimension is, since the full-rank invariant makes every axis either 1 or the domain's extent. Nothing is ever read through it, the emptiness is carried by the domain, and the map is degenerate in exactly the way a size-1 array is — which this format already collapses. So it collapses the same way, and the extension is gone. The loader still recovers the axis from the domain for an empty array arriving from a producer that does emit one, since ndsel does not forbid it; that path just no longer has a first-party caller. Checked against tensorstore 0.1.84: every canonical body from 4000 randomized transforms loads into `ts.IndexTransform`, and every one re-emits itself unchanged. No spec change needed. Assisted-by: ClaudeCode:claude-fable-5 * fix(indexing): address lazy array review findings Assisted-by: Codex:gpt-5 * fix(zarr-indexing): keep an empty masked result masked whichever parts are in force An empty view is now answered without reading the source, and the shortcut reached for the array namespace's own `empty`, which knows nothing about masks. An unpartitioned empty view over a masked source therefore came back a plain array while the same view partitioned came back masked — no cells either way, so no value changed, but the caller's type depended on how the read had been divided, which `result()` promises it never does. A masked source goes through `_output_buffer`, which is the branch that knows. Assisted-by: ClaudeCode:claude-fable-5 * docs(indexing): design chunk projection API Assisted-by: Codex:gpt-5.6 * feat(indexing): add reusable chunk plans Assisted-by: Codex:gpt-5.6 * feat(indexing): project chunks through paired transforms Assisted-by: Codex:gpt-5.6 * refactor(indexing): build lazy parts from projections Assisted-by: Codex:gpt-5.6 * refactor(indexing): expose projection-only chunk planning Assisted-by: Codex:gpt-5.6 * docs(indexing): design visual indexing guide Assisted-by: Codex:gpt-5.6 * docs(indexing): explain coordinate origins Assisted-by: Codex:gpt-5.6 * docs(indexing): motivate negative chunk coordinates Assisted-by: Codex:gpt-5.6 * feat(indexing-docs): add SVG diagram renderer Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): harden diagram rendering Assisted-by: Codex:gpt-5.6 * feat(indexing-docs): add accessible guide diagrams Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): correct guide figure semantics Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): prevent selection label overlap Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): validate arrow label offsets Assisted-by: Codex:gpt-5.6 * test(indexing-docs): add executable guide examples Assisted-by: Codex:gpt-5.6 * docs(indexing): add NumPy-first visual tour Assisted-by: Codex:gpt-5.6 * docs(indexing): explain chunk projections visually Assisted-by: Codex:gpt-5.6 * docs(indexing): add indexing and integration references Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): satisfy strict example typing Assisted-by: Codex:gpt-5.6 * docs(indexing): connect visual guide to reference docs Assisted-by: Codex:gpt-5.6 * fix(indexing-docs): source landing quickstart from example Assisted-by: Codex:gpt-5.6 * ci(indexing): verify executable visual docs Assisted-by: Codex:gpt-5.6 * fix(indexing): address visual guide review Assisted-by: Codex:gpt-5.6 * fix(indexing): improve chunk overlay on phones Assisted-by: Codex:gpt-5.6 * docs(indexing): design system-memory chunk cache example Assisted-by: Codex:gpt-5.6 * docs(indexing): introduce half-open intervals Assisted-by: Codex:gpt-5.6 * test(indexing): narrow diagram label elements Assisted-by: Codex:gpt-5.6 * docs(indexing): demonstrate a system-memory chunk cache Assisted-by: Codex:gpt-5.6 * chore(indexing): stop tracking design specs Assisted-by: Codex:gpt-5.6 * feat(indexing): apply and invert transforms Assisted-by: Codex:gpt-5.6 * fix(indexing): handle scalar and wide transform coordinates Assisted-by: Codex:gpt-5.6 * docs(indexing): explain the chunk-cache lifecycle Assisted-by: Codex:gpt-5.6 * fix(indexing): keep lifecycle diagram readable Assisted-by: Codex:gpt-5.6 * fix(indexing): keep lifecycle caption stationary Assisted-by: Codex:gpt-5.6 * fix(indexing): keep guide diagrams readable Assisted-by: Codex:gpt-5.6 * test(indexing): enforce unique guide figure wrappers Assisted-by: Codex:gpt-5.6 * test(indexing): scan all guide sources for figure duplicates Assisted-by: Codex:gpt-5.6 * docs(indexing): distinguish cache indexing modes Assisted-by: Codex:gpt-5.6 * docs(indexing): scope lazy examples to package Assisted-by: Codex:gpt-5.6 * fix(indexing): expose docs modules to root tests Assisted-by: Codex:gpt-5.6 * docs(indexing): omit text from diagram legends Assisted-by: Codex:gpt-5.6 * docs(indexing): strengthen chunk outlines Assisted-by: Codex:gpt-5.6 * docs(indexing): label unselected chunk cells Assisted-by: Codex:gpt-5.6 * fix(indexing): show coordinates in basic selection Assisted-by: Codex:gpt-5.6 * docs(indexing): clarify coordinate-value mapping Assisted-by: Codex:gpt-5.6 * docs(indexing): simplify half-open intervals Assisted-by: Codex:gpt-5.6 * docs(indexing): explain ordered concatenation Assisted-by: Codex:gpt-5.6 * docs(indexing): consolidate visual guide Assisted-by: Codex:gpt-5.6 * docs(indexing): clarify basic selection figure Assisted-by: Codex:gpt-5.6 * docs(indexing): simplify coordinate introduction Assisted-by: Codex:gpt-5.6 * docs(indexing): explain result axis construction Assisted-by: Codex:gpt-5.6 * docs(indexing): enclose slice result axis Assisted-by: Codex:gpt-5.6 * fix(indexing): clarify result array comparison Assisted-by: Codex:gpt-5.6 * fix(indexing): preserve tutorial result ranks Assisted-by: Codex:gpt-5.6 * docs(indexing): promote chunk cache example Assisted-by: Codex:gpt-5.6 * docs(indexing): render chunk cache source Assisted-by: Codex:gpt-5.6 * fix(indexing): close final correctness gaps Assisted-by: Codex:gpt-5.6 * docs(indexing): replace diagrams with ascii Assisted-by: Codex:gpt-5.6 * chore(indexing): remove svg diagram pipeline Assisted-by: Codex:gpt-5.6 * docs(indexing): simplify guide navigation Assisted-by: Codex:gpt-5.6 * fix(indexing): align selection diagram columns Assisted-by: Codex:gpt-5.6 * feat(indexing): add explicit array readers Assisted-by: Codex:gpt-5.6 * refactor(indexing): resolve lazy arrays through readers Assisted-by: Codex:gpt-5.6 * docs(indexing): exercise readers in chunk cache example Assisted-by: Codex:gpt-5.6 * fix(indexing): preserve cache request event ordering Assisted-by: Codex:gpt-5.6 * refactor(indexing): remove indexing capability taxonomy Assisted-by: Codex:gpt-5.6 * docs(indexing): explain explicit reader execution Assisted-by: Codex:gpt-5.6 * test(indexing): avoid constructor spelling assertion Assisted-by: Codex:gpt-5.6 * fix(indexing): complete reader migration Assisted-by: Codex:gpt-5.6 * fix(indexing): keep reader helpers private Assisted-by: Codex:gpt-5.6 * docs(indexing): finalize reader safety contract Assisted-by: Codex:gpt-5.6 * feat(indexing): add compact chunk grids Assisted-by: Codex:gpt-5.6 * fix(indexing): check affine coordinate arithmetic Assisted-by: Codex:gpt-5.6 * fix(indexing): validate composed constants Assisted-by: Codex:gpt-5.6 * fix(indexing): expose grid size representation Assisted-by: Codex:gpt-5.6 * fix(indexing): validate direct advanced selections Assisted-by: Codex:gpt-5.6 * fix(indexing): handle boolean list masks Assisted-by: Codex:gpt-5.6 * fix(indexing): project sparse affine selections directly Assisted-by: Codex:gpt-5.6 * fix(indexing): normalize chunk planner positions Assisted-by: Codex:gpt-5.6 * fix(indexing): lower reader transforms through bounded slabs Assisted-by: Codex:gpt-5.6 * refactor(indexing): expose global partition read context Assisted-by: Codex:gpt-5.6 * fix(indexing): resolve read context annotations Assisted-by: Codex:gpt-5.6 * feat(indexing): reuse prepared partition plans Assisted-by: Codex:gpt-5.6 * fix(indexing): validate prepared partition coverage Assisted-by: Codex:gpt-5.6 * test(indexing): enforce transform materialization laws Assisted-by: Codex:gpt-5.6 * docs(indexing): clarify chunk projection locality Assisted-by: Codex:gpt-5.6 * fix(indexing): harden transform and partition validation Assisted-by: Codex:gpt-5.6 * docs(indexing): separate examples from snippets Assisted-by: Codex:gpt-5.6 * feat(indexing): support index protocol selectors Assisted-by: Codex:gpt-5.6 * feat(indexing): compose fancy selections without restriction A second oindex/vindex/mask step may now land on any axis of an already-fancy view, including axes an existing index array merely broadcasts along. Array-carrying transforms route through compose() instead of being rewritten in place: the selection is applied to an identity transform over the current domain (same dialect by construction) and chained on, which evaluates the existing index arrays at the new coordinates. The in-place reindex machinery and its fancy-after-fancy guard are deleted. Resolution classifies transforms by structure (the new public index_array_structure): pure per-axis outer products keep the orthogonal resolvers; correlated maps, mixtures, and index arrays sharing an input axis (diagonal gathers) all take the pointwise path, whose intersect and lower stages now broadcast per-map blocks instead of assuming full-block index arrays. Only hand-built affine diagonals (an index array and a slice map bound to the same axis) remain unsupported. This removes the crash where a slice-only vindex step (view.lazy.vindex[...] or vindex[..., scalar]) after a correlated gather misclassified the gather as orthogonal and failed at result(), and fixes a stale input_dimension surviving integer indexing of an empty map's pinned axis. Also, from the same review: result(parts=...) raises ValueError instead of AssertionError when supplied parts do not tile the view; with_parts and with_parts_per_axis raise the documented ValueError for non-iterable input; prepared-part validation uses plain assignment for box parts; and __dask_tokenize__ digests the canonical transform body instead of embedding it, keeping tokens small for large fancy selections. The testing state machine now draws any number of fancy steps per chain. Assisted-by: ClaudeCode:claude-fable-5 * refactor(indexing)!: retire ArrayMap.input_dimension What an index-array map depends on is now read from one place: its full-rank array's shape, whose non-singleton axes are the dependency axes. The retired field pinned the orthogonal axis redundantly and could contradict the array it rode on; every bug found in two adversarial review rounds traced back to its bookkeeping (stale values surviving reindexing, misclassification of correlated maps, dangling axes after integer indexing). The one shape the field disambiguated - a single-coordinate, all-singleton array - is normalized away instead: the selection and composition layers build it as the ConstantMap it equals (output_map.array_map_or_constant), exactly as the JSON serializer has always collapsed it on the wire. A consequence is that a length-1 fancy selection now classifies as a box (is_box, bounding_box, strides), which is the semantically sharper answer. Hand-built all-singleton, empty, or shared-axis ArrayMaps remain valid values and resolve through the pointwise path. Fallout removed with the field: the post-init consistency validation, the basic-indexing renumbering of pinned axes, the JSON loader's global dependency-axis reconstruction, and composition's binding carry-through. The sorted 1-D chunk-planning fast path now applies to either fancy spelling, since the flavors coincide in one dimension. The wire format is unaffected - it never carried the field. BREAKING: ArrayMap.__init__ no longer accepts input_dimension, and the attribute is gone; array_map_dependent_axis now answers from the shape alone. Assisted-by: ClaudeCode:claude-fable-5 * feat(zarr-indexing): a reader for sources that only accept unit-step slices BasicReader reads the minimum by pushing strided and descending selections down as positive-step slices, which assumes the source accepts any step. Integrating a zarrs-backed array showed how common the narrower contract is: FFI bindings and HTTP range endpoints support nothing but slice(start, stop, 1), leaving every such backend to rewrite the same cover-and-restride lowering in its facade. UnitStepReader moves that lowering behind the Reader boundary. The decomposition covers each DimensionMap with the smallest ascending unit-step slab and replays the original stride in the residual — the same move the basic decomposition already makes for direction, extended to magnitude. The residual lowering needed no change: positive strides slice the block, descending ones were always gathered. The cost is explicit in the docstring: a strided selection over-reads its cover by the stride factor, bounded by partitioning the wrapping LazyArray. The existing reader contract cases and the affine-overflow parity test now run across all three built-in readers, through a source that rejects anything but ascending unit-step slices inside its bounds. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): the dense-box re-partition idiom is_box and strides() exist so a consumer can choose a read strategy, but the three lines that act on them were only discoverable by deriving them. The integrations guide now states the policy: a dense box resolves best as one backend slab read — re-partition to the base shape and let the backend dispatch, decode in parallel, and partial-decode shards on its own side — while strided boxes and gathers keep the partitioning, which bounds every cover by one part and makes hull-sized reads of sparse selections structurally impossible. The snippet is executable and pins both regimes by observed reads: the corner gather touches four single cells, never the hull; the dense box is exactly one call. A closing subsection points sources that only accept unit-step slices at unit_step_reader. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-indexing): re-vendor ndsel conformance corpus at 49b9e1db Re-vendor from zarr-developers/ndsel main (49b9e1db1ca93c55f320b025a666367de87a9014, merge of ndsel PR #3). Only transform.json changed vs the previously vendored 92d6a32d: two new fixtures pin empty index_array serialization — normalize carries an empty index_array verbatim rather than rewriting it to a constant map, while a producer SHOULD collapse it to a constant output map, which zarr_indexing.json already does. All other corpus files are byte-identical. Assisted-by: ClaudeCode:claude-fable-5 * ci(zarr-indexing): run the tensorstore parity tests test_ndsel_tensorstore.py skipped everywhere because tensorstore was in no dependency group or workflow. Add a dedicated step to the test job that overlays tensorstore (>=0.1.84, wheels cover the whole 3.12-3.14 matrix) and runs the two parity modules; the main pytest run stays byte-identical to the local canonical invocation. Verified locally against tensorstore 0.1.85 on CPython 3.14: 88 passed, 0 skipped. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-indexing): derive the docs include graph instead of registering it The doc-example tests accreted during the documentation build-out as one-off guardrails: hand-maintained registries of snippet regions, exact heading and navigation strings, substring pins on teaching prose, and tombstones for migrations that already happened. Each new snippet had to be registered by hand, and editing a sentence could fail CI. The registries also missed the one failure they existed to prevent: a page including a region nobody registered was invisible to them. The suite now states two kinds of contract and nothing else. Structural: every '--8<--' include in the rendered markdown — discovered by scanning, so new snippets are covered the day they are written — resolves to exactly one file with a balanced, non-empty region, and every snippet executes, its inline assertions serving as the value check where expected values were previously duplicated into test tables. Behavioral: the pattern matrix, the wrapped-source contract, and the chunk-cache lifecycle keep their tests, because an example cannot assert its own error paths. pymdownx.snippets gains check_paths: true, so the strict docs build now fails on an unresolvable include instead of silently rendering nothing. Verified both directions: a deliberately broken region name fails the scan test (naming page, file, and region) and the docs build. Editorial choices — section order, wording, nav — return to review, where they belong. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): lead every page with the simple idea The guide opened with two meta-paragraphs and an annotated table of contents; the one-sentence mental model — lazy indexing builds a view, planning partitions it, result() materializes it — sat below them. It opens with that sentence now, and says plainly that the first four sections serve anyone indexing arrays while the last two serve integrators, so most readers know they can stop early. Advanced material moves out of the beginner path. Negative-origin domains, grid prepending, and the EdgeDimensionGrid/DimensionGridLike comparison sat in section two, before the reader had met a transform; they now live in the design notes as 'Negative-origin domains and prependable grids', linked from the two places that want them. The paired-projection section introduces the cell domain concretely — a table with one row per selected cell, chunk-local address on one side, result position on the other — before naming it. The landing page gains the missing why: many arrays support only plain slicing, and this package grafts the full NumPy dialect onto them. The pattern reference leads with the selection matrix readers come for and moves the positions-vs-literal-coordinates table after it. The materialization warning becomes a list, and captions that narrated their own code are cut. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): document every public method of the public API Every public class, method, and property reachable from the package root now carries a docstring stating its contract: the coordinate frame it speaks (global source, chunk-grid, or zero-origin chunk-local), whether a chunk length is the declared codec size or the boundary-clipped data extent, what is bounds-checked and what extrapolates, and which inputs raise which errors. Protocol members (DimensionGridLike, DimensionGrid) are written as implementer obligations, since the docstring is the contract a third-party grid must satisfy. Dataclass-generated __init__ methods are left to their class docstrings; adding a docstring there would mean hand-writing the constructor for no behavioral reason. Docstrings only — no code, signature, or existing-docstring changes. Verified: an introspection audit over __all__ reports zero public members without docstrings; the full suite and pyright are unchanged. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): attribute docstrings for every public field The previous pass documented methods and properties; dataclass and TypedDict fields — IndexDomain.exclusive_max and 41 siblings across fifteen classes — carried no per-attribute documentation, only prose in their class docstrings. Each public field now has an attribute docstring stating what the value means, its coordinate frame or units, and the invariant it carries (literal bounds may be negative, edges are declared codec sizes unclipped by extent, derived fields say what they are derived from, wire bounds admit the infinities the engine refuses to lower). Found by an AST audit, since attribute docstrings are invisible to runtime introspection; that audit now reports zero undocumented public fields. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): the input/output-to-request/source dictionary, and each map kind in NumPy terms An IndexTransform is a function between coordinate spaces, and its fields speak function vocabulary (input, output) while every array-minded reader speaks request and source. The confusion this causes is concentrated in one word: 'output' looks like data, but names the output side of the coordinate function — which is where values are read FROM, since data flows against the arrow. The transform section of the guide and the transform API page now state the dictionary outright, in a two-row table, at the moment a reader first meets the fields, along with why the neutral names exist: composition, where an interior transform has neither a request nor a source side. The three output map kinds are now demonstrated executably against their NumPy counterparts: DimensionMap against basic and negative-step slices, ArrayMap against fancy indexing with order and duplicates preserved, and ConstantMap against numpy.broadcast_to — stated as the value-faithful counterpart precisely because no NumPy selection spells a retained constant axis; an integer index drops it, and a repeated fancy index matches the values while degrading the description to a coordinate list. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): flat landing page — motivation, example, links The grid cards misrendered (misaligned card bodies) and earned their keep poorly: two navigation targets dressed as a layout feature. The landing page now follows the shape convention of projects like pydantic — motivation paragraph, install, one quickstart with a sentence stating the lazy/eager boundary, then a single annotated link list. The two cards' start-here targets survive as the visual guide entry, which names both audiences and their entry points in one line each. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): a single-point bounds error never mentions a batch apply() delegates to the vectorized kernel shared with apply_many() — the right direction, since the batch path is the hot one — but the kernel's diagnostic leaked through it: apply((11,)) on a [-10, 10) domain reported 'point at batch position ()', naming a batch the caller never formed, from a private frame the caller never called. The kernel now raises an internal structured signal (dimension, value, bounds, batch position) and each public entry formats it in its own vocabulary, 'from None' so the traceback ends at the API layer: apply says 'coordinate 11 on input dimension 0 is outside the domain [-10, 10)'; apply_many keeps the batch-position form, where that context is exactly right. Message-only change; BoundsCheckError remains the type on both paths. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): annotation syntax in docstring type slots Parameters and Returns entries now state types as annotations — Sequence[int], tuple[int, ...], numpy.typing.NDArray[numpy.intp] — instead of prose like 'sequence of int' or bare 'tuple'. The annotation is the type's one precise spelling, matches the signature beside it, and names the exact shape where prose left it to the description (both boundary functions' bare 'tuple' entries now state their element structure). Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): the chunk cache example is named for what it is The example was born napari_chunk_cache, and the docs spent two bold disclaimers insisting it is a napari-like consumer, not a napari integration — while the nav entry, section headings, and class names had already settled on 'system-memory chunk cache'. A name that needs disclaimers is the wrong name. The directory, script, and docs page are now system_memory_chunk_cache, matching everything else; napari remains where it belongs, in prose, as the motivating access pattern. Path-only rename: no code, region names, or prose claims change. The derived include-graph test and the strict docs build (check_paths) verify every include and link followed the move. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-indexing): an empty domain reads as empty through every reader Composing a fancy selection onto an empty-domain view emits an ArrayMap that is legitimately empty along the vanished axis — a shape the package promises resolves like any other. The resolvers broke that promise: _correlated_map_coords tried to reshape the 0-size array to its non-zero singleton block axes and raised ValueError from all three built-in readers on a direct read_into, a sequence the pre-composition engine handled. LazyArray.result() masked it only through its own size-0 short-circuit. _lower now answers an empty domain first — nothing is selected, so no resolver needs to evaluate maps that may be empty along vanished axes — and the correlated path independently returns no coordinates for an empty broadcast block. Found by an adversarial review fuzzing composed selection chains (3 of 400 random chains hit it); the regression test pins the exact public-API reproduction across all three readers. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-indexing): execute the CLI examples; strict builds guard anchors and nav Two claims from the doc-test restructure were false, and an adversarial review proved both empirically. First, the lazy_indexing_* examples were said to run under the repository-root example runner; that runner globs only the root examples directory, so the two CLI examples ran under no test at all. They now run as subprocesses here, the dask one skipping where dask is absent. Second, the module docstring claimed mkdocs --strict covered the deleted anchor and nav guards; a strict build passed with a deliberately broken cross-page anchor and with a page omitted from nav, because both are INFO-level by default. mkdocs.yml now sets those validations to warn, which strict promotes to errors — verified failing on a broken anchor and passing clean. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): the vindex error's real contract, and neutral map vocabulary VindexInvalidSelectionError's docstring claimed it covered every non-coordinate vindex form; in fact only the wrapper's validation raises it, only for slices — other invalid entries raise plain IndexError, and the engine-level IndexTransform.vindex accepts residual slice dimensions without raising. The docstring now states the actual raise site. The transform-algebra docstrings (transform, output_map, composition, json) also drop 'storage' for neutral input/output vocabulary: a transform's output side is just output coordinates — in a composition chain an interior transform has no storage side at all. The request/source/storage translation stays where it belongs, in the guide's vocabulary table and the endpoint layers (readers, grids) that really do face arrays. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): an executable Examples section on every public object Every public class and function in the package root now carries a numpydoc Examples section in doctest form: 37 new examples, each a small intuition-builder in the shape the IndexTransform walk-through set — the domain is the result's coordinates, the output maps are the rule, and where an object corresponds to a NumPy indexing concept the example demonstrates the equivalence (DimensionMap against a slice, ArrayMap against fancy indexing with duplicates surviving, ConstantMap against numpy.broadcast_to, compose against chained slicing). The error classes demonstrate their actual raise; the wire types round-trip real bodies. The examples are enforced, not decorative — and closing that loop exposed that the package's existing doctests were never collected anywhere: the package pyproject shadows the repository root's pytest configuration, and no invocation named src. The package config now enables --doctest-modules with the root's option flags, testpaths includes src/zarr_indexing, and the justfile recipe and CI workflow collect it explicitly. 1280 tests pass, 45 of them doctests. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): apply and apply_many say what they locate, not their signature 'Map one input coordinate to an output coordinate' restates the type signature in prose — any function maps inputs to outputs. The summaries now speak the class docstring's array-indexing frame: apply maps a coordinate of the domain (a result cell) to the source coordinate its value is read from, by evaluating each output map; apply_many is the batch form. Both gain a doctest locating cells of the [::2] transform, and both state that no data is touched — this is the coordinate arrow, running result to source. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): method summaries never lean on the ambiguous naked input/output 'input' and 'output' are reserved algebra terms in this package, and in a method summary they collide with ordinary function-speak: identity's 'input coordinate i maps to output coordinate i' reads equally as the algebra statement and as a vacuous description of any function — the same trap apply's summary fell into. identity, intersect, and translate now speak the array frame instead: every result cell reads the source at its own address; keep only the cells whose source coordinates fall inside the box; shift the source coordinates every cell reads. Field docstrings and class summaries keep the naked terms where no call is in sight and the technical reading is the only one available. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): drop non-behavior statements that disambiguate nothing 'No data is touched' in apply/apply_many and 'without I/O' on the oindex/vindex accessors stated what the functions do not do without clarifying what they do: nothing about a coordinate lookup, or about an accessor documented to return a new transform, suggests data movement. Removed. The statements that earn their negation stay: the module thesis, the class contract, and __getitem__ — where subscription syntax genuinely suggests an eager read to anyone arriving from zarr. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): a flat sidebar — sections must earn their existence 'Use lazy indexing' held one page: a disclosure triangle and a competing label with no organization gained. 'Practical reference' classified nothing and grouped two pages serving different audiences — the exact split the guide's opening and the landing page's annotated links already route explicitly. Both dissolve into top-level entries. Examples and API Reference keep their sections, being the only real collections at this site's size; the ndsel wire format and design notes gain explicit labels and sit in the for-builders tail before the API. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): the three guide pages are one Guide section Visual guide, indexing patterns, and integration boundaries sat at top level with the same rank as the fifteen-page API Reference — three pages wearing category clothes. They are one collection, and the docs/guide/ directory said so all along: learn it, look it up, apply it at the…
HTTP server that exposes stores, arrays, groups (#3732) * implement store server and node server * update tests and add v2 -> v3 example * add __all__ and clean up tests * add docs and changelog * add proper server * rework examples (simplify) and make server a context manager * minor tweaks to server * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * fix: adapt server branch to current starlette/uvicorn starlette 1.3 deprecated using httpx with its TestClient, which the warnings-as-errors filter turns into a collection error; add httpx2 to the test dependency group. uvicorn 0.51 removed Server.install_signal_handlers and now skips signal-handler setup off the main thread natively, so drop the monkey-patch workaround. Assisted-by: ClaudeCode:claude-fable-5 * feat(zarr-server): scaffold packages/zarr-server Assisted-by: ClaudeCode:claude-fable-5 * feat(zarr-server): move server code, tests, and example into the package The moved code now uses only public zarr API: zarr.buffer.cpu replaces zarr.core.buffer.cpu, spec-defined key names are inlined, chunk-key encodings are duck-typed on their spec names, and the shard grid shape is computed locally from public Array attributes. Also mirrors the root repo's [tool.numpydoc_validation] override in the package's pyproject.toml, since numpydoc-validation resolves config from the nearest pyproject.toml and would otherwise apply its stricter default checks to the moved docstrings. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-server): package README; retarget root changelog fragment at the core fix Assisted-by: ClaudeCode:claude-fable-5 * refactor!: remove the HTTP server from zarr core The server now lives in packages/zarr-server (published as zarr-server). The feature never shipped in a zarr release, so there is no deprecation shim. The decode_chunk_key strictness fix stays in zarr core. Assisted-by: ClaudeCode:claude-fable-5 * ci(zarr-server): add package test and release workflows Assisted-by: ClaudeCode:claude-fable-5 * chore(zarr-server): refresh package lockfile after root server-extra removal The package lock embeds the workspace-root zarr project's metadata; Task 4 removed the root server extra and httpx2 test dep after this lock was first generated. Assisted-by: ClaudeCode:claude-fable-5 * fix: reject path-traversal segments in zarr-server request handling store_app (and thus serve_store) passed the request path straight to the store without validation, since the is_valid_node_key gate only ran for node_app. Starlette percent-decodes path params, so a request like GET /..%2fsecret.txt arrived as literal ".." and LocalStore resolved it outside the store root, allowing arbitrary file read via GET and arbitrary file write via PUT. Closes this path-traversal vulnerability by rejecting any "." or ".." path segment in _handle_request before the store is touched, for both store_app and node_app. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-server): close absolute-key path-traversal bypass in request guard The path-traversal guard in _handle_request only rejected "." and ".." segments. A percent-encoded leading slash (e.g. "/%2fetc%2fhostname") decodes to an absolute path param ("/etc/hostname"), whose split() produces an empty leading segment with no "." or ".." segment, so the guard let it through. LocalStore resolves an absolute key by discarding its configured root, allowing arbitrary filesystem read/write outside the store. Reject empty segments too, closing the bypass. Assisted-by: ClaudeCode:claude-fable-5 * ci(zarr-server): bump actions/attest to v4.2.0 to match sibling workflow Keeps the zarr-server release workflow's pinned action SHAs in sync with the dependabot bump that landed in zarr-metadata-release.yml via the merge. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-server): fold backslash separators and reject drive-qualified keys The traversal guard in _handle_request split only on "/", so backslash- separated segments like "..\\..\\win.ini" and drive-qualified or UNC-rooted keys like "C:/Windows/win.ini" or "\\host\share\x" passed through untouched. On Windows, LocalStore joins keys onto its root via pathlib, which discards the root entirely for a drive-qualified or rooted key -- turning percent-encoded backslash paths into arbitrary file read/write. Fold backslashes to "/" before the segment check (mirroring zarr's own normalize_path) and add an ntpath.splitdrive check to catch drive letters and UNC prefixes that don't produce empty/"."/".." segments. Added TDD coverage that fails against the old guard (via a store double that raises if get/set is ever called) and passes against the fix. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-server): fix vacuous two-level traversal test The `/..%2f..%2fsecret.txt` case in test_encoded_traversal_variants_ return_404 climbed two levels from tmp_path/store_root to tmp_path/.., but the secret was written at tmp_path/secret.txt (one level up) -- so the case passed for the wrong reason (no such file, not "guard blocked it") even with the traversal guard deleted entirely. Nest the store root exactly `climb_depth` directories below tmp_path per case, so every case's ".." segments resolve to tmp_path/secret.txt. Verified by mutation: copied _serve.py to a scratch dir (never the repo), deleted the guard body, and ran TestPathTraversalProtection against it via PYTHONPATH. Before this fix, 12/13 traversal tests failed against the mutant (1 false pass -- this vacuous case). After this fix, 13/13 fail against the mutant, and all still pass against the real guard. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-server): bound BackgroundServer shutdown instead of blocking forever shutdown() joined the server thread with no timeout, and uvicorn's graceful wait is itself unbounded: force_exit is only set by a signal handler uvicorn deliberately skips off the main thread. A client mid request could wedge __exit__ unrecoverably. Bound uvicorn's graceful wait with timeout_graceful_shutdown and fall back to force_exit if the thread outlives it, both driven by a new shutdown_timeout parameter on serve_store/serve_node. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-server)!: only accept HTTP methods the handler implements _handle_request special-cased PUT and let every other verb fall through to the read path, so a server built with DELETE/POST/PATCH answered them with the key's contents and changed nothing. HTTPMethod advertised all of them. Narrow HTTPMethod to GET/PUT/HEAD and reject anything else when the app is built. Doing this before the first release avoids narrowing a published type later. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-server): document client deps; lint the package's changelog in CI The README's round-trip example reads back through FsspecStore, which needs an HTTP-capable fsspec that zarr-server does not depend on, so a clean install failed on the PyPI landing page's headline snippet. Same for examples/serve.py under a plain interpreter, which needs httpx. check_changelogs.yml also never visited packages/zarr-server/changes, so a malformed fragment would have passed PR CI and failed at release time. Assisted-by: ClaudeCode:claude-fable-5 * test(zarr-server): add docs dependency group and cover the README round-trip Reading a served array back with zarr.open_array(url) routes through FsspecStore -- zarr's only URL-string backend -- so the README's headline example needs an HTTP-capable fsspec that the package itself has no reason to depend on. Carry that in a docs group, mirroring the root project's group for running examples, and use it to test the round-trip in CI so the example cannot rot. Assisted-by: ClaudeCode:claude-fable-5 * refactor(zarr-http-server)!: rename zarr-server to zarr-http-server The package is HTTP-specific end to end -- Starlette/ASGI, byte-range headers, CORS, HTTP verbs -- so the unqualified name overclaimed its scope and squatted the generic name that a future transport (an S3-compatible or WebDAV frontend) would want. Renames the distribution, the zarr_http_server module, the package directory, the zarr_http_server-v* release tags, both workflows and their artifact and PyPI environment names. Nothing is published yet, so this costs nothing now and would be permanent after the first upload. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-http-server): reject an empty methods set instead of failing open Starlette's Route treats a falsy `methods` as "match every method", and an empty set passes the unsupported-verb check trivially, so store_app(store, methods=set()) served GET, DELETE, POST, PATCH and TRACE alike and accepted PUT writes -- the opposite of what the caller asked for, and the same fail-open the method narrowing set out to close. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-http-server): report the bound port, unblock the loop, 404 unopenable children Three defects the adversarial re-review reproduced: BackgroundServer reported the requested port, so port=0 produced http://host:0 while the socket was bound elsewhere -- clients following the documented url reached an unrelated service. Group key validation opened children through zarr's synchronous API directly on the event loop, serializing every concurrent request behind it (20 concurrent requests: 10.5s, now 1.1s) and outlasting the shutdown timeout. Move it to a worker thread. Child lookup caught only KeyError, so a corrupt metadata document or a codec from an uninstalled plugin surfaced as 500 and made a whole subtree unservable during ordinary operation. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-http-server): close the S5-S13 follow-ups from the adversarial review Hostile keys the store cannot express (embedded NUL, over-long names) now answer 404 instead of surfacing ValueError/OSError as a 500. PUT bodies are capped at DEFAULT_MAX_BODY_SIZE and answer 413 past it; Store.set takes a whole Buffer, so a body cannot be streamed and one request would otherwise size the server's memory. 206 responses carry Content-Range, and a range beyond the end or an inverted one answers 416 rather than an empty 206. A 0-d v2 array's sole chunk, stored under "0", is now servable: the key decodes to a 1-tuple no 0-d grid could match, so it needs the array's dimensionality to disambiguate. Array and group metadata key sets are split, so a v2 group no longer claims .zarray as its own; bind failures report the likely cause instead of a bare timeout; and serve_* gain a background: bool overload. The two out-of-bounds chunk tests planted no data, so they passed for the wrong reason -- mutating the bounds check to return True left the suite green. They now plant data at the out-of-grid key, and that mutation fails 3 tests. Adds the missing coverage for this branch's one core change, and widens the workflow's path filter to src/zarr, since the package resolves zarr from the repo root and a core change can break it. Assisted-by: ClaudeCode:claude-fable-5 * fix(zarr-http-server): stop reporting I/O failures as misses; enforce the body cap while reading The round-3 review found the previous round's error handling was too broad and its body cap too narrow. except (ValueError, OSError) around store.get and store.set caught the whole errno family, so EACCES, EROFS and ENOSPC all answered 404. Under the v3 spec an absent chunk is an uninitialized one and a reader is right to substitute the array's fill value, so 404 asserts something about the store's contents: an unreadable chunk answered that way has a correct client silently materialize fill values over data that exists, and a failed write looks like a pointless one rather than a failure. Only ENAMETOOLONG and EINVAL now mean "this key names nothing"; everything else surfaces. NUL bytes are rejected in the guard instead, so the store's own errors always mean real I/O trouble. max_body_size only consulted Content-Length, which a chunked request does not send, so request.body() buffered the whole thing before the check: 256 MiB passed a 1 KiB cap at 572 MiB peak RSS. The body is now read incrementally and abandoned at the cap -- the same attack peaks at 59 MiB. Also: a range too wide to allocate answers 416 rather than raising MemoryError; the force_exit fallback join is bounded, so shutdown cannot outlast its timeout; DEFAULT_MAX_BODY_SIZE is importable, since it is the documented default of three public functions; and _make_starlette_app loses a parameter it never read. Adds the coverage the review found missing: shard-grid bounds (the mutant served an out-of-grid shard key against a green suite), wrong-arity chunk keys, the chunked body path, I/O failures as 5xx, and a parser-level assertion for inverted ranges, which the status code alone could not distinguish on a MemoryStore. The drive-letter check stays unconditional. It over-rejects a first-segment node name like a:b, which is legal on POSIX -- but ntpath treats any single character before a colon as a drive, so there is no safe subset, and the guard is a string gate in front of an arbitrary Store whose path semantics this package cannot know. Assisted-by: ClaudeCode:claude-fable-5 * fix(ci): silence BLE001 on two intentional broad excepts ruff 0.16 selects BLE001 under the root config's `B` prefix, so the package's `uvx ruff check .` job failed on two deliberate blind excepts. Both are intentional and already documented, so they get targeted noqa comments, matching how #4213 handled the same rule in the core tree. Assisted-by: ClaudeCode:claude-opus-5 * chore: drop stale changelog fragment for the chunk key fix The DefaultChunkKeyEncoding.decode_chunk_key fix this fragment described was split out into #4219 and has already shipped -- its text is in docs/release-notes.md verbatim. Leaving the fragment here would emit the same paragraph a second time under a #3732 link, for a change this branch no longer contains. Assisted-by: ClaudeCode:claude-opus-5 * ci: check zarr-http-server changelogs and align action pins check_changelogs.yml validated the root, zarr-metadata and zarr-indexing changes/ directories but not zarr-http-server's, so the new package's fragments were the only ones whose filenames went unchecked. The two new workflows were also written before #4241 bumped the actions group, so they pinned older checkout/setup-uv/attest/pypi-publish SHAs than their siblings. Bump them to the versions main already uses; the release workflow is now identical to zarr-metadata's modulo the package name. Assisted-by: ClaudeCode:claude-opus-5 * docs(http-server): add justfile and Read the Docs scaffold Brings the package in line with zarr-metadata and zarr-indexing, which each own a justfile and a separate Read the Docs site. The docs content is a scaffold -- an overview page and an API reference over the public namespace -- meant to be filled in later; the point is to get the site wired up and building under --strict now. The `docs` dependency group previously held the runtime deps for the README examples, but Read the Docs and `just docs-check` both expect it to carry the mkdocs toolchain, as it does in the sibling packages. Those example deps move to a new `examples` group and the test job follows. CI gains a `docs` job matching the siblings', so a scaffold that stops building fails the gate. The repo-root .readthedocs.yaml skips PR builds confined to this package now that it has its own site, and the root docs nav links out to it. Creating the Read the Docs project itself is a manual step: point its configuration-file path at packages/zarr-http-server/.readthedocs.yaml. Assisted-by: ClaudeCode:claude-opus-5 * fix(http-server): correct silent-corruption and HTTP-conformance defects Two of these answered a request successfully while leaving the client with data that does not exist. Chunk keys were validated by decoding coordinates and bounds-checking them, never by re-encoding. `int` is lenient in ways a store key is not -- leading zeros, a leading `+`/`-`, surrounding whitespace, underscore separators, non-ASCII decimal digits -- so `c/00/00` decoded to (0, 0) and validated, then went to the store verbatim. A PUT answered 204 and stored the body under a key no reader looks up: success reported, data invisible. Validation now requires the key to equal `metadata.encode_chunk_key(coords)`, which makes the accepted set exactly the set zarr can read. Decoding delegates to the encoding's own decoder rather than reimplementing the default/v2 grammars, so a new or third-party chunk key encoding works without changes here. Resolving a group child caught bare `Exception` and returned False, which turned an unreadable child -- EACCES, EIO, a corrupt metadata document, a missing codec plugin -- into 404. Under the v3 spec an absent chunk is an uninitialized one, so a correct reader answers that 404 by substituting the fill value over data that exists. Only KeyError is caught now; a key that could not be judged surfaces as 5xx rather than being reported absent. The rest are conformance fixes on the same request path: - A Range header the server cannot use is now ignored with a 200 rather than refused with a 416, per RFC 9110 §14.2. This covers an unrecognized unit and a multi-range request, both legal to send. - A suffix range resolves against the object's size, so a 206 always carries the Content-Range that RFC 9110 §15.3.7 requires. Sharding reads a shard index this way, so the header was missing on a hot path. - A last-byte-pos wider than the store can materialize is clamped to the end of the object per §14.1.2 instead of raising out of the store as a 500. - A byte position is parsed as 1*DIGIT rather than by `int`, which accepted `+0`, ` 0` and `0_0`. - PUT to a read-only store answers 403 instead of letting the store's ValueError surface as a 500. Assisted-by: ClaudeCode:claude-opus-5 * test(http-server): property tests over a real endpoint Adds hypothesis properties that drive a real uvicorn server over a socket, so the assertions cover what only exists on the wire: header parsing, method dispatch, status codes. Each property checks the response *and* the backing store. That pairing is what the previous suite could not do: a PUT to a non-canonical chunk key answered 204 and wrote a key no reader consults, which a response-only assertion cannot see. Refused requests assert the store is byte-for-byte unchanged; accepted writes assert the bytes landed under the key the client named and that a zarr client reads back the values. Keys are generated in two families -- in-band (the node's metadata and the canonical spelling of each chunk key in its grid) and out-of-band (non-canonical spellings, out-of-grid coordinates, traversal probes, a sibling node's keys). Two details worth keeping: - Traversal probes are percent-encoded. An HTTP client resolves dot-segments before sending, so httpx turns "../secret" into "/secret" and a literal probe asserts nothing; encoded, it reaches the server and Starlette decodes it back into a real ".." segment. - The matrix includes LocalStore, not just MemoryStore. MemoryStore slices a `bytes` and accepts any range bound, so it cannot distinguish a clamped over-wide range from a refused one -- mutation-testing the suite showed that property passing against deliberately broken code until a filesystem-backed server was added. Verified by reverting each fix in turn and confirming the corresponding property fails. Assisted-by: ClaudeCode:claude-opus-5 * fix(http-server): stop reading EINVAL from a store as a missing key `_names_nothing` reclassified two errnos as absence so a client could not turn a freely chosen key into a 5xx. `ENAMETOOLONG` earns that: it is the store answering about the name -- nothing can be stored under a name it cannot express -- and `encode_chunk_key` never produces a segment near a filesystem's length limit, so it is unreachable for real data. `EINVAL` does not. It is POSIX's catch-all, reachable on a perfectly ordinary short key through a bad seek or an unsupported filesystem feature, and under the v3 spec an absent chunk is an uninitialized one -- so answering 404 has a correct reader write fill values over a chunk that exists but could not be read. That is the same defect already fixed in the group-child lookup and in chunk key validation: reporting "something went wrong" as "it is not there". Nothing exercised EINVAL in practice. A NUL in a key raises ValueError and is rejected before the store anyway, and an over-long key raises ENAMETOOLONG, so this narrows the guard to the case that was doing the work. The traversal, absolute-key and drive-letter guards deliberately stay where they are rather than deferring to the store. LocalStore.get/set are `self.root / key` with no validation -- zarr's normalize_path applies at the StorePath layer, not to raw store keys -- so `../sibling.txt` and an absolute key both write outside the store root. This server is the component that feeds a Store unvalidated strings from the network, so it is the component that has to reject them. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): stop sealing over CORSMiddleware and uvicorn.Config Wrapping an API means taking responsibility for its parameters, not hiding the ones we did not think to name. Two wrappers were doing the latter. `CorsOptions` carried 2 of `CORSMiddleware`'s 8 parameters, so `allow_headers`, `allow_credentials`, `allow_origin_regex`, `allow_private_network`, `expose_headers` and `max_age` were unreachable without bypassing this package. It now mirrors the full signature, with every key optional. Two of the defaults are ours rather than Starlette's, because the server knows what its caller should not have to. It emits `Content-Range` on every ranged response, which is not a CORS-safelisted response header, so `expose_headers` defaults to `["Content-Range"]` -- without it a browser client could read the bytes but not learn which bytes it got, which made the suffix-range Content-Range fix invisible to exactly the clients CORS exists for. It accepts a `Range` request header, so `allow_headers` defaults to `["Range"]`; Starlette's empty default answered a preflight naming Range with 400. Defaults apply only to absent keys, so an explicit `expose_headers: []` means "expose nothing". `_start_server` passed 4 of `uvicorn.Config`'s 52 parameters, which put TLS, `proxy_headers`/`forwarded_allow_ips`, `root_path`, `log_level`, `limit_concurrency` and unix-socket binds out of reach entirely. A `uvicorn_options: Mapping[str, object] | None` is merged over the three options set here, so a caller key wins. uvicorn ships no TypedDict for Config -- its only TypedDicts are ASGI protocol events -- so the mapping is hand-typed and the cast is confined to the call site. Un-sealing uvicorn makes two BackgroundServer attributes reachable that could previously only be one thing. `url` now reports the scheme actually in use, so configuring TLS yields https, and `host`/`port`/`url` are None for a uds or fd bind rather than naming an address nothing is listening on. Assisted-by: ClaudeCode:claude-opus-5 * fix(http-server): lifecycle, HEAD cost, and CORS/route method agreement Clears the findings left open from the review. Lifecycle. `_start_server` raised on its startup timeout without ever signalling the server, so the thread went on to bind the port and serve forever as a daemon with no handle to stop it -- and a retry on the same port then failed with the other error. It now sets should_exit and force_exit and joins before raising. `shutdown()` returned normally when the thread survived both joins, reporting success for a server still bound and still serving; it now raises. The first join also matched uvicorn's own `timeout_graceful_shutdown` exactly, and uvicorn spends ~0.2s tearing down before that wait even begins, so the join always expired first and escalated to force_exit on the orderly path -- which makes uvicorn skip ASGI lifespan shutdown. The join now outlasts the graceful bound by a margin, and reads that bound from the config so it stays correct when a caller sets it through uvicorn_options. HEAD. A HEAD body is discarded at the wire, but HEAD fell through to the GET handler, so answering one transferred the whole value: measured at 10 MB read to report a length. It is now answered from `Store.getsize` -- a stat on a filesystem store, an info call on a remote one -- and reads zero bytes. HEAD is served whenever GET is, which is what Starlette does and what RFC 9110 asks of an origin server; the README and docstrings said otherwise and now say so. CORS. `cors_options["allow_methods"]` was passed through unchecked, so an app could advertise methods its route rejects: a browser caches that preflight and every later cross-origin call fails with 405 after a successful handshake. Advertising an unserved method is now a ValueError at construction, consistent with how unsupported `methods` are already rejected, and `"*"` expands to what is actually served rather than to every verb Starlette knows. An absent `allow_methods` is left alone -- widening it to everything served would newly advertise PUT cross-origin on a write-enabled app that never asked for it. Media type. The JSON content type was keyed off a third hardcoded "zarr.json", so a v2 array's `.zarray` was served as octet-stream. It is now derived from the same tables that decide which keys a node owns. Also documents that `store_app` does not validate keys: it proxies the raw key space and has no array semantics to check against, so a client that misspells a chunk key gets a successful write to a key no reader consults. `node_app` rejects that with 404. Assisted-by: ClaudeCode:claude-opus-5 * ci(http-server): run the justfile's recipes instead of copies of them The workflow repeated the commands the justfile already defines -- `uvx ruff check .` and the mypy invocation were byte-identical copies, and the pytest step differed only by the sync that precedes it. Two definitions of the same verb drift silently: renaming the `docs` dependency group to `examples` required the same edit in both places, and updating only one would have left `just check` and CI testing different things with nothing failing. CI now calls `just test`, `just lint` and `just typecheck` (it already called `just docs-check`), keeping the python matrix and caching, which are genuinely CI's concern. This matches zarr-metadata, whose workflow already states the arrangement; zarr-indexing remains half-converted. Delegating also meant fixing what the shared recipe would otherwise spread: `just lint` ran an unpinned `uvx ruff`, which is precisely how this job broke before -- ruff 0.16 began selecting BLE001 under the root config's `B` prefix and failed on rules the pre-commit-pinned ruff never enforced, with no code change to blame. The recipe now pins the same version .pre-commit-config.yaml does, so the local gate, the pre-commit gate and CI enforce one standard. Assisted-by: ClaudeCode:claude-opus-5 * ci: build the zarr-http-server docs on pre-push A dead cross-reference or a nav entry pointing at a removed file only fails at `mkdocs build --strict`, which until now happened first in CI. This catches it before the code leaves the machine. Scoped deliberately. `stages: [pre-push]` overrides the repo default of running on every commit: this builds the whole site, which is too slow to pay per commit and is only actionable before pushing. `files:` limits it to changes that touch the package, and `pass_filenames: false` because mkdocs builds a site rather than a list of files. It delegates to `just docs-check` so the build has one definition shared with CI, and is added to `ci.skip` alongside mypy for the same reason that one is skipped: pre-commit.ci's runners have neither `uv` nor the repo checkout needed to resolve the environment. The zarr-http-server workflow covers it there. Note this hook and CI still declare their toolchains separately -- the hook shells out to the local `just`/`uv`, CI installs them itself. That is inherent to pre-commit.ci not being able to run them, and is the same trade already accepted for mypy. Also refreshes packages/zarr-http-server/uv.lock, which references the root project's dependency groups, for the hypothesis and uv bumps that arrived with the main merge. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): make the read-only guarantee explicit and enforced Read-only was already the default -- `store_app(store)` answers 405 to PUT, POST, DELETE and PATCH, and POST is unconfigurable because there is no handler behavior for it -- but nothing said so and little pinned it. Only PUT was covered against the default app; POST, DELETE and PATCH were covered only against a fixture built with writes enabled, so "the default app is read-only" was not actually a tested claim. Adds a test class covering both layers the guarantee rests on: `methods`, which decides what the route answers, and the store, which decides whether a write could succeed at all. Each refusal also asserts the value is unchanged, matching the property tests -- a 405 that still wrote would otherwise pass. Serving PUT from a read-only store is now a ValueError at construction. A store's `read_only` is fixed when it is built, so that combination can never succeed; it previously surfaced as a 403 to whichever client tried to write first, long after whoever misconfigured it had moved on. The handler's 403 stays as a backstop for a store whose read_only is not fixed, and the test for it builds the app through the private builder since the public entry points now reject the combination. Documents `store.with_read_only(True)` as the categorical recipe: it is the stronger of the two layers because it holds even if the HTTP layer is misconfigured. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): name the read-only and read-write method sets `READ_ONLY_METHODS` and `READ_WRITE_METHODS` let a call site say which it is, rather than leaving that to the presence or absence of an argument. The read-only one is exactly the default, so passing it changes nothing except that the intent is written down. The value is the other direction: a writable app must name a method set, so `grep -r 'methods='` finds every place that opts into writes -- which is what makes a deployment auditable without a separate read-only entry point. Both are frozensets, so one caller cannot widen the default for every other, and both name HEAD explicitly: Starlette serves it wherever GET goes, and a constant that omitted it would misdescribe the route. `methods` now accepts any `AbstractSet`, which is what lets a frozenset constant be passed where a `set` was previously required. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): model read-only methods in the type domain Renames the constants to READ_ONLY_HTTP_METHODS / READ_WRITE_HTTP_METHODS so they say what kind of method they hold, matching the HTTPMethod type they are drawn from. Adds `ReadOnlyHTTPMethod = Literal["GET", "HEAD"]`, which moves the distinction from a runtime convention to something a checker enforces: a `frozenset[ReadOnlyHTTPMethod]` cannot contain "PUT", so a read-only interface can be declared rather than merely configured. Verified against mypy --strict -- assigning either a set containing "PUT" or READ_WRITE_HTTP_METHODS to that annotation is an error, while READ_ONLY_HTTP_METHODS is accepted. HTTPMethod is now the union of that and a private `_WriteHTTPMethod` rather than a third hand-written list of the same strings, and both constants plus _SUPPORTED_METHODS are derived from the Literals via get_args. The runtime sets and the static types therefore cannot disagree about what this server serves: widening a Literal is the only edit needed, and a test pins the contents so that widening is deliberate. Assisted-by: ClaudeCode:claude-opus-5 * docs(http-server): add a notebook example, and run both examples in tests The only example used `with serve_node(...)`, which is the one form that cannot work in a notebook: it shuts the server down when the cell ends, so anyone copying it gets a dead server by the next cell. Nothing in the package mentioned notebooks at all. Adds examples/serve_notebook.ipynb covering the lifecycle a kernel needs -- start with background=True and keep the handle, use it across cells, then shutdown() -- plus metadata and chunk reads, a byte range, and a refused PUT. Two arguments carry it: background=True runs uvicorn in a daemon thread with its own loop so the kernel's loop is untouched, and port=0 means re-running a start cell picks a new port instead of failing with "address already in use". A README section says the same in prose. The notebook is executed by the suite through nbclient, in a real kernel, and asserts its own expectations, so a behavior change fails there rather than in someone's notebook. Verified by mutation: making writes the default breaks the notebook's `assert refused.status_code == 405` and surfaces as a CellExecutionError naming the cell. examples/serve.py is now executed too, which required fixing the same fixed-port footgun the notebook section warns about -- it bound 8000, so it failed if anything else held that port. It runs in-process rather than under `uv run`, because its inline script metadata resolves zarr-http-server from git and would test main instead of the working tree. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): expose serve() for arbitrary ASGI apps Serving two nodes did not need two servers, but the only way to run several was to reach past this package: `serve_store`/`serve_node` each take exactly one store or node, so a composed app had no route to the background-server ergonomics -- `port=0` into `server.url`, and `shutdown()` -- only a blocking `uvicorn.run`. `_start_server` already did this for any Starlette app; it was just private. `serve(app, ...)` makes it public, with the same background/blocking overloads the shorthands have. `serve_store` and `serve_node` now delegate to it and stay, because they are the common case and are what the docs and examples use; retiring them is still available later. The split the pair muddles is now visible: what an app *serves* (`methods`, `cors_options`, `max_body_size`) is settled when the app is built, and `serve` only decides how it runs. Documents the three ways to serve several nodes -- serve their common parent group, serve the whole store, or mount separate apps and run the result -- with tests covering mounted nodes in *separate* stores, that each mount serves only its own data, and that `serve` runs the composed app in the background. Also derives the bounded-shutdown test's threshold from the timeouts that produce it. It hard-coded 3.0s, which was generous when shutdown could take at most 2x shutdown_timeout and marginal once the join margin was added -- it began failing under load rather than at the moment the constant changed. Assisted-by: ClaudeCode:claude-opus-5 * refactor(http-server)!: split blocking and background into two functions `background: bool` decided whether a call returns immediately with a handle or never returns at all -- the largest difference a call site can have, hidden in a keyword. The return type depended on it too, which is why every runner carried three @overload stanzas: nine in total, all of them working around that one flag, and `background=False` returned None, a value meaningless half the time. `serve(app)` now blocks and `serve_background(app)` returns a BackgroundServer. Neither needs an overload. Splitting forced the shorthand question, since the axes multiply: keeping serve_store/serve_node alongside two modes means six runner functions. The public surface is instead two builders and two runners -- `serve_background(store_app(store))` replaces `serve_store(store, background=True)`. That is one more call, and it puts the two halves where they belong: what an app serves is settled when it is built, and the runner only decides how it runs. _serve.py drops from 1301 to ~1100 lines with the duplicated signatures and docstrings gone. `serve_background` defaults to `port=0` where `serve` defaults to 8000. Deliberate: a background server is reached through `server.url`, and a fixed default makes starting a second one -- or re-running a notebook cell -- fail on a collision, while a blocking server usually wants a port others already know. BREAKING CHANGE: serve_store and serve_node are removed. The package is unreleased, so nothing depends on them yet. Assisted-by: ClaudeCode:claude-opus-5 * feat(http-server): default both runners to port="auto" `serve` defaulted to 8000 and `serve_background` to 0, which read as an arbitrary disagreement between two sibling functions about a shared parameter. Both now default to `"auto"`: prefer 8000, fall back to any free port if it is taken, and report the result through `server.url` and uvicorn's own startup line. What makes the fallback safe is that it applies only to the default. An explicit port still binds exactly that or fails, because a caller who names one usually has a proxy or a container port mapping expecting the server there -- silently moving would break it while looking healthy. `port=0` keeps its OS meaning of "any free port, no preference". The port is bound here and the socket handed to `Server.run(sockets=...)` rather than probing for a free port and passing uvicorn the number: probing releases the port before uvicorn claims it, which is the bind-then-close race that makes "find a free port" helpers flaky. Holding the socket means nothing can take it in between. `Config.port` is set to what was actually bound, so uvicorn's "running on ..." line does not name a port it is not serving. Two cases the mechanism has to respect: a `uds` or `fd` bind in uvicorn_options skips the TCP bind entirely, and the address family comes from `getaddrinfo` rather than a hard-coded AF_INET, which would bind the wrong family for an IPv6 host. Both are covered by tests, as is that an explicit taken port still raises. Assisted-by: ClaudeCode:claude-opus-5 * revert: drop changes to files this package does not own The PR should not reach outside packages/zarr-http-server and .github, and four files did. Two were unrelated churn: docs/api/zarr/experimental.md renamed a heading in the *core* zarr docs about zarr.experimental.cache_store, left over from when this server lived at zarr.experimental.serve, and uv.lock carried an idna bump nothing here asked for. Reverted; `uv lock --check` is clean. Two were premature rather than wrong. mkdocs.yml added a nav link to zarr-http-server.readthedocs.io and .readthedocs.yaml skipped the repo-root docs build for changes confined to this package. Both belong with a Read the Docs project that does not exist yet -- until it does, the nav link 404s and the build skip means neither site builds the package's docs. mkdocs.yml is also what GitHub reported a conflict on, which is what surfaced this. They should land in the follow-up that creates the RTD project. What remains outside the package is two root files that have nowhere else to live: the pre-push docs hook, since a pre-commit hook is necessarily repo-level, and a comment-only change to pyproject.toml noting that the release workflow's `zarr_http_server-v*` tags are among those the `git describe --match v*` filter exists to exclude. Assisted-by: ClaudeCode:claude-opus-5
feat: add the zarr-indexing package (TensorStore-style index transfor… …ms, ndsel wire format) (#4196) * fix: byte-order handling for structured dtypes in the bytes codec (#220) * fix: byte-order handling for structured dtypes in the bytes codec The bytes codec neither byte-swapped structured-dtype fields to its configured endian on encode (numpy reports byteorder '|' for void dtypes, so the top-level byteorder comparison never detected a mismatch) nor honored its endian when decoding, silently corrupting any structured data whose field byte order differed from the stored one (e.g. virtual references to external big-endian data). Encode now detects byte-order mismatches by comparing full dtypes via newbyteorder, and decode reinterprets raw bytes in the stored byte order before converting to the data type's declared byte order, so the stored layout (codec state) and the in-memory layout (array data type) are independent. Closes #4141 Assisted-by: ClaudeCode:claude-fable-5 * test: fold structured byte-order cases into existing bytes codec tests Extend test_endian's parametrization with structured dtypes and test_bytes_codec_sync_roundtrip with endian/dtype parametrization plus stored-layout and decoded-dtype assertions, instead of adding parallel test functions for the same properties. Assisted-by: ClaudeCode:claude-fable-5 * refactor: rename stored_dtype to view_dtype in BytesCodec decode The variable is the dtype used to view the raw chunk bytes (byte order from the codec's endian configuration), not a property of the stored data or of the returned buffer, which always carries the array's declared dtype. Assisted-by: ClaudeCode:claude-fable-5 * docs: note that the decode-side byte-order conversion copies the chunk Assisted-by: ClaudeCode:claude-fable-5 * feat: add the zarr-indexing package (TensorStore-style index transforms) Standalone workspace package extracted from the lazy-indexing branch (#3906): composable, lazy coordinate transforms (IndexTransform / IndexDomain / output maps), dependency-aware chunk resolution against a DimensionGridLike protocol, and an ndsel-conformant JSON wire format validated against the vendored conformance corpus. zarr itself does not depend on zarr-indexing yet — the runtime wiring lands separately once 0.1.0 is published. The package is numpy-only; its tests exercise chunk resolution against zarr's concrete ChunkGrid, so they run from the workspace root (uv sync --all-packages). Assisted-by: ClaudeCode:claude-fable-5 * style: conventional submodule import in the chunk-resolution tests Assisted-by: ClaudeCode:claude-fable-5 * perf(zarr-indexing): joint chunk enumeration for correlated vindex maps Candidate-chunk enumeration took the cartesian product of each correlated ArrayMap's per-dimension distinct chunk ids and relied on intersect() to filter untouched combinations. For a diagonal selection of P scattered points that is P**2 intersect calls — quadratic in the number of selected points, the same workload shape as #4174 (400 points: ~2.6s; 10k points: ~30min). Group correlated maps jointly instead: broadcast their per-point chunk ids, take the distinct rows (np.unique(axis=0), O(P log P)), and enumerate exactly the touched combinations. Candidate slots now carry chunk-coordinate tuples covering one or more output dimensions; orthogonal/constant/slice dimensions keep their existing per-dimension candidates. 400-point diagonal resolution drops from 2628ms to 14ms and scales linearly. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): standalone documentation site; add package justfile Mirror the treatment zarr-metadata received in #4208/#4210 onto zarr-indexing: a self-contained mkdocs site under the package (own mkdocs.yml, landing page, ndsel wire-format guide, mkdocstrings page per module, and .readthedocs.yaml for a dedicated RTD project), so the package presents as a separate project with docs versioned by its own zarr_indexing-v* release tags rather than zarr-python's. The zarr-python site's API Reference nav links out to it, and each RTD project now skips PR builds that do not touch its half of the repo. The package gains a pinned docs dependency group, a docs build job in its CI workflow, and a justfile with package-scoped dev recipes. Two recipes deviate from the zarr-metadata original by design: - `test` runs against the workspace-root environment (`uv run --project ../.. --all-packages --group test`), because the chunk-resolution tests exercise this package against zarr's chunk grids and `zarr` is deliberately not a dependency of this package. - `typecheck` uses plain `pyright`, unpinned and on the default interpreter, mirroring this package's own CI invocation. The zarr-metadata pin exists for a PEP 661 sentinel regression that zarr-indexing's sources do not hit. composition.py gains the module docstring the other modules already have, since mkdocstrings renders it as the page introduction. Assisted-by: ClaudeCode:claude-fable-5 * chore: drop the already-released 4141 changelog fragment The bytes-codec byte-order fix this fragment describes shipped upstream and its entry is already in docs/release-notes.md; the fragment survived on this branch only as a rebase remnant, and would emit a duplicate entry in the next release. Assisted-by: ClaudeCode:claude-fable-5 * docs(zarr-indexing): canonicalize ndsel references to zarr-developers/ndsel Also aligns the zarr-indexing workflow's setup-uv pin (v8.3.2) with the rest of the repo. The vendored-corpus sha is present upstream; the historical d-v-b/ndsel#1 PR reference stays as provenance. Assisted-by: ClaudeCode:claude-fable-5 * chore: drop the root uv-workspace wiring for zarr-indexing Per review: the root pyproject.toml should not change in this PR. The package now operates fully standalone (like zarr-metadata); the test invocations layer the package into the repo-root environment as an editable overlay instead (python -m pytest, since a base-env console script would not see the overlay). Assisted-by: ClaudeCode:claude-fable-5
docs: dev blog, performance examples, and compiled 3.3.0 release notes ( #4191) Adds a development blog to the documentation site with a 3.3.0 release post covering the FusedCodecPipeline and sharded partial-read coalescing, plus two runnable examples referenced by the post (examples/codec_pipeline_performance, examples/sharding_coalescing). Compiles every pending changelog fragment into a single 3.3.0 release notes section dated 2026-07-30, merging it with the section compiled earlier in #4148, and empties changes/ — making this commit suitable to tag as v3.3.0. Assisted-by: ClaudeCode:claude-fable-5
chore(zarr-metadata): build 0.4.0 changelog (#4211) Consume the pending news fragments (the #4119 model layer's feature and removal notes, plus a new doc fragment for the standalone documentation site and justfile from #4208/#4210) into CHANGELOG.md via towncrier for the zarr_metadata-v0.4.0 release. Assisted-by: ClaudeCode:claude-fable-5
PreviousNext