Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 68 additions & 0 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,74 @@
> `**Status:**`/`**Confidence:**` line. A correction gets its own new,
> dated entry that references the one it corrects — the storno rule.

## 2026-08-17 — E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1

**Status:** DOCTRINE (operator-stated, scope confirmed). **Confidence:** High —
four directives + three posters, restated and confirmed in session.

The blast radius, recorded because a session that reads this repo as "a faster
Java binding to a Rust library" will make locally-sensible decisions that are
globally wrong:

1. **The middle of the Java data stack is deleted, not wrapped.** Today:
App → DTO/ORM → Gremlin/TinkerPop → JanusGraph → Cassandra → Elastic /
ClickHouse / Lucene = six components, five serialization boundaries, three
mental models. After: **one** explicit ABI boundary, **zero** serialization
boundaries. The middleware and side-car analytics tiers do not get wrapped —
lance-graph + ndarray under one Panama membrane already *are* the traversal,
analytics and search substrate. *"Java als low-code Oberfläche, ABI als
Wahrheit."*
2. **Objects are eliminated, not optimized.** 10⁹ logical entities ⇒ **0** Java
objects: no header tax, no GC churn, masks instead of pointers, survivors
only touch heavy data. Valhalla's role is narrow and already measured here —
it makes the *tiny descriptor vocabulary* free (≤8 B flattens; the 16 B
entity does not), which is exactly why entities stay native and descriptors
stay `record`-shaped.
3. **The trust boundary collapses with the data boundary.** Mask-first: the
RBAC/ABAC clamp composes BEFORE execution, the scan runs on authorized lanes
only, and only aggregates/projections leave. Security enforced at the source
is a *consequence* of zero-copy, not a feature bolted on.
4. **The migration asymmetry is the weapon.** The developer-visible diff is
`stream().filter(λ)` → `.where(Field.gt(...))`; everything underneath changes
universe. Hence the standing rule: **the ABI is a machine membrane and never
the product API** — the product is the illusion that ordinary Java just works
at 10⁹ objects.

Operator's compression: *"Java Panama and Valhalla become the supraconductor
over lance-graph ABI shaped SoA substrate."* Supraconductor is precise — current
(the query) flows with no resistance (no allocation, no GC, no serialization)
through a thin familiar surface.

**Consequence for review:** any proposal that adds a serialization step, a
per-element crossing, an object materialization, or a post-filter security check
is not a tradeoff to weigh — it contradicts the thesis and is rejected.

## 2026-08-17 — E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1

**Status:** CORRECTION (of my own framing). **Confidence:** High — operator
correction, acted on the same session.

I answered the `simd_soa` question by measuring `MultiLaneColumn` against the
**flat three-lane fixture**, found two real API mismatches, and recorded a
"declined for now" verdict. The operator corrected the frame: *"the whole point
is Java should optimize the SoA layout — we won't dismiss the initial plans
just because you found it doesn't apply for unorganized non-SoA."*

The technical findings were right and are unchanged (see the entry below); the
**conclusion drawn from them was scoped wrong**. The flat fixture was always
scaffolding — `docs/abi.md` §10 and `architecture.md` said so from PR #1 ("the
generic fixture in this first slice was deliberately chosen … so the membrane's
physics could be proven independent of graph semantics"). Measuring a
substrate-shaped tool against the scaffolding and concluding "not yet" inverted
which one was provisional.

**The generalizable failure:** when a proposal doesn't fit the *current* code,
check whether the proposal is early or whether the **code is the placeholder**.
Here the code was the placeholder, and the right move was to build the real
shape (the 512-byte row store, W2, shipped same session) rather than defer the
tool. A "declined, revisit later" verdict is only honest when the thing it was
measured against is the thing that stays.

## 2026-08-17 — E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1

**Status:** DECISION (declined refactor, with the trigger for revisiting named).
Expand Down
45 changes: 45 additions & 0 deletions .claude/board/INTEGRATION_PLANS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,48 @@
## 2026-08-17 — consumer-{world-trades,bricks-analytics,graph-traversal}-v1 (PLANS; the three W5 consumer examples)

Plans: `.claude/plans/consumer-world-trades-v1.md`,
`consumer-bricks-analytics-v1.md`, `consumer-graph-traversal-v1.md`.

One operator poster made runnable per plan, all three over the SAME
substrate, each exercising a different face: the fluent domain API with
zero object allocation (trades); mask-first authorization where the RBAC
clamp composes BEFORE execution and only aggregates leave (bricks); and
traversal as facet addressing with crossings that scale with HOPS, not
rows (graph). Each carries its own falsifier set, including the
anti-vacuity requirement that a result be neither empty nor total.

**Sequencing:** all three are gated on `lgj-soa-substrate-v1` W3 (the
Java `RowStore` facade); after that they are independently shippable in
any order and none blocks the others. **Iron rule recorded in all three:**
a consumer example that needs a new ABI symbol goes back through the
substrate plan's wave process — the membrane never grows from the consumer
side.

**Status: PLANNED.**

## 2026-08-17 — lgj-soa-substrate-v1 (PLAN; the lance-graph-shaped SoA substrate)

Plan: `.claude/plans/lgj-soa-substrate-v1.md`. Successor to
`lgj-vertical-slice-v1` (COMPLETE, PRs #1-#4).

**What it covers:** the real layout — 64K × 512-byte rows, 32 facet lanes
of (4-byte classid + 12-byte payload) — wired end to end, in five waves:
W1 ndarray primitives (`iter_u32x16`, `eq_u32_strided_to_mask`), W2 the
Rust row store (`LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`,
`lgj_op_eq_classid`, `lgj_row_facet_match`, ABI minor 2, `abi.md` §11),
W3 the Java `RowStore` facade, W4 a Vector-API-vs-crossing bench on the
REAL layout, W5 the three consumer examples.

**Framing decision on record:** this plan exists because the flat
three-lane fixture was always scaffolding
(`E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`). The doctrine
it serves — the middle tier is deleted rather than wrapped, objects are
eliminated rather than optimized, security collapses into the data
boundary — is `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`.

**Status: ACTIVE.** W1 shipped (ndarray PR #279); W2 shipped (84/84,
18/18 symbols); W3 is the next action.

## 2026-08-17 — lgj-vertical-slice-v1 (PLAN; the first Panama×Valhalla×ndarray::simd proof)

Plan: `.claude/plans/lgj-vertical-slice-v1.md`. Active plan index — this
Expand Down
50 changes: 50 additions & 0 deletions .claude/board/LATEST_STATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,53 @@
## 2026-08-17 (Slice 2) — the SoA row store is REAL: ABI minor 2, W1+W2 shipped

**The reframing that started it** (operator, three directives): the flat
three-lane fixture was always scaffolding; Java is meant to optimize the *SoA
layout*; serialization is abandoned outright in favor of lance-graph's 64K
zero-copy concurrency + the ndarray SIMD polyfill; Panama+Valhalla are "the
supraconductor over lance-graph ABI shaped SoA substrate". Doctrine on the
board as `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`; my own mis-scoped
"declined" verdict corrected in
`E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`.

**Layout now in code** (operator-stated reference): 64K × **512 B rows, 32
facet lanes of 16 B = 4-byte LE classid + 12-byte payload**, the lance-graph
V3 content-blind facet. Java's own view may differ — these bytes are the
substrate truth. Full statement: `.claude/knowledge/soa-row-store-layout.md`.

- **W1 (ndarray PR #279, open):** `MultiLaneColumn::iter_u32x16`/`len_u32x16`
(the u32 lane whose absence was the real blocker) + `eq_u32_strided_to_mask`
(the AoS-facet classid scan, overflow-checked bounds). `simd_int_ops` 46/46,
`simd_soa` 15/15, `simd` 263/263, clippy/fmt clean, both x86 arms.
- **W2 (this repo):** `rowstore.rs` + `LGJ_RESOURCE_ROWSTORE` +
`lgj_rowstore_open` + `lgj_op_eq_classid` + `lgj_row_facet_match`; facet
lanes ride the **unchanged** `LgjLaneDesc` (`stride_bytes` has carried this
since minor 1). ABI **minor 1→2**, `docs/abi.md` §11 written, and the §1/§7
"14 symbols" count corrected (its own list already enumerated 15; the real
number is now 18, verified by `nm -D`). `cargo test` **84/84**, clippy
`-D warnings` + fmt clean.
- **Parity, three independent ways:** each SIMD kernel vs its independent
scalar reference over 10 row counts × 2 seeds × 4 facets × 4 needles, then
both cross-checked against `RowStore::classid_at`. Two-sided falsifier proves
payload bytes carrying the needle's bit pattern never satisfy a classid
match, and that a real classid match does fire.
- **`byte_len` semantics tightened** to the exact covered span
`(len-1)*stride + elem_bytes` — a full-stride final window would have let
Java bound a segment past the allocation's end on a facet lane. Contiguous
lanes unchanged.
- **Masks parent onto row stores**, so the entire existing mask algebra applies
with no new surface (proven end-to-end through the membrane).

**Planned and written this session:** `.claude/plans/lgj-soa-substrate-v1.md`
(the W1–W5 wave plan) plus one plan per consumer example —
`consumer-world-trades-v1.md` (zero-object fluent domain API),
`consumer-bricks-analytics-v1.md` (mask-first RBAC, fail-closed, aggregates
only), `consumer-graph-traversal-v1.md` (traversal as facet addressing,
crossings ∝ hops). Iron rule in all three: **a consumer example never grows the
membrane** — a needed symbol goes back through the wave process.

**Next:** W3, the Java `RowStore` facade (structured `MemoryLayout`,
minor-≥2 gate, `FacetMatchView`, generator-transcribing parity test).

## 2026-08-17 (later) — Phase I docs written, fusion re-run merged, simd_soa question answered (PR #4)

- **All four synthesis docs shipped** (`docs/architecture.md`,
Expand Down
20 changes: 20 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,26 @@
> anti-pattern the imported board rules name. Backfilled below in one
> pass rather than left stale; PR #4 onward gets its entry at merge time.

## PR #4 — Phase I synthesis docs + fusion re-run + board hygiene (merged 2026-08-17, squash `bd92c58`)

- **Added:** `docs/{architecture,panama,valhalla-lab,execution-boundary}.md`
(D-LGJ-I DONE — synthesis, each claim tied to its proving artifact);
the fusion-sweep 256-row re-run (`RESULTS.md` rewritten from
`jmh-results-merged.csv`, `TABLES.md` mechanically generated from the
same file); refreshed Valhalla lab result files (findings unchanged).
- **Locked:** the fusion self-correction — "fused ≈ unfused" was true
only at 65,536 rows; at 256 rows × 8 predicates unfused/fused = 2.99×.
The `MultiLaneColumn` decision
(`E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1`): declined
for flat lanes, earmarked for the 512-byte row-store slice; operator
layout reference (64K × 512 B, 32 × (4 B classid + 12 B)) recorded.
- **Deferred:** `NodeRow`/`WideFieldMask` wiring (unchanged);
`MultiLaneColumn` adoption gated on that slice.
- **Docs:** the four docs ARE the deliverable; board updated in-commit,
incl. this file's #1-#3 backfill (lapse owned above).
- **Confidence:** High — docs-only + measured data; both bot reviewers
(cursor, codex) hit usage limits and did not run.

## PR #3 — Vector API bench: real JMH, cross-checked (merged 2026-08-17, squash)

- **Added:** `bench/` — real JMH 1.37 suite (Components A/B/C/E:
Expand Down
13 changes: 13 additions & 0 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,16 @@ and D-LGJ-G are the only rows still open; they are deliberately NOT blocking
PR #1 (the core slice is independently complete and green) and will land as
their own PR once the Lab agent finishes and is reviewed with the same
rigor.

## lgj-soa-substrate-v1 — the lance-graph-shaped SoA substrate (2026-08-17)

Plan: `.claude/plans/lgj-soa-substrate-v1.md`. The 512-byte row / 32-facet
layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`.

| D-id | Deliverable | Status |
|---|---|---|
| D-LGJ-W1 | ndarray: `MultiLaneColumn::iter_u32x16`/`len_u32x16` + `eq_u32_strided_to_mask` (W1a contract) | **DONE 2026-08-17** — ndarray PR #279; `simd_int_ops` 46/46 (5 new strided tests incl. two `should_panic` bounds/overflow arms + stride-4 parity against the contiguous primitive), `simd_soa` 15/15, full `simd` 263/263, doctests, clippy `-D warnings` + fmt clean |
| D-LGJ-W2 | lgj-abi row store: `rowstore.rs`, `LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`, strided facet lanes through the unchanged `LgjLaneDesc`, `lgj_op_eq_classid`, `lgj_row_facet_match`, ABI minor 1→2, `docs/abi.md` §11 | **DONE 2026-08-17** — `cargo test` **84/84**, clippy/fmt clean, release build exports **18/18** symbols (`nm -D`). Parity: both kernels vs independent scalar references over 10 row counts × 2 seeds × 4 facets × 4 needles, cross-checked a THIRD way against `RowStore::classid_at`. Two-sided payload-vs-classid falsifier. End-to-end membrane test covers describe → predicate → mask algebra → count → facet-match → lifecycle |
| D-LGJ-W3 | Java `RowStore` facade: structured `MemoryLayout`, minor-≥2 gate, `FacetMatchView`, parity test transcribing the generator | **NEXT** |
| D-LGJ-W4 | Bench Component F: Vector API facet scan vs the crossing, on the REAL layout | Queued |
| D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | Planned, gated on W3 |
88 changes: 88 additions & 0 deletions .claude/knowledge/soa-row-store-layout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# The SoA row store layout — the substrate everything converges on

> **READ BY:** `abi-membrane-warden`, `simd-savant`, `panama-bridge-engineer`,
> `java-surface-warden`, and any session touching `rowstore.rs`,
> `docs/abi.md` §11, the Java `RowStore` facade, or a consumer example.
> **MANDATORY** before proposing any change to row geometry, facet
> semantics, or the lane map.

## The layout (operator-stated, 2026-08-17)

> *"the 64k x 512 bytes SoA layout is enforced everywhere in lance-graph
> (32 Lanes each 4 bytes classview+12 bytes). For Java the layout might
> differ just for reference."*

```
row (512 B) = 32 × facet (16 B)
facet (16 B) = classid (4 B, little-endian u32) ++ payload (12 B)
```

This is the lance-graph **V3 content-blind facet** — the same shape the
sibling repos pin as canon (`E-V3-FACET-4-PLUS-12`; the 12 bytes are an
axis-grouped byte register read as `6×(u8:u8)` / `4×(u8:u8:u8)` /
`3×(u8:u8:u8:u8)` per the ClassView, never widened). `lgj-abi` treats the
12 bytes as opaque **on purpose**: the ABI is a machine membrane and the
payload's *reading* is a ClassView concern one layer up.

**"For Java the layout might differ" is load-bearing.** The Java side is
free to project a different view (a structured `MemoryLayout`, a different
field grouping, a Valhalla-shaped descriptor vocabulary). These bytes are
the substrate truth; the Java view is a *reading* of them. Nothing in the
Java facade may assume its own view is the storage layout.

## The two readings, one buffer, zero copies

| reading | how it is addressed | who uses it |
|---|---|---|
| **row** | row `r` = bytes `r*512..(r+1)*512`; facet `f` at `+f*16` | `MultiLaneColumn::iter_u32x16` (4 facets per 64-B chunk), Java's structured layout |
| **facet lane** | strided u32 column: `first_offset = f*16`, `stride = 512` | `eq_u32_strided_to_mask`, `LgjLaneDesc` (which has carried `stride_bytes` since minor 1) |

Neither is a copy. The buffer is one `Arc<[u8]>`; a clone is a refcount
bump. **There is no serialization anywhere in this stack** — that is the
whole point (operator: *"abandon any use of serialization in favor of
lance-graph 64k concurrency zero copy and ndarray SIMD polyfill"*), and it
composes with lance-graph's own doctrine that an SoA envelope is zero-copy
from creation to Lance tombstone.

## Facts a session must not re-derive

- **`n_rows * 512` is always a multiple of 64** — so
`MultiLaneColumn::new` is infallible here *by construction*, not by
luck. Pinned by `the_buffer_is_exactly_n_times_512_bytes`.
- **Classids sit at `U32x16` positions 0, 4, 8, 12** of each 64-byte
chunk. The `& 0x1111` mask in the facet-match kernel is what keeps
payload bytes from ever satisfying a classid predicate — pinned
two-sided by `facet_match_ignores_needle_patterns_in_payload_bytes`.
- **`byte_len` is the EXACT covered span** `(len-1)*stride + elem_bytes`,
never `len * stride`. A facet lane's base sits `f*16` into the buffer, so
a full-stride final window would let Java bound a segment past the
allocation's end. Contiguous lanes reduce to the old formula unchanged.
- **`facet` ≠ `lane_id`.** Lane 0 is the raw buffer; facet `f`'s lane id is
`1 + f`. `lgj_op_eq_classid` takes a **facet index**. Pinned by the
end-to-end test asserting facet 32 is invalid while lane 32 is valid.
- **Masks parent onto row stores** exactly as onto patterns — both are
read-only, row-shaped resources — so the entire existing mask algebra
(`and`/`or`/`count`/`describe`, direct Java word writes) applies with no
new surface.
- **Alignment, honestly:** the base is `u8`-aligned (`Arc<[u8]>` promises
no more on stable Rust). Rows are strided at 512. Nothing here needs
more — Panama has unaligned value layouts, and every `ndarray::simd` load
is a register fill. The `align(64)` base guarantee arrives with real
`NodeRow` (`#[repr(C, align(64))]`) wiring, not before.

## Why `MultiLaneColumn` fits HERE and not in the flat fixture

Recorded because the answer flipped once already
(`E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1`):

- It requires `len % 64 == 0` with **no tail arm**. The flat fixture's
lanes are caller-sized `n_rows` of 4/8-byte elements — arbitrary. The
row store's buffer is `n*512` — always conforming.
- Its typed iterators are 64-byte chunk views, which is *exactly* a
four-facet group and *not* a natural fit for a flat column scan (where
`simd_int_ops`' own group-plus-scalar-tail loop is the right shape).

So both consumers are correct and neither is a workaround: use
`simd_int_ops` primitives for flat columns, `MultiLaneColumn` for the row
store. The u32 lane (`iter_u32x16`) was added to ndarray specifically to
close the gap that made the second impossible.
Loading