diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 4c7400c..542dd59 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -4,6 +4,74 @@ > `**Status:**`/`**Confidence:**` line. A correction gets its own new, > dated entry that references the one it corrects — the storno rule. +## 2026-08-17 — E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1 + +**Status:** DOCTRINE (operator-stated, scope confirmed). **Confidence:** High — +four directives + three posters, restated and confirmed in session. + +The blast radius, recorded because a session that reads this repo as "a faster +Java binding to a Rust library" will make locally-sensible decisions that are +globally wrong: + +1. **The middle of the Java data stack is deleted, not wrapped.** Today: + App → DTO/ORM → Gremlin/TinkerPop → JanusGraph → Cassandra → Elastic / + ClickHouse / Lucene = six components, five serialization boundaries, three + mental models. After: **one** explicit ABI boundary, **zero** serialization + boundaries. The middleware and side-car analytics tiers do not get wrapped — + lance-graph + ndarray under one Panama membrane already *are* the traversal, + analytics and search substrate. *"Java als low-code Oberfläche, ABI als + Wahrheit."* +2. **Objects are eliminated, not optimized.** 10⁹ logical entities ⇒ **0** Java + objects: no header tax, no GC churn, masks instead of pointers, survivors + only touch heavy data. Valhalla's role is narrow and already measured here — + it makes the *tiny descriptor vocabulary* free (≤8 B flattens; the 16 B + entity does not), which is exactly why entities stay native and descriptors + stay `record`-shaped. +3. **The trust boundary collapses with the data boundary.** Mask-first: the + RBAC/ABAC clamp composes BEFORE execution, the scan runs on authorized lanes + only, and only aggregates/projections leave. Security enforced at the source + is a *consequence* of zero-copy, not a feature bolted on. +4. **The migration asymmetry is the weapon.** The developer-visible diff is + `stream().filter(λ)` → `.where(Field.gt(...))`; everything underneath changes + universe. Hence the standing rule: **the ABI is a machine membrane and never + the product API** — the product is the illusion that ordinary Java just works + at 10⁹ objects. + +Operator's compression: *"Java Panama and Valhalla become the supraconductor +over lance-graph ABI shaped SoA substrate."* Supraconductor is precise — current +(the query) flows with no resistance (no allocation, no GC, no serialization) +through a thin familiar surface. + +**Consequence for review:** any proposal that adds a serialization step, a +per-element crossing, an object materialization, or a post-filter security check +is not a tradeoff to weigh — it contradicts the thesis and is rejected. + +## 2026-08-17 — E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1 + +**Status:** CORRECTION (of my own framing). **Confidence:** High — operator +correction, acted on the same session. + +I answered the `simd_soa` question by measuring `MultiLaneColumn` against the +**flat three-lane fixture**, found two real API mismatches, and recorded a +"declined for now" verdict. The operator corrected the frame: *"the whole point +is Java should optimize the SoA layout — we won't dismiss the initial plans +just because you found it doesn't apply for unorganized non-SoA."* + +The technical findings were right and are unchanged (see the entry below); the +**conclusion drawn from them was scoped wrong**. The flat fixture was always +scaffolding — `docs/abi.md` §10 and `architecture.md` said so from PR #1 ("the +generic fixture in this first slice was deliberately chosen … so the membrane's +physics could be proven independent of graph semantics"). Measuring a +substrate-shaped tool against the scaffolding and concluding "not yet" inverted +which one was provisional. + +**The generalizable failure:** when a proposal doesn't fit the *current* code, +check whether the proposal is early or whether the **code is the placeholder**. +Here the code was the placeholder, and the right move was to build the real +shape (the 512-byte row store, W2, shipped same session) rather than defer the +tool. A "declined, revisit later" verdict is only honest when the thing it was +measured against is the thing that stays. + ## 2026-08-17 — E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1 **Status:** DECISION (declined refactor, with the trigger for revisiting named). diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 4419559..65043dd 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,48 @@ +## 2026-08-17 — consumer-{world-trades,bricks-analytics,graph-traversal}-v1 (PLANS; the three W5 consumer examples) + +Plans: `.claude/plans/consumer-world-trades-v1.md`, +`consumer-bricks-analytics-v1.md`, `consumer-graph-traversal-v1.md`. + +One operator poster made runnable per plan, all three over the SAME +substrate, each exercising a different face: the fluent domain API with +zero object allocation (trades); mask-first authorization where the RBAC +clamp composes BEFORE execution and only aggregates leave (bricks); and +traversal as facet addressing with crossings that scale with HOPS, not +rows (graph). Each carries its own falsifier set, including the +anti-vacuity requirement that a result be neither empty nor total. + +**Sequencing:** all three are gated on `lgj-soa-substrate-v1` W3 (the +Java `RowStore` facade); after that they are independently shippable in +any order and none blocks the others. **Iron rule recorded in all three:** +a consumer example that needs a new ABI symbol goes back through the +substrate plan's wave process — the membrane never grows from the consumer +side. + +**Status: PLANNED.** + +## 2026-08-17 — lgj-soa-substrate-v1 (PLAN; the lance-graph-shaped SoA substrate) + +Plan: `.claude/plans/lgj-soa-substrate-v1.md`. Successor to +`lgj-vertical-slice-v1` (COMPLETE, PRs #1-#4). + +**What it covers:** the real layout — 64K × 512-byte rows, 32 facet lanes +of (4-byte classid + 12-byte payload) — wired end to end, in five waves: +W1 ndarray primitives (`iter_u32x16`, `eq_u32_strided_to_mask`), W2 the +Rust row store (`LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`, +`lgj_op_eq_classid`, `lgj_row_facet_match`, ABI minor 2, `abi.md` §11), +W3 the Java `RowStore` facade, W4 a Vector-API-vs-crossing bench on the +REAL layout, W5 the three consumer examples. + +**Framing decision on record:** this plan exists because the flat +three-lane fixture was always scaffolding +(`E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`). The doctrine +it serves — the middle tier is deleted rather than wrapped, objects are +eliminated rather than optimized, security collapses into the data +boundary — is `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`. + +**Status: ACTIVE.** W1 shipped (ndarray PR #279); W2 shipped (84/84, +18/18 symbols); W3 is the next action. + ## 2026-08-17 — lgj-vertical-slice-v1 (PLAN; the first Panama×Valhalla×ndarray::simd proof) Plan: `.claude/plans/lgj-vertical-slice-v1.md`. Active plan index — this diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 66f2e2e..941243c 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,53 @@ +## 2026-08-17 (Slice 2) — the SoA row store is REAL: ABI minor 2, W1+W2 shipped + +**The reframing that started it** (operator, three directives): the flat +three-lane fixture was always scaffolding; Java is meant to optimize the *SoA +layout*; serialization is abandoned outright in favor of lance-graph's 64K +zero-copy concurrency + the ndarray SIMD polyfill; Panama+Valhalla are "the +supraconductor over lance-graph ABI shaped SoA substrate". Doctrine on the +board as `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`; my own mis-scoped +"declined" verdict corrected in +`E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`. + +**Layout now in code** (operator-stated reference): 64K × **512 B rows, 32 +facet lanes of 16 B = 4-byte LE classid + 12-byte payload**, the lance-graph +V3 content-blind facet. Java's own view may differ — these bytes are the +substrate truth. Full statement: `.claude/knowledge/soa-row-store-layout.md`. + +- **W1 (ndarray PR #279, open):** `MultiLaneColumn::iter_u32x16`/`len_u32x16` + (the u32 lane whose absence was the real blocker) + `eq_u32_strided_to_mask` + (the AoS-facet classid scan, overflow-checked bounds). `simd_int_ops` 46/46, + `simd_soa` 15/15, `simd` 263/263, clippy/fmt clean, both x86 arms. +- **W2 (this repo):** `rowstore.rs` + `LGJ_RESOURCE_ROWSTORE` + + `lgj_rowstore_open` + `lgj_op_eq_classid` + `lgj_row_facet_match`; facet + lanes ride the **unchanged** `LgjLaneDesc` (`stride_bytes` has carried this + since minor 1). ABI **minor 1→2**, `docs/abi.md` §11 written, and the §1/§7 + "14 symbols" count corrected (its own list already enumerated 15; the real + number is now 18, verified by `nm -D`). `cargo test` **84/84**, clippy + `-D warnings` + fmt clean. +- **Parity, three independent ways:** each SIMD kernel vs its independent + scalar reference over 10 row counts × 2 seeds × 4 facets × 4 needles, then + both cross-checked against `RowStore::classid_at`. Two-sided falsifier proves + payload bytes carrying the needle's bit pattern never satisfy a classid + match, and that a real classid match does fire. +- **`byte_len` semantics tightened** to the exact covered span + `(len-1)*stride + elem_bytes` — a full-stride final window would have let + Java bound a segment past the allocation's end on a facet lane. Contiguous + lanes unchanged. +- **Masks parent onto row stores**, so the entire existing mask algebra applies + with no new surface (proven end-to-end through the membrane). + +**Planned and written this session:** `.claude/plans/lgj-soa-substrate-v1.md` +(the W1–W5 wave plan) plus one plan per consumer example — +`consumer-world-trades-v1.md` (zero-object fluent domain API), +`consumer-bricks-analytics-v1.md` (mask-first RBAC, fail-closed, aggregates +only), `consumer-graph-traversal-v1.md` (traversal as facet addressing, +crossings ∝ hops). Iron rule in all three: **a consumer example never grows the +membrane** — a needed symbol goes back through the wave process. + +**Next:** W3, the Java `RowStore` facade (structured `MemoryLayout`, +minor-≥2 gate, `FacetMatchView`, generator-transcribing parity test). + ## 2026-08-17 (later) — Phase I docs written, fusion re-run merged, simd_soa question answered (PR #4) - **All four synthesis docs shipped** (`docs/architecture.md`, diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index eacf076..b6451d4 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -8,6 +8,26 @@ > anti-pattern the imported board rules name. Backfilled below in one > pass rather than left stale; PR #4 onward gets its entry at merge time. +## PR #4 — Phase I synthesis docs + fusion re-run + board hygiene (merged 2026-08-17, squash `bd92c58`) + +- **Added:** `docs/{architecture,panama,valhalla-lab,execution-boundary}.md` + (D-LGJ-I DONE — synthesis, each claim tied to its proving artifact); + the fusion-sweep 256-row re-run (`RESULTS.md` rewritten from + `jmh-results-merged.csv`, `TABLES.md` mechanically generated from the + same file); refreshed Valhalla lab result files (findings unchanged). +- **Locked:** the fusion self-correction — "fused ≈ unfused" was true + only at 65,536 rows; at 256 rows × 8 predicates unfused/fused = 2.99×. + The `MultiLaneColumn` decision + (`E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1`): declined + for flat lanes, earmarked for the 512-byte row-store slice; operator + layout reference (64K × 512 B, 32 × (4 B classid + 12 B)) recorded. +- **Deferred:** `NodeRow`/`WideFieldMask` wiring (unchanged); + `MultiLaneColumn` adoption gated on that slice. +- **Docs:** the four docs ARE the deliverable; board updated in-commit, + incl. this file's #1-#3 backfill (lapse owned above). +- **Confidence:** High — docs-only + measured data; both bot reviewers + (cursor, codex) hit usage limits and did not run. + ## PR #3 — Vector API bench: real JMH, cross-checked (merged 2026-08-17, squash) - **Added:** `bench/` — real JMH 1.37 suite (Components A/B/C/E: diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 0f6883e..fd5f122 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -28,3 +28,16 @@ and D-LGJ-G are the only rows still open; they are deliberately NOT blocking PR #1 (the core slice is independently complete and green) and will land as their own PR once the Lab agent finishes and is reviewed with the same rigor. + +## lgj-soa-substrate-v1 — the lance-graph-shaped SoA substrate (2026-08-17) + +Plan: `.claude/plans/lgj-soa-substrate-v1.md`. The 512-byte row / 32-facet +layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`. + +| D-id | Deliverable | Status | +|---|---|---| +| D-LGJ-W1 | ndarray: `MultiLaneColumn::iter_u32x16`/`len_u32x16` + `eq_u32_strided_to_mask` (W1a contract) | **DONE 2026-08-17** — ndarray PR #279; `simd_int_ops` 46/46 (5 new strided tests incl. two `should_panic` bounds/overflow arms + stride-4 parity against the contiguous primitive), `simd_soa` 15/15, full `simd` 263/263, doctests, clippy `-D warnings` + fmt clean | +| D-LGJ-W2 | lgj-abi row store: `rowstore.rs`, `LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`, strided facet lanes through the unchanged `LgjLaneDesc`, `lgj_op_eq_classid`, `lgj_row_facet_match`, ABI minor 1→2, `docs/abi.md` §11 | **DONE 2026-08-17** — `cargo test` **84/84**, clippy/fmt clean, release build exports **18/18** symbols (`nm -D`). Parity: both kernels vs independent scalar references over 10 row counts × 2 seeds × 4 facets × 4 needles, cross-checked a THIRD way against `RowStore::classid_at`. Two-sided payload-vs-classid falsifier. End-to-end membrane test covers describe → predicate → mask algebra → count → facet-match → lifecycle | +| D-LGJ-W3 | Java `RowStore` facade: structured `MemoryLayout`, minor-≥2 gate, `FacetMatchView`, parity test transcribing the generator | **NEXT** | +| D-LGJ-W4 | Bench Component F: Vector API facet scan vs the crossing, on the REAL layout | Queued | +| D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | Planned, gated on W3 | diff --git a/.claude/knowledge/soa-row-store-layout.md b/.claude/knowledge/soa-row-store-layout.md new file mode 100644 index 0000000..86484b3 --- /dev/null +++ b/.claude/knowledge/soa-row-store-layout.md @@ -0,0 +1,88 @@ +# The SoA row store layout — the substrate everything converges on + +> **READ BY:** `abi-membrane-warden`, `simd-savant`, `panama-bridge-engineer`, +> `java-surface-warden`, and any session touching `rowstore.rs`, +> `docs/abi.md` §11, the Java `RowStore` facade, or a consumer example. +> **MANDATORY** before proposing any change to row geometry, facet +> semantics, or the lane map. + +## The layout (operator-stated, 2026-08-17) + +> *"the 64k x 512 bytes SoA layout is enforced everywhere in lance-graph +> (32 Lanes each 4 bytes classview+12 bytes). For Java the layout might +> differ just for reference."* + +``` +row (512 B) = 32 × facet (16 B) +facet (16 B) = classid (4 B, little-endian u32) ++ payload (12 B) +``` + +This is the lance-graph **V3 content-blind facet** — the same shape the +sibling repos pin as canon (`E-V3-FACET-4-PLUS-12`; the 12 bytes are an +axis-grouped byte register read as `6×(u8:u8)` / `4×(u8:u8:u8)` / +`3×(u8:u8:u8:u8)` per the ClassView, never widened). `lgj-abi` treats the +12 bytes as opaque **on purpose**: the ABI is a machine membrane and the +payload's *reading* is a ClassView concern one layer up. + +**"For Java the layout might differ" is load-bearing.** The Java side is +free to project a different view (a structured `MemoryLayout`, a different +field grouping, a Valhalla-shaped descriptor vocabulary). These bytes are +the substrate truth; the Java view is a *reading* of them. Nothing in the +Java facade may assume its own view is the storage layout. + +## The two readings, one buffer, zero copies + +| reading | how it is addressed | who uses it | +|---|---|---| +| **row** | row `r` = bytes `r*512..(r+1)*512`; facet `f` at `+f*16` | `MultiLaneColumn::iter_u32x16` (4 facets per 64-B chunk), Java's structured layout | +| **facet lane** | strided u32 column: `first_offset = f*16`, `stride = 512` | `eq_u32_strided_to_mask`, `LgjLaneDesc` (which has carried `stride_bytes` since minor 1) | + +Neither is a copy. The buffer is one `Arc<[u8]>`; a clone is a refcount +bump. **There is no serialization anywhere in this stack** — that is the +whole point (operator: *"abandon any use of serialization in favor of +lance-graph 64k concurrency zero copy and ndarray SIMD polyfill"*), and it +composes with lance-graph's own doctrine that an SoA envelope is zero-copy +from creation to Lance tombstone. + +## Facts a session must not re-derive + +- **`n_rows * 512` is always a multiple of 64** — so + `MultiLaneColumn::new` is infallible here *by construction*, not by + luck. Pinned by `the_buffer_is_exactly_n_times_512_bytes`. +- **Classids sit at `U32x16` positions 0, 4, 8, 12** of each 64-byte + chunk. The `& 0x1111` mask in the facet-match kernel is what keeps + payload bytes from ever satisfying a classid predicate — pinned + two-sided by `facet_match_ignores_needle_patterns_in_payload_bytes`. +- **`byte_len` is the EXACT covered span** `(len-1)*stride + elem_bytes`, + never `len * stride`. A facet lane's base sits `f*16` into the buffer, so + a full-stride final window would let Java bound a segment past the + allocation's end. Contiguous lanes reduce to the old formula unchanged. +- **`facet` ≠ `lane_id`.** Lane 0 is the raw buffer; facet `f`'s lane id is + `1 + f`. `lgj_op_eq_classid` takes a **facet index**. Pinned by the + end-to-end test asserting facet 32 is invalid while lane 32 is valid. +- **Masks parent onto row stores** exactly as onto patterns — both are + read-only, row-shaped resources — so the entire existing mask algebra + (`and`/`or`/`count`/`describe`, direct Java word writes) applies with no + new surface. +- **Alignment, honestly:** the base is `u8`-aligned (`Arc<[u8]>` promises + no more on stable Rust). Rows are strided at 512. Nothing here needs + more — Panama has unaligned value layouts, and every `ndarray::simd` load + is a register fill. The `align(64)` base guarantee arrives with real + `NodeRow` (`#[repr(C, align(64))]`) wiring, not before. + +## Why `MultiLaneColumn` fits HERE and not in the flat fixture + +Recorded because the answer flipped once already +(`E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1`): + +- It requires `len % 64 == 0` with **no tail arm**. The flat fixture's + lanes are caller-sized `n_rows` of 4/8-byte elements — arbitrary. The + row store's buffer is `n*512` — always conforming. +- Its typed iterators are 64-byte chunk views, which is *exactly* a + four-facet group and *not* a natural fit for a flat column scan (where + `simd_int_ops`' own group-plus-scalar-tail loop is the right shape). + +So both consumers are correct and neither is a workaround: use +`simd_int_ops` primitives for flat columns, `MultiLaneColumn` for the row +store. The u32 lane (`iter_u32x16`) was added to ndarray specifically to +close the gap that made the second impossible. diff --git a/.claude/plans/consumer-bricks-analytics-v1.md b/.claude/plans/consumer-bricks-analytics-v1.md new file mode 100644 index 0000000..dafd8a6 --- /dev/null +++ b/.claude/plans/consumer-bricks-analytics-v1.md @@ -0,0 +1,67 @@ +# consumer-bricks-analytics-v1 — mask-first security over the row store + +> **Status: PLANNED** (2026-08-17). W5 consumer example #2, from the +> operator's "OGAR-Bricks + lance-graph + Panama + Valhalla" poster. +> Gated on `lgj-soa-substrate-v1.md` W3. + +## What it proves + +The poster's structural claim: **security is a mask composed BEFORE +execution, not a post-filter on rows that already crossed the trust +boundary.** Runnable shape: + +```java +var orders = Bricks.table(Orders.class); +var result = orders + .where(Orders.REGION.eq("EU")) + .where(Orders.YEAR.eq(2026)) + .authorize(currentRole) // ← the mask-first clamp, BEFORE execution + .groupBy(Orders.PRODUCT) + .sum(Orders.REVENUE); +``` + +Only aggregates leave the boundary; raw rows never do. + +## Design (constraints, not code) + +- **`authorize(role)` composes an ADDITIONAL predicate into the SAME lazy + chain** — it is not a separate enforcement pass over already-fetched + data. Concretely: a role resolves to an allowed-facet-classid set (a + small, deterministic table — no new ABI symbol, expressible as an OR of + `Orders..eq(...)` over the existing `where` vocabulary, or as a + precomputed mask ANDed in via the existing `lgj_mask_and`). Either + encoding is legal; the falsifier below is what matters, not the + mechanism. +- **RBAC clamp happens where the OTHER predicates happen** — inside the ONE + fused crossing. There is no code path where a row's bytes are readable by + Java before the mask that would exclude it has been applied. This is the + operational meaning of "mask-first" and it is what the falsifier checks. +- **Reuses the row-store facet-match kernel for the "which fields visible" + half** — a role that can see some facets of a row but not others (partial + visibility) is a `lgj_row_facet_match`-shaped question; a role that can + see some ROWS but not others is a mask-composition question. This example + demonstrates the row-visibility case (simpler, no new kernel needed); + facet-level field masking is named as a documented extension, not built + here. + +## Falsifiers + +1. **No raw-row read before authorize.** Instrument (or structurally + prove via the API surface — no accessor exists that reads a row's bytes + before a terminal op runs) that between `.where(...)` calls and the + terminal aggregate, zero bytes of any EXCLUDED row are ever read into a + Java-visible value. The `LazinessTest` 0-crossings-while-composing + proof already gives half of this "for free" — extend it to prove + `authorize()` composes rather than executes. +2. **A caller who never calls `authorize()` gets an explicit refusal**, not + an unauthorized default (fail-closed — mirrors the a2ui-rs/lance-graph + RBAC doctrine already in this workspace's CLAUDE.md: "a missing/narrow + role mask never falls back to emit everything"). +3. **Two roles, same query, different counts** — a real two-sided + discrimination test (a role that can see 0 rows of a restricted region + must count 0; a role that can see all of it must match the unrestricted + query) — the anti-vacuity discipline from this workspace's falsifier + rules, applied to authorization instead of a table filter. +4. **Aggregate-only egress**: the public return type of a `Bricks` query is + never `Row`/`Trade`-shaped when a `groupBy`/`sum` terminal was used — + checked by the API surface test, same mechanism as `ApiSurfaceTest`. diff --git a/.claude/plans/consumer-graph-traversal-v1.md b/.claude/plans/consumer-graph-traversal-v1.md new file mode 100644 index 0000000..1aad892 --- /dev/null +++ b/.claude/plans/consumer-graph-traversal-v1.md @@ -0,0 +1,70 @@ +# consumer-graph-traversal-v1 — traversal as facet addressing, no middleware + +> **Status: PLANNED** (2026-08-17). W5 consumer example #3, from the +> operator's "Java Graph Stack (Heute) vs Project Panama + lance-graph +> (richtig gemacht)" poster. Gated on `lgj-soa-substrate-v1.md` W3. + +## What it proves + +The poster's BEFORE chain is six components and five serialization +boundaries: App → DTO/ORM → TinkerPop/Gremlin → JanusGraph → Cassandra → +Elastic/ClickHouse/Lucene. The AFTER chain is **one** explicit ABI boundary +and **zero** serialization boundaries — *"Java als low-code Oberfläche, ABI +als Wahrheit."* + +Runnable shape (a 2-hop neighbourhood, entirely in masks): + +```java +var g = Graph.open(store); +var friendsOfFriends = g.from(seedRows) // a mask + .hop(Edge.KNOWS) // facet-addressed, one crossing + .hop(Edge.KNOWS) + .minus(seedRows) + .count(); +``` + +No `Vertex` object, no `Edge` object, no Gremlin step compiler, no +serialization between hops — a hop is a mask transformation over the SAME +un-copied bytes. + +## Design (constraints, not code) + +- **An edge is an ADDRESS, not an object.** In the 512-byte row, a facet's + 4-byte classid names *what kind of relation this slot holds*; the 12-byte + payload carries the target address. That is exactly the workspace canon + ("a relation is a class; an edge's predicate is a classid reference" — + MedCare-rs commitment #10 / the OGAR EdgeBlock doctrine) expressed in the + facet register. `Edge.KNOWS` is therefore a *classid constant*, and + "which of this row's slots are KNOWS edges" is precisely + `lgj_row_facet_match` — already built in W2. +- **A hop is: facet-match → decode targets → build the next mask.** The + first half exists. The second half (target decode + scatter into a mask) + is the ONE genuinely new capability this example needs, and it is a + *bulk* operation by construction (one crossing per hop, work ∝ rows). If + it cannot be expressed with the existing symbols, it goes back through + `lgj-soa-substrate-v1.md`'s wave process as a proposed W6 ABI addition — + **not** added ad hoc from the consumer side. Naming it here is the + design decision; building it is gated. +- **The comparison is the point.** This example carries an explicit + boundary-count table in its output: components traversed, serialization + boundaries crossed, objects allocated — measured, against the poster's + BEFORE column as the stated baseline (which is cited as *architecture*, + not benchmarked here; we do not claim measured numbers for a JanusGraph + stack we did not run). + +## Falsifiers + +1. **Hop correctness**: the 2-hop neighbourhood equals a transcribed, + plain-Java breadth-first walk over the same generated data — an + independent computation, not a golden blob. +2. **Zero serialization**: no `byte[]`, no `toArray`, no JSON/proto on the + hop path — enforced the way the bench enforces it (an explicit rule in + the test, plus the API surface test for the public types). +3. **Crossings scale with HOPS, not with rows or edges**: instrument the + downcall count and assert it equals the hop count (+1 terminal), + independent of row count — the anti-JNI property, stated as an + assertion rather than as prose. +4. **Anti-vacuity**: the seed set, the 1-hop set and the 2-hop set must be + three *different, non-empty, non-total* sizes — a traversal that + returned everything or nothing would satisfy a naive equality test while + proving nothing. diff --git a/.claude/plans/consumer-world-trades-v1.md b/.claude/plans/consumer-world-trades-v1.md new file mode 100644 index 0000000..c66e40d --- /dev/null +++ b/.claude/plans/consumer-world-trades-v1.md @@ -0,0 +1,58 @@ +# consumer-world-trades-v1 — "One Billion Objects. Zero Objects." + +> **Status: PLANNED** (2026-08-17). W5 consumer example #1, from the +> operator's "One Billion Objects in Java — Before vs After" poster. +> Gated on `lgj-soa-substrate-v1.md` W3 (the Java `RowStore` facade). + +## What it proves + +The poster's AFTER column, runnable: + +```java +var trades = World.open(Trade.class); // ← a RowStore, not a loadTrades() +long count = trades + .where(Trade.QUANTITY.gt(1000)) + .where(Trade.VENUE.eq(XETRA)) + .where(Trade.PRICE.gt(threshold)) + .count(); +``` + +Developer sees: familiar fluent Java, domain language, no serialization. +What actually happens: compose lens (0 crossings) → one fused evaluation → +packed mask → count. **Java objects allocated for N logical trades: 0** — +and that number is asserted by measurement +(`getThreadAllocatedBytes`, the valhalla-lab instrument), not claimed. + +## Design (constraints, not code) + +- **`World.open(Class)` is a schema binding, not a loader.** The class + is a *description*: static typed field descriptors (`Trade.QUANTITY`) + carrying (facet index, element kind, offset-within-facet). It maps the + domain vocabulary onto the 32-facet row; no instance of `Trade` is ever + constructed. This is the poster's "ClassView (Semantics)" cell scaled to + the fixture — a REAL lance-graph ClassView binding replaces it in a + later slice without changing consumer code. +- **Field descriptors are the Valhalla-shaped vocabulary** — tiny, + identity-free, `record`-shaped, ≤8B payload where possible (the measured + flattening cliff), migrating to `value record` by one word when JEP 401 + ships. The three-truths lab already proved this is the ONE place + Valhalla pays here. +- **Reuses `View`/`Predicate`/`Mask` machinery** — the fluent chain stays + lazy (0 crossings to compose, LazinessTest discipline), fuses to one + plan, crosses once. No new membrane surface expected; if one turns out + to be needed (e.g. a fused plan over facet lanes), it goes through the + substrate plan's wave process first. +- **The demo scale is honest**: 64K rows in-repo CI; the 10⁶+ row arm runs + as a bench/example, not a unit test. + +## Falsifiers + +1. Count parity: the fluent chain's answer == a transcribed-generator + recomputation in plain Java (no substrate involvement). +2. Zero-allocation: measured allocated bytes for the query path below a + fixed small constant (the descriptors + the mask handle), regardless of + row count — the row-count-independence IS the assertion. +3. Laziness: crossing count 0 while composing, exactly 1 at the terminal + (the LazinessTest instrument, reused). +4. API surface: reflection test — nothing in the consumer-visible API + mentions FFM, facet indices, or lane ids. diff --git a/.claude/plans/lgj-soa-substrate-v1.md b/.claude/plans/lgj-soa-substrate-v1.md new file mode 100644 index 0000000..6feed96 --- /dev/null +++ b/.claude/plans/lgj-soa-substrate-v1.md @@ -0,0 +1,82 @@ +# lgj-soa-substrate-v1 — Slice 2: the lance-graph-shaped SoA substrate + +> **Status: ACTIVE** (2026-08-17). Successor to `lgj-vertical-slice-v1.md`, +> which is COMPLETE (PRs #1–#4 merged). Operator directives that reframed +> this slice, in order: +> 1. *"the 64k x 512 bytes SoA layout is enforced everywhere in lance-graph +> (32 Lanes each 4 bytes classview+12 bytes). For Java the layout might +> differ — just for reference."* +> 2. *"Java should optimize the SoA layout — we won't dismiss the initial +> plans just because it doesn't apply for unorganized non-SoA; that's +> the whole point about project Panama."* +> 3. *"Abandon any use of serialization in favor of lance-graph 64k +> concurrency zero copy and ndarray SIMD polyfill — the low code low +> migration cost experience."* +> 4. *"Java Panama and Valhalla become the supraconductor over lance-graph +> ABI shaped SoA substrate."* +> +> Blast radius (operator-confirmed, three posters): this is not a faster +> binding — it is the deletion of the Java data stack's middle tier (the +> ORM/DTO layer, graph middleware, serialization frameworks, side-car +> analytics) in favor of ONE ABI boundary over ONE substrate, with the JVM +> kept as the familiar low-migration-cost surface. The formula: +> `ClassView → WideFieldMask → Meta Gate (64K) → SIMD Sweep → SoA Lanes → +> Survivors → Seal & Persist (Lance)`. + +## The design waves + +| wave | deliverable | status | +|---|---|---| +| **W1** | ndarray: `MultiLaneColumn` u32 lane (`iter_u32x16`) + `eq_u32_strided_to_mask` (W1a contract: parity tests, re-exports, both x86 arms) | **DONE** — ndarray PR #279; 46+15 tests, clippy/fmt clean | +| **W2** | lgj-abi row store: `LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`, facet lanes via the EXISTING `LgjLaneDesc` (`stride=512`), `lgj_op_eq_classid` (row masks composing with the existing algebra), `lgj_row_facet_match` (per-row facet bitsets into a caller buffer via `MultiLaneColumn`), ABI minor 1→2, `docs/abi.md` §11 | **DONE** — 84/84 incl. end-to-end membrane test, 18/18 symbols via `nm -D` | +| **W3** | Java `RowStore` facade: no FFM in public signatures; structured `MemoryLayout` (`sequence(32, struct(u32 classid, 12B payload))`); minor-≥2 gate; `FacetMatchView` zero-copy accessor over a Java-arena segment; `RowStoreParityTest` transcribing the generator | OPEN — next | +| **W4** | Bench Component F: Java Vector API per-row facet scan (one `IntVector` 16-lane chunk = 4 facets, same algorithm as the Rust kernel) vs `lgj_row_facet_match` crossing vs scalar VarHandle walk — the "where does execution belong" question re-asked on the REAL layout | OPEN | +| **W5** | The three consumer examples (own plan files, below) | PLANNED | + +Wave rule (house style): one wave = one reviewable PR; gates run centrally +(orchestrator only — agents never run cargo); every safety property lands +disable-verified, every measured claim lands with its reproduction command. + +## What W2 locked (so W3+ doesn't re-derive it) + +- **Layout truth:** `ROW_BYTES=512`, `ROW_FACETS=32`, `FACET_BYTES=16`, + classid = leading LE u32. Generator: 2 SplitMix64 draws per facet + (`a`→classid via `(a>>>33)&0xF`, `b`+low-`a` → payload), 64 draws/row. +- **Lane map:** lane 0 = raw U8 contiguous; lane `1+f` = facet `f` classid, + U32, stride 512. `byte_len` = exact covered span + `(len-1)*stride + elem_bytes` — never rounds up past the allocation. +- **Masks parent onto row stores** exactly as onto patterns; the whole + existing mask algebra applies unchanged (proven in + `the_rowstore_slice_end_to_end_through_the_membrane`). +- **Carrier:** `Arc<[u8]>`, base u8-aligned (honest limit — the align(64) + guarantee arrives with real `NodeRow` wiring); `n*512 % 64 == 0` by + construction is what makes `MultiLaneColumn::new` infallible here. + +## The three consumer examples (W5) — one plan file each + +Each is one poster made runnable, on the SAME substrate, each exercising a +different face of it: + +| plan | poster | face of the substrate | +|---|---|---| +| `consumer-world-trades-v1.md` | "One Billion Objects in Java" | the fluent domain API: `World.open(...)` → schema-named fields → `.where().count()`, zero objects | +| `consumer-bricks-analytics-v1.md` | "OGAR-Bricks done right" | mask-first security: RBAC clamp BEFORE execution, survivors-only, aggregates leave | +| `consumer-graph-traversal-v1.md` | "Java Graph Stack (richtig gemacht)" | facet edges as addresses: traversal = facet-match + mask hops, no middleware | + +Sequencing: any order after W3; each is independently shippable; none +blocks the others. All three consume ONLY the public Java facade — a +consumer plan that needs a new ABI symbol goes back through this plan's +wave process instead of growing the membrane ad hoc. + +## Falsification obligations carried forward + +- W3 parity: Java recomputes classids from the transcribed generator AND + reads them back through the raw-lane segment — two independent paths to + the same numbers. +- W3 disable-run: break the minor-version gate (require ≥ 3) and prove + load fails; restore. +- W4 cross-check before timing: all three arms must agree on every + facet-match bitset before any timing is reported (the `Data.crossCheck` + discipline). +- Every consumer example ends with an assertion computed independently of + the substrate (transcribed-generator arithmetic), never a golden blob. diff --git a/docs/abi.md b/docs/abi.md index 8db9b72..3f7a665 100644 --- a/docs/abi.md +++ b/docs/abi.md @@ -62,8 +62,10 @@ cannot disagree with itself. The ABI is a **machine membrane**. It is not the product. The product is the Java semantic API (see `architecture.md`). Therefore: -- It is **small** — currently 14 symbols. Growth is a design smell to be argued - for, not a default. +- It is **small** — currently 18 symbols (minor 2; the "14" this line carried + at minor 1 was arithmetic drift — the §7 list it referred to already + enumerated 15). Growth is a design smell to be argued for, not a default; + minor 2's three additions are argued in §11. - It is **bulk-only**. Every call must be capable of doing work proportional to `n_rows` (see §6 — the anti-JNI rule). - It speaks **resource, lane, view, mask, operation, descriptor, status, @@ -76,7 +78,7 @@ semantic API (see `architecture.md`). Therefore: ``` LGJ_ABI_MAJOR = 0 // incompatible change ⇒ bump; Java refuses to load -LGJ_ABI_MINOR = 1 // additive change ⇒ bump; older Java may still load +LGJ_ABI_MINOR = 2 // additive change ⇒ bump; older Java may still load LGJ_MAGIC = 0x4C_47_4A_5F_41_42_49_00 // "LGJ_ABI\0" big-endian-read ``` @@ -177,7 +179,7 @@ bounded description that the FFM layer turns into a `MemorySegment`: pub struct LgjLaneDesc { pub addr: u64, // physics — never surfaced in the public Java API pub len_elems: u64, - pub byte_len: u64, + pub byte_len: u64, // exact covered span: (len-1)*stride + elem_bytes; 0 when empty pub owner: u64, // owning resource handle pub epoch: u64, // liveness stamp; Java re-checks before use pub elem_kind: u32, // LgjElemKind @@ -267,7 +269,7 @@ predicates or rows are involved. The unfused per-predicate ops are retained only so the fused path can be benchmarked *against* something and so parity can be checked predicate-by-predicate. -## 7. The function surface (14 symbols) +## 7. The function surface (18 symbols) All symbols are prefixed `lgj_`. All return `i32` status except the manifest getter. `out_*` parameters are written only on `OK`. @@ -407,3 +409,78 @@ Named so their absence is a decision on record rather than an oversight: ABI-compatible: `WideFieldMask`'s canonical `[u64]` chunks *are* this ABI's `MASK_WORD` lane, and `NodeRow`'s `16|16|480` `#[repr(C, align(64))]` layout is already a legal lane description. + +## 11. The SoA row store (ABI minor ≥ 2) + +The substrate layout the whole stack converges on — operator-stated reference +(2026-08-17): **64K rows × 512 bytes per row, 32 facet lanes of 16 bytes each +(4-byte little-endian classid + 12-byte payload)**, the lance-graph V3 +content-blind facet shape, enforced everywhere on the Rust side. The Java +side's *view* may differ; these bytes are the substrate truth. One buffer, +zero serialization: every access — Rust kernel or Java segment read — is a +*reading* of the same bytes. + +### Resource + +``` +LGJ_RESOURCE_ROWSTORE = 3 +i32 lgj_rowstore_open(u64 n_rows, u64 seed, u64* out_handle) +``` + +Deterministic generator (normative; two SplitMix64 draws per facet, `a` then +`b`, 64 draws per row — full statement in `rowstore.rs`'s doc): + +``` +classid = (a >>> 33) & 0xF // same recipe as the fixture's class lane +payload = le64(b) ++ le32(a & 0xFFFFFFFF) +``` + +### Lanes (described through the UNCHANGED `LgjLaneDesc` — `stride_bytes` +anticipated this since minor 1) + +| lane id | what | kind | stride | flags | +|---|---|---|---|---| +| `0` | the raw buffer, `n_rows * 512` bytes | `U8` | 1 | `READABLE \| CONTIGUOUS` | +| `1 + f` (f in `0..32`) | facet `f`'s classid column | `U32` | 512 | `READABLE` | + +`byte_len` is the **exact covered span** `(len_elems - 1) * stride_bytes + +elem_bytes` (0 when empty) — for a strided facet lane this deliberately does +NOT round up to `len * stride`, because the lane's base sits `f*16` into the +buffer and a full-stride final window would let Java bound a segment past the +allocation's end. For contiguous lanes the formula reduces to +`len * elem_bytes`, unchanged from minor 1. + +### Operations + +``` +i32 lgj_op_eq_classid(u64 res, u32 facet, u32 needle, u64 dst_mask) +i32 lgj_row_facet_match(u64 res, u32 needle, u32* out, u64 out_len_elems) +``` + +- `lgj_op_eq_classid` overwrites `dst_mask` with the row mask + `classid(facet, row) == needle`. `facet` is a facet index `0..32`, not a + lane id. The result is an ordinary mask: `lgj_mask_create` accepts a row + store as parent (masks may parent onto a pattern OR a row store — both are + read-only row-shaped resources), and the whole §7 mask algebra + (`and`/`or`/`count`/`describe`) applies unchanged. +- `lgj_row_facet_match` writes, for every row, a `u32` bitset of which of its + 32 facets carry `needle` as classid — into the **caller's** buffer (a + Java-arena segment; zero-copy out, nothing serialized). Capacity is checked + BEFORE anything is written (`MASK_LENGTH_MISMATCH` on a short buffer). + +### SIMD provenance (unchanged §8 rule, applied) + +`lgj_op_eq_classid` routes through `ndarray::simd::eq_u32_strided_to_mask` +(scalar strided loads — at stride 512 each element is on its own cache line, +so the walk is memory-bound and SIMD earns its keep in the 16-wide compare). +`lgj_row_facet_match` wraps the store's bytes in +`ndarray::simd::MultiLaneColumn` (an `Arc` refcount bump, no copy) and answers +four facets per 64-byte chunk with one `U32x16::eq_bitmask`. + +### Alignment (stated honestly) + +The buffer base is `u8`-aligned (`Arc<[u8]>`; stable Rust promises no more). +Rows are 512-byte strided within it. Nothing in this slice needs more — Java +reads via `JAVA_INT_UNALIGNED`-class layouts, and every `ndarray::simd` load +is a register fill. The 64-byte-aligned base guarantee arrives with the real +`NodeRow` (`#[repr(C, align(64))]`) wiring. diff --git a/native/lgj-abi/src/abi.rs b/native/lgj-abi/src/abi.rs index 80d8bf6..d919922 100644 --- a/native/lgj-abi/src/abi.rs +++ b/native/lgj-abi/src/abi.rs @@ -31,7 +31,12 @@ use core::mem::{align_of, size_of}; pub const LGJ_ABI_MAJOR: u32 = 0; /// Additive change ⇒ bump. Older Java may still load (`minor >= expected`). -pub const LGJ_ABI_MINOR: u32 = 1; +/// +/// Minor **2** (2026-08-17): the SoA row store — `LGJ_RESOURCE_ROWSTORE`, +/// `lgj_rowstore_open`, `lgj_op_eq_classid`, `lgj_row_facet_match`, and +/// strided facet lanes described through the (unchanged) `LgjLaneDesc`. +/// Purely additive; a minor-1 Java loads and sees none of it. +pub const LGJ_ABI_MINOR: u32 = 2; /// `"LGJ_ABI\0"` read big-endian. /// @@ -139,8 +144,13 @@ pub const LGJ_FLAG_CONTIGUOUS: u32 = 1 << 2; // §5 resource kinds /// A pattern (the SoA fixture): id/class/value lanes, read-only. pub const LGJ_RESOURCE_PATTERN: u32 = 1; -/// A mask: one `MASK_WORD` lane, writable, owned by a parent pattern. +/// A mask: one `MASK_WORD` lane, writable, owned by a parent pattern or +/// row store. pub const LGJ_RESOURCE_MASK: u32 = 2; +/// A SoA row store (abi.md §11): `n_rows × 512` bytes, 32 facet lanes of +/// (4-byte LE classid + 12-byte payload); 1 raw `U8` lane + 32 strided +/// `U32` classid lanes, all read-only. ABI minor ≥ 2. +pub const LGJ_RESOURCE_ROWSTORE: u32 = 3; // §7 mask_create initial states /// `lgj_mask_create(initial = 0)` — no rows set. @@ -201,7 +211,11 @@ pub struct LgjLaneDesc { pub addr: u64, /// Number of elements. pub len_elems: u64, - /// `len_elems * stride_bytes`. + /// Exact covered span: `(len_elems - 1) * stride_bytes + elem_bytes`, `0` + /// when empty. Reduces to `len_elems * elem_bytes` for contiguous lanes; + /// for a strided facet lane it deliberately does NOT round up to + /// `len * stride` (abi.md §11 — a full-stride final window would let Java + /// bound a segment past the allocation's end). pub byte_len: u64, /// Owning resource handle. pub owner: u64, diff --git a/native/lgj-abi/src/exports.rs b/native/lgj-abi/src/exports.rs index 050bf60..e2dd2aa 100644 --- a/native/lgj-abi/src/exports.rs +++ b/native/lgj-abi/src/exports.rs @@ -104,6 +104,40 @@ pub unsafe extern "C" fn lgj_pattern_open(n_rows: u64, seed: u64, out_handle: *m }) } +/// Build the deterministic SoA **row store** (abi.md §11) and return its +/// handle: `n_rows × 512` bytes, 32 facets of (4-byte LE classid + 12-byte +/// payload) per row. ABI minor ≥ 2. +/// +/// Bulk by construction; the generation algorithm is normative — see +/// [`crate::rowstore::RowStore`]. Lanes: `0` = the raw `U8` buffer +/// (contiguous), `1..=32` = per-facet classid lanes (`U32`, stride 512). +/// # Safety +/// +/// A null pointer is *handled*, not UB: it returns `NULL_ARGUMENT`. Beyond +/// that, `out_handle` must be a valid, aligned, writable `u64`. Written only +/// on success. +/// +/// `unsafe` here is a note to Rust callers linking the `rlib`. The JVM, +/// which is the real caller, has no such concept — it upholds the same +/// contract by construction, because every pointer it passes comes from a +/// `MemorySegment` whose size and alignment it derived from the manifest. +#[no_mangle] +pub unsafe extern "C" fn lgj_rowstore_open(n_rows: u64, seed: u64, out_handle: *mut u64) -> i32 { + guard(|| { + if out_handle.is_null() { + return LGJ_ERR_NULL_ARGUMENT; + } + match registry::open_rowstore(n_rows, seed) { + Ok(h) => { + // SAFETY: non-null (checked above); written only on success. + unsafe { *out_handle = h }; + LGJ_OK + } + Err(e) => e, + } + }) +} + /// Free a resource: its lanes are dropped, its generation is bumped, and its /// children begin failing with `PARENT_CLOSED`. /// @@ -145,12 +179,15 @@ pub unsafe extern "C" fn lgj_resource_info(handle: u64, out: *mut LgjResourceInf // Lanes // ─────────────────────────────────────────────────────────────────────────── -/// Describe one lane of a **pattern**: `0 = ids (U64)`, `1 = classes (U32)`, -/// `2 = values (I32)`. +/// Describe one lane of a **pattern** (`0 = ids (U64)`, `1 = classes (U32)`, +/// `2 = values (I32)`) or of a **row store** (`0 = raw U8 buffer, +/// contiguous; 1..=32 = facet classid lanes, U32, stride 512` — the strided +/// case `stride_bytes` existed for since ABI 0.1). /// -/// All pattern lanes are `READABLE | CONTIGUOUS` and never `WRITABLE` -/// (abi.md §7). The returned `addr` is stable until `lgj_close` — lanes are -/// allocated once and never moved or resized (§4). +/// All these lanes are `READABLE` and never `WRITABLE` (abi.md §7); the +/// contiguous flag is set exactly when `stride_bytes == elem_bytes`. The +/// returned `addr` is stable until `lgj_close` — buffers are allocated once +/// and never moved or resized (§4). /// # Safety /// /// A null pointer is *handled*, not UB: it returns `NULL_ARGUMENT`. Beyond @@ -170,29 +207,53 @@ pub unsafe extern "C" fn lgj_lane_describe( if out.is_null() { return LGJ_ERR_NULL_ARGUMENT; } - let entry = match registry::resolve_kind(handle, LGJ_RESOURCE_PATTERN) { + let entry = match registry::resolve(handle) { Ok(e) => e, Err(e) => return e, }; - let fixture = match entry.fixture() { - Some(f) => f, - None => return LGJ_ERR_WRONG_RESOURCE_KIND, - }; - let (addr, len_elems, kind) = match fixture.lane_raw(lane_id) { - Some(t) => t, - None => return LGJ_ERR_INVALID_LANE, - }; + // (addr, len_elems, kind, stride_bytes, contiguous) — patterns are + // always contiguous; a row store's facet lanes are the strided case. + let (addr, len_elems, kind, stride_bytes, contiguous) = + if let Some(fixture) = entry.fixture() { + match fixture.lane_raw(lane_id) { + Some((a, n, k)) => (a, n, k, k.elem_bytes(), true), + None => return LGJ_ERR_INVALID_LANE, + } + } else if let Some(store) = entry.rowstore() { + match store.lane_raw(lane_id) { + Some(t) => t, + None => return LGJ_ERR_INVALID_LANE, + } + } else { + // A mask: its word lane is described by lgj_mask_describe. + return LGJ_ERR_WRONG_RESOURCE_KIND; + }; let elem_bytes = kind.elem_bytes(); + let mut flags = LGJ_FLAG_READABLE; + if contiguous { + flags |= LGJ_FLAG_CONTIGUOUS; + } + // Exact covered span: from the lane's base to the END of its LAST + // element — `(len-1)*stride + elem_bytes`. For a contiguous lane this + // is `len * elem_bytes` exactly as before; for a strided facet lane it + // deliberately does NOT round up to `len * stride`, because a facet + // lane's base sits `facet*16` into the buffer and a full-stride final + // window would let Java bound a segment past the allocation's end. + let byte_len = if len_elems == 0 { + 0 + } else { + (len_elems - 1) * stride_bytes as u64 + elem_bytes as u64 + }; let desc = LgjLaneDesc { addr, len_elems, - byte_len: len_elems * elem_bytes as u64, + byte_len, owner: handle, epoch: entry.epoch, elem_kind: kind as u32, elem_bytes, - stride_bytes: elem_bytes, - flags: crate::fixture::Fixture::lane_flags(), + stride_bytes, + flags, }; // SAFETY: non-null; `LgjLaneDesc` is `#[repr(C)]`, 56 bytes, and that // size is asserted at compile time and reported by the manifest. @@ -544,6 +605,111 @@ pub extern "C" fn lgj_op_gt_i32(res: u64, lane_id: u32, threshold: i32, dst_mask }) } +// ─────────────────────────────────────────────────────────────────────────── +// Row-store bulk predicates (ABI minor ≥ 2) +// ─────────────────────────────────────────────────────────────────────────── + +/// One crossing: **overwrites** `dst_mask` with the row mask +/// `classid(facet, row) == needle` over a row store's facet lane. +/// +/// `facet` is the facet index `0..32`, NOT a lane id (lane id = facet + 1). +/// The resulting mask is an ordinary mask resource: it composes with +/// `lgj_mask_and`/`or`, counts with `lgj_mask_count`, and its words are +/// directly readable/writable through `lgj_mask_describe` — the whole +/// existing mask algebra applies unchanged to row stores. +#[no_mangle] +pub extern "C" fn lgj_op_eq_classid(res: u64, facet: u32, needle: u32, dst_mask: u64) -> i32 { + guard(|| { + let store_entry = match registry::resolve_kind(res, LGJ_RESOURCE_ROWSTORE) { + Ok(e) => e, + Err(e) => return e, + }; + let store = match store_entry.rowstore() { + Some(s) => s, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + if facet >= crate::rowstore::ROW_FACETS { + return LGJ_ERR_INVALID_LANE; + } + let (mask, _parent) = match registry::resolve_mask_with_parent(dst_mask) { + Ok(t) => t, + Err(e) => return e, + }; + if mask.n_rows != store_entry.n_rows { + return LGJ_ERR_MASK_LENGTH_MISMATCH; + } + let mut g = match mask.write_mask() { + Some(g) => g, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + kernels::simd_rowstore_classid_mask( + store.as_bytes(), + facet as usize * crate::rowstore::FACET_BYTES as usize, + store_entry.n_rows as usize, + needle, + &mut g.words, + ); + clear_tail_bits(&mut g.words, store_entry.n_rows); + LGJ_OK + }) +} + +/// One crossing: for every row, which of its 32 facets carry `needle` as +/// classid — one `u32` bitset per row, written into the **caller's** buffer +/// (a Java-arena segment of `n_rows` ints; zero-copy out, nothing +/// serialized). +/// +/// `out_len_elems` is the capacity of `out` in `u32` elements; it must be at +/// least the store's row count or the call fails with `MASK_LENGTH_MISMATCH` +/// before anything is written. The first `n_rows` elements are fully +/// overwritten; elements past `n_rows` are untouched. +/// # Safety +/// +/// A null pointer is *handled*, not UB: it returns `NULL_ARGUMENT`. Beyond +/// that, `out` must point to at least `out_len_elems` writable, 4-byte-aligned +/// `u32`s (Java passes a segment it allocated with that layout). +/// +/// `unsafe` here is a note to Rust callers linking the `rlib`. The JVM, +/// which is the real caller, has no such concept — it upholds the same +/// contract by construction, because every pointer it passes comes from a +/// `MemorySegment` whose size and alignment it derived from the manifest. +#[no_mangle] +pub unsafe extern "C" fn lgj_row_facet_match( + res: u64, + needle: u32, + out: *mut u32, + out_len_elems: u64, +) -> i32 { + guard(|| { + if out.is_null() { + return LGJ_ERR_NULL_ARGUMENT; + } + let store_entry = match registry::resolve_kind(res, LGJ_RESOURCE_ROWSTORE) { + Ok(e) => e, + Err(e) => return e, + }; + let store = match store_entry.rowstore() { + Some(s) => s, + None => return LGJ_ERR_WRONG_RESOURCE_KIND, + }; + let n_rows = store_entry.n_rows; + if out_len_elems < n_rows { + return LGJ_ERR_MASK_LENGTH_MISMATCH; + } + let n = match usize::try_from(n_rows) { + Ok(n) => n, + Err(_) => return LGJ_ERR_LENGTH_OVERFLOW, + }; + // SAFETY: non-null (checked), and the caller guarantees at least + // `out_len_elems >= n_rows` writable u32s at `out` — Java passes a + // segment whose element count it allocated. The slice is built over + // exactly the prefix this call overwrites. + let out_slice = unsafe { std::slice::from_raw_parts_mut(out, n) }; + kernels::simd_rowstore_facet_match(&store.bytes_arc(), n, needle, out_slice); + LGJ_OK + }) +} + // ─────────────────────────────────────────────────────────────────────────── // The fused plan — N predicates, ONE crossing // ─────────────────────────────────────────────────────────────────────────── diff --git a/native/lgj-abi/src/kernels.rs b/native/lgj-abi/src/kernels.rs index 9e6d03b..7342384 100644 --- a/native/lgj-abi/src/kernels.rs +++ b/native/lgj-abi/src/kernels.rs @@ -98,6 +98,81 @@ pub fn simd_popcount(words: &[u64]) -> u64 { ndarray::simd::popcount_batch_u64(words) } +/// Row mask over one facet-classid lane of a 512-byte row store: +/// `out_words[row-th bit] = (classid of facet at first_offset in row == needle)`. +/// +/// Routes through `ndarray::simd::eq_u32_strided_to_mask` — the strided +/// AoS-facet scan (LE `u32` at `first_offset + row * 512`). The primitive +/// owns bounds checking (overflow-checked, panics rather than reading out of +/// bounds) and the trailing-bits-zero guarantee. +#[inline] +pub fn simd_rowstore_classid_mask( + bytes: &[u8], + first_offset: usize, + n_rows: usize, + needle: u32, + out_words: &mut [u64], +) { + ndarray::simd::eq_u32_strided_to_mask( + bytes, + first_offset, + crate::rowstore::ROW_BYTES as usize, + n_rows, + needle, + out_words, + ); +} + +/// Per-row facet-match: `out[row]` gets bit `f` set iff facet `f`'s classid +/// in that row equals `needle` — "which facets of this node carry class X", +/// one `u32` answer per row, written into the caller's buffer. +/// +/// This is the [`ndarray::simd::MultiLaneColumn`] consumer: the store's +/// `Arc<[u8]>` is wrapped WITHOUT copying (the Arc clone is a refcount bump), +/// and each 64-byte chunk — four 16-byte facets — is answered by ONE +/// `U32x16::eq_bitmask` against the broadcast needle, masked to the classid +/// positions 0/4/8/12 and folded into 4 facet bits. Eight chunks per row +/// assemble the row's 32-bit answer. +/// +/// # Panics +/// +/// Panics if `bytes.len() != n_rows * 512` or `out.len() < n_rows` — caller +/// bugs inside this crate, not reachable from the membrane (the export +/// validates first). +pub fn simd_rowstore_facet_match( + bytes: &std::sync::Arc<[u8]>, + n_rows: usize, + needle: u32, + out: &mut [u32], +) { + use crate::rowstore::ROW_BYTES; + assert_eq!(bytes.len(), n_rows * ROW_BYTES as usize); + assert!(out.len() >= n_rows); + + // n*512 is always a multiple of 64 (rowstore tests pin this), so `new` + // cannot fail — and the construction shares the bytes, never copies them. + let col = ndarray::simd::MultiLaneColumn::new(std::sync::Arc::clone(bytes)) + .expect("rowstore buffer is a multiple of 64 bytes by construction"); + let needle_v = ndarray::simd::U32x16::from_array([needle; 16]); + + // Fully overwrite, same contract as every mask writer in this crate: the + // per-chunk fold below ORs, so stale caller bits must not survive. + for o in out.iter_mut().take(n_rows) { + *o = 0; + } + + const CHUNKS_PER_ROW: usize = (ROW_BYTES / 64) as usize; // 8 + for (c, chunk) in col.iter_u32x16().enumerate() { + // Classids sit at u32 positions 0/4/8/12 of the 16-lane chunk; the + // other twelve lanes are payload bytes that must never contribute. + let m = chunk.eq_bitmask(needle_v) & 0x1111; + let facet_bits = (m & 1) | ((m >> 4) & 1) << 1 | ((m >> 8) & 1) << 2 | ((m >> 12) & 1) << 3; + let row = c / CHUNKS_PER_ROW; + let chunk_in_row = c % CHUNKS_PER_ROW; + out[row] |= (facet_bits as u32) << (4 * chunk_in_row); + } +} + // ─────────────────────────────────────────────────────────────────────────── // Scalar reference — INDEPENDENT of ndarray. Do not "simplify" by calling the // wrappers above; the independence IS the test. @@ -161,6 +236,43 @@ pub fn scalar_popcount(words: &[u64]) -> u64 { n } +/// Reference strided classid → row mask. Plain byte reads, no ndarray. +pub fn scalar_rowstore_classid_mask( + bytes: &[u8], + first_offset: usize, + n_rows: usize, + needle: u32, + out_words: &mut [u64], +) { + for w in out_words.iter_mut() { + *w = 0; + } + for row in 0..n_rows { + let off = first_offset + row * crate::rowstore::ROW_BYTES as usize; + let v = u32::from_le_bytes([bytes[off], bytes[off + 1], bytes[off + 2], bytes[off + 3]]); + if v == needle { + out_words[row / 64] |= 1u64 << (row % 64); + } + } +} + +/// Reference per-row facet match. Plain byte reads, no ndarray. +pub fn scalar_rowstore_facet_match(bytes: &[u8], n_rows: usize, needle: u32, out: &mut [u32]) { + use crate::rowstore::{FACET_BYTES, ROW_BYTES, ROW_FACETS}; + for (row, o) in out.iter_mut().enumerate().take(n_rows) { + let mut bits = 0u32; + for f in 0..ROW_FACETS { + let off = row * ROW_BYTES as usize + f as usize * FACET_BYTES as usize; + let v = + u32::from_le_bytes([bytes[off], bytes[off + 1], bytes[off + 2], bytes[off + 3]]); + if v == needle { + bits |= 1 << f; + } + } + *o = bits; + } +} + // ─────────────────────────────────────────────────────────────────────────── // Which path a call takes // ─────────────────────────────────────────────────────────────────────────── @@ -387,6 +499,110 @@ mod tests { ); } + /// The row-store parity falsifier: both new SIMD kernels against their + /// independent scalar references, over real generated stores at row + /// counts straddling every boundary (16-lane groups via 4-facet chunks, + /// 64-bit words, and the 8-chunks-per-row fold). + #[test] + fn rowstore_kernels_match_their_scalar_references() { + use crate::rowstore::RowStore; + for n in [0u64, 1, 2, 15, 16, 17, 63, 64, 65, 200] { + for seed in [0u64, 0xABCD] { + let s = RowStore::generate(n, seed).unwrap(); + let bytes = s.bytes_arc(); + + for facet in [0u32, 1, 15, 31] { + for needle in [0u32, 7, 15, 42] { + let n_words = mask_words_for(n) as usize; + let mut a = vec![u64::MAX; n_words]; + let mut b = vec![u64::MAX; n_words]; + let first_offset = (facet as usize) * crate::rowstore::FACET_BYTES as usize; + simd_rowstore_classid_mask( + &bytes, + first_offset, + n as usize, + needle, + &mut a, + ); + scalar_rowstore_classid_mask( + &bytes, + first_offset, + n as usize, + needle, + &mut b, + ); + assert_eq!(a, b, "classid_mask n={n} facet={facet} needle={needle}"); + // Cross-check against the store's own scalar accessor, + // a THIRD independent computation. + for row in 0..n { + let bit = (a[(row / 64) as usize] >> (row % 64)) & 1; + let expect = (s.classid_at(row, facet) == needle) as u64; + assert_eq!(bit, expect, "row {row}"); + } + } + } + + for needle in [0u32, 7, 15] { + let mut a = vec![u32::MAX; n as usize]; + let mut b = vec![u32::MAX; n as usize]; + simd_rowstore_facet_match(&bytes, n as usize, needle, &mut a); + scalar_rowstore_facet_match(&bytes, n as usize, needle, &mut b); + assert_eq!(a, b, "facet_match n={n} needle={needle}"); + // Consistency with the per-facet masks: bit f of row r in + // facet_match must equal row r's bit in facet f's mask. + if n > 0 { + for facet in [0u32, 31] { + let mut m = vec![0u64; mask_words_for(n) as usize]; + scalar_rowstore_classid_mask( + &bytes, + facet as usize * crate::rowstore::FACET_BYTES as usize, + n as usize, + needle, + &mut m, + ); + for row in 0..n as usize { + let via_match = (a[row] >> facet) & 1; + let via_mask = ((m[row / 64] >> (row % 64)) & 1) as u32; + assert_eq!(via_match, via_mask, "row {row} facet {facet}"); + } + } + } + } + } + } + } + + /// The facet-match fold must never let PAYLOAD bytes match: plant the + /// needle's bit pattern inside a payload and prove it does not fire. + #[test] + fn facet_match_ignores_needle_patterns_in_payload_bytes() { + use crate::rowstore::RowStore; + let s = RowStore::generate(4, 0x5EED).unwrap(); + let mut bytes = s.as_bytes().to_vec(); + let needle = 0xDEAD_BEEFu32; + // Row 2, facet 5: put the needle in PAYLOAD positions (offsets +4 and + // +12), and a non-matching classid at +0. + let base = 2 * 512 + 5 * 16; + bytes[base..base + 4].copy_from_slice(&1u32.to_le_bytes()); + bytes[base + 4..base + 8].copy_from_slice(&needle.to_le_bytes()); + bytes[base + 12..base + 16].copy_from_slice(&needle.to_le_bytes()); + let bytes: std::sync::Arc<[u8]> = std::sync::Arc::from(bytes); + + let mut out = vec![0u32; 4]; + simd_rowstore_facet_match(&bytes, 4, needle, &mut out); + assert_eq!( + (out[2] >> 5) & 1, + 0, + "payload bytes must never satisfy a classid match" + ); + // And the twin: planting it in the CLASSID position does fire. + let mut bytes2 = s.as_bytes().to_vec(); + bytes2[base..base + 4].copy_from_slice(&needle.to_le_bytes()); + let bytes2: std::sync::Arc<[u8]> = std::sync::Arc::from(bytes2); + simd_rowstore_facet_match(&bytes2, 4, needle, &mut out); + assert_eq!((out[2] >> 5) & 1, 1, "a real classid match must fire"); + } + #[test] fn non_aliasing_mask_ops_agree_with_assign_forms() { let a = vec![0xF0F0_F0F0_F0F0_F0F0u64, 0x00FF]; diff --git a/native/lgj-abi/src/lib.rs b/native/lgj-abi/src/lib.rs index 952e87f..83644b6 100644 --- a/native/lgj-abi/src/lib.rs +++ b/native/lgj-abi/src/lib.rs @@ -51,6 +51,7 @@ pub mod exports; pub mod fixture; pub mod kernels; pub mod registry; +pub mod rowstore; // Re-export the ABI vocabulary at the crate root for convenience in tests and // for any Rust consumer that links the `rlib` rather than the `cdylib`. @@ -89,6 +90,18 @@ mod integration_tests { pub fn reduce_sum_i32(r: u64, lane: u32, m: u64, out: *mut i64) -> i32 { unsafe { lgj_reduce_sum_i32(r, lane, m, out) } } + pub fn rowstore_open(n: u64, seed: u64, out: *mut u64) -> i32 { + unsafe { lgj_rowstore_open(n, seed, out) } + } + pub fn resource_info(h: u64, out: *mut LgjResourceInfo) -> i32 { + unsafe { lgj_resource_info(h, out) } + } + pub fn lane_describe(h: u64, lane: u32, out: *mut LgjLaneDesc) -> i32 { + unsafe { lgj_lane_describe(h, lane, out) } + } + pub fn row_facet_match(r: u64, needle: u32, out: *mut u32, cap: u64) -> i32 { + unsafe { lgj_row_facet_match(r, needle, out, cap) } + } } fn open(n: u64, seed: u64) -> u64 { @@ -233,6 +246,106 @@ mod integration_tests { } } + /// The SoA row store, end to end through the membrane (ABI minor ≥ 2): + /// open → describe → classid predicate → mask algebra → count → + /// facet-match into a caller buffer — everything independently recomputed + /// from the documented generator, exactly as the Java parity test will. + #[test] + fn the_rowstore_slice_end_to_end_through_the_membrane() { + use crate::rowstore::{RowStore, LANE_FACET_BASE, ROW_BYTES}; + let n = 1000u64; + let seed = 0xC0FFEE; + let mut s = 0u64; + assert_eq!(call::rowstore_open(n, seed, &mut s), LGJ_OK); + + // Resource self-description. + let mut info = LgjResourceInfo::default(); + assert_eq!(call::resource_info(s, &mut info), LGJ_OK); + assert_eq!(info.kind, LGJ_RESOURCE_ROWSTORE); + assert_eq!(info.lane_count, 33); + assert_eq!(info.n_rows, n); + + // Raw lane: contiguous U8, exactly n*512 bytes. + let mut d = LgjLaneDesc::default(); + assert_eq!(call::lane_describe(s, 0, &mut d), LGJ_OK); + assert_eq!(d.elem_kind, LgjElemKind::U8 as u32); + assert_eq!(d.len_elems, n * ROW_BYTES); + assert_eq!(d.byte_len, n * ROW_BYTES); + assert_ne!(d.flags & LGJ_FLAG_CONTIGUOUS, 0); + + // Facet lane 7: strided U32, stride 512, and the exact-span rule — + // the described window must END at the buffer's last classid, never + // a full stride past it (Java bounds a segment from this number). + assert_eq!(call::lane_describe(s, LANE_FACET_BASE + 7, &mut d), LGJ_OK); + assert_eq!(d.elem_kind, LgjElemKind::U32 as u32); + assert_eq!(d.len_elems, n); + assert_eq!(d.stride_bytes, 512); + assert_eq!(d.byte_len, (n - 1) * 512 + 4); + assert_eq!(d.flags & LGJ_FLAG_CONTIGUOUS, 0); + // Lane 34 does not exist (1 raw + 32 facets = ids 0..=32). + assert_eq!(call::lane_describe(s, 34, &mut d), LGJ_ERR_INVALID_LANE); + + // classid predicate on facet 7 → an ordinary mask, counted natively… + let m = mask(s, LGJ_MASK_INIT_EMPTY); + assert_eq!(lgj_op_eq_classid(s, 7, 9, m), LGJ_OK); + let mut count = 0u64; + assert_eq!(call::mask_count(m, &mut count), LGJ_OK); + + // …and recomputed independently from the documented generator. + let store = RowStore::generate(n, seed).unwrap(); + let want = (0..n).filter(|&r| store.classid_at(r, 7) == 9).count() as u64; + assert_eq!(count, want); + assert!(count > 0 && count < n, "must be a middling selection"); + + // Facet out of range is a status, and the facet/lane-id distinction + // holds: facet 32 is invalid even though LANE id 32 (facet 31) exists. + assert_eq!(lgj_op_eq_classid(s, 32, 9, m), LGJ_ERR_INVALID_LANE); + + // The mask composes with the EXISTING algebra: AND facet-7==9 with + // facet-0==9 and verify against the independent recomputation. + let m2 = mask(s, LGJ_MASK_INIT_EMPTY); + assert_eq!(lgj_op_eq_classid(s, 0, 9, m2), LGJ_OK); + assert_eq!(lgj_mask_and(m, m2, m), LGJ_OK); + assert_eq!(call::mask_count(m, &mut count), LGJ_OK); + let want_and = (0..n) + .filter(|&r| store.classid_at(r, 7) == 9 && store.classid_at(r, 0) == 9) + .count() as u64; + assert_eq!(count, want_and); + + // Facet-match into a caller-owned buffer (what Java allocates in its + // own arena): per-row 32-bit facet sets, verified row by row. + let mut out = vec![u32::MAX; n as usize]; + assert_eq!(call::row_facet_match(s, 9, out.as_mut_ptr(), n), LGJ_OK); + for r in 0..n { + let mut want_bits = 0u32; + for f in 0..32u32 { + if store.classid_at(r, f) == 9 { + want_bits |= 1 << f; + } + } + assert_eq!(out[r as usize], want_bits, "row {r}"); + } + // A too-small caller buffer is rejected BEFORE anything is written. + let mut short = vec![0xABAB_ABABu32; 10]; + assert_eq!( + call::row_facet_match(s, 9, short.as_mut_ptr(), 10), + LGJ_ERR_MASK_LENGTH_MISMATCH + ); + assert!( + short.iter().all(|&x| x == 0xABAB_ABAB), + "untouched on failure" + ); + + // Lifecycle: same rules as every resource. + assert_eq!(lgj_close(m2), LGJ_OK); + assert_eq!(lgj_close(m), LGJ_OK); + assert_eq!(lgj_close(s), LGJ_OK); + assert_eq!( + call::row_facet_match(s, 9, out.as_mut_ptr(), n), + LGJ_ERR_INVALID_HANDLE + ); + } + /// Concurrent mask binops that name the same masks in *opposite* orders — /// the shape that would deadlock without address-ordered locking. #[test] diff --git a/native/lgj-abi/src/registry.rs b/native/lgj-abi/src/registry.rs index a8cd04a..dda345f 100644 --- a/native/lgj-abi/src/registry.rs +++ b/native/lgj-abi/src/registry.rs @@ -49,6 +49,7 @@ use std::sync::{Arc, OnceLock, RwLock, RwLockReadGuard, RwLockWriteGuard}; use crate::abi::*; use crate::fixture::{Fixture, PATTERN_LANE_COUNT}; +use crate::rowstore::{RowStore, ROWSTORE_LANE_COUNT}; /// The mutable half of a mask: the packed row bits. #[derive(Debug)] @@ -65,6 +66,9 @@ pub struct MaskWords { pub enum Payload { /// A read-only SoA fixture — no lock needed, because no ABI path mutates it. Pattern(Fixture), + /// A read-only SoA row store (abi.md §11) — likewise lock-free: the + /// `Arc<[u8]>` buffer is immutable for the resource's whole life. + RowStore(RowStore), /// Mutable mask words behind their own lock. Mask(RwLock), } @@ -96,7 +100,15 @@ impl ResourceEntry { pub fn fixture(&self) -> Option<&Fixture> { match &self.payload { Payload::Pattern(f) => Some(f), - Payload::Mask(_) => None, + _ => None, + } + } + + /// `Some(&RowStore)` iff this is a row store. + pub fn rowstore(&self) -> Option<&RowStore> { + match &self.payload { + Payload::RowStore(s) => Some(s), + _ => None, } } @@ -104,7 +116,7 @@ impl ResourceEntry { pub fn mask(&self) -> Option<&RwLock> { match &self.payload { Payload::Mask(m) => Some(m), - Payload::Pattern(_) => None, + _ => None, } } @@ -127,6 +139,7 @@ impl ResourceEntry { kind: self.kind, lane_count: match self.kind { LGJ_RESOURCE_PATTERN => PATTERN_LANE_COUNT, + LGJ_RESOURCE_ROWSTORE => ROWSTORE_LANE_COUNT, // A mask exposes exactly one MASK_WORD lane. _ => 1, }, @@ -278,9 +291,29 @@ pub fn open_pattern(n_rows: u64, seed: u64) -> Result { }) } +/// Create a row-store resource from the deterministic generator (abi.md §11). +pub fn open_rowstore(n_rows: u64, seed: u64) -> Result { + let store = RowStore::generate(n_rows, seed).ok_or(LGJ_ERR_LENGTH_OVERFLOW)?; + insert(ResourceEntry { + kind: LGJ_RESOURCE_ROWSTORE, + epoch: next_epoch(), + n_rows, + parent: 0, + parent_gen: 0, + payload: Payload::RowStore(store), + }) +} + /// Create a mask over `parent`, all bits `0` or all bits `1`. +/// +/// A mask's parent may be a pattern OR a row store — both are read-only +/// row-shaped resources, and a mask is a row selection over either. A mask +/// over a mask stays rejected. pub fn create_mask(parent_handle: u64, initial: u32) -> Result { - let parent = resolve_kind(parent_handle, LGJ_RESOURCE_PATTERN)?; + let parent = resolve(parent_handle)?; + if !matches!(parent.kind, LGJ_RESOURCE_PATTERN | LGJ_RESOURCE_ROWSTORE) { + return Err(LGJ_ERR_WRONG_RESOURCE_KIND); + } let n_rows = parent.n_rows; let n_words = usize::try_from(mask_words_for(n_rows)).map_err(|_| LGJ_ERR_LENGTH_OVERFLOW)?; @@ -486,6 +519,40 @@ mod tests { close(p).unwrap(); } + #[test] + fn rowstore_opens_and_describes_itself() { + let h = open_rowstore(70, 3).unwrap(); + let e = resolve_kind(h, LGJ_RESOURCE_ROWSTORE).unwrap(); + let info = e.info(); + assert_eq!(info.kind, LGJ_RESOURCE_ROWSTORE); + assert_eq!(info.lane_count, ROWSTORE_LANE_COUNT); + assert_eq!(info.n_rows, 70); + assert!(e.rowstore().is_some()); + assert!(e.fixture().is_none()); + close(h).unwrap(); + } + + /// A mask parents onto a row store exactly as onto a pattern — same + /// row-count sizing, same tail rule, same parent-liveness propagation. + #[test] + fn mask_over_rowstore_works_and_tracks_parent_liveness() { + let s = open_rowstore(70, 1).unwrap(); + let m = create_mask(s, LGJ_MASK_INIT_ALL).unwrap(); + { + let e = resolve(m).unwrap(); + let g = e.read_mask().unwrap(); + assert_eq!(g.words.len(), 2); + assert_eq!(g.words[1], 0x3F, "tail past row 70 must be zero"); + } + assert!(resolve_mask_with_parent(m).is_ok()); + close(s).unwrap(); + assert_eq!( + resolve_mask_with_parent(m).unwrap_err(), + LGJ_ERR_PARENT_CLOSED + ); + close(m).unwrap(); + } + /// Locking distinct masks in address order must not deadlock regardless of /// the order the caller names them in. #[test] diff --git a/native/lgj-abi/src/rowstore.rs b/native/lgj-abi/src/rowstore.rs new file mode 100644 index 0000000..0d3e8af --- /dev/null +++ b/native/lgj-abi/src/rowstore.rs @@ -0,0 +1,292 @@ +//! The SoA row store — the lance-graph-shaped substrate (abi.md §11). +//! +//! Where [`crate::fixture`] proved the membrane over three flat lanes, this +//! module carries the layout the whole stack actually converges on — the +//! operator-stated reference (2026-08-17): **64K rows × 512 bytes per row, +//! read as 32 lanes of 16 bytes each: a 4-byte little-endian classid plus a +//! 12-byte payload** (the lance-graph V3 content-blind facet). The Java side +//! may lay its *view* out differently; these bytes are the substrate truth. +//! +//! # One buffer, two readings, zero copies +//! +//! The store is ONE `Arc<[u8]>` of `n_rows * 512` bytes. Everything else is a +//! *reading* of those bytes, never a copy: +//! +//! - **Row reading** — row `r` is bytes `r*512 .. (r+1)*512`; facet `f` of +//! row `r` is the 16 bytes at `r*512 + f*16`, its classid the leading LE +//! `u32`. This is what `iter_u8x64`/`iter_u32x16`-style chunk scans and +//! Java's structured `MemoryLayout` both address. +//! - **Facet-lane reading** — classid lane `f` is a strided `u32` column: +//! `first_offset = f*16`, `stride = 512`, `count = n_rows`. This is what +//! [`crate::abi::LgjLaneDesc::stride_bytes`] has described since ABI 0.1 — +//! the descriptor anticipated this module. +//! +//! `Arc<[u8]>` is the deliberate carrier: its heap buffer never moves for the +//! Arc's whole life (the §4 allocation-stability guarantee), a clone is a +//! refcount bump (the kernels wrap the same bytes in an +//! `ndarray::simd::MultiLaneColumn` without copying), and shared immutable +//! ownership is exactly the one-writer-per-resource concurrency shape the +//! 64K-mailbox model wants. +//! +//! # Alignment (stated honestly) +//! +//! Rows are 512-byte *strided* within the buffer, but the buffer's base is +//! only `u8`-aligned — `Arc<[u8]>` cannot promise more on stable Rust. +//! Nothing in this slice needs more: Panama reads are alignment-agnostic +//! (`ValueLayout.JAVA_INT_UNALIGNED` exists precisely for this), and every +//! `ndarray::simd` load goes through `from_array`-style register fills. The +//! 64-byte-aligned guarantee arrives with the real `NodeRow` +//! (`#[repr(C, align(64))]`) wiring, not here. + +/// Bytes per row: 32 facets × 16 bytes. +pub const ROW_BYTES: u64 = 512; +/// Facet lanes per row. +pub const ROW_FACETS: u32 = 32; +/// Bytes per facet: 4-byte classid + 12-byte payload. +pub const FACET_BYTES: u64 = 16; +/// The classid is the facet's leading little-endian `u32`. +pub const FACET_CLASSID_BYTES: u64 = 4; +/// Classid cardinality the generator produces: `0..16` (same recipe as the +/// flat fixture, so predicates select the same middling fraction). +pub const ROWSTORE_CLASS_CARDINALITY: u64 = 16; + +/// Lane id of the raw whole-buffer lane (`U8`, contiguous, `n_rows * 512` +/// elements). +pub const LANE_RAW: u32 = 0; +/// Lane id of facet `f`'s classid lane is `LANE_FACET_BASE + f`. +pub const LANE_FACET_BASE: u32 = 1; +/// Total describable lanes: 1 raw + 32 facet classid lanes. +pub const ROWSTORE_LANE_COUNT: u32 = 1 + ROW_FACETS; + +use std::sync::Arc; + +use crate::fixture::SplitMix64; + +/// The SoA row store: one shared, immutable, address-stable byte buffer. +/// +/// # The generation algorithm — NORMATIVE +/// +/// Like [`crate::fixture::Fixture`], the Java parity test recomputes +/// expectations from this description alone, so it is a contract: +/// +/// ```text +/// rng = SplitMix64(seed) // state = seed, no warm-up draws +/// for row in 0 .. n_rows: // ascending +/// for facet in 0 .. 32: // ascending within the row +/// a = rng.next_u64() // FIRST draw of the facet +/// b = rng.next_u64() // SECOND draw of the facet +/// base = row*512 + facet*16 +/// bytes[base .. base+4 ] = le32( (a >>> 33) & 0xF ) // classid +/// bytes[base+4 .. base+12] = le64( b ) // payload +/// bytes[base+12 .. base+16] = le32( a & 0xFFFFFFFF ) // payload +/// ``` +/// +/// Two draws per facet, `a` before `b`, 64 draws per row. The classid recipe +/// `(a >>> 33) & 0xF` is byte-identical to the flat fixture's class lane, so +/// a classid predicate selects the same ≈1/16 fraction here. +pub struct RowStore { + /// Logical row count. + pub n_rows: u64, + /// The seed the buffer was generated from. + pub seed: u64, + bytes: Arc<[u8]>, +} + +impl std::fmt::Debug for RowStore { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("RowStore") + .field("n_rows", &self.n_rows) + .field("seed", &self.seed) + .finish_non_exhaustive() + } +} + +impl RowStore { + /// Build the store. Allocates the buffer exactly once. + /// + /// Returns `None` if `n_rows * 512` overflows or cannot be allocated + /// (the caller maps that to `LENGTH_OVERFLOW` / `ALLOCATION_FAILED`). + pub fn generate(n_rows: u64, seed: u64) -> Option { + let n = usize::try_from(n_rows).ok()?; + let byte_len = n.checked_mul(ROW_BYTES as usize)?; + + let mut bytes = Vec::new(); + bytes.try_reserve_exact(byte_len).ok()?; + bytes.resize(byte_len, 0u8); + + let mut rng = SplitMix64::new(seed); + for row in 0..n { + for facet in 0..ROW_FACETS as usize { + let a = rng.next_u64(); + let b = rng.next_u64(); + let base = row * ROW_BYTES as usize + facet * FACET_BYTES as usize; + let classid = ((a >> 33) & (ROWSTORE_CLASS_CARDINALITY - 1)) as u32; + bytes[base..base + 4].copy_from_slice(&classid.to_le_bytes()); + bytes[base + 4..base + 12].copy_from_slice(&b.to_le_bytes()); + bytes[base + 12..base + 16].copy_from_slice(&(a as u32).to_le_bytes()); + } + } + + Some(Self { + n_rows, + seed, + bytes: Arc::from(bytes), + }) + } + + /// The whole buffer as a byte slice. Zero-copy; the address is stable for + /// the store's life (see the module header). + pub fn as_bytes(&self) -> &[u8] { + &self.bytes + } + + /// A cheap shared handle to the same bytes — what the kernels wrap in a + /// `MultiLaneColumn` without copying. + pub fn bytes_arc(&self) -> Arc<[u8]> { + Arc::clone(&self.bytes) + } + + /// The classid of facet `facet` in row `row` — the scalar (one-element) + /// read, used by tests and the scalar reference kernels. Bulk access goes + /// through the lanes, never through a loop over this. + pub fn classid_at(&self, row: u64, facet: u32) -> u32 { + let base = (row * ROW_BYTES + facet as u64 * FACET_BYTES) as usize; + u32::from_le_bytes([ + self.bytes[base], + self.bytes[base + 1], + self.bytes[base + 2], + self.bytes[base + 3], + ]) + } + + /// `(base address, len_elems, elem_kind, stride_bytes, contiguous)` for a + /// lane id, or `None` for an out-of-range id (⇒ `INVALID_LANE`). + pub fn lane_raw(&self, lane_id: u32) -> Option<(u64, u64, crate::abi::LgjElemKind, u32, bool)> { + use crate::abi::LgjElemKind; + if lane_id == LANE_RAW { + return Some(( + self.bytes.as_ptr() as u64, + self.bytes.len() as u64, + LgjElemKind::U8, + 1, + true, + )); + } + let facet = lane_id.checked_sub(LANE_FACET_BASE)?; + if facet >= ROW_FACETS { + return None; + } + // Classid lane f: strided u32 column at first_offset f*16, stride 512. + let addr = self.bytes.as_ptr() as u64 + facet as u64 * FACET_BYTES; + Some((addr, self.n_rows, LgjElemKind::U32, ROW_BYTES as u32, false)) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn generation_is_deterministic_and_seed_sensitive() { + let a = RowStore::generate(64, 42).unwrap(); + let b = RowStore::generate(64, 42).unwrap(); + let c = RowStore::generate(64, 43).unwrap(); + assert_eq!(a.as_bytes(), b.as_bytes()); + assert_ne!(a.as_bytes(), c.as_bytes()); + } + + /// The normative algorithm, recomputed independently from the doc-comment + /// description (a transcription, exactly what the Java test will do). + #[test] + fn the_documented_generator_is_the_actual_generator() { + let n = 5u64; + let seed = 0xABCD; + let store = RowStore::generate(n, seed).unwrap(); + + let mut rng = SplitMix64::new(seed); + for row in 0..n { + for facet in 0..ROW_FACETS { + let a = rng.next_u64(); + let b = rng.next_u64(); + let expect_class = ((a >> 33) & 0xF) as u32; + assert_eq!(store.classid_at(row, facet), expect_class); + let base = (row * ROW_BYTES + facet as u64 * FACET_BYTES) as usize; + assert_eq!(&store.as_bytes()[base + 4..base + 12], &b.to_le_bytes()); + assert_eq!( + &store.as_bytes()[base + 12..base + 16], + &(a as u32).to_le_bytes() + ); + } + } + } + + #[test] + fn the_buffer_is_exactly_n_times_512_bytes() { + for n in [0u64, 1, 7, 64] { + let s = RowStore::generate(n, 1).unwrap(); + assert_eq!(s.as_bytes().len() as u64, n * ROW_BYTES); + // …which is always a multiple of 64: the MultiLaneColumn + // precondition holds BY CONSTRUCTION, never by luck. + assert_eq!(s.as_bytes().len() % 64, 0); + } + } + + #[test] + fn lane_map_covers_raw_plus_32_facets_and_nothing_else() { + let s = RowStore::generate(16, 9).unwrap(); + let (addr0, len0, kind0, stride0, contig0) = s.lane_raw(LANE_RAW).unwrap(); + assert_eq!(addr0, s.as_bytes().as_ptr() as u64); + assert_eq!(len0, 16 * ROW_BYTES); + assert_eq!(kind0, crate::abi::LgjElemKind::U8); + assert_eq!(stride0, 1); + assert!(contig0); + + for f in 0..ROW_FACETS { + let (addr, len, kind, stride, contig) = s.lane_raw(LANE_FACET_BASE + f).unwrap(); + assert_eq!(addr, s.as_bytes().as_ptr() as u64 + f as u64 * FACET_BYTES); + assert_eq!(len, 16); + assert_eq!(kind, crate::abi::LgjElemKind::U32); + assert_eq!(stride, ROW_BYTES as u32); + assert!(!contig); + } + assert!(s.lane_raw(LANE_FACET_BASE + ROW_FACETS).is_none()); + assert!(s.lane_raw(u32::MAX).is_none()); + } + + /// Classids must use their full 0..16 range in every facet lane — a + /// constant lane would make every classid predicate vacuous. + #[test] + fn every_facet_lane_uses_the_full_classid_range() { + let s = RowStore::generate(4096, 7).unwrap(); + for facet in [0u32, 1, 15, 31] { + let mut seen = [false; 16]; + for row in 0..s.n_rows { + let c = s.classid_at(row, facet); + assert!(c < 16); + seen[c as usize] = true; + } + assert!( + seen.iter().all(|&x| x), + "facet {facet} must hit all 16 classids at n=4096" + ); + } + } + + #[test] + fn addresses_are_stable_across_reads() { + let s = RowStore::generate(32, 3).unwrap(); + let first = s.lane_raw(LANE_FACET_BASE + 5).unwrap().0; + for _ in 0..100 { + assert_eq!(s.lane_raw(LANE_FACET_BASE + 5).unwrap().0, first); + } + // And the Arc handle shares, never copies. + assert_eq!(s.bytes_arc().as_ptr(), s.as_bytes().as_ptr()); + } + + #[test] + fn zero_rows_is_legal_and_empty() { + let s = RowStore::generate(0, 1).unwrap(); + assert!(s.as_bytes().is_empty()); + assert_eq!(s.lane_raw(LANE_FACET_BASE).unwrap().1, 0); + } +}