diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 82ebc12..03a1066 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -4,6 +4,97 @@ > `**Status:**`/`**Confidence:**` line. A correction gets its own new, > dated entry that references the one it corrects — the storno rule. +## 2026-08-18 (later still) — "the generator exists" ≠ "Java can reach it": the membrane gap the prior entry's own resolution note missed + +**Status:** FINDING + a correction of the entry directly below this one. +**Confidence:** High — measured (`nm -D` before/after, full Rust + Java test +suites, two disable-runs). + +### What the prior entry's "RESOLVED" undersold + +The STOP-condition entry below this one — and `wave-consumer-graph.md`'s own +header — declared the graph wave DISPATCHABLE once +`RowStore::generate_with_edges` existed and its numbers were pinned. True at +the Rust level, but incomplete: `Engine.openRowStore` / +`registry::open_rowstore` (the only path Java has to a `RowStore`) call +plain `RowStore::generate` unconditionally. **No `extern "C"` symbol for the +edge-bearing generator existed.** A worker briefed to build `Graph.java` +against `RowStore.open` would have had no way to reach non-vacuous data at +all — the exact STOP condition this wave was declared clear of, just moved +one layer up the stack, and it would have surfaced as a live "the file I +need is outside my scope" STOP report from a Sonnet worker mid-dispatch +rather than being caught here, before any worker spawned. + +Same shape as the finding this whole sub-arc started from: checking the +*mechanism* (does a hop composition exist) without checking the *path* +(can a caller actually reach the data the mechanism needs). Twice now in +one wave. + +### The fix — the wave's own D1b rule, applied to itself + +`wave-consumer-graph.md`'s Decision D1 already states the rule for exactly +this shape of gap: *"a new ABI symbol... must go through the substrate wave +process FIRST as its own W-tier PR (the consumer-never-grows-the-membrane +rule)."* Applied it to the row-store CONSTRUCTOR, not just the hop op D1 +was originally about — same rule, same reasoning: growing the membrane is +orchestrator/W-tier work, never a consumer worker's ad hoc addition. + +`lgj_rowstore_open_with_edges` (ABI minor 2→3, `docs/abi.md` §12): +`registry::open_rowstore_with_edges` + the `extern "C"` export mirror +`open_rowstore`/`lgj_rowstore_open` symbol-for-symbol — same +`LGJ_RESOURCE_ROWSTORE` kind, same lane shape, no new mask op, purely an +alternative constructor. Java: `Downcalls.rowstoreOpenWithEdges` + +`Engine.openRowStoreWithEdges` (`Abi.requireMinor(3)`, matching the row +store's own minor-2 gate pattern) + the public `RowStore.openWithEdges` +factory. + +### The strongest new result: Java independently reproduces the D1a hop, not just the classid stream + +Added to `RowStoreParityTest` rather than deferred to G1/G2: a **Java-side +transcription of the exact D1a mechanism** (`lgj_row_facet_match` crossing + +raw lane-0 payload decode, zero new ABI op — the mechanism the wave's own +Decision D1 chose to start with) at the SAME parameters as the Rust-pinned +regression (`n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, +radius=25`). Result: **19 rows at 1 hop, 29 at 2 hops — identical to the +Rust side, to the row.** This is a stronger falsifier than the Rust +regression alone: it proves the membrane doesn't just carry the same +classid stream, it carries the same *edge structure*, read through the +exact mechanism a real `Graph.hop()` will use. + +Two disable-runs, both red-then-green: +1. **Registry level:** hardcoded `open_rowstore_with_edges`'s inner call to + pass `edge_classid = 0` regardless of the argument → the new + `out_of_range_edge_classid_matches_plain_open_through_the_registry` test + (which passes `edge_classid = 16`, expecting parity with plain `open`) + went red, because classid 0 with `gate_mask = 0x0` genuinely writes + structured edges the plain generator never would. Restored, 93/93. +2. **Java level:** forced the hop transcription's classid-match check to + always skip (`if (true) { continue; }`) → 1-hop and 2-hop both collapsed + to 0 and the anti-vacuity assertion failed exactly as expected. Restored, + 194/194. + +### Gates + +`cargo test` 93/93 (+3 over the prior entry's 90), clippy `-D warnings` + +`fmt --check` clean, release build exports `lgj_rowstore_open_with_edges` +(`nm -D`, confirmed present). Java `AllTests` 194/194 (+6, all in +`RowStoreParityTest`) — full suite re-run, not just the new section, since +the stale top-level `target/release/liblgj_abi.so` (pre-dating this pass, +minor 2) initially made EVERY suite fail at class-init (`Downcalls` +eagerly resolves all method handles including the new one) until rebuilt +with `CARGO_TARGET_DIR=$ROOT/target cargo build --release` per the +documented build convention — a real, if brief, self-inflicted false +alarm, not a substrate defect; recorded so a future session doesn't +re-diagnose the same eager-resolution behavior as a bug. + +### Consequence + +The graph-consumer wave is now genuinely dispatchable — the substrate is +proven at BOTH the Rust generator level (prior entry) and the Java +membrane level (this entry), through the exact D1a mechanism the wave +already chose. G1 (traversal facade) and G2 (falsifier tests) are next, +not yet spawned. + ## 2026-08-18 (measured) — the graph-consumer STOP condition was real, and is now cleared **Status:** FINDING + a correction of MY OWN earlier claim. **Confidence:** diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index 4a91d0b..08ee19b 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,43 @@ +## 2026-08-18 (later still) — the graph wave's ABI gap, found and closed before dispatch: `lgj_rowstore_open_with_edges` (minor 3) + +Picked up the graph-consumer wave on "everything on track?" — it was marked +DISPATCHABLE by the entry below, and idle capacity while ruff_r2il works its +own PR2/3 track was worth using. Before spawning G1/G2, checked what Java +would actually call to reach `RowStore::generate_with_edges` — and it was +nothing: `Engine.openRowStore`/`registry::open_rowstore` only ever call plain +`RowStore::generate`; no `extern "C"` symbol for the edge-bearing generator +existed anywhere. The prior pass's STOP-condition-RESOLVED note proved the +*generator*, not the *membrane path to it* — a real gap that would have +surfaced mid-dispatch as "G1's scope requires touching a file outside your +scope" the moment a worker tried to open an edge-bearing store from Java. + +Closed it as the wave's own D1b rule requires: "a new ABI symbol... must go +through the substrate wave process FIRST as its own W-tier PR" — done here as +orchestrator work (genuinely new ABI surface, not consumer scope), not +delegated. `lgj_rowstore_open_with_edges` (ABI minor 2→3, docs/abi.md §12): +byte-identical resource kind and lane shape to `lgj_rowstore_open`, purely an +alternative constructor, following the row store's own minor-2 +`requireMinor` gating pattern exactly. `cargo test` 93/93 (+3), Java `AllTests` +194/194 (+6). Full record (including the two disable-runs and the strongest +result — Java independently reproducing the D1a hop mechanism's exact pinned +numbers through raw-segment reads, not just the classid stream): see +`STATUS_BOARD.md` D-LGJ-W6, `EPIPHANIES.md`. + +Not yet dispatched: G1 (traversal facade) and G2 (falsifier tests) are next, +now against a genuinely complete substrate rather than one proven only at the +Rust generator level. + +Meanwhile on ruff: PR #100 (upstream 1500-commit catch-up, explicitly marked +"baseline, not for merge" in its own body) merged anyway by the operator — +its own CI never completed (cancelled on every commit), but incidentally +fixed a 13-day-old `main`-CI lint failure (`prek`/shellcheck on +`.github/workflows/ci.yaml:443`) that predated this whole arc and was +unrelated to it. PR #101 (`ruff_r2il` PR2 first slice — §12 corpus profile +resolving O1, `RefinedTruthSink`) merged cleanly on top, ~1.5h turnaround, +CI green through `prek` on the resulting `main` run — the standing lint issue +stayed fixed. Neither event required action here; recorded for continuity +since both landed mid-session. + ## 2026-08-18 (measured) — W5c's real blocker found + cleared: RowStore::generate_with_edges Asked what was buildable while ruff_r2il PR2/PR3 are blocked. Was about to diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index c69b5d9..e6e38b2 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -40,4 +40,5 @@ layout wired end to end. Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED | D-LGJ-W2 | lgj-abi row store: `rowstore.rs`, `LGJ_RESOURCE_ROWSTORE`, `lgj_rowstore_open`, strided facet lanes through the unchanged `LgjLaneDesc`, `lgj_op_eq_classid`, `lgj_row_facet_match`, ABI minor 1→2, `docs/abi.md` §11 | **DONE 2026-08-17** — `cargo test` **84/84**, clippy/fmt clean, release build exports **18/18** symbols (`nm -D`). Parity: both kernels vs independent scalar references over 10 row counts × 2 seeds × 4 facets × 4 needles, cross-checked a THIRD way against `RowStore::classid_at`. Two-sided payload-vs-classid falsifier. End-to-end membrane test covers describe → predicate → mask algebra → count → facet-match → lifecycle | | D-LGJ-W3 | Java `RowStore` facade: structured `MemoryLayout`, minor-≥2 gate, `FacetMatchView`, parity test transcribing the generator | **DONE 2026-08-17** — dispatched per `.claude/waves/wave-substrate-w3-w4.md` (3 Sonnet workers, disjoint scopes: FFM membrane extension / public facade / tests), orchestrator-integrated. `javac -Xlint:all` clean (same 7 pre-existing `[restricted]` warnings, zero new). `AllTests` **185/185** (was 132; +53 new checks: 29 parity + 24 lifetime). **One real bug caught by the suite and fixed**: `FacetMatchView.rowCount()` was missing the closed-store guard `matchesOf`/`cardinality` both had — a stale row count was readable after the owning store closed. Fixed, re-verified. Both mandated disable-runs ran red-then-green: (1) `Abi.requireMinor` inflated by 1 → exactly `RowStoreParityTest`+`RowStoreLifetimeTest` failed, all 8 other suites stayed green; (2) the pure-Java generator's a/b draw order swapped in `RowStoreParityTest` → exactly that suite broke (17/29), `RowStoreLifetimeTest` (generator-independent) stayed green — confirming the parity test is a real falsifier, not decorative. `Mask.source()` retyped `NativePattern → NativeResource` (new interface) so a `Mask` can parent onto either a `NativePattern` or a `RowStore` — zero call-site breakage (verified: no existing caller bound the narrower type) | | D-LGJ-W4 | Bench Component F: Vector API facet scan vs the crossing, on the REAL layout | **DONE 2026-08-17** — 1 Sonnet worker (F_RowStoreFacetScan + RowStoreData + Kernels facet-match arms, cross-check-in-@Setup discipline), orchestrator-run JMH: 9/9 combos, cross-checks green at every row count. **Finding: Component C's direction survives, its margin collapses** — Vector API wins the 32-facet strided scan at every row count but by 2.51×/1.92×/1.14× (4K/65K/1M rows) vs C's 56×; at 512 MiB traversed all three arms converge on memory bandwidth. Native arm's per-call allocation asymmetry disclosed in §F with a named follow-up (`facetMatchesInto`), not hidden. summarise.sh extended with the F table (and the old 'E/F' section retitled 'E' — a real naming collision); tables regenerated from the merged CSV | -| D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | **trades DONE 2026-08-17** — `consumers/trades/` (own compile unit, core consumed as a third-party would): `Trade` (schema-not-entity: zero public ctors, zero instance fields, reflection-forced construction still throws), `World.open` → the existing lazy `View` under domain names, zero new membrane surface. TradesParityTest 12/12 (chain vs transcribed-generator recomputation at 1K+64K rows; 0 crossings composing / 1 at terminal THROUGH the domain vocabulary; reflection guard). TradesAllocationTest 3/3 — **the poster's number, measured: 240 bytes/query, IDENTICAL at 64K and 1M rows** (row-count independence is the thesis assertion; 64 KiB absolute backstop). Disable-run: VENUE pointed at the wrong lane → the membrane's own LANE_KIND_MISMATCH rejected it (the binding is checked, not trusted); restored green. **bricks DONE 2026-08-17** — `consumers/bricks/` (2 Sonnet workers K1/K2 per `.claude/waves/wave-consumer-bricks.md`): mask-first RBAC where `authorize(Role)` is a real natively-evaluated predicate in the SAME lazy chain as `where(...)` (`Role.EU_ONLY` = `REGION.eq(EU)`, `DENY_ALL` = `REGION.eq(0xFFFF)` — a genuine impossible predicate, not a Java branch), fail-closed (`UnauthorizedQueryException` BEFORE any crossing; no default-allow path exists), aggregate-only egress (every public method returns `BricksQuery`/`long`/`Map` — structurally no row-shaped type). BricksAuthTest **62/62**: parity vs transcribed generator at 1K+64K; RBAC-as-predicate equivalence (EU_ONLY result == GLOBAL+explicit-where); DENY_ALL counts 0 while paying a real crossing; crossing arithmetic — count()=1, sumBy()=**32 crossings (16 groups × 2: plan_eval + lgj_reduce_sum_i32), IDENTICAL at both row counts** (the thesis: crossings ∝ groups, never rows — the measured 32 corrected K1's "1 per group" Javadoc claim, a real finding about sum-terminal cost); reflection guards. Disable-run: `requireAuthorized` short-circuited → **exactly the 3 can-fire fail-closed checks red, 59 green**; restored, 62/62. Core suite unaffected (188/188). graph still shelved on the D1 ruling + edge-generator substrate change | +| D-LGJ-W5 | Three consumer examples (trades / bricks / graph) — one plan file each | **trades DONE 2026-08-17** — `consumers/trades/` (own compile unit, core consumed as a third-party would): `Trade` (schema-not-entity: zero public ctors, zero instance fields, reflection-forced construction still throws), `World.open` → the existing lazy `View` under domain names, zero new membrane surface. TradesParityTest 12/12 (chain vs transcribed-generator recomputation at 1K+64K rows; 0 crossings composing / 1 at terminal THROUGH the domain vocabulary; reflection guard). TradesAllocationTest 3/3 — **the poster's number, measured: 240 bytes/query, IDENTICAL at 64K and 1M rows** (row-count independence is the thesis assertion; 64 KiB absolute backstop). Disable-run: VENUE pointed at the wrong lane → the membrane's own LANE_KIND_MISMATCH rejected it (the binding is checked, not trusted); restored green. **bricks DONE 2026-08-17** — `consumers/bricks/` (2 Sonnet workers K1/K2 per `.claude/waves/wave-consumer-bricks.md`): mask-first RBAC where `authorize(Role)` is a real natively-evaluated predicate in the SAME lazy chain as `where(...)` (`Role.EU_ONLY` = `REGION.eq(EU)`, `DENY_ALL` = `REGION.eq(0xFFFF)` — a genuine impossible predicate, not a Java branch), fail-closed (`UnauthorizedQueryException` BEFORE any crossing; no default-allow path exists), aggregate-only egress (every public method returns `BricksQuery`/`long`/`Map` — structurally no row-shaped type). BricksAuthTest **62/62**: parity vs transcribed generator at 1K+64K; RBAC-as-predicate equivalence (EU_ONLY result == GLOBAL+explicit-where); DENY_ALL counts 0 while paying a real crossing; crossing arithmetic — count()=1, sumBy()=**32 crossings (16 groups × 2: plan_eval + lgj_reduce_sum_i32), IDENTICAL at both row counts** (the thesis: crossings ∝ groups, never rows — the measured 32 corrected K1's "1 per group" Javadoc claim, a real finding about sum-terminal cost); reflection guards. Disable-run: `requireAuthorized` short-circuited → **exactly the 3 can-fire fail-closed checks red, 59 green**; restored, 62/62. Core suite unaffected (188/188). **graph: substrate now fully unblocked (D-LGJ-W6, this pass) — dispatch is the next action, not yet executed** | +| D-LGJ-W6 | Edge-bearing row store ABI addition (`lgj_rowstore_open_with_edges`, minor 2→3, docs/abi.md §12) — the D1b-shaped "must land as its own W-tier PR before the consumer wave" the graph wave itself named | **DONE 2026-08-18** — orchestrator-authored (genuinely new ABI surface, not consumer-scope work): `registry::open_rowstore_with_edges` + `lgj_rowstore_open_with_edges` (mirrors `lgj_rowstore_open` exactly: same resource kind, same lane shape, no new mask op — purely an alternative constructor), `Engine.openRowStoreWithEdges`/`Abi.requireMinor(3)`, `RowStore.openWithEdges`. `cargo test` **93/93** (+3: registry-level open/describe, out-of-range-classid-matches-plain, radius-overflow-rejected), clippy/fmt clean, release build exports the new symbol (`nm -D`). Java: `AllTests` **194/194** (+6, all in `RowStoreParityTest`) — the strongest new result is a cross-language reproduction of the D1a hop mechanism itself: Java facet-matches + raw-lane-0 payload decode (zero new ABI op) reaches the EXACT same measured hop counts already pinned as a Rust regression (10-row seed → 19 at 1 hop → 29 at 2 hops, `n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, radius=25`) — proving the two sides of the membrane see identical edge structure, not merely identical classids. Two disable-runs, both red-then-green: (1) registry-level, a classid-not-threaded bug (`open_rowstore_with_edges` hardcoded classid `0`) caught by the out-of-range-parity test; (2) Java-level, the hop's classid-match condition forced to always skip → 1-hop/2-hop both went to 0 and the anti-vacuity assertion failed, exactly as expected. Caught mid-dispatch: the ABI-facing symbol did not exist before this pass (only the bare `RowStore::generate_with_edges` Rust function did, from the prior session) — the graph wave's own STOP-condition-RESOLVED note undersold what was still missing; closed here rather than discovered by G1/G2 mid-flight | diff --git a/docs/abi.md b/docs/abi.md index 3f7a665..ec8153e 100644 --- a/docs/abi.md +++ b/docs/abi.md @@ -62,10 +62,10 @@ cannot disagree with itself. The ABI is a **machine membrane**. It is not the product. The product is the Java semantic API (see `architecture.md`). Therefore: -- It is **small** — currently 18 symbols (minor 2; the "14" this line carried +- It is **small** — currently 19 symbols (minor 3; the "14" this line carried at minor 1 was arithmetic drift — the §7 list it referred to already enumerated 15). Growth is a design smell to be argued for, not a default; - minor 2's three additions are argued in §11. + minor 2's three additions are argued in §11, minor 3's one addition in §12. - It is **bulk-only**. Every call must be capable of doing work proportional to `n_rows` (see §6 — the anti-JNI rule). - It speaks **resource, lane, view, mask, operation, descriptor, status, @@ -78,7 +78,7 @@ semantic API (see `architecture.md`). Therefore: ``` LGJ_ABI_MAJOR = 0 // incompatible change ⇒ bump; Java refuses to load -LGJ_ABI_MINOR = 2 // additive change ⇒ bump; older Java may still load +LGJ_ABI_MINOR = 3 // additive change ⇒ bump; older Java may still load LGJ_MAGIC = 0x4C_47_4A_5F_41_42_49_00 // "LGJ_ABI\0" big-endian-read ``` @@ -269,7 +269,7 @@ predicates or rows are involved. The unfused per-predicate ops are retained only so the fused path can be benchmarked *against* something and so parity can be checked predicate-by-predicate. -## 7. The function surface (18 symbols) +## 7. The function surface (19 symbols) All symbols are prefixed `lgj_`. All return `i32` status except the manifest getter. `out_*` parameters are written only on `OK`. @@ -484,3 +484,65 @@ Rows are 512-byte strided within it. Nothing in this slice needs more — Java reads via `JAVA_INT_UNALIGNED`-class layouts, and every `ndarray::simd` load is a register fill. The 64-byte-aligned base guarantee arrives with the real `NodeRow` (`#[repr(C, align(64))]`) wiring. + +## 12. The edge-bearing row store (ABI minor ≥ 3) + +`consumer-graph-traversal-v1.md`'s falsifiers need a 1-2 hop BFS that is +*non-vacuous* — plain `lgj_rowstore_open`'s payload is uniform noise, so any +hop over it saturates to nearly every row within one or two steps +(`.claude/harvest/graph_density_probe.rs`'s own measurement). This is an +alternative **constructor**, not a new resource kind, new lane shape, or new +mask op — `LGJ_RESOURCE_ROWSTORE` and the whole §11 lane/operation surface +apply to its output unchanged. + +### Resource + +``` +i32 lgj_rowstore_open_with_edges(u64 n_rows, u64 seed, + u32 edge_classid, u64 edge_gate_mask, + u32 edge_radius, u64* out_handle) +``` + +Reuses §11's classid stream **byte-for-byte** — same two SplitMix64 draws per +facet, same `classid = (a >>> 33) & 0xF`. For a facet whose classid equals +`edge_classid` AND whose draw clears the sparsity gate (`a & edge_gate_mask +== 0`), the 12-byte payload instead carries a **structured target row**: + +``` +span = 2 * edge_radius + 1 +offset = (b % span) - edge_radius // signed, |offset| <= edge_radius +target = (row + offset) mod n_rows // bounded-local-neighbourhood +payload_lo64 = target as u64 // hi32 forced to 0 (the flag +payload_hi32 = 0 // a hop reads to know "structured") +``` + +Every other facet (classid mismatch, or gate not cleared) is byte-identical +to `lgj_rowstore_open`'s plain draw (`payload = le64(b) ++ le32(a & +0xFFFFFFFF)`) — so an **out-of-range `edge_classid`** (one that never occurs +in the classid stream, e.g. `16`) reproduces `lgj_rowstore_open` exactly, +which is how `RowStore::generate_with_edges`'s own test suite proves the two +generators share one code path rather than drifting apart. + +Density is governed by `edge_gate_mask`: the gate probability is +`1 / (16 * (edge_gate_mask + 1))` (16 candidate classids × the mask's extra +selectivity), so `edge_gate_mask = 0` is the densest edge-bearing setting. +`edge_radius` must be `< n_rows` (an unsatisfiable bound is +`LGJ_ERR_LENGTH_OVERFLOW`, matching every other overflow-shaped rejection in +this generator family). + +### A hop, at the Java layer (no new op — composition of existing symbols) + +A hop is `lgj_row_facet_match(edge_classid)` (§11, unchanged) to find which +of a row's facets carry the edge classid, followed by a **Java-side** decode +of each matched facet's `payload_lo64` as the target row and a scatter into +the next mask via `lgj_mask_describe`'s WRITABLE lane (§7) — one crossing per +hop, zero new ABI surface. See `consumer-graph-traversal-v1.md` Decision D1a. +A native `lgj_hop` symbol (D1b) remains a future ABI minor if Java-side +scatter measurably dominates; it is not part of this minor. + +### SIMD provenance (unchanged §8 rule, applied) + +The generator's per-facet loop is scalar (SplitMix64 is inherently +sequential per draw); the sparsity gate and target-row arithmetic are cheap +integer ops on that same sequential stream — no new SIMD kernel, matching +`RowStore::generate`'s own scalar generation loop. diff --git a/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java b/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java index b823b9f..e9f799e 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/RowStore.java @@ -63,6 +63,32 @@ public static RowStore open(long nRows, long seed) { return new RowStore(h, Engine.rowCount(h)); } + /** + * Open {@code nRows} rows generated deterministically from {@code seed}, with a sparse, gated + * subset of {@code edgeClassid}-matching facets carrying a bounded-local-neighbourhood target + * row instead of raw noise (docs/abi.md §12) — what a non-vacuous BFS traversal needs; {@link + * #open} alone produces uniform-random payloads that saturate any hop within one or two steps. + * + *

Byte-identical classid stream to {@link #open}. An {@code edgeClassid} that never occurs + * in the classid stream (e.g. {@code 16}, one past the 4-bit range) reproduces {@link #open} + * exactly. + * + * @param edgeGateMask sparsity gate: a facet is edge-shaped iff {@code a & edgeGateMask == 0} + * on its underlying draw; {@code 0} is the densest setting + * @param edgeRadius bounds how far a structured target may land from its source row; must be + * {@code < nRows} + * @throws NativeLibraryNotFoundException if the native artifact is not present + * @throws AbiMismatchException if it is present but ABI minor < 3 + */ + public static RowStore openWithEdges(long nRows, long seed, int edgeClassid, + long edgeGateMask, int edgeRadius) { + if (nRows < 0) { + throw new IllegalArgumentException("nRows must be >= 0, was " + nRows); + } + long h = Engine.openRowStoreWithEdges(nRows, seed, edgeClassid, edgeGateMask, edgeRadius); + return new RowStore(h, Engine.rowCount(h)); + } + /** How many rows this resource holds. */ @Override public long rowCount() { diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java index 45cb8e6..ad0a4a2 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Downcalls.java @@ -109,6 +109,13 @@ private Downcalls() {} FunctionDescriptor.of(ValueLayout.JAVA_INT, ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT, ValueLayout.ADDRESS, ValueLayout.JAVA_LONG)); + // Edge-bearing row store (docs/abi.md §12, ABI minor 3). One extra constructor over the + // minor-2 shape above — no new lane, no new op. + private static final MethodHandle ROWSTORE_OPEN_WITH_EDGES = mh("lgj_rowstore_open_with_edges", + FunctionDescriptor.of(ValueLayout.JAVA_INT, + ValueLayout.JAVA_LONG, ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT, + ValueLayout.JAVA_LONG, ValueLayout.JAVA_INT, ValueLayout.ADDRESS)); + private static MethodHandle mh(String symbol, FunctionDescriptor descriptor) { MemorySegment addr = Abi.lookup().find(symbol).orElseThrow(() -> new LanceGraphException("the native library exports no symbol '" + symbol @@ -355,6 +362,26 @@ public static long rowstoreOpen(long nRows, long seed, MemorySegment outHandle) return outHandle.get(ValueLayout.JAVA_LONG, 0); } + /** + * Build the edge-bearing SoA row store (docs/abi.md §12). Byte-identical classid stream to + * {@link #rowstoreOpen}; a sparse, gated subset of {@code edgeClassid}-matching facets carries + * a bounded-local-neighbourhood target row instead of raw noise. + */ + public static long rowstoreOpenWithEdges(long nRows, long seed, int edgeClassid, + long edgeGateMask, int edgeRadius, + MemorySegment outHandle) { + crossed(); + int st; + try { + st = (int) ROWSTORE_OPEN_WITH_EDGES.invokeExact( + nRows, seed, edgeClassid, edgeGateMask, edgeRadius, outHandle); + } catch (Throwable t) { + throw wrap("lgj_rowstore_open_with_edges", t); + } + Status.check("lgj_rowstore_open_with_edges", st); + return outHandle.get(ValueLayout.JAVA_LONG, 0); + } + /** * Overwrites {@code dstMask} with {@code classid(facet, row) == needle}. {@code facet} is a * facet index {@code 0..32} into the row's 32 facet lanes, not a lane id diff --git a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java index 69cf2a1..0db3a18 100644 --- a/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java +++ b/java/src/main/java/com/adaworldapi/lancegraph/internal/ffm/Engine.java @@ -179,6 +179,25 @@ public static long openRowStore(long nRows, long seed) { return Downcalls.rowstoreOpen(nRows, seed, s.out); } + /** + * Open the edge-bearing SoA row store (docs/abi.md §12): byte-identical classid stream to + * {@link #openRowStore}, plus a sparse, gated subset of {@code edgeClassid}-matching facets + * carrying a bounded-local-neighbourhood target row instead of raw noise — what a non-vacuous + * BFS over the row store needs. Returns the resource handle. Requires ABI minor >= 3. + * + * @param edgeGateMask sparsity gate: a facet is edge-shaped iff {@code a & edgeGateMask == 0} + * on its draw; {@code 0} is the densest setting + * @param edgeRadius bounds how far a structured target may land from its source row; must be + * {@code < nRows} + */ + public static long openRowStoreWithEdges(long nRows, long seed, int edgeClassid, + long edgeGateMask, int edgeRadius) { + Abi.requireMinor(3); + Scratch s = SCRATCH.get(); + return Downcalls.rowstoreOpenWithEdges(nRows, seed, edgeClassid, edgeGateMask, edgeRadius, + s.out); + } + /** * Overwrite {@code dstMask} with {@code classid(facet, row) == classId} for every row of * {@code store} (docs/abi.md §11). {@code facet} is a facet index {@code 0..32} into the row's diff --git a/java/src/test/java/com/adaworldapi/lancegraph/RowStoreParityTest.java b/java/src/test/java/com/adaworldapi/lancegraph/RowStoreParityTest.java index eaede48..60ad1f5 100644 --- a/java/src/test/java/com/adaworldapi/lancegraph/RowStoreParityTest.java +++ b/java/src/test/java/com/adaworldapi/lancegraph/RowStoreParityTest.java @@ -210,5 +210,105 @@ public static void run(Checks c) { c.eq("all " + (n1000 * FACETS_PER_ROW) + " classids read through ROW_LAYOUT match the" + " transcribed generator", 0, layoutMismatches); } + + // ── the edge-bearing generator (docs/abi.md §12) ────────────────────────────────────── + // + // Two claims, both genuinely new (not a restatement of the sections above): (1) an + // out-of-range edgeClassid reproduces RowStore.open's classid stream exactly, THROUGH the + // Java facade rather than only at the Rust generator level; (2) a Java-side hop — the D1a + // mechanism `consumer-graph-traversal-v1.md` chose (facet-match crossing + raw-segment + // payload decode, zero new ABI op) — reproduces the SAME measured hop counts already + // pinned as a Rust regression (rowstore.rs's `measured_hop_counts_are_...` test), proving + // the two sides of the membrane see identical edge structure, not merely identical + // classids. + + c.section("openWithEdges: an out-of-range edgeClassid matches RowStore.open's classid stream"); + int nEdgeParity = 500; + int[][] plainClassids = generateClassids(nEdgeParity, seed); + try (RowStore edged = RowStore.openWithEdges(nEdgeParity, seed, 16, 0x0, 5)) { + c.eq("row count is unaffected", nEdgeParity, edged.rowCount()); + long classidMismatches = 0; + for (int classId : new int[] {0, 9, 15}) { + long expected = countFacetClassid(plainClassids, 7, classId); + try (Mask mask = edged.maskOfFacetClass(new FacetId(7), classId)) { + if (mask.count() != expected) { + classidMismatches++; + } + } + } + c.eq("classid 16 never occurs in the 4-bit stream, so every facet-7 classid count" + + " matches the plain generator exactly", 0, classidMismatches); + } + + c.section("openWithEdges: a Java-side hop reproduces the pinned Rust regression"); + // Exact parameters of rowstore.rs's own `measured_hop_counts_are_three_distinct_non_empty_ + // non_total_sizes`: n=2000, seed=0xF00D_CAFE, edge_classid=0, gate_mask=0x0, radius=25, + // seed set = i*37+5 for i in 0..10. + long hopN = 2000; + int edgeClassid = 0; + try (RowStore hopStore = RowStore.openWithEdges(hopN, 0xF00D_CAFEL, edgeClassid, 0x0L, 25)) { + java.lang.foreign.MemorySegment raw = + com.adaworldapi.lancegraph.internal.ffm.Engine.describeLane(hopStore.handle(), 0) + .segment(); + long rowBytes2 = com.adaworldapi.lancegraph.internal.ffm.Layouts.ROW_LAYOUT.byteSize(); + + java.util.function.Function hop = from -> { + boolean[] seenRows = new boolean[(int) hopN]; + java.util.List out = new java.util.ArrayList<>(); + for (long row : from) { + for (int facet = 0; facet < FACETS_PER_ROW; facet++) { + long classOff = com.adaworldapi.lancegraph.internal.ffm.Layouts.ROW_LAYOUT + .byteOffset( + java.lang.foreign.MemoryLayout.PathElement.sequenceElement( + facet), + java.lang.foreign.MemoryLayout.PathElement.groupElement( + "classid")); + int classid = raw.get(java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED, + row * rowBytes2 + classOff); + if (classid != edgeClassid) { + continue; + } + long payloadOff = com.adaworldapi.lancegraph.internal.ffm.Layouts.ROW_LAYOUT + .byteOffset( + java.lang.foreign.MemoryLayout.PathElement.sequenceElement( + facet), + java.lang.foreign.MemoryLayout.PathElement.groupElement( + "payload")); + long base = row * rowBytes2 + payloadOff; + int hi32 = raw.get(java.lang.foreign.ValueLayout.JAVA_INT_UNALIGNED, + base + 8); + if (hi32 != 0) { + continue; + } + long target = raw.get(java.lang.foreign.ValueLayout.JAVA_LONG_UNALIGNED, + base); + if (target >= 0 && target < hopN && !seenRows[(int) target]) { + seenRows[(int) target] = true; + out.add(target); + } + } + } + long[] arr = new long[out.size()]; + for (int i = 0; i < arr.length; i++) { + arr[i] = out.get(i); + } + return arr; + }; + + long[] seedRows = new long[10]; + for (int i = 0; i < 10; i++) { + seedRows[i] = i * 37L + 5; + } + long[] oneHop = hop.apply(seedRows); + long[] twoHop = hop.apply(oneHop); + + c.eq("seed set size", 10, seedRows.length); + c.eq("1-hop set size matches the Rust-pinned regression", 19, oneHop.length); + c.eq("2-hop set size matches the Rust-pinned regression", 29, twoHop.length); + c.that("three different, non-empty, non-total sizes (anti-vacuity)", + seedRows.length != oneHop.length && oneHop.length != twoHop.length + && oneHop.length > 0 && twoHop.length > 0 + && oneHop.length < hopN && twoHop.length < hopN); + } } } diff --git a/native/lgj-abi/src/abi.rs b/native/lgj-abi/src/abi.rs index d919922..2bdda6a 100644 --- a/native/lgj-abi/src/abi.rs +++ b/native/lgj-abi/src/abi.rs @@ -36,7 +36,15 @@ pub const LGJ_ABI_MAJOR: u32 = 0; /// `lgj_rowstore_open`, `lgj_op_eq_classid`, `lgj_row_facet_match`, and /// strided facet lanes described through the (unchanged) `LgjLaneDesc`. /// Purely additive; a minor-1 Java loads and sees none of it. -pub const LGJ_ABI_MINOR: u32 = 2; +/// +/// Minor **3** (2026-08-18): `lgj_rowstore_open_with_edges` — the +/// edge-bearing row-store generator (abi.md §12, +/// `RowStore::generate_with_edges`), unblocking `consumer-graph-traversal-v1.md`. +/// Byte-identical `LGJ_RESOURCE_ROWSTORE` resource kind, no new lane shape, +/// no new mask op; purely an alternative constructor. A minor-2 Java loads +/// fine and simply cannot call the new symbol (`Abi.requireMinor(3)` gates +/// it, matching the row store's own minor-2 gate pattern). +pub const LGJ_ABI_MINOR: u32 = 3; /// `"LGJ_ABI\0"` read big-endian. /// diff --git a/native/lgj-abi/src/exports.rs b/native/lgj-abi/src/exports.rs index e2dd2aa..6f80945 100644 --- a/native/lgj-abi/src/exports.rs +++ b/native/lgj-abi/src/exports.rs @@ -138,6 +138,58 @@ pub unsafe extern "C" fn lgj_rowstore_open(n_rows: u64, seed: u64, out_handle: * }) } +/// Build the edge-bearing SoA **row store** (abi.md §12) and return its +/// handle: byte-identical classid stream to [`lgj_rowstore_open`], plus a +/// sparse, gated subset of `edge_classid`-matching facets carrying a +/// bounded-local-neighbourhood target row (docs/abi.md §12; the mechanism +/// `consumer-graph-traversal-v1.md`'s hop falsifiers need). ABI minor ≥ 3. +/// +/// `edge_gate_mask` selects sparsity (a facet is edge-shaped iff `a & +/// edge_gate_mask == 0` on its draw — abi.md §12); `edge_radius` bounds how +/// far a structured target may land from its source row. An out-of-range +/// `edge_classid` (one that never occurs in the plain classid stream) +/// reproduces [`lgj_rowstore_open`] byte-for-byte — proven by +/// `RowStore::generate_with_edges`'s own test suite. +/// # Safety +/// +/// A null pointer is *handled*, not UB: it returns `NULL_ARGUMENT`. Beyond +/// that, `out_handle` must be a valid, aligned, writable `u64`. Written only +/// on success. +/// +/// `unsafe` here is a note to Rust callers linking the `rlib`. The JVM, +/// which is the real caller, has no such concept — it upholds the same +/// contract by construction, because every pointer it passes comes from a +/// `MemorySegment` whose size and alignment it derived from the manifest. +#[no_mangle] +pub unsafe extern "C" fn lgj_rowstore_open_with_edges( + n_rows: u64, + seed: u64, + edge_classid: u32, + edge_gate_mask: u64, + edge_radius: u32, + out_handle: *mut u64, +) -> i32 { + guard(|| { + if out_handle.is_null() { + return LGJ_ERR_NULL_ARGUMENT; + } + match registry::open_rowstore_with_edges( + n_rows, + seed, + edge_classid, + edge_gate_mask, + edge_radius, + ) { + Ok(h) => { + // SAFETY: non-null (checked above); written only on success. + unsafe { *out_handle = h }; + LGJ_OK + } + Err(e) => e, + } + }) +} + /// Free a resource: its lanes are dropped, its generation is bumped, and its /// children begin failing with `PARENT_CLOSED`. /// diff --git a/native/lgj-abi/src/registry.rs b/native/lgj-abi/src/registry.rs index dda345f..f9b278a 100644 --- a/native/lgj-abi/src/registry.rs +++ b/native/lgj-abi/src/registry.rs @@ -304,6 +304,32 @@ pub fn open_rowstore(n_rows: u64, seed: u64) -> Result { }) } +/// Create a row-store resource from the edge-bearing generator (abi.md §12, +/// ABI minor 3): identical classid stream to [`open_rowstore`], plus a +/// sparse, gated subset of `edge_classid`-matching facets carrying a +/// bounded-local-neighbourhood target row instead of raw noise — the +/// mechanism `consumer-graph-traversal-v1.md` needs a non-vacuous BFS at +/// all. See [`RowStore::generate_with_edges`] for the exact generator. +pub fn open_rowstore_with_edges( + n_rows: u64, + seed: u64, + edge_classid: u32, + edge_gate_mask: u64, + edge_radius: u32, +) -> Result { + let store = + RowStore::generate_with_edges(n_rows, seed, edge_classid, edge_gate_mask, edge_radius) + .ok_or(LGJ_ERR_LENGTH_OVERFLOW)?; + insert(ResourceEntry { + kind: LGJ_RESOURCE_ROWSTORE, + epoch: next_epoch(), + n_rows, + parent: 0, + parent_gen: 0, + payload: Payload::RowStore(store), + }) +} + /// Create a mask over `parent`, all bits `0` or all bits `1`. /// /// A mask's parent may be a pattern OR a row store — both are read-only @@ -532,6 +558,53 @@ mod tests { close(h).unwrap(); } + /// The registry-level twin of `rowstore.rs`'s own `generate_with_edges` + /// tests: proves the ABI-facing constructor (abi.md §12) reaches the same + /// generator through the full `open_rowstore_with_edges` → registry → + /// `ResourceEntry` path, not just the bare `RowStore` type. + #[test] + fn rowstore_with_edges_opens_and_describes_itself() { + let h = open_rowstore_with_edges(70, 3, 0, 0x0, 10).unwrap(); + let e = resolve_kind(h, LGJ_RESOURCE_ROWSTORE).unwrap(); + let info = e.info(); + assert_eq!(info.kind, LGJ_RESOURCE_ROWSTORE); + assert_eq!(info.lane_count, ROWSTORE_LANE_COUNT); + assert_eq!(info.n_rows, 70); + assert!(e.rowstore().is_some()); + close(h).unwrap(); + } + + /// abi.md §12's own claim, re-proven at the registry boundary: an + /// `edge_classid` outside the 4-bit classid range reproduces + /// `open_rowstore` byte-for-byte through the SAME entry point a Java + /// caller uses — not merely at the underlying `RowStore::generate*` + /// level, which `rowstore.rs` already covers. + #[test] + fn out_of_range_edge_classid_matches_plain_open_through_the_registry() { + let plain = open_rowstore(200, 0xABCD).unwrap(); + let edged = open_rowstore_with_edges(200, 0xABCD, 16, 0x0, 5).unwrap(); + let pe = resolve_kind(plain, LGJ_RESOURCE_ROWSTORE).unwrap(); + let ee = resolve_kind(edged, LGJ_RESOURCE_ROWSTORE).unwrap(); + assert_eq!( + pe.rowstore().unwrap().as_bytes(), + ee.rowstore().unwrap().as_bytes() + ); + close(plain).unwrap(); + close(edged).unwrap(); + } + + /// abi.md §12's overflow rule at the registry boundary: `edge_radius >= + /// n_rows` is `LGJ_ERR_LENGTH_OVERFLOW`, matching every other + /// overflow-shaped rejection in this generator family (`open_rowstore`'s + /// own `LGJ_ERR_LENGTH_OVERFLOW` on a too-large `n_rows`). + #[test] + fn radius_that_cannot_fit_is_rejected_through_the_registry() { + assert_eq!( + open_rowstore_with_edges(10, 1, 0, 0, 10), + Err(LGJ_ERR_LENGTH_OVERFLOW) + ); + } + /// A mask parents onto a row store exactly as onto a pattern — same /// row-count sizing, same tail rule, same parent-liveness propagation. #[test]