Skip to content
Merged
30 changes: 30 additions & 0 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,36 @@
> `**Status:**`/`**Confidence:**` line. A correction gets its own new,
> dated entry that references the one it corrects — the storno rule.

## 2026-08-17 — E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1

**Status:** DECISION (declined refactor, with the trigger for revisiting named).
**Confidence:** High — decided by reading `ndarray/src/simd_soa.rs`'s full API, not by taste.

Operator suggestion: "if you use SoA, calling simd_soa.rs would make sense" — should
`native/lgj-abi/src/kernels.rs` route through `ndarray::simd_soa::MultiLaneColumn` (the canonical
`Arc<[u8]>` SoA carrier) instead of raw `&[u32]`/`&[i32]` slices? **Answer: not for today's
flat-lane fixture; yes for the future 512-byte row-store slice.** Two concrete API mismatches,
not a style call:

1. **No tail handling.** `MultiLaneColumn::new()` hard-requires `len % 64 == 0`; every `iter_*`
yields only full 64-byte chunks via `as_chunks::<64>()` — no remainder arm. The
`simd_int_ops` primitives this project consumes do the opposite by design: full 16-lane
groups + a scalar tail for arbitrary caller-chosen `n_rows`. Wrapping the fixture's lanes in
`MultiLaneColumn` would force 64-byte padding on every allocation, bought for nothing.
2. **No `u32` lane.** `MultiLaneColumn` ships u8x64/f32x16/f64x8/u64x8/i32x16/i64x8 iterators —
no u32. The fixture's `ids`/`classes` are `u32` (`eq_u32_to_mask`).

So `kernels.rs` already calls the correct layer: the `ndarray::simd_int_ops` primitives own their
chunking internally. `MultiLaneColumn` sits *above* that layer, for uniform pre-padded columns.

**Where it DOES fit — the operator-stated layout reference (recorded verbatim so it survives):**
"the 64k x 512 bytes SoA layout is enforced everywhere in lance-graph (32 Lanes each 4 bytes
classview+12 bytes). For Java the layout might differ — just for reference." A 512-byte,
64-byte-aligned row store (32 × 16-byte V3 facets) is padded/aligned *by construction* — no tail
problem — and each row is a natural `iter_u8x64` chunk-of-chunks. When the real
`NodeRow`/facet slice replaces the generic fixture (`docs/abi.md` §10, `docs/architecture.md`
"where a real graph slice would attach"), `MultiLaneColumn` is the type to reach for. Not before.

## 2026-08-17 — E-LGJ-VECTOR-API-BEATS-THE-CROSSING-1

**Status:** FINDING. **Confidence:** High (real JMH 1.37, `Data.crossCheck()` guards every fork,
Expand Down
24 changes: 24 additions & 0 deletions .claude/board/LATEST_STATE.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
## 2026-08-17 (later) — Phase I docs written, fusion re-run merged, simd_soa question answered (PR #4)

- **All four synthesis docs shipped** (`docs/architecture.md`,
`docs/panama.md`, `docs/valhalla-lab.md`, `docs/execution-boundary.md`)
— D-LGJ-I DONE. Each cites the proving artifact instead of restating it.
- **Fusion sweep re-run with a 256-row arm** (`./run.sh E_`): the first
pass's "fusion does nothing" (true at 65,536 rows, where kernel time
dominates) is false at small rows — unfused/fused grows 0.95× → 2.99×
at 256 rows × 8 predicates, because per-crossing overhead dominates
there. `RESULTS.md` rewritten from `jmh-results-merged.csv` (A/B/C from
the full sweep + E from the re-run), `TABLES.md` mechanically generated
from the same file. Valhalla lab result files refreshed by a same-box
re-run; findings unchanged.
- **`MultiLaneColumn` question answered** (operator: "if you use SoA,
calling simd_soa.rs would make sense"): declined for the flat-lane
fixture (64-byte-multiple constraint + no u32 lane — two concrete API
mismatches), earmarked for the 512-byte row-store slice where it fits
by construction. Operator layout reference recorded: 64K × 512 B rows,
32 lanes × (4 B classid + 12 B), enforced everywhere in lance-graph;
Java-side layout may differ. See
`E-LGJ-SIMD-SOA-IS-FOR-THE-ROW-STORE-NOT-THE-FLAT-LANES-1`.
- **PR_ARC_INVENTORY backfilled** for merged PRs #1-#3 (hygiene lapse
owned in the file itself).

## 2026-08-17 — D-LGJ-AUDIT complete, core vertical slice VERIFIED GREEN, PR #1 opened

### Current Contract Inventory — the vertical slice is real and green
Expand Down
62 changes: 52 additions & 10 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,56 @@
# updatable in place — corrections append as new dated lines; reversals
# get their own PR entry)

_No PR has been opened against this repository yet — session 1, 2026-08-17._
> **Hygiene lapse, owned (2026-08-17):** PRs #1-#3 merged without their
> entries landing in the same commit — the exact retroactive-hygiene
> anti-pattern the imported board rules name. Backfilled below in one
> pass rather than left stale; PR #4 onward gets its entry at merge time.

The first entry in this file will be written when the first PR against
`lance-graph-java` merges (expected: the vertical slice on
`claude/lance-graph-java-panama-valhalla-sus9w8` → `main`, once
`ISS-LGJ-FANOUT-UNREVIEWED` closes). Until then, ground truth for
in-progress work lives on `LATEST_STATE.md` (current contract inventory),
`STATUS_BOARD.md` (per-D-id status), and `AGENT_LOG.md` (what actually
happened) — this file stays empty rather than backfilled with a
pre-registration entry that would misrepresent something as merged before
it is.
## PR #3 — Vector API bench: real JMH, cross-checked (merged 2026-08-17, squash)

- **Added:** `bench/` — real JMH 1.37 suite (Components A/B/C/E:
downcall overhead, segment access, execution boundary sweep 64→4.2M
rows, fusion/planning), `Data.crossCheck()` gating every fork,
`summarise.sh` mechanical table generator, `RESULTS.md`, raw
run logs + CSV.
- **Locked:** the headline finding — Java Vector API zero-copy on the
native segment beats the native crossing at every row count tested
(56.4× → 1.33×); native beats Java *scalar* only past ~4K-16K rows.
Recorded as `E-LGJ-VECTOR-API-BEATS-THE-CROSSING-1`.
- **Deferred:** fusion sweep ran at 65,536 rows only (repaid post-merge
by the E_ re-run with a 256-row arm — see PR #4).
- **Docs:** `bench/README.md`, board updates.
- **Confidence:** High — 50/50 rows, 0 failures, two independent
computations of the same CSV agree.

## PR #2 — Valhalla lab: three-truths, causal isolation, 3 reproducers (merged 2026-08-17, squash)

- **Added:** `valhalla-lab/` — shared/stable/valhalla trees, self-verifying
`run.sh` (vocab-diff honesty gate + flattening-flag causal isolation),
`docs/three-truths.md`, reproducers R1/R2/R3 with observed outputs.
- **Locked:** the 8-byte array-flattening cliff (R2, VM-confirmed);
native-one-crossing beats hydration ~38-57× on BOTH JDKs; production
API adopts zero Valhalla-only mechanisms — migration stays
`record` → `value record`, one word per type.
- **Deferred:** nothing; the lab is complete for this vocabulary.
- **Docs:** lab README + three-truths; board updates.
- **Confidence:** High — one real defect (`Class::isValue()` not on
JDK 26) found by compile failure and fixed before landing.

## PR #1 — Core vertical slice: ABI contract, native crate, Java facade (merged 2026-08-17, squash)

- **Added:** `docs/abi.md` (normative, 14 symbols / 4 repr(C) types /
13 status codes / generation-checked handles); `native/lgj-abi`
(72/72, clippy/fmt clean, 14/14 exported symbols via `nm -D`);
`java/` facade + FFM membrane (132/132, reflection-enforced zero-FFM
public surface); 5 new `ndarray::simd` primitives under the W1a
contract (41/41); the `.claude/` ensemble + board.
- **Locked:** disable-verified generation check (exactly 2 tests red
when broken, 70 green); the manifest cross-check rejects a real wrong
`.so`; laziness measured (0 crossings to build, exactly 1 to
evaluate); target-cpu=x86-64-v4 divergence recorded.
- **Deferred:** real graph types (`NodeRow`/`WideFieldMask`) — generic
fixture first, by design (`docs/abi.md` §10).
- **Docs:** `docs/abi.md`, knowledge docs, board.
- **Confidence:** High — one real audit violation (`ndarray::hpc`
import) found and fixed pre-merge; recorded in EPIPHANIES.
2 changes: 1 addition & 1 deletion .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ list.
| D-LGJ-F | Valhalla lab — three-truths method on the small semantic value vocabulary | **DONE 2026-08-17** — `valhalla-lab/`: 4 experiments + a self-verifying `run.sh` (mechanically diffs the two `Vocab.java`s modulo the `value` keyword before trusting the A/B) + 3 causal-isolation runs (escape-analysis off; `UseArrayFlattening`/`UseFieldFlattening` toggled independently). 3 real Valhalla limitations reproduced and filed under `reproducers/` (R1: `@NullRestricted` field on an identity class is a `VerifyError`, javac's fault — no source form expresses required strict-field order; **R2: array flattening has a hard 8-byte payload cliff, VM-confirmed via `-XX:+PrintFlatArrayLayout`** — `LaneId`/`Ordinal`/`MaskId` (≤8B) flatten, `RowRange`/`Row` (16B) do not, so "Valhalla helps descriptors not entities" is a measured VM cutoff, not a hand-wave, and `RowRange` landing on the wrong side is flagged as the one place the expectation was too optimistic; R3: the densest null-restricted array form is `jdk.internal`-only and generics erase flattening entirely — `Foo!` type syntax confirmed NOT to parse, matching the archaeology finding). 1 real defect found + fixed before landing (see `EPIPHANIES.md`). None of the three limitations changed the production API — the migration path stays exactly `record` → `value record` | I |
| D-LGJ-G | Java Vector API comparative bench vs Panama→`ndarray::simd` | **DONE 2026-08-17** — real JMH 1.37 (fork+warmup+blackholes confirmed in the log), 50/50 rows, 0 failures, `Data.crossCheck()` guards every fork. **Headline (Component C, single predicate, zero-copy `IntVector.fromMemorySegment`): the Java Vector API beats the native crossing at EVERY row count tested, 64 to 4,194,304** — 56.4x at small sizes down to 1.3-1.4x at the largest. Native beats a plain Java scalar loop only past ~4,096-16,384 rows. Component E: SIMD-vs-scalar is the biggest lever measured (10.8x-31.1x); fused vs unfused are within noise of each other at 65,536 rows (crossing-count guarantee matters more than measured time here, since Component A puts one downcall at ~22ns). Independently cross-checked: hand-written `RESULTS.md` numbers verified byte-for-byte against `summarise.sh`'s mechanically-generated tables from the same CSV | I |
| D-LGJ-H | Falsification: handle lifecycle (adversarial), SIMD/scalar parity, Java/native parity | **DONE 2026-08-17, all scopes closed** — Rust+Java core (D-LGJ-C disable-verification, D-LGJ-E `FusionParityTest`/`LifetimeTest`); Valhalla lab (`run.sh`'s vocab-honesty self-check + causal-isolation runs); bench (`Data.crossCheck()` on every fork, `summarise.sh` cross-check) | I |
| D-LGJ-I | Docs: `architecture.md`, `panama.md`, `valhalla-lab.md`, `execution-boundary.md` | **Unblocked** — F and G both landed; next action | — |
| D-LGJ-I | Docs: `architecture.md`, `panama.md`, `valhalla-lab.md`, `execution-boundary.md` | **DONE 2026-08-17** — all four written as synthesis (each cites the artifact that proves its claim rather than restating it); `execution-boundary.md` additionally records the three structural hot-path facts (zero-copy precision incl. the lance-graph `SoaEnvelope` inheritance, no-thread-pool/caller-threads-are-the-parallelism, `array_windows`/`array_chunks` precisely traced as un-invoked at any input size). Ships in PR #4 with the fusion-sweep 256-row re-run merged into `RESULTS.md`/`TABLES.md` | — |
| D-LGJ-AUDIT | Mechanical post-fan-out audit: `grep` for `ndarray::hpc` imports, any `.h`/`cbindgen`/`jextract` artifact, any FFM type leaking into public Java API | **DONE 2026-08-17** — 1 real violation found (`kernels.rs::simd_popcount` used the internal `ndarray::hpc::bitwise` path), fixed in place; everything else confirmed to be the one sanctioned exception or explanatory prose | closed D-LGJ-C/D/E for the core |

### Reading this table
Expand Down
Loading