Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions .claude/board/EPIPHANIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,57 @@
> `**Status:**`/`**Confidence:**` line. A correction gets its own new,
> dated entry that references the one it corrects — the storno rule.

## 2026-08-17 — E-LGJ-VALHALLA-MEASURED-NOT-ASSUMED-1

**Status:** FINDING. **Confidence:** High (real numbers, both JDKs actually
run, reproducible via `valhalla-lab/README.md`).

The mandatory N-objects-vs-N-values-vs-1-lane experiment
(`.claude/knowledge/valhalla-three-truths-method.md`'s "one experiment that
must never be skipped") ran on both real JDKs. Headline, on 65,536 rows,
identical question, identical answer on every path:

| | native, one crossing | hydrate 65,536 `Row`, then scan |
|---|---:|---:|
| stable JDK 26 | 19.5 µs, 289 KiB | 746 µs, 2.00 MiB |
| Valhalla JDK 27 EA | 15.7 µs, 289.5 KiB | 900 µs, 2.50 MiB |

**The thesis's prediction held, and the reason why is itself a measured
finding, not an assumption:** `LaneId` (one field) measured `FLAT` under
Valhalla via the real VM query `ValueClass.isFlatArray` (2.90 B/element vs
16.00 B on stable — ~5.5× smaller), but `Row` (multiple fields) measured
**`NOT-FLAT`** even under Valhalla, and its per-row heap cost (40.01 B) was
*larger* than the stable JDK's own record-array cost (32.01 B). Valhalla
genuinely helps a single-field descriptor; it did not flatten the
multi-field materialization the thesis explicitly said to check rather
than assume away.

**One real defect found and fixed before this landed** — a bug of the
falsifiability-discipline-caught-it, not the happy-path-hid-it kind. The
first version of `IdentityExperiment` and the stable-JDK `Platform` called
`Class::isValue()` directly on four vocabulary types, with a comment
incorrectly asserting *"Class::isValue is final API on JDK 26."* It does
not exist there at all — confirmed by a real `javac` compile failure, not
by re-reading documentation. Fixed by routing every identity query through
`Platform.isValueClass(Class<?>)`: the stable half answers `false`
honestly (a JDK with no value-class concept can never produce one — the
answer is exact, not a guess, unlike the genuinely-unknowable
`arrayFlatness` case the same file already handles correctly), the
Valhalla half answers with the real `type.isValue()`. The correction
mirrors `E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1`'s finding about
`kernels.rs`: an agent's own doc comment stated the WRONG fact confidently
one line above the code that relied on it, and only compiling both
variants for real (not trusting the report that they "should" compile)
caught it.

**Two javac usage facts worth keeping** (real dead ends this session hit
and resolved, recorded so a future session doesn't re-hit them):
`--release N` cannot be combined with `--add-exports` for a system module
(a hard javac restriction, not a bug) — use `-source N` instead when
compiling for the same JDK you'll run on; and `--enable-preview` requires
an explicit `-source`/`--release` to be present at all, it is not
self-sufficient.

## 2026-08-17 — E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1

**Status:** FINDING. **Confidence:** High (measured, not asserted — every
Expand Down
4 changes: 2 additions & 2 deletions .claude/board/STATUS_BOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ list.
| D-LGJ-C | `native/lgj-abi` — manifest, generation-checked registry, generic SoA fixture, kernels, `extern "C"` surface | **DONE 2026-08-17** — `cargo test` **72/72**, `clippy -D warnings` clean, `fmt --check` clean, release build → 14/14 symbols verified via `nm -D`. **Disable-verified**: the registry's generation check was short-circuited and exactly the 2 tests that should catch it went red, 70 stayed green; restored, re-verified 72/72 | D, H |
| D-LGJ-D | Java FFM membrane `internal/ffm` | **DONE 2026-08-17** — compiles clean with `-Xlint:all`; 7 `[restricted]` warnings, all in `internal/ffm/*` or a test deliberately exercising it; `AbiContractTest` 7/7 incl. proving the manifest cross-check genuinely rejects a wrong `.so` (`libz.so.1` loads but is refused for exporting no `lgj_abi_manifest`) | E |
| D-LGJ-E | Java public facade (`NativePattern`/`View`/`Predicate`/`Pattern`/`Mask`) | **DONE 2026-08-17** — `AllTests` **132/132**: `ApiSurfaceTest` (reflection-enforced zero-FFM-leakage), `SmokeTest` 14/14, `FixtureParityTest` 30/30 (Java independently recomputes expected counts from the transcribed generator), `FusionParityTest` 31/31 (fused/unfused/scalar bit-identical across 6 row-count shapes), `LazinessTest` 8/8 (empirically: 0 crossings to build a 16-condition chain, exactly 1 to evaluate it, independent of rows up to 1,000,000 — the thesis's central claim, measured), `NarrowingTest` 16/16, `LifetimeTest` 23/23 | F, G |
| D-LGJ-F | Valhalla lab — three-truths method on the small semantic value vocabulary | **In flight** — sequenced after E, now reading the real Java types; deferred to a follow-up PR, not blocking PR #1 | I |
| D-LGJ-G | Java Vector API comparative bench vs Panama→`ndarray::simd` | **In flight** — real JMH jars fetched (`jmh-core`/`jmh-generator-annprocess`/`jopt-simple`/`commons-math3`) to `bench/lib/` (gitignored); no bench source written yet; deferred to the same follow-up PR as F | I |
| D-LGJ-F | Valhalla lab — three-truths method on the small semantic value vocabulary | **DONE 2026-08-17** — `valhalla-lab/`: 4 experiments + a self-verifying `run.sh` (mechanically diffs the two `Vocab.java`s modulo the `value` keyword before trusting the A/B) + 3 causal-isolation runs (escape-analysis off; `UseArrayFlattening`/`UseFieldFlattening` toggled independently). 3 real Valhalla limitations reproduced and filed under `reproducers/` (R1: `@NullRestricted` field on an identity class is a `VerifyError`, javac's fault — no source form expresses required strict-field order; **R2: array flattening has a hard 8-byte payload cliff, VM-confirmed via `-XX:+PrintFlatArrayLayout`** — `LaneId`/`Ordinal`/`MaskId` (≤8B) flatten, `RowRange`/`Row` (16B) do not, so "Valhalla helps descriptors not entities" is a measured VM cutoff, not a hand-wave, and `RowRange` landing on the wrong side is flagged as the one place the expectation was too optimistic; R3: the densest null-restricted array form is `jdk.internal`-only and generics erase flattening entirely — `Foo!` type syntax confirmed NOT to parse, matching the archaeology finding). 1 real defect found + fixed before landing (see `EPIPHANIES.md`). None of the three limitations changed the production API — the migration path stays exactly `record` → `value record` | I |
| D-LGJ-G | Java Vector API comparative bench vs Panama→`ndarray::simd` | **In flight** — real JMH + JOL jars fetched (`jmh-core`/`jmh-generator-annprocess`/`jopt-simple`/`commons-math3`/`jol-core`) to `bench/lib/` (gitignored); no bench source written yet; the ONLY remaining open row | I |
| D-LGJ-H | Falsification: handle lifecycle (adversarial), SIMD/scalar parity, Java/native parity | **DONE 2026-08-17 for the Rust+Java core** — see D-LGJ-C's disable-verification and D-LGJ-E's `FusionParityTest`/`LifetimeTest`. Re-opens for F/G once the Lab lands | I |
| D-LGJ-I | Docs: `architecture.md`, `panama.md`, `valhalla-lab.md`, `execution-boundary.md` | **Queued** — gated on F/G landing (the docs synthesize Lab results, not just the core) | — |
| D-LGJ-AUDIT | Mechanical post-fan-out audit: `grep` for `ndarray::hpc` imports, any `.h`/`cbindgen`/`jextract` artifact, any FFM type leaking into public Java API | **DONE 2026-08-17** — 1 real violation found (`kernels.rs::simd_popcount` used the internal `ndarray::hpc::bitwise` path), fixed in place; everything else confirmed to be the one sanctioned exception or explanatory prose | closed D-LGJ-C/D/E for the core |
Expand Down
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ Cargo.lock.bak
/bench/out/
/bench/lib/*.jar
/valhalla-lab/out/
# run.sh's own compiled output (results/*-api, results/*-lab class trees) — the
# .txt/.diff/.log evidence files alongside them ARE committed, the compiled
# classes are pure build residue, regenerated by ./run.sh on demand.
/valhalla-lab/results/*/

# Downloaded JDKs and artifacts (never committed — see docs/abi.md and
# .claude/knowledge/jdk-toolchain-facts.md for how to obtain them)
Expand Down
100 changes: 100 additions & 0 deletions valhalla-lab/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# The Valhalla laboratory

The three-truths method (`.claude/knowledge/valhalla-three-truths-method.md`) applied to this
project's small semantic value vocabulary — `LaneId`, `Ordinal`, `MaskId`, `RowRange`, `Row` — and
to the mission's mandatory headline experiment: does Valhalla rescue per-entity materialization at
65,536-row scale, or only the tiny descriptor vocabulary around it?

**Same experiment source, compiled twice** — once against a stable JDK where the vocabulary types
are plain `record`s, once against the JEP 401 early-access JDK where they are `value record`s — so
the comparison is genuinely apples-to-apples, not two different programs.

## Layout

```
src/shared/ experiment logic, byte-identical on both compiles
src/stable/ Vocab.java (record), Containers.java, Platform.java — the stable-JDK half of the A/B
src/valhalla/ Vocab.java (value record), Containers.java, Platform.java — the Valhalla half
```

`Platform` is the one seam between them: same signatures on both sides, so `src/shared/` never
branches on which platform it's running on except by asking `Platform` — never by calling a
Valhalla-only API (like `Class::isValue` or `jdk.internal.value.ValueClass`) directly. That is a
real rule, not a style preference: `Class::isValue` does not exist at all on a stable JDK, so a
direct call would fail to *compile* the stable half, not just report the wrong answer.

`NativeAccess` (in `src/shared/`, package `com.adaworldapi.lancegraph`) is a read-only, split-package
escape hatch into the shipped library's package-private handle — documented in the file itself. It
exists because the lab has to build the very thing the thesis says you should not build (65,536 Java
objects) from the *same bytes* the native kernel reads, or the comparison proves nothing. Nothing
under `java/` changes to support this.

## Build and run

### Stable half (JDK 26 GA, plain `record`s)

```sh
javac -d out-stable $(find ../java/src/main/java src/shared src/stable -name '*.java')

java --enable-native-access=ALL-UNNAMED \
-Dlgj.library=../target/release/liblgj_abi.so \
-cp out-stable com.adaworldapi.lancegraph.lab.RunAll
```

### Valhalla half (the JEP 401 EA build, `value record`s)

`--release` cannot be combined with `--add-exports` (a real javac restriction — `--release` uses a
stricter cross-compilation module model). Use `-source` instead when compiling *for* the JDK you are
also running on, which is the case here.

```sh
javac --enable-preview -source 27 \
--add-exports java.base/jdk.internal.value=ALL-UNNAMED \
--add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \
-d out-valhalla $(find ../java/src/main/java src/shared src/valhalla -name '*.java')

java --enable-preview --enable-native-access=ALL-UNNAMED \
--add-exports java.base/jdk.internal.value=ALL-UNNAMED \
--add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \
-Dlgj.library=../target/release/liblgj_abi.so \
-cp out-valhalla com.adaworldapi.lancegraph.lab.RunAll
```

Both need the JDK paths from `.claude/knowledge/jdk-toolchain-facts.md` — do not use `/usr/bin/java`
(JDK 21, no value classes at all) for either.

## What each experiment measures

| Class | Question |
|---|---|
| `IdentityExperiment` | Truth (a), semantic: is identity actually unobservable? `Class::isValue`, reference equality, array flatness, `synchronized` legality — measured on both platforms, asked to agree everywhere except reference equality (which no caller in the production API uses). |
| `FootprintExperiment` | Truth (b)/(c), representation: per-object bytes, array layout, field flattening, call-argument passing — via `jol-core`'s real VM instrumentation where available, allocation-delta measurement elsewhere. |
| `FfmAddressingExperiment` | Is the wrapper free where it actually touches native memory — a `RowRange`/`Ordinal` around an FFM offset vs a bare `long`? |
| `ThesisExperiment` | The mandatory headline: 65,536 rows as (1) one native lane + one packed mask + one crossing, vs (2)/(3) hydrated Java objects, on the SAME question and the SAME answer. Heap cost and wall time, both platforms. |

## Measured headline (2026-08-17, this environment)

Real numbers from a real run — reproduce with the commands above before citing a different number.

| | native, one crossing | hydrate 65,536 `Row`, then scan |
|---|---:|---:|
| stable JDK 26 | 19.5 µs, 289 KiB Java-side | 746 µs, 2.00 MiB |
| Valhalla (JDK 27 EA) | 15.7 µs, 289.5 KiB Java-side | 900 µs, 2.50 MiB |

The native path wins by roughly **38–57×** on time and **7–9×** on Java heap, on **both** platforms —
Valhalla does not close this gap, because `Row` (multiple fields) measured `NOT-FLAT` even under
Valhalla, while the single-field `LaneId` measured `FLAT` (2.90 B/element vs 16.00 B on stable, ~5.5×
smaller). This is the mission thesis's prediction, confirmed rather than assumed: **Valhalla helps
the tiny descriptor vocabulary; it does not rescue per-entity materialization at this scale.** See
`IdentityExperiment`'s and `FootprintExperiment`'s full output for the field-by-field evidence.

## A defect found and fixed while wiring this up

The first version of `IdentityExperiment`/stable `Platform` called `Class::isValue()` directly for
four of the five vocabulary types (`Ordinal`/`MaskId`/`RowRange`/`Row`), with a comment incorrectly
claiming it was "final API on JDK 26." It is not — `javac` on JDK 26 GA does not have that method at
all, confirmed by a real compile failure, not by reading documentation. Fixed by routing every
identity query through `Platform.isValueClass(Class<?>)`, which the stable half answers `false` (a
JDK with no value-class concept can never produce one, so the answer is exact, not a guess) and the
Valhalla half answers with the real `type.isValue()`. See `EPIPHANIES.md`
`E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1` for the audit discipline this caught it under.
4 changes: 4 additions & 0 deletions valhalla-lab/reproducers/R1-observed.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Picked up JAVA_TOOL_OPTIONS:
identity container FAILED: java.lang.VerifyError
All strict final fields must be initialized before super(): 1 field(s), lane:LR1_NullRestrictedFieldInIdentityClass$LaneId; in R1_NullRestrictedFieldInIdentityClass$Descriptor
value container: LaneId[index=1] (works)
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
// Reproducer R1 — @NullRestricted on a field of an ORDINARY (identity) class fails at class load.
// javac emits the fields' initialisers AFTER the super() call; the VM demands strict fields be
// assigned BEFORE it. There is no @Strict in this build for javac to key on and no source form
// that expresses the required order, so the combination is unreachable from Java source.
//
// javac --enable-preview -source 27 -target 27 \
// --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED -d out R1_*.java
// java --enable-preview \
// --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED -cp out R1_NullRestrictedFieldInIdentityClass
import jdk.internal.vm.annotation.NullRestricted;

public class R1_NullRestrictedFieldInIdentityClass {
static value record LaneId(int index) {}

/** An ordinary class that wants a flat LaneId field. Compiles. Does not load. */
static final class Descriptor {
@NullRestricted final LaneId lane;
Descriptor(int i) { this.lane = new LaneId(i); }
}

/** The workaround: make the CONTAINER a value class too. Its fields are then strict already. */
static value class ValueDescriptor {
@NullRestricted final LaneId lane;
ValueDescriptor(int i) { this.lane = new LaneId(i); }
}

public static void main(String[] a) {
try {
System.out.println("identity container: " + new Descriptor(1).lane);
} catch (Throwable t) {
System.out.println("identity container FAILED: " + t.getClass().getName());
System.out.println(" " + String.valueOf(t.getMessage()).lines().findFirst().orElse(""));
}
System.out.println("value container: " + new ValueDescriptor(1).lane + " (works)");
}
}
8 changes: 8 additions & 0 deletions valhalla-lab/reproducers/R2-observed.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Picked up JAVA_TOOL_OPTIONS:
type payload NR-nonAtomic NR-atomic nullable-atomic
P4 4 B true true true
P8i 8 B true true false
P8l 8 B true true false
P12 12 B false false false
P16 16 B false false false
P16l 16 B false false false
40 changes: 40 additions & 0 deletions valhalla-lab/reproducers/R2_FlatteningCliff.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
// Reproducer R2 — array flattening stops at an 8-byte payload in this build.
// Sweeps payload shapes and asks the VM directly via ValueClass.isFlatArray for all three
// array flavours. Every shape wider than 8 bytes is NOT flattened, in any flavour.
//
// javac --enable-preview -source 27 -target 27 \
// --add-exports java.base/jdk.internal.value=ALL-UNNAMED -d out R2_FlatteningCliff.java
// java --enable-preview --add-exports java.base/jdk.internal.value=ALL-UNNAMED \
// -cp out R2_FlatteningCliff
// Add -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlatArrayLayout to see the VM's own layout log.
import jdk.internal.value.ValueClass;

public class R2_FlatteningCliff {
static value record P4(int a) {} // 4 B
static value record P8i(int a, int b) {} // 8 B
static value record P8l(long a) {} // 8 B
static value record P12(long a, int b) {} // 12 B
static value record P16(long a, int b, int c) {} // 16 B <- the shape of a real entity
static value record P16l(long a, long b) {} // 16 B

record Case(String name, int payload, Class<?> type, Object init) {}

public static void main(String[] x) {
Case[] cases = {
new Case("P4", 4, P4.class, new P4(0)),
new Case("P8i", 8, P8i.class, new P8i(0, 0)),
new Case("P8l", 8, P8l.class, new P8l(0)),
new Case("P12", 12, P12.class, new P12(0, 0)),
new Case("P16", 16, P16.class, new P16(0, 0, 0)),
new Case("P16l",16, P16l.class, new P16l(0, 0)),
};
System.out.printf("%-6s %-8s %-16s %-16s %s%n",
"type", "payload", "NR-nonAtomic", "NR-atomic", "nullable-atomic");
for (Case c : cases) {
System.out.printf("%-6s %5d B %-16s %-16s %s%n", c.name(), c.payload(),
ValueClass.isFlatArray(ValueClass.newNullRestrictedNonAtomicArray(c.type(), 16, c.init())),
ValueClass.isFlatArray(ValueClass.newNullRestrictedAtomicArray(c.type(), 16, c.init())),
ValueClass.isFlatArray(ValueClass.newNullableAtomicArray(c.type(), 16)));
}
}
}
6 changes: 6 additions & 0 deletions valhalla-lab/reproducers/R3-bang-syntax-observed.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes=
/tmp/Bang.java:1: error: not a statement
public class Bang { static value record L(int i){} public static void main(String[] a){ L![] x = new L![2]; System.out.println(x.length);} }
^
/tmp/Bang.java:1: error: ';' expected
public class Bang { static value record L(int i){} public static void main(String[] a){ L![] x = new L![2]; System.out.println(x.length);} }
Loading