diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index 05eeae2..d3cb4fb 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -4,6 +4,57 @@ > `**Status:**`/`**Confidence:**` line. A correction gets its own new, > dated entry that references the one it corrects — the storno rule. +## 2026-08-17 — E-LGJ-VALHALLA-MEASURED-NOT-ASSUMED-1 + +**Status:** FINDING. **Confidence:** High (real numbers, both JDKs actually +run, reproducible via `valhalla-lab/README.md`). + +The mandatory N-objects-vs-N-values-vs-1-lane experiment +(`.claude/knowledge/valhalla-three-truths-method.md`'s "one experiment that +must never be skipped") ran on both real JDKs. Headline, on 65,536 rows, +identical question, identical answer on every path: + +| | native, one crossing | hydrate 65,536 `Row`, then scan | +|---|---:|---:| +| stable JDK 26 | 19.5 µs, 289 KiB | 746 µs, 2.00 MiB | +| Valhalla JDK 27 EA | 15.7 µs, 289.5 KiB | 900 µs, 2.50 MiB | + +**The thesis's prediction held, and the reason why is itself a measured +finding, not an assumption:** `LaneId` (one field) measured `FLAT` under +Valhalla via the real VM query `ValueClass.isFlatArray` (2.90 B/element vs +16.00 B on stable — ~5.5× smaller), but `Row` (multiple fields) measured +**`NOT-FLAT`** even under Valhalla, and its per-row heap cost (40.01 B) was +*larger* than the stable JDK's own record-array cost (32.01 B). Valhalla +genuinely helps a single-field descriptor; it did not flatten the +multi-field materialization the thesis explicitly said to check rather +than assume away. + +**One real defect found and fixed before this landed** — a bug of the +falsifiability-discipline-caught-it, not the happy-path-hid-it kind. The +first version of `IdentityExperiment` and the stable-JDK `Platform` called +`Class::isValue()` directly on four vocabulary types, with a comment +incorrectly asserting *"Class::isValue is final API on JDK 26."* It does +not exist there at all — confirmed by a real `javac` compile failure, not +by re-reading documentation. Fixed by routing every identity query through +`Platform.isValueClass(Class)`: the stable half answers `false` +honestly (a JDK with no value-class concept can never produce one — the +answer is exact, not a guess, unlike the genuinely-unknowable +`arrayFlatness` case the same file already handles correctly), the +Valhalla half answers with the real `type.isValue()`. The correction +mirrors `E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1`'s finding about +`kernels.rs`: an agent's own doc comment stated the WRONG fact confidently +one line above the code that relied on it, and only compiling both +variants for real (not trusting the report that they "should" compile) +caught it. + +**Two javac usage facts worth keeping** (real dead ends this session hit +and resolved, recorded so a future session doesn't re-hit them): +`--release N` cannot be combined with `--add-exports` for a system module +(a hard javac restriction, not a bug) — use `-source N` instead when +compiling for the same JDK you'll run on; and `--enable-preview` requires +an explicit `-source`/`--release` to be present at all, it is not +self-sufficient. + ## 2026-08-17 — E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1 **Status:** FINDING. **Confidence:** High (measured, not asserted — every diff --git a/.claude/board/STATUS_BOARD.md b/.claude/board/STATUS_BOARD.md index 112599f..f4be4bc 100644 --- a/.claude/board/STATUS_BOARD.md +++ b/.claude/board/STATUS_BOARD.md @@ -13,8 +13,8 @@ list. | D-LGJ-C | `native/lgj-abi` — manifest, generation-checked registry, generic SoA fixture, kernels, `extern "C"` surface | **DONE 2026-08-17** — `cargo test` **72/72**, `clippy -D warnings` clean, `fmt --check` clean, release build → 14/14 symbols verified via `nm -D`. **Disable-verified**: the registry's generation check was short-circuited and exactly the 2 tests that should catch it went red, 70 stayed green; restored, re-verified 72/72 | D, H | | D-LGJ-D | Java FFM membrane `internal/ffm` | **DONE 2026-08-17** — compiles clean with `-Xlint:all`; 7 `[restricted]` warnings, all in `internal/ffm/*` or a test deliberately exercising it; `AbiContractTest` 7/7 incl. proving the manifest cross-check genuinely rejects a wrong `.so` (`libz.so.1` loads but is refused for exporting no `lgj_abi_manifest`) | E | | D-LGJ-E | Java public facade (`NativePattern`/`View`/`Predicate`/`Pattern`/`Mask`) | **DONE 2026-08-17** — `AllTests` **132/132**: `ApiSurfaceTest` (reflection-enforced zero-FFM-leakage), `SmokeTest` 14/14, `FixtureParityTest` 30/30 (Java independently recomputes expected counts from the transcribed generator), `FusionParityTest` 31/31 (fused/unfused/scalar bit-identical across 6 row-count shapes), `LazinessTest` 8/8 (empirically: 0 crossings to build a 16-condition chain, exactly 1 to evaluate it, independent of rows up to 1,000,000 — the thesis's central claim, measured), `NarrowingTest` 16/16, `LifetimeTest` 23/23 | F, G | -| D-LGJ-F | Valhalla lab — three-truths method on the small semantic value vocabulary | **In flight** — sequenced after E, now reading the real Java types; deferred to a follow-up PR, not blocking PR #1 | I | -| D-LGJ-G | Java Vector API comparative bench vs Panama→`ndarray::simd` | **In flight** — real JMH jars fetched (`jmh-core`/`jmh-generator-annprocess`/`jopt-simple`/`commons-math3`) to `bench/lib/` (gitignored); no bench source written yet; deferred to the same follow-up PR as F | I | +| D-LGJ-F | Valhalla lab — three-truths method on the small semantic value vocabulary | **DONE 2026-08-17** — `valhalla-lab/`: 4 experiments + a self-verifying `run.sh` (mechanically diffs the two `Vocab.java`s modulo the `value` keyword before trusting the A/B) + 3 causal-isolation runs (escape-analysis off; `UseArrayFlattening`/`UseFieldFlattening` toggled independently). 3 real Valhalla limitations reproduced and filed under `reproducers/` (R1: `@NullRestricted` field on an identity class is a `VerifyError`, javac's fault — no source form expresses required strict-field order; **R2: array flattening has a hard 8-byte payload cliff, VM-confirmed via `-XX:+PrintFlatArrayLayout`** — `LaneId`/`Ordinal`/`MaskId` (≤8B) flatten, `RowRange`/`Row` (16B) do not, so "Valhalla helps descriptors not entities" is a measured VM cutoff, not a hand-wave, and `RowRange` landing on the wrong side is flagged as the one place the expectation was too optimistic; R3: the densest null-restricted array form is `jdk.internal`-only and generics erase flattening entirely — `Foo!` type syntax confirmed NOT to parse, matching the archaeology finding). 1 real defect found + fixed before landing (see `EPIPHANIES.md`). None of the three limitations changed the production API — the migration path stays exactly `record` → `value record` | I | +| D-LGJ-G | Java Vector API comparative bench vs Panama→`ndarray::simd` | **In flight** — real JMH + JOL jars fetched (`jmh-core`/`jmh-generator-annprocess`/`jopt-simple`/`commons-math3`/`jol-core`) to `bench/lib/` (gitignored); no bench source written yet; the ONLY remaining open row | I | | D-LGJ-H | Falsification: handle lifecycle (adversarial), SIMD/scalar parity, Java/native parity | **DONE 2026-08-17 for the Rust+Java core** — see D-LGJ-C's disable-verification and D-LGJ-E's `FusionParityTest`/`LifetimeTest`. Re-opens for F/G once the Lab lands | I | | D-LGJ-I | Docs: `architecture.md`, `panama.md`, `valhalla-lab.md`, `execution-boundary.md` | **Queued** — gated on F/G landing (the docs synthesize Lab results, not just the core) | — | | D-LGJ-AUDIT | Mechanical post-fan-out audit: `grep` for `ndarray::hpc` imports, any `.h`/`cbindgen`/`jextract` artifact, any FFM type leaking into public Java API | **DONE 2026-08-17** — 1 real violation found (`kernels.rs::simd_popcount` used the internal `ndarray::hpc::bitwise` path), fixed in place; everything else confirmed to be the one sanctioned exception or explanatory prose | closed D-LGJ-C/D/E for the core | diff --git a/.gitignore b/.gitignore index 0d6f235..8c7e5b4 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,10 @@ Cargo.lock.bak /bench/out/ /bench/lib/*.jar /valhalla-lab/out/ +# run.sh's own compiled output (results/*-api, results/*-lab class trees) — the +# .txt/.diff/.log evidence files alongside them ARE committed, the compiled +# classes are pure build residue, regenerated by ./run.sh on demand. +/valhalla-lab/results/*/ # Downloaded JDKs and artifacts (never committed — see docs/abi.md and # .claude/knowledge/jdk-toolchain-facts.md for how to obtain them) diff --git a/valhalla-lab/README.md b/valhalla-lab/README.md new file mode 100644 index 0000000..8255c41 --- /dev/null +++ b/valhalla-lab/README.md @@ -0,0 +1,100 @@ +# The Valhalla laboratory + +The three-truths method (`.claude/knowledge/valhalla-three-truths-method.md`) applied to this +project's small semantic value vocabulary — `LaneId`, `Ordinal`, `MaskId`, `RowRange`, `Row` — and +to the mission's mandatory headline experiment: does Valhalla rescue per-entity materialization at +65,536-row scale, or only the tiny descriptor vocabulary around it? + +**Same experiment source, compiled twice** — once against a stable JDK where the vocabulary types +are plain `record`s, once against the JEP 401 early-access JDK where they are `value record`s — so +the comparison is genuinely apples-to-apples, not two different programs. + +## Layout + +``` +src/shared/ experiment logic, byte-identical on both compiles +src/stable/ Vocab.java (record), Containers.java, Platform.java — the stable-JDK half of the A/B +src/valhalla/ Vocab.java (value record), Containers.java, Platform.java — the Valhalla half +``` + +`Platform` is the one seam between them: same signatures on both sides, so `src/shared/` never +branches on which platform it's running on except by asking `Platform` — never by calling a +Valhalla-only API (like `Class::isValue` or `jdk.internal.value.ValueClass`) directly. That is a +real rule, not a style preference: `Class::isValue` does not exist at all on a stable JDK, so a +direct call would fail to *compile* the stable half, not just report the wrong answer. + +`NativeAccess` (in `src/shared/`, package `com.adaworldapi.lancegraph`) is a read-only, split-package +escape hatch into the shipped library's package-private handle — documented in the file itself. It +exists because the lab has to build the very thing the thesis says you should not build (65,536 Java +objects) from the *same bytes* the native kernel reads, or the comparison proves nothing. Nothing +under `java/` changes to support this. + +## Build and run + +### Stable half (JDK 26 GA, plain `record`s) + +```sh +javac -d out-stable $(find ../java/src/main/java src/shared src/stable -name '*.java') + +java --enable-native-access=ALL-UNNAMED \ + -Dlgj.library=../target/release/liblgj_abi.so \ + -cp out-stable com.adaworldapi.lancegraph.lab.RunAll +``` + +### Valhalla half (the JEP 401 EA build, `value record`s) + +`--release` cannot be combined with `--add-exports` (a real javac restriction — `--release` uses a +stricter cross-compilation module model). Use `-source` instead when compiling *for* the JDK you are +also running on, which is the case here. + +```sh +javac --enable-preview -source 27 \ + --add-exports java.base/jdk.internal.value=ALL-UNNAMED \ + --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \ + -d out-valhalla $(find ../java/src/main/java src/shared src/valhalla -name '*.java') + +java --enable-preview --enable-native-access=ALL-UNNAMED \ + --add-exports java.base/jdk.internal.value=ALL-UNNAMED \ + --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED \ + -Dlgj.library=../target/release/liblgj_abi.so \ + -cp out-valhalla com.adaworldapi.lancegraph.lab.RunAll +``` + +Both need the JDK paths from `.claude/knowledge/jdk-toolchain-facts.md` — do not use `/usr/bin/java` +(JDK 21, no value classes at all) for either. + +## What each experiment measures + +| Class | Question | +|---|---| +| `IdentityExperiment` | Truth (a), semantic: is identity actually unobservable? `Class::isValue`, reference equality, array flatness, `synchronized` legality — measured on both platforms, asked to agree everywhere except reference equality (which no caller in the production API uses). | +| `FootprintExperiment` | Truth (b)/(c), representation: per-object bytes, array layout, field flattening, call-argument passing — via `jol-core`'s real VM instrumentation where available, allocation-delta measurement elsewhere. | +| `FfmAddressingExperiment` | Is the wrapper free where it actually touches native memory — a `RowRange`/`Ordinal` around an FFM offset vs a bare `long`? | +| `ThesisExperiment` | The mandatory headline: 65,536 rows as (1) one native lane + one packed mask + one crossing, vs (2)/(3) hydrated Java objects, on the SAME question and the SAME answer. Heap cost and wall time, both platforms. | + +## Measured headline (2026-08-17, this environment) + +Real numbers from a real run — reproduce with the commands above before citing a different number. + +| | native, one crossing | hydrate 65,536 `Row`, then scan | +|---|---:|---:| +| stable JDK 26 | 19.5 µs, 289 KiB Java-side | 746 µs, 2.00 MiB | +| Valhalla (JDK 27 EA) | 15.7 µs, 289.5 KiB Java-side | 900 µs, 2.50 MiB | + +The native path wins by roughly **38–57×** on time and **7–9×** on Java heap, on **both** platforms — +Valhalla does not close this gap, because `Row` (multiple fields) measured `NOT-FLAT` even under +Valhalla, while the single-field `LaneId` measured `FLAT` (2.90 B/element vs 16.00 B on stable, ~5.5× +smaller). This is the mission thesis's prediction, confirmed rather than assumed: **Valhalla helps +the tiny descriptor vocabulary; it does not rescue per-entity materialization at this scale.** See +`IdentityExperiment`'s and `FootprintExperiment`'s full output for the field-by-field evidence. + +## A defect found and fixed while wiring this up + +The first version of `IdentityExperiment`/stable `Platform` called `Class::isValue()` directly for +four of the five vocabulary types (`Ordinal`/`MaskId`/`RowRange`/`Row`), with a comment incorrectly +claiming it was "final API on JDK 26." It is not — `javac` on JDK 26 GA does not have that method at +all, confirmed by a real compile failure, not by reading documentation. Fixed by routing every +identity query through `Platform.isValueClass(Class)`, which the stable half answers `false` (a +JDK with no value-class concept can never produce one, so the answer is exact, not a guess) and the +Valhalla half answers with the real `type.isValue()`. See `EPIPHANIES.md` +`E-LGJ-CORE-SLICE-GREEN-DISABLE-VERIFIED-1` for the audit discipline this caught it under. diff --git a/valhalla-lab/reproducers/R1-observed.txt b/valhalla-lab/reproducers/R1-observed.txt new file mode 100644 index 0000000..f03f8ba --- /dev/null +++ b/valhalla-lab/reproducers/R1-observed.txt @@ -0,0 +1,4 @@ +Picked up JAVA_TOOL_OPTIONS: +identity container FAILED: java.lang.VerifyError + All strict final fields must be initialized before super(): 1 field(s), lane:LR1_NullRestrictedFieldInIdentityClass$LaneId; in R1_NullRestrictedFieldInIdentityClass$Descriptor +value container: LaneId[index=1] (works) diff --git a/valhalla-lab/reproducers/R1_NullRestrictedFieldInIdentityClass.java b/valhalla-lab/reproducers/R1_NullRestrictedFieldInIdentityClass.java new file mode 100644 index 0000000..eaee0da --- /dev/null +++ b/valhalla-lab/reproducers/R1_NullRestrictedFieldInIdentityClass.java @@ -0,0 +1,36 @@ +// Reproducer R1 — @NullRestricted on a field of an ORDINARY (identity) class fails at class load. +// javac emits the fields' initialisers AFTER the super() call; the VM demands strict fields be +// assigned BEFORE it. There is no @Strict in this build for javac to key on and no source form +// that expresses the required order, so the combination is unreachable from Java source. +// +// javac --enable-preview -source 27 -target 27 \ +// --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED -d out R1_*.java +// java --enable-preview \ +// --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED -cp out R1_NullRestrictedFieldInIdentityClass +import jdk.internal.vm.annotation.NullRestricted; + +public class R1_NullRestrictedFieldInIdentityClass { + static value record LaneId(int index) {} + + /** An ordinary class that wants a flat LaneId field. Compiles. Does not load. */ + static final class Descriptor { + @NullRestricted final LaneId lane; + Descriptor(int i) { this.lane = new LaneId(i); } + } + + /** The workaround: make the CONTAINER a value class too. Its fields are then strict already. */ + static value class ValueDescriptor { + @NullRestricted final LaneId lane; + ValueDescriptor(int i) { this.lane = new LaneId(i); } + } + + public static void main(String[] a) { + try { + System.out.println("identity container: " + new Descriptor(1).lane); + } catch (Throwable t) { + System.out.println("identity container FAILED: " + t.getClass().getName()); + System.out.println(" " + String.valueOf(t.getMessage()).lines().findFirst().orElse("")); + } + System.out.println("value container: " + new ValueDescriptor(1).lane + " (works)"); + } +} diff --git a/valhalla-lab/reproducers/R2-observed.txt b/valhalla-lab/reproducers/R2-observed.txt new file mode 100644 index 0000000..ccb773d --- /dev/null +++ b/valhalla-lab/reproducers/R2-observed.txt @@ -0,0 +1,8 @@ +Picked up JAVA_TOOL_OPTIONS: +type payload NR-nonAtomic NR-atomic nullable-atomic +P4 4 B true true true +P8i 8 B true true false +P8l 8 B true true false +P12 12 B false false false +P16 16 B false false false +P16l 16 B false false false diff --git a/valhalla-lab/reproducers/R2_FlatteningCliff.java b/valhalla-lab/reproducers/R2_FlatteningCliff.java new file mode 100644 index 0000000..3ecaec4 --- /dev/null +++ b/valhalla-lab/reproducers/R2_FlatteningCliff.java @@ -0,0 +1,40 @@ +// Reproducer R2 — array flattening stops at an 8-byte payload in this build. +// Sweeps payload shapes and asks the VM directly via ValueClass.isFlatArray for all three +// array flavours. Every shape wider than 8 bytes is NOT flattened, in any flavour. +// +// javac --enable-preview -source 27 -target 27 \ +// --add-exports java.base/jdk.internal.value=ALL-UNNAMED -d out R2_FlatteningCliff.java +// java --enable-preview --add-exports java.base/jdk.internal.value=ALL-UNNAMED \ +// -cp out R2_FlatteningCliff +// Add -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlatArrayLayout to see the VM's own layout log. +import jdk.internal.value.ValueClass; + +public class R2_FlatteningCliff { + static value record P4(int a) {} // 4 B + static value record P8i(int a, int b) {} // 8 B + static value record P8l(long a) {} // 8 B + static value record P12(long a, int b) {} // 12 B + static value record P16(long a, int b, int c) {} // 16 B <- the shape of a real entity + static value record P16l(long a, long b) {} // 16 B + + record Case(String name, int payload, Class type, Object init) {} + + public static void main(String[] x) { + Case[] cases = { + new Case("P4", 4, P4.class, new P4(0)), + new Case("P8i", 8, P8i.class, new P8i(0, 0)), + new Case("P8l", 8, P8l.class, new P8l(0)), + new Case("P12", 12, P12.class, new P12(0, 0)), + new Case("P16", 16, P16.class, new P16(0, 0, 0)), + new Case("P16l",16, P16l.class, new P16l(0, 0)), + }; + System.out.printf("%-6s %-8s %-16s %-16s %s%n", + "type", "payload", "NR-nonAtomic", "NR-atomic", "nullable-atomic"); + for (Case c : cases) { + System.out.printf("%-6s %5d B %-16s %-16s %s%n", c.name(), c.payload(), + ValueClass.isFlatArray(ValueClass.newNullRestrictedNonAtomicArray(c.type(), 16, c.init())), + ValueClass.isFlatArray(ValueClass.newNullRestrictedAtomicArray(c.type(), 16, c.init())), + ValueClass.isFlatArray(ValueClass.newNullableAtomicArray(c.type(), 16))); + } + } +} diff --git a/valhalla-lab/reproducers/R3-bang-syntax-observed.txt b/valhalla-lab/reproducers/R3-bang-syntax-observed.txt new file mode 100644 index 0000000..d4ac6d5 --- /dev/null +++ b/valhalla-lab/reproducers/R3-bang-syntax-observed.txt @@ -0,0 +1,6 @@ +Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= +/tmp/Bang.java:1: error: not a statement +public class Bang { static value record L(int i){} public static void main(String[] a){ L![] x = new L![2]; System.out.println(x.length);} } + ^ +/tmp/Bang.java:1: error: ';' expected +public class Bang { static value record L(int i){} public static void main(String[] a){ L![] x = new L![2]; System.out.println(x.length);} } diff --git a/valhalla-lab/reproducers/R3-observed.txt b/valhalla-lab/reproducers/R3-observed.txt new file mode 100644 index 0000000..847ea2d --- /dev/null +++ b/valhalla-lab/reproducers/R3-observed.txt @@ -0,0 +1,4 @@ +(1) new LaneId[8] flat=true accepts null=true (nullable-flat: pays for a null marker) +(2) LaneId![] DOES NOT PARSE ? no null-restricted type syntax +(3) ValueClass.newNullRestricted... flat=true accepts null=false (jdk.internal, needs --add-exports) +(4) List.toArray() flat=false ? generics erase to Object[]; the flattening is undone at the collection boundary diff --git a/valhalla-lab/reproducers/R3_NoSupportedFlatSurface.java b/valhalla-lab/reproducers/R3_NoSupportedFlatSurface.java new file mode 100644 index 0000000..16f4ab1 --- /dev/null +++ b/valhalla-lab/reproducers/R3_NoSupportedFlatSurface.java @@ -0,0 +1,54 @@ +// Reproducer R3 — the ideal API cannot be expressed in supported Java. +// +// The API wants to say "an array of LaneId, densely packed, no nulls". Three ways to ask, and +// what each actually yields on this build: +// (1) `new LaneId[n]` -> flat, but NULLABLE-flat: it still accepts null and +// pays for a null marker, so it is not the densest +// encoding — and for a payload > 8 B it is not flat +// at all (see R2). +// (2) `LaneId![] a = new LaneId![n]` -> DOES NOT PARSE (no null-restricted type syntax), so +// the density in (3) has no supported spelling. +// (3) ValueClass.newNullRestrictedNonAtomicArray -> densest, but jdk.internal + --add-exports. +// (4) `List` -> generics erase; the flattening is undone at the +// collection boundary regardless of (1)-(3). +// +// So a supported API can get *some* flattening for small payloads and can never get the densest +// form, and any generic container discards it. That is the deficiency: not that flattening is +// missing, but that the API cannot ASK for it. +// +// javac --enable-preview -source 27 -target 27 \ +// --add-exports java.base/jdk.internal.value=ALL-UNNAMED -d out R3_NoSupportedFlatSurface.java +// java --enable-preview --add-exports java.base/jdk.internal.value=ALL-UNNAMED \ +// -cp out R3_NoSupportedFlatSurface +import jdk.internal.value.ValueClass; +import java.util.ArrayList; +import java.util.List; + +public class R3_NoSupportedFlatSurface { + static value record LaneId(int index) {} + + public static void main(String[] a) { + LaneId[] supported = new LaneId[8]; + System.out.println("(1) new LaneId[8] flat=" + ValueClass.isFlatArray(supported) + + " accepts null=" + tryNull(supported) + " (nullable-flat: pays for a null marker)"); + + // (2) `LaneId![] x = new LaneId![8];` <-- uncomment to see: this syntax does not exist. + System.out.println("(2) LaneId![] DOES NOT PARSE — no null-restricted type syntax"); + + Object[] internal = ValueClass.newNullRestrictedNonAtomicArray(LaneId.class, 8, new LaneId(0)); + System.out.println("(3) ValueClass.newNullRestricted... flat=" + ValueClass.isFlatArray(internal) + + " accepts null=" + tryNull(internal) + + " (jdk.internal, needs --add-exports)"); + + List generic = new ArrayList<>(); + for (int i = 0; i < 8; i++) generic.add(new LaneId(i)); + Object[] backing = generic.toArray(); + System.out.println("(4) List.toArray() flat=" + ValueClass.isFlatArray(backing) + + " — generics erase to Object[]; the flattening is undone at the collection boundary"); + } + + private static boolean tryNull(Object[] arr) { + try { Object keep = arr[0]; arr[0] = null; arr[0] = keep; return true; } + catch (Throwable t) { return false; } + } +} diff --git a/valhalla-lab/reproducers/README.md b/valhalla-lab/reproducers/README.md new file mode 100644 index 0000000..2543170 --- /dev/null +++ b/valhalla-lab/reproducers/README.md @@ -0,0 +1,159 @@ +# Reproducers — Valhalla limitations hit while expressing the ideal API + +Three limitations were hit. **None of them changed the API.** Where the ideal shape could not be +expressed, that fact is recorded here and the production types stayed as they are — distorting a +public API to fit a preview VM's current budget would bake a temporary constraint into a permanent +surface. + +Each reproducer is a single self-contained file with its command line in the header comment, and a +`*-observed.txt` holding the exact output that file produced on this box. + +| # | Limitation | Belongs to | +|---|---|---| +| [R1](#r1) | `@NullRestricted` field in an ordinary class fails at class load | **javac** | +| [R2](#r2) | Array flattening stops at an 8-byte payload | **HotSpot / Valhalla** | +| [R3](#r3) | The densest layout has no supported spelling, and generics discard it | **Valhalla (language + libraries)** | + +Environment for every observation below: `openjdk 27-jep401ea3+1-1`, Linux x86-64, +Intel Xeon @ 2.10 GHz (4 vCPU, AVX-512). + +--- + +## R1 — `@NullRestricted` on a field of an identity class {#r1} + +**File:** `R1_NullRestrictedFieldInIdentityClass.java` · **Observed:** `R1-observed.txt` + +**Desired semantics.** An ordinary class holds a value-typed field flat — the field is never +null, so no reference and no header should be needed: + +```java +final class Descriptor { // ordinary identity class + @NullRestricted final LaneId lane; // want: 4 bytes inline + Descriptor(int i) { this.lane = new LaneId(i); } +} +``` + +**Ordinary Java (JDK 26).** Compiles and runs; the field is a reference. The annotation does not +exist, so the question cannot even be asked. + +**Valhalla (JDK 27 EA).** Compiles, then fails at class load: + +``` +java.lang.VerifyError: All strict final fields must be initialized before super(): + 1 field(s), lane:LR1_...$LaneId; in R1_...$Descriptor +``` + +**Why.** A null-restricted field is a *strict* field: the VM requires it to be assigned before the +`super()` call. javac emits field initialisers *after* `super()`, and this build has no `@Strict` +annotation for javac to key on — `jdk.internal.vm.annotation` here contains `NullRestricted` and +`LooselyConsistentValue` but no `Strict`. There is no Java source form that expresses the required +order, so the combination is unreachable from source. + +**Workaround, and its cost.** Make the container itself a `value class`; its fields are then +implicitly strict and it works. That is what `src/valhalla/.../Containers.java` does. The cost is +that the workaround is not always available: a container that legitimately has identity — anything +mutable, anything used as a lock, anything with lifecycle — cannot become a value class, and +therefore cannot hold a flat field at all on this build. + +**Consequence for this project.** None yet, and that is luck rather than design: the descriptor +types (`Field` and friends) happen to be immutable. Had `NativePattern` — which is genuinely an +identity object, it owns a native resource and closes it — wanted a flat `LaneId` field, there +would be no way to write it. + +--- + +## R2 — array flattening stops at an 8-byte payload {#r2} + +**File:** `R2_FlatteningCliff.java` · **Observed:** `R2-observed.txt` + +**Desired semantics.** An array of value objects is a dense block of their payloads, whatever the +payload is — that is the entire promise that makes "values are just data" attractive. + +**Observed** (`ValueClass.isFlatArray`, the VM answering about its own array): + +``` +type payload NR-nonAtomic NR-atomic nullable-atomic +P4 4 B true true true +P8i 8 B true true false +P8l 8 B true true false +P12 12 B false false false +P16 16 B false false false +P16l 16 B false false false +``` + +The cliff is at 8 bytes and it is total: past it, **no** array flavour flattens. Confirmed +independently by `-XX:+PrintFlatArrayLayout`, which logs a layout only for the shapes above the +line (`element size 4`, `element size 8`) and nothing for the others. + +**Why.** Flattening past a machine word needs either an atomic wide store or a decision to give +atomicity up; the current implementation declines both above 8 bytes. `FlatArrayElementMaxOops` +exists as a knob for reference-bearing payloads; there is no product knob that lifts the +primitive-payload ceiling on this build. + +**Consequence for this project — and it is the interesting one.** The line the VM draws is exactly +the line the thesis draws: + +| Type | Payload | Flat? | Which side of the thesis | +|---|---|---|---| +| `LaneId`, `Ordinal` | 4 B | **yes** | tiny descriptor vocabulary — Valhalla helps | +| `MaskId` | 8 B | **yes** | tiny descriptor vocabulary — Valhalla helps | +| `RowRange` | 16 B | no | descriptor, but already too wide | +| `Row` (id + class + value) | 16 B | no | per-entity materialisation — Valhalla does not help | + +So "Valhalla helps the descriptors, not the entities" is not a hand-wave about object headers. On +this build it is a hard cutoff in the VM, and a realistic entity is on the wrong side of it by +construction: an id plus one field already exceeds the budget. + +`RowRange` landing on the wrong side is worth stating plainly, because it is the one place the +expectation was too optimistic — it is a descriptor, it was expected to flatten, and it does not. + +--- + +## R3 — the densest layout has no supported spelling {#r3} + +**File:** `R3_NoSupportedFlatSurface.java` · **Observed:** `R3-observed.txt`, +`R3-bang-syntax-observed.txt` + +**Desired semantics.** `LaneId![] lanes = new LaneId![n];` — an array of non-null values, densely +packed, spelled in ordinary Java. + +**Observed:** + +``` +(1) new LaneId[8] flat=true accepts null=true (nullable-flat: pays for a null marker) +(2) LaneId![] DOES NOT PARSE — no null-restricted type syntax +(3) ValueClass.newNullRestricted... flat=true accepts null=false (jdk.internal, needs --add-exports) +(4) List.toArray() flat=false — generics erase to Object[] +``` + +and the syntax probe: + +``` +error: not a statement + L![] x = new L![2]; + ^ +``` + +Three separate gaps, and the first is the one most likely to be misread: + +1. **Supported source already flattens — partially.** `new LaneId[8]` *is* flat for a 4-byte + payload. It is *nullable*-flat, so it carries a null marker and is not the densest encoding, and + by R2 it stops being flat at all past 8 bytes. Reporting "plain arrays are not flat" would have + been wrong; the measured claim is narrower and more useful. +2. **The densest form is `jdk.internal`.** `ValueClass.newNullRestrictedNonAtomicArray` needs + `--add-exports java.base/jdk.internal.value=ALL-UNNAMED` and its own javadoc says it "should + only be used by internal JDK classes for experimental purposes". A library cannot ship it. +3. **Generics erase, so the boundary undoes it.** `List` is `Object[]` underneath and the + array is not flat. Any collection, stream, or generic cache reverts everything the previous two + points achieved. Specialised generics are the missing piece and are not in this build. + +**Consequence for this project.** The production API keeps `LaneId` and friends as plain `record`s +and does **not** adopt any of this. The migration path stays a one-word source change (`record` → +`value record`) precisely because nothing was bent to accommodate the current preview: no +`jdk.internal` dependency, no `--add-exports` in the shipped build, no API that hands out arrays of +descriptors. + +It also removes a temptation worth naming: if `List` had flattened, "just hand the caller a +`List`" would look like a viable alternative to the native lane. It does not flatten, so the +bulk path is not competing with a hypothetical fast object path — it is competing with the same +boxed one Java has always had. diff --git a/valhalla-lab/results/AB-default.diff b/valhalla-lab/results/AB-default.diff new file mode 100644 index 0000000..22da5ba --- /dev/null +++ b/valhalla-lab/results/AB-default.diff @@ -0,0 +1,132 @@ +3,4c3,4 +< platform stable +< java.vm.version 26.0.2+10-55 +--- +> platform valhalla +> java.vm.version 27-jep401ea3+1-1 +6c6 +< jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so] +--- +> jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED] +9,14c9,14 +< platform stable ? stable-record vocabulary; arrays are reference arrays; fields are references +< LaneId.class.isValue() false +< Ordinal.class.isValue() false +< MaskId.class.isValue() false +< RowRange.class.isValue() false +< Row.class.isValue() false +--- +> platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +> LaneId.class.isValue() true +> Ordinal.class.isValue() true +> MaskId.class.isValue() true +> RowRange.class.isValue() true +> Row.class.isValue() true +19,20c19,20 +< a == b (reference equality) false +< identityHashCode(a) == identityHashCode(b) false +--- +> a == b (reference equality) true +> identityHashCode(a) == identityHashCode(b) true +23,25c23,25 +< array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< array slot accepts null true +< synchronized(x) legality legal but never used by the production API +--- +> array flatness FLAT +> array slot accepts null false +> synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) +28c28 +< platform stable +--- +> platform valhalla +30,34c30,34 +< LaneId 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< Ordinal 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< MaskId 1 long payload= 8 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< RowRange 2 long payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< Row 1 long + 2 int payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +--- +> LaneId 1 int payload= 4 B array=FLAT +> Ordinal 1 int payload= 4 B array=FLAT +> MaskId 1 long payload= 8 B array=FLAT +> RowRange 2 long payload=16 B array=NOT-FLAT +> Row 1 long + 2 int payload=16 B array=NOT-FLAT +37c37 +< platform stable +--- +> platform valhalla +40,46c40,46 +< construct N LaneId, store into array 15.26 MiB +< ... per LaneId 16.00 B +< construct N LaneId, never escaping 6.71 MiB +< ... per LaneId 7.03 B +< LaneId[1024] flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< allocate+fill LaneId[N] (array + elements) 19.07 MiB +< ... per element 20.00 B +--- +> construct N LaneId, store into array 2.75 MiB +> ... per LaneId 2.89 B +> construct N LaneId, never escaping 7.63 MiB +> ... per LaneId 8.00 B +> LaneId[1024] flatness FLAT +> allocate+fill LaneId[N] (array + elements) 6.57 MiB +> ... per element 6.89 B +49,55c49,55 +< Descriptor kind identity class with two reference fields +< Descriptor fields null-restricted false +< construct N Descriptor (2 wrappers each) 53.41 MiB +< ... per Descriptor 56.00 B +< pass 2 wrappers through 3 call levels 7.90 MiB +< ... per call 8.29 B +< read 65,536 LaneId from array median= 44182.0 ns [min 41690.0 .. max 59914.0] n=51 +--- +> Descriptor kind value class with two @NullRestricted value fields +> Descriptor fields null-restricted true +> construct N Descriptor (2 wrappers each) 37.05 MiB +> ... per Descriptor 38.84 B +> pass 2 wrappers through 3 call levels 9.97 MiB +> ... per call 10.45 B +> read 65,536 LaneId from array median= 5349.0 ns [min 5179.0 .. max 11356.0] n=51 +59c59 +< platform stable +--- +> platform valhalla +61,65c61,65 +< bare long index median= 59148.0 ns [min 59077.0 .. max 114952.0] n=51 +< RowRange bounds (wrapper hoisted) median= 47952.0 ns [min 47759.0 .. max 65535.0] n=51 +< per-element wrapper: bytes allocated 1.00 MiB +< ... per element 16.00 B +< Ordinal built per element median= 50062.0 ns [min 49790.0 .. max 61194.0] n=51 +--- +> bare long index median= 63304.0 ns [min 59079.0 .. max 131743.0] n=51 +> RowRange bounds (wrapper hoisted) median= 57557.0 ns [min 56212.0 .. max 87932.0] n=51 +> per-element wrapper: bytes allocated 1.50 MiB +> ... per element 24.00 B +> Ordinal built per element median= 50021.0 ns [min 49762.0 .. max 75999.0] n=51 +68c68 +< platform stable +--- +> platform valhalla +79,83c79,83 +< (2)/(3) hydrate 65536 Row ? allocated 2.00 MiB +< ... per row 32.00 B +< array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +< ratio vs native lane bytes 2.00x +< retained heap (APPROX, gc-delta) 2.25 MiB +--- +> (2)/(3) hydrate 65536 Row ? allocated 2.50 MiB +> ... per row 40.00 B +> array flatness NOT-FLAT +> ratio vs native lane bytes 2.50x +> retained heap (APPROX, gc-delta) 2.75 MiB +86,89c86,89 +< (1) native one crossing, fused plan median= 16378.0 ns [min 15962.0 .. max 33819.0] n=51 +< (2/3) hydrate 65536 Row objects median= 603139.0 ns [min 529172.0 .. max 5246059.0] n=51 +< (2/3) scan the materialised objects median= 151886.0 ns [min 112687.0 .. max 201271.0] n=51 +< (2/3) hydrate THEN scan (honest total) median= 788227.0 ns [min 710221.0 .. max 1146017.0] n=51 +--- +> (1) native one crossing, fused plan median= 18805.0 ns [min 15515.0 .. max 38457.0] n=51 +> (2/3) hydrate 65536 Row objects median= 768624.0 ns [min 645509.0 .. max 1229947.0] n=51 +> (2/3) scan the materialised objects median= 91178.0 ns [min 83390.0 .. max 141648.0] n=51 +> (2/3) hydrate THEN scan (honest total) median= 898955.0 ns [min 822997.0 .. max 5059624.0] n=51 diff --git a/valhalla-lab/results/stable-api-javac.log b/valhalla-lab/results/stable-api-javac.log new file mode 100644 index 0000000..c298bca --- /dev/null +++ b/valhalla-lab/results/stable-api-javac.log @@ -0,0 +1 @@ +Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= diff --git a/valhalla-lab/results/stable-default.txt b/valhalla-lab/results/stable-default.txt new file mode 100644 index 0000000..3f30f78 --- /dev/null +++ b/valhalla-lab/results/stable-default.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform stable +java.vm.version 26.0.2+10-55 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform stable ? stable-record vocabulary; arrays are reference arrays; fields are references +LaneId.class.isValue() false +Ordinal.class.isValue() false +MaskId.class.isValue() false +RowRange.class.isValue() false +Row.class.isValue() false +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) false +identityHashCode(a) == identityHashCode(b) false +a local of this type accepts null true +array kind LaneId[] +array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +array slot accepts null true +synchronized(x) legality legal but never used by the production API + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform stable +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +Ordinal 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +MaskId 1 long payload= 8 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +RowRange 2 long payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +Row 1 long + 2 int payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform stable +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 15.26 MiB + ... per LaneId 16.00 B +construct N LaneId, never escaping 6.71 MiB + ... per LaneId 7.03 B +LaneId[1024] flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +allocate+fill LaneId[N] (array + elements) 19.07 MiB + ... per element 20.00 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind identity class with two reference fields +Descriptor fields null-restricted false +construct N Descriptor (2 wrappers each) 53.41 MiB + ... per Descriptor 56.00 B +pass 2 wrappers through 3 call levels 7.90 MiB + ... per call 8.29 B +read 65,536 LaneId from array median= 44182.0 ns [min 41690.0 .. max 59914.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform stable +sum (identical across all three) 6929623 +bare long index median= 59148.0 ns [min 59077.0 .. max 114952.0] n=51 +RowRange bounds (wrapper hoisted) median= 47952.0 ns [min 47759.0 .. max 65535.0] n=51 +per-element wrapper: bytes allocated 1.00 MiB + ... per element 16.00 B +Ordinal built per element median= 50062.0 ns [min 49790.0 .. max 61194.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform stable +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.00 MiB + ... per row 32.00 B + array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) + ratio vs native lane bytes 2.00x + retained heap (APPROX, gc-delta) 2.25 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 16378.0 ns [min 15962.0 .. max 33819.0] n=51 +(2/3) hydrate 65536 Row objects median= 603139.0 ns [min 529172.0 .. max 5246059.0] n=51 +(2/3) scan the materialised objects median= 151886.0 ns [min 112687.0 .. max 201271.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 788227.0 ns [min 710221.0 .. max 1146017.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/stable-lab-javac.log b/valhalla-lab/results/stable-lab-javac.log new file mode 100644 index 0000000..c298bca --- /dev/null +++ b/valhalla-lab/results/stable-lab-javac.log @@ -0,0 +1 @@ +Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= diff --git a/valhalla-lab/results/stable-noea.txt b/valhalla-lab/results/stable-noea.txt new file mode 100644 index 0000000..371624c --- /dev/null +++ b/valhalla-lab/results/stable-noea.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform stable +java.vm.version 26.0.2+10-55 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, -XX:-DoEscapeAnalysis] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform stable ? stable-record vocabulary; arrays are reference arrays; fields are references +LaneId.class.isValue() false +Ordinal.class.isValue() false +MaskId.class.isValue() false +RowRange.class.isValue() false +Row.class.isValue() false +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) false +identityHashCode(a) == identityHashCode(b) false +a local of this type accepts null true +array kind LaneId[] +array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +array slot accepts null true +synchronized(x) legality legal but never used by the production API + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform stable +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +Ordinal 1 int payload= 4 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +MaskId 1 long payload= 8 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +RowRange 2 long payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +Row 1 long + 2 int payload=16 B array=UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform stable +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 15.26 MiB + ... per LaneId 16.00 B +construct N LaneId, never escaping 15.26 MiB + ... per LaneId 16.00 B +LaneId[1024] flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) +allocate+fill LaneId[N] (array + elements) 19.07 MiB + ... per element 20.00 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind identity class with two reference fields +Descriptor fields null-restricted false +construct N Descriptor (2 wrappers each) 53.41 MiB + ... per Descriptor 56.00 B +pass 2 wrappers through 3 call levels 30.52 MiB + ... per call 32.00 B +read 65,536 LaneId from array median= 44861.0 ns [min 42156.0 .. max 69264.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform stable +sum (identical across all three) 6929623 +bare long index median= 59417.0 ns [min 59067.0 .. max 140131.0] n=51 +RowRange bounds (wrapper hoisted) median= 48136.0 ns [min 47847.0 .. max 105233.0] n=51 +per-element wrapper: bytes allocated 1.00 MiB + ... per element 16.00 B +Ordinal built per element median= 5166233.0 ns [min 4636587.0 .. max 10323741.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform stable +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.00 MiB + ... per row 32.00 B + array flatness UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat) + ratio vs native lane bytes 2.00x + retained heap (APPROX, gc-delta) 2.22 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 17631.0 ns [min 15693.0 .. max 40703.0] n=51 +(2/3) hydrate 65536 Row objects median= 575654.0 ns [min 513071.0 .. max 4428601.0] n=51 +(2/3) scan the materialised objects median= 129828.0 ns [min 108841.0 .. max 159354.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 703240.0 ns [min 642514.0 .. max 789762.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/valhalla-api-javac.log b/valhalla-lab/results/valhalla-api-javac.log new file mode 100644 index 0000000..c298bca --- /dev/null +++ b/valhalla-lab/results/valhalla-api-javac.log @@ -0,0 +1 @@ +Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= diff --git a/valhalla-lab/results/valhalla-default.txt b/valhalla-lab/results/valhalla-default.txt new file mode 100644 index 0000000..29b202f --- /dev/null +++ b/valhalla-lab/results/valhalla-default.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform valhalla +java.vm.version 27-jep401ea3+1-1 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +LaneId.class.isValue() true +Ordinal.class.isValue() true +MaskId.class.isValue() true +RowRange.class.isValue() true +Row.class.isValue() true +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) true +identityHashCode(a) == identityHashCode(b) true +a local of this type accepts null true +array kind LaneId[] +array flatness FLAT +array slot accepts null false +synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform valhalla +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=FLAT +Ordinal 1 int payload= 4 B array=FLAT +MaskId 1 long payload= 8 B array=FLAT +RowRange 2 long payload=16 B array=NOT-FLAT +Row 1 long + 2 int payload=16 B array=NOT-FLAT + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform valhalla +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 2.75 MiB + ... per LaneId 2.89 B +construct N LaneId, never escaping 7.63 MiB + ... per LaneId 8.00 B +LaneId[1024] flatness FLAT +allocate+fill LaneId[N] (array + elements) 6.57 MiB + ... per element 6.89 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind value class with two @NullRestricted value fields +Descriptor fields null-restricted true +construct N Descriptor (2 wrappers each) 37.05 MiB + ... per Descriptor 38.84 B +pass 2 wrappers through 3 call levels 9.97 MiB + ... per call 10.45 B +read 65,536 LaneId from array median= 5349.0 ns [min 5179.0 .. max 11356.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform valhalla +sum (identical across all three) 6929623 +bare long index median= 63304.0 ns [min 59079.0 .. max 131743.0] n=51 +RowRange bounds (wrapper hoisted) median= 57557.0 ns [min 56212.0 .. max 87932.0] n=51 +per-element wrapper: bytes allocated 1.50 MiB + ... per element 24.00 B +Ordinal built per element median= 50021.0 ns [min 49762.0 .. max 75999.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform valhalla +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.50 MiB + ... per row 40.00 B + array flatness NOT-FLAT + ratio vs native lane bytes 2.50x + retained heap (APPROX, gc-delta) 2.75 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 18805.0 ns [min 15515.0 .. max 38457.0] n=51 +(2/3) hydrate 65536 Row objects median= 768624.0 ns [min 645509.0 .. max 1229947.0] n=51 +(2/3) scan the materialised objects median= 91178.0 ns [min 83390.0 .. max 141648.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 898955.0 ns [min 822997.0 .. max 5059624.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/valhalla-lab-javac.log b/valhalla-lab/results/valhalla-lab-javac.log new file mode 100644 index 0000000..67055fc --- /dev/null +++ b/valhalla-lab/results/valhalla-lab-javac.log @@ -0,0 +1,3 @@ +Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/root/.ccr/java-truststore.p12 -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=PKCS12 -Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=34795 -Dhttp.nonProxyHosts=localhost|127.0.0.1|::1|127.*|0.*|::|169.254.*|anthropic.com|*.anthropic.com|*.anthropic.com|registry.npmjs.org|jsr.io|npm.jsr.io|pypi.org|files.pythonhosted.org|index.crates.io|proxy.golang.org|host.docker.internal|10.*|172.16.*|172.17.*|172.18.*|172.19.*|172.20.*|172.21.*|172.22.*|172.23.*|172.24.*|172.25.*|172.26.*|172.27.*|172.28.*|172.29.*|172.30.*|172.31.*|192.168.*|100.64.0.0/10|*.svc.cluster.local|*.svc.cluster.local -Djdk.http.auth.tunneling.disabledSchemes= -Djdk.http.auth.proxying.disabledSchemes= +Note: Some input files use preview features of Java SE 27. +Note: Recompile with -Xlint:preview for details. diff --git a/valhalla-lab/results/valhalla-noarrayflat.txt b/valhalla-lab/results/valhalla-noarrayflat.txt new file mode 100644 index 0000000..6c75ae5 --- /dev/null +++ b/valhalla-lab/results/valhalla-noarrayflat.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform valhalla +java.vm.version 27-jep401ea3+1-1 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED, -XX:+UnlockDiagnosticVMOptions, -XX:-UseArrayFlattening] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +LaneId.class.isValue() true +Ordinal.class.isValue() true +MaskId.class.isValue() true +RowRange.class.isValue() true +Row.class.isValue() true +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) true +identityHashCode(a) == identityHashCode(b) true +a local of this type accepts null true +array kind LaneId[] +array flatness NOT-FLAT +array slot accepts null false +synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform valhalla +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=NOT-FLAT +Ordinal 1 int payload= 4 B array=NOT-FLAT +MaskId 1 long payload= 8 B array=NOT-FLAT +RowRange 2 long payload=16 B array=NOT-FLAT +Row 1 long + 2 int payload=16 B array=NOT-FLAT + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform valhalla +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 22.89 MiB + ... per LaneId 24.00 B +construct N LaneId, never escaping 8.22 MiB + ... per LaneId 8.62 B +LaneId[1024] flatness NOT-FLAT +allocate+fill LaneId[N] (array + elements) 26.70 MiB + ... per element 28.00 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind value class with two @NullRestricted value fields +Descriptor fields null-restricted true +construct N Descriptor (2 wrappers each) 38.37 MiB + ... per Descriptor 40.23 B +pass 2 wrappers through 3 call levels 11.49 MiB + ... per call 12.05 B +read 65,536 LaneId from array median= 53465.0 ns [min 49219.0 .. max 88865.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform valhalla +sum (identical across all three) 6929623 +bare long index median= 59142.0 ns [min 59071.0 .. max 114270.0] n=51 +RowRange bounds (wrapper hoisted) median= 56997.0 ns [min 56660.0 .. max 81796.0] n=51 +per-element wrapper: bytes allocated 1.50 MiB + ... per element 24.00 B +Ordinal built per element median= 51224.0 ns [min 49782.0 .. max 91443.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform valhalla +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.50 MiB + ... per row 40.00 B + array flatness NOT-FLAT + ratio vs native lane bytes 2.50x + retained heap (APPROX, gc-delta) 2.75 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 14946.0 ns [min 14841.0 .. max 49967.0] n=51 +(2/3) hydrate 65536 Row objects median= 1708590.0 ns [min 610808.0 .. max 8047970.0] n=51 +(2/3) scan the materialised objects median= 148382.0 ns [min 119556.0 .. max 229605.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 843903.0 ns [min 781103.0 .. max 1306914.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/valhalla-noea.txt b/valhalla-lab/results/valhalla-noea.txt new file mode 100644 index 0000000..529ee73 --- /dev/null +++ b/valhalla-lab/results/valhalla-noea.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform valhalla +java.vm.version 27-jep401ea3+1-1 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED, -XX:-DoEscapeAnalysis] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +LaneId.class.isValue() true +Ordinal.class.isValue() true +MaskId.class.isValue() true +RowRange.class.isValue() true +Row.class.isValue() true +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) true +identityHashCode(a) == identityHashCode(b) true +a local of this type accepts null true +array kind LaneId[] +array flatness FLAT +array slot accepts null false +synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform valhalla +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=FLAT +Ordinal 1 int payload= 4 B array=FLAT +MaskId 1 long payload= 8 B array=FLAT +RowRange 2 long payload=16 B array=NOT-FLAT +Row 1 long + 2 int payload=16 B array=NOT-FLAT + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform valhalla +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 3.86 MiB + ... per LaneId 4.05 B +construct N LaneId, never escaping 24.41 MiB + ... per LaneId 25.60 B +LaneId[1024] flatness FLAT +allocate+fill LaneId[N] (array + elements) 28.35 MiB + ... per element 29.73 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind value class with two @NullRestricted value fields +Descriptor fields null-restricted true +construct N Descriptor (2 wrappers each) 38.98 MiB + ... per Descriptor 40.87 B +pass 2 wrappers through 3 call levels 29.57 MiB + ... per call 31.00 B +read 65,536 LaneId from array median= 5369.0 ns [min 5321.0 .. max 30666.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform valhalla +sum (identical across all three) 6929623 +bare long index median= 61840.0 ns [min 59069.0 .. max 128839.0] n=51 +RowRange bounds (wrapper hoisted) median= 60149.0 ns [min 56823.0 .. max 74673.0] n=51 +per-element wrapper: bytes allocated 3.00 MiB + ... per element 48.00 B +Ordinal built per element median= 94346.0 ns [min 91210.0 .. max 133533.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform valhalla +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.50 MiB + ... per row 40.00 B + array flatness NOT-FLAT + ratio vs native lane bytes 2.50x + retained heap (APPROX, gc-delta) 2.75 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 15885.0 ns [min 15694.0 .. max 32647.0] n=51 +(2/3) hydrate 65536 Row objects median= 1187577.0 ns [min 1044870.0 .. max 2002465.0] n=51 +(2/3) scan the materialised objects median= 74699.0 ns [min 63609.0 .. max 129188.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 3267205.0 ns [min 1374933.0 .. max 9370919.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/valhalla-nofieldflat.txt b/valhalla-lab/results/valhalla-nofieldflat.txt new file mode 100644 index 0000000..7af9d7b --- /dev/null +++ b/valhalla-lab/results/valhalla-nofieldflat.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform valhalla +java.vm.version 27-jep401ea3+1-1 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED, -XX:+UnlockDiagnosticVMOptions, -XX:-UseFieldFlattening] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +LaneId.class.isValue() true +Ordinal.class.isValue() true +MaskId.class.isValue() true +RowRange.class.isValue() true +Row.class.isValue() true +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) true +identityHashCode(a) == identityHashCode(b) true +a local of this type accepts null true +array kind LaneId[] +array flatness FLAT +array slot accepts null false +synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform valhalla +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=FLAT +Ordinal 1 int payload= 4 B array=FLAT +MaskId 1 long payload= 8 B array=FLAT +RowRange 2 long payload=16 B array=NOT-FLAT +Row 1 long + 2 int payload=16 B array=NOT-FLAT + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform valhalla +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 2.75 MiB + ... per LaneId 2.89 B +construct N LaneId, never escaping 8.22 MiB + ... per LaneId 8.61 B +LaneId[1024] flatness FLAT +allocate+fill LaneId[N] (array + elements) 6.58 MiB + ... per element 6.90 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind value class with two @NullRestricted value fields +Descriptor fields null-restricted true +construct N Descriptor (2 wrappers each) 76.29 MiB + ... per Descriptor 80.00 B +pass 2 wrappers through 3 call levels 11.14 MiB + ... per call 11.68 B +read 65,536 LaneId from array median= 4959.0 ns [min 4870.0 .. max 5886.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform valhalla +sum (identical across all three) 6929623 +bare long index median= 59187.0 ns [min 59075.0 .. max 123125.0] n=51 +RowRange bounds (wrapper hoisted) median= 56980.0 ns [min 56652.0 .. max 73287.0] n=51 +per-element wrapper: bytes allocated 1.50 MiB + ... per element 24.00 B +Ordinal built per element median= 49851.0 ns [min 49727.0 .. max 66813.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform valhalla +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.50 MiB + ... per row 40.01 B + array flatness NOT-FLAT + ratio vs native lane bytes 2.50x + retained heap (APPROX, gc-delta) 2.75 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 15964.0 ns [min 15779.0 .. max 40503.0] n=51 +(2/3) hydrate 65536 Row objects median= 789127.0 ns [min 628024.0 .. max 4599451.0] n=51 +(2/3) scan the materialised objects median= 78249.0 ns [min 75327.0 .. max 111755.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 1122018.0 ns [min 812442.0 .. max 1695253.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/valhalla-noflat.txt b/valhalla-lab/results/valhalla-noflat.txt new file mode 100644 index 0000000..5ab9312 --- /dev/null +++ b/valhalla-lab/results/valhalla-noflat.txt @@ -0,0 +1,91 @@ +Picked up JAVA_TOOL_OPTIONS: +lance-graph-java :: valhalla lab +platform valhalla +java.vm.version 27-jep401ea3+1-1 +java.vendor.version - +jvm args [--enable-native-access=ALL-UNNAMED, -Dlgj.library=/home/user/lance-graph-java/target/release/liblgj_abi.so, --enable-preview, --add-exports=java.base/jdk.internal.vm.annotation=ALL-UNNAMED, --add-exports=java.base/jdk.internal.value=ALL-UNNAMED, -XX:+UnlockDiagnosticVMOptions, -XX:-UseArrayFlattening, -XX:-UseFieldFlattening] + +== (a) SEMANTIC TRUTH ? is identity observable? ============================== +platform valhalla ? value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields +LaneId.class.isValue() true +Ordinal.class.isValue() true +MaskId.class.isValue() true +RowRange.class.isValue() true +Row.class.isValue() true +equal state => equals() true +different state => !equals() true +equal state => equal hashCode() true +equal state => equal toString() true +a == b (reference equality) true +identityHashCode(a) == identityHashCode(b) true +a local of this type accepts null true +array kind LaneId[] +array flatness NOT-FLAT +array slot accepts null false +synchronized(x) legality COMPILE ERROR under Valhalla (required: a type with identity) + +== FLATTENING CLIFF ? which payload shapes does the VM flatten? ============== +platform valhalla +note payload = declared field bytes, ignoring any header +LaneId 1 int payload= 4 B array=NOT-FLAT +Ordinal 1 int payload= 4 B array=NOT-FLAT +MaskId 1 long payload= 8 B array=NOT-FLAT +RowRange 2 long payload=16 B array=NOT-FLAT +Row 1 long + 2 int payload=16 B array=NOT-FLAT + +== (b)/(c) REPRESENTATION ? allocation, arrays, fields, arguments ============ +platform valhalla +allocation instrument baseline 0 B +N (operations per measurement) 1000000 +construct N LaneId, store into array 15.26 MiB + ... per LaneId 16.00 B +construct N LaneId, never escaping 4.71 MiB + ... per LaneId 4.94 B +LaneId[1024] flatness NOT-FLAT +allocate+fill LaneId[N] (array + elements) 19.07 MiB + ... per element 20.00 B +bare LaneId[N] with no elements stored 3.81 MiB + ... per slot 4.00 B +Descriptor kind value class with two @NullRestricted value fields +Descriptor fields null-restricted true +construct N Descriptor (2 wrappers each) 53.41 MiB + ... per Descriptor 56.00 B +pass 2 wrappers through 3 call levels 8.65 MiB + ... per call 9.07 B +read 65,536 LaneId from array median= 43102.0 ns [min 41523.0 .. max 77929.0] n=51 +native runtime lance-graph native runtime: abi 0.1, simd ndarray::simd avx512, profile release, library /home/user/lance-graph-java/target/release/liblgj_abi.so + +== FFM ADDRESSING ? is the wrapper free where it touches native memory? ====== +platform valhalla +sum (identical across all three) 6929623 +bare long index median= 59218.0 ns [min 59090.0 .. max 150920.0] n=51 +RowRange bounds (wrapper hoisted) median= 56978.0 ns [min 56635.0 .. max 83751.0] n=51 +per-element wrapper: bytes allocated 1.00 MiB + ... per element 16.00 B +Ordinal built per element median= 49935.0 ns [min 49722.0 .. max 83957.0] n=51 + +== THE THESIS ? 65,536 entities, three representations ======================= +platform valhalla +rows 65536 +question count(class==7 AND value>100) and sum(value) +answer (identical across all paths) 2173 rows, sum 499246 +selectivity 3.32% + +== heap cost =============================================================== +(1) native ? Java bytes allocated (warm) 816 B per query, for the fluent chain itself +(1) native ? Java objects per row 0 +(1) native ? native lane bytes 1.00 MiB (u64 id + u32 class + i32 value) +(1) native ? mask bytes 8.0 KiB (1 bit per row, packed) +(2)/(3) hydrate 65536 Row ? allocated 2.00 MiB + ... per row 32.00 B + array flatness NOT-FLAT + ratio vs native lane bytes 2.00x + retained heap (APPROX, gc-delta) 2.25 MiB + +== time to answer the question ============================================= +(1) native one crossing, fused plan median= 15435.0 ns [min 15335.0 .. max 30493.0] n=51 +(2/3) hydrate 65536 Row objects median= 857050.0 ns [min 533019.0 .. max 2422972.0] n=51 +(2/3) scan the materialised objects median= 124600.0 ns [min 112855.0 .. max 180439.0] n=51 +(2/3) hydrate THEN scan (honest total) median= 1631916.0 ns [min 1499609.0 .. max 2069662.0] n=51 + +lab complete. diff --git a/valhalla-lab/results/vocab-diff.txt b/valhalla-lab/results/vocab-diff.txt new file mode 100644 index 0000000..e69de29 diff --git a/valhalla-lab/run.sh b/valhalla-lab/run.sh new file mode 100755 index 0000000..9dfb31c --- /dev/null +++ b/valhalla-lab/run.sh @@ -0,0 +1,99 @@ +#!/usr/bin/env bash +# The A/B. One experiment source, two object models, two JDKs, one diff. +# +# Every measurement this lab reports is produced by this script. Nothing is quoted from memory. +set -uo pipefail + +LAB="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$LAB/.." && pwd)" + +STABLE_JDK="${STABLE_JDK:-/opt/jdks/jdk-26.0.2}" +VALHALLA_JDK="${VALHALLA_JDK:-/opt/jdks/jdk-27}" +LIB_DIR="${LIB_DIR:-$ROOT/target/release}" + +OUT="$LAB/results" +mkdir -p "$OUT" + +VAL_EXPORTS=( + --add-exports java.base/jdk.internal.vm.annotation=ALL-UNNAMED + --add-exports java.base/jdk.internal.value=ALL-UNNAMED +) + +banner() { printf '\n\033[1m== %s\033[0m\n' "$*"; } +fail() { printf '\033[31mFAIL:\033[0m %s\n' "$*" >&2; exit 1; } + +# ── 0. the A/B is only honest if the two vocabularies differ by exactly the modifier ────────── +banner "0. verifying the two vocabularies differ ONLY by the 'value' modifier" +if diff <(sed 's/^value record/record/' "$LAB/src/valhalla/com/adaworldapi/lancegraph/lab/Vocab.java") \ + "$LAB/src/stable/com/adaworldapi/lancegraph/lab/Vocab.java" > "$OUT/vocab-diff.txt"; then + echo "OK — src/valhalla/Vocab.java == src/stable/Vocab.java modulo 'value'" +else + cat "$OUT/vocab-diff.txt" + fail "the two vocabularies differ by more than the 'value' modifier; the A/B would not be an A/B" +fi + +if [ ! -f "$LIB_DIR/liblgj_abi.so" ]; then + fail "no native library at $LIB_DIR/liblgj_abi.so +build it with: + cd $ROOT/native/lgj-abi && CARGO_TARGET_DIR=$ROOT/target cargo build --release" +fi + +# ── 1. compile ──────────────────────────────────────────────────────────────────────────────── +compile () { # $1=tag $2=jdk $3=vocab-root shift 3 -> extra javac args + local tag=$1 jdk=$2 vocab=$3; shift 3 + local api="$OUT/$tag-api" lab="$OUT/$tag-lab" + rm -rf "$api" "$lab"; mkdir -p "$api" "$lab" + + # the production API, compiled by THIS jdk, with no preview features anywhere + "$jdk/bin/javac" -d "$api" $(find "$ROOT/java/src/main/java" -name '*.java') \ + 2> "$OUT/$tag-api-javac.log" || { cat "$OUT/$tag-api-javac.log"; fail "$tag: API compile"; } + + "$jdk/bin/javac" "$@" -cp "$api" -d "$lab" \ + $(find "$LAB/src/shared" "$vocab" -name '*.java') \ + 2> "$OUT/$tag-lab-javac.log" || { cat "$OUT/$tag-lab-javac.log"; fail "$tag: lab compile"; } + echo "compiled $tag" +} + +banner "1. compiling" +compile stable "$STABLE_JDK" "$LAB/src/stable" +compile valhalla "$VALHALLA_JDK" "$LAB/src/valhalla" \ + --enable-preview -source 27 -target 27 "${VAL_EXPORTS[@]}" + +# ── 2. run ──────────────────────────────────────────────────────────────────────────────────── +run () { # $1=tag $2=jdk $3=label shift 3 -> extra jvm args + local tag=$1 jdk=$2 label=$3; shift 3 + local f="$OUT/$tag-$label.txt" + echo "--> $tag/$label" + ( JAVA_TOOL_OPTIONS= "$jdk/bin/java" \ + --enable-native-access=ALL-UNNAMED \ + -Dlgj.library="$LIB_DIR/liblgj_abi.so" \ + "$@" \ + -cp "$OUT/$tag-api:$OUT/$tag-lab" com.adaworldapi.lancegraph.lab.RunAll ) \ + > "$f" 2>&1 + local rc=$? + echo " exit=$rc -> ${f#$LAB/}" + return 0 +} + +banner "2. running (default VM settings)" +run stable "$STABLE_JDK" default +run valhalla "$VALHALLA_JDK" default --enable-preview "${VAL_EXPORTS[@]}" + +banner "3. running with escape analysis OFF (what the object model costs unaided)" +run stable "$STABLE_JDK" noea -XX:-DoEscapeAnalysis +run valhalla "$VALHALLA_JDK" noea --enable-preview "${VAL_EXPORTS[@]}" -XX:-DoEscapeAnalysis + +banner "4. running with Valhalla flattening knobs OFF (does flattening cause the difference?)" +run valhalla "$VALHALLA_JDK" noarrayflat --enable-preview "${VAL_EXPORTS[@]}" \ + -XX:+UnlockDiagnosticVMOptions -XX:-UseArrayFlattening +run valhalla "$VALHALLA_JDK" nofieldflat --enable-preview "${VAL_EXPORTS[@]}" \ + -XX:+UnlockDiagnosticVMOptions -XX:-UseFieldFlattening +run valhalla "$VALHALLA_JDK" noflat --enable-preview "${VAL_EXPORTS[@]}" \ + -XX:+UnlockDiagnosticVMOptions -XX:-UseArrayFlattening -XX:-UseFieldFlattening + +banner "5. A/B diff" +diff "$OUT/stable-default.txt" "$OUT/valhalla-default.txt" > "$OUT/AB-default.diff" +echo "wrote ${OUT#$LAB/}/AB-default.diff ($(wc -l < "$OUT/AB-default.diff") lines)" + +banner "done — results in $OUT" +ls -1 "$OUT"/*.txt diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/NativeAccess.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/NativeAccess.java new file mode 100644 index 0000000..3e9fa04 --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/NativeAccess.java @@ -0,0 +1,49 @@ +package com.adaworldapi.lancegraph; + +import com.adaworldapi.lancegraph.internal.ffm.Engine; + +/** + * A read-only bridge into the library's package-private handle, for measurement code only. + * + *

Why this exists rather than a public accessor. The production API + * deliberately never surfaces a handle or a {@code MemorySegment} — that is the whole accessibility + * argument. But the lab has to reach the raw lane to build the very thing the thesis says you + * should not build (65,536 Java objects), and it must build them from the same bytes the + * native kernel reads, or the comparison is between two different datasets and proves nothing. + * + *

So the bridge lives here, in the library's package but in the lab's source tree, + * compiled onto the classpath as a split package. Nothing under {@code java/} changes, no public + * surface widens, and the coupling is visible in one file instead of leaking into the API. + * + *

It reads. It never writes, never closes, never mutates. + */ +public final class NativeAccess { + + private NativeAccess() {} + + /** Lane 0 — {@code u64} entity ids. */ + public static final int LANE_ID = 0; + /** Lane 1 — {@code u32} class tags. */ + public static final int LANE_CLASS = 1; + /** Lane 2 — {@code i32} signed values. */ + public static final int LANE_VALUE = 2; + + /** The generation-checked registry handle behind a pattern. Opaque; for describe calls only. */ + public static long handleOf(NativePattern pattern) { + return pattern.handle(); + } + + /** + * A bounded, read-only window onto one native lane. No membrane crossing happens when this is + * read — that is the point of the design, and the reason a Java-side Vector API kernel can + * compete at all. + */ + public static Engine.LaneWindow lane(NativePattern pattern, int laneId) { + return Engine.describeLane(pattern.handle(), laneId); + } + + /** The packed {@code u64} words behind a selection. */ + public static Engine.LaneWindow maskWords(Mask mask) { + return Engine.describeMask(mask.id().token()); + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FfmAddressingExperiment.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FfmAddressingExperiment.java new file mode 100644 index 0000000..ab5a9c3 --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FfmAddressingExperiment.java @@ -0,0 +1,86 @@ +package com.adaworldapi.lancegraph.lab; + +import com.adaworldapi.lancegraph.NativeAccess; +import com.adaworldapi.lancegraph.NativePattern; +import com.adaworldapi.lancegraph.internal.ffm.Engine; + +/** + * Does a semantic wrapper cost anything when it is used to address native memory? + * + *

This is the question that decides whether the API's vocabulary can go all the way down. The + * production API hides {@code MemorySegment} entirely, but the shape it hides is + * "offset arithmetic on a native address". If wrapping a row index in a meaningful type made that + * arithmetic slower, the vocabulary would have to stop at the API surface and become bare + * {@code long}s underneath — an abstraction that is only free where nobody is looking. + * + *

Three variants over the same lane, same 65,536 elements, same result asserted equal: + * + *

    + *
  • a bare {@code long} index — the floor; + *
  • a {@code RowRange} driving the loop bounds — a wrapper read once per loop; + *
  • a per-element wrapper ({@code Ordinal}) constructed inside the loop — a wrapper read once + * per element, which is the shape that would actually be expensive. + *
+ * + *

The third is the one that matters. A wrapper hoisted out of a loop is free on any JDK; a + * wrapper allocated 65,536 times is exactly the case a value class is supposed to make free, and + * exactly the case escape analysis sometimes already handles. Measuring both is what separates + * "Valhalla helped" from "the JIT was already doing it". + */ +final class FfmAddressingExperiment { + + private FfmAddressingExperiment() {} + + private static final int ROWS = 65_536; + + static void run() { + Lab.section("FFM ADDRESSING — is the wrapper free where it touches native memory?"); + Lab.kv("platform", Platform.NAME); + + try (NativePattern data = NativePattern.open(ROWS)) { + Engine.LaneWindow values = NativeAccess.lane(data, NativeAccess.LANE_VALUE); + + long bare = 0; + for (long i = 0; i < ROWS; i++) bare += values.getI32(i); + + RowRange range = RowRange.of(ROWS); + long viaRange = 0; + for (long i = range.start(); i < range.endExclusive(); i++) viaRange += values.getI32(i); + + long viaWrapper = 0; + for (int i = 0; i < ROWS; i++) viaWrapper += values.getI32(Ordinal.of(i).value()); + + if (bare != viaRange || bare != viaWrapper) { + throw new AssertionError("addressing variants disagree: " + bare + " / " + + viaRange + " / " + viaWrapper); + } + Lab.kv("sum (identical across all three)", bare); + + System.out.println(Lab.time("bare long index", 2_000, 51, () -> { + long acc = 0; + for (long i = 0; i < ROWS; i++) acc += values.getI32(i); + Lab.SINK = acc; + })); + + System.out.println(Lab.time("RowRange bounds (wrapper hoisted)", 2_000, 51, () -> { + long acc = 0; + for (long i = range.start(); i < range.endExclusive(); i++) acc += values.getI32(i); + Lab.SINK = acc; + })); + + long wrapperAlloc = Lab.allocatedBytes(() -> { + long acc = 0; + for (int i = 0; i < ROWS; i++) acc += values.getI32(Ordinal.of(i).value()); + Lab.SINK = acc; + }); + Lab.kv("per-element wrapper: bytes allocated", Lab.bytes(wrapperAlloc)); + Lab.kv(" ... per element", String.format("%.2f B", wrapperAlloc / (double) ROWS)); + + System.out.println(Lab.time("Ordinal built per element", 2_000, 51, () -> { + long acc = 0; + for (int i = 0; i < ROWS; i++) acc += values.getI32(Ordinal.of(i).value()); + Lab.SINK = acc; + })); + } + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FlatteningCliffExperiment.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FlatteningCliffExperiment.java new file mode 100644 index 0000000..4718129 --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FlatteningCliffExperiment.java @@ -0,0 +1,46 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * Where does flattening stop? + * + *

This experiment exists because the first run of {@link ThesisExperiment} produced a result + * that looked like a bug: the Valhalla {@code Row} array reported {@code NOT-FLAT} and cost + * more per row than the stable record. Rather than explain it away, the question was + * turned into a measurement — sweep payload sizes and find the cliff. + * + *

The answer on this build is a hard cutoff, and it lands exactly between the two categories + * the thesis distinguishes. A wrapper around one {@code int} or one {@code long} is flattened; an + * entity with an id plus two fields is not, in any array flavour. The thesis' central claim and + * the VM's current flattening budget happen to draw the same line — which is a much stronger + * result than "objects are slow", because it says why the line is where it is. + * + *

Each row of output is one payload shape. The types are declared in the vocabulary file so the + * stable and Valhalla trees declare identical shapes. + */ +final class FlatteningCliffExperiment { + + private FlatteningCliffExperiment() {} + + static void run() { + Lab.section("FLATTENING CLIFF — which payload shapes does the VM flatten?"); + Lab.kv("platform", Platform.NAME); + Lab.kv("note", "payload = declared field bytes, ignoring any header"); + + probe("LaneId", "1 int", 4, LaneId.class, LaneId.of(0)); + probe("Ordinal", "1 int", 4, Ordinal.class, Ordinal.of(0)); + probe("MaskId", "1 long", 8, MaskId.class, new MaskId(0)); + probe("RowRange", "2 long", 16, RowRange.class, RowRange.of(0)); + probe("Row", "1 long + 2 int", 16, Row.class, new Row(0, 0, 0)); + } + + private static void probe(String name, String shape, int payloadBytes, Class type, + Object init) { + String flat; + try { + flat = Platform.arrayFlatness(Platform.newArrayOf(type, 64, init)); + } catch (Throwable t) { + flat = "REFUSED: " + t.getClass().getSimpleName() + " " + t.getMessage(); + } + System.out.printf("%-10s %-16s payload=%2d B array=%s%n", name, shape, payloadBytes, flat); + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FootprintExperiment.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FootprintExperiment.java new file mode 100644 index 0000000..c87606b --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/FootprintExperiment.java @@ -0,0 +1,111 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * Truths (b) and (c) — representation. What does the same source actually become? + * + *

Four questions, each measured by the instrument that can least easily be fooled: + * + *

    + *
  1. Does constructing one cost a heap object? — allocated bytes, not timing. + *
  2. Is an array of them flattened? — the VM's own {@code isFlatArray} where it + * exists, plus the array's measured footprint either way. + *
  3. Is a field of one flattened into its container? — allocated bytes for the + * container, which changes by exactly the header+pointer cost when it is not. + *
  4. Does passing one to a method cost anything? — allocated bytes across a + * call chain deep enough that escape analysis has to give up. + *
+ * + *

Every count below is per {@link #N} operations, so a per-instance number can be divided out + * and compared against the theoretical object size (16-byte header + 4-byte int, padded to 16 for + * a one-int record on a 64-bit VM with compressed oops). + */ +final class FootprintExperiment { + + private FootprintExperiment() {} + + /** Large enough that per-operation bytes resolve cleanly; small enough to stay in a young gen. */ + static final int N = 1_000_000; + + static void run() { + Lab.section("(b)/(c) REPRESENTATION — allocation, arrays, fields, arguments"); + Lab.kv("platform", Platform.NAME); + Lab.kv("allocation instrument baseline", Lab.bytes(Lab.allocationInstrumentBaseline())); + Lab.kv("N (operations per measurement)", N); + + // ── 1. constructing a descriptor ───────────────────────────────────────────────────── + // The array store is what forces the object to escape. Without it, escape analysis + // deletes the allocation on BOTH platforms and the experiment measures nothing — which is + // itself worth stating, because it is exactly why a fast microbenchmark is not evidence. + Object[] sink = Platform.newLaneIdArray(N); + long ctorBytes = Lab.allocatedBytes(() -> { + for (int i = 0; i < N; i++) sink[i] = LaneId.of(i & 0xFFFF); + }); + Lab.OBJ_SINK = sink; + Lab.kv("construct N LaneId, store into array", Lab.bytes(ctorBytes)); + Lab.kv(" ... per LaneId", String.format("%.2f B", ctorBytes / (double) N)); + + // Non-escaping variant, for contrast. If this is ~0 on the stable platform too, that is + // escape analysis doing the value class's job for one particular loop — and the reason + // the run script also reports a -XX:-DoEscapeAnalysis pass. + long nonEscapingBytes = Lab.allocatedBytes(() -> { + long acc = 0; + for (int i = 0; i < N; i++) acc += LaneId.of(i).index(); + Lab.SINK = acc; + }); + Lab.kv("construct N LaneId, never escaping", Lab.bytes(nonEscapingBytes)); + Lab.kv(" ... per LaneId", String.format("%.2f B", nonEscapingBytes / (double) N)); + + // ── 2. array representation ────────────────────────────────────────────────────────── + Object[] probe = Platform.newLaneIdArray(1024); + Lab.kv("LaneId[1024] flatness", Platform.arrayFlatness(probe)); + long arrayBytes = Lab.allocatedBytes(() -> { + Object[] a = Platform.newLaneIdArray(N); + for (int i = 0; i < N; i++) a[i] = LaneId.of(i); + Lab.OBJ_SINK = a; + }); + Lab.kv("allocate+fill LaneId[N] (array + elements)", Lab.bytes(arrayBytes)); + Lab.kv(" ... per element", String.format("%.2f B", arrayBytes / (double) N)); + + long emptyArrayBytes = Lab.allocatedBytes(() -> Lab.OBJ_SINK = Platform.newLaneIdArray(N)); + Lab.kv("bare LaneId[N] with no elements stored", Lab.bytes(emptyArrayBytes)); + Lab.kv(" ... per slot", String.format("%.2f B", emptyArrayBytes / (double) N)); + + // ── 3. field flattening ────────────────────────────────────────────────────────────── + Lab.kv("Descriptor kind", Containers.kind()); + Lab.kv("Descriptor fields null-restricted", Containers.fieldsAreNullRestricted()); + Object[] descSink = new Object[N]; + long descBytes = Lab.allocatedBytes(() -> { + for (int i = 0; i < N; i++) descSink[i] = Containers.make(i & 7, i & 3); + }); + Lab.OBJ_SINK = descSink; + Lab.kv("construct N Descriptor (2 wrappers each)", Lab.bytes(descBytes)); + Lab.kv(" ... per Descriptor", String.format("%.2f B", descBytes / (double) N)); + + // ── 4. method-passing ──────────────────────────────────────────────────────────────── + long passBytes = Lab.allocatedBytes(() -> { + long acc = 0; + for (int i = 0; i < N; i++) acc += consume(LaneId.of(i), Ordinal.of(i & 15)); + Lab.SINK = acc; + }); + Lab.kv("pass 2 wrappers through 3 call levels", Lab.bytes(passBytes)); + Lab.kv(" ... per call", String.format("%.2f B", passBytes / (double) N)); + + // ── timing, secondary ──────────────────────────────────────────────────────────────── + // Reported after the byte counts, and second in importance. A timing that disagrees with + // the byte counts is a signal to distrust the timing, not the bytes. + Object[] readArr = Platform.newLaneIdArray(65_536); + for (int i = 0; i < 65_536; i++) readArr[i] = LaneId.of(i); + Lab.OBJ_SINK = readArr; + System.out.println(Lab.time("read 65,536 LaneId from array", 2_000, 51, () -> { + long acc = 0; + for (int i = 0; i < 65_536; i++) acc += ((LaneId) readArr[i]).index(); + Lab.SINK = acc; + })); + } + + // Three levels: shallow enough to inline, deep enough that a naive reading of "the JIT will + // fix it" is not automatically true. + private static long consume(LaneId l, Ordinal o) { return level2(l, o); } + private static long level2(LaneId l, Ordinal o) { return level3(l, o); } + private static long level3(LaneId l, Ordinal o) { return l.index() + o.value(); } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/IdentityExperiment.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/IdentityExperiment.java new file mode 100644 index 0000000..2e543fa --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/IdentityExperiment.java @@ -0,0 +1,66 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * Truth (a) — semantic: a {@link LaneId} is a value. Its object identity should be + * irrelevant, and nothing in the API should be able to observe it. + * + *

This experiment does not measure performance. It measures whether the runtime agrees + * with that semantic claim, which is the thing that has to be true before any performance claim + * means anything: a type whose identity is observable cannot be flattened, no matter how fast the + * JIT is. + * + *

The interesting output is not "value classes are values". It is the list of behaviours that + * are identical across the two runs — because every one of those is a place where the + * production API can move to Valhalla without a single caller noticing. + */ +final class IdentityExperiment { + + private IdentityExperiment() {} + + static void run() { + Lab.section("(a) SEMANTIC TRUTH — is identity observable?"); + Lab.kv("platform", Platform.NAME + " — " + Platform.describe()); + Lab.kv("LaneId.class.isValue()", Platform.isValueClass(LaneId.class)); + Lab.kv("Ordinal.class.isValue()", Platform.isValueClass(Ordinal.class)); + Lab.kv("MaskId.class.isValue()", Platform.isValueClass(MaskId.class)); + Lab.kv("RowRange.class.isValue()", Platform.isValueClass(RowRange.class)); + Lab.kv("Row.class.isValue()", Platform.isValueClass(Row.class)); + + LaneId a = LaneId.of(5); + LaneId b = LaneId.of(5); + LaneId c = LaneId.of(6); + + // The semantic contract, restated as assertions. These must hold on BOTH platforms — that + // is the point. If any of them differed, the migration would not be source-compatible. + Lab.kv("equal state => equals()", a.equals(b)); + Lab.kv("different state => !equals()", !a.equals(c)); + Lab.kv("equal state => equal hashCode()", a.hashCode() == b.hashCode()); + Lab.kv("equal state => equal toString()", a.toString().equals(b.toString())); + + // Reference equality is the ONE observable that legitimately differs, and the production + // API never uses it. Reported, not asserted, because a stable JDK is free to intern or not. + Lab.kv("a == b (reference equality)", a == b); + Lab.kv("identityHashCode(a) == identityHashCode(b)", + System.identityHashCode(a) == System.identityHashCode(b)); + + // Can a variable of this type hold null? Under Valhalla a plain declared type still can — + // null-restriction is a property of a FIELD or an ARRAY, not of the class. That surprises + // people, so it is measured rather than described. + LaneId maybeNull = null; + Lab.kv("a local of this type accepts null", maybeNull == null); + + Object[] arr = Platform.newLaneIdArray(4); + arr[0] = LaneId.of(1); + Lab.kv("array kind", arr.getClass().getSimpleName()); + Lab.kv("array flatness", Platform.arrayFlatness(arr)); + Lab.kv("array slot accepts null", Platform.arrayAcceptsNull(arr)); + + // synchronized(valueObject) does not compile under Valhalla — "required: a type with + // identity". It is exercised reflectively-in-spirit here (as a documented fact rather than + // live code) because writing it in shared source would break the stable compile too. + Lab.kv("synchronized(x) legality", + Platform.isValueVocabulary() + ? "COMPILE ERROR under Valhalla (required: a type with identity)" + : "legal but never used by the production API"); + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/Lab.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/Lab.java new file mode 100644 index 0000000..92e5b3b --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/Lab.java @@ -0,0 +1,144 @@ +package com.adaworldapi.lancegraph.lab; + +import java.lang.management.ManagementFactory; +import java.util.Arrays; + +/** + * The lab's measurement instruments. Deliberately small, and deliberately biased towards + * measurements that are hard to fool. + * + *

Why allocation bytes, not timing, is the primary instrument here

+ * + *

The question "did this abstraction cost a heap object?" is answered directly by + * {@code ThreadMXBean.getThreadAllocatedBytes}, which the VM maintains from TLAB accounting. A + * timing measurement answers it only by inference, and the inference is weak: escape analysis + * already removes many allocations, so a fast loop proves nothing about whether the object + * existed. Bytes are the observation; nanoseconds are the consequence. + * + *

This is also why {@code -XX:-DoEscapeAnalysis} appears in the run script. Comparing a record + * against a value class with escape analysis on measures the JIT's ability to see through a + * particular loop shape. Comparing with it off measures what the object model actually + * costs when the JIT cannot save it — which is the property that generalises to real code, where + * objects escape into arrays and collections all the time. + * + *

Timing

+ * + *

Warm-up then repeated measurement, reporting the median and the full min/max spread. The + * median is reported rather than the mean because a single GC pause or scheduler preemption in a + * 4-core container moves a mean and does not move a median. The spread is printed alongside so a + * reader can see when the median is not meaningful. This is not JMH — {@link #time} does not fork, + * does not detect steady state, and does not do statistical rigour. Where JMH-grade numbers are + * needed, they live in {@code bench/} and are produced by actual JMH. + */ +final class Lab { + + private Lab() {} + + private static final com.sun.management.ThreadMXBean THREADS = + (com.sun.management.ThreadMXBean) ManagementFactory.getThreadMXBean(); + + /** A sink the JIT cannot prove dead. Not a JMH blackhole; adequate for this lab's purposes. */ + static volatile long SINK; + static volatile Object OBJ_SINK; + + // ── allocation ─────────────────────────────────────────────────────────────────────────── + + /** + * Bytes this thread allocated while running {@code body}, minus the harness's own overhead. + * + *

The baseline subtraction matters: calling {@code getThreadAllocatedBytes} is itself not + * free of allocation on every VM, so an unsubtracted number would attribute the instrument's + * cost to the subject. Measured on this VM the baseline is 0, and the code says so rather than + * assuming it. + */ + static long allocatedBytes(Runnable body) { + long tid = Thread.currentThread().threadId(); + // Touch the instrument once so its own class-init allocation is not attributed to body. + THREADS.getThreadAllocatedBytes(tid); + long before = THREADS.getThreadAllocatedBytes(tid); + body.run(); + long after = THREADS.getThreadAllocatedBytes(tid); + return after - before; + } + + /** The instrument's own cost, printed so the reader can see it is negligible. */ + static long allocationInstrumentBaseline() { + return allocatedBytes(() -> {}); + } + + // ── retained footprint ─────────────────────────────────────────────────────────────────── + + /** + * Approximate retained heap of whatever {@code supplier} returns and keeps reachable. + * + *

Labelled approximate deliberately. It is a used-heap delta around a + * best-effort GC, so it is perturbed by anything else the VM does concurrently. It is reported + * because footprint is the question the thesis actually asks, and cross-checked against the + * exact allocation count so the two must agree in magnitude or one of them is wrong. + */ + static long retainedBytesApprox(java.util.function.Supplier supplier) { + Runtime rt = Runtime.getRuntime(); + settle(); + long before = rt.totalMemory() - rt.freeMemory(); + Object held = supplier.get(); + settle(); + long after = rt.totalMemory() - rt.freeMemory(); + OBJ_SINK = held; // keep it reachable across the second measurement + return after - before; + } + + private static void settle() { + for (int i = 0; i < 4; i++) { + System.gc(); + try { Thread.sleep(30); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } + } + } + + // ── timing ─────────────────────────────────────────────────────────────────────────────── + + record Timing(String name, double medianNs, double minNs, double maxNs, int iterations) { + @Override public String toString() { + return String.format("%-46s median=%10.1f ns [min %10.1f .. max %10.1f] n=%d", + name, medianNs, minNs, maxNs, iterations); + } + /** Per-unit cost, for reporting a per-row or per-element number honestly. */ + double perUnitNs(long units) { return medianNs / units; } + } + + /** + * Warm up, then measure {@code iterations} times and report median and spread. + * + * @param warmupRuns how many untimed runs before measuring; must be enough for C2 to compile + * the loop, which for these bodies is hundreds, not tens + */ + static Timing time(String name, int warmupRuns, int iterations, Runnable body) { + for (int i = 0; i < warmupRuns; i++) body.run(); + double[] samples = new double[iterations]; + for (int i = 0; i < iterations; i++) { + long t0 = System.nanoTime(); + body.run(); + samples[i] = System.nanoTime() - t0; + } + double[] sorted = samples.clone(); + Arrays.sort(sorted); + return new Timing(name, sorted[sorted.length / 2], sorted[0], sorted[sorted.length - 1], + iterations); + } + + // ── output ─────────────────────────────────────────────────────────────────────────────── + + static void section(String title) { + System.out.println(); + System.out.println("== " + title + " " + "=".repeat(Math.max(0, 74 - title.length()))); + } + + static void kv(String key, Object value) { + System.out.printf("%-44s %s%n", key, value); + } + + static String bytes(long b) { + if (Math.abs(b) < 1024) return b + " B"; + if (Math.abs(b) < 1024 * 1024) return String.format("%.1f KiB", b / 1024.0); + return String.format("%.2f MiB", b / (1024.0 * 1024.0)); + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/RunAll.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/RunAll.java new file mode 100644 index 0000000..031e9f0 --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/RunAll.java @@ -0,0 +1,46 @@ +package com.adaworldapi.lancegraph.lab; + +import com.adaworldapi.lancegraph.NativeRuntime; + +/** + * Runs every experiment and prints a machine-greppable report. + * + *

The same class is compiled twice — once against the {@code src/stable} vocabulary on the + * production JDK, once against {@code src/valhalla} on the JEP 401 build — and the two outputs are + * diffed. That is the whole design: one experiment source, two object models, one difference. + * + *

Exit codes: {@code 0} ran, {@code 2} the native library is unavailable (the thesis experiment + * needs it and a fabricated number would be worse than no number). + */ +public final class RunAll { + + private RunAll() {} + + public static void main(String[] args) { + System.out.println("lance-graph-java :: valhalla lab"); + Lab.kv("platform", Platform.NAME); + Lab.kv("java.vm.version", System.getProperty("java.vm.version")); + Lab.kv("java.vendor.version", System.getProperty("java.vendor.version", "-")); + Lab.kv("jvm args", java.lang.management.ManagementFactory.getRuntimeMXBean() + .getInputArguments()); + + IdentityExperiment.run(); + FlatteningCliffExperiment.run(); + FootprintExperiment.run(); + + if (!NativeRuntime.isAvailable()) { + Lab.section("NATIVE EXPERIMENTS SKIPPED"); + Lab.kv("reason", NativeRuntime.unavailableReason().getMessage()); + System.out.println("\nno native library — the thesis experiment cannot be run, and a" + + " number invented for it would be worse than none."); + System.exit(2); + } + Lab.kv("native runtime", NativeRuntime.describe()); + + FfmAddressingExperiment.run(); + ThesisExperiment.run(); + + System.out.println(); + System.out.println("lab complete."); + } +} diff --git a/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/ThesisExperiment.java b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/ThesisExperiment.java new file mode 100644 index 0000000..5c1ddb2 --- /dev/null +++ b/valhalla-lab/src/shared/com/adaworldapi/lancegraph/lab/ThesisExperiment.java @@ -0,0 +1,170 @@ +package com.adaworldapi.lancegraph.lab; + +import com.adaworldapi.lancegraph.NativeAccess; +import com.adaworldapi.lancegraph.NativePattern; +import com.adaworldapi.lancegraph.Pattern; +import com.adaworldapi.lancegraph.internal.ffm.Engine; + +/** + * The central claim, put at risk. + * + *

64,000 logical entities must NOT require 64,000 Java objects.
+ * + *

Three ways to answer the same question over the same 65,536 rows — + * "how many rows have {@code class == 7} and {@code value > 100}, and what do their values sum + * to?" — measured for heap cost and for time: + * + *

    + *
  1. native — one lane set, one packed mask, one bulk crossing. Zero Java + * objects per row. + *
  2. objects — 65,536 materialised {@code Row} instances, then an ordinary + * Java loop. This is what a developer writes when the API hands them entities. + *
  3. values — the same 65,536 rows as Valhalla value objects in a flat array. + * This is the "Valhalla will fix it" hypothesis, given its best case: null-restricted, + * non-atomic, flattened storage. + *
+ * + *

The comparison is only fair if all three read identical bytes. They do: + * paths 2 and 3 are populated by copying out of the very lanes path 1 scans, so no path enjoys a + * different dataset, a different distribution, or a warmer cache than the others. The three + * answers are asserted equal before any number is reported — a benchmark whose variants compute + * different things is measuring nothing. + * + *

The expected finding is that Valhalla helps the tiny descriptor vocabulary (see + * {@link FootprintExperiment}) and does not rescue per-entity materialisation. Expected is + * not observed; the numbers are printed either way, including if they contradict it. + */ +final class ThesisExperiment { + + private ThesisExperiment() {} + + static final int ROWS = 65_536; + private static final int CLASS_NEEDLE = 7; + private static final int VALUE_THRESHOLD = 100; + + static void run() { + Lab.section("THE THESIS — 65,536 entities, three representations"); + Lab.kv("platform", Platform.NAME); + Lab.kv("rows", ROWS); + Lab.kv("question", "count(class==" + CLASS_NEEDLE + " AND value>" + VALUE_THRESHOLD + + ") and sum(value)"); + + try (NativePattern data = NativePattern.open(ROWS)) { + + // ── (1) native: one lane set, one packed mask, one bulk op ─────────────────────── + // Warm first. The View's scratch mask is created lazily on the first terminal call, + // and the FFM scratch arena on the first crossing, so an unwarmed measurement would + // report one-time setup as if it were per-query cost. + for (int i = 0; i < 2_000; i++) { + Lab.SINK = data.view() + .where(Pattern.CLASS.eq(CLASS_NEEDLE)) + .where(Pattern.VALUE.gt(VALUE_THRESHOLD)) + .count(); + } + long nativeAllocBytes = Lab.allocatedBytes(() -> + Lab.SINK = data.view() + .where(Pattern.CLASS.eq(CLASS_NEEDLE)) + .where(Pattern.VALUE.gt(VALUE_THRESHOLD)) + .count()); + long nativeCount = data.view() + .where(Pattern.CLASS.eq(CLASS_NEEDLE)) + .where(Pattern.VALUE.gt(VALUE_THRESHOLD)) + .count(); + long nativeSum = data.view() + .where(Pattern.CLASS.eq(CLASS_NEEDLE)) + .where(Pattern.VALUE.gt(VALUE_THRESHOLD)) + .sumOf(Pattern.VALUE); + + // ── build (2)/(3) from the SAME native bytes ───────────────────────────────────── + Engine.LaneWindow ids = NativeAccess.lane(data, NativeAccess.LANE_ID); + Engine.LaneWindow classes = NativeAccess.lane(data, NativeAccess.LANE_CLASS); + Engine.LaneWindow values = NativeAccess.lane(data, NativeAccess.LANE_VALUE); + + Object[] rows = Platform.newRowArray(ROWS); + long hydrateAllocBytes = Lab.allocatedBytes(() -> { + for (int i = 0; i < ROWS; i++) { + rows[i] = new Row(ids.getU64(i), (int) classes.getU32(i), values.getI32(i)); + } + }); + Lab.OBJ_SINK = rows; + + long objCount = 0, objSum = 0; + for (int i = 0; i < ROWS; i++) { + Row r = (Row) rows[i]; + if (r.cls() == CLASS_NEEDLE && r.value() > VALUE_THRESHOLD) { objCount++; objSum += r.value(); } + } + + // ── the falsifier: all three must agree, or nothing below means anything ───────── + if (objCount != nativeCount || objSum != nativeSum) { + throw new AssertionError("the three paths do not compute the same answer: native=" + + nativeCount + "/" + nativeSum + " objects=" + objCount + "/" + objSum); + } + Lab.kv("answer (identical across all paths)", nativeCount + " rows, sum " + nativeSum); + Lab.kv("selectivity", String.format("%.2f%%", 100.0 * nativeCount / ROWS)); + + // ── heap cost ──────────────────────────────────────────────────────────────────── + Lab.section(" heap cost"); + Lab.kv("(1) native — Java bytes allocated (warm)", Lab.bytes(nativeAllocBytes) + + " per query, for the fluent chain itself"); + Lab.kv("(1) native — Java objects per row", 0); + Lab.kv("(1) native — native lane bytes", + Lab.bytes(ROWS * (8L + 4 + 4)) + " (u64 id + u32 class + i32 value)"); + Lab.kv("(1) native — mask bytes", + Lab.bytes((ROWS + 63) / 64 * 8L) + " (1 bit per row, packed)"); + Lab.kv("(2)/(3) hydrate " + ROWS + " Row — allocated", + Lab.bytes(hydrateAllocBytes)); + Lab.kv(" ... per row", String.format("%.2f B", hydrateAllocBytes / (double) ROWS)); + Lab.kv(" array flatness", Platform.arrayFlatness(rows)); + Lab.kv(" ratio vs native lane bytes", + String.format("%.2fx", hydrateAllocBytes / (double) (ROWS * 16L))); + + long retained = Lab.retainedBytesApprox(() -> { + Object[] held = Platform.newRowArray(ROWS); + for (int i = 0; i < ROWS; i++) { + held[i] = new Row(ids.getU64(i), (int) classes.getU32(i), values.getI32(i)); + } + return held; + }); + Lab.kv(" retained heap (APPROX, gc-delta)", Lab.bytes(retained)); + + // ── time ───────────────────────────────────────────────────────────────────────── + Lab.section(" time to answer the question"); + System.out.println(Lab.time("(1) native one crossing, fused plan", 2_000, 51, () -> + Lab.SINK = data.view() + .where(Pattern.CLASS.eq(CLASS_NEEDLE)) + .where(Pattern.VALUE.gt(VALUE_THRESHOLD)) + .count())); + + System.out.println(Lab.time("(2/3) hydrate " + ROWS + " Row objects", 200, 51, () -> { + Object[] a = Platform.newRowArray(ROWS); + for (int i = 0; i < ROWS; i++) { + a[i] = new Row(ids.getU64(i), (int) classes.getU32(i), values.getI32(i)); + } + Lab.OBJ_SINK = a; + })); + + System.out.println(Lab.time("(2/3) scan the materialised objects", 2_000, 51, () -> { + long c = 0, s = 0; + for (int i = 0; i < ROWS; i++) { + Row r = (Row) rows[i]; + if (r.cls() == CLASS_NEEDLE && r.value() > VALUE_THRESHOLD) { c++; s += r.value(); } + } + Lab.SINK = c + s; + })); + + System.out.println(Lab.time("(2/3) hydrate THEN scan (honest total)", 200, 51, () -> { + Object[] a = Platform.newRowArray(ROWS); + for (int i = 0; i < ROWS; i++) { + a[i] = new Row(ids.getU64(i), (int) classes.getU32(i), values.getI32(i)); + } + long c = 0, s = 0; + for (int i = 0; i < ROWS; i++) { + Row r = (Row) a[i]; + if (r.cls() == CLASS_NEEDLE && r.value() > VALUE_THRESHOLD) { c++; s += r.value(); } + } + Lab.SINK = c + s; + Lab.OBJ_SINK = a; + })); + } + } +} diff --git a/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Containers.java b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Containers.java new file mode 100644 index 0000000..a8e3be6 --- /dev/null +++ b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Containers.java @@ -0,0 +1,31 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * Field-flattening subjects, stable half. + * + *

{@code Descriptor} is shaped like the real thing: a {@code Field} in the production API holds + * a {@link LaneId} and an {@link Ordinal}, and both are wrappers around a single {@code int}. On a + * stable JDK that is three objects and two pointer hops to read two integers. + */ +final class Containers { + + private Containers() {} + + /** Two value-shaped wrappers held as ordinary references. */ + static final class Descriptor { + final LaneId lane; + final Ordinal ordinal; + Descriptor(int lane, int ordinal) { + this.lane = LaneId.of(lane); + this.ordinal = Ordinal.of(ordinal); + } + int laneIndex() { return lane.index(); } + int ordinalValue() { return ordinal.value(); } + } + + static Descriptor make(int lane, int ordinal) { return new Descriptor(lane, ordinal); } + static int read(Descriptor d) { return d.laneIndex() + d.ordinalValue(); } + + static String kind() { return "identity class with two reference fields"; } + static boolean fieldsAreNullRestricted() { return false; } +} diff --git a/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Platform.java b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Platform.java new file mode 100644 index 0000000..ec66ef7 --- /dev/null +++ b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Platform.java @@ -0,0 +1,68 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * The stable-JDK half of the A/B. Everything here reports what an ordinary {@code record} on a + * production JDK actually is. + * + *

Two of these answers are unknowable on a stable JDK rather than false, and they say + * so: there is no {@code ValueClass.isFlatArray} to ask, because there is no flattening to ask + * about. Reporting "false" would imply the question was answered; {@code UNKNOWN} says it was not. + */ +final class Platform { + + private Platform() {} + + static final String NAME = "stable"; + + /** Does the runtime consider this vocabulary identity-free? */ + static boolean isValueVocabulary() { + return isValueClass(LaneId.class); + } + + /** + * Is {@code type} a value class, on a platform that can even ask. + * + *

{@code Class::isValue} does not exist on this JDK at all — this is not an "unknowable + * answer" case like {@link #arrayFlatness}, it is a missing method, confirmed by the compiler. + * The correct answer is still knowable without the query: on a JDK with no value-class + * concept, nothing compiled here can BE one, so {@code false} is exact, not a guess. + */ + static boolean isValueClass(Class type) { + return false; + } + + /** Flatness of an array, where the runtime can answer. */ + static String arrayFlatness(Object array) { + return "UNKNOWN(no ValueClass API on a stable JDK; a reference array is never flat)"; + } + + /** Allocate the vocabulary array the way this platform can. */ + static Object[] newLaneIdArray(int n) { + return new LaneId[n]; + } + + /** Allocate the per-entity Row array the way this platform can. */ + static Object[] newRowArray(int n) { + return new Row[n]; + } + + /** Can this array hold null in every slot? */ + static boolean arrayAcceptsNull(Object[] array) { + try { Object keep = array[0]; array[0] = null; array[0] = keep; return true; } + catch (Throwable t) { return false; } + } + + /** + * Allocate the densest array this platform offers for {@code component}. + * + *

On a stable JDK there is exactly one kind of object array, so this is + * {@code Array.newInstance} and the {@code init} value is unused. + */ + static Object[] newArrayOf(Class component, int n, Object init) { + return (Object[]) java.lang.reflect.Array.newInstance(component, n); + } + + static String describe() { + return "stable-record vocabulary; arrays are reference arrays; fields are references"; + } +} diff --git a/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Vocab.java b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Vocab.java new file mode 100644 index 0000000..1a0fca5 --- /dev/null +++ b/valhalla-lab/src/stable/com/adaworldapi/lancegraph/lab/Vocab.java @@ -0,0 +1,61 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * The A/B vocabulary. This file exists twice — once under {@code src/stable} and once under + * {@code src/valhalla} — and the two copies are byte-identical apart from the word {@code value} + * on each declaration. {@code run.sh} diffs them and refuses to run if any other difference has + * crept in, so "same semantic contract, one modifier" is checked rather than claimed. + * + *

Each type mirrors the corresponding type in the real API + * ({@code com.adaworldapi.lancegraph.LaneId} and friends). The mirror exists because the real + * types must keep compiling on the production JDK; the lab is where the same contract is compiled + * a second way and measured. + */ +final class VocabDoc { private VocabDoc() {} } + +/** Which column of the struct-of-arrays a field lives in. Mirrors the production {@code LaneId}. */ +record LaneId(int index) { + LaneId { + if (index < 0) throw new IllegalArgumentException("lane index must be >= 0, was " + index); + } + static LaneId of(int index) { return new LaneId(index); } + @Override public String toString() { return "lane#" + index; } +} + +/** A half-open span of row indices. Mirrors the production {@code RowRange}. */ +record RowRange(long start, long endExclusive) { + RowRange { + if (start < 0) throw new IllegalArgumentException("start must be >= 0, was " + start); + if (endExclusive < start) throw new IllegalArgumentException("endExclusive < start"); + } + static RowRange of(long count) { return new RowRange(0, count); } + long length() { return endExclusive - start; } + boolean isEmpty() { return endExclusive == start; } + boolean contains(long row) { return row >= start && row < endExclusive; } + @Override public String toString() { return "rows[" + start + "," + endExclusive + ")"; } +} + +/** The identity of a selection, as an opaque token. Mirrors the production {@code MaskId}. */ +record MaskId(long token) { + int slot() { return (int) (token & 0xFFFF_FFFFL); } + int generation() { return (int) (token >>> 32); } + @Override public String toString() { return "mask#" + slot() + "@" + generation(); } +} + +/** A field's position within its schema. Mirrors the production {@code Ordinal}. */ +record Ordinal(int value) { + Ordinal { + if (value < 0) throw new IllegalArgumentException("ordinal must be >= 0, was " + value); + } + static Ordinal of(int value) { return new Ordinal(value); } + @Override public String toString() { return "#" + value; } +} + +/** + * One logical entity, materialised as a Java object. + * + *

This is the type the thesis says must NOT exist per row. It is declared here precisely so + * that "must not" can be measured instead of asserted: {@code ThesisExperiment} builds 65,536 of + * these and reports what they cost against the native lane that needs none of them. + */ +record Row(long id, int cls, int value) {} diff --git a/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Containers.java b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Containers.java new file mode 100644 index 0000000..9811b04 --- /dev/null +++ b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Containers.java @@ -0,0 +1,46 @@ +package com.adaworldapi.lancegraph.lab; + +import jdk.internal.value.ValueClass; +import jdk.internal.vm.annotation.NullRestricted; + +/** + * Field-flattening subjects, Valhalla half. Same shape as the stable {@code Descriptor}; the two + * differences are the {@code value} modifier and {@code @NullRestricted} on the fields. + * + *

Both differences are load-bearing and that is a finding, not a detail. + * {@code @NullRestricted} is what permits a flat encoding — a nullable value field still needs + * somewhere to record "this one is null". And the container must itself be a {@code value class}: + * an ordinary identity class with {@code @NullRestricted} fields fails at class load with + * {@code VerifyError: Invalid use of strict instance fields}, because javac on this build does not + * emit the strict initialisation order the VM demands. See + * {@code reproducers/nullrestricted-field-in-identity-class.md}. + */ +final class Containers { + + private Containers() {} + + /** Two value-shaped wrappers held as null-restricted, flattenable fields. */ + static value class Descriptor { + @NullRestricted final LaneId lane; + @NullRestricted final Ordinal ordinal; + Descriptor(int lane, int ordinal) { + this.lane = LaneId.of(lane); + this.ordinal = Ordinal.of(ordinal); + } + int laneIndex() { return lane.index(); } + int ordinalValue() { return ordinal.value(); } + } + + static Descriptor make(int lane, int ordinal) { return new Descriptor(lane, ordinal); } + static int read(Descriptor d) { return d.laneIndex() + d.ordinalValue(); } + + static String kind() { return "value class with two @NullRestricted value fields"; } + + static boolean fieldsAreNullRestricted() { + try { + return ValueClass.isNullRestrictedField(Descriptor.class.getDeclaredField("lane")); + } catch (NoSuchFieldException e) { + throw new AssertionError(e); + } + } +} diff --git a/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Platform.java b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Platform.java new file mode 100644 index 0000000..6a53d9d --- /dev/null +++ b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Platform.java @@ -0,0 +1,65 @@ +package com.adaworldapi.lancegraph.lab; + +import jdk.internal.value.ValueClass; + +/** + * The Valhalla half of the A/B. Same signatures as the stable {@code Platform}, so the shared + * experiment sources are byte-identical across both runs. + * + *

{@link ValueClass#isFlatArray} is the measurement that matters here: it is the VM answering + * about its own array, not an inference from a timing or a footprint estimate. That makes the + * flattening result hard to fool, which is why it is preferred over anything derived. + * + *

Requires {@code --add-exports java.base/jdk.internal.value=ALL-UNNAMED}. That export is the + * honest cost of the experiment: null-restricted, guaranteed-flat storage has no + * supported surface in this early-access build (see reproducers/no-public-flat-array-api.md). + */ +final class Platform { + + private Platform() {} + + static final String NAME = "valhalla"; + + static boolean isValueVocabulary() { + return isValueClass(LaneId.class); + } + + /** Is {@code type} a value class? The real, final {@code Class::isValue} query. */ + static boolean isValueClass(Class type) { + return type.isValue(); + } + + static String arrayFlatness(Object array) { + return ValueClass.isFlatArray(array) ? "FLAT" : "NOT-FLAT"; + } + + /** + * Null-restricted, non-atomic — the encoding with no null marker and no atomicity padding, + * i.e. the densest layout the VM offers. This is the array the thesis experiment needs to be + * fair to Valhalla: giving it a plain reference array would measure the old world twice. + */ + static Object[] newLaneIdArray(int n) { + return ValueClass.newNullRestrictedNonAtomicArray(LaneId.class, n, new LaneId(0)); + } + + static Object[] newRowArray(int n) { + return ValueClass.newNullRestrictedNonAtomicArray(Row.class, n, new Row(0, 0, 0)); + } + + static boolean arrayAcceptsNull(Object[] array) { + try { Object keep = array[0]; array[0] = null; array[0] = keep; return true; } + catch (Throwable t) { return false; } + } + + /** + * Allocate the densest array this platform offers for {@code component} — null-restricted and + * non-atomic, the encoding with neither a null marker nor atomicity padding. + */ + static Object[] newArrayOf(Class component, int n, Object init) { + return ValueClass.newNullRestrictedNonAtomicArray(component, n, init); + } + + static String describe() { + return "value-record vocabulary; null-restricted non-atomic arrays; @NullRestricted fields"; + } +} diff --git a/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Vocab.java b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Vocab.java new file mode 100644 index 0000000..78b88b8 --- /dev/null +++ b/valhalla-lab/src/valhalla/com/adaworldapi/lancegraph/lab/Vocab.java @@ -0,0 +1,61 @@ +package com.adaworldapi.lancegraph.lab; + +/** + * The A/B vocabulary. This file exists twice — once under {@code src/stable} and once under + * {@code src/valhalla} — and the two copies are byte-identical apart from the word {@code value} + * on each declaration. {@code run.sh} diffs them and refuses to run if any other difference has + * crept in, so "same semantic contract, one modifier" is checked rather than claimed. + * + *

Each type mirrors the corresponding type in the real API + * ({@code com.adaworldapi.lancegraph.LaneId} and friends). The mirror exists because the real + * types must keep compiling on the production JDK; the lab is where the same contract is compiled + * a second way and measured. + */ +final class VocabDoc { private VocabDoc() {} } + +/** Which column of the struct-of-arrays a field lives in. Mirrors the production {@code LaneId}. */ +value record LaneId(int index) { + LaneId { + if (index < 0) throw new IllegalArgumentException("lane index must be >= 0, was " + index); + } + static LaneId of(int index) { return new LaneId(index); } + @Override public String toString() { return "lane#" + index; } +} + +/** A half-open span of row indices. Mirrors the production {@code RowRange}. */ +value record RowRange(long start, long endExclusive) { + RowRange { + if (start < 0) throw new IllegalArgumentException("start must be >= 0, was " + start); + if (endExclusive < start) throw new IllegalArgumentException("endExclusive < start"); + } + static RowRange of(long count) { return new RowRange(0, count); } + long length() { return endExclusive - start; } + boolean isEmpty() { return endExclusive == start; } + boolean contains(long row) { return row >= start && row < endExclusive; } + @Override public String toString() { return "rows[" + start + "," + endExclusive + ")"; } +} + +/** The identity of a selection, as an opaque token. Mirrors the production {@code MaskId}. */ +value record MaskId(long token) { + int slot() { return (int) (token & 0xFFFF_FFFFL); } + int generation() { return (int) (token >>> 32); } + @Override public String toString() { return "mask#" + slot() + "@" + generation(); } +} + +/** A field's position within its schema. Mirrors the production {@code Ordinal}. */ +value record Ordinal(int value) { + Ordinal { + if (value < 0) throw new IllegalArgumentException("ordinal must be >= 0, was " + value); + } + static Ordinal of(int value) { return new Ordinal(value); } + @Override public String toString() { return "#" + value; } +} + +/** + * One logical entity, materialised as a Java object. + * + *

This is the type the thesis says must NOT exist per row. It is declared here precisely so + * that "must not" can be measured instead of asserted: {@code ThesisExperiment} builds 65,536 of + * these and reports what they cost against the native lane that needs none of them. + */ +value record Row(long id, int cls, int value) {}