Skip to content

Valhalla lab: three-truths method, causal isolation, 3 real reproducers - #2

Merged
AdaWorldAPI merged 4 commits into
mainfrom
claude/lance-graph-java-panama-valhalla-sus9w8
Aug 17, 2026
Merged

Valhalla lab: three-truths method, causal isolation, 3 real reproducers#2
AdaWorldAPI merged 4 commits into
mainfrom
claude/lance-graph-java-panama-valhalla-sus9w8

Conversation

@AdaWorldAPI

@AdaWorldAPI AdaWorldAPI commented Aug 17, 2026

Copy link
Copy Markdown
Owner

Summary

Completes D-LGJ-F. Applies the three-truths method (.claude/knowledge/valhalla-three-truths-method.md) to this project's small semantic value vocabulary (LaneId/Ordinal/MaskId/RowRange/Row) and the mission's mandatory headline experiment.

One experiment source (src/shared/), compiled twice against real JDKs — stable JDK 26 GA (plain record) and the official JEP 401 early-access binary (value record) — via a self-verifying run.sh that mechanically diffs the two Vocab.java files modulo the value keyword before trusting the A/B is honest.

Experiments: IdentityExperiment (is identity actually unobservable), FootprintExperiment (real per-object/array/field bytes), FfmAddressingExperiment (is the wrapper free where it touches native memory), ThesisExperiment (the mandatory headline: 65,536 rows as one native lane vs hydrated Java objects, on both platforms).

Causal isolation, not just observation: three additional run.sh passes toggle escape-analysis and UseArrayFlattening/UseFieldFlattening independently, to isolate which mechanism drives a measured difference rather than inferring it.

Three real Valhalla limitations, none of which changed the production API

Filed under reproducers/, each with a minimal self-contained file, its exact command line, and the observed output:

  • R1 — @NullRestricted field on an identity class is a VerifyError at class load. javac's fault: it emits field initializers after super(), the VM demands strict fields before it, and there's no source form to express the required order.
  • R2 — array flattening has a hard 8-byte payload cliff, confirmed via -XX:+PrintFlatArrayLayout. LaneId/Ordinal/MaskId (≤8B) flatten; RowRange/Row (16B) do not. This turns "Valhalla helps descriptors, not entities" from a hand-wave into a measured VM cutoff — and RowRange landing on the wrong side is flagged honestly as the one place the expectation was too optimistic.
  • R3 — the densest null-restricted array form is jdk.internal-only (unshippable), and generics erase flattening entirely (List<LaneId> is Object[] underneath). Foo! null-restricted type syntax confirmed not to parse, matching the earlier archaeology finding.

The production API adopts none of this — the migration path stays exactly record → value record, one word, no jdk.internal dependency, no --add-exports in the shipped build.

Measured headline

native, one crossing hydrate 65,536 Row, then scan
stable JDK 26 19.5 µs, 289 KiB 746 µs, 2.00 MiB
Valhalla JDK 27 EA 15.7 µs, 289.5 KiB 900 µs, 2.50 MiB

Native wins ~38–57× on time, ~7–9× on Java heap, on both platforms.

A real defect found and fixed before landing

IdentityExperiment and the stable-JDK Platform called Class::isValue() directly on four vocabulary types, with a comment incorrectly claiming it was "final API on JDK 26." It does not exist there at all — confirmed by a real javac compile failure. Fixed by routing every query through Platform.isValueClass(Class<?>), answered honestly per platform (stable: false, exact rather than a guess, since a JDK with no value-class concept can never produce one; Valhalla: the real type.isValue()). See .claude/board/EPIPHANIES.md E-LGJ-VALHALLA-MEASURED-NOT-ASSUMED-1.

What's deliberately NOT in this PR

bench/ (D-LGJ-G, the Java Vector API comparative benchmark) — real JMH + JOL jars are fetched but no source yet. It's the only remaining open row on .claude/board/STATUS_BOARD.md and will land in its own follow-up PR.

Test plan

  • ./run.sh (the lab's own self-verifying script): all 8 passes exit 0 — vocab-honesty check, stable compile+run, valhalla compile+run, escape-analysis-off ×2, both flattening flags toggled independently ×3
  • Manually re-verified: AB-default.diff shows only platform-dependent lines differ (identity/flatness/reference-equality) — the semantic contract (equals/hashCode/toString) is identical across both platforms
  • Manually re-verified the flattening-cliff causal isolation numbers by hand against the raw results/*.txt files before citing them
  • Audited: no cross-tree edits into java/ or native/ (confirmed via git status), no .h/cbindgen/jextract references

Generated by Claude Code

Generated by Claude Code

@AdaWorldAPI
Core vertical slice: docs/abi.md contract, native/lgj-abi, Java facade
546b17a 
Ships the fully verified core of the Panama x ndarray::simd x Valhalla
vertical slice (Phases A-E of the mission plan):

- docs/abi.md: the normative Rust<->Java ABI contract, written before
  either side was implemented so both could be checked against one
  frozen doc instead of each other.
- Five new ndarray::simd primitives (eq_u32_to_mask, gt_i32_to_mask,
  mask_and/mask_or(_assign), masked_sum_i32), added under ndarray's own
  W1a consumer contract.
- native/lgj-abi: the Rust ABI crate. Generation-checked handle
  registry, generic SoA fixture, bulk kernels routed exclusively
  through ndarray::simd, 14-symbol extern "C" surface. 72/72 tests
  green, clippy/fmt clean, and the registry's core safety check was
  disable-verified (short-circuited, confirmed exactly the two
  guarding tests go red, restored).
- java/: the Panama membrane (internal/ffm, never exposed publicly)
  and the public semantic facade (NativePattern/View/Predicate/
  Pattern/Mask). 132/132 checks green across 8 suites, including a
  reflection-enforced ApiSurfaceTest that mechanically proves zero FFM
  types ever reach a public signature, and a LazinessTest that
  empirically proves the thesis: building a chain costs zero
  crossings, evaluating it costs exactly one, independent of row
  count up to 1,000,000.
- .claude/: a 6-agent ensemble, 6 knowledge docs, and a full board
  (LATEST_STATE/STATUS_BOARD/AGENT_LOG/EPIPHANIES/TECH_DEBT/ISSUES/
  PR_ARC_INVENTORY/INTEGRATION_PLANS/CODEX_REVIEW_CHECKLIST), all
  scoped to this repo's actual seams.

A mechanical audit (D-LGJ-AUDIT) found and fixed the one real rule
violation before this commit: kernels.rs::simd_popcount was calling
the internal ndarray::hpc::bitwise path instead of the sanctioned
ndarray::simd re-export.

Deliberately NOT included: the Valhalla lab (valhalla-lab/) and the
Vector API benchmark harness (bench/) — still in flight, tracked as
open STATUS_BOARD.md rows, to land in a follow-up PR once reviewed
with the same rigor as this slice.

Generated by [Claude Code](https://claude.ai/code)
@AdaWorldAPI
Merge main (bootstrap commit) into dev branch to establish shared his…
a1e65cc 
…tory

main was bootstrapped from an isolated clone (README + .gitignore only,
to establish the default branch before any feature work existed) and
has no common ancestor with this branch. Merging here so PR #1 can
target main. dev branch's .gitignore (comprehensive) is kept over
main's (a subset); main's README.md is taken as-is since this branch
has none yet.

# Conflicts:
#	.gitignore
@AdaWorldAPI
Valhalla lab: three-truths method, causal isolation, 3 real reproducers
bdd40b2 
Completes D-LGJ-F. One experiment source (src/shared/), compiled twice
against real JDKs -- stable JDK 26 GA (record) and the official JEP 401
early-access binary (value record) -- via a self-verifying run.sh that
mechanically diffs the two Vocab.java files modulo the 'value' keyword
before trusting the A/B is honest.

Experiments: IdentityExperiment (semantic truth -- is identity actually
unobservable), FootprintExperiment (real per-object/array/field bytes
via allocation-delta + JOL where available), FfmAddressingExperiment
(is the wrapper free where it touches native memory), ThesisExperiment
(the mandatory headline: 65,536 rows as one native lane vs hydrated
Java objects, on both platforms).

Causal isolation via three additional run.sh passes: escape analysis
off, and UseArrayFlattening/UseFieldFlattening toggled independently --
isolates which flag actually drives the measured difference rather than
inferring it.

Three real Valhalla limitations reproduced and filed under
reproducers/, none of which changed the production API:
- R1: @NullRestricted field on an identity class is a VerifyError
  (javac's fault -- no source form expresses the required strict-field
  init order relative to super())
- R2: array flattening has a hard 8-byte payload cliff, confirmed via
  -XX:+PrintFlatArrayLayout. LaneId/Ordinal/MaskId (<=8B) flatten;
  RowRange/Row (16B) do not. This turns "Valhalla helps descriptors,
  not entities" from a hand-wave into a measured VM cutoff -- and
  RowRange landing on the wrong side is flagged as the one place the
  expectation was too optimistic.
- R3: the densest null-restricted array form is jdk.internal-only and
  generics erase flattening entirely; Foo! null-restricted type syntax
  confirmed not to parse, matching the earlier archaeology finding.

One real defect found and fixed before landing: IdentityExperiment and
the stable Platform called Class::isValue() directly on four vocabulary
types with a comment incorrectly claiming it was "final API on JDK 26"
-- it does not exist there at all, confirmed by a real javac failure.
Fixed by routing every query through Platform.isValueClass(Class<?>),
answered honestly per platform.

Generated by [Claude Code](https://claude.ai/code)
@AdaWorldAPI
AdaWorldAPI marked this pull request as ready for review August 17, 2026 21:30
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Aug 17, 2026

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@cursor

cursor Bot commented Aug 17, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_7e33a800-a5b4-4012-8a00-f5005c12f063)

@AdaWorldAPI
Merge main (squash of PR #1) to fix history divergence for PR #2
317ee83 
The squash-merge of PR #1 created a new commit (617e9d1) on main whose
content is identical to this branch's own 546b17a but is a different
commit object, so git/GitHub could not recognize PR #2's branch as
containing main's latest state, reporting mergeable_state=dirty despite
no actual content conflict. Merging explicitly to fix.

# Conflicts:
#	.claude/board/EPIPHANIES.md
#	.claude/board/STATUS_BOARD.md
@AdaWorldAPI
AdaWorldAPI merged commit 60e2607 into main Aug 17, 2026
AdaWorldAPI added a commit that referenced this pull request Aug 17, 2026
@AdaWorldAPI
Merge main (squash of PR #2) to keep dev branch history aligned
ea4753c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AdaWorldAPI