Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .claude/board/INTEGRATION_PLANS.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,20 @@
## 2026-08-17 — ogar-machine-v1 (EXPLORATORY; the population emulator)

Plan: `.claude/plans/ogar-machine-v1.md`. Captured from the operator's
second archived ChatGPT context (the first was convergent confirmation;
this one is a genuinely NEW workload): one row = one machine STATE,
control flow becomes population masks ("which machines currently execute
ADD?"), Ghidra P-code as the normalized guest ISA
(`AdaWorldAPI/ghidra` now attached + shallow-cloned at
`/workspace/ghidra`), differential migration testing as the killer demo
(65,536 worlds through legacy XOR replacement → the divergent few),
Lance as the time machine (via `lance-graph-hydrate`, lance-graph
#957/#958 — inherited, never re-implemented). Strong/weak claims
separated in the plan; gated on W3 + one W5 example + Ghidra
archaeology + probe P-M1.

**Status: NAMED, not scheduled.** W3 remains the next action.

## 2026-08-17 — consumer-{world-trades,bricks-analytics,graph-traversal}-v1 (PLANS; the three W5 consumer examples)

Plans: `.claude/plans/consumer-world-trades-v1.md`,
Expand Down
29 changes: 29 additions & 0 deletions .claude/board/PR_ARC_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,35 @@
> anti-pattern the imported board rules name. Backfilled below in one
> pass rather than left stale; PR #4 onward gets its entry at merge time.

## PR #5 — SoA row store: 512B rows, 32 facet lanes, ABI minor 2 (merged 2026-08-17, squash `78aa60e`)

Companion: **AdaWorldAPI/ndarray#279** (W1), merged first — `lgj-abi`'s
`kernels.rs` calls `iter_u32x16` / `eq_u32_strided_to_mask` from it.

- **Added:** `native/lgj-abi/src/rowstore.rs` (one `Arc<[u8]>`, two
readings, zero copies, normative SplitMix64 generator);
`LGJ_RESOURCE_ROWSTORE` + `lgj_rowstore_open` + `lgj_op_eq_classid` +
`lgj_row_facet_match`; `docs/abi.md` §11; the W1–W5 wave plan and three
consumer-example plans; `.claude/knowledge/soa-row-store-layout.md`.
- **Locked:** the 512 B / 32 × (4 B classid + 12 B payload) layout as
substrate truth (Java's view may differ); facet lanes ride the
**unchanged** `LgjLaneDesc`; masks parent onto row stores so the whole
existing mask algebra applies with no new surface; `byte_len` is the
exact covered span `(len-1)*stride + elem_bytes` (the old `len*stride`
form would have let Java bound a segment past the allocation's end);
ABI minor 1→2 and the §1/§7 symbol count corrected 14→18 (`nm -D`).
Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`;
self-correction: `E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`.
- **Deferred:** `align(64)` base (stated honestly — arrives with real
`NodeRow`); payload semantics (a ClassView concern one layer up); fused
plans over facet lanes (W6, only if measurement asks).
- **Docs:** `abi.md` §11 + 4 plan files + 1 knowledge doc + full board.
- **Confidence:** High — 84/84, clippy/fmt clean, 18/18 symbols; both new
kernels parity-checked against independent scalar references over
10 row counts × 2 seeds × 4 facets × 4 needles and cross-checked a third
way; two-sided payload-vs-classid falsifier. Both bot reviewers (cursor,
codex) hit usage limits and did not run.

## PR #4 — Phase I synthesis docs + fusion re-run + board hygiene (merged 2026-08-17, squash `bd92c58`)

- **Added:** `docs/{architecture,panama,valhalla-lab,execution-boundary}.md`
Expand Down
109 changes: 109 additions & 0 deletions .claude/knowledge/prior-art-and-the-layout-bridge-claim.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Prior art, and the claim that distinguishes this project

> **READ BY:** anyone writing positioning/architecture prose, the consumer-
> example authors (W5), and any session tempted to cite the operator's
> archived ChatGPT discussion (2026-08-17, predates this repo's build) as
> design authority. That discussion is assessed HERE, once — its residual
> value extracted, its errors pinned — so it is not re-mined.

## Assessment verdict

The discussion **converged independently on the architecture this repo then
built and measured** — layout-centric bridge, engine-chooses-the-kernel,
"Java understands the layout, Rust owns the physics," a brutally small
vertical-slice MVP. Convergence from an independent derivation is mild
evidence the shape is right, and zero evidence about anything the
discussion *assumed but never measured*. Three parts remain valuable; two
parts are wrong relative to what this repo has since measured or mandates.

## 1. The distinction that holds up: callability vs shared executable layout

The competitive landscape, correctly characterized there and worth keeping:

| bridge | what it is | what it solves |
|---|---|---|
| JNI / `jni-rs` | env pointers, object handles, marshaling ceremony | callability |
| `j4rs` | higher-level interop layer | callability |
| UniFFI | binding generator (Kotlin/Swift/Python official; Java external, FFM-based) | callability |
| Panama FFM (JEP 454, final since JDK 22) | the *mechanism* — `MemorySegment`/`MemoryLayout`/`Linker` | reachability, not semantics |

All of these are **function bridges**: Java calls a native function, values
marshal across. This project's claim is a **layout bridge**: Java and Rust
*execute over the same bytes*, and the contract is the layout itself
(`docs/abi.md`), runtime-proven by the manifest cross-check. The
discussion's phrase for the moat — solving *"shared executable layout"*
where existing bridges solve *"callability"* — is the crispest one-line
positioning this project has and is worth using verbatim.

## 2. Where the discussion's sketch was WEAKER than what was built

Its `OgarAbiPage` sketch (`schema_key, row_count, stride, lane_count,
flags, data_ptr`) carries a bare `data_ptr` with **no lifetime story at
all** — no generation check, no epoch, no owner, no parent-liveness, no
close semantics. That is precisely the machinery this repo's registry
provides and **disable-verifies** (`abi.md` §4: no code path in which a
stale handle dereferences freed memory). A page descriptor without a
liveness protocol is a use-after-free with documentation. Keep this as the
standing answer to "couldn't the ABI just be one page struct?"

## 3. The genuinely forward-looking extractable: the schema key as the join point

> "One ABI key connects: Java MemoryLayout, Rust repr(C)/SoA, ClassView,
> ontology predicate, low-code block, Lance column projection, SIMD
> kernel."

This is the OGAR classid doctrine — *classid is pure address; the magic is
what it resolves to* — arriving from an independent direction, which is
worth something. Concretely for this repo: today a resource names its
layout contract only *implicitly* (`kind == ROWSTORE` ⇒ the §11 geometry).
When the real ClassView slice lands, `LgjResourceInfo` (and/or
`LgjLaneDesc`) should gain an explicit **schema/classid field** so a
resource names *which* layout contract its bytes obey — additive, one
minor bump, and it is what makes "one key, many projections" literal at
the membrane. Filed as the W6 consideration in
`.claude/plans/lgj-soa-substrate-v1.md`.

## 4. The claims discipline worth adopting verbatim

The discussion's own guard against its headline numbers: never state
"2000×" as an engine claim. The architecture turns *pathological baselines*
(object-per-edge allocation, boxed values, string predicates, virtual
dispatch, JNI marshaling, JSON serialization, per-row Java loops) into
dense native kernels — so deltas are **baseline-dependent**: 10×, 100×,
1000×+ depending on how bad the baseline was. This repo's bench already
practices the stronger form (measure, publish the reproduction command,
state the noise floor); the W5 consumer examples MUST phrase any
comparison this way, and the graph-traversal plan already cites the
six-component BEFORE stack *as architecture, not as a benchmarked number*.

## 5. Two corrections — pin these so the text is never cited naively

1. **"Rust/RISC-speed execution through the membrane" assumes the native
side always wins. Measured false here.** Component C: the Java Vector
API, zero-copy on the *same* native segment, beats the native crossing
at **every** row count tested (64 → 4.2M) for a single predicate; the
crossing pays off for *composed, fused* work (Component E, 10.8–31.1×).
The real split is finer than the discussion's frame and lives in
`docs/execution-boundary.md`. Any prose inheriting the discussion's
framing must inherit the measurement instead.
2. **Its ndarray paragraph is about UPSTREAM crates.io ndarray** ("an
ergonomic n-dimensional array crate … `matrixmultiply` underneath, not
a SIMD layer"). The **AdaWorldAPI fork** this stack mandates is a
different artifact: `ndarray::simd` IS the SIMD polyfill layer
(dispatched AVX-512/AVX2/NEON/wasm/scalar under the W1a consumer
contract), and this project's §8 provenance rule depends on that. Do
not import the discussion's correction against the fork. (Its nightly
caveat about `std::simd` portable SIMD is true and is exactly why the
polyfill exists on stable.)
3. *(minor, was already true)* Java's Vector API is still incubating —
consistent with this repo's `--add-modules jdk.incubator.vector` and
noted in `jdk-toolchain-facts.md`.

## The slogan shelf (used sparingly, they are earned now)

- *"Java understands the layout. Rust owns the physics."* — matches
`architecture.md` §layers; safe to use.
- *"Panama gives Java fingers; this ABI gives it bones, tendons, and a
nervous system."* — rhetorically fine; keep it out of normative docs.
- *"Existing bridges solve callability; this solves shared executable
layout."* — the best of the three; use in positioning.
13 changes: 13 additions & 0 deletions .claude/plans/lgj-soa-substrate-v1.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,24 @@
| **W3** | Java `RowStore` facade: no FFM in public signatures; structured `MemoryLayout` (`sequence(32, struct(u32 classid, 12B payload))`); minor-≥2 gate; `FacetMatchView` zero-copy accessor over a Java-arena segment; `RowStoreParityTest` transcribing the generator | OPEN — next |
| **W4** | Bench Component F: Java Vector API per-row facet scan (one `IntVector` 16-lane chunk = 4 facets, same algorithm as the Rust kernel) vs `lgj_row_facet_match` crossing vs scalar VarHandle walk — the "where does execution belong" question re-asked on the REAL layout | OPEN |
| **W5** | The three consumer examples (own plan files, below) | PLANNED |
| **W6** *(named, not scheduled)* | ClassView wiring — and with it an explicit **schema/classid field** on `LgjResourceInfo`/`LgjLaneDesc` (additive, one minor bump), so a resource names WHICH layout contract its bytes obey instead of implying it via `kind`. Provenance + rationale: `.claude/knowledge/prior-art-and-the-layout-bridge-claim.md` §3 ("one key, many projections" made literal at the membrane). Also the `align(64)` base guarantee (real `NodeRow`) and, only if measurement asks, fused plans over facet lanes | NAMED |

Wave rule (house style): one wave = one reviewable PR; gates run centrally
(orchestrator only — agents never run cargo); every safety property lands
disable-verified, every measured claim lands with its reproduction command.

**Cross-repo dependency note (2026-08-17, operator-flagged):** lance-graph
**#957** (merged) minted `crates/lance-graph-hydrate` — the generic
SoA→S3→volume→Lance hydration pattern (four-state lifecycle,
hydrate-aside/publish-by-rename, warm markers, dirty detection), minted in
lance-graph *specifically so consumers inherit it as a path/git dependency
rather than re-implement*. **#958** (open, another session's PR) is its 5+3
council hardening fast-follow. Consequence here: when this substrate's
persistence slice arrives (the "Seal & Persist (Lance)" column of the
formula, and `ogar-machine-v1.md`'s time-machine storage), the hydration
path is `lance-graph-hydrate` — inherited, never re-derived. Do not design
a hydration mechanism in this repo.

## What W2 locked (so W3+ doesn't re-derive it)

- **Layout truth:** `ROW_BYTES=512`, `ROW_FACETS=32`, `FACET_BYTES=16`,
Expand Down
116 changes: 116 additions & 0 deletions .claude/plans/ogar-machine-v1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
# ogar-machine-v1 — the population emulator (EXPLORATORY, named not scheduled)

> **Status: NAMED** (2026-08-17). Source: the operator's second archived
> ChatGPT context, assessed here once so it is not re-mined. Unlike the
> first archived discussion (convergent confirmation —
> `.claude/knowledge/prior-art-and-the-layout-bridge-claim.md`), this one
> proposes a **genuinely new workload** for the shipped substrate, and the
> operator has attached `AdaWorldAPI/ghidra` to the session for it.
> Nothing here reorders the wave plan: W3 (Java `RowStore` facade) remains
> the next action; this plan is the shelf the idea sits on, with its
> strong/weak claims already separated.

## The inversion, in one sentence

A VM is `state + transition → new state`; instead of one machine executing
instructions sequentially, **one row is one machine STATE**, and the
substrate executes each *operation* across the whole population at once:

```
65,536 execution contexts (rows)
→ classify current opcode (a classid-style scan → population masks)
→ per-population bulk execution (SIMD ADD over the ADD mask, gathers
over the LOAD mask, mask updates for
BRANCH)
→ next state
```

The VM never asks *"what does machine 17 execute?"* — it asks *"which
machines currently execute ADD?"* **Control flow becomes population
masks.** That is not an analogy to the shipped substrate; it is literally
its op set: rows, classid scans, mask algebra, masked bulk ops, survivors.

## The claim discipline (the discussion's own, kept sharp)

- **Weak claim, explicitly rejected**: "emulate a Pentium faster than
QEMU." One interactive instance is branchy and sequential; SIMD is not
fairy dust for `A then B then C`. Do not build toward this and do not
let a demo imply it.
- **Strong claim, the actual target**: *from Java, explore 65,536
executions of an unsupported binary simultaneously, time-travel them,
XOR them against the replacement, and return only the worlds where
behavior diverges.* Every element of that sentence maps to something
this stack already does well (population masks / Lance versioning /
mask XOR / survivors-only inspection).

## Ghidra's role — front-end compiler, never a peer emulator

Do NOT implement x86/ARM/68k. Ghidra lifts the legacy binary **once** to
P-code (a small normalized op set: COPY, LOAD, STORE, BRANCH, CBRANCH,
CALL, RETURN, INT_ADD/SUB/MULT/AND/OR/XOR, …); the OGAR Machine executes
P-code only. `legacy.exe → Ghidra (once) → normalized program image →
population execution`. This is the same shape as the workspace's
ruff→OGAR harvest arms: an existing analyzer becomes the transcoder
front-end, and the substrate executes the normalized IR. First concrete
archaeology step in the fresh clone: locate the P-code opcode enum and
SLEIGH lifting surface, and size the *real* op set (the list above is the
discussion's sketch, not a verified inventory).

## The four load-bearing design rules captured from the discussion

1. **Differential migration testing is the killer demo.** Same 65,536
input worlds through the legacy machine and the replacement;
`legacy_output[] XOR new_output[] → divergence_mask`; show the 15 of
65,536 worlds the rewrite broke, click into one, see the concrete
state. (Note for a future warden pass: outcome-XOR here is a
*comparison*, not a state-transition kernel — it does not touch
lance-graph's `I-SUBSTRATE-MARKOV` XOR restriction, which governs
transition bundling.)
2. **Lance is the time machine.** Per-cycle sealed diffs (changed
registers / changed pages), `machine.at(cycle)`, `machine.diff(a, b)` —
deterministic rewind as a *consequence* of the substrate, matching the
workspace's episodic-=-Lance-versions doctrine. No bespoke snapshot
format.
3. **Memory purity resistance** (the discussion's own best guard): guest
RAM does NOT go into 512-byte rows. Semantic machine state (PC, regs,
flags, device state, identities, relations) → SoA lanes; the memory
image (pages) → a dense backing store the lens points at. This mirrors
the substrate's own key/value split — meaning and addressing in the
graph, bulk bytes dense and compressible.
4. **Semantic shims erode the emulator.** Recognize stable external
surfaces (USER/GDI/KERNEL/ODBC/filesystem/registry) and progressively
replace instruction execution with semantic operations — 90/10 →
40/60 → 5/95. The endpoint is not `binary → reconstructed source` but
`binary → normalized behavior machine`, which is the OGAR transpile
doctrine's 85/15 split arrived at by *running* the program instead of
parsing it.

Also named, further out: carrying BOTH branch populations at a CBRANCH
(state forking) drifts toward symbolic execution / abstract interpretation
done as masks over concrete states — file under "unthinkable until the
plain version works."

## Mapping onto shipped primitives (why this is not science fiction)

| OGAR Machine need | shipped today |
|---|---|
| 65,536 contexts as rows | the row store (64K × 512 B was the design point) |
| opcode classification → masks | `eq_u32`-family scans + `lgj_op_eq_classid` shape |
| population dispatch | mask algebra (`and`/`or`/`count`), masks-parent-on-rowstore |
| divergence mask | mask XOR (a small W-tier ABI addition when needed — goes through the wave process, never ad hoc) |
| survivors-only inspection | the whole thesis (PR #1's laziness/mask discipline) |
| time travel | Lance versioning (not yet wired here; lance-graph owns it) |

## Gate — what must exist before ANY OGAR Machine code

1. W3 (Java facade) and at least one W5 consumer example shipped, so the
consumer-never-grows-the-membrane rule has a proven workflow.
2. Ghidra archaeology: the real P-code op inventory + how a lifted
program image serializes (sized, not sketched).
3. **Probe P-M1** (the first falsifiable step, deliberately tiny): ONE
hand-written P-code program (a dozen ops, one CBRANCH), 64K input
worlds, executed by (a) a scalar reference interpreter and (b) the
population path over the row store — bit-identical final states
required. Two-sided divergence falsifier: plant one known behavioral
difference between two program variants and require the divergence
mask to find exactly that world set, no more, no fewer.