diff --git a/.claude/board/INTEGRATION_PLANS.md b/.claude/board/INTEGRATION_PLANS.md index 65043dd..806f532 100644 --- a/.claude/board/INTEGRATION_PLANS.md +++ b/.claude/board/INTEGRATION_PLANS.md @@ -1,3 +1,20 @@ +## 2026-08-17 — ogar-machine-v1 (EXPLORATORY; the population emulator) + +Plan: `.claude/plans/ogar-machine-v1.md`. Captured from the operator's +second archived ChatGPT context (the first was convergent confirmation; +this one is a genuinely NEW workload): one row = one machine STATE, +control flow becomes population masks ("which machines currently execute +ADD?"), Ghidra P-code as the normalized guest ISA +(`AdaWorldAPI/ghidra` now attached + shallow-cloned at +`/workspace/ghidra`), differential migration testing as the killer demo +(65,536 worlds through legacy XOR replacement → the divergent few), +Lance as the time machine (via `lance-graph-hydrate`, lance-graph +#957/#958 — inherited, never re-implemented). Strong/weak claims +separated in the plan; gated on W3 + one W5 example + Ghidra +archaeology + probe P-M1. + +**Status: NAMED, not scheduled.** W3 remains the next action. + ## 2026-08-17 — consumer-{world-trades,bricks-analytics,graph-traversal}-v1 (PLANS; the three W5 consumer examples) Plans: `.claude/plans/consumer-world-trades-v1.md`, diff --git a/.claude/board/PR_ARC_INVENTORY.md b/.claude/board/PR_ARC_INVENTORY.md index b6451d4..e4cb265 100644 --- a/.claude/board/PR_ARC_INVENTORY.md +++ b/.claude/board/PR_ARC_INVENTORY.md @@ -8,6 +8,35 @@ > anti-pattern the imported board rules name. Backfilled below in one > pass rather than left stale; PR #4 onward gets its entry at merge time. +## PR #5 — SoA row store: 512B rows, 32 facet lanes, ABI minor 2 (merged 2026-08-17, squash `78aa60e`) + +Companion: **AdaWorldAPI/ndarray#279** (W1), merged first — `lgj-abi`'s +`kernels.rs` calls `iter_u32x16` / `eq_u32_strided_to_mask` from it. + +- **Added:** `native/lgj-abi/src/rowstore.rs` (one `Arc<[u8]>`, two + readings, zero copies, normative SplitMix64 generator); + `LGJ_RESOURCE_ROWSTORE` + `lgj_rowstore_open` + `lgj_op_eq_classid` + + `lgj_row_facet_match`; `docs/abi.md` §11; the W1–W5 wave plan and three + consumer-example plans; `.claude/knowledge/soa-row-store-layout.md`. +- **Locked:** the 512 B / 32 × (4 B classid + 12 B payload) layout as + substrate truth (Java's view may differ); facet lanes ride the + **unchanged** `LgjLaneDesc`; masks parent onto row stores so the whole + existing mask algebra applies with no new surface; `byte_len` is the + exact covered span `(len-1)*stride + elem_bytes` (the old `len*stride` + form would have let Java bound a segment past the allocation's end); + ABI minor 1→2 and the §1/§7 symbol count corrected 14→18 (`nm -D`). + Doctrine: `E-LGJ-THE-MIDDLE-TIER-IS-DELETED-NOT-WRAPPED-1`; + self-correction: `E-LGJ-THE-FLAT-FIXTURE-WAS-SCAFFOLDING-NOT-THE-TARGET-1`. +- **Deferred:** `align(64)` base (stated honestly — arrives with real + `NodeRow`); payload semantics (a ClassView concern one layer up); fused + plans over facet lanes (W6, only if measurement asks). +- **Docs:** `abi.md` §11 + 4 plan files + 1 knowledge doc + full board. +- **Confidence:** High — 84/84, clippy/fmt clean, 18/18 symbols; both new + kernels parity-checked against independent scalar references over + 10 row counts × 2 seeds × 4 facets × 4 needles and cross-checked a third + way; two-sided payload-vs-classid falsifier. Both bot reviewers (cursor, + codex) hit usage limits and did not run. + ## PR #4 — Phase I synthesis docs + fusion re-run + board hygiene (merged 2026-08-17, squash `bd92c58`) - **Added:** `docs/{architecture,panama,valhalla-lab,execution-boundary}.md` diff --git a/.claude/knowledge/prior-art-and-the-layout-bridge-claim.md b/.claude/knowledge/prior-art-and-the-layout-bridge-claim.md new file mode 100644 index 0000000..36c771c --- /dev/null +++ b/.claude/knowledge/prior-art-and-the-layout-bridge-claim.md @@ -0,0 +1,109 @@ +# Prior art, and the claim that distinguishes this project + +> **READ BY:** anyone writing positioning/architecture prose, the consumer- +> example authors (W5), and any session tempted to cite the operator's +> archived ChatGPT discussion (2026-08-17, predates this repo's build) as +> design authority. That discussion is assessed HERE, once — its residual +> value extracted, its errors pinned — so it is not re-mined. + +## Assessment verdict + +The discussion **converged independently on the architecture this repo then +built and measured** — layout-centric bridge, engine-chooses-the-kernel, +"Java understands the layout, Rust owns the physics," a brutally small +vertical-slice MVP. Convergence from an independent derivation is mild +evidence the shape is right, and zero evidence about anything the +discussion *assumed but never measured*. Three parts remain valuable; two +parts are wrong relative to what this repo has since measured or mandates. + +## 1. The distinction that holds up: callability vs shared executable layout + +The competitive landscape, correctly characterized there and worth keeping: + +| bridge | what it is | what it solves | +|---|---|---| +| JNI / `jni-rs` | env pointers, object handles, marshaling ceremony | callability | +| `j4rs` | higher-level interop layer | callability | +| UniFFI | binding generator (Kotlin/Swift/Python official; Java external, FFM-based) | callability | +| Panama FFM (JEP 454, final since JDK 22) | the *mechanism* — `MemorySegment`/`MemoryLayout`/`Linker` | reachability, not semantics | + +All of these are **function bridges**: Java calls a native function, values +marshal across. This project's claim is a **layout bridge**: Java and Rust +*execute over the same bytes*, and the contract is the layout itself +(`docs/abi.md`), runtime-proven by the manifest cross-check. The +discussion's phrase for the moat — solving *"shared executable layout"* +where existing bridges solve *"callability"* — is the crispest one-line +positioning this project has and is worth using verbatim. + +## 2. Where the discussion's sketch was WEAKER than what was built + +Its `OgarAbiPage` sketch (`schema_key, row_count, stride, lane_count, +flags, data_ptr`) carries a bare `data_ptr` with **no lifetime story at +all** — no generation check, no epoch, no owner, no parent-liveness, no +close semantics. That is precisely the machinery this repo's registry +provides and **disable-verifies** (`abi.md` §4: no code path in which a +stale handle dereferences freed memory). A page descriptor without a +liveness protocol is a use-after-free with documentation. Keep this as the +standing answer to "couldn't the ABI just be one page struct?" + +## 3. The genuinely forward-looking extractable: the schema key as the join point + +> "One ABI key connects: Java MemoryLayout, Rust repr(C)/SoA, ClassView, +> ontology predicate, low-code block, Lance column projection, SIMD +> kernel." + +This is the OGAR classid doctrine — *classid is pure address; the magic is +what it resolves to* — arriving from an independent direction, which is +worth something. Concretely for this repo: today a resource names its +layout contract only *implicitly* (`kind == ROWSTORE` ⇒ the §11 geometry). +When the real ClassView slice lands, `LgjResourceInfo` (and/or +`LgjLaneDesc`) should gain an explicit **schema/classid field** so a +resource names *which* layout contract its bytes obey — additive, one +minor bump, and it is what makes "one key, many projections" literal at +the membrane. Filed as the W6 consideration in +`.claude/plans/lgj-soa-substrate-v1.md`. + +## 4. The claims discipline worth adopting verbatim + +The discussion's own guard against its headline numbers: never state +"2000×" as an engine claim. The architecture turns *pathological baselines* +(object-per-edge allocation, boxed values, string predicates, virtual +dispatch, JNI marshaling, JSON serialization, per-row Java loops) into +dense native kernels — so deltas are **baseline-dependent**: 10×, 100×, +1000×+ depending on how bad the baseline was. This repo's bench already +practices the stronger form (measure, publish the reproduction command, +state the noise floor); the W5 consumer examples MUST phrase any +comparison this way, and the graph-traversal plan already cites the +six-component BEFORE stack *as architecture, not as a benchmarked number*. + +## 5. Two corrections — pin these so the text is never cited naively + +1. **"Rust/RISC-speed execution through the membrane" assumes the native + side always wins. Measured false here.** Component C: the Java Vector + API, zero-copy on the *same* native segment, beats the native crossing + at **every** row count tested (64 → 4.2M) for a single predicate; the + crossing pays off for *composed, fused* work (Component E, 10.8–31.1×). + The real split is finer than the discussion's frame and lives in + `docs/execution-boundary.md`. Any prose inheriting the discussion's + framing must inherit the measurement instead. +2. **Its ndarray paragraph is about UPSTREAM crates.io ndarray** ("an + ergonomic n-dimensional array crate … `matrixmultiply` underneath, not + a SIMD layer"). The **AdaWorldAPI fork** this stack mandates is a + different artifact: `ndarray::simd` IS the SIMD polyfill layer + (dispatched AVX-512/AVX2/NEON/wasm/scalar under the W1a consumer + contract), and this project's §8 provenance rule depends on that. Do + not import the discussion's correction against the fork. (Its nightly + caveat about `std::simd` portable SIMD is true and is exactly why the + polyfill exists on stable.) +3. *(minor, was already true)* Java's Vector API is still incubating — + consistent with this repo's `--add-modules jdk.incubator.vector` and + noted in `jdk-toolchain-facts.md`. + +## The slogan shelf (used sparingly, they are earned now) + +- *"Java understands the layout. Rust owns the physics."* — matches + `architecture.md` §layers; safe to use. +- *"Panama gives Java fingers; this ABI gives it bones, tendons, and a + nervous system."* — rhetorically fine; keep it out of normative docs. +- *"Existing bridges solve callability; this solves shared executable + layout."* — the best of the three; use in positioning. diff --git a/.claude/plans/lgj-soa-substrate-v1.md b/.claude/plans/lgj-soa-substrate-v1.md index 6feed96..ea3b6af 100644 --- a/.claude/plans/lgj-soa-substrate-v1.md +++ b/.claude/plans/lgj-soa-substrate-v1.md @@ -32,11 +32,24 @@ | **W3** | Java `RowStore` facade: no FFM in public signatures; structured `MemoryLayout` (`sequence(32, struct(u32 classid, 12B payload))`); minor-≥2 gate; `FacetMatchView` zero-copy accessor over a Java-arena segment; `RowStoreParityTest` transcribing the generator | OPEN — next | | **W4** | Bench Component F: Java Vector API per-row facet scan (one `IntVector` 16-lane chunk = 4 facets, same algorithm as the Rust kernel) vs `lgj_row_facet_match` crossing vs scalar VarHandle walk — the "where does execution belong" question re-asked on the REAL layout | OPEN | | **W5** | The three consumer examples (own plan files, below) | PLANNED | +| **W6** *(named, not scheduled)* | ClassView wiring — and with it an explicit **schema/classid field** on `LgjResourceInfo`/`LgjLaneDesc` (additive, one minor bump), so a resource names WHICH layout contract its bytes obey instead of implying it via `kind`. Provenance + rationale: `.claude/knowledge/prior-art-and-the-layout-bridge-claim.md` §3 ("one key, many projections" made literal at the membrane). Also the `align(64)` base guarantee (real `NodeRow`) and, only if measurement asks, fused plans over facet lanes | NAMED | Wave rule (house style): one wave = one reviewable PR; gates run centrally (orchestrator only — agents never run cargo); every safety property lands disable-verified, every measured claim lands with its reproduction command. +**Cross-repo dependency note (2026-08-17, operator-flagged):** lance-graph +**#957** (merged) minted `crates/lance-graph-hydrate` — the generic +SoA→S3→volume→Lance hydration pattern (four-state lifecycle, +hydrate-aside/publish-by-rename, warm markers, dirty detection), minted in +lance-graph *specifically so consumers inherit it as a path/git dependency +rather than re-implement*. **#958** (open, another session's PR) is its 5+3 +council hardening fast-follow. Consequence here: when this substrate's +persistence slice arrives (the "Seal & Persist (Lance)" column of the +formula, and `ogar-machine-v1.md`'s time-machine storage), the hydration +path is `lance-graph-hydrate` — inherited, never re-derived. Do not design +a hydration mechanism in this repo. + ## What W2 locked (so W3+ doesn't re-derive it) - **Layout truth:** `ROW_BYTES=512`, `ROW_FACETS=32`, `FACET_BYTES=16`, diff --git a/.claude/plans/ogar-machine-v1.md b/.claude/plans/ogar-machine-v1.md new file mode 100644 index 0000000..1fb9519 --- /dev/null +++ b/.claude/plans/ogar-machine-v1.md @@ -0,0 +1,116 @@ +# ogar-machine-v1 — the population emulator (EXPLORATORY, named not scheduled) + +> **Status: NAMED** (2026-08-17). Source: the operator's second archived +> ChatGPT context, assessed here once so it is not re-mined. Unlike the +> first archived discussion (convergent confirmation — +> `.claude/knowledge/prior-art-and-the-layout-bridge-claim.md`), this one +> proposes a **genuinely new workload** for the shipped substrate, and the +> operator has attached `AdaWorldAPI/ghidra` to the session for it. +> Nothing here reorders the wave plan: W3 (Java `RowStore` facade) remains +> the next action; this plan is the shelf the idea sits on, with its +> strong/weak claims already separated. + +## The inversion, in one sentence + +A VM is `state + transition → new state`; instead of one machine executing +instructions sequentially, **one row is one machine STATE**, and the +substrate executes each *operation* across the whole population at once: + +``` +65,536 execution contexts (rows) + → classify current opcode (a classid-style scan → population masks) + → per-population bulk execution (SIMD ADD over the ADD mask, gathers + over the LOAD mask, mask updates for + BRANCH) + → next state +``` + +The VM never asks *"what does machine 17 execute?"* — it asks *"which +machines currently execute ADD?"* **Control flow becomes population +masks.** That is not an analogy to the shipped substrate; it is literally +its op set: rows, classid scans, mask algebra, masked bulk ops, survivors. + +## The claim discipline (the discussion's own, kept sharp) + +- **Weak claim, explicitly rejected**: "emulate a Pentium faster than + QEMU." One interactive instance is branchy and sequential; SIMD is not + fairy dust for `A then B then C`. Do not build toward this and do not + let a demo imply it. +- **Strong claim, the actual target**: *from Java, explore 65,536 + executions of an unsupported binary simultaneously, time-travel them, + XOR them against the replacement, and return only the worlds where + behavior diverges.* Every element of that sentence maps to something + this stack already does well (population masks / Lance versioning / + mask XOR / survivors-only inspection). + +## Ghidra's role — front-end compiler, never a peer emulator + +Do NOT implement x86/ARM/68k. Ghidra lifts the legacy binary **once** to +P-code (a small normalized op set: COPY, LOAD, STORE, BRANCH, CBRANCH, +CALL, RETURN, INT_ADD/SUB/MULT/AND/OR/XOR, …); the OGAR Machine executes +P-code only. `legacy.exe → Ghidra (once) → normalized program image → +population execution`. This is the same shape as the workspace's +ruff→OGAR harvest arms: an existing analyzer becomes the transcoder +front-end, and the substrate executes the normalized IR. First concrete +archaeology step in the fresh clone: locate the P-code opcode enum and +SLEIGH lifting surface, and size the *real* op set (the list above is the +discussion's sketch, not a verified inventory). + +## The four load-bearing design rules captured from the discussion + +1. **Differential migration testing is the killer demo.** Same 65,536 + input worlds through the legacy machine and the replacement; + `legacy_output[] XOR new_output[] → divergence_mask`; show the 15 of + 65,536 worlds the rewrite broke, click into one, see the concrete + state. (Note for a future warden pass: outcome-XOR here is a + *comparison*, not a state-transition kernel — it does not touch + lance-graph's `I-SUBSTRATE-MARKOV` XOR restriction, which governs + transition bundling.) +2. **Lance is the time machine.** Per-cycle sealed diffs (changed + registers / changed pages), `machine.at(cycle)`, `machine.diff(a, b)` — + deterministic rewind as a *consequence* of the substrate, matching the + workspace's episodic-=-Lance-versions doctrine. No bespoke snapshot + format. +3. **Memory purity resistance** (the discussion's own best guard): guest + RAM does NOT go into 512-byte rows. Semantic machine state (PC, regs, + flags, device state, identities, relations) → SoA lanes; the memory + image (pages) → a dense backing store the lens points at. This mirrors + the substrate's own key/value split — meaning and addressing in the + graph, bulk bytes dense and compressible. +4. **Semantic shims erode the emulator.** Recognize stable external + surfaces (USER/GDI/KERNEL/ODBC/filesystem/registry) and progressively + replace instruction execution with semantic operations — 90/10 → + 40/60 → 5/95. The endpoint is not `binary → reconstructed source` but + `binary → normalized behavior machine`, which is the OGAR transpile + doctrine's 85/15 split arrived at by *running* the program instead of + parsing it. + +Also named, further out: carrying BOTH branch populations at a CBRANCH +(state forking) drifts toward symbolic execution / abstract interpretation +done as masks over concrete states — file under "unthinkable until the +plain version works." + +## Mapping onto shipped primitives (why this is not science fiction) + +| OGAR Machine need | shipped today | +|---|---| +| 65,536 contexts as rows | the row store (64K × 512 B was the design point) | +| opcode classification → masks | `eq_u32`-family scans + `lgj_op_eq_classid` shape | +| population dispatch | mask algebra (`and`/`or`/`count`), masks-parent-on-rowstore | +| divergence mask | mask XOR (a small W-tier ABI addition when needed — goes through the wave process, never ad hoc) | +| survivors-only inspection | the whole thesis (PR #1's laziness/mask discipline) | +| time travel | Lance versioning (not yet wired here; lance-graph owns it) | + +## Gate — what must exist before ANY OGAR Machine code + +1. W3 (Java facade) and at least one W5 consumer example shipped, so the + consumer-never-grows-the-membrane rule has a proven workflow. +2. Ghidra archaeology: the real P-code op inventory + how a lifted + program image serializes (sized, not sketched). +3. **Probe P-M1** (the first falsifiable step, deliberately tiny): ONE + hand-written P-code program (a dozen ops, one CBRANCH), 64K input + worlds, executed by (a) a scalar reference interpreter and (b) the + population path over the row store — bit-identical final states + required. Two-sided divergence falsifier: plant one known behavioral + difference between two program variants and require the divergence + mask to find exactly that world set, no more, no fewer.