diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c4dfe9f..cd1922b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -38,6 +38,48 @@ jobs: cargo test --all --offline sudo iptables -F OUTPUT || true + # Every platform a release ships a binary for is tested here. + # + # `check` above stays on Linux and owns the things that do not vary by platform — + # formatting, clippy, and the network-egress gate, which needs iptables. This job + # owns what does vary: path handling, the temporary directory, line endings, and + # whether the CLI actually runs. Releases shipped Windows and macOS binaries that no + # job had ever executed; a portability break would have reached users first. + platform: + name: test (${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + os: [windows-latest, macos-latest] + steps: + # Before checkout, deliberately: Windows runners convert LF to CRLF on the way + # in, which would mean testing fixtures no user's repository actually contains + # and shifting every byte offset the index records. Configuring git afterwards + # would be too late. + - name: Check out with the line endings as committed + if: runner.os == 'Windows' + run: git config --global core.autocrlf false + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + + - name: Test + run: cargo test --all + + # The unit tests can pass while the binary cannot start. This is the same + # end-to-end path `bench-smoke` runs on Linux, reduced to what proves the CLI + # works on this platform at all. + - name: The CLI runs end to end + # bash, not the Windows default: PowerShell only propagates the *last* + # command's exit code, so a failing `init` here would pass the step silently. + shell: bash + run: | + set -euo pipefail + cargo run --release -p reify-cli -- -C fixtures/minierp init + cargo run --release -p reify-cli -- -C fixtures/minierp index + cargo run --release -p reify-cli -- -C fixtures/minierp --json context "strategic account discount" + deny: runs-on: ubuntu-latest steps: diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index d9e2831..cadba9c 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -33,10 +33,13 @@ jobs: fail-fast: false matrix: include: - - { os: macos-latest, target: aarch64-apple-darwin } - - { os: macos-latest, target: x86_64-apple-darwin } - - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu } - - { os: ubuntu-latest, target: aarch64-unknown-linux-gnu } + - { os: macos-latest, target: aarch64-apple-darwin } + - { os: macos-latest, target: x86_64-apple-darwin } + - { os: ubuntu-latest, target: x86_64-unknown-linux-gnu } + - { os: ubuntu-latest, target: aarch64-unknown-linux-gnu } + # Mature business systems are disproportionately maintained on Windows, + # which is exactly the codebase this tool is for. + - { os: windows-latest, target: x86_64-pc-windows-msvc, ext: .exe } runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 @@ -59,7 +62,7 @@ jobs: run: | NAME="reify-${{ github.ref_name || inputs.tag }}-${{ matrix.target }}" mkdir -p "dist/$NAME" - cp "target/${{ matrix.target }}/release/reify" "dist/$NAME/" + cp "target/${{ matrix.target }}/release/reify${{ matrix.ext }}" "dist/$NAME/" cp README.md LICENSE "dist/$NAME/" tar -C dist -czf "dist/$NAME.tar.gz" "$NAME" shasum -a 256 "dist/$NAME.tar.gz" > "dist/$NAME.tar.gz.sha256" 2>/dev/null \ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..9094603 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,39 @@ +# Project agent memory + +This file is the project's committed home for project-intrinsic agent knowledge: build, test, release, architecture, and sharp-edge notes that should travel with the code. + +## Checks + +`cargo fmt --all`, `cargo clippy --workspace --all-targets` (CI treats warnings as +errors) and `cargo test --workspace` must all be clean. CI additionally runs the test +suite with network egress blocked — `crates/reify/tests/offline.rs` fails the build if a +networking crate enters the dependency tree. + +## The standard this repository is held to + +`crates/reify-bench` is the most load-bearing thing here, and its value is its +intellectual honesty rather than its numbers: steel-manned baselines, falsification +conditions written down *before* the run, Wilson intervals reported next to the +admission that they overlap, provider failures excluded rather than scored as misses, +and one rule from `metrics.rs` — *a metric that cannot be defined precisely does not get +reported*. Metric definitions live in `docs/metrics.md`. Reports are **generated**, never +hand-edited; a number that was not re-measured is corrected with a dated note rather than +silently regenerated (see the top of `benchmarks/REPORT-medusa.md`). + +When a fitted parameter fails held-out validation, the fit is published and the default +reverts — `HISTORY_PRIOR_WEIGHT` in `crates/reify/src/context.rs` is the worked example. + +## Decisions already measured + +`reify verify` — a post-flight check reporting what an agent's patch missed — was +measured before being written and **failed** its pre-registered condition on Rust, Python +and Go. `benchmarks/REPORT-verify.md` has the numbers; `reify-bench verify-eval` +reproduces them in about two minutes with no model. Do not build it on the `CALLS` graph +alone without re-running that benchmark and beating those numbers. + +## Maintaining this file + +Keep this file for knowledge useful to almost every future agent session in this project. +Do not repeat what the codebase already shows; point to the authoritative file or command instead. +Prefer rewriting or pruning existing entries over appending new ones. +When updating this file, preserve this bar for all agents and keep entries concise. diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d88967..ba82bd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,137 @@ Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). This project adheres to semantic versioning once it reaches 1.0; before then, minor versions may break the store schema, and `reify index --force` rebuilds it. +## [0.2.3] - 2026-08-25 + +### Fixed +- **The index lock did nothing on Windows.** `process_is_alive` was implemented for + unix and stubbed to `false` everywhere else. For the *current* process that means + every lock looks stale, so the lock reclaims itself and stops excluding anything: + two `reify index` runs could proceed against the same store. Windows users of 0.2.2 + have this; there is no configuration that avoids it, and upgrading is the fix. + Implemented with `OpenProcess` + `WaitForSingleObject`, declared by hand for one + question asked once, the same way `kill` already was. The remaining + `not(any(unix, windows))` arm now returns `true`: without a liveness check the lock + cannot be trusted, and refusing to reclaim a lock costs one user less than letting + two indexers share a store costs everyone. +- **One file could take the whole answer.** Relevance spreads along edges, so every + member of a file that matched the task loosely arrived holding a plausible score and + nothing bounded how many of them were admitted. Measured on Medusa, a task about + applying a discount twice filled 13 of 20 slots with one HTTP router and one + arithmetic helper, and the promotion service that actually had to change ranked + eighteenth. No file may now claim more than four symbol slots. +- **The reading plan collapsed onto a single file.** `next_reads` drained each ranked + file in turn, so the top file consumed every slot: two of three sampled Medusa tasks + produced six entries naming one file. Spans are now drawn a file at a time in rounds, + so every ranked file earns one before any earns a second. +- Tests, fixtures and mocks outranked the code they exercise. A test names the domain + vocabulary as densely as its implementation and a reader cannot edit it to change + behaviour; on Medusa a promotion spec *and* its fixture both outranked the promotion + service. They now give up half their score — a penalty, not an exclusion, because + "fix the failing test for X" is a real task. Tunable as `test_path_penalty`. +- `install.sh` did not verify the checksum it advertised. The `curl | sh` path now + checks the published SHA-256 before unpacking, which is what `reify upgrade` already + did; a mismatch installs nothing. +- The README shipped its quickstart, language switcher and table of contents twice. +- The Roadmap and Status sections are gone from all three READMEs. Both were prose that + needed hand-editing every release to stay true, and the Status section had already + drifted — it still quoted `reify why` at 205 ms three versions after the measurement + changed. What they said that is durable lives in the benchmark reports and this file. +- `Cargo.toml` pointed `documentation` at `docs.rs/reify`, which belongs to an + unrelated crate published in 2021. + +### Added +- **Windows and macOS are tested in CI.** Releases shipped binaries for both and no job + had ever executed either, so a portability break would have reached users before it + reached us. A `platform` job now runs the full suite and an end-to-end CLI smoke on + `windows-latest` and `macos-latest`; `check` stays on Linux and keeps what does not + vary by platform — formatting, clippy, and the network-egress gate that needs + iptables. It found the lock bug above in its first run, by failing to recognise its + own process as alive. Two details are handled rather than discovered later: Windows + checkouts convert LF to CRLF, so `core.autocrlf false` is set *before* checkout; and + the smoke step runs under bash, because PowerShell propagates only the last command's + exit code and a failing `init` would otherwise leave the step green. +- **Windows x86_64 binaries.** `reify upgrade` works there too: Windows keeps a handle + on the running image, so the old binary is renamed aside rather than overwritten, and + restored if the replacement fails. `install.sh` recognises Git Bash, MSYS2 and Cygwin. +- Three more MCP tools — `reify_explain`, `reify_flow`, `reify_conflicts` — because + those are the capabilities no other retriever offers and MCP is how most clients + reach Reify at all. All six schemas still cost under the 600-token ceiling the + original three were held to, which is still asserted by a test. `reify_preflight` was + considered and left out: it answers the same question as `reify_why` for an agent. + +### Changed +- Retrieval measured before and after on all three brownfield repositories, same + machine, same task sets. **Hit rate improved in every repository and both + conditions**; grep baselines are byte-identical, which is the control. + + | three rounds | before | after | + |---|--:|--:| + | medusa (TypeScript, n=40) | 27.5% | **42.5%** | + | openmrs (Java, n=22) | 54.5% | **59.1%** | + | ofbiz (Java, n=40) | 77.5% | **85.0%** | + + Stated plainly because it is a trade: mean reciprocal rank rose on medusa + (0.097 → 0.136) and fell on openmrs (0.278 → 0.208) and ofbiz (0.461 → 0.392). + Spreading the top slots across more files means that when the right file was already + ranked first it now shares the window. That is the intended direction — a file never + offered cannot be used, while a file at rank six is still in the context — and it is + the failure the end-to-end SWE-bench run diagnosed: retrieval was already ahead and + the model still did no better, because of what the window contained and in what + order. +- **SWE-bench Verified re-run in full, all 500 instances**, and every published + retrieval figure moved up. Both grep baselines came back byte-identical to the + previous run, which is the control. + + | retrieval, n=500 | before | after | + |---|--:|--:| + | reify, one round | 66.0% | **72.6%** | + | reify, three rounds | 84.6% | **87.0%** | + | one round, offered *every* touched file | 59.0% | **65.4%** | + | three rounds, offered *every* touched file | 77.0% | **81.4%** | + + Paired against content-grep, one round now wins 342 and loses 12 (was 310 / 13); + three rounds win 406 and lose 4 (was 395 / 5), exact McNemar p ≈ 9 × 10⁻¹¹⁵. Six of + twelve repositories improved and none regressed — the largest are pytest 84% → 95%, + matplotlib 91% → 97% and sphinx 75% → 80%. MRR moved the other way, 0.45 → 0.43 at + three rounds, for the reason given above. Median tokens rose from 3,466 to 3,670 at + one round, still below content-grep's 3,998. + +- **Stage 2, the end-to-end result, re-run and now ahead** — 101 instances graded under + both arms by the official SWE-bench harness, up from 63. + + | resolved | | | + |---|--:|---| + | BM25 | 67.3% | 68 resolved, 1 empty patch | + | **Reify** | **73.3%** | 74 resolved, 2 empty patches | + | | | paired 12–6, exact McNemar **p = 0.24** | + + Reify resolves six more issues and wins twice as many disagreements as it loses, but + at this sample size that is **not statistically significant**: the honest reading is + *ahead and not yet proven*, and the README says so before it says the percentages. + The previous run was a tie, and the one before that a loss. + + These absolute rates are **not** comparable to the 23.8% published earlier. That run + used DeepSeek, whose account ran out of balance mid-project; this one uses Claude + Sonnet, and a stronger model lifts both arms. What survives a model change is the + paired comparison, because both arms always answer the same instance with the same + model. Per-instance outcomes are committed in + `benchmarks/swe/results/stage2-endtoend.json`. + +### Fixed (reproduction) +- The committed stage-2 driver was **not the script that produced the published + numbers**. `benchmarks/swe/stage2c.py` fed the model whole files; the run behind the + figures used `reify context --for-edit` regions, which is the entire reason the + earlier loss became a tie. Anyone following the repository's own instructions would + have failed to reproduce its own results. Replaced by `benchmarks/swe/stage2.py`, + which is the driver that produced the numbers above, with the model as a parameter. +- An epistemic status shared by every row of a section is stated once on the section + heading instead of repeated down the page. Symbols are `CONFIRMED` by construction, + so `[confirmed]` appeared on all twenty code rows of every answer, where it carried + no information and invited the wrong reading — it attests that a symbol was parsed + from source, never that it is the right place to change. A section with mixed + statuses still badges every row, and machine-readable output is unchanged. + ## [0.2.2] - 2026-08-22 ### Fixed diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..a9d4d26 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,2 @@ + +@AGENTS.md diff --git a/Cargo.lock b/Cargo.lock index cdeb299..77706d8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -780,7 +780,7 @@ checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" [[package]] name = "reify" -version = "0.2.2" +version = "0.2.3" dependencies = [ "anyhow", "blake3", @@ -813,7 +813,7 @@ dependencies = [ [[package]] name = "reify-bench" -version = "0.2.2" +version = "0.2.3" dependencies = [ "anyhow", "clap", @@ -826,7 +826,7 @@ dependencies = [ [[package]] name = "reify-cli" -version = "0.2.2" +version = "0.2.3" dependencies = [ "anyhow", "clap", diff --git a/Cargo.toml b/Cargo.toml index d5e19a3..6bf2c54 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -3,13 +3,15 @@ resolver = "2" members = ["crates/reify", "crates/reify-cli", "crates/reify-bench"] [workspace.package] -version = "0.2.2" +version = "0.2.3" edition = "2021" license = "Apache-2.0" description = "A local knowledge engine that gives AI coding agents the smallest context they need to change mature business systems correctly." repository = "https://github.com/lambiengcode/reify" -homepage = "https://github.com/lambiengcode/reify" -documentation = "https://docs.rs/reify" +homepage = "https://lambiengcode.github.io/reify/" +# Not docs.rs/reify: the `reify` name on crates.io belongs to an unrelated crate +# published in 2021, so that URL sends readers to somebody else's software. +documentation = "https://lambiengcode.github.io/reify/docs.html" readme = "README.md" keywords = ["ai", "agents", "code-search", "knowledge-graph", "codebase"] categories = ["development-tools", "command-line-utilities"] diff --git a/README.md b/README.md index eb74d70..7fa0459 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,9 @@ Release Documentation Apache-2.0 - SWE-bench retrieval 84.6% + SWE-bench retrieval 87.0% network calls: 0 + platforms: macOS, Linux, Windows

@@ -27,12 +28,12 @@ Cursor Codex OpenCode - MCP + MCP

- On SWE-bench Verified, Reify puts the file that had to change in front of the model 84.6% of the time — grep manages 6.6% · 500 real issues, someone else's benchmark · never opens a socket
- A real model on 142 tasks from real merged commits across ERPNext, OFBiz, OpenMRS and Medusa, each index built at a commit before those changes existed. That is retrieval: the right file, in front of the model. On end-to-end patch correctness a BM25 baseline currently resolves more issues than Reify does, and the section saying so is as prominent as this one. Full writeup · reproduce it. + On SWE-bench Verified, Reify puts the file that had to change in front of the model 87.0% of the time — grep manages 6.6% · 500 real issues, someone else's benchmark · never opens a socket
+ A real model on 142 tasks from real merged commits across ERPNext, OFBiz, OpenMRS and Medusa, each index built at a commit before those changes existed. That is retrieval: the right file, in front of the model. On end-to-end patch correctness Reify is ahead of a BM25 baseline but not significantly so, and the section saying so is as prominent as this one. Full writeup · reproduce it.

@@ -48,15 +49,16 @@ ```bash curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh cd your-repository -reify init --write-agent-instructions # wires your agent through AGENTS.md / CLAUDE.md -reify index # 4.6 s for 5,000 files; 0.7 s after one edit +reify doctor # should you even use this? it will say no +reify install --yes # detects your agents and wires each one +reify index # 4.2 s for 5,000 files; 0.5 s after one edit reify context "the change you are about to make" --toon ``` One static binary — no daemon, no config, no API key, and every release ships a -SHA-256 checksum that `reify upgrade` verifies before installing. Changed your mind? -`reify uninstall` removes the binary and `reify uninit` cleans one repository, both -showing their plan first. Per-agent wiring, hooks and MCP: Install. +SHA-256 checksum that both the installer above and `reify upgrade` verify before +anything is unpacked. Per-agent wiring, hooks, MCP and how to leave cleanly: +Install.

English · Tiếng Việt · 简体中文 @@ -67,39 +69,10 @@ showing their plan first. Per-agent wiring, hooks and MCP: In **Contents** - [Two minutes to first answer](#two-minutes-to-first-answer) · [the one-person problem](#the-one-person-problem) · [what it gives you](#what-it-actually-gives-you) · [before / after](#before--after) -- **Numbers:** [SWE-bench Verified](#swe-bench-verified) · [end to end](#end-to-end-a-tie-and-what-it-took-to-get-there) · [four repositories](#four-repositories-chosen-to-hurt) · [where it doesn't work](#where-it-doesnt-work) +- **Numbers:** [SWE-bench Verified](#swe-bench-verified) · [end to end](#end-to-end-ahead-but-not-yet-proven) · [four repositories](#four-repositories-chosen-to-hurt) · [where it doesn't work](#where-it-doesnt-work) - **Using it:** [install](#install) · [wire it into your agent](#wire-it-into-your-agent) · [commands](#commands) · [privacy](#privacy) - **Under it:** [how it works](#how-it-works) · [what it reads](#what-it-reads) · [multilingual](#multilingual) · [architecture](#architecture) · [reproducing the benchmark](#reproducing-the-benchmark) -- [FAQ](#faq) · [development](#development) · [roadmap](#roadmap) · [status](#status) · [license](#license) - -## Two minutes to first answer - -```bash -curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh -cd your-repository -reify init --write-agent-instructions # wires your agent through AGENTS.md / CLAUDE.md -reify index # 4.6 s for 5,000 files; 0.7 s after one edit -reify context "the change you are about to make" --toon -``` - -One static binary — no daemon, no config, no API key, and every release ships a -SHA-256 checksum that `reify upgrade` verifies before installing. Changed your mind? -`reify uninstall` removes the binary and `reify uninit` cleans one repository, both -showing their plan first. Per-agent wiring, hooks and MCP: Install. - -

- English · Tiếng Việt · 简体中文 -

- ---- - -**Contents** - -- [Two minutes to first answer](#two-minutes-to-first-answer) · [the one-person problem](#the-one-person-problem) · [what it gives you](#what-it-actually-gives-you) · [before / after](#before--after) -- **Numbers:** [SWE-bench Verified](#swe-bench-verified) · [end to end](#end-to-end-a-tie-and-what-it-took-to-get-there) · [four repositories](#four-repositories-chosen-to-hurt) · [where it doesn't work](#where-it-doesnt-work) -- **Using it:** [install](#install) · [wire it into your agent](#wire-it-into-your-agent) · [commands](#commands) · [privacy](#privacy) -- **Under it:** [how it works](#how-it-works) · [what it reads](#what-it-reads) · [multilingual](#multilingual) · [architecture](#architecture) · [reproducing the benchmark](#reproducing-the-benchmark) -- [FAQ](#faq) · [development](#development) · [roadmap](#roadmap) · [status](#status) · [license](#license) +- [FAQ](#faq) · [development](#development) · [license](#license) ## The one-person problem @@ -182,12 +155,12 @@ the right answer is the set of files the accepted fix actually touched. |---|--:|--:|--:|--:| | grep, content | 6.6% [4.7–9.1] | 0.06 | 5.6% | 3,998 | | grep, paths | 9.0% [6.8–11.8] | 0.06 | 7.8% | 3,996 | -| **reify**, one round | **66.0%** [61.7–70.0] | 0.43 | 59.0% | **3,466** | -| **reify**, three rounds | **84.6%** [81.2–87.5] | 0.45 | 77.0% | 9,174 | +| **reify**, one round | **72.6%** [68.5–76.3] | 0.42 | 65.4% | **3,670** | +| **reify**, three rounds | **87.0%** [83.8–89.7] | 0.43 | 81.4% | 9,549 | -**A single round of Reify beats grep on 310 instances and loses on 13 — while spending -fewer tokens** (3,466 against 3,998). Three rounds win 395 to 5 (exact McNemar -p ≈ 7 × 10⁻¹¹⁰). This is not a close measurement, and it is the cleanest number in this +**A single round of Reify beats grep on 342 instances and loses on 12 — while spending +fewer tokens** (3,670 against 3,998). Three rounds win 406 to 4 (exact McNemar +p ≈ 9 × 10⁻¹¹⁵). This is not a close measurement, and it is the cleanest number in this README precisely because the tasks, the repositories and the ground truth all came from somewhere else. @@ -195,10 +168,10 @@ Per repository, three rounds against content-grep: | | grep | reify ×3 | | | grep | reify ×3 | |---|--:|--:|---|---|--:|--:| -| django (n=231) | 6% | **88%** | | astropy (n=22) | 0% | **77%** | -| sympy (n=75) | 7% | **77%** | | xarray (n=22) | 9% | **91%** | -| sphinx (n=44) | 7% | **75%** | | pytest (n=19) | 26% | **84%** | -| matplotlib (n=34) | 0% | **91%** | | pylint (n=10) | 10% | **60%** | +| django (n=231) | 6% | **90%** | | astropy (n=22) | 0% | **77%** | +| sympy (n=75) | 7% | **79%** | | xarray (n=22) | 9% | **95%** | +| sphinx (n=44) | 7% | **80%** | | pytest (n=19) | 26% | **95%** | +| matplotlib (n=34) | 0% | **97%** | | pylint (n=10) | 10% | **60%** | | scikit-learn (n=32) | 9% | **88%** | | requests (n=8) | 0% | **100%** | **What this does and does not show.** It measures retrieval — whether the files that @@ -210,25 +183,39 @@ retriever offers, and every arm here ran against the same index at the same comm Reproduce it with the driver in [`benchmarks/swe/`](benchmarks/swe/). -### End to end: a tie, and what it took to get there +### End to end: ahead, but not yet proven Retrieval is not the final claim — resolving the issue is. The same benchmark, run through the SWE-bench paper's own protocol (one model, one budget, the retriever as the only difference), with every patch judged by the **official SWE-bench harness**: -| resolved the issue, 63 instances graded under both arms | | | +| resolved the issue, 101 instances graded under both arms | | | |---|--:|---| -| BM25 | 23.8% | | -| **Reify** | **23.8%** | paired 6–6, p = 1.0 | - -A tie — and worth stating plainly, because the first attempt was a **loss**: 11.1% -against 18.1%. What closed it is the interesting part. - -Reify retrieved the right file far more often (77% against BM25's 60%) and the model -still did worse. Rebuilding the exact prompts showed why: a context window filled with -whole files in rank order spends itself on whatever ranks first, and Reify's ranking is -blind to file size where BM25's has length normalisation built in. **The gold file was -retrieved and then never shown** — visible in only 27% of prompts against BM25's 40%. +| BM25 | 67.3% | 68 resolved, 1 empty patch | +| **Reify** | **73.3%** | 74 resolved, 2 empty patches | +| | | paired 12–6, exact McNemar **p = 0.24** | + +**Read that p-value before the percentages.** Reify resolved six more issues and won +twice as many disagreements as it lost — but at this sample size that is not a +statistically significant result. The honest summary is *ahead and not yet proven*, not +a win. Two hundred more instances would settle it; today's evidence supports "Reify does +not cost you anything end to end, and probably helps", nothing stronger. + +**These absolute rates are not comparable to earlier ones published here.** That run +used DeepSeek; this one uses Claude Sonnet, because the DeepSeek account ran out of +balance mid-project. A stronger model lifts both arms — the earlier run resolved about +24% on each side. What survives a model change is the *paired* comparison, because both +arms always answer the same instance with the same model, and that is what the table +above reports. Raw per-instance outcomes: [`benchmarks/swe/results/stage2-endtoend.json`](benchmarks/swe/results/stage2-endtoend.json). + +It has not always been ahead. The first attempt was a **loss**, and what closed the gap +is the interesting part. + +Reify retrieved the right file far more often and the model still did worse. Rebuilding +the exact prompts showed why: a context window filled with whole files in rank order +spends itself on whatever ranks first, and file-rank order is blind to file size where +BM25 has length normalisation built in. **The gold file was retrieved and then never +shown** — visible in only 27% of prompts against BM25's 40%. `reify context --for-edit` fixes that at the source: regions padded to whole definitions, the file's imports included once, overlapping regions merged, budget still @@ -240,14 +227,18 @@ hard. Nothing retrieved is lost at the window any more: | Reify, whole files | 76.7% | 26.7% | | **Reify `--for-edit`** | **80.0%** | **56.7%** | -Two fixes were tried and **rejected on evidence**: a per-file cap made visibility worse -(a truncated file is not editable either), and cost-aware ranking cut retrieval by seven -points while buying nothing once regions made file size irrelevant. +Capping how much of the window a single file may claim was tried early, **rejected on +evidence**, and only later adopted in the form that works. Truncating a file's *content* +made things worse, because a truncated file is not editable either. Capping how many +*symbols* one file contributes to the selection — upstream of the window, leaving each +region whole — is what improved retrieval, and it is what ships today. Cost-aware +ranking was tried and rejected outright: it cut retrieval by seven points while buying +nothing once regions made file size irrelevant. -So: Reify wins retrieval decisively and ties on final patch success. The remaining -constraint is the patch-writing loop, not the context — both arms cap near 24%, and -about a fifth of attempts produce no usable edit at all. Beating BM25 *significantly* on -resolution would need Reify to win two of every three disagreements; today it wins half. +So: Reify wins retrieval decisively and is ahead, inconclusively, on final patch +success. The remaining constraint is the patch-writing loop rather than the context — +both arms leave roughly a quarter of issues unresolved with the file that had to change +sitting in the prompt. ## Four repositories chosen to hurt @@ -392,27 +383,41 @@ Three things that only break once you leave Latin script, each of which broke he curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh ``` -Prebuilt binaries for macOS (Apple Silicon and Intel) and Linux (x86_64 and aarch64). +Prebuilt binaries for **macOS** (Apple Silicon and Intel), **Linux** (x86_64 and +aarch64) and **Windows** (x86_64). On Windows the line above works as written in Git +Bash, MSYS2 or WSL; from PowerShell, take the `x86_64-pc-windows-msvc` archive from +[the latest release](https://github.com/lambiengcode/reify/releases/latest), verify its +`.sha256`, and put `reify.exe` somewhere on your `PATH`. + +Every one of those platforms runs the full test suite in CI, and the CLI is exercised +end to end on each — a binary is not published for a platform nothing has executed. + Or build from source: ```bash cargo install --path crates/reify-cli ``` -Then, in any repository: +Then, in the repository you want it for: ```bash -reify init # tells you what it will and won't index, and why -reify index # 4.6s for 5,000 files; 0.7s after you edit one +reify doctor # is this repository one Reify helps? it is willing to say no +reify install # shows what it found and what it would wire; --yes applies it ``` +`install` detects the agents actually configured in the repository — `AGENTS.md`, +`CLAUDE.md`, `.cursor/`, `.clinerules/` and the rest — and wires each one the +[cheapest way that works](#wire-it-into-your-agent). An agent installed on the machine +but not configured here is reported, not written to, and nothing outside the repository +is touched. `--mcp` opts into MCP instead, and says what that costs before it writes. + **Stay current, leave cleanly.** `reify upgrade` replaces the binary with the latest release — through `curl` and `tar` as visible subprocesses, never an embedded HTTP client, with the checksum verified before anything is installed; `--check` only asks, and `REIFY_OFFLINE=1` refuses the command outright. `reify uninstall --yes` removes the binary and nothing else; `reify uninit --yes` removes one repository's `.reify/` store -and the instruction block `init` wrote. Both show their plan first when run without -`--yes`. +and every agent integration `init` or `install` wrote. Both show their plan first when +run without `--yes`.
Shell completions @@ -431,33 +436,34 @@ reify completions fish > ~/.config/fish/completions/reify.fish reify init --write-agent-instructions ``` -Appends a six-line block to `AGENTS.md` or `CLAUDE.md`. No protocol, no server, no -per-turn schema tax — this is the level the benchmark measured. For tools that read a -different file (`.cursorrules`, `CONVENTIONS.md`, `.windsurfrules`, `.clinerules/`), -paste the same four lines: +Appends this block to `AGENTS.md` or `CLAUDE.md`. No protocol, no server, no per-turn +schema tax — this is the level the benchmark measured. For tools that read a different +file (`.cursorrules`, `CONVENTIONS.md`, `.windsurfrules`, `.clinerules/`), paste it +there instead: ```markdown -Before changing code here, run `reify context "" --toon`. +## Before changing code in this repository + +Run `reify context "" --toon` and read its output first. Run `reify why :` before modifying unfamiliar logic. Run `reify impact ""` before changing anything shared. -Treat INFERRED claims as leads to verify, not facts. + +Claims marked `INFERRED` are leads to verify against their citation, not facts. +If `conflicts` is non-empty, resolve the disagreement before changing behaviour. ``` -**MCP**, if you prefer it: `reify serve --mcp` exposes three tools — `reify_context`, -`reify_why`, `reify_impact` — and three is the whole surface. A server's schemas are -re-sent every turn of every session, so a tool built to save context should not charge -rent to deliver it; a test asserts they cost under 600 tokens. +**MCP**, if you prefer it: `reify serve --mcp` exposes six tools — `reify_context`, +`reify_why`, `reify_impact`, `reify_explain`, `reify_flow` and `reify_conflicts` — and +six is the whole surface. A server's schemas are re-sent every turn of every session, +so a tool built to save context should not charge rent to deliver it; a test asserts +they cost under 600 tokens, which six still fit inside. **A model is optional and off** until you name a command in `.reify/llm.toml` (`command = ["ollama", "run", "llama3"]`). Reify writes the prompt to its stdin. See [Privacy](#privacy) for why that is a command and not an HTTP client.
-Shell completions, and a pre-edit risk hook - -```bash -reify completions zsh > ~/.zfunc/_reify # also bash, fish -``` +A pre-edit risk hook, and keeping the index fresh Inject a risk header before every edit, under 300 tokens, asserted by a test because it runs on every edit. Non-blocking by default: a hook that blocks edits gets uninstalled, @@ -489,7 +495,7 @@ chmod +x .git/hooks/post-merge && cp .git/hooks/post-merge .git/hooks/post-check |---|---| | `reify context ""` | The minimum knowledge for a change, plus a reading plan. **The one that matters.** `--toon` emits the agent format | | `reify why :` | What this is, what calls it, what data it touches, what changed it | -| `reify impact ""` | What depends on it — including through the database, where no call edge exists | +| `reify impact ""` | What depends on it — callers, importers, and coupling through the database where no call edge exists | | `reify explain ""` | A business concept across every language, table and file it appears in | | `reify flow ""` | The call sequence that carries out a business process | | `reify conflicts` | Documentation that disagrees with the code | @@ -498,9 +504,11 @@ chmod +x .git/hooks/post-merge && cp .git/hooks/post-merge .git/hooks/post-check | `reify preflight ` | A risk header for an editor hook | | `reify report` | System scorecard | | `reify status` | Freshness, coverage, and what was skipped | +| `reify doctor` | Should this repository use Reify at all? Runs before there is an index, and will say no | +| `reify install [--yes]` | Detect the agents configured here and wire each one. Shows its plan first; `--mcp` opts into MCP instead | | `reify llm status \| preview` | Is a model configured, and exactly what would be sent | | `reify upgrade [--check]` | Replace this binary with the latest release. The only networked command; refused under `REIFY_OFFLINE=1` | -| `reify uninstall --yes` \| `uninit --yes` | Remove the binary \| one repository's store and instruction block | +| `reify uninstall --yes` \| `uninit --yes` | Remove the binary \| one repository's store and everything `install` wrote | | `reify serve --mcp` | Model Context Protocol over stdio | | `reify completions ` | Completion script | @@ -565,32 +573,19 @@ ERPNext, 5,064 files, 8-core M-series laptop. | peak memory, full index | 224 MB | | store size | 47 MB (33% of a 144 MB working tree) | -A full index took **78 seconds** until the full-text index was keyed by node id. `uid` is `UNINDEXED` in FTS5, so `DELETE ... WHERE uid = ?` scanned the whole table once per node — quadratic, and invisible until it was timed per stage. Editing one file took **5.9 seconds** until the repository-wide stages learned to skip when their inputs are provably unchanged. - -Reindexing was **2× slower** until two things stopped being done repository-wide for -a one-line edit. Discovery read and hashed all 5,285 files on every run — 222 ms of -reading to find the handful that moved — and now `stat`s past anything whose size and -modification time are unchanged, hashing the rest across all cores. Reference -resolution reloaded and re-resolved all **144,309** references, 167 ms to resolve and -145 ms to commit, regardless of how little changed; it now re-resolves only references -whose *name* the edit added or removed, plus those inside the edited files, which is -provably the whole affected set. Measured against the previous build on the same -machine: full index 6.75 s → 4.25 s, no-op reindex 256 ms → 101 ms, one file edited -974 ms → 486 ms. - -`reify why` was **1.5 seconds** on a blobless clone, and returned a *worse* answer than -it does now. `git log -L` needs the file's blob at every revision it walks, and on a -partial clone those blobs are not local — so git was silently fetching them from the -remote, one query costing 29.5 s of network and 0.07 s of work. The subprocess now runs -with `GIT_NO_LAZY_FETCH=1`: git answers from local objects or fails, and either way the -command returns in milliseconds. Eleven of twelve sampled symbols used to hit the -timeout; none do. - -That fix is also why the privacy claim below is true of the whole process tree rather -than just this binary. Reify never opened a socket; the git it spawned did. - -`REIFY_TIMING=1 reify index` prints the per-stage breakdown that found every one of -these. +Those numbers are the end of a long optimisation, not a first draft: a full index took +**78 seconds** and a one-file reindex **5.9 seconds** before the stages learned to skip +work whose inputs are provably unchanged. `REIFY_TIMING=1 reify index` prints the +per-stage breakdown that found every one of them, and [CHANGELOG.md](CHANGELOG.md) has +the before-and-after for each. + +One of those fixes is load-bearing for the privacy claim below rather than for speed. +`reify why` runs `git log -L`, which needs the file's blob at every revision it walks — +and on a blobless clone those blobs are not local, so git was silently **fetching them +from the remote**: 29.5 s of network for 0.07 s of work. Every `git` invocation now sets +`GIT_NO_LAZY_FETCH=1`, so it answers from local objects or fails. Reify never opened a +socket; the git it spawned did. That is why the guarantee below is stated over the whole +process tree and not just this binary. ## Reproducing the benchmark @@ -616,6 +611,25 @@ reify-bench chart --results "Mine=results/" --out assets/ The task set is frozen before any condition runs. The report includes a **"Where Reify lost"** section listing every task the baseline won, and it is a required part of the document rather than an optional one. +### The benchmark that killed a feature + +`reify verify` — a post-flight check that reads an agent's diff and reports what the +patch missed — was measured before it was written. The harness withholds one hunk from +a real merged commit, asks the graph what the patch missed, then runs the *complete* +commit through the same query, where every finding is a false positive by construction. +Model-free, deterministic, 116s for three repositories. + +```bash +reify-bench verify-eval --repo --out results/verify- --until +reify-bench verify-report --results "name=results/verify-" --out benchmarks/REPORT-verify.md +``` + +It failed its pre-registered condition on all three: the graph finds the omitted file +often enough — `omission_recall` 0.40 on Go, 0.50 on Rust, 0.10 on Python — but reports +4.4 to 23.5 findings against commits that are already complete. A `CALLS` edge says a +caller exists; it does not say the caller needed changing. **The feature was not +built.** [Full writeup](benchmarks/REPORT-verify.md). + ## Development ```bash @@ -655,7 +669,8 @@ which is why every answer comes with a line number instead of a similarity score **My repo is 3,000 lines. Should I use it?** No. Use ripgrep. Under roughly 20k LOC Reify buys you nothing a grep and a scroll wheel -don't. +don't. `reify doctor` applies that floor and three other signals to your repository, before +you index it. **Does it send my proprietary code anywhere?** It cannot. There is no HTTP client in the binary, and a test fails the build if one @@ -667,20 +682,6 @@ Probably not. Detection requires five conditions at once and is biased hard towa because a conflict detector that cries wolf gets switched off in week two and takes its true positives with it. -## Roadmap - -The first improvement pass is done. The history prior (every merged -commit is a labelled example: message ≈ ticket, changed files = answer), test-to-code -edges, iterative refinement and a fourth repository all shipped; the weight fit failed -its held-out validation and was reverted per its own pre-registration; and the -scorecard stands at one of seven targets met, each number printed next to its bar. The -open problem is the modern-TypeScript case, where nothing yet closes the vocabulary -gap between how people describe UI changes and how the code spells them. - -## Status - -Early, and measured. Known misses, all documented rather than buried: the store is 33% of the working tree against a 5% target, `reify why` is 205 ms against 20 ms, and Windows is untested. - ## License [Apache-2.0](LICENSE). Patent grant included, so an agent vendor can actually ship it. diff --git a/README.vi.md b/README.vi.md index 92f8592..f4a742f 100644 --- a/README.vi.md +++ b/README.vi.md @@ -18,8 +18,9 @@ Release Documentation Apache-2.0 - SWE-bench retrieval 84.6% + SWE-bench retrieval 87.0% network calls: 0 + platforms: macOS, Linux, Windows

@@ -27,11 +28,11 @@ Cursor Codex OpenCode - MCP + MCP

- Trên SWE-bench Verified, Reify đặt đúng file cần sửa trước mặt mô hình 84,6% số lần — grep chỉ được 6,6% · 500 issue thật, benchmark của người khác · không bao giờ mở socket
+ Trên SWE-bench Verified, Reify đặt đúng file cần sửa trước mặt mô hình 87,0% số lần — grep chỉ được 6,6% · 500 issue thật, benchmark của người khác · không bao giờ mở socket
Một mô hình thật, 142 task lấy từ commit đã merge thật ở ERPNext, OFBiz, OpenMRS và Medusa; mỗi index được dựng tại một commit trước khi những thay đổi đó tồn tại. Đó là truy xuất: đúng file, đặt trước mặt mô hình. Còn về tính đúng đắn của bản vá đầu-cuối, hiện một baseline BM25 giải quyết được nhiều issue hơn Reify, và mục nói điều đó cũng nổi bật đúng như mục này. Bản viết đầy đủ · tự chạy lại.

@@ -70,7 +71,7 @@ repository, cả hai đều in kế hoạch trước. Nối từng agent, hook v - **Số liệu:** [SWE-bench Verified](#swebench) · [bốn repository](#numbers) · [chỗ nó không chạy được](#where-it-doesnt-work) - **Dùng nó:** [cài đặt](#install) · [nối vào agent](#other-agents) · [các lệnh](#commands) · [quyền riêng tư](#privacy) - **Bên dưới:** [cách hoạt động](#how-it-works) · [nó đọc gì](#what-it-reads) · [đa ngôn ngữ](#multilingual) · [kiến trúc](#architecture) -- [Câu hỏi thường gặp](#faq) · [phát triển](#development) · [lộ trình](#roadmap) · [giấy phép](#license) +- [Câu hỏi thường gặp](#faq) · [phát triển](#development) · [giấy phép](#license) ## Vấn đề phụ thuộc một người @@ -169,22 +170,22 @@ thông thường; đáp án đúng là tập file mà bản vá được chấp |---|--:|--:|--:|--:| | grep, content | 6.6% [4.7–9.1] | 0.06 | 5.6% | 3,998 | | grep, paths | 9.0% [6.8–11.8] | 0.06 | 7.8% | 3,996 | -| **reify**, một vòng | **66.0%** [61.7–70.0] | 0.43 | 59.0% | **3,466** | -| **reify**, ba vòng | **84.6%** [81.2–87.5] | 0.45 | 77.0% | 9,174 | +| **reify**, một vòng | **72.6%** [68.5–76.3] | 0.42 | 65.4% | **3,670** | +| **reify**, ba vòng | **87.0%** [83.8–89.7] | 0.43 | 81.4% | 9,549 | -**Một vòng Reify duy nhất thắng grep trên 310 instance và thua 13 — trong khi tiêu ít -token hơn** (3.466 so với 3.998). Ba vòng thắng 395–5 (McNemar chính xác -p ≈ 7 × 10⁻¹¹⁰). Đây không phải phép đo sát nút, và nó là con số sạch nhất trong tài +**Một vòng Reify duy nhất thắng grep trên 342 instance và thua 12 — trong khi tiêu ít +token hơn** (3.670 so với 3.998). Ba vòng thắng 406–4 (McNemar chính xác +p ≈ 9 × 10⁻¹¹⁵). Đây không phải phép đo sát nút, và nó là con số sạch nhất trong tài liệu này chính bởi vì đề bài, repository lẫn đáp án đều đến từ nơi khác. Theo từng repository, ba vòng so với grep nội dung: | | grep | reify ×3 | | | grep | reify ×3 | |---|--:|--:|---|---|--:|--:| -| django (n=231) | 6% | **88%** | | astropy (n=22) | 0% | **77%** | -| sympy (n=75) | 7% | **77%** | | xarray (n=22) | 9% | **91%** | -| sphinx (n=44) | 7% | **75%** | | pytest (n=19) | 26% | **84%** | -| matplotlib (n=34) | 0% | **91%** | | pylint (n=10) | 10% | **60%** | +| django (n=231) | 6% | **90%** | | astropy (n=22) | 0% | **77%** | +| sympy (n=75) | 7% | **79%** | | xarray (n=22) | 9% | **95%** | +| sphinx (n=44) | 7% | **80%** | | pytest (n=19) | 26% | **95%** | +| matplotlib (n=34) | 0% | **97%** | | pylint (n=10) | 10% | **60%** | | scikit-learn (n=32) | 9% | **88%** | | requests (n=8) | 0% | **100%** | **Điều này chứng minh gì và không chứng minh gì.** Nó đo *truy xuất* — liệu những file @@ -197,26 +198,38 @@ hình, không ảnh hưởng tới việc một bộ truy xuất đề xuất fi [`benchmarks/swe/`](benchmarks/swe/). -### Đầu-cuối: hoà, và cái giá để đến được đó +### Đầu-cuối: đang dẫn trước, nhưng chưa được chứng minh Truy xuất không phải tuyên bố cuối cùng — giải quyết được issue mới là. Cùng benchmark đó, chạy qua chính giao thức của bài báo SWE-bench (một mô hình, một ngân sách, bộ truy xuất là khác biệt duy nhất), mọi bản vá do **bộ chấm chính thức của SWE-bench** phán quyết: -| giải quyết được issue, 63 instance chấm ở cả hai nhánh | | | +| giải quyết được issue, 101 instance chấm ở cả hai nhánh | | | |---|--:|---| -| BM25 | 23,8% | | -| **Reify** | **23,8%** | 6–6, p = 1,0 | - -Hoà — và cần nói thẳng, vì lần thử đầu tiên là **thua**: 11,1% so với 18,1%. Điều thú vị -nằm ở chỗ đã khép lại khoảng cách đó bằng cách nào. - -Reify tìm đúng file thường xuyên hơn hẳn (77% so với 60% của BM25) mà mô hình vẫn làm tệ -hơn. Dựng lại đúng những prompt đó cho thấy lý do: một cửa sổ ngữ cảnh được đổ đầy bằng -nguyên cả file theo thứ tự xếp hạng sẽ tiêu hết vào thứ đứng đầu, và cách xếp hạng của -Reify mù trước kích thước file, trong khi BM25 có chuẩn hoá độ dài ngay trong công thức. -**File đúng đã được tìm ra rồi không bao giờ được cho xem** — chỉ hiện diện trong 27% số -prompt, so với 40% của BM25. +| BM25 | 67,3% | 68 giải quyết được, 1 bản vá rỗng | +| **Reify** | **73,3%** | 74 giải quyết được, 2 bản vá rỗng | +| | | 12–6, McNemar chính xác **p = 0,24** | + +**Hãy đọc p-value trước khi đọc phần trăm.** Reify giải quyết được nhiều hơn sáu issue và +thắng gấp đôi số lần bất đồng — nhưng ở cỡ mẫu này đó chưa phải kết quả có ý nghĩa thống +kê. Cách tóm tắt trung thực là *đang dẫn trước và chưa được chứng minh*, không phải một +chiến thắng. + +**Các con số tuyệt đối này không so sánh được với những con số từng công bố ở đây.** +Lần chạy đó dùng DeepSeek; lần này dùng Claude Sonnet, vì tài khoản DeepSeek đã hết số dư +giữa chừng. Một mô hình mạnh hơn nâng cả hai nhánh — lần trước mỗi bên khoảng 24%. Thứ +sống sót qua việc đổi mô hình là phép so sánh *theo cặp*, vì cả hai nhánh luôn trả lời +cùng một instance bằng cùng một mô hình. Kết quả thô từng instance: +[`benchmarks/swe/results/stage2-endtoend.json`](benchmarks/swe/results/stage2-endtoend.json). + +Không phải lúc nào cũng dẫn trước. Lần thử đầu tiên là **thua**, và điều thú vị nằm ở chỗ +đã khép lại khoảng cách đó bằng cách nào. + +Reify tìm đúng file thường xuyên hơn hẳn mà mô hình vẫn làm tệ hơn. Dựng lại đúng những +prompt đó cho thấy lý do: một cửa sổ ngữ cảnh được đổ đầy bằng nguyên cả file theo thứ tự +xếp hạng sẽ tiêu hết vào thứ đứng đầu, và thứ tự đó mù trước kích thước file, trong khi +BM25 có chuẩn hoá độ dài ngay trong công thức. **File đúng đã được tìm ra rồi không bao +giờ được cho xem** — chỉ hiện diện trong 27% số prompt, so với 40% của BM25. `reify context --for-edit` sửa từ gốc: các vùng được nới ra thành định nghĩa trọn vẹn, phần import của file được đưa vào một lần, các vùng chồng nhau được gộp, ngân sách vẫn @@ -228,8 +241,11 @@ cứng. Không còn mất gì ở cửa sổ nữa: | Reify, nguyên file | 76,7% | 26,7% | | **Reify `--for-edit`** | **80,0%** | **56,7%** | -Hai cách sửa đã bị **bác bỏ dựa trên bằng chứng**: giới hạn theo từng file làm mọi thứ tệ -hơn (một file bị cắt cụt thì cũng không sửa được), và xếp hạng theo chi phí làm giảm truy +Việc giới hạn phần cửa sổ mà một file được chiếm đã từng bị **bác bỏ dựa trên bằng chứng**, +rồi sau đó mới được áp dụng ở dạng có hiệu quả. Cắt cụt *nội dung* file làm mọi thứ tệ hơn, +vì một file bị cắt cụt thì cũng không sửa được. Giới hạn số *ký hiệu* mà một file đóng góp +vào phần được chọn — ở phía trên cửa sổ, giữ nguyên vẹn từng vùng — mới là thứ cải thiện +truy xuất, và đó là thứ đang được ship. Xếp hạng theo chi phí thì bị bác bỏ hẳn: nó làm giảm truy xuất bảy điểm mà chẳng đem lại gì khi các vùng đã khiến kích thước file không còn quan trọng. Vậy: Reify thắng rõ ràng ở truy xuất và hoà ở kết quả vá cuối cùng. Ràng buộc còn lại nằm @@ -388,7 +404,16 @@ Ba thứ chỉ vỡ khi bạn rời khỏi hệ chữ Latinh, và cả ba đều curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh ``` -Có sẵn binary dựng trước cho macOS (Apple Silicon và Intel) và Linux (x86_64 và aarch64). +Có sẵn binary dựng trước cho **macOS** (Apple Silicon và Intel), **Linux** (x86_64 và +aarch64) và **Windows** (x86_64). Trên Windows, dòng lệnh trên chạy được nguyên vẹn +trong Git Bash, MSYS2 hoặc WSL; từ PowerShell, hãy tải archive `x86_64-pc-windows-msvc` +ở [bản phát hành mới nhất](https://github.com/lambiengcode/reify/releases/latest), kiểm +tra `.sha256` của nó, rồi đặt `reify.exe` vào một thư mục nằm trong `PATH`. + +Mọi nền tảng trong số đó đều chạy toàn bộ test suite trên CI, và CLI được chạy thử +đầu-cuối trên từng nền tảng — không phát hành binary cho một nền tảng mà chưa có gì +từng chạy trên đó. + Hoặc build từ mã nguồn: ```bash @@ -481,7 +506,7 @@ Treat INFERRED claims as leads to verify, not facts. reify serve --mcp ``` -Ba công cụ — `reify_context`, `reify_why`, `reify_impact` — và ba là toàn bộ bề mặt. Schema của một MCP server bị gửi lại mỗi lượt của mỗi phiên, nên một công cụ sinh ra để tiết kiệm ngữ cảnh thì không nên thu tiền thuê chỗ để giao hàng. Một test khẳng định các schema tốn dưới 600 token. +Sáu công cụ — `reify_context`, `reify_why`, `reify_impact`, `reify_explain`, `reify_flow` và `reify_conflicts` — và sáu là toàn bộ bề mặt. Schema của một MCP server bị gửi lại mỗi lượt của mỗi phiên, nên một công cụ sinh ra để tiết kiệm ngữ cảnh thì không nên thu tiền thuê chỗ để giao hàng. Một test khẳng định các schema tốn dưới 600 token. ### Tuỳ chọn: dùng một mô hình @@ -571,7 +596,7 @@ ERPNext, 5.064 file, laptop chip M 8 nhân. | index lại, sửa một file | 0,7 giây | | `reify context` | 57 ms | | `reify impact` | 0,2 ms | -| `reify why` | 205 ms — do gọi tiến trình con `git log -L`; khoảng 5 ms nếu bỏ nó | +| `reify why` | 87 ms trung vị, 168 ms tệ nhất — do gọi tiến trình con `git log -L`; khoảng 5 ms nếu bỏ nó | | bộ nhớ đỉnh, index đầy đủ | 224 MB | | dung lượng kho | 47 MB (33% của cây làm việc 144 MB) | @@ -656,7 +681,7 @@ Không. Dùng ripgrep đi. Dưới khoảng 20 nghìn dòng code, Reify không c Không thể. Trong binary không có HTTP client nào, và một test sẽ làm hỏng build nếu có một cái xuất hiện. Nếu bạn cấu hình nhà cung cấp mô hình, đó là lệnh do bạn chọn, và `reify llm preview` cho bạn xem chính xác từng byte trước. **Sao `reify why` chậm hơn mọi lệnh khác?** -Nó gọi ra `git log -L` để lấy lịch sử theo dòng chính xác. 205 ms khi có, khoảng 5 ms khi không. Vẫn nằm trong danh sách cần cải thiện. +Nó gọi ra `git log -L` để lấy lịch sử theo dòng chính xác. 87 ms khi có, khoảng 5 ms khi không. Vẫn nằm trong danh sách cần cải thiện. **Lệnh conflicts không tìm thấy gì trong repo của tôi. Nó hỏng à?** Chắc là không. Việc phát hiện đòi hỏi năm điều kiện cùng đúng một lúc và được thiên lệch mạnh về phía im lặng, bởi vì một bộ phát hiện mâu thuẫn hay báo động giả sẽ bị tắt ngay tuần thứ hai và mang theo cả những cảnh báo đúng của nó. Nó tìm thấy 0 trên ERPNext — repo gần như không có văn bản đặc tả — và đúng 1 trên fixture, nơi có một cái được cài sẵn. @@ -664,20 +689,6 @@ Chắc là không. Việc phát hiện đòi hỏi năm điều kiện cùng đ **"Reify" nghĩa là gì?** Là biến một thứ trừu tượng thành cụ thể. Kiến thức vốn luôn ở đó; chỉ là nó chưa từng là một file. -## Lộ trình - -Đợt cải tiến đầu tiên đã xong. Tiên nghiệm từ lịch sử (mỗi commit đã merge là một ví dụ -có nhãn: message ≈ ticket, file thay đổi = đáp án), cạnh nối test với code, tinh chỉnh -lặp vòng và một repository thứ tư đều đã lên; phần fit trọng số trượt khâu kiểm định -trên tập giữ riêng và đã được khôi phục về mặc định đúng theo cam kết đăng ký trước; và -bảng điểm dừng ở một trên bảy mục tiêu, mỗi con số được in ngay cạnh ngưỡng của nó. Bài -toán còn mở là trường hợp TypeScript hiện đại, nơi chưa có gì lấp được khoảng cách từ -vựng giữa cách người ta mô tả thay đổi giao diện và cách code viết ra chúng. - -## Trạng thái - -Còn sớm, và có đo đạc. Những điểm chưa đạt, đều được ghi rõ chứ không giấu: kho lưu trữ chiếm 33% cây làm việc so với mục tiêu 5%, `reify why` mất 205 ms so với mục tiêu 20 ms, và Windows chưa được kiểm thử. - ## Giấy phép [Apache-2.0](LICENSE). Có kèm cấp quyền sáng chế, nên một nhà cung cấp agent thực sự có thể ship nó. diff --git a/README.zh.md b/README.zh.md index 0d7f5fb..d4134f7 100644 --- a/README.zh.md +++ b/README.zh.md @@ -18,8 +18,9 @@ Release Documentation Apache-2.0 - SWE-bench retrieval 84.6% + SWE-bench retrieval 87.0% network calls: 0 + platforms: macOS, Linux, Windows

@@ -27,11 +28,11 @@ Cursor Codex OpenCode - MCP + MCP

- 在 SWE-bench Verified 上，Reify 有 84.6% 的概率把必须改动的文件送到模型面前 —— grep 只有 6.6% · 500 个真实 issue，别人的基准 · 从不打开任何 socket
+ 在 SWE-bench Verified 上，Reify 有 87.0% 的概率把必须改动的文件送到模型面前 —— grep 只有 6.6% · 500 个真实 issue，别人的基准 · 从不打开任何 socket
真实模型，142 个任务，全部取自 ERPNext、OFBiz、OpenMRS 和 Medusa 中真实合并的提交；每个索引都构建在这些改动尚不存在的提交上。那是检索：把正确的文件送到模型面前。而在端到端的补丁正确性上，目前一个 BM25 基线解决的 issue 比 Reify 更多，说明这一点的那一节与本节同样醒目。完整报告 · 自行复现。

@@ -70,7 +71,7 @@ reify context "你即将进行的改动" --toon - **数据：** [SWE-bench Verified](#swebench) · [四个代码库](#numbers) · [它失效的地方](#where-it-doesnt-work) - **使用：** [安装](#install) · [接入 agent](#other-agents) · [命令](#commands) · [隐私](#privacy) - **底层：** [工作原理](#how-it-works) · [它能读什么](#what-it-reads) · [多语言](#multilingual) · [架构](#architecture) -- [常见问题](#faq) · [开发](#development) · [路线图](#roadmap) · [许可证](#license) +- [常见问题](#faq) · [开发](#development) · [许可证](#license) ## 只有一个人懂的问题 @@ -163,11 +164,11 @@ $ reify why erpnext/selling/doctype/sales_order/sales_order.py:812 |---|--:|--:|--:|--:| | grep, content | 6.6% [4.7–9.1] | 0.06 | 5.6% | 3,998 | | grep, paths | 9.0% [6.8–11.8] | 0.06 | 7.8% | 3,996 | -| **reify**, 单轮 | **66.0%** [61.7–70.0] | 0.43 | 59.0% | **3,466** | -| **reify**, 三轮 | **84.6%** [81.2–87.5] | 0.45 | 77.0% | 9,174 | +| **reify**, 单轮 | **72.6%** [68.5–76.3] | 0.42 | 65.4% | **3,670** | +| **reify**, 三轮 | **87.0%** [83.8–89.7] | 0.43 | 81.4% | 9,549 | -**单轮 Reify 在 310 个实例上胜过 grep，仅在 13 个上落败 —— 而且花的 token 更少** -（3,466 对 3,998）。三轮则是 395 比 5（精确 McNemar 检验 p ≈ 7 × 10⁻¹¹⁰）。这不是一次 +**单轮 Reify 在 342 个实例上胜过 grep，仅在 12 个上落败 —— 而且花的 token 更少** +（3,670 对 3,998）。三轮则是 406 比 4（精确 McNemar 检验 p ≈ 9 × 10⁻¹¹⁵）。这不是一次 势均力敌的测量，而且它是本文档中最干净的数字，正因为任务、代码库和标准答案全都来自 别处。 @@ -175,10 +176,10 @@ $ reify why erpnext/selling/doctype/sales_order/sales_order.py:812 | | grep | reify ×3 | | | grep | reify ×3 | |---|--:|--:|---|---|--:|--:| -| django (n=231) | 6% | **88%** | | astropy (n=22) | 0% | **77%** | -| sympy (n=75) | 7% | **77%** | | xarray (n=22) | 9% | **91%** | -| sphinx (n=44) | 7% | **75%** | | pytest (n=19) | 26% | **84%** | -| matplotlib (n=34) | 0% | **91%** | | pylint (n=10) | 10% | **60%** | +| django (n=231) | 6% | **90%** | | astropy (n=22) | 0% | **77%** | +| sympy (n=75) | 7% | **79%** | | xarray (n=22) | 9% | **95%** | +| sphinx (n=44) | 7% | **80%** | | pytest (n=19) | 26% | **95%** | +| matplotlib (n=34) | 0% | **97%** | | pylint (n=10) | 10% | **60%** | | scikit-learn (n=32) | 9% | **88%** | | requests (n=8) | 0% | **100%** | **它证明了什么，没证明什么。** 它测的是*检索* —— 必须改动的文件是否被送到模型面前 —— @@ -188,23 +189,33 @@ $ reify why erpnext/selling/doctype/sales_order/sales_order.py:812 [`benchmarks/swe/`](benchmarks/swe/) 中的驱动脚本复现。 -### 端到端：打平，以及走到这一步的代价 +### 端到端：领先，但尚未被证明 检索不是最终的主张 —— 解决 issue 才是。同一份基准，跑在 SWE-bench 论文自己的协议上 （一个模型、一份预算，检索器是唯一变量），每个补丁都由 **SWE-bench 官方评测器**判定： -| 解决了 issue，两组都判定过的 63 个实例 | | | +| 解决了 issue，两组都判定过的 101 个实例 | | | |---|--:|---| -| BM25 | 23.8% | | -| **Reify** | **23.8%** | 6–6，p = 1.0 | +| BM25 | 67.3% | 解决 68 个，1 个空补丁 | +| **Reify** | **73.3%** | 解决 74 个，2 个空补丁 | +| | | 12–6，精确 McNemar **p = 0.24** | -打平 —— 而且必须说清楚，因为第一次尝试是**落败**：11.1% 对 18.1%。有意思的是它是怎么被 -追平的。 +**先看 p 值，再看百分比。** Reify 多解决了六个 issue，在双方结果不一致的实例上赢的次数 +是输的两倍 —— 但在这个样本量下，这还不是统计显著的结果。诚实的说法是*领先且尚未被证明*， +而不是一场胜利。 -Reify 找到正确文件的频率高得多（77% 对 BM25 的 60%），模型却表现更差。重建那些 prompt -后原因很清楚：一个按排名顺序用整份文件填满的上下文窗口，会把额度全花在排第一的东西上， -而 Reify 的排名对文件大小视而不见，BM25 的公式里却自带长度归一化。**正确的文件被检索到 -了，然后从未被展示** —— 只出现在 27% 的 prompt 中，而 BM25 是 40%。 +**这些绝对数字无法与本文档此前公布的数字相比。** 那次用的是 DeepSeek；这次用的是 +Claude Sonnet，因为 DeepSeek 账户中途余额耗尽。更强的模型会同时抬高两组 —— 上一次两边 +都在 24% 左右。换模型之后仍然成立的是*配对*比较，因为两组永远用同一个模型回答同一个实例。 +逐实例的原始结果： +[`benchmarks/swe/results/stage2-endtoend.json`](benchmarks/swe/results/stage2-endtoend.json)。 + +它并非一直领先。第一次尝试是**落败**，有意思的是它是怎么被追平的。 + +Reify 找到正确文件的频率高得多，模型却表现更差。重建那些 prompt 后原因很清楚：一个按排名 +顺序用整份文件填满的上下文窗口，会把额度全花在排第一的东西上，而那个顺序对文件大小视而 +不见，BM25 的公式里却自带长度归一化。**正确的文件被检索到了，然后从未被展示** —— 只出现 +在 27% 的 prompt 中，而 BM25 是 40%。 `reify context --for-edit` 从源头解决：区域扩展到完整定义、文件的 import 只包含一次、 重叠区域合并、预算依然是硬约束。检索到的东西不再在窗口处丢失： @@ -215,11 +226,13 @@ Reify 找到正确文件的频率高得多（77% 对 BM25 的 60%），模型却 | Reify，整份文件 | 76.7% | 26.7% | | **Reify `--for-edit`** | **80.0%** | **56.7%** | -有两种修法**依据证据被否决**：按文件设上限反而更糟（被截断的文件同样无法编辑），而 -成本感知排名让检索下降七个百分点，且在区域化让文件大小失去意义之后毫无收益。 +限制单个文件能占据多少窗口，早期曾**依据证据被否决**，后来才以真正有效的形式被采纳。截断 +文件*内容*会让情况更糟，因为被截断的文件同样无法编辑。限制单个文件向选择结果贡献多少个 +*符号* —— 在窗口之上、且每个区域保持完整 —— 才是真正改善检索的做法，也是今天所发布的版本。 +成本感知排名则被彻底否决：它让检索下降七个百分点，且在区域化让文件大小失去意义之后毫无收益。 -所以：Reify 在检索上明显胜出，在最终补丁成功率上打平。剩下的瓶颈在写补丁的循环，而不在 -上下文 —— 两组都卡在 24% 附近。 +所以：Reify 在检索上明显胜出，在最终补丁成功率上领先但结论不充分。剩下的瓶颈在写补丁的 +循环，而不在上下文 —— 两组都还有约四分之一的 issue 未能解决，尽管该改的文件就在 prompt 里。 ## 数据：在四个刻意挑难的代码库上 @@ -378,7 +391,14 @@ C/C++、Kotlin，外加 SQL。每一种都有一个测试断言它能产出容 curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh ``` -提供 macOS（Apple Silicon 与 Intel）和 Linux（x86_64 与 aarch64）的预编译二进制。 +提供 **macOS**（Apple Silicon 与 Intel）、**Linux**（x86_64 与 aarch64）和 **Windows** +（x86_64）的预编译二进制。在 Windows 上，上面这行命令在 Git Bash、MSYS2 或 WSL 中可以 +原样运行；若使用 PowerShell，请从[最新发布](https://github.com/lambiengcode/reify/releases/latest) +下载 `x86_64-pc-windows-msvc` 压缩包，校验其 `.sha256`，再把 `reify.exe` 放到 `PATH` 中。 + +以上每个平台都会在 CI 上跑完整的测试套件，并逐一做端到端的 CLI 验证 —— 不会为一个从未 +被执行过的平台发布二进制。 + 也可以从源码构建： ```bash @@ -473,7 +493,8 @@ Treat INFERRED claims as leads to verify, not facts. reify serve --mcp ``` -三个工具 —— `reify_context`、`reify_why`、`reify_impact` —— 三个就是全部接口。MCP 服务端 +六个工具 —— `reify_context`、`reify_why`、`reify_impact`、`reify_explain`、 +`reify_flow`、`reify_conflicts` —— 六个就是全部接口。MCP 服务端 的 schema 会在每个会话的每一轮被重新发送，所以一个为了节省上下文而生的工具，不该为了 送货再收一笔租金。有测试断言这些 schema 的开销低于 600 个 token。 @@ -668,19 +689,6 @@ Fixture 位于 [`fixtures/minierp`](fixtures/) —— 一个小型业务系统 **"reify" 是什么意思？** 把抽象之物变得具体。这些知识一直都在，只是从来没成为一个文件。 -## 路线图 - -第一轮改进已经完成。历史先验（每个已合并的提交都是一个带标注的样本：提交信息 ≈ 工单， -改动文件 = 答案）、测试到代码的边、迭代式精修，以及第四个代码库都已落地；权重拟合没有 -通过留出验证，按其事前登记被回退；记分卡停在七项目标中达成一项，每个数字都印在它的门槛 -旁边。尚未解决的问题是现代 TypeScript 的情形：人们描述界面改动的说法，和代码里的写法 -之间，那道词汇鸿沟目前还没有任何东西能填上。 - -## 项目状态 - -尚早，但有实测。已知的未达标项，全部写明而非掩埋：存储占工作区的 33%，目标是 5%； -`reify why` 耗时 205 毫秒，目标是 20 毫秒；Windows 尚未测试。 - ## 许可证 [Apache-2.0](LICENSE)。含专利授权，所以 agent 厂商真的可以把它发出去。 diff --git a/benchmarks/REPORT-medusa.md b/benchmarks/REPORT-medusa.md index d88c1af..0dba335 100644 --- a/benchmarks/REPORT-medusa.md +++ b/benchmarks/REPORT-medusa.md @@ -1,5 +1,23 @@ # Reify Brownfield Benchmark +> **Correction, 2026-08-24** — added by hand; the rest of this file is generated. +> +> The *With a model in the loop* section below was produced with a prompt that named +> the wrong repository. `reify-bench`'s prompt template hard-coded the literal +> `(ERPNext)`, so every row of that table told the model it was working on ERPNext +> while asking it about **medusa**. Everything above that section is model-free and +> is unaffected. +> +> **Scope.** The misnaming sits in the preamble every condition shares, so it does +> not obviously favour one arm, and the test that the prompt differs between +> conditions only in its CONTEXT block still holds. It is a validity defect all the +> same: those hit rates are not what a correctly-prompted model would produce, and +> should not be read as such. The table is left exactly as measured rather than +> regenerated, because re-running it needs the provider and the pinned checkout. +> +> Fixed in `crates/reify-bench/src/agent.rs`: `prompt()` now takes the repository +> name from `TaskSet::repository`. + Generated by `reify-bench report`. Every number here is computed from `outcomes.json` in the same directory; nothing is entered by hand. diff --git a/benchmarks/REPORT-ofbiz.md b/benchmarks/REPORT-ofbiz.md index bfef115..eddbf9d 100644 --- a/benchmarks/REPORT-ofbiz.md +++ b/benchmarks/REPORT-ofbiz.md @@ -1,5 +1,23 @@ # Reify Brownfield Benchmark +> **Correction, 2026-08-24** — added by hand; the rest of this file is generated. +> +> The *With a model in the loop* section below was produced with a prompt that named +> the wrong repository. `reify-bench`'s prompt template hard-coded the literal +> `(ERPNext)`, so every row of that table told the model it was working on ERPNext +> while asking it about **ofbiz**. Everything above that section is model-free and +> is unaffected. +> +> **Scope.** The misnaming sits in the preamble every condition shares, so it does +> not obviously favour one arm, and the test that the prompt differs between +> conditions only in its CONTEXT block still holds. It is a validity defect all the +> same: those hit rates are not what a correctly-prompted model would produce, and +> should not be read as such. The table is left exactly as measured rather than +> regenerated, because re-running it needs the provider and the pinned checkout. +> +> Fixed in `crates/reify-bench/src/agent.rs`: `prompt()` now takes the repository +> name from `TaskSet::repository`. + Generated by `reify-bench report`. Every number here is computed from `outcomes.json` in the same directory; nothing is entered by hand. diff --git a/benchmarks/REPORT-openmrs.md b/benchmarks/REPORT-openmrs.md index 0f55612..9e23ce7 100644 --- a/benchmarks/REPORT-openmrs.md +++ b/benchmarks/REPORT-openmrs.md @@ -1,5 +1,23 @@ # Reify Brownfield Benchmark +> **Correction, 2026-08-24** — added by hand; the rest of this file is generated. +> +> The *With a model in the loop* section below was produced with a prompt that named +> the wrong repository. `reify-bench`'s prompt template hard-coded the literal +> `(ERPNext)`, so every row of that table told the model it was working on ERPNext +> while asking it about **openmrs**. Everything above that section is model-free and +> is unaffected. +> +> **Scope.** The misnaming sits in the preamble every condition shares, so it does +> not obviously favour one arm, and the test that the prompt differs between +> conditions only in its CONTEXT block still holds. It is a validity defect all the +> same: those hit rates are not what a correctly-prompted model would produce, and +> should not be read as such. The table is left exactly as measured rather than +> regenerated, because re-running it needs the provider and the pinned checkout. +> +> Fixed in `crates/reify-bench/src/agent.rs`: `prompt()` now takes the repository +> name from `TaskSet::repository`. + Generated by `reify-bench report`. Every number here is computed from `outcomes.json` in the same directory; nothing is entered by hand. diff --git a/benchmarks/REPORT-verify.md b/benchmarks/REPORT-verify.md new file mode 100644 index 0000000..a781d4c --- /dev/null +++ b/benchmarks/REPORT-verify.md @@ -0,0 +1,144 @@ +# Can the graph tell that a patch is incomplete? + +Generated by `reify-bench verify-report`. Every number is computed from the `verify-summary.json` files named below; nothing is entered by hand. + +This benchmark exists to decide one thing: whether `reify verify` — a post-flight check that reads an agent's diff and reports what the patch missed — is worth building on Reify's call graph. It is model-free, deterministic, and costs nothing per run. + +## Construction + +For each merged commit that passes the retrieval benchmark's filters and touches at least two indexable files: + +1. the parent tree is extracted and indexed, so the change is absent from the index by construction; +2. one file's **only** hunk is withheld — the *omission*. Removing it removes that file from the patch entirely, so a citation of it cannot be an echo of a hunk still present. Among the files with exactly one hunk, the last by path order is chosen; the choice is arbitrary, fixed, and made before any checker runs; +3. the truncated patch goes to the checker; +4. **the same commit goes to the checker complete.** A merged commit is complete by definition, so every finding there is a false positive. This control is not optional: without it the metric would reward a checker that simply shouts. + +The checker is not `reify verify`, which does not exist. It is the shipped graph query — *symbols changed by this diff, minus symbols present in the diff, where an inbound `CALLS` edge exists at distance 1* — reached through `reify::query::impact`. That deliberately measures the **substrate**, which is the number the decision needs. + +## Pre-registered falsification condition + +> If `omission_recall` on this substrate is below **0.25**, or `false_alarm_rate` is above **0.1 per commit**, the `reify verify` feature does not get built on this substrate. + +Stated in `crates/reify-bench/src/metrics.rs` before the first run and not moved since. A result that kills the feature is a result. + +## Results + +| Metric | reify | django | gh-cli | +|---|---:|---:|---:| +| Most indexed language | rust | python | go | +| Trials | 4 | 20 | 20 | +| `omission_recall` | **0.50** (0.15–0.85) | **0.10** (0.03–0.30) | **0.40** (0.22–0.61) | +| …attributable to the omission | 0.00 (0.00–0.49) | 0.05 (0.01–0.24) | 0.15 (0.05–0.36) | +| `omission_recall_symbol` | — (0 scorable) | 0.12 (0.03–0.36) over 16 | 0.31 (0.14–0.56) over 16 | +| Omitted files a caller query *could* cite | 3/4 | 20/20 | 19/20 | +| `false_alarm_rate` (per complete commit) | **23.5** | **6.9** | **4.4** | +| Complete commits with ≥1 false alarm | 4/4 (0.51–1.00) | 18/20 (0.70–0.97) | 15/20 (0.53–0.89) | +| `findings_per_diff` (median) | 19 | 1 | 2 | +| `verify_tokens` (median) | 422 | 36 | 48 | +| `verify_latency_ms` (median) | 1 | 1 | 0 | +| Index per trial, ms (median) | 151 | 4732 | 809 | +| Whole run, wall clock | 1s | 97s | 18s | +| Pre-registered verdict | **do not build** | **do not build** | **do not build** | + +## What the numbers say + +**reify** — false_alarm_rate 23.50 > 0.10 + +**django** — omission_recall 0.10 < 0.25; false_alarm_rate 6.90 > 0.10 + +**gh-cli** — false_alarm_rate 4.45 > 0.10 + +Every repository fails the pre-registered condition, so **`reify verify` does not get built on this substrate**. The condition was written down before the first run precisely so this outcome could not be argued away afterwards. + +**It fails on noise, not on blindness.** 3 of 3 repositories exceed the false-alarm ceiling; 1 of 3 fall below the recall floor (a repository can fail both). The graph does find the omitted file often enough to be interesting; what it cannot do is stay quiet about a patch that is already complete. + +**The negative control takes most of the headline back.** `omission_recall` counts a citation of the omitted file whether or not the complete commit is cited too. The attributable row counts only citations the complete commit does *not* produce, and it is the smaller number in every repository here. The gap is the checker citing a file it would have cited anyway — which is not detection, however it reads next to the label. + +**The ceiling is not what binds.** A finding is a caller, so the omitted file can only be cited if something in it calls out of itself. In the least favourable repository here that holds for 75% of omissions, so the edges mostly exist and `omission_recall` is not capped by their absence. The gap between that row and the recall row is a *ranking* gap, not a coverage one. + +**The noise is structural, not marginal.** `false_alarm_rate` is findings per commit that is complete by construction. A `CALLS` edge says a caller exists; it does not say the caller needed changing. Nothing in the graph distinguishes a changed signature from an edit inside a body, so every caller of every touched symbol is a candidate. That is a property of the edge, and no rewriting of the query around the same edge removes it. + +## Cost and determinism + +No model, no network, no provider key: the whole run is a git extract, an index and a graph query. Total wall clock for everything in this report is **116s**, dominated by re-indexing one parent tree per trial. The query itself is the `verify_latency_ms` row — single-digit milliseconds. + +Each run is deterministic given a fixed `HEAD`: task selection, the omission rule and the query contain no randomness and no tunable threshold. A run against a repository whose history is still moving — this one, for instance — should pin the window with `--until `, or the trial set moves with the branch. + +```bash +reify-bench verify-eval --repo --out results/verify- --until +reify-bench verify-report --results "name=results/verify-" --out benchmarks/REPORT-verify.md +``` + +## Limitations + +1. **Small samples.** The intervals are wide and are printed beside every rate. Where two repositories differ by less than their intervals, they have not been shown to differ. +2. **The omission-selection rule has a direction.** "Last by path order, among files with exactly one hunk" is arbitrary but not neutral: in a repository laid out as `src/` and `tests/`, path order lands on `tests/`. Counted across every run here, 17 of 44 omissions sit under a path segment named `test` or `tests`. The rule was fixed before any run and has not been changed since; every omitted file is named in the appendix, so the effect is checkable rather than described. +3. **`CALLS` at distance 1 only.** `impact` also propagates two hops and crosses into the data layer. Widening the query would raise recall and raise the false-alarm rate with it — the trade this benchmark measures rather than pre-empts. +4. **A checker, not the feature.** `reify verify` could use a signature diff, type information, or the model. This measures the substrate those would all stand on. +5. **Parent trees are extracted with `git archive`**, so the indexed tree has no git history and no co-change edges. The checker uses neither; a checker that did would need re-measuring. +6. **Ground truth is one commit's hunks.** A change that could correctly have been made elsewhere scores as a miss. +7. **`impact`'s own bounds are inherited, not bypassed.** It stops at 60 affected nodes and walks depth-first to two hops, so on a widely-called symbol some direct callers can be crowded out by second-hop ones. That is the shipped query's behaviour and measuring around it would measure something that does not exist. + +## Appendix: every trial + +`could cite` is whether the omitted file calls out of itself at all — the ceiling for that trial. `cited` is findings on the truncated patch, `noise` is findings on the same commit complete. + +### reify (`git@github.com:lambiengcode/reify.git`, commit `0b0bcf5cf5fc25f4c7325f108a0474e1d2895cda`) + +| Trial | Omitted file | could cite | hit | attributable | cited | noise | +|---|---|---|---|---|---:|---:| +| `v-9af59e47` | `crates/reify/tests/fixtures.rs` | yes | yes | no | 19 | 19 | +| `v-2b7bad4c` | `assets/make-logo.py` | no | no | no | 1 | 1 | +| `v-deb46ef8` | `crates/reify-cli/src/render.rs` | yes | no | no | 4 | 4 | +| `v-7dd36dae` | `crates/reify-bench/src/conditions.rs` | yes | yes | no | 70 | 70 | + +### django (`git@github.com:django/django`, commit `0b40210e4808937a7c0922e8b7502bff4752faa3`) + +| Trial | Omitted file | could cite | hit | attributable | cited | noise | +|---|---|---|---|---|---:|---:| +| `v-d992705f` | `tests/basic/models.py` | yes | no | no | 1 | 1 | +| `v-be6cf832` | `django/db/models/sql/query.py` | yes | no | no | 1 | 13 | +| `v-c72f5fb4` | `tests/validators/tests.py` | yes | no | no | 5 | 5 | +| `v-1a001208` | `tests/ordering/tests.py` | yes | no | no | 0 | 0 | +| `v-07d4f69c` | `tests/migrations/test_operations.py` | yes | no | no | 0 | 0 | +| `v-febefb17` | `django/db/models/base.py` | yes | no | no | 1 | 2 | +| `v-082b3df4` | `tests/admin_changelist/tests.py` | yes | no | no | 1 | 1 | +| `v-6df8fe3b` | `tests/admin_changelist/tests.py` | yes | no | no | 3 | 3 | +| `v-616e8c52` | `tests/admin_views/tests.py` | yes | no | no | 1 | 1 | +| `v-89e82866` | `tests/admin_views/tests.py` | yes | no | no | 1 | 1 | +| `v-47511a21` | `tests/admin_utils/tests.py` | yes | yes | no | 15 | 14 | +| `v-27137e65` | `django/test/signals.py` | yes | no | no | 6 | 6 | +| `v-94653491` | `tests/field_defaults/models.py` | yes | no | no | 2 | 2 | +| `v-ca14173f` | `tests/urlpatterns/tests.py` | yes | no | no | 1 | 1 | +| `v-c9ff757a` | `tests/bulk_create/tests.py` | yes | no | no | 1 | 1 | +| `v-2936a0a9` | `tests/admin_views/tests.py` | yes | no | no | 10 | 10 | +| `v-92e1d9e3` | `tests/admin_views/tests.py` | yes | no | no | 1 | 1 | +| `v-92470ad3` | `tests/admin_utils/tests.py` | yes | yes | yes | 11 | 10 | +| `v-4ea38d54` | `django/contrib/admin/options.py` | yes | no | no | 61 | 61 | +| `v-6fc81500` | `django/core/handlers/exception.py` | yes | no | no | 5 | 5 | + +### gh-cli (`git@github.com:cli/cli`, commit `5d3c4817f1619213951dbf15031bad04acb88392`) + +| Trial | Omitted file | could cite | hit | attributable | cited | noise | +|---|---|---|---|---|---:|---:| +| `v-e4efbc42` | `pkg/cmd/copilot/copilot_test.go` | yes | yes | yes | 2 | 1 | +| `v-1e04dab8` | `pkg/cmd/copilot/copilot.go` | yes | no | no | 0 | 1 | +| `v-2e9fedd3` | `git/client_test.go` | yes | yes | no | 7 | 7 | +| `v-a6bcd08d` | `pkg/cmd/project/item-add/item_add.go` | yes | no | no | 0 | 1 | +| `v-efe3f165` | `pkg/cmd/project/shared/queries/resolve_fields_test.go` | yes | yes | yes | 29 | 28 | +| `v-688751de` | `pkg/cmd/pr/checkout/checkout_test.go` | yes | yes | no | 2 | 2 | +| `v-9f14d1ac` | `pkg/cmd/pr/checkout/checkout_test.go` | yes | yes | no | 3 | 2 | +| `v-d5f4bed3` | `pkg/cmd/skills/update/update.go` | yes | no | no | 2 | 5 | +| `v-74e77914` | `internal/codespaces/connection/connection.go` | yes | no | no | 14 | 14 | +| `v-f1d11210` | `pkg/cmd/skills/install/install_test.go` | yes | no | no | 2 | 2 | +| `v-751dc5e0` | `pkg/cmd/release/shared/fetch.go` | yes | no | no | 0 | 6 | +| `v-517dae6a` | `internal/skills/registry/registry_test.go` | yes | no | no | 0 | 0 | +| `v-8d2b059e` | `pkg/cmd/discussion/view/view.go` | yes | no | no | 0 | 0 | +| `v-2618999b` | `pkg/cmd/discussion/client/client_impl_test.go` | yes | no | no | 0 | 0 | +| `v-e2d150da` | `pkg/cmd/discussion/edit/edit.go` | yes | no | no | 1 | 2 | +| `v-c1f3c1a1` | `pkg/cmd/discussion/view/view.go` | yes | no | no | 0 | 0 | +| `v-b1029009` | `pkg/cmd/discussion/edit/edit.go` | yes | yes | no | 5 | 5 | +| `v-16a20347` | `pkg/cmd/skills/update/update_test.go` | yes | yes | yes | 2 | 1 | +| `v-fb748cb2` | `pkg/cmd/skills/preview/preview_test.go` | yes | yes | no | 13 | 12 | +| `v-a44721d2` | `internal/prompter/echo_linux_test.go` | no | no | no | 0 | 0 | + diff --git a/benchmarks/results/verify-django/verify-environment.json b/benchmarks/results/verify-django/verify-environment.json new file mode 100644 index 0000000..7147bf0 --- /dev/null +++ b/benchmarks/results/verify-django/verify-environment.json @@ -0,0 +1,25 @@ +{ + "after": null, + "candidates_rejected": 3, + "checker": "symbols changed by this diff, minus symbols present in the diff, where an inbound CALLS edge exists at distance 1, via reify::query::impact", + "count": 20, + "head": "0b40210e4808937a7c0922e8b7502bff4752faa3", + "languages": [ + [ + "python", + 2014 + ], + [ + "javascript", + 16 + ] + ], + "origin": "git@github.com:django/django", + "reify_version": "0.2.2", + "repository": "/private/tmp/claude-501/-Users-lambiengcode--treehouse-reify-2e416f-2-reify/a9d50d0b-cce2-4027-a129-e64262e800a7/scratchpad/django", + "scan": 400, + "token_counts": "estimated by reify heuristic-v1", + "trials": 20, + "until": null, + "wall_clock_ms": 96781 +} \ No newline at end of file diff --git a/benchmarks/results/verify-django/verify-outcomes.json b/benchmarks/results/verify-django/verify-outcomes.json new file mode 100644 index 0000000..6adbaa0 --- /dev/null +++ b/benchmarks/results/verify-django/verify-outcomes.json @@ -0,0 +1,663 @@ +[ + { + "task": "v-d992705f", + "commit": "d992705f9eb56199dc474b77af16474e5ce3d2ab", + "omission_file": "tests/basic/models.py", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 5, + "verify_tokens": 23, + "verify_latency_ms": 1, + "index_ms": 4264, + "cited": [ + "django/__init__.py:8" + ], + "cited_on_complete": [ + "django/__init__.py:8" + ] + }, + { + "task": "v-be6cf832", + "commit": "be6cf832293779d8aeaeddca8c47a37ba1530898", + "omission_file": "django/db/models/sql/query.py", + "omission_symbol": "django/db/models/sql/query.py:1775", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 13, + "changed_symbols": 3, + "verify_tokens": 28, + "verify_latency_ms": 1, + "index_ms": 4761, + "cited": [ + "django/db/models/sql/compiler.py:757" + ], + "cited_on_complete": [ + "django/db/models/sql/compiler.py:757", + "django/db/models/sql/query.py:1344", + "django/db/models/sql/query.py:1891", + "django/db/models/sql/query.py:2327", + "django/db/models/sql/query.py:2588", + "tests/composite_pk/test_names_to_path.py:114", + "tests/composite_pk/test_names_to_path.py:18", + "tests/composite_pk/test_names_to_path.py:27", + "tests/composite_pk/test_names_to_path.py:51", + "tests/composite_pk/test_names_to_path.py:78", + "tests/composite_pk/test_names_to_path.py:9", + "tests/queries/test_query.py:193", + "tests/queries/test_query.py:203" + ] + }, + { + "task": "v-c72f5fb4", + "commit": "c72f5fb4793f2cac66d76e3cbd590499ad85b89e", + "omission_file": "tests/validators/tests.py", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 5, + "false_alarms": 5, + "changed_symbols": 1, + "verify_tokens": 121, + "verify_latency_ms": 0, + "index_ms": 4727, + "cited": [ + "django/contrib/admin/utils.py:433", + "django/db/models/fields/__init__.py:2720", + "django/forms/fields.py:772", + "tests/forms_tests/tests/test_validators.py:127", + "tests/forms_tests/tests/test_validators.py:75" + ], + "cited_on_complete": [ + "django/contrib/admin/utils.py:433", + "django/db/models/fields/__init__.py:2720", + "django/forms/fields.py:772", + "tests/forms_tests/tests/test_validators.py:127", + "tests/forms_tests/tests/test_validators.py:75" + ] + }, + { + "task": "v-1a001208", + "commit": "1a001208b0b9d79b15b27ca04d94f31f3d55d5ea", + "omission_file": "tests/ordering/tests.py", + "omission_symbol": "tests/ordering/tests.py:729", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 2, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 4732, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-07d4f69c", + "commit": "07d4f69c94a0e32c583b3aee5daf48fd81b4cd69", + "omission_file": "tests/migrations/test_operations.py", + "omission_symbol": "tests/migrations/test_operations.py:2576", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 4, + "verify_tokens": 17, + "verify_latency_ms": 1, + "index_ms": 5098, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-febefb17", + "commit": "febefb175e03352e5aeb2ed827024bacab96cf16", + "omission_file": "django/db/models/base.py", + "omission_symbol": "django/db/models/base.py:1553", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 2, + "changed_symbols": 3, + "verify_tokens": 36, + "verify_latency_ms": 1, + "index_ms": 4756, + "cited": [ + "tests/validation/test_unique.py:85" + ], + "cited_on_complete": [ + "django/db/models/base.py:1469", + "tests/validation/test_unique.py:85" + ] + }, + { + "task": "v-082b3df4", + "commit": "082b3df4067c3899dd4d57e8c2eca5baea9d07bb", + "omission_file": "tests/admin_changelist/tests.py", + "omission_symbol": "tests/admin_changelist/tests.py:1810", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 2, + "verify_tokens": 33, + "verify_latency_ms": 0, + "index_ms": 4691, + "cited": [ + "django/contrib/admin/templatetags/admin_list.py:334" + ], + "cited_on_complete": [ + "django/contrib/admin/templatetags/admin_list.py:334" + ] + }, + { + "task": "v-6df8fe3b", + "commit": "6df8fe3bc1879265958b8e59c637a4145995e93c", + "omission_file": "tests/admin_changelist/tests.py", + "omission_symbol": "tests/admin_changelist/tests.py:1796", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 3, + "false_alarms": 3, + "changed_symbols": 1, + "verify_tokens": 65, + "verify_latency_ms": 0, + "index_ms": 5047, + "cited": [ + "tests/model_inheritance/tests.py:217", + "tests/model_inheritance/tests.py:667", + "tests/model_inheritance/tests.py:677" + ], + "cited_on_complete": [ + "tests/model_inheritance/tests.py:217", + "tests/model_inheritance/tests.py:667", + "tests/model_inheritance/tests.py:677" + ] + }, + { + "task": "v-616e8c52", + "commit": "616e8c52ded7f4c7b00cae5a95f5a5d12a6a39b9", + "omission_file": "tests/admin_views/tests.py", + "omission_symbol": "tests/admin_views/tests.py:576", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 31, + "verify_latency_ms": 0, + "index_ms": 4650, + "cited": [ + "django/contrib/admin/options.py:2048" + ], + "cited_on_complete": [ + "django/contrib/admin/options.py:2048" + ] + }, + { + "task": "v-89e82866", + "commit": "89e82866dc2746383c336c7b10e050b9da3ae1ef", + "omission_file": "tests/admin_views/tests.py", + "omission_symbol": "tests/admin_views/tests.py:3144", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 32, + "verify_latency_ms": 0, + "index_ms": 4735, + "cited": [ + "django/contrib/admin/options.py:2048" + ], + "cited_on_complete": [ + "django/contrib/admin/options.py:2048" + ] + }, + { + "task": "v-47511a21", + "commit": "47511a21026cdd721d8fbf8571cc079bc38bb46d", + "omission_file": "tests/admin_utils/tests.py", + "omission_symbol": "tests/admin_utils/tests.py:243", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": true, + "findings": 15, + "false_alarms": 14, + "changed_symbols": 1, + "verify_tokens": 388, + "verify_latency_ms": 0, + "index_ms": 4693, + "cited": [ + "django/contrib/admin/helpers.py:269", + "django/contrib/admin/templatetags/admin_list.py:205", + "tests/admin_utils/tests.py:132", + "tests/admin_utils/tests.py:181", + "tests/admin_utils/tests.py:199", + "tests/admin_utils/tests.py:205", + "tests/admin_utils/tests.py:210", + "tests/admin_utils/tests.py:220", + "tests/admin_utils/tests.py:235", + "tests/admin_utils/tests.py:243", + "tests/admin_utils/tests.py:260", + "tests/admin_utils/tests.py:277", + "tests/admin_utils/tests.py:285", + "tests/postgres_tests/test_array.py:1550", + "tests/postgres_tests/test_array.py:1559" + ], + "cited_on_complete": [ + "django/contrib/admin/helpers.py:269", + "django/contrib/admin/templatetags/admin_list.py:205", + "tests/admin_utils/tests.py:132", + "tests/admin_utils/tests.py:181", + "tests/admin_utils/tests.py:199", + "tests/admin_utils/tests.py:205", + "tests/admin_utils/tests.py:210", + "tests/admin_utils/tests.py:220", + "tests/admin_utils/tests.py:235", + "tests/admin_utils/tests.py:260", + "tests/admin_utils/tests.py:277", + "tests/admin_utils/tests.py:285", + "tests/postgres_tests/test_array.py:1550", + "tests/postgres_tests/test_array.py:1559" + ] + }, + { + "task": "v-27137e65", + "commit": "27137e655e442e81095f1f8f77ff3870d9fdf169", + "omission_file": "django/test/signals.py", + "omission_symbol": "django/test/signals.py:146", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 6, + "false_alarms": 6, + "changed_symbols": 3, + "verify_tokens": 140, + "verify_latency_ms": 1, + "index_ms": 4656, + "cited": [ + "django/utils/formats.py:62", + "django/utils/translation/trans_real.py:495", + "django/utils/translation/trans_real.py:564", + "django/views/i18n.py:30", + "tests/i18n/tests.py:2066", + "tests/i18n/tests.py:2168" + ], + "cited_on_complete": [ + "django/utils/formats.py:62", + "django/utils/translation/trans_real.py:495", + "django/utils/translation/trans_real.py:564", + "django/views/i18n.py:30", + "tests/i18n/tests.py:2066", + "tests/i18n/tests.py:2168" + ] + }, + { + "task": "v-94653491", + "commit": "9465349120ef8a0b0689e12bcbfd05f3d173ebdf", + "omission_file": "tests/field_defaults/models.py", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 2, + "false_alarms": 2, + "changed_symbols": 2, + "verify_tokens": 44, + "verify_latency_ms": 0, + "index_ms": 5012, + "cited": [ + "django/db/models/base.py:1022", + "django/db/models/base.py:950" + ], + "cited_on_complete": [ + "django/db/models/base.py:1022", + "django/db/models/base.py:950" + ] + }, + { + "task": "v-ca14173f", + "commit": "ca14173f968cf36115f22d6c6785f738de4391ed", + "omission_file": "tests/urlpatterns/tests.py", + "omission_symbol": "tests/urlpatterns/tests.py:443", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 30, + "verify_latency_ms": 0, + "index_ms": 4749, + "cited": [ + "django/urls/utils.py:199" + ], + "cited_on_complete": [ + "django/urls/utils.py:199" + ] + }, + { + "task": "v-c9ff757a", + "commit": "c9ff757a55392b1f50968eb89fe775f6155168d8", + "omission_file": "tests/bulk_create/tests.py", + "omission_symbol": "tests/bulk_create/tests.py:442", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 31, + "verify_latency_ms": 1, + "index_ms": 4688, + "cited": [ + "django/db/models/query.py:817" + ], + "cited_on_complete": [ + "django/db/models/query.py:817" + ] + }, + { + "task": "v-2936a0a9", + "commit": "2936a0a99719e3c3777039a0d6968deecb55c752", + "omission_file": "tests/admin_views/tests.py", + "omission_symbol": "tests/admin_views/tests.py:802", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 10, + "false_alarms": 10, + "changed_symbols": 5, + "verify_tokens": 225, + "verify_latency_ms": 1, + "index_ms": 4754, + "cited": [ + "django/contrib/admin/helpers.py:204", + "django/contrib/admin/helpers.py:381", + "django/contrib/admin/templatetags/admin_list.py:88", + "django/contrib/admin/views/main.py:375", + "django/contrib/admin/views/main.py:424", + "tests/admin_utils/tests.py:353", + "tests/admin_utils/tests.py:399", + "tests/admin_utils/tests.py:420", + "tests/admin_utils/tests.py:436", + "tests/admin_utils/tests.py:458" + ], + "cited_on_complete": [ + "django/contrib/admin/helpers.py:204", + "django/contrib/admin/helpers.py:381", + "django/contrib/admin/templatetags/admin_list.py:88", + "django/contrib/admin/views/main.py:375", + "django/contrib/admin/views/main.py:424", + "tests/admin_utils/tests.py:353", + "tests/admin_utils/tests.py:399", + "tests/admin_utils/tests.py:420", + "tests/admin_utils/tests.py:436", + "tests/admin_utils/tests.py:458" + ] + }, + { + "task": "v-92e1d9e3", + "commit": "92e1d9e3619ae5274a64b38f26177064486892f2", + "omission_file": "tests/admin_views/tests.py", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 3, + "verify_tokens": 34, + "verify_latency_ms": 0, + "index_ms": 4666, + "cited": [ + "django/contrib/admin/templatetags/admin_list.py:348" + ], + "cited_on_complete": [ + "django/contrib/admin/templatetags/admin_list.py:348" + ] + }, + { + "task": "v-92470ad3", + "commit": "92470ad3742524902b29769d2c822dbe791630db", + "omission_file": "tests/admin_utils/tests.py", + "omission_symbol": "tests/admin_utils/tests.py:132", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": true, + "symbol_hit": true, + "findings": 11, + "false_alarms": 10, + "changed_symbols": 3, + "verify_tokens": 217, + "verify_latency_ms": 1, + "index_ms": 4680, + "cited": [ + "django/contrib/admin/helpers.py:269", + "django/contrib/admin/templatetags/admin_list.py:326", + "django/contrib/sites/management.py:11", + "tests/admin_utils/tests.py:132", + "tests/contenttypes_tests/test_views.py:29", + "tests/gis_tests/geoapp/test_feeds.py:16", + "tests/gis_tests/geoapp/test_sitemaps.py:18", + "tests/sites_tests/tests.py:143", + "tests/sites_tests/tests.py:196", + "tests/sites_tests/tests.py:24", + "tests/sites_tests/tests.py:311" + ], + "cited_on_complete": [ + "django/contrib/admin/helpers.py:269", + "django/contrib/admin/templatetags/admin_list.py:326", + "django/contrib/sites/management.py:11", + "tests/contenttypes_tests/test_views.py:29", + "tests/gis_tests/geoapp/test_feeds.py:16", + "tests/gis_tests/geoapp/test_sitemaps.py:18", + "tests/sites_tests/tests.py:143", + "tests/sites_tests/tests.py:196", + "tests/sites_tests/tests.py:24", + "tests/sites_tests/tests.py:311" + ] + }, + { + "task": "v-4ea38d54", + "commit": "4ea38d54c10e0f44e189605c217a55cdfe9fdde8", + "omission_file": "django/contrib/admin/options.py", + "omission_symbol": "django/contrib/admin/options.py:2558", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 61, + "false_alarms": 61, + "changed_symbols": 4, + "verify_tokens": 1315, + "verify_latency_ms": 2, + "index_ms": 4639, + "cited": [ + "django/contrib/admin/sites.py:325", + "tests/admin_checks/tests.py:1024", + "tests/admin_checks/tests.py:1042", + "tests/admin_checks/tests.py:1049", + "tests/admin_checks/tests.py:294", + "tests/admin_checks/tests.py:309", + "tests/admin_checks/tests.py:324", + "tests/admin_checks/tests.py:340", + "tests/admin_checks/tests.py:365", + "tests/admin_checks/tests.py:382", + "tests/admin_checks/tests.py:398", + "tests/admin_checks/tests.py:405", + "tests/admin_checks/tests.py:413", + "tests/admin_checks/tests.py:435", + "tests/admin_checks/tests.py:456", + "tests/admin_checks/tests.py:474", + "tests/admin_checks/tests.py:489", + "tests/admin_checks/tests.py:508", + "tests/admin_checks/tests.py:533", + "tests/admin_checks/tests.py:548", + "tests/admin_checks/tests.py:570", + "tests/admin_checks/tests.py:594", + "tests/admin_checks/tests.py:618", + "tests/admin_checks/tests.py:642", + "tests/admin_checks/tests.py:666", + "tests/admin_checks/tests.py:681", + "tests/admin_checks/tests.py:699", + "tests/admin_checks/tests.py:718", + "tests/admin_checks/tests.py:729", + "tests/admin_checks/tests.py:741", + "tests/admin_checks/tests.py:748", + "tests/admin_checks/tests.py:759", + "tests/admin_checks/tests.py:770", + "tests/admin_checks/tests.py:787", + "tests/admin_checks/tests.py:794", + "tests/admin_checks/tests.py:810", + "tests/admin_checks/tests.py:827", + "tests/admin_checks/tests.py:842", + "tests/admin_checks/tests.py:853", + "tests/admin_checks/tests.py:860", + "tests/admin_checks/tests.py:881", + "tests/admin_checks/tests.py:900", + "tests/admin_checks/tests.py:907", + "tests/admin_checks/tests.py:914", + "tests/admin_checks/tests.py:930", + "tests/admin_checks/tests.py:946", + "tests/admin_checks/tests.py:967", + "tests/admin_checks/tests.py:982", + "tests/admin_checks/tests.py:999", + "tests/admin_registration/tests.py:21", + "tests/admin_views/test_adminsite.py:106", + "tests/admin_views/tests.py:9338", + "tests/generic_inline_admin/tests.py:314", + "tests/generic_inline_admin/tests.py:344", + "tests/generic_inline_admin/tests.py:419", + "tests/generic_inline_admin/tests.py:422", + "tests/modeladmin/test_actions.py:30", + "tests/modeladmin/test_actions.py:67", + "tests/modeladmin/test_actions.py:89", + "tests/modeladmin/test_checks.py:18", + "tests/modeladmin/test_checks.py:1811" + ], + "cited_on_complete": [ + "django/contrib/admin/sites.py:325", + "tests/admin_checks/tests.py:1024", + "tests/admin_checks/tests.py:1042", + "tests/admin_checks/tests.py:1049", + "tests/admin_checks/tests.py:294", + "tests/admin_checks/tests.py:309", + "tests/admin_checks/tests.py:324", + "tests/admin_checks/tests.py:340", + "tests/admin_checks/tests.py:365", + "tests/admin_checks/tests.py:382", + "tests/admin_checks/tests.py:398", + "tests/admin_checks/tests.py:405", + "tests/admin_checks/tests.py:413", + "tests/admin_checks/tests.py:435", + "tests/admin_checks/tests.py:456", + "tests/admin_checks/tests.py:474", + "tests/admin_checks/tests.py:489", + "tests/admin_checks/tests.py:508", + "tests/admin_checks/tests.py:533", + "tests/admin_checks/tests.py:548", + "tests/admin_checks/tests.py:570", + "tests/admin_checks/tests.py:594", + "tests/admin_checks/tests.py:618", + "tests/admin_checks/tests.py:642", + "tests/admin_checks/tests.py:666", + "tests/admin_checks/tests.py:681", + "tests/admin_checks/tests.py:699", + "tests/admin_checks/tests.py:718", + "tests/admin_checks/tests.py:729", + "tests/admin_checks/tests.py:741", + "tests/admin_checks/tests.py:748", + "tests/admin_checks/tests.py:759", + "tests/admin_checks/tests.py:770", + "tests/admin_checks/tests.py:787", + "tests/admin_checks/tests.py:794", + "tests/admin_checks/tests.py:810", + "tests/admin_checks/tests.py:827", + "tests/admin_checks/tests.py:842", + "tests/admin_checks/tests.py:853", + "tests/admin_checks/tests.py:860", + "tests/admin_checks/tests.py:881", + "tests/admin_checks/tests.py:900", + "tests/admin_checks/tests.py:907", + "tests/admin_checks/tests.py:914", + "tests/admin_checks/tests.py:930", + "tests/admin_checks/tests.py:946", + "tests/admin_checks/tests.py:967", + "tests/admin_checks/tests.py:982", + "tests/admin_checks/tests.py:999", + "tests/admin_registration/tests.py:21", + "tests/admin_views/test_adminsite.py:106", + "tests/admin_views/tests.py:9338", + "tests/generic_inline_admin/tests.py:314", + "tests/generic_inline_admin/tests.py:344", + "tests/generic_inline_admin/tests.py:419", + "tests/generic_inline_admin/tests.py:422", + "tests/modeladmin/test_actions.py:30", + "tests/modeladmin/test_actions.py:67", + "tests/modeladmin/test_actions.py:89", + "tests/modeladmin/test_checks.py:18", + "tests/modeladmin/test_checks.py:1811" + ] + }, + { + "task": "v-6fc81500", + "commit": "6fc8150005256db2052b01812d65dff737563a1b", + "omission_file": "django/core/handlers/exception.py", + "omission_symbol": "django/core/handlers/exception.py:41", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 5, + "false_alarms": 5, + "changed_symbols": 6, + "verify_tokens": 113, + "verify_latency_ms": 1, + "index_ms": 4772, + "cited": [ + "django/contrib/staticfiles/handlers.py:103", + "django/core/handlers/asgi.py:228", + "django/test/client.py:220", + "tests/asgi/urls.py:60", + "tests/staticfiles_tests/test_handlers.py:18" + ], + "cited_on_complete": [ + "django/contrib/staticfiles/handlers.py:103", + "django/core/handlers/asgi.py:228", + "django/test/client.py:220", + "tests/asgi/urls.py:60", + "tests/staticfiles_tests/test_handlers.py:18" + ] + } +] \ No newline at end of file diff --git a/benchmarks/results/verify-django/verify-summary.json b/benchmarks/results/verify-django/verify-summary.json new file mode 100644 index 0000000..ec5f410 --- /dev/null +++ b/benchmarks/results/verify-django/verify-summary.json @@ -0,0 +1,36 @@ +{ + "tasks": 20, + "omission_recall": 0.1, + "omission_recall_ci": [ + 0.027865902, + 0.3010382 + ], + "omission_recall_attributable": 0.05, + "omission_recall_attributable_ci": [ + 0.008881226, + 0.2361359 + ], + "reachable_omissions": 20, + "omission_recall_reachable": 0.1, + "omission_recall_reachable_ci": [ + 0.027865902, + 0.3010382 + ], + "symbol_scorable": 16, + "omission_recall_symbol": 0.125, + "omission_recall_symbol_ci": [ + 0.034976766, + 0.3602333 + ], + "false_alarm_rate": 6.9, + "commits_with_a_false_alarm": 18, + "false_alarm_share_ci": [ + 0.69896173, + 0.9721341 + ], + "median_findings_per_diff": 1, + "median_verify_tokens": 36, + "median_verify_latency_ms": 1, + "median_index_ms": 4732, + "diffs_resolving_to_nothing": 0 +} \ No newline at end of file diff --git a/benchmarks/results/verify-django/verify-tasks.json b/benchmarks/results/verify-django/verify-tasks.json new file mode 100644 index 0000000..e61c2ec --- /dev/null +++ b/benchmarks/results/verify-django/verify-tasks.json @@ -0,0 +1,2386 @@ +{ + "repository": "/private/tmp/claude-501/-Users-lambiengcode--treehouse-reify-2e416f-2-reify/a9d50d0b-cce2-4027-a129-e64262e800a7/scratchpad/django", + "head": "0b40210e4808937a7c0922e8b7502bff4752faa3", + "generated_from_commits": 400, + "rejected": [ + [ + "cef2346abf8d6e9e61a5a3599fbcf72163e6a6e5", + "no file changed by exactly one hunk" + ], + [ + "f1949c1f9758947ade984c895ff16bef46f56520", + "no file changed by exactly one hunk" + ], + [ + "60121939f6b225c7a719dd561e372e1d8e5e2c4a", + "no file changed by exactly one hunk" + ] + ], + "tasks": [ + { + "id": "v-d992705f", + "commit": "d992705f9eb56199dc474b77af16474e5ce3d2ab", + "parent": "504d1f12cd5879177295fecd2ba4da1001a0930f", + "date": "2026-08-08", + "prompt": "Fixed #37259 -- Restored support for old-signature Model.from_db overrides.", + "complete": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 6, + "old_len": 7, + "changed_lines": [ + 9 + ] + }, + { + "old_start": 52, + "old_len": 6, + "changed_lines": [ + 55 + ] + }, + { + "old_start": 63, + "old_len": 6, + "changed_lines": [ + 66 + ] + }, + { + "old_start": 123, + "old_len": 6, + "changed_lines": [ + 126 + ] + }, + { + "old_start": 142, + "old_len": 11, + "changed_lines": [ + 145, + 149 + ] + }, + { + "old_start": 213, + "old_len": 13, + "changed_lines": [ + 216, + 220, + 221, + 222 + ] + }, + { + "old_start": 3044, + "old_len": 6, + "changed_lines": [ + 3047 + ] + }, + { + "old_start": 3063, + "old_len": 11, + "changed_lines": [ + 3066, + 3070 + ] + } + ] + }, + { + "path": "tests/basic/models.py", + "created": false, + "hunks": [ + { + "old_start": 61, + "old_len": 3, + "changed_lines": [ + 64 + ] + } + ] + }, + { + "path": "tests/basic/tests.py", + "created": false, + "hunks": [ + { + "old_start": 24, + "old_len": 6, + "changed_lines": [ + 27 + ] + }, + { + "old_start": 31, + "old_len": 6, + "changed_lines": [ + 34 + ] + }, + { + "old_start": 1120, + "old_len": 3, + "changed_lines": [ + 1123 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 6, + "old_len": 7, + "changed_lines": [ + 9 + ] + }, + { + "old_start": 52, + "old_len": 6, + "changed_lines": [ + 55 + ] + }, + { + "old_start": 63, + "old_len": 6, + "changed_lines": [ + 66 + ] + }, + { + "old_start": 123, + "old_len": 6, + "changed_lines": [ + 126 + ] + }, + { + "old_start": 142, + "old_len": 11, + "changed_lines": [ + 145, + 149 + ] + }, + { + "old_start": 213, + "old_len": 13, + "changed_lines": [ + 216, + 220, + 221, + 222 + ] + }, + { + "old_start": 3044, + "old_len": 6, + "changed_lines": [ + 3047 + ] + }, + { + "old_start": 3063, + "old_len": 11, + "changed_lines": [ + 3066, + 3070 + ] + } + ] + }, + { + "path": "tests/basic/tests.py", + "created": false, + "hunks": [ + { + "old_start": 24, + "old_len": 6, + "changed_lines": [ + 27 + ] + }, + { + "old_start": 31, + "old_len": 6, + "changed_lines": [ + 34 + ] + }, + { + "old_start": 1120, + "old_len": 3, + "changed_lines": [ + 1123 + ] + } + ] + } + ] + }, + "omission_file": "tests/basic/models.py", + "omission_line": 64 + }, + { + "id": "v-be6cf832", + "commit": "be6cf832293779d8aeaeddca8c47a37ba1530898", + "parent": "c72f5fb4793f2cac66d76e3cbd590499ad85b89e", + "date": "2026-08-13", + "prompt": "Fixed #37274 -- Allowed transforms in order_by after alias.", + "complete": { + "files": [ + { + "path": "django/db/models/sql/compiler.py", + "created": false, + "hunks": [ + { + "old_start": 1052, + "old_len": 6, + "changed_lines": [ + 1055 + ] + } + ] + }, + { + "path": "django/db/models/sql/query.py", + "created": false, + "hunks": [ + { + "old_start": 1800, + "old_len": 8, + "changed_lines": [ + 1803, + 1804 + ] + } + ] + }, + { + "path": "tests/annotations/tests.py", + "created": false, + "hunks": [ + { + "old_start": 1481, + "old_len": 6, + "changed_lines": [ + 1484 + ] + }, + { + "old_start": 1536, + "old_len": 9, + "changed_lines": [ + 1539, + 1541 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/sql/compiler.py", + "created": false, + "hunks": [ + { + "old_start": 1052, + "old_len": 6, + "changed_lines": [ + 1055 + ] + } + ] + }, + { + "path": "tests/annotations/tests.py", + "created": false, + "hunks": [ + { + "old_start": 1481, + "old_len": 6, + "changed_lines": [ + 1484 + ] + }, + { + "old_start": 1536, + "old_len": 9, + "changed_lines": [ + 1539, + 1541 + ] + } + ] + } + ] + }, + "omission_file": "django/db/models/sql/query.py", + "omission_line": 1803 + }, + { + "id": "v-c72f5fb4", + "commit": "c72f5fb4793f2cac66d76e3cbd590499ad85b89e", + "parent": "f7610bda78afb13ee395dfee2445805d8c7ad0f6", + "date": "2026-08-14", + "prompt": "Fixed #37279 -- Rejected null characters in URLValidator.", + "complete": { + "files": [ + { + "path": "django/core/validators.py", + "created": false, + "hunks": [ + { + "old_start": 152, + "old_len": 7, + "changed_lines": [ + 155 + ] + } + ] + }, + { + "path": "tests/validators/tests.py", + "created": false, + "hunks": [ + { + "old_start": 265, + "old_len": 6, + "changed_lines": [ + 268 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/core/validators.py", + "created": false, + "hunks": [ + { + "old_start": 152, + "old_len": 7, + "changed_lines": [ + 155 + ] + } + ] + } + ] + }, + "omission_file": "tests/validators/tests.py", + "omission_line": 268 + }, + { + "id": "v-1a001208", + "commit": "1a001208b0b9d79b15b27ca04d94f31f3d55d5ea", + "parent": "3436cf9bce84bb1f6877ad96819637366b27b719", + "date": "2026-08-14", + "prompt": "Fixed #37278 -- Made QuerySet.totally_ordered understand aliases to pure Col/ColPairs.", + "complete": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 26, + "old_len": 7, + "changed_lines": [ + 29 + ] + }, + { + "old_start": 2048, + "old_len": 7, + "changed_lines": [ + 2051 + ] + }, + { + "old_start": 2058, + "old_len": 7, + "changed_lines": [ + 2061 + ] + }, + { + "old_start": 2068, + "old_len": 18, + "changed_lines": [ + 2071, + 2072, + 2073, + 2074, + 2075, + 2076, + 2077, + 2078, + 2079, + 2082 + ] + }, + { + "old_start": 2097, + "old_len": 7, + "changed_lines": [ + 2100 + ] + } + ] + }, + { + "path": "tests/composite_pk/test_order_by.py", + "created": false, + "hunks": [ + { + "old_start": 70, + "old_len": 3, + "changed_lines": [ + 73 + ] + } + ] + }, + { + "path": "tests/ordering/models.py", + "created": false, + "hunks": [ + { + "old_start": 85, + "old_len": 10, + "changed_lines": [ + 88, + 92 + ] + } + ] + }, + { + "path": "tests/ordering/tests.py", + "created": false, + "hunks": [ + { + "old_start": 726, + "old_len": 6, + "changed_lines": [ + 729 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 26, + "old_len": 7, + "changed_lines": [ + 29 + ] + }, + { + "old_start": 2048, + "old_len": 7, + "changed_lines": [ + 2051 + ] + }, + { + "old_start": 2058, + "old_len": 7, + "changed_lines": [ + 2061 + ] + }, + { + "old_start": 2068, + "old_len": 18, + "changed_lines": [ + 2071, + 2072, + 2073, + 2074, + 2075, + 2076, + 2077, + 2078, + 2079, + 2082 + ] + }, + { + "old_start": 2097, + "old_len": 7, + "changed_lines": [ + 2100 + ] + } + ] + }, + { + "path": "tests/composite_pk/test_order_by.py", + "created": false, + "hunks": [ + { + "old_start": 70, + "old_len": 3, + "changed_lines": [ + 73 + ] + } + ] + }, + { + "path": "tests/ordering/models.py", + "created": false, + "hunks": [ + { + "old_start": 85, + "old_len": 10, + "changed_lines": [ + 88, + 92 + ] + } + ] + } + ] + }, + "omission_file": "tests/ordering/tests.py", + "omission_line": 729 + }, + { + "id": "v-07d4f69c", + "commit": "07d4f69c94a0e32c583b3aee5daf48fd81b4cd69", + "parent": "4ee04972e7f9163dbdf5a7c36330e3379187e187", + "date": "2026-08-08", + "prompt": "Fixed #37260 -- Made alterations between Python on_delete options noops.", + "complete": { + "files": [ + { + "path": "django/db/models/fields/related.py", + "created": false, + "hunks": [ + { + "old_start": 601, + "old_len": 6, + "changed_lines": [ + 604 + ] + } + ] + }, + { + "path": "tests/migrations/test_operations.py", + "created": false, + "hunks": [ + { + "old_start": 2573, + "old_len": 6, + "changed_lines": [ + 2576 + ] + } + ] + }, + { + "path": "tests/schema/tests.py", + "created": false, + "hunks": [ + { + "old_start": 635, + "old_len": 10, + "changed_lines": [ + 638, + 642 + ] + }, + { + "old_start": 4926, + "old_len": 7, + "changed_lines": [ + 4929 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/fields/related.py", + "created": false, + "hunks": [ + { + "old_start": 601, + "old_len": 6, + "changed_lines": [ + 604 + ] + } + ] + }, + { + "path": "tests/schema/tests.py", + "created": false, + "hunks": [ + { + "old_start": 635, + "old_len": 10, + "changed_lines": [ + 638, + 642 + ] + }, + { + "old_start": 4926, + "old_len": 7, + "changed_lines": [ + 4929 + ] + } + ] + } + ] + }, + "omission_file": "tests/migrations/test_operations.py", + "omission_line": 2576 + }, + { + "id": "v-febefb17", + "commit": "febefb175e03352e5aeb2ed827024bacab96cf16", + "parent": "812c08bd4e9da7b74ab9ee0db83da58a6da48d19", + "date": "2026-08-14", + "prompt": "Fixed #37248 -- Skipped unique validation of a dynamic DatabaseDefault expression.", + "complete": { + "files": [ + { + "path": "django/db/models/base.py", + "created": false, + "hunks": [ + { + "old_start": 1562, + "old_len": 9, + "changed_lines": [ + 1565, + 1566, + 1567 + ] + } + ] + }, + { + "path": "django/db/models/constraints.py", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 7, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 605, + "old_len": 6, + "changed_lines": [ + 608 + ] + } + ] + }, + { + "path": "tests/constraints/models.py", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 5, + "changed_lines": [ + 2 + ] + }, + { + "old_start": 175, + "old_len": 3, + "changed_lines": [ + 178 + ] + } + ] + }, + { + "path": "tests/constraints/tests.py", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 1503, + "old_len": 3, + "changed_lines": [ + 1506 + ] + } + ] + }, + { + "path": "tests/validation/models.py", + "created": false, + "hunks": [ + { + "old_start": 2, + "old_len": 7, + "changed_lines": [ + 5 + ] + }, + { + "old_start": 52, + "old_len": 6, + "changed_lines": [ + 55 + ] + } + ] + }, + { + "path": "tests/validation/test_unique.py", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 17, + "changed_lines": [ + 7, + 14, + 18 + ] + }, + { + "old_start": 160, + "old_len": 6, + "changed_lines": [ + 163 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/constraints.py", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 7, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 605, + "old_len": 6, + "changed_lines": [ + 608 + ] + } + ] + }, + { + "path": "tests/constraints/models.py", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 5, + "changed_lines": [ + 2 + ] + }, + { + "old_start": 175, + "old_len": 3, + "changed_lines": [ + 178 + ] + } + ] + }, + { + "path": "tests/constraints/tests.py", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 1503, + "old_len": 3, + "changed_lines": [ + 1506 + ] + } + ] + }, + { + "path": "tests/validation/models.py", + "created": false, + "hunks": [ + { + "old_start": 2, + "old_len": 7, + "changed_lines": [ + 5 + ] + }, + { + "old_start": 52, + "old_len": 6, + "changed_lines": [ + 55 + ] + } + ] + }, + { + "path": "tests/validation/test_unique.py", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 17, + "changed_lines": [ + 7, + 14, + 18 + ] + }, + { + "old_start": 160, + "old_len": 6, + "changed_lines": [ + 163 + ] + } + ] + } + ] + }, + "omission_file": "django/db/models/base.py", + "omission_line": 1565 + }, + { + "id": "v-082b3df4", + "commit": "082b3df4067c3899dd4d57e8c2eca5baea9d07bb", + "parent": "6df8fe3bc1879265958b8e59c637a4145995e93c", + "date": "2026-08-10", + "prompt": "Fixed #37270 -- Fixed incorrect values for second-degree relations in ModelAdmin.list_display.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 228, + "old_len": 7, + "changed_lines": [ + 231 + ] + }, + { + "old_start": 246, + "old_len": 11, + "changed_lines": [ + 249, + 250, + 253 + ] + } + ] + }, + { + "path": "tests/admin_changelist/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + }, + { + "path": "tests/admin_changelist/tests.py", + "created": false, + "hunks": [ + { + "old_start": 1807, + "old_len": 6, + "changed_lines": [ + 1810 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 228, + "old_len": 7, + "changed_lines": [ + 231 + ] + }, + { + "old_start": 246, + "old_len": 11, + "changed_lines": [ + 249, + 250, + 253 + ] + } + ] + }, + { + "path": "tests/admin_changelist/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_changelist/tests.py", + "omission_line": 1810 + }, + { + "id": "v-6df8fe3b", + "commit": "6df8fe3bc1879265958b8e59c637a4145995e93c", + "parent": "616e8c52ded7f4c7b00cae5a95f5a5d12a6a39b9", + "date": "2026-08-10", + "prompt": "Fixed #24580 -- Tested FK values with __html__ in ModelAdmin.list_display.", + "complete": { + "files": [ + { + "path": "tests/admin_changelist/models.py", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + } + ] + }, + { + "path": "tests/admin_changelist/tests.py", + "created": false, + "hunks": [ + { + "old_start": 1793, + "old_len": 6, + "changed_lines": [ + 1796 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "tests/admin_changelist/models.py", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_changelist/tests.py", + "omission_line": 1796 + }, + { + "id": "v-616e8c52", + "commit": "616e8c52ded7f4c7b00cae5a95f5a5d12a6a39b9", + "parent": "2b4c88b2ce753a44b5ba867e5e52a20e07b258e6", + "date": "2026-08-08", + "prompt": "Fixed #37264 -- Handled further malformed _source_model values in admin popups.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/options.py", + "created": false, + "hunks": [ + { + "old_start": 1594, + "old_len": 10, + "changed_lines": [ + 1597, + 1599, + 1600 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 576, + "old_len": 23, + "changed_lines": [ + 579, + 581, + 582, + 583, + 584, + 585, + 586, + 587, + 588, + 589, + 590, + 591, + 592, + 593, + 594, + 595 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/options.py", + "created": false, + "hunks": [ + { + "old_start": 1594, + "old_len": 10, + "changed_lines": [ + 1597, + 1599, + 1600 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_views/tests.py", + "omission_line": 579 + }, + { + "id": "v-89e82866", + "commit": "89e82866dc2746383c336c7b10e050b9da3ae1ef", + "parent": "dfc52e53f1d19a2730854d68b602fb4dba8bf0c5", + "date": "2026-03-01", + "prompt": "Fixed #29969 -- Omitted inlines without add permission on save-as-new.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/options.py", + "created": false, + "hunks": [ + { + "old_start": 2646, + "old_len": 6, + "changed_lines": [ + 2649 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 3141, + "old_len": 6, + "changed_lines": [ + 3144 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/options.py", + "created": false, + "hunks": [ + { + "old_start": 2646, + "old_len": 6, + "changed_lines": [ + 2649 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_views/tests.py", + "omission_line": 3144 + }, + { + "id": "v-47511a21", + "commit": "47511a21026cdd721d8fbf8571cc079bc38bb46d", + "parent": "d2e59b77fe18de318a8272c2a7bbc798d84d1d0d", + "date": "2026-07-13", + "prompt": "Fixed CVE-2026-15920 -- Made display_for_field validate URLs before rendering admin links.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 7, + "old_len": 8, + "changed_lines": [ + 10, + 11 + ] + }, + { + "old_start": 464, + "old_len": 6, + "changed_lines": [ + 467 + ] + } + ] + }, + { + "path": "tests/admin_utils/tests.py", + "created": false, + "hunks": [ + { + "old_start": 240, + "old_len": 6, + "changed_lines": [ + 243 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 7, + "old_len": 8, + "changed_lines": [ + 10, + 11 + ] + }, + { + "old_start": 464, + "old_len": 6, + "changed_lines": [ + 467 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_utils/tests.py", + "omission_line": 243 + }, + { + "id": "v-27137e65", + "commit": "27137e655e442e81095f1f8f77ff3870d9fdf169", + "parent": "f1949c1f9758947ade984c895ff16bef46f56520", + "date": "2026-07-10", + "prompt": "Fixed CVE-2026-15337 -- Mitigated potential DoS in check_for_language.", + "complete": { + "files": [ + { + "path": "django/test/signals.py", + "created": false, + "hunks": [ + { + "old_start": 153, + "old_len": 7, + "changed_lines": [ + 156 + ] + } + ] + }, + { + "path": "django/utils/translation/trans_real.py", + "created": false, + "hunks": [ + { + "old_start": 31, + "old_len": 9, + "changed_lines": [ + 34, + 35, + 36 + ] + }, + { + "old_start": 65, + "old_len": 7, + "changed_lines": [ + 68 + ] + }, + { + "old_start": 462, + "old_len": 19, + "changed_lines": [ + 465, + 472, + 473, + 477 + ] + } + ] + }, + { + "path": "tests/i18n/tests.py", + "created": false, + "hunks": [ + { + "old_start": 59, + "old_len": 7, + "changed_lines": [ + 62 + ] + }, + { + "old_start": 2081, + "old_len": 6, + "changed_lines": [ + 2084 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/utils/translation/trans_real.py", + "created": false, + "hunks": [ + { + "old_start": 31, + "old_len": 9, + "changed_lines": [ + 34, + 35, + 36 + ] + }, + { + "old_start": 65, + "old_len": 7, + "changed_lines": [ + 68 + ] + }, + { + "old_start": 462, + "old_len": 19, + "changed_lines": [ + 465, + 472, + 473, + 477 + ] + } + ] + }, + { + "path": "tests/i18n/tests.py", + "created": false, + "hunks": [ + { + "old_start": 59, + "old_len": 7, + "changed_lines": [ + 62 + ] + }, + { + "old_start": 2081, + "old_len": 6, + "changed_lines": [ + 2084 + ] + } + ] + } + ] + }, + "omission_file": "django/test/signals.py", + "omission_line": 156 + }, + { + "id": "v-94653491", + "commit": "9465349120ef8a0b0689e12bcbfd05f3d173ebdf", + "parent": "8c83e9c0ea39099b213478a893cbebe9faa837ba", + "date": "2026-07-28", + "prompt": "Fixed #37238 -- Prevented fallback to python default for a pk with a db_default.", + "complete": { + "files": [ + { + "path": "django/db/models/base.py", + "created": false, + "hunks": [ + { + "old_start": 1094, + "old_len": 7, + "changed_lines": [ + 1097 + ] + } + ] + }, + { + "path": "tests/field_defaults/models.py", + "created": false, + "hunks": [ + { + "old_start": 68, + "old_len": 3, + "changed_lines": [ + 71 + ] + } + ] + }, + { + "path": "tests/field_defaults/tests.py", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + }, + { + "old_start": 137, + "old_len": 6, + "changed_lines": [ + 140 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/base.py", + "created": false, + "hunks": [ + { + "old_start": 1094, + "old_len": 7, + "changed_lines": [ + 1097 + ] + } + ] + }, + { + "path": "tests/field_defaults/tests.py", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + }, + { + "old_start": 137, + "old_len": 6, + "changed_lines": [ + 140 + ] + } + ] + } + ] + }, + "omission_file": "tests/field_defaults/models.py", + "omission_line": 71 + }, + { + "id": "v-ca14173f", + "commit": "ca14173f968cf36115f22d6c6785f738de4391ed", + "parent": "1c5927f04a853c79ac9b098eab92fb328ff9e4ad", + "date": "2026-07-30", + "prompt": "Fixed #37240 -- Fixed simplify_regex with multiple unnamed groups.", + "complete": { + "files": [ + { + "path": "django/urls/utils.py", + "created": false, + "hunks": [ + { + "old_start": 136, + "old_len": 12, + "changed_lines": [ + 139, + 142, + 143, + 144 + ] + } + ] + }, + { + "path": "tests/urlpatterns/tests.py", + "created": false, + "hunks": [ + { + "old_start": 454, + "old_len": 6, + "changed_lines": [ + 457 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/urls/utils.py", + "created": false, + "hunks": [ + { + "old_start": 136, + "old_len": 12, + "changed_lines": [ + 139, + 142, + 143, + 144 + ] + } + ] + } + ] + }, + "omission_file": "tests/urlpatterns/tests.py", + "omission_line": 457 + }, + { + "id": "v-c9ff757a", + "commit": "c9ff757a55392b1f50968eb89fe775f6155168d8", + "parent": "2936a0a99719e3c3777039a0d6968deecb55c752", + "date": "2026-07-27", + "prompt": "Fixed #37234 -- Fixed bulk_create for late-saved related primary keys.", + "complete": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 747, + "old_len": 6, + "changed_lines": [ + 750 + ] + }, + { + "old_start": 757, + "old_len": 7, + "changed_lines": [ + 760 + ] + } + ] + }, + { + "path": "tests/bulk_create/tests.py", + "created": false, + "hunks": [ + { + "old_start": 439, + "old_len": 6, + "changed_lines": [ + 442 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/db/models/query.py", + "created": false, + "hunks": [ + { + "old_start": 747, + "old_len": 6, + "changed_lines": [ + 750 + ] + }, + { + "old_start": 757, + "old_len": 7, + "changed_lines": [ + 760 + ] + } + ] + } + ] + }, + "omission_file": "tests/bulk_create/tests.py", + "omission_line": 442 + }, + { + "id": "v-2936a0a9", + "commit": "2936a0a99719e3c3777039a0d6968deecb55c752", + "parent": "e1feeee45ea8bcd4325554c9b94fcd75fcd8dbdc", + "date": "2026-01-03", + "prompt": "Fixed #27752 -- Fixed ordering by Model.__str__ in the admin.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 372, + "old_len": 7, + "changed_lines": [ + 375 + ] + } + ] + }, + { + "path": "django/contrib/admin/views/main.py", + "created": false, + "hunks": [ + { + "old_start": 359, + "old_len": 7, + "changed_lines": [ + 362 + ] + } + ] + }, + { + "path": "tests/admin_utils/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + }, + { + "path": "tests/admin_utils/tests.py", + "created": false, + "hunks": [ + { + "old_start": 445, + "old_len": 6, + "changed_lines": [ + 448 + ] + } + ] + }, + { + "path": "tests/admin_views/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 799, + "old_len": 6, + "changed_lines": [ + 802 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 372, + "old_len": 7, + "changed_lines": [ + 375 + ] + } + ] + }, + { + "path": "django/contrib/admin/views/main.py", + "created": false, + "hunks": [ + { + "old_start": 359, + "old_len": 7, + "changed_lines": [ + 362 + ] + } + ] + }, + { + "path": "tests/admin_utils/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + }, + { + "path": "tests/admin_utils/tests.py", + "created": false, + "hunks": [ + { + "old_start": 445, + "old_len": 6, + "changed_lines": [ + 448 + ] + } + ] + }, + { + "path": "tests/admin_views/models.py", + "created": false, + "hunks": [ + { + "old_start": 47, + "old_len": 6, + "changed_lines": [ + 50 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_views/tests.py", + "omission_line": 802 + }, + { + "id": "v-92e1d9e3", + "commit": "92e1d9e3619ae5274a64b38f26177064486892f2", + "parent": "50e5264a458961134d34d6340a00d9a1b269df7a", + "date": "2026-07-27", + "prompt": "Fixed #37233 -- Prevented sort controls for unordered __str__ admin columns.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 116, + "old_len": 7, + "changed_lines": [ + 119 + ] + } + ] + }, + { + "path": "tests/admin_changelist/tests.py", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 7, + "changed_lines": [ + 7 + ] + }, + { + "old_start": 115, + "old_len": 6, + "changed_lines": [ + 118 + ] + }, + { + "old_start": 1726, + "old_len": 6, + "changed_lines": [ + 1729 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 4323, + "old_len": 6, + "changed_lines": [ + 4326 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 116, + "old_len": 7, + "changed_lines": [ + 119 + ] + } + ] + }, + { + "path": "tests/admin_changelist/tests.py", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 7, + "changed_lines": [ + 7 + ] + }, + { + "old_start": 115, + "old_len": 6, + "changed_lines": [ + 118 + ] + }, + { + "old_start": 1726, + "old_len": 6, + "changed_lines": [ + 1729 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_views/tests.py", + "omission_line": 4326 + }, + { + "id": "v-92470ad3", + "commit": "92470ad3742524902b29769d2c822dbe791630db", + "parent": "09c8b50bc8e59f7ec2d97df1bfb3fbd3ae0d4522", + "date": "2026-07-26", + "prompt": "Fixed #37230 -- Fixed a crash for second-degree relations in ModelAdmin.list_display.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 7, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 226, + "old_len": 6, + "changed_lines": [ + 229 + ] + } + ] + }, + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 314, + "old_len": 9, + "changed_lines": [ + 317, + 318, + 319 + ] + } + ] + }, + { + "path": "tests/admin_utils/models.py", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + } + ] + }, + { + "path": "tests/admin_utils/tests.py", + "created": false, + "hunks": [ + { + "old_start": 164, + "old_len": 6, + "changed_lines": [ + 167 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/templatetags/admin_list.py", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 7, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 226, + "old_len": 6, + "changed_lines": [ + 229 + ] + } + ] + }, + { + "path": "django/contrib/admin/utils.py", + "created": false, + "hunks": [ + { + "old_start": 314, + "old_len": 9, + "changed_lines": [ + 317, + 318, + 319 + ] + } + ] + }, + { + "path": "tests/admin_utils/models.py", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + } + ] + } + ] + }, + "omission_file": "tests/admin_utils/tests.py", + "omission_line": 167 + }, + { + "id": "v-4ea38d54", + "commit": "4ea38d54c10e0f44e189605c217a55cdfe9fdde8", + "parent": "8a162076e1988ddf9453edfb8329d5df1573dc38", + "date": "2026-06-10", + "prompt": "Fixed #37160 -- Made admin views raise PermissionDenied consistently.", + "complete": { + "files": [ + { + "path": "django/contrib/admin/options.py", + "created": false, + "hunks": [ + { + "old_start": 2563, + "old_len": 14, + "changed_lines": [ + 2566, + 2571, + 2572, + 2573 + ] + } + ] + }, + { + "path": "django/contrib/admin/sites.py", + "created": false, + "hunks": [ + { + "old_start": 9, + "old_len": 7, + "changed_lines": [ + 12 + ] + }, + { + "old_start": 256, + "old_len": 10, + "changed_lines": [ + 259, + 260, + 261, + 262 + ] + }, + { + "old_start": 290, + "old_len": 7, + "changed_lines": [ + 293 + ] + }, + { + "old_start": 451, + "old_len": 6, + "changed_lines": [ + 454 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 3380, + "old_len": 6, + "changed_lines": [ + 3383 + ] + }, + { + "old_start": 3527, + "old_len": 6, + "changed_lines": [ + 3530 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/admin/sites.py", + "created": false, + "hunks": [ + { + "old_start": 9, + "old_len": 7, + "changed_lines": [ + 12 + ] + }, + { + "old_start": 256, + "old_len": 10, + "changed_lines": [ + 259, + 260, + 261, + 262 + ] + }, + { + "old_start": 290, + "old_len": 7, + "changed_lines": [ + 293 + ] + }, + { + "old_start": 451, + "old_len": 6, + "changed_lines": [ + 454 + ] + } + ] + }, + { + "path": "tests/admin_views/tests.py", + "created": false, + "hunks": [ + { + "old_start": 3380, + "old_len": 6, + "changed_lines": [ + 3383 + ] + }, + { + "old_start": 3527, + "old_len": 6, + "changed_lines": [ + 3530 + ] + } + ] + } + ] + }, + "omission_file": "django/contrib/admin/options.py", + "omission_line": 2566 + }, + { + "id": "v-6fc81500", + "commit": "6fc8150005256db2052b01812d65dff737563a1b", + "parent": "2a5da9d00555beef8e5f6307cfcbfc029d45491e", + "date": "2026-07-18", + "prompt": "Fixed #36027 -- Made error response rendering thread-sensitive.", + "complete": { + "files": [ + { + "path": "django/contrib/staticfiles/handlers.py", + "created": false, + "hunks": [ + { + "old_start": 59, + "old_len": 7, + "changed_lines": [ + 62 + ] + } + ] + }, + { + "path": "django/core/handlers/exception.py", + "created": false, + "hunks": [ + { + "old_start": 43, + "old_len": 7, + "changed_lines": [ + 46 + ] + } + ] + }, + { + "path": "tests/asgi/tests.py", + "created": false, + "hunks": [ + { + "old_start": 30, + "old_len": 7, + "changed_lines": [ + 33 + ] + }, + { + "old_start": 497, + "old_len": 6, + "changed_lines": [ + 500 + ] + } + ] + }, + { + "path": "tests/asgi/urls.py", + "created": false, + "hunks": [ + { + "old_start": 2, + "old_len": 6, + "changed_lines": [ + 5 + ] + }, + { + "old_start": 51, + "old_len": 6, + "changed_lines": [ + 54 + ] + }, + { + "old_start": 73, + "old_len": 5, + "changed_lines": [ + 76 + ] + } + ] + }, + { + "path": "tests/staticfiles_tests/test_handlers.py", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 6, + "changed_lines": [ + 1, + 3 + ] + }, + { + "old_start": 12, + "old_len": 6, + "changed_lines": [ + 15 + ] + }, + { + "old_start": 23, + "old_len": 11, + "changed_lines": [ + 26, + 28, + 31 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "django/contrib/staticfiles/handlers.py", + "created": false, + "hunks": [ + { + "old_start": 59, + "old_len": 7, + "changed_lines": [ + 62 + ] + } + ] + }, + { + "path": "tests/asgi/tests.py", + "created": false, + "hunks": [ + { + "old_start": 30, + "old_len": 7, + "changed_lines": [ + 33 + ] + }, + { + "old_start": 497, + "old_len": 6, + "changed_lines": [ + 500 + ] + } + ] + }, + { + "path": "tests/asgi/urls.py", + "created": false, + "hunks": [ + { + "old_start": 2, + "old_len": 6, + "changed_lines": [ + 5 + ] + }, + { + "old_start": 51, + "old_len": 6, + "changed_lines": [ + 54 + ] + }, + { + "old_start": 73, + "old_len": 5, + "changed_lines": [ + 76 + ] + } + ] + }, + { + "path": "tests/staticfiles_tests/test_handlers.py", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 6, + "changed_lines": [ + 1, + 3 + ] + }, + { + "old_start": 12, + "old_len": 6, + "changed_lines": [ + 15 + ] + }, + { + "old_start": 23, + "old_len": 11, + "changed_lines": [ + 26, + 28, + 31 + ] + } + ] + } + ] + }, + "omission_file": "django/core/handlers/exception.py", + "omission_line": 46 + } + ] +} \ No newline at end of file diff --git a/benchmarks/results/verify-gh-cli/verify-environment.json b/benchmarks/results/verify-gh-cli/verify-environment.json new file mode 100644 index 0000000..71e5b3c --- /dev/null +++ b/benchmarks/results/verify-gh-cli/verify-environment.json @@ -0,0 +1,25 @@ +{ + "after": null, + "candidates_rejected": 23, + "checker": "symbols changed by this diff, minus symbols present in the diff, where an inbound CALLS edge exists at distance 1, via reify::query::impact", + "count": 20, + "head": "5d3c4817f1619213951dbf15031bad04acb88392", + "languages": [ + [ + "go", + 891 + ], + [ + "javascript", + 3 + ] + ], + "origin": "git@github.com:cli/cli", + "reify_version": "0.2.2", + "repository": "/private/tmp/claude-501/-Users-lambiengcode--treehouse-reify-2e416f-2-reify/a9d50d0b-cce2-4027-a129-e64262e800a7/scratchpad/gh-cli", + "scan": 400, + "token_counts": "estimated by reify heuristic-v1", + "trials": 20, + "until": null, + "wall_clock_ms": 17946 +} \ No newline at end of file diff --git a/benchmarks/results/verify-gh-cli/verify-outcomes.json b/benchmarks/results/verify-gh-cli/verify-outcomes.json new file mode 100644 index 0000000..ae0f482 --- /dev/null +++ b/benchmarks/results/verify-gh-cli/verify-outcomes.json @@ -0,0 +1,560 @@ +[ + { + "task": "v-e4efbc42", + "commit": "e4efbc42ccfb1f50c2b97e7b864eb5fc5bcc97f0", + "omission_file": "pkg/cmd/copilot/copilot_test.go", + "omission_symbol": "pkg/cmd/copilot/copilot_test.go:597", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": true, + "symbol_hit": true, + "findings": 2, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 47, + "verify_latency_ms": 0, + "index_ms": 778, + "cited": [ + "pkg/cmd/copilot/copilot.go:42", + "pkg/cmd/copilot/copilot_test.go:597" + ], + "cited_on_complete": [ + "pkg/cmd/copilot/copilot.go:42" + ] + }, + { + "task": "v-1e04dab8", + "commit": "1e04dab89cf10a2eab5b238d27fb1d7cb94b8af4", + "omission_file": "pkg/cmd/copilot/copilot.go", + "omission_symbol": "pkg/cmd/copilot/copilot.go:134", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 840, + "cited": [], + "cited_on_complete": [ + "pkg/cmd/copilot/copilot.go:42" + ] + }, + { + "task": "v-2e9fedd3", + "commit": "2e9fedd3aaeb9bc5f044f2d825f565d98083fc56", + "omission_file": "git/client_test.go", + "omission_symbol": "git/client_test.go:1429", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 7, + "false_alarms": 7, + "changed_symbols": 5, + "verify_tokens": 141, + "verify_latency_ms": 1, + "index_ms": 886, + "cited": [ + "git/client_test.go:1316", + "pkg/cmd/issue/develop/develop.go:178", + "pkg/cmd/issue/issue.go:23", + "pkg/cmd/pr/close/close.go:65", + "pkg/cmd/pr/merge/merge.go:380", + "pkg/cmd/pr/merge/merge.go:505", + "pkg/cmd/repo/sync/sync.go:221" + ], + "cited_on_complete": [ + "git/client_test.go:1316", + "pkg/cmd/issue/develop/develop.go:178", + "pkg/cmd/issue/issue.go:23", + "pkg/cmd/pr/close/close.go:65", + "pkg/cmd/pr/merge/merge.go:380", + "pkg/cmd/pr/merge/merge.go:505", + "pkg/cmd/repo/sync/sync.go:221" + ] + }, + { + "task": "v-a6bcd08d", + "commit": "a6bcd08d07d1cbcb17cfce497c0cf261a966f703", + "omission_file": "pkg/cmd/project/item-add/item_add.go", + "omission_symbol": "pkg/cmd/project/item-add/item_add.go:126", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 1, + "changed_symbols": 0, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 832, + "cited": [], + "cited_on_complete": [ + "pkg/cmd/project/item-add/item_add.go:84" + ] + }, + { + "task": "v-efe3f165", + "commit": "efe3f165dd297c85fff11473dbf586f2d39fbf86", + "omission_file": "pkg/cmd/project/shared/queries/resolve_fields_test.go", + "omission_symbol": "pkg/cmd/project/shared/queries/resolve_fields_test.go:31", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": true, + "symbol_hit": true, + "findings": 29, + "false_alarms": 28, + "changed_symbols": 7, + "verify_tokens": 618, + "verify_latency_ms": 2, + "index_ms": 809, + "cited": [ + "pkg/cmd/project/field-list/field_list_test.go:186", + "pkg/cmd/project/field-list/field_list_test.go:278", + "pkg/cmd/project/field-list/field_list_test.go:370", + "pkg/cmd/project/field-list/field_list_test.go:452", + "pkg/cmd/project/field-list/field_list_test.go:517", + "pkg/cmd/project/field-list/field_list_test.go:91", + "pkg/cmd/project/item-edit/item_edit.go:263", + "pkg/cmd/project/item-list/item_list_test.go:102", + "pkg/cmd/project/item-list/item_list_test.go:212", + "pkg/cmd/project/item-list/item_list_test.go:319", + "pkg/cmd/project/item-list/item_list_test.go:426", + "pkg/cmd/project/item-list/item_list_test.go:523", + "pkg/cmd/project/item-list/item_list_test.go:632", + "pkg/cmd/project/item-list/item_list_test.go:722", + "pkg/cmd/project/list/list_test.go:174", + "pkg/cmd/project/list/list_test.go:252", + "pkg/cmd/project/list/list_test.go:332", + "pkg/cmd/project/list/list_test.go:402", + "pkg/cmd/project/list/list_test.go:470", + "pkg/cmd/project/list/list_test.go:549", + "pkg/cmd/project/list/list_test.go:599", + "pkg/cmd/project/list/list_test.go:678", + "pkg/cmd/project/list/list_test.go:725", + "pkg/cmd/project/list/list_test.go:772", + "pkg/cmd/project/list/list_test.go:812", + "pkg/cmd/project/list/list_test.go:851", + "pkg/cmd/project/list/list_test.go:891", + "pkg/cmd/project/list/list_test.go:95", + "pkg/cmd/project/shared/queries/resolve_fields_test.go:31" + ], + "cited_on_complete": [ + "pkg/cmd/project/field-list/field_list_test.go:186", + "pkg/cmd/project/field-list/field_list_test.go:278", + "pkg/cmd/project/field-list/field_list_test.go:370", + "pkg/cmd/project/field-list/field_list_test.go:452", + "pkg/cmd/project/field-list/field_list_test.go:517", + "pkg/cmd/project/field-list/field_list_test.go:91", + "pkg/cmd/project/item-edit/item_edit.go:263", + "pkg/cmd/project/item-list/item_list_test.go:102", + "pkg/cmd/project/item-list/item_list_test.go:212", + "pkg/cmd/project/item-list/item_list_test.go:319", + "pkg/cmd/project/item-list/item_list_test.go:426", + "pkg/cmd/project/item-list/item_list_test.go:523", + "pkg/cmd/project/item-list/item_list_test.go:632", + "pkg/cmd/project/item-list/item_list_test.go:722", + "pkg/cmd/project/list/list_test.go:174", + "pkg/cmd/project/list/list_test.go:252", + "pkg/cmd/project/list/list_test.go:332", + "pkg/cmd/project/list/list_test.go:402", + "pkg/cmd/project/list/list_test.go:470", + "pkg/cmd/project/list/list_test.go:549", + "pkg/cmd/project/list/list_test.go:599", + "pkg/cmd/project/list/list_test.go:678", + "pkg/cmd/project/list/list_test.go:725", + "pkg/cmd/project/list/list_test.go:772", + "pkg/cmd/project/list/list_test.go:812", + "pkg/cmd/project/list/list_test.go:851", + "pkg/cmd/project/list/list_test.go:891", + "pkg/cmd/project/list/list_test.go:95" + ] + }, + { + "task": "v-688751de", + "commit": "688751de2ce8d610ce76cd0608930e7912509ed3", + "omission_file": "pkg/cmd/pr/checkout/checkout_test.go", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 2, + "false_alarms": 2, + "changed_symbols": 1, + "verify_tokens": 50, + "verify_latency_ms": 0, + "index_ms": 812, + "cited": [ + "pkg/cmd/pr/checkout/checkout.go:40", + "pkg/cmd/pr/checkout/checkout_test.go:179" + ], + "cited_on_complete": [ + "pkg/cmd/pr/checkout/checkout.go:40", + "pkg/cmd/pr/checkout/checkout_test.go:179" + ] + }, + { + "task": "v-9f14d1ac", + "commit": "9f14d1ac675f25a75d4b940dc88e733f06398e76", + "omission_file": "pkg/cmd/pr/checkout/checkout_test.go", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 3, + "false_alarms": 2, + "changed_symbols": 2, + "verify_tokens": 72, + "verify_latency_ms": 0, + "index_ms": 811, + "cited": [ + "pkg/cmd/pr/checkout/checkout.go:40", + "pkg/cmd/pr/checkout/checkout_test.go:1075", + "pkg/cmd/pr/checkout/checkout_test.go:179" + ], + "cited_on_complete": [ + "pkg/cmd/pr/checkout/checkout.go:40", + "pkg/cmd/pr/checkout/checkout_test.go:179" + ] + }, + { + "task": "v-d5f4bed3", + "commit": "d5f4bed3f49dbbc5d5a2e7bb76ba9f25ba7ba574", + "omission_file": "pkg/cmd/skills/update/update.go", + "omission_symbol": "pkg/cmd/skills/update/update.go:66", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 2, + "false_alarms": 5, + "changed_symbols": 3, + "verify_tokens": 50, + "verify_latency_ms": 0, + "index_ms": 804, + "cited": [ + "pkg/cmd/skills/install/install_test.go:30", + "pkg/cmd/skills/skills.go:17" + ], + "cited_on_complete": [ + "pkg/cmd/skills/install/install_test.go:30", + "pkg/cmd/skills/skills.go:17", + "pkg/cmd/skills/update/update_test.go:25", + "pkg/cmd/skills/update/update_test.go:43", + "pkg/cmd/skills/update/update_test.go:54" + ] + }, + { + "task": "v-74e77914", + "commit": "74e779140c472bc380ce57978853480805dc16b7", + "omission_file": "internal/codespaces/connection/connection.go", + "omission_symbol": "internal/codespaces/connection/connection.go:26", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 14, + "false_alarms": 14, + "changed_symbols": 10, + "verify_tokens": 296, + "verify_latency_ms": 2, + "index_ms": 803, + "cited": [ + "internal/codespaces/portforwarder/port_forwarder.go:63", + "internal/codespaces/states.go:40", + "pkg/cmd/codespace/jupyter.go:32", + "pkg/cmd/codespace/logs.go:35", + "pkg/cmd/codespace/ports.go:233", + "pkg/cmd/codespace/ports.go:312", + "pkg/cmd/codespace/ports.go:53", + "pkg/cmd/codespace/ports_test.go:104", + "pkg/cmd/codespace/ports_test.go:14", + "pkg/cmd/codespace/ports_test.go:69", + "pkg/cmd/codespace/ports_test.go:91", + "pkg/cmd/codespace/rebuild.go:43", + "pkg/cmd/codespace/ssh.go:165", + "pkg/cmd/codespace/ssh.go:552" + ], + "cited_on_complete": [ + "internal/codespaces/portforwarder/port_forwarder.go:63", + "internal/codespaces/states.go:40", + "pkg/cmd/codespace/jupyter.go:32", + "pkg/cmd/codespace/logs.go:35", + "pkg/cmd/codespace/ports.go:233", + "pkg/cmd/codespace/ports.go:312", + "pkg/cmd/codespace/ports.go:53", + "pkg/cmd/codespace/ports_test.go:104", + "pkg/cmd/codespace/ports_test.go:14", + "pkg/cmd/codespace/ports_test.go:69", + "pkg/cmd/codespace/ports_test.go:91", + "pkg/cmd/codespace/rebuild.go:43", + "pkg/cmd/codespace/ssh.go:165", + "pkg/cmd/codespace/ssh.go:552" + ] + }, + { + "task": "v-f1d11210", + "commit": "f1d112104821b055bb0c5656f2989f9213db71f6", + "omission_file": "pkg/cmd/skills/install/install_test.go", + "omission_symbol": "pkg/cmd/skills/install/install_test.go:303", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 2, + "false_alarms": 2, + "changed_symbols": 1, + "verify_tokens": 50, + "verify_latency_ms": 0, + "index_ms": 820, + "cited": [ + "pkg/cmd/skills/install/install.go:250", + "pkg/cmd/skills/install/install.go:482" + ], + "cited_on_complete": [ + "pkg/cmd/skills/install/install.go:250", + "pkg/cmd/skills/install/install.go:482" + ] + }, + { + "task": "v-751dc5e0", + "commit": "751dc5e0383f08d1d6a211c97d0479aedb39726b", + "omission_file": "pkg/cmd/release/shared/fetch.go", + "omission_symbol": "pkg/cmd/release/shared/fetch.go:187", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 6, + "changed_symbols": 1, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 821, + "cited": [], + "cited_on_complete": [ + "pkg/cmd/release/delete-asset/delete_asset.go:59", + "pkg/cmd/release/delete/delete.go:67", + "pkg/cmd/release/download/download.go:136", + "pkg/cmd/release/edit/edit.go:92", + "pkg/cmd/release/upload/upload.go:77", + "pkg/cmd/release/view/view.go:75" + ] + }, + { + "task": "v-517dae6a", + "commit": "517dae6a938d4efe73f1219873ebcc74cf4febe1", + "omission_file": "internal/skills/registry/registry_test.go", + "omission_symbol": "internal/skills/registry/registry_test.go:40", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 0, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 816, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-8d2b059e", + "commit": "8d2b059e07f71c17068f2286617f23286e05e0c0", + "omission_file": "pkg/cmd/discussion/view/view.go", + "omission_symbol": "pkg/cmd/discussion/view/view.go:96", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 2, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 810, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-2618999b", + "commit": "2618999bcb6c85d4554638937dc80c322a76d593", + "omission_file": "pkg/cmd/discussion/client/client_impl_test.go", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 2, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 777, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-e2d150da", + "commit": "e2d150da420b8bf84f3097f6fad3bbb715ea1cb4", + "omission_file": "pkg/cmd/discussion/edit/edit.go", + "omission_symbol": "pkg/cmd/discussion/edit/edit.go:125", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 1, + "false_alarms": 2, + "changed_symbols": 2, + "verify_tokens": 30, + "verify_latency_ms": 0, + "index_ms": 777, + "cited": [ + "pkg/cmd/discussion/create/create.go:34" + ], + "cited_on_complete": [ + "pkg/cmd/discussion/create/create.go:34", + "pkg/cmd/discussion/edit/edit.go:42" + ] + }, + { + "task": "v-c1f3c1a1", + "commit": "c1f3c1a164ab67436d525769adbce0fd67dd5e70", + "omission_file": "pkg/cmd/discussion/view/view.go", + "omission_symbol": "pkg/cmd/discussion/view/view.go:95", + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": false, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 1, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 801, + "cited": [], + "cited_on_complete": [] + }, + { + "task": "v-b1029009", + "commit": "b1029009dbfbcf4240472097e12614dbc3cdcd19", + "omission_file": "pkg/cmd/discussion/edit/edit.go", + "omission_symbol": "pkg/cmd/discussion/edit/edit.go:125", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": true, + "findings": 5, + "false_alarms": 5, + "changed_symbols": 6, + "verify_tokens": 115, + "verify_latency_ms": 1, + "index_ms": 776, + "cited": [ + "pkg/cmd/auth/shared/git_credential.go:80", + "pkg/cmd/auth/shared/gitcredentials/updater_test.go:43", + "pkg/cmd/auth/shared/gitcredentials/updater_test.go:64", + "pkg/cmd/discussion/create/create.go:34", + "pkg/cmd/discussion/edit/edit.go:125" + ], + "cited_on_complete": [ + "pkg/cmd/auth/shared/git_credential.go:80", + "pkg/cmd/auth/shared/gitcredentials/updater_test.go:43", + "pkg/cmd/auth/shared/gitcredentials/updater_test.go:64", + "pkg/cmd/discussion/create/create.go:34", + "pkg/cmd/discussion/edit/edit.go:42" + ] + }, + { + "task": "v-16a20347", + "commit": "16a20347dd33b8f67abd8990ab0940d35f522233", + "omission_file": "pkg/cmd/skills/update/update_test.go", + "omission_symbol": "pkg/cmd/skills/update/update_test.go:313", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": true, + "symbol_hit": true, + "findings": 2, + "false_alarms": 1, + "changed_symbols": 1, + "verify_tokens": 48, + "verify_latency_ms": 0, + "index_ms": 753, + "cited": [ + "pkg/cmd/skills/update/update.go:66", + "pkg/cmd/skills/update/update_test.go:313" + ], + "cited_on_complete": [ + "pkg/cmd/skills/update/update.go:66" + ] + }, + { + "task": "v-fb748cb2", + "commit": "fb748cb2bf3a434ff12f5268c9983a65310c6520", + "omission_file": "pkg/cmd/skills/preview/preview_test.go", + "omission_symbol": "pkg/cmd/skills/preview/preview_test.go:112", + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": true, + "findings": 13, + "false_alarms": 12, + "changed_symbols": 4, + "verify_tokens": 306, + "verify_latency_ms": 1, + "index_ms": 776, + "cited": [ + "pkg/cmd/root/root.go:63", + "pkg/cmd/skills/install/install.go:232", + "pkg/cmd/skills/install/install_test.go:2134", + "pkg/cmd/skills/preview/preview_test.go:112", + "pkg/cmd/skills/preview/preview_test.go:1157", + "pkg/cmd/skills/preview/preview_test.go:24", + "pkg/cmd/skills/preview/preview_test.go:408", + "pkg/cmd/skills/preview/preview_test.go:419", + "pkg/cmd/skills/preview/preview_test.go:484", + "pkg/cmd/skills/preview/preview_test.go:696", + "pkg/cmd/skills/preview/preview_test.go:871", + "pkg/cmd/skills/preview/preview_test.go:948", + "pkg/cmd/skills/skills.go:16" + ], + "cited_on_complete": [ + "pkg/cmd/root/root.go:63", + "pkg/cmd/skills/install/install.go:232", + "pkg/cmd/skills/install/install_test.go:2134", + "pkg/cmd/skills/preview/preview_test.go:1157", + "pkg/cmd/skills/preview/preview_test.go:24", + "pkg/cmd/skills/preview/preview_test.go:408", + "pkg/cmd/skills/preview/preview_test.go:419", + "pkg/cmd/skills/preview/preview_test.go:484", + "pkg/cmd/skills/preview/preview_test.go:696", + "pkg/cmd/skills/preview/preview_test.go:871", + "pkg/cmd/skills/preview/preview_test.go:948", + "pkg/cmd/skills/skills.go:16" + ] + }, + { + "task": "v-a44721d2", + "commit": "a44721d233be9a2f6f0b5ee5c4f71274acb8d296", + "omission_file": "internal/prompter/echo_linux_test.go", + "omission_symbol": null, + "omission_file_reachable": false, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 0, + "false_alarms": 0, + "changed_symbols": 0, + "verify_tokens": 17, + "verify_latency_ms": 0, + "index_ms": 768, + "cited": [], + "cited_on_complete": [] + } +] \ No newline at end of file diff --git a/benchmarks/results/verify-gh-cli/verify-summary.json b/benchmarks/results/verify-gh-cli/verify-summary.json new file mode 100644 index 0000000..5332877 --- /dev/null +++ b/benchmarks/results/verify-gh-cli/verify-summary.json @@ -0,0 +1,36 @@ +{ + "tasks": 20, + "omission_recall": 0.4, + "omission_recall_ci": [ + 0.21880396, + 0.6134221 + ], + "omission_recall_attributable": 0.15, + "omission_recall_attributable_ci": [ + 0.05236779, + 0.36042333 + ], + "reachable_omissions": 19, + "omission_recall_reachable": 0.42105263, + "omission_recall_reachable_ci": [ + 0.2314162, + 0.63724446 + ], + "symbol_scorable": 16, + "omission_recall_symbol": 0.3125, + "omission_recall_symbol_ci": [ + 0.14164433, + 0.55596066 + ], + "false_alarm_rate": 4.45, + "commits_with_a_false_alarm": 15, + "false_alarm_share_ci": [ + 0.53129494, + 0.88813996 + ], + "median_findings_per_diff": 2, + "median_verify_tokens": 48, + "median_verify_latency_ms": 0, + "median_index_ms": 809, + "diffs_resolving_to_nothing": 3 +} \ No newline at end of file diff --git a/benchmarks/results/verify-gh-cli/verify-tasks.json b/benchmarks/results/verify-gh-cli/verify-tasks.json new file mode 100644 index 0000000..c36e535 --- /dev/null +++ b/benchmarks/results/verify-gh-cli/verify-tasks.json @@ -0,0 +1,2881 @@ +{ + "repository": "/private/tmp/claude-501/-Users-lambiengcode--treehouse-reify-2e416f-2-reify/a9d50d0b-cce2-4027-a129-e64262e800a7/scratchpad/gh-cli", + "head": "5d3c4817f1619213951dbf15031bad04acb88392", + "generated_from_commits": 399, + "rejected": [ + [ + "92ae1de0355368b3d6d1c361362dc524fe98498b", + "no file changed by exactly one hunk" + ], + [ + "5e6aa5ab275ad25a8999871f7f86aeda6349099b", + "no file changed by exactly one hunk" + ], + [ + "c2ad3b0eb7ead66eec3a8a61239e10a765332ec7", + "no file changed by exactly one hunk" + ], + [ + "b130a9be5b0f0db2e41ddd82044ef6c256c93625", + "no file changed by exactly one hunk" + ], + [ + "7b681a4e8b67d203ccdaab5ff54570bdf6ca3669", + "fewer than two indexable files" + ], + [ + "954ffc37e5931a9621868a50f05c8fb17e288f19", + "no file changed by exactly one hunk" + ], + [ + "cce391b663aa488b8b4f681616b52dc8f2e3d531", + "fewer than two indexable files" + ], + [ + "70bb306bd25eb407f90eabefd98824aed62cf519", + "no file changed by exactly one hunk" + ], + [ + "01bcd474447b5034da20686c838ae4b0cf2b23f5", + "no file changed by exactly one hunk" + ], + [ + "d63ab8ddd8696194f3fd49a726eb6d60dd3cdfa5", + "no file changed by exactly one hunk" + ], + [ + "5d77247a5a77b4ee1e9516f83da010dd9a0d08c3", + "no file changed by exactly one hunk" + ], + [ + "9f2da1186132c0ba891767939c81fb6e785b67dc", + "no file changed by exactly one hunk" + ], + [ + "da68cb8f6f597cfc3838cf40f89ecc01f4e53233", + "fewer than two indexable files" + ], + [ + "797effe0295914e03f4b40cb873276da1d3b7e93", + "fewer than two indexable files" + ], + [ + "7bd67a840456436a1edc0f80dc85687f29b277af", + "fewer than two indexable files" + ], + [ + "57008e797097e9b275ba7d1849ac750e762743e1", + "fewer than two indexable files" + ], + [ + "8b73951ca75c6495a776e36da9fb48cc93000866", + "fewer than two indexable files" + ], + [ + "d3a153872b9774009d620d0f3e12f993a21765eb", + "fewer than two indexable files" + ], + [ + "51b765381870dbf62390e0f7a297cc57ea771db7", + "fewer than two indexable files" + ], + [ + "97d1cbd9fc5499a2f804d991b353821e17b19eb7", + "no file changed by exactly one hunk" + ], + [ + "00fc8c923ab3f321a412e1c89e425c107122f0bb", + "no file changed by exactly one hunk" + ], + [ + "d9eb0627dceeb49b2943fa992414eb185787d02e", + "no file changed by exactly one hunk" + ], + [ + "601dd346b00b357a0541239fb80b34c3795e7c33", + "no file changed by exactly one hunk" + ] + ], + "tasks": [ + { + "id": "v-e4efbc42", + "commit": "e4efbc42ccfb1f50c2b97e7b864eb5fc5bcc97f0", + "parent": "046c222048e780a707bacea18fc6dc0d0865c7a5", + "date": "2026-08-21", + "prompt": "Narrow Copilot newline fix scope", + "complete": { + "files": [ + { + "path": "pkg/cmd/copilot/copilot.go", + "created": false, + "hunks": [ + { + "old_start": 156, + "old_len": 7, + "changed_lines": [ + 159 + ] + } + ] + }, + { + "path": "pkg/cmd/copilot/copilot_test.go", + "created": false, + "hunks": [ + { + "old_start": 620, + "old_len": 14, + "changed_lines": [ + 623, + 624, + 625, + 626, + 627, + 628, + 629, + 630 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/copilot/copilot.go", + "created": false, + "hunks": [ + { + "old_start": 156, + "old_len": 7, + "changed_lines": [ + 159 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/copilot/copilot_test.go", + "omission_line": 623 + }, + { + "id": "v-1e04dab8", + "commit": "1e04dab89cf10a2eab5b238d27fb1d7cb94b8af4", + "parent": "a255baf71d13fe5947a4eb7ad521ffd412d64cee", + "date": "2026-08-20", + "prompt": "Fix Copilot install warning newlines", + "complete": { + "files": [ + { + "path": "pkg/cmd/copilot/copilot.go", + "created": false, + "hunks": [ + { + "old_start": 152, + "old_len": 11, + "changed_lines": [ + 155, + 159 + ] + } + ] + }, + { + "path": "pkg/cmd/copilot/copilot_test.go", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 591, + "old_len": 30, + "changed_lines": [ + 594, + 595, + 596, + 597, + 598, + 601, + 602, + 603, + 604, + 605, + 607, + 608, + 609, + 610, + 611, + 612, + 614, + 615, + 616, + 617 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/copilot/copilot_test.go", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + }, + { + "old_start": 591, + "old_len": 30, + "changed_lines": [ + 594, + 595, + 596, + 597, + 598, + 601, + 602, + 603, + 604, + 605, + 607, + 608, + 609, + 610, + 611, + 612, + 614, + 615, + 616, + 617 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/copilot/copilot.go", + "omission_line": 155 + }, + { + "id": "v-2e9fedd3", + "commit": "2e9fedd3aaeb9bc5f044f2d825f565d98083fc56", + "parent": "a526307b621c90dca18734bfedaa5533318edd1a", + "date": "2026-08-12", + "prompt": "Add worktree checkout to issue develop", + "complete": { + "files": [ + { + "path": "git/client.go", + "created": false, + "hunks": [ + { + "old_start": 642, + "old_len": 6, + "changed_lines": [ + 645 + ] + } + ] + }, + { + "path": "git/client_test.go", + "created": false, + "hunks": [ + { + "old_start": 1426, + "old_len": 6, + "changed_lines": [ + 1429 + ] + } + ] + }, + { + "path": "pkg/cmd/issue/develop/develop.go", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 32, + "old_len": 6, + "changed_lines": [ + 35 + ] + }, + { + "old_start": 66, + "old_len": 6, + "changed_lines": [ + 69 + ] + }, + { + "old_start": 91, + "old_len": 6, + "changed_lines": [ + 94 + ] + }, + { + "old_start": 120, + "old_len": 6, + "changed_lines": [ + 123 + ] + }, + { + "old_start": 133, + "old_len": 6, + "changed_lines": [ + 136 + ] + }, + { + "old_start": 353, + "old_len": 16, + "changed_lines": [ + 356, + 357, + 360, + 364, + 365 + ] + } + ] + }, + { + "path": "pkg/cmd/issue/develop/develop_test.go", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 6, + "changed_lines": [ + 7 + ] + }, + { + "old_start": 24, + "old_len": 6, + "changed_lines": [ + 27 + ] + }, + { + "old_start": 58, + "old_len": 6, + "changed_lines": [ + 61 + ] + }, + { + "old_start": 106, + "old_len": 6, + "changed_lines": [ + 109 + ] + }, + { + "old_start": 138, + "old_len": 6, + "changed_lines": [ + 141 + ] + }, + { + "old_start": 767, + "old_len": 3, + "changed_lines": [ + 770 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "git/client.go", + "created": false, + "hunks": [ + { + "old_start": 642, + "old_len": 6, + "changed_lines": [ + 645 + ] + } + ] + }, + { + "path": "pkg/cmd/issue/develop/develop.go", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 32, + "old_len": 6, + "changed_lines": [ + 35 + ] + }, + { + "old_start": 66, + "old_len": 6, + "changed_lines": [ + 69 + ] + }, + { + "old_start": 91, + "old_len": 6, + "changed_lines": [ + 94 + ] + }, + { + "old_start": 120, + "old_len": 6, + "changed_lines": [ + 123 + ] + }, + { + "old_start": 133, + "old_len": 6, + "changed_lines": [ + 136 + ] + }, + { + "old_start": 353, + "old_len": 16, + "changed_lines": [ + 356, + 357, + 360, + 364, + 365 + ] + } + ] + }, + { + "path": "pkg/cmd/issue/develop/develop_test.go", + "created": false, + "hunks": [ + { + "old_start": 4, + "old_len": 6, + "changed_lines": [ + 7 + ] + }, + { + "old_start": 24, + "old_len": 6, + "changed_lines": [ + 27 + ] + }, + { + "old_start": 58, + "old_len": 6, + "changed_lines": [ + 61 + ] + }, + { + "old_start": 106, + "old_len": 6, + "changed_lines": [ + 109 + ] + }, + { + "old_start": 138, + "old_len": 6, + "changed_lines": [ + 141 + ] + }, + { + "old_start": 767, + "old_len": 3, + "changed_lines": [ + 770 + ] + } + ] + } + ] + }, + "omission_file": "git/client_test.go", + "omission_line": 1429 + }, + { + "id": "v-a6bcd08d", + "commit": "a6bcd08d07d1cbcb17cfce497c0cf261a966f703", + "parent": "e83adbc0642994fae7c39a9a012eb34b8c81f4f1", + "date": "2026-08-03", + "prompt": "Fix item-add output for non-TTY", + "complete": { + "files": [ + { + "path": "pkg/cmd/project/item-add/item_add.go", + "created": false, + "hunks": [ + { + "old_start": 124, + "old_len": 10, + "changed_lines": [ + 127, + 128, + 131 + ] + } + ] + }, + { + "path": "pkg/cmd/project/item-add/item_add_test.go", + "created": false, + "hunks": [ + { + "old_start": 8, + "old_len": 6, + "changed_lines": [ + 11 + ] + }, + { + "old_start": 539, + "old_len": 3, + "changed_lines": [ + 542 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/project/item-add/item_add_test.go", + "created": false, + "hunks": [ + { + "old_start": 8, + "old_len": 6, + "changed_lines": [ + 11 + ] + }, + { + "old_start": 539, + "old_len": 3, + "changed_lines": [ + 542 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/project/item-add/item_add.go", + "omission_line": 127 + }, + { + "id": "v-efe3f165", + "commit": "efe3f165dd297c85fff11473dbf586f2d39fbf86", + "parent": "ae66a1c02e08366858f3070664f493afbe0cdf18", + "date": "2026-07-22", + "prompt": "Add named field columns to gh project item-list", + "complete": { + "files": [ + { + "path": "pkg/cmd/project/item-list/item_list.go", + "created": false, + "hunks": [ + { + "old_start": 21, + "old_len": 6, + "changed_lines": [ + 24 + ] + }, + { + "old_start": 55, + "old_len": 6, + "changed_lines": [ + 58 + ] + }, + { + "old_start": 71, + "old_len": 6, + "changed_lines": [ + 74 + ] + }, + { + "old_start": 101, + "old_len": 6, + "changed_lines": [ + 104 + ] + }, + { + "old_start": 142, + "old_len": 15, + "changed_lines": [ + 145, + 148, + 153 + ] + }, + { + "old_start": 162, + "old_len": 8, + "changed_lines": [ + 165, + 170 + ] + } + ] + }, + { + "path": "pkg/cmd/project/item-list/item_list_test.go", + "created": false, + "hunks": [ + { + "old_start": 61, + "old_len": 6, + "changed_lines": [ + 64 + ] + }, + { + "old_start": 95, + "old_len": 6, + "changed_lines": [ + 98 + ] + }, + { + "old_start": 734, + "old_len": 3, + "changed_lines": [ + 737 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/queries.go", + "created": false, + "hunks": [ + { + "old_start": 6, + "old_len": 6, + "changed_lines": [ + 9 + ] + }, + { + "old_start": 472, + "old_len": 6, + "changed_lines": [ + 475 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/queries_test.go", + "created": false, + "hunks": [ + { + "old_start": 680, + "old_len": 3, + "changed_lines": [ + 683 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/resolve_fields.go", + "created": false, + "hunks": [ + { + "old_start": 85, + "old_len": 6, + "changed_lines": [ + 88 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/resolve_fields_test.go", + "created": false, + "hunks": [ + { + "old_start": 60, + "old_len": 14, + "changed_lines": [ + 63, + 70 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/project/item-list/item_list.go", + "created": false, + "hunks": [ + { + "old_start": 21, + "old_len": 6, + "changed_lines": [ + 24 + ] + }, + { + "old_start": 55, + "old_len": 6, + "changed_lines": [ + 58 + ] + }, + { + "old_start": 71, + "old_len": 6, + "changed_lines": [ + 74 + ] + }, + { + "old_start": 101, + "old_len": 6, + "changed_lines": [ + 104 + ] + }, + { + "old_start": 142, + "old_len": 15, + "changed_lines": [ + 145, + 148, + 153 + ] + }, + { + "old_start": 162, + "old_len": 8, + "changed_lines": [ + 165, + 170 + ] + } + ] + }, + { + "path": "pkg/cmd/project/item-list/item_list_test.go", + "created": false, + "hunks": [ + { + "old_start": 61, + "old_len": 6, + "changed_lines": [ + 64 + ] + }, + { + "old_start": 95, + "old_len": 6, + "changed_lines": [ + 98 + ] + }, + { + "old_start": 734, + "old_len": 3, + "changed_lines": [ + 737 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/queries.go", + "created": false, + "hunks": [ + { + "old_start": 6, + "old_len": 6, + "changed_lines": [ + 9 + ] + }, + { + "old_start": 472, + "old_len": 6, + "changed_lines": [ + 475 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/queries_test.go", + "created": false, + "hunks": [ + { + "old_start": 680, + "old_len": 3, + "changed_lines": [ + 683 + ] + } + ] + }, + { + "path": "pkg/cmd/project/shared/queries/resolve_fields.go", + "created": false, + "hunks": [ + { + "old_start": 85, + "old_len": 6, + "changed_lines": [ + 88 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/project/shared/queries/resolve_fields_test.go", + "omission_line": 63 + }, + { + "id": "v-688751de", + "commit": "688751de2ce8d610ce76cd0608930e7912509ed3", + "parent": "a5eea131501c535d0527eb61ade5969bd85d0ff3", + "date": "2026-07-22", + "prompt": "Harden worktree submodule prefixing and cover cmd.Dir stripping", + "complete": { + "files": [ + { + "path": "pkg/cmd/pr/checkout/checkout.go", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 166, + "old_len": 8, + "changed_lines": [ + 169, + 170 + ] + } + ] + }, + { + "path": "pkg/cmd/pr/checkout/checkout_test.go", + "created": false, + "hunks": [ + { + "old_start": 1170, + "old_len": 3, + "changed_lines": [ + 1173 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/pr/checkout/checkout.go", + "created": false, + "hunks": [ + { + "old_start": 5, + "old_len": 6, + "changed_lines": [ + 8 + ] + }, + { + "old_start": 166, + "old_len": 8, + "changed_lines": [ + 169, + 170 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/pr/checkout/checkout_test.go", + "omission_line": 1173 + }, + { + "id": "v-9f14d1ac", + "commit": "9f14d1ac675f25a75d4b940dc88e733f06398e76", + "parent": "9fc654ee0985d08c2d9076785c6993da885435a4", + "date": "2026-07-22", + "prompt": "Simplify submodule worktree prefix to inline conditional", + "complete": { + "files": [ + { + "path": "pkg/cmd/pr/checkout/checkout.go", + "created": false, + "hunks": [ + { + "old_start": 160, + "old_len": 7, + "changed_lines": [ + 163 + ] + }, + { + "old_start": 359, + "old_len": 23, + "changed_lines": [ + 362, + 363, + 364, + 365, + 366, + 367, + 368, + 369, + 370, + 371, + 372, + 373, + 374, + 375, + 376, + 377, + 378 + ] + } + ] + }, + { + "path": "pkg/cmd/pr/checkout/checkout_test.go", + "created": false, + "hunks": [ + { + "old_start": 1071, + "old_len": 19, + "changed_lines": [ + 1074, + 1075, + 1076, + 1077, + 1078, + 1079, + 1080, + 1081, + 1082, + 1083, + 1084, + 1085, + 1086, + 1087, + 1088, + 1089 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/pr/checkout/checkout.go", + "created": false, + "hunks": [ + { + "old_start": 160, + "old_len": 7, + "changed_lines": [ + 163 + ] + }, + { + "old_start": 359, + "old_len": 23, + "changed_lines": [ + 362, + 363, + 364, + 365, + 366, + 367, + 368, + 369, + 370, + 371, + 372, + 373, + 374, + 375, + 376, + 377, + 378 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/pr/checkout/checkout_test.go", + "omission_line": 1074 + }, + { + "id": "v-d5f4bed3", + "commit": "d5f4bed3f49dbbc5d5a2e7bb76ba9f25ba7ba574", + "parent": "2b970995a3c63b7a7600f242d55d15185a82ed8b", + "date": "2026-07-13", + "prompt": "Add Grok skill host support", + "complete": { + "files": [ + { + "path": "internal/skills/registry/registry.go", + "created": false, + "hunks": [ + { + "old_start": 188, + "old_len": 6, + "changed_lines": [ + 191 + ] + } + ] + }, + { + "path": "internal/skills/registry/registry_test.go", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + }, + { + "old_start": 159, + "old_len": 6, + "changed_lines": [ + 162 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 89, + "old_len": 7, + "changed_lines": [ + 92 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/update/update.go", + "created": false, + "hunks": [ + { + "old_start": 80, + "old_len": 7, + "changed_lines": [ + 83 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "internal/skills/registry/registry.go", + "created": false, + "hunks": [ + { + "old_start": 188, + "old_len": 6, + "changed_lines": [ + 191 + ] + } + ] + }, + { + "path": "internal/skills/registry/registry_test.go", + "created": false, + "hunks": [ + { + "old_start": 23, + "old_len": 6, + "changed_lines": [ + 26 + ] + }, + { + "old_start": 159, + "old_len": 6, + "changed_lines": [ + 162 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 89, + "old_len": 7, + "changed_lines": [ + 92 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/skills/update/update.go", + "omission_line": 83 + }, + { + "id": "v-74e77914", + "commit": "74e779140c472bc380ce57978853480805dc16b7", + "parent": "6dae3077b89c9858c5778c0b37a116b1091f8782", + "date": "2026-07-02", + "prompt": "Fix concurrent map writes in codespace port forwarding", + "complete": { + "files": [ + { + "path": "internal/codespaces/connection/connection.go", + "created": false, + "hunks": [ + { + "old_start": 30, + "old_len": 6, + "changed_lines": [ + 33 + ] + } + ] + }, + { + "path": "internal/codespaces/portforwarder/port_forwarder.go", + "created": false, + "hunks": [ + { + "old_start": 36, + "old_len": 7, + "changed_lines": [ + 39 + ] + }, + { + "old_start": 54, + "old_len": 7, + "changed_lines": [ + 57 + ] + }, + { + "old_start": 108, + "old_len": 6, + "changed_lines": [ + 111 + ] + }, + { + "old_start": 166, + "old_len": 18, + "changed_lines": [ + 169, + 170, + 171, + 172, + 173, + 174, + 175, + 176, + 177, + 178, + 179, + 180 + ] + }, + { + "old_start": 243, + "old_len": 6, + "changed_lines": [ + 246 + ] + }, + { + "old_start": 253, + "old_len": 22, + "changed_lines": [ + 256, + 258, + 263, + 269, + 272 + ] + } + ] + }, + { + "path": "internal/codespaces/portforwarder/port_forwarder_test.go", + "created": false, + "hunks": [ + { + "old_start": 7, + "old_len": 6, + "changed_lines": [ + 10 + ] + }, + { + "old_start": 31, + "old_len": 26, + "changed_lines": [ + 34, + 35, + 36, + 40, + 41, + 42, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53 + ] + }, + { + "old_start": 96, + "old_len": 9, + "changed_lines": [ + 99, + 100, + 101 + ] + }, + { + "old_start": 131, + "old_len": 9, + "changed_lines": [ + 134, + 135, + 136 + ] + }, + { + "old_start": 163, + "old_len": 39, + "changed_lines": [ + 166, + 167, + 168, + 171, + 172, + 173, + 176, + 177, + 178, + 181, + 183, + 184, + 185, + 188, + 189, + 190, + 192, + 193, + 196, + 197, + 199 + ] + }, + { + "old_start": 230, + "old_len": 38, + "changed_lines": [ + 233, + 234, + 235, + 238, + 239, + 240, + 243, + 244, + 245, + 249, + 251, + 252, + 253, + 256, + 257, + 258, + 259, + 260, + 261, + 262, + 263, + 264, + 265, + 266 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "internal/codespaces/portforwarder/port_forwarder.go", + "created": false, + "hunks": [ + { + "old_start": 36, + "old_len": 7, + "changed_lines": [ + 39 + ] + }, + { + "old_start": 54, + "old_len": 7, + "changed_lines": [ + 57 + ] + }, + { + "old_start": 108, + "old_len": 6, + "changed_lines": [ + 111 + ] + }, + { + "old_start": 166, + "old_len": 18, + "changed_lines": [ + 169, + 170, + 171, + 172, + 173, + 174, + 175, + 176, + 177, + 178, + 179, + 180 + ] + }, + { + "old_start": 243, + "old_len": 6, + "changed_lines": [ + 246 + ] + }, + { + "old_start": 253, + "old_len": 22, + "changed_lines": [ + 256, + 258, + 263, + 269, + 272 + ] + } + ] + }, + { + "path": "internal/codespaces/portforwarder/port_forwarder_test.go", + "created": false, + "hunks": [ + { + "old_start": 7, + "old_len": 6, + "changed_lines": [ + 10 + ] + }, + { + "old_start": 31, + "old_len": 26, + "changed_lines": [ + 34, + 35, + 36, + 40, + 41, + 42, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53 + ] + }, + { + "old_start": 96, + "old_len": 9, + "changed_lines": [ + 99, + 100, + 101 + ] + }, + { + "old_start": 131, + "old_len": 9, + "changed_lines": [ + 134, + 135, + 136 + ] + }, + { + "old_start": 163, + "old_len": 39, + "changed_lines": [ + 166, + 167, + 168, + 171, + 172, + 173, + 176, + 177, + 178, + 181, + 183, + 184, + 185, + 188, + 189, + 190, + 192, + 193, + 196, + 197, + 199 + ] + }, + { + "old_start": 230, + "old_len": 38, + "changed_lines": [ + 233, + 234, + 235, + 238, + 239, + 240, + 243, + 244, + 245, + 249, + 251, + 252, + 253, + 256, + 257, + 258, + 259, + 260, + 261, + 262, + 263, + 264, + 265, + 266 + ] + } + ] + } + ] + }, + "omission_file": "internal/codespaces/connection/connection.go", + "omission_line": 33 + }, + { + "id": "v-f1d11210", + "commit": "f1d112104821b055bb0c5656f2989f9213db71f6", + "parent": "326faaac8b5c3b736ddc6bfa573cc97dc6452a24", + "date": "2026-07-02", + "prompt": "honor --dir without agent prompt", + "complete": { + "files": [ + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 915, + "old_len": 6, + "changed_lines": [ + 918 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/install/install_test.go", + "created": false, + "hunks": [ + { + "old_start": 473, + "old_len": 6, + "changed_lines": [ + 476 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 915, + "old_len": 6, + "changed_lines": [ + 918 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/skills/install/install_test.go", + "omission_line": 476 + }, + { + "id": "v-751dc5e0", + "commit": "751dc5e0383f08d1d6a211c97d0479aedb39726b", + "parent": "dd26eb39b04db5bd282509787d2b8fbff11ba694", + "date": "2026-06-24", + "prompt": "don't let a failed draft lookup mask a found release", + "complete": { + "files": [ + { + "path": "pkg/cmd/release/download/download_test.go", + "created": false, + "hunks": [ + { + "old_start": 218, + "old_len": 6, + "changed_lines": [ + 221 + ] + } + ] + }, + { + "path": "pkg/cmd/release/shared/fetch.go", + "created": false, + "hunks": [ + { + "old_start": 201, + "old_len": 15, + "changed_lines": [ + 204, + 205, + 206, + 207, + 208, + 211, + 212 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/release/download/download_test.go", + "created": false, + "hunks": [ + { + "old_start": 218, + "old_len": 6, + "changed_lines": [ + 221 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/release/shared/fetch.go", + "omission_line": 204 + }, + { + "id": "v-517dae6a", + "commit": "517dae6a938d4efe73f1219873ebcc74cf4febe1", + "parent": "70bb306bd25eb407f90eabefd98824aed62cf519", + "date": "2026-06-18", + "prompt": "install universal agent to ~/.agents/skills", + "complete": { + "files": [ + { + "path": "internal/skills/registry/registry.go", + "created": false, + "hunks": [ + { + "old_start": 299, + "old_len": 7, + "changed_lines": [ + 302 + ] + } + ] + }, + { + "path": "internal/skills/registry/registry_test.go", + "created": false, + "hunks": [ + { + "old_start": 125, + "old_len": 6, + "changed_lines": [ + 128 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "internal/skills/registry/registry.go", + "created": false, + "hunks": [ + { + "old_start": 299, + "old_len": 7, + "changed_lines": [ + 302 + ] + } + ] + } + ] + }, + "omission_file": "internal/skills/registry/registry_test.go", + "omission_line": 128 + }, + { + "id": "v-8d2b059e", + "commit": "8d2b059e07f71c17068f2286617f23286e05e0c0", + "parent": "869c044391ba73f1adca31f52d4b025430006242", + "date": "2026-06-10", + "prompt": "fix: error when --comments is used with a comment argument", + "complete": { + "files": [ + { + "path": "pkg/cmd/discussion/view/view.go", + "created": false, + "hunks": [ + { + "old_start": 174, + "old_len": 6, + "changed_lines": [ + 177 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/view/view_test.go", + "created": false, + "hunks": [ + { + "old_start": 190, + "old_len": 14, + "changed_lines": [ + 193, + 194, + 195, + 196, + 197, + 198, + 199, + 200 + ] + }, + { + "old_start": 881, + "old_len": 7, + "changed_lines": [ + 884 + ] + }, + { + "old_start": 910, + "old_len": 7, + "changed_lines": [ + 913 + ] + }, + { + "old_start": 940, + "old_len": 7, + "changed_lines": [ + 943 + ] + }, + { + "old_start": 1003, + "old_len": 7, + "changed_lines": [ + 1006 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/discussion/view/view_test.go", + "created": false, + "hunks": [ + { + "old_start": 190, + "old_len": 14, + "changed_lines": [ + 193, + 194, + 195, + 196, + 197, + 198, + 199, + 200 + ] + }, + { + "old_start": 881, + "old_len": 7, + "changed_lines": [ + 884 + ] + }, + { + "old_start": 910, + "old_len": 7, + "changed_lines": [ + 913 + ] + }, + { + "old_start": 940, + "old_len": 7, + "changed_lines": [ + 943 + ] + }, + { + "old_start": 1003, + "old_len": 7, + "changed_lines": [ + 1006 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/discussion/view/view.go", + "omission_line": 177 + }, + { + "id": "v-2618999b", + "commit": "2618999bcb6c85d4554638937dc80c322a76d593", + "parent": "4166ecf2cada209714af79387a5243b334af764b", + "date": "2026-06-06", + "prompt": "feat: add comment manipulation methods", + "complete": { + "files": [ + { + "path": "pkg/cmd/discussion/client/client.go", + "created": false, + "hunks": [ + { + "old_start": 22, + "old_len": 4, + "changed_lines": [ + 25 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl.go", + "created": false, + "hunks": [ + { + "old_start": 1075, + "old_len": 3, + "changed_lines": [ + 1078 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl_test.go", + "created": false, + "hunks": [ + { + "old_start": 3599, + "old_len": 3, + "changed_lines": [ + 3602 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_mock.go", + "created": false, + "hunks": [ + { + "old_start": 18, + "old_len": 12, + "changed_lines": [ + 21, + 24, + 27 + ] + }, + { + "old_start": 45, + "old_len": 6, + "changed_lines": [ + 48 + ] + }, + { + "old_start": 52, + "old_len": 12, + "changed_lines": [ + 55, + 58, + 61 + ] + }, + { + "old_start": 79, + "old_len": 8, + "changed_lines": [ + 82, + 84 + ] + }, + { + "old_start": 88, + "old_len": 6, + "changed_lines": [ + 91 + ] + }, + { + "old_start": 95, + "old_len": 6, + "changed_lines": [ + 98 + ] + }, + { + "old_start": 162, + "old_len": 9, + "changed_lines": [ + 165, + 167 + ] + }, + { + "old_start": 172, + "old_len": 6, + "changed_lines": [ + 175 + ] + }, + { + "old_start": 210, + "old_len": 6, + "changed_lines": [ + 213 + ] + }, + { + "old_start": 246, + "old_len": 6, + "changed_lines": [ + 249 + ] + }, + { + "old_start": 533, + "old_len": 3, + "changed_lines": [ + 536 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/discussion/client/client.go", + "created": false, + "hunks": [ + { + "old_start": 22, + "old_len": 4, + "changed_lines": [ + 25 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl.go", + "created": false, + "hunks": [ + { + "old_start": 1075, + "old_len": 3, + "changed_lines": [ + 1078 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_mock.go", + "created": false, + "hunks": [ + { + "old_start": 18, + "old_len": 12, + "changed_lines": [ + 21, + 24, + 27 + ] + }, + { + "old_start": 45, + "old_len": 6, + "changed_lines": [ + 48 + ] + }, + { + "old_start": 52, + "old_len": 12, + "changed_lines": [ + 55, + 58, + 61 + ] + }, + { + "old_start": 79, + "old_len": 8, + "changed_lines": [ + 82, + 84 + ] + }, + { + "old_start": 88, + "old_len": 6, + "changed_lines": [ + 91 + ] + }, + { + "old_start": 95, + "old_len": 6, + "changed_lines": [ + 98 + ] + }, + { + "old_start": 162, + "old_len": 9, + "changed_lines": [ + 165, + 167 + ] + }, + { + "old_start": 172, + "old_len": 6, + "changed_lines": [ + 175 + ] + }, + { + "old_start": 210, + "old_len": 6, + "changed_lines": [ + 213 + ] + }, + { + "old_start": 246, + "old_len": 6, + "changed_lines": [ + 249 + ] + }, + { + "old_start": 533, + "old_len": 3, + "changed_lines": [ + 536 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/discussion/client/client_impl_test.go", + "omission_line": 3602 + }, + { + "id": "v-e2d150da", + "commit": "e2d150da420b8bf84f3097f6fad3bbb715ea1cb4", + "parent": "9d413e769a3604194364421db34bcc0128696d09", + "date": "2026-06-09", + "prompt": "remove redundant error wrapping on ListCategories", + "complete": { + "files": [ + { + "path": "pkg/cmd/discussion/create/create.go", + "created": false, + "hunks": [ + { + "old_start": 113, + "old_len": 7, + "changed_lines": [ + 116 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/create/create_test.go", + "created": false, + "hunks": [ + { + "old_start": 242, + "old_len": 7, + "changed_lines": [ + 245 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/edit/edit.go", + "created": false, + "hunks": [ + { + "old_start": 175, + "old_len": 7, + "changed_lines": [ + 178 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/discussion/create/create.go", + "created": false, + "hunks": [ + { + "old_start": 113, + "old_len": 7, + "changed_lines": [ + 116 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/create/create_test.go", + "created": false, + "hunks": [ + { + "old_start": 242, + "old_len": 7, + "changed_lines": [ + 245 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/discussion/edit/edit.go", + "omission_line": 178 + }, + { + "id": "v-c1f3c1a1", + "commit": "c1f3c1a164ab67436d525769adbce0fd67dd5e70", + "parent": "e61df0721a01641e15c463a8cbb829fa40242eec", + "date": "2026-06-08", + "prompt": "add missing repo flag override", + "complete": { + "files": [ + { + "path": "pkg/cmd/discussion/list/list.go", + "created": false, + "hunks": [ + { + "old_start": 129, + "old_len": 6, + "changed_lines": [ + 132 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/view/view.go", + "created": false, + "hunks": [ + { + "old_start": 189, + "old_len": 6, + "changed_lines": [ + 192 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/discussion/list/list.go", + "created": false, + "hunks": [ + { + "old_start": 129, + "old_len": 6, + "changed_lines": [ + 132 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/discussion/view/view.go", + "omission_line": 192 + }, + { + "id": "v-b1029009", + "commit": "b1029009dbfbcf4240472097e12614dbc3cdcd19", + "parent": "d6a089d5ce30e5b09bac22338f7949ef7ebbd36f", + "date": "2026-06-05", + "prompt": "handle partial failure on create/update label mutations", + "complete": { + "files": [ + { + "path": "pkg/cmd/discussion/client/client.go", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl.go", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 6, + "changed_lines": [ + 4 + ] + }, + { + "old_start": 925, + "old_len": 6, + "changed_lines": [ + 928 + ] + }, + { + "old_start": 957, + "old_len": 12, + "changed_lines": [ + 960, + 963, + 965 + ] + }, + { + "old_start": 974, + "old_len": 9, + "changed_lines": [ + 977, + 980 + ] + }, + { + "old_start": 1021, + "old_len": 12, + "changed_lines": [ + 1024, + 1027, + 1029 + ] + }, + { + "old_start": 1038, + "old_len": 5, + "changed_lines": [ + 1041 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl_test.go", + "created": false, + "hunks": [ + { + "old_start": 2813, + "old_len": 7, + "changed_lines": [ + 2816 + ] + }, + { + "old_start": 2866, + "old_len": 7, + "changed_lines": [ + 2869 + ] + }, + { + "old_start": 2885, + "old_len": 6, + "changed_lines": [ + 2888 + ] + }, + { + "old_start": 3509, + "old_len": 6, + "changed_lines": [ + 3512 + ] + }, + { + "old_start": 3526, + "old_len": 6, + "changed_lines": [ + 3529 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/create/create.go", + "created": false, + "hunks": [ + { + "old_start": 188, + "old_len": 6, + "changed_lines": [ + 191 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/edit/edit.go", + "created": false, + "hunks": [ + { + "old_start": 210, + "old_len": 6, + "changed_lines": [ + 213 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/discussion/client/client.go", + "created": false, + "hunks": [ + { + "old_start": 16, + "old_len": 6, + "changed_lines": [ + 19 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl.go", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 6, + "changed_lines": [ + 4 + ] + }, + { + "old_start": 925, + "old_len": 6, + "changed_lines": [ + 928 + ] + }, + { + "old_start": 957, + "old_len": 12, + "changed_lines": [ + 960, + 963, + 965 + ] + }, + { + "old_start": 974, + "old_len": 9, + "changed_lines": [ + 977, + 980 + ] + }, + { + "old_start": 1021, + "old_len": 12, + "changed_lines": [ + 1024, + 1027, + 1029 + ] + }, + { + "old_start": 1038, + "old_len": 5, + "changed_lines": [ + 1041 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/client/client_impl_test.go", + "created": false, + "hunks": [ + { + "old_start": 2813, + "old_len": 7, + "changed_lines": [ + 2816 + ] + }, + { + "old_start": 2866, + "old_len": 7, + "changed_lines": [ + 2869 + ] + }, + { + "old_start": 2885, + "old_len": 6, + "changed_lines": [ + 2888 + ] + }, + { + "old_start": 3509, + "old_len": 6, + "changed_lines": [ + 3512 + ] + }, + { + "old_start": 3526, + "old_len": 6, + "changed_lines": [ + 3529 + ] + } + ] + }, + { + "path": "pkg/cmd/discussion/create/create.go", + "created": false, + "hunks": [ + { + "old_start": 188, + "old_len": 6, + "changed_lines": [ + 191 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/discussion/edit/edit.go", + "omission_line": 213 + }, + { + "id": "v-16a20347", + "commit": "16a20347dd33b8f67abd8990ab0940d35f522233", + "parent": "55808753070c023c45b40592fbfc624d8f03a759", + "date": "2026-05-20", + "prompt": "fix warning message to make it clear", + "complete": { + "files": [ + { + "path": "pkg/cmd/skills/update/update.go", + "created": false, + "hunks": [ + { + "old_start": 338, + "old_len": 7, + "changed_lines": [ + 341 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/update/update_test.go", + "created": false, + "hunks": [ + { + "old_start": 508, + "old_len": 7, + "changed_lines": [ + 511 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "pkg/cmd/skills/update/update.go", + "created": false, + "hunks": [ + { + "old_start": 338, + "old_len": 7, + "changed_lines": [ + 341 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/skills/update/update_test.go", + "omission_line": 511 + }, + { + "id": "v-fb748cb2", + "commit": "fb748cb2bf3a434ff12f5268c9983a65310c6520", + "parent": "00fc8c923ab3f321a412e1c89e425c107122f0bb", + "date": "2026-05-19", + "prompt": "add logic to preview too", + "complete": { + "files": [ + { + "path": "internal/skills/discovery/discovery.go", + "created": false, + "hunks": [ + { + "old_start": 390, + "old_len": 6, + "changed_lines": [ + 393 + ] + } + ] + }, + { + "path": "internal/skills/discovery/discovery_test.go", + "created": false, + "hunks": [ + { + "old_start": 1526, + "old_len": 6, + "changed_lines": [ + 1529 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 551, + "old_len": 23, + "changed_lines": [ + 554, + 555, + 556, + 557, + 558, + 559, + 560, + 561, + 562, + 563, + 564, + 565, + 566, + 567, + 568, + 569, + 570 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/preview/preview.go", + "created": false, + "hunks": [ + { + "old_start": 69, + "old_len": 6, + "changed_lines": [ + 72 + ] + }, + { + "old_start": 82, + "old_len": 6, + "changed_lines": [ + 85 + ] + }, + { + "old_start": 153, + "old_len": 25, + "changed_lines": [ + 156, + 157, + 158, + 159, + 160, + 161, + 163, + 164, + 165, + 166, + 168, + 169, + 170, + 172, + 173, + 174 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/preview/preview_test.go", + "created": false, + "hunks": [ + { + "old_start": 261, + "old_len": 6, + "changed_lines": [ + 264 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "internal/skills/discovery/discovery.go", + "created": false, + "hunks": [ + { + "old_start": 390, + "old_len": 6, + "changed_lines": [ + 393 + ] + } + ] + }, + { + "path": "internal/skills/discovery/discovery_test.go", + "created": false, + "hunks": [ + { + "old_start": 1526, + "old_len": 6, + "changed_lines": [ + 1529 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/install/install.go", + "created": false, + "hunks": [ + { + "old_start": 551, + "old_len": 23, + "changed_lines": [ + 554, + 555, + 556, + 557, + 558, + 559, + 560, + 561, + 562, + 563, + 564, + 565, + 566, + 567, + 568, + 569, + 570 + ] + } + ] + }, + { + "path": "pkg/cmd/skills/preview/preview.go", + "created": false, + "hunks": [ + { + "old_start": 69, + "old_len": 6, + "changed_lines": [ + 72 + ] + }, + { + "old_start": 82, + "old_len": 6, + "changed_lines": [ + 85 + ] + }, + { + "old_start": 153, + "old_len": 25, + "changed_lines": [ + 156, + 157, + 158, + 159, + 160, + 161, + 163, + 164, + 165, + 166, + 168, + 169, + 170, + 172, + 173, + 174 + ] + } + ] + } + ] + }, + "omission_file": "pkg/cmd/skills/preview/preview_test.go", + "omission_line": 264 + }, + { + "id": "v-a44721d2", + "commit": "a44721d233be9a2f6f0b5ee5c4f71274acb8d296", + "parent": "9c4184de6f8c208a11e4329b90fa9844efd728e9", + "date": "2026-05-07", + "prompt": "Add explicit build tags to platform-specific echo test files", + "complete": { + "files": [ + { + "path": "internal/prompter/echo_darwin_test.go", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 3, + "changed_lines": [ + 1 + ] + } + ] + }, + { + "path": "internal/prompter/echo_linux_test.go", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 3, + "changed_lines": [ + 1 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "internal/prompter/echo_darwin_test.go", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 3, + "changed_lines": [ + 1 + ] + } + ] + } + ] + }, + "omission_file": "internal/prompter/echo_linux_test.go", + "omission_line": 1 + } + ] +} \ No newline at end of file diff --git a/benchmarks/results/verify-reify/verify-environment.json b/benchmarks/results/verify-reify/verify-environment.json new file mode 100644 index 0000000..ea65ff7 --- /dev/null +++ b/benchmarks/results/verify-reify/verify-environment.json @@ -0,0 +1,25 @@ +{ + "after": null, + "candidates_rejected": 0, + "checker": "symbols changed by this diff, minus symbols present in the diff, where an inbound CALLS edge exists at distance 1, via reify::query::impact", + "count": 60, + "head": "0b0bcf5cf5fc25f4c7325f108a0474e1d2895cda", + "languages": [ + [ + "rust", + 31 + ], + [ + "python", + 12 + ] + ], + "origin": "git@github.com:lambiengcode/reify.git", + "reify_version": "0.2.2", + "repository": ".", + "scan": 4000, + "token_counts": "estimated by reify heuristic-v1", + "trials": 4, + "until": "870905b", + "wall_clock_ms": 867 +} \ No newline at end of file diff --git a/benchmarks/results/verify-reify/verify-outcomes.json b/benchmarks/results/verify-reify/verify-outcomes.json new file mode 100644 index 0000000..9d85313 --- /dev/null +++ b/benchmarks/results/verify-reify/verify-outcomes.json @@ -0,0 +1,270 @@ +[ + { + "task": "v-9af59e47", + "commit": "9af59e472b5bac3dae35323bcac223c922b89882", + "omission_file": "crates/reify/tests/fixtures.rs", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 19, + "false_alarms": 19, + "changed_symbols": 2, + "verify_tokens": 422, + "verify_latency_ms": 1, + "index_ms": 151, + "cited": [ + "crates/reify-cli/src/main.rs:212", + "crates/reify/benches/queries.rs:28", + "crates/reify/benches/queries.rs:44", + "crates/reify/src/context.rs:1685", + "crates/reify/src/context.rs:2023", + "crates/reify/src/index.rs:1143", + "crates/reify/src/index.rs:1151", + "crates/reify/src/index.rs:1282", + "crates/reify/src/index.rs:1300", + "crates/reify/src/index.rs:1326", + "crates/reify/src/index.rs:1456", + "crates/reify/src/index.rs:1486", + "crates/reify/src/index.rs:1510", + "crates/reify/src/index.rs:1530", + "crates/reify/src/index.rs:1674", + "crates/reify/src/query.rs:1182", + "crates/reify/tests/fixtures.rs:292", + "crates/reify/tests/fixtures.rs:30", + "crates/reify/tests/fixtures.rs:312" + ], + "cited_on_complete": [ + "crates/reify-cli/src/main.rs:212", + "crates/reify/benches/queries.rs:28", + "crates/reify/benches/queries.rs:44", + "crates/reify/src/context.rs:1685", + "crates/reify/src/context.rs:2023", + "crates/reify/src/index.rs:1143", + "crates/reify/src/index.rs:1151", + "crates/reify/src/index.rs:1282", + "crates/reify/src/index.rs:1300", + "crates/reify/src/index.rs:1326", + "crates/reify/src/index.rs:1456", + "crates/reify/src/index.rs:1486", + "crates/reify/src/index.rs:1510", + "crates/reify/src/index.rs:1530", + "crates/reify/src/index.rs:1674", + "crates/reify/src/query.rs:1182", + "crates/reify/tests/fixtures.rs:292", + "crates/reify/tests/fixtures.rs:30", + "crates/reify/tests/fixtures.rs:312" + ] + }, + { + "task": "v-2b7bad4c", + "commit": "2b7bad4ccaca00cc8d751047361ade9a5066c91b", + "omission_file": "assets/make-logo.py", + "omission_symbol": null, + "omission_file_reachable": false, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 1, + "false_alarms": 1, + "changed_symbols": 2, + "verify_tokens": 25, + "verify_latency_ms": 0, + "index_ms": 151, + "cited": [ + "assets/make-social-preview.py:136" + ], + "cited_on_complete": [ + "assets/make-social-preview.py:136" + ] + }, + { + "task": "v-deb46ef8", + "commit": "deb46ef835c1694324e161b1055759c391f883a0", + "omission_file": "crates/reify-cli/src/render.rs", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": false, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 4, + "false_alarms": 4, + "changed_symbols": 2, + "verify_tokens": 89, + "verify_latency_ms": 0, + "index_ms": 149, + "cited": [ + "crates/reify-cli/src/main.rs:167", + "crates/reify-cli/src/mcp.rs:231", + "crates/reify-cli/src/mcp.rs:240", + "crates/reify-cli/src/mcp.rs:61" + ], + "cited_on_complete": [ + "crates/reify-cli/src/main.rs:167", + "crates/reify-cli/src/mcp.rs:231", + "crates/reify-cli/src/mcp.rs:240", + "crates/reify-cli/src/mcp.rs:61" + ] + }, + { + "task": "v-7dd36dae", + "commit": "7dd36daec587f5c19afe6cab881886da530d3d15", + "omission_file": "crates/reify-bench/src/conditions.rs", + "omission_symbol": null, + "omission_file_reachable": true, + "file_hit": true, + "file_hit_attributable": false, + "symbol_hit": null, + "findings": 70, + "false_alarms": 70, + "changed_symbols": 8, + "verify_tokens": 1557, + "verify_latency_ms": 2, + "index_ms": 144, + "cited": [ + "crates/reify-bench/src/conditions.rs:146", + "crates/reify-bench/src/conditions.rs:151", + "crates/reify-bench/src/conditions.rs:216", + "crates/reify-bench/src/main.rs:120", + "crates/reify-bench/src/main.rs:372", + "crates/reify-cli/src/main.rs:174", + "crates/reify-cli/src/mcp.rs:122", + "crates/reify/benches/queries.rs:74", + "crates/reify/src/concepts.rs:1030", + "crates/reify/src/concepts.rs:1145", + "crates/reify/src/concepts.rs:1371", + "crates/reify/src/concepts.rs:1426", + "crates/reify/src/concepts.rs:392", + "crates/reify/src/concepts.rs:472", + "crates/reify/src/concepts.rs:505", + "crates/reify/src/concepts.rs:701", + "crates/reify/src/context.rs:1086", + "crates/reify/src/context.rs:1107", + "crates/reify/src/context.rs:1133", + "crates/reify/src/context.rs:1152", + "crates/reify/src/context.rs:1169", + "crates/reify/src/context.rs:1193", + "crates/reify/src/context.rs:1285", + "crates/reify/src/context.rs:1300", + "crates/reify/src/context.rs:1326", + "crates/reify/src/context.rs:1352", + "crates/reify/src/context.rs:1375", + "crates/reify/src/context.rs:1411", + "crates/reify/src/context.rs:1433", + "crates/reify/src/context.rs:1458", + "crates/reify/src/context.rs:1481", + "crates/reify/src/context.rs:1494", + "crates/reify/src/context.rs:172", + "crates/reify/src/discover.rs:121", + "crates/reify/src/extract/code.rs:1214", + "crates/reify/src/extract/code.rs:1249", + "crates/reify/src/extract/code.rs:31", + "crates/reify/src/extract/code.rs:602", + "crates/reify/src/extract/code.rs:834", + "crates/reify/src/extract/code.rs:851", + "crates/reify/src/extract/code.rs:876", + "crates/reify/src/extract/code.rs:895", + "crates/reify/src/extract/code.rs:909", + "crates/reify/src/extract/code.rs:923", + "crates/reify/src/extract/docs.rs:30", + "crates/reify/src/extract/schema.rs:199", + "crates/reify/src/extract/schema.rs:207", + "crates/reify/src/extract/schema.rs:215", + "crates/reify/src/extract/sqlish.rs:146", + "crates/reify/src/extract/sqlish.rs:167", + "crates/reify/src/gitlog.rs:208", + "crates/reify/src/gitlog.rs:527", + "crates/reify/src/gitlog.rs:538", + "crates/reify/src/gitlog.rs:553", + "crates/reify/src/gitlog.rs:571", + "crates/reify/src/index.rs:1484", + "crates/reify/src/index.rs:1518", + "crates/reify/src/index.rs:1565", + "crates/reify/src/index.rs:1580", + "crates/reify/src/index.rs:209", + "crates/reify/src/index.rs:609", + "crates/reify/src/index.rs:892", + "crates/reify/src/rules.rs:619", + "crates/reify/src/store.rs:997", + "crates/reify/tests/fixtures.rs:109", + "crates/reify/tests/fixtures.rs:143", + "crates/reify/tests/fixtures.rs:159", + "crates/reify/tests/fixtures.rs:190", + "crates/reify/tests/fixtures.rs:210", + "crates/reify/tests/fixtures.rs:231" + ], + "cited_on_complete": [ + "crates/reify-bench/src/conditions.rs:146", + "crates/reify-bench/src/conditions.rs:151", + "crates/reify-bench/src/conditions.rs:216", + "crates/reify-bench/src/main.rs:120", + "crates/reify-bench/src/main.rs:372", + "crates/reify-cli/src/main.rs:174", + "crates/reify-cli/src/mcp.rs:122", + "crates/reify/benches/queries.rs:74", + "crates/reify/src/concepts.rs:1030", + "crates/reify/src/concepts.rs:1145", + "crates/reify/src/concepts.rs:1371", + "crates/reify/src/concepts.rs:1426", + "crates/reify/src/concepts.rs:392", + "crates/reify/src/concepts.rs:472", + "crates/reify/src/concepts.rs:505", + "crates/reify/src/concepts.rs:701", + "crates/reify/src/context.rs:1086", + "crates/reify/src/context.rs:1107", + "crates/reify/src/context.rs:1133", + "crates/reify/src/context.rs:1152", + "crates/reify/src/context.rs:1169", + "crates/reify/src/context.rs:1193", + "crates/reify/src/context.rs:1285", + "crates/reify/src/context.rs:1300", + "crates/reify/src/context.rs:1326", + "crates/reify/src/context.rs:1352", + "crates/reify/src/context.rs:1375", + "crates/reify/src/context.rs:1411", + "crates/reify/src/context.rs:1433", + "crates/reify/src/context.rs:1458", + "crates/reify/src/context.rs:1481", + "crates/reify/src/context.rs:1494", + "crates/reify/src/context.rs:172", + "crates/reify/src/discover.rs:121", + "crates/reify/src/extract/code.rs:1214", + "crates/reify/src/extract/code.rs:1249", + "crates/reify/src/extract/code.rs:31", + "crates/reify/src/extract/code.rs:602", + "crates/reify/src/extract/code.rs:834", + "crates/reify/src/extract/code.rs:851", + "crates/reify/src/extract/code.rs:876", + "crates/reify/src/extract/code.rs:895", + "crates/reify/src/extract/code.rs:909", + "crates/reify/src/extract/code.rs:923", + "crates/reify/src/extract/docs.rs:30", + "crates/reify/src/extract/schema.rs:199", + "crates/reify/src/extract/schema.rs:207", + "crates/reify/src/extract/schema.rs:215", + "crates/reify/src/extract/sqlish.rs:146", + "crates/reify/src/extract/sqlish.rs:167", + "crates/reify/src/gitlog.rs:208", + "crates/reify/src/gitlog.rs:527", + "crates/reify/src/gitlog.rs:538", + "crates/reify/src/gitlog.rs:553", + "crates/reify/src/gitlog.rs:571", + "crates/reify/src/index.rs:1484", + "crates/reify/src/index.rs:1518", + "crates/reify/src/index.rs:1565", + "crates/reify/src/index.rs:1580", + "crates/reify/src/index.rs:209", + "crates/reify/src/index.rs:609", + "crates/reify/src/index.rs:892", + "crates/reify/src/rules.rs:619", + "crates/reify/src/store.rs:997", + "crates/reify/tests/fixtures.rs:109", + "crates/reify/tests/fixtures.rs:143", + "crates/reify/tests/fixtures.rs:159", + "crates/reify/tests/fixtures.rs:190", + "crates/reify/tests/fixtures.rs:210", + "crates/reify/tests/fixtures.rs:231" + ] + } +] \ No newline at end of file diff --git a/benchmarks/results/verify-reify/verify-summary.json b/benchmarks/results/verify-reify/verify-summary.json new file mode 100644 index 0000000..3a384b1 --- /dev/null +++ b/benchmarks/results/verify-reify/verify-summary.json @@ -0,0 +1,33 @@ +{ + "tasks": 4, + "omission_recall": 0.5, + "omission_recall_ci": [ + 0.15003571, + 0.84996426 + ], + "omission_recall_attributable": 0.0, + "omission_recall_attributable_ci": [ + 0.0, + 0.48990002 + ], + "reachable_omissions": 3, + "omission_recall_reachable": 0.6666667, + "omission_recall_reachable_ci": [ + 0.20765498, + 0.9385097 + ], + "symbol_scorable": 0, + "omission_recall_symbol": null, + "omission_recall_symbol_ci": null, + "false_alarm_rate": 23.5, + "commits_with_a_false_alarm": 4, + "false_alarm_share_ci": [ + 0.5101, + 1.0 + ], + "median_findings_per_diff": 19, + "median_verify_tokens": 422, + "median_verify_latency_ms": 1, + "median_index_ms": 151, + "diffs_resolving_to_nothing": 0 +} \ No newline at end of file diff --git a/benchmarks/results/verify-reify/verify-tasks.json b/benchmarks/results/verify-reify/verify-tasks.json new file mode 100644 index 0000000..aeb7218 --- /dev/null +++ b/benchmarks/results/verify-reify/verify-tasks.json @@ -0,0 +1,852 @@ +{ + "repository": ".", + "head": "0b0bcf5cf5fc25f4c7325f108a0474e1d2895cda", + "generated_from_commits": 48, + "rejected": [], + "tasks": [ + { + "id": "v-9af59e47", + "commit": "9af59e472b5bac3dae35323bcac223c922b89882", + "parent": "8c9e91d5b8a9dece2dc4edea705b96cb5dde6954", + "date": "2026-08-24", + "prompt": "a repository whose history git cannot read still indexes", + "complete": { + "files": [ + { + "path": "crates/reify/src/index.rs", + "created": false, + "hunks": [ + { + "old_start": 189, + "old_len": 6, + "changed_lines": [ + 192 + ] + }, + { + "old_start": 615, + "old_len": 7, + "changed_lines": [ + 618 + ] + } + ] + }, + { + "path": "crates/reify/tests/fixtures.rs", + "created": false, + "hunks": [ + { + "old_start": 365, + "old_len": 3, + "changed_lines": [ + 368 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "crates/reify/src/index.rs", + "created": false, + "hunks": [ + { + "old_start": 189, + "old_len": 6, + "changed_lines": [ + 192 + ] + }, + { + "old_start": 615, + "old_len": 7, + "changed_lines": [ + 618 + ] + } + ] + } + ] + }, + "omission_file": "crates/reify/tests/fixtures.rs", + "omission_line": 368 + }, + { + "id": "v-2b7bad4c", + "commit": "2b7bad4ccaca00cc8d751047361ade9a5066c91b", + "parent": "5a0d4c70e0962f82125f21e9bbb50c1abede9fa0", + "date": "2026-08-22", + "prompt": "a hand-drawn mascot, and one master it all derives from", + "complete": { + "files": [ + { + "path": "assets/make-logo.py", + "created": false, + "hunks": [ + { + "old_start": 1, + "old_len": 96, + "changed_lines": [ + 2, + 3, + 4, + 5, + 6, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93 + ] + } + ] + }, + { + "path": "assets/make-social-preview.py", + "created": false, + "hunks": [ + { + "old_start": 17, + "old_len": 6, + "changed_lines": [ + 20 + ] + }, + { + "old_start": 59, + "old_len": 23, + "changed_lines": [ + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78 + ] + }, + { + "old_start": 97, + "old_len": 13, + "changed_lines": [ + 100, + 105, + 106 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "assets/make-social-preview.py", + "created": false, + "hunks": [ + { + "old_start": 17, + "old_len": 6, + "changed_lines": [ + 20 + ] + }, + { + "old_start": 59, + "old_len": 23, + "changed_lines": [ + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78 + ] + }, + { + "old_start": 97, + "old_len": 13, + "changed_lines": [ + 100, + 105, + 106 + ] + } + ] + } + ] + }, + "omission_file": "assets/make-logo.py", + "omission_line": 2 + }, + { + "id": "v-deb46ef8", + "commit": "deb46ef835c1694324e161b1055759c391f883a0", + "parent": "cbfcad5a1756473d49576a47242de80392514a43", + "date": "2026-08-21", + "prompt": "TOON output for agents — 57% fewer tokens than the JSON envelope, measured cost in the header", + "complete": { + "files": [ + { + "path": "crates/reify-cli/src/main.rs", + "created": false, + "hunks": [ + { + "old_start": 76, + "old_len": 6, + "changed_lines": [ + 79 + ] + }, + { + "old_start": 212, + "old_len": 6, + "changed_lines": [ + 215 + ] + }, + { + "old_start": 223, + "old_len": 6, + "changed_lines": [ + 226 + ] + }, + { + "old_start": 435, + "old_len": 7, + "changed_lines": [ + 438 + ] + } + ] + }, + { + "path": "crates/reify-cli/src/mcp.rs", + "created": false, + "hunks": [ + { + "old_start": 143, + "old_len": 6, + "changed_lines": [ + 146 + ] + }, + { + "old_start": 167, + "old_len": 7, + "changed_lines": [ + 170 + ] + } + ] + }, + { + "path": "crates/reify-cli/src/render.rs", + "created": false, + "hunks": [ + { + "old_start": 863, + "old_len": 3, + "changed_lines": [ + 866 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "crates/reify-cli/src/main.rs", + "created": false, + "hunks": [ + { + "old_start": 76, + "old_len": 6, + "changed_lines": [ + 79 + ] + }, + { + "old_start": 212, + "old_len": 6, + "changed_lines": [ + 215 + ] + }, + { + "old_start": 223, + "old_len": 6, + "changed_lines": [ + 226 + ] + }, + { + "old_start": 435, + "old_len": 7, + "changed_lines": [ + 438 + ] + } + ] + }, + { + "path": "crates/reify-cli/src/mcp.rs", + "created": false, + "hunks": [ + { + "old_start": 143, + "old_len": 6, + "changed_lines": [ + 146 + ] + }, + { + "old_start": 167, + "old_len": 7, + "changed_lines": [ + 170 + ] + } + ] + } + ] + }, + "omission_file": "crates/reify-cli/src/render.rs", + "omission_line": 866 + }, + { + "id": "v-7dd36dae", + "commit": "7dd36daec587f5c19afe6cab881886da530d3d15", + "parent": "b65dcf2cb0aa1ca550c24549ae19901d7162e1ce", + "date": "2026-08-21", + "prompt": "verbatim identifier lookup, stemmed prefix search, file-aggregate ordering, offer cutoff; bench: rank audit", + "complete": { + "files": [ + { + "path": "crates/reify-bench/src/conditions.rs", + "created": false, + "hunks": [ + { + "old_start": 328, + "old_len": 3, + "changed_lines": [ + 331 + ] + } + ] + }, + { + "path": "crates/reify-bench/src/main.rs", + "created": false, + "hunks": [ + { + "old_start": 80, + "old_len": 6, + "changed_lines": [ + 83 + ] + }, + { + "old_start": 166, + "old_len": 12, + "changed_lines": [ + 169, + 175 + ] + }, + { + "old_start": 212, + "old_len": 99, + "changed_lines": [ + 215, + 216, + 217, + 218, + 219, + 220, + 224, + 225, + 226, + 227, + 228, + 229, + 230, + 231, + 232, + 233, + 235, + 240, + 241, + 242, + 243, + 244, + 245, + 246, + 247, + 248, + 249, + 250, + 251, + 252, + 253, + 254, + 255, + 256, + 257, + 258, + 259, + 260, + 261, + 262, + 263, + 264, + 265, + 266, + 267, + 268, + 269, + 270, + 271, + 272, + 273, + 274, + 275, + 276, + 277, + 278, + 279, + 280, + 281, + 282, + 283, + 284, + 285, + 286, + 287, + 288, + 289, + 290, + 292, + 293, + 299, + 303, + 304, + 305, + 306, + 307 + ] + }, + { + "old_start": 318, + "old_len": 6, + "changed_lines": [ + 321 + ] + } + ] + }, + { + "path": "crates/reify/src/context.rs", + "created": false, + "hunks": [ + { + "old_start": 56, + "old_len": 11, + "changed_lines": [ + 59, + 64 + ] + }, + { + "old_start": 141, + "old_len": 6, + "changed_lines": [ + 144 + ] + }, + { + "old_start": 151, + "old_len": 6, + "changed_lines": [ + 154 + ] + }, + { + "old_start": 340, + "old_len": 10, + "changed_lines": [ + 343, + 346 + ] + }, + { + "old_start": 361, + "old_len": 7, + "changed_lines": [ + 364 + ] + }, + { + "old_start": 425, + "old_len": 6, + "changed_lines": [ + 428 + ] + }, + { + "old_start": 846, + "old_len": 32, + "changed_lines": [ + 849, + 852, + 853, + 854, + 856, + 862, + 863, + 864, + 865, + 867, + 868, + 869, + 870, + 871, + 872, + 874 + ] + } + ] + }, + { + "path": "crates/reify/src/store.rs", + "created": false, + "hunks": [ + { + "old_start": 1097, + "old_len": 7, + "changed_lines": [ + 1100 + ] + }, + { + "old_start": 1548, + "old_len": 6, + "changed_lines": [ + 1551 + ] + } + ] + } + ] + }, + "truncated": { + "files": [ + { + "path": "crates/reify-bench/src/main.rs", + "created": false, + "hunks": [ + { + "old_start": 80, + "old_len": 6, + "changed_lines": [ + 83 + ] + }, + { + "old_start": 166, + "old_len": 12, + "changed_lines": [ + 169, + 175 + ] + }, + { + "old_start": 212, + "old_len": 99, + "changed_lines": [ + 215, + 216, + 217, + 218, + 219, + 220, + 224, + 225, + 226, + 227, + 228, + 229, + 230, + 231, + 232, + 233, + 235, + 240, + 241, + 242, + 243, + 244, + 245, + 246, + 247, + 248, + 249, + 250, + 251, + 252, + 253, + 254, + 255, + 256, + 257, + 258, + 259, + 260, + 261, + 262, + 263, + 264, + 265, + 266, + 267, + 268, + 269, + 270, + 271, + 272, + 273, + 274, + 275, + 276, + 277, + 278, + 279, + 280, + 281, + 282, + 283, + 284, + 285, + 286, + 287, + 288, + 289, + 290, + 292, + 293, + 299, + 303, + 304, + 305, + 306, + 307 + ] + }, + { + "old_start": 318, + "old_len": 6, + "changed_lines": [ + 321 + ] + } + ] + }, + { + "path": "crates/reify/src/context.rs", + "created": false, + "hunks": [ + { + "old_start": 56, + "old_len": 11, + "changed_lines": [ + 59, + 64 + ] + }, + { + "old_start": 141, + "old_len": 6, + "changed_lines": [ + 144 + ] + }, + { + "old_start": 151, + "old_len": 6, + "changed_lines": [ + 154 + ] + }, + { + "old_start": 340, + "old_len": 10, + "changed_lines": [ + 343, + 346 + ] + }, + { + "old_start": 361, + "old_len": 7, + "changed_lines": [ + 364 + ] + }, + { + "old_start": 425, + "old_len": 6, + "changed_lines": [ + 428 + ] + }, + { + "old_start": 846, + "old_len": 32, + "changed_lines": [ + 849, + 852, + 853, + 854, + 856, + 862, + 863, + 864, + 865, + 867, + 868, + 869, + 870, + 871, + 872, + 874 + ] + } + ] + }, + { + "path": "crates/reify/src/store.rs", + "created": false, + "hunks": [ + { + "old_start": 1097, + "old_len": 7, + "changed_lines": [ + 1100 + ] + }, + { + "old_start": 1548, + "old_len": 6, + "changed_lines": [ + 1551 + ] + } + ] + } + ] + }, + "omission_file": "crates/reify-bench/src/conditions.rs", + "omission_line": 331 + } + ] +} \ No newline at end of file diff --git a/benchmarks/swe/README.md b/benchmarks/swe/README.md index d2d2470..60fc787 100644 --- a/benchmarks/swe/README.md +++ b/benchmarks/swe/README.md @@ -28,12 +28,12 @@ baseline over the same repository. The model returns SEARCH/REPLACE edit blocks, are applied so `git diff` yields a prediction the official harness can judge. ```bash -python3 stage2c.py # generate patches, both arms +STAGE2_MODEL=sonnet python3 stage2.py # both arms ./eval_batched.sh # grade them, batched python3 diagnose2.py # paired result + why ``` -`stage2c.py` numbers every line of the retrieved context and asks for line-range +`stage2.py` numbers every line of the retrieved context and asks for line-range replacements. That detail is load-bearing: asked for exact SEARCH text instead, the model reproduces these famous repositories from memory and ~45% of patches fail to apply. With line ranges it is ~1%. diff --git a/benchmarks/swe/results/stage1-retrieval.txt b/benchmarks/swe/results/stage1-retrieval.txt index 499a1ae..c072454 100644 --- a/benchmarks/swe/results/stage1-retrieval.txt +++ b/benchmarks/swe/results/stage1-retrieval.txt @@ -3,19 +3,19 @@ SWE-bench Verified retrieval — 500 instances, budget 4000 tok condition hit (any gold file offered) MRR full recall B-content-grep 6.6% [4.7%-9.1%] n=500 0.06 5.6% C-path-grep 9.0% [6.8%-11.8%] n=500 0.06 7.8% -R-reify 66.0% [61.7%-70.0%] n=500 0.43 59.0% -R-reify-iter3 84.6% [81.2%-87.5%] n=500 0.45 77.0% +R-reify 72.6% [68.5%-76.3%] n=500 0.42 65.4% +R-reify-iter3 87.0% [83.8%-89.7%] n=500 0.43 81.4% per-repo, reify×3 vs grep (hit rate): astropy grep 0% reify×3 77% (n=22) - django grep 6% reify×3 88% (n=231) - matplotlib grep 0% reify×3 91% (n=34) + django grep 6% reify×3 90% (n=231) + matplotlib grep 0% reify×3 97% (n=34) mwaskom grep 0% reify×3 100% (n=2) pallets grep 0% reify×3 100% (n=1) psf grep 0% reify×3 100% (n=8) - pydata grep 9% reify×3 91% (n=22) + pydata grep 9% reify×3 95% (n=22) pylint-dev grep 10% reify×3 60% (n=10) - pytest-dev grep 26% reify×3 84% (n=19) + pytest-dev grep 26% reify×3 95% (n=19) scikit-learn grep 9% reify×3 88% (n=32) - sphinx-doc grep 7% reify×3 75% (n=44) - sympy grep 7% reify×3 77% (n=75) + sphinx-doc grep 7% reify×3 80% (n=44) + sympy grep 7% reify×3 79% (n=75) diff --git a/benchmarks/swe/results/stage2-endtoend.json b/benchmarks/swe/results/stage2-endtoend.json index 3f2e01f..3bbcf99 100644 --- a/benchmarks/swe/results/stage2-endtoend.json +++ b/benchmarks/swe/results/stage2-endtoend.json @@ -1,222 +1,42 @@ { - "reify": { - "resolved_ids": [ - "django__django-11133", - "django__django-13670", - "django__django-15467", - "pallets__flask-5014", - "scikit-learn__scikit-learn-14894", - "scikit-learn__scikit-learn-26323", - "sphinx-doc__sphinx-10673", - "sphinx-doc__sphinx-8475", - "sympy__sympy-16886" + "protocol": "SWE-bench paper protocol: one model, one budget, retriever is the only difference", + "model": "claude-sonnet-5 (via `claude -p`)", + "note": "Model differs from the earlier DeepSeek run, which ran out of balance. Absolute rates are NOT comparable to the previously published 23.8% tie; the paired arm-vs-arm comparison within this run is what stands.", + "graded_under_both_arms": 101, + "reify": { + "resolved": 74, + "empty_patches": 2 + }, + "bm25": { + "resolved": 68, + "empty_patches": 1 + }, + "paired": { + "reify_only": 12, + "bm25_only": 6, + "both": 62, + "exact_mcnemar_p": 0.2379 + }, + "reify_only_ids": [ + "astropy__astropy-8707", + "django__django-11206", + "django__django-14534", + "django__django-14999", + "matplotlib__matplotlib-25311", + "mwaskom__seaborn-3069", + "pallets__flask-5014", + "psf__requests-5414", + "pylint-dev__pylint-7277", + "pytest-dev__pytest-6197", + "sphinx-doc__sphinx-10449", + "sympy__sympy-13031" ], - "unresolved_ids": [ - "astropy__astropy-13236", - "astropy__astropy-7671", - "astropy__astropy-8707", - "astropy__astropy-8872", - "django__django-10973", - "django__django-11141", - "django__django-11149", - "django__django-11206", - "django__django-11451", - "django__django-11477", - "django__django-11490", - "django__django-11551", - "django__django-12262", - "django__django-12713", - "django__django-12965", - "django__django-13109", - "django__django-13121", - "django__django-13212", - "django__django-13344", - "django__django-13512", - "django__django-13933", - "django__django-14011", - "django__django-14534", - "django__django-14631", - "django__django-14725", - "django__django-14999", - "django__django-15161", - "django__django-15382", - "django__django-15851", - "django__django-15916", - "django__django-15987", - "django__django-16255", - "django__django-16454", - "django__django-16493", - "django__django-16612", - "django__django-16662", - "django__django-16950", - "matplotlib__matplotlib-23314", - "matplotlib__matplotlib-24026", - "matplotlib__matplotlib-24570", - "matplotlib__matplotlib-25311", - "mwaskom__seaborn-3069", - "psf__requests-2931", - "psf__requests-5414", - "pydata__xarray-3993", - "pydata__xarray-4094", - "pydata__xarray-6938", - "pydata__xarray-7233", - "pylint-dev__pylint-4970", - "pylint-dev__pylint-7277", - "pytest-dev__pytest-5787", - "pytest-dev__pytest-6197", - "pytest-dev__pytest-7324", - "pytest-dev__pytest-7490", - "scikit-learn__scikit-learn-12682", - "scikit-learn__scikit-learn-13124", - "scikit-learn__scikit-learn-13142", - "scikit-learn__scikit-learn-13328", - "sphinx-doc__sphinx-10449", - "sphinx-doc__sphinx-10614", - "sphinx-doc__sphinx-7748", - "sphinx-doc__sphinx-7889", - "sphinx-doc__sphinx-8056", - "sphinx-doc__sphinx-9367", - "sphinx-doc__sphinx-9591", - "sympy__sympy-13031", - "sympy__sympy-13615", - "sympy__sympy-13647", - "sympy__sympy-13757", - "sympy__sympy-13877", - "sympy__sympy-17655", - "sympy__sympy-18199", - "sympy__sympy-19346", - "sympy__sympy-19954", - "sympy__sympy-20801", - "sympy__sympy-21847" - ], - "empty_patch_ids": [ - "django__django-10914", - "django__django-11734", - "django__django-13809", - "django__django-13925", - "django__django-14053", - "django__django-14351", - "django__django-14500", - "django__django-14752", - "django__django-15525", - "django__django-16801", - "matplotlib__matplotlib-21568", - "matplotlib__matplotlib-24870", - "matplotlib__matplotlib-25479", - "sympy__sympy-18211", - "sympy__sympy-19495", - "sympy__sympy-24562" - ], - "error_ids": [] - }, - "bm25": { - "resolved_ids": [ - "django__django-10914", - "django__django-11133", - "django__django-11551", - "django__django-13670", - "django__django-14752", - "django__django-15467", - "django__django-16255", - "django__django-16493", - "django__django-16801", - "matplotlib__matplotlib-24570", - "pylint-dev__pylint-4970", - "scikit-learn__scikit-learn-12682", - "scikit-learn__scikit-learn-26323", - "sympy__sympy-13647", - "sympy__sympy-13877", - "sympy__sympy-16886" - ], - "unresolved_ids": [ - "astropy__astropy-13236", - "astropy__astropy-7671", - "astropy__astropy-8707", - "astropy__astropy-8872", - "django__django-10973", - "django__django-11141", - "django__django-11149", - "django__django-11206", - "django__django-11451", - "django__django-11477", - "django__django-11490", - "django__django-11734", - "django__django-12713", - "django__django-13109", - "django__django-13121", - "django__django-13212", - "django__django-13512", - "django__django-13809", - "django__django-13933", - "django__django-14351", - "django__django-14500", - "django__django-14534", - "django__django-14631", - "django__django-14725", - "django__django-14999", - "django__django-15161", - "django__django-15851", - "django__django-15916", - "django__django-15987", - "django__django-16612", - "django__django-16662", - "django__django-16950", - "matplotlib__matplotlib-21568", - "matplotlib__matplotlib-23314", - "matplotlib__matplotlib-24026", - "matplotlib__matplotlib-24870", - "matplotlib__matplotlib-25311", - "matplotlib__matplotlib-25479", - "mwaskom__seaborn-3069", - "pallets__flask-5014", - "psf__requests-2931", - "psf__requests-5414", - "pydata__xarray-3993", - "pydata__xarray-4094", - "pydata__xarray-6938", - "pydata__xarray-7233", - "pylint-dev__pylint-7277", - "pytest-dev__pytest-6197", - "pytest-dev__pytest-7490", - "scikit-learn__scikit-learn-13124", - "scikit-learn__scikit-learn-13142", - "scikit-learn__scikit-learn-13328", - "scikit-learn__scikit-learn-14894", - "sphinx-doc__sphinx-10449", - "sphinx-doc__sphinx-10614", - "sphinx-doc__sphinx-7748", - "sphinx-doc__sphinx-7889", - "sphinx-doc__sphinx-8056", - "sphinx-doc__sphinx-8475", - "sphinx-doc__sphinx-9367", - "sphinx-doc__sphinx-9591", - "sympy__sympy-13757", - "sympy__sympy-17655", - "sympy__sympy-18199", - "sympy__sympy-19495", - "sympy__sympy-19954", - "sympy__sympy-21847", - "sympy__sympy-24562" - ], - "empty_patch_ids": [ - "django__django-12262", - "django__django-12965", - "django__django-13344", - "django__django-13925", - "django__django-14011", - "django__django-14053", - "django__django-15382", - "django__django-15525", - "django__django-16454", - "pytest-dev__pytest-5787", - "pytest-dev__pytest-7324", - "sphinx-doc__sphinx-10673", - "sympy__sympy-13031", - "sympy__sympy-13615", - "sympy__sympy-18211", - "sympy__sympy-19346", - "sympy__sympy-20801" - ], - "error_ids": [] - } -} \ No newline at end of file + "bm25_only_ids": [ + "django__django-14752", + "django__django-15382", + "matplotlib__matplotlib-24570", + "pytest-dev__pytest-5787", + "sphinx-doc__sphinx-8056", + "sphinx-doc__sphinx-9591" + ] +} diff --git a/benchmarks/swe/stage2c.py b/benchmarks/swe/stage2.py similarity index 72% rename from benchmarks/swe/stage2c.py rename to benchmarks/swe/stage2.py index 30d8fb3..7fa8014 100644 --- a/benchmarks/swe/stage2c.py +++ b/benchmarks/swe/stage2.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Stage 2: retrieval-augmented patch generation on SWE-bench Verified. +"""Stage 2 (edit mode): the reify arm feeds REGIONS, the bm25 arm whole files. The SWE-bench paper's own protocol: one model, one budget, and the *retriever* is the only thing that changes between arms. Arm `reify` fills the context from @@ -8,12 +8,14 @@ (far more reliably applied than a hand-written unified diff), which are applied to the tree so `git diff` produces the prediction the official harness will judge. -Usage: stage2b.py +Usage: stage2.py +Model: $STAGE2_MODEL (default sonnet), passed to `claude -p`. """ -import json, math, pathlib, re, subprocess, sys, time +import json, math, os, pathlib, re, subprocess, sys, time from collections import Counter R = "/Users/lambiengcode/Documents/reify/projects/reify/target/release" +MODEL = os.environ.get("STAGE2_MODEL", "sonnet") CTX_CHARS = 48_000 # ~12k tokens of retrieved code, per arm, identical for both repo_dir = pathlib.Path(sys.argv[1]).resolve() @@ -71,18 +73,51 @@ def bm25_files(problem, k=8): scored.sort(key=lambda x: (-x[0], x[1])) return [p for _, p in scored[:k]] -def reify_files(problem, k=8): - r = sh([f"{R}/reify", "context", problem[:4000], "--budget", "4000", "--json"], - cwd=repo_dir, timeout=300) +def reify_regions(problem): + """Reify in edit mode: (path, start, end) regions padded to whole definitions. + + This is what `reify context` actually produces. Feeding whole files instead is what + cost the earlier run: one large file consumed the window and the file that mattered + never arrived at the model at all. + """ + r = sh([f"{R}/reify", "context", problem[:4000], "--budget", "4000", + "--for-edit", "--json"], cwd=repo_dir, timeout=300) if r.returncode: return [] data = json.loads(r.stdout) - seen, out = set(), [] + out, seen = [], set() for item in data.get("next_reads", []) + data.get("code", []): - p = item.get("path", "") - if p and p not in seen and (repo_dir / p).is_file(): - seen.add(p); out.append(p) - return out[:k] + p_, lines = item.get("path", ""), item.get("lines", "") + if not p_ or "-" not in lines or not (repo_dir / p_).is_file(): + continue + try: + a, b = (int(x) for x in lines.split("-", 1)) + except ValueError: + continue + if (p_, a, b) in seen: + continue + seen.add((p_, a, b)); out.append((p_, a, b)) + return out + + +def region_block(regions): + """Assemble the prompt from regions, each charged for its own size.""" + parts, used = [], 0 + for path, a, b in regions: + try: + lines = (repo_dir / path).read_text(errors="ignore").split("\n") + except OSError: + continue + a = max(1, a); b = min(len(lines), b) + if a > b: + continue + chunk = "\n".join(f"{i:5d}| {lines[i-1]}" for i in range(a, b + 1)) + if used + len(chunk) > CTX_CHARS: + break + parts.append(f"### File: {path} (lines {a}-{b})\n```python\n{chunk}\n```") + used += len(chunk) + return "\n\n".join(parts), used + def context_block(files): """Same character budget for both arms, so the model's cost is matched.""" @@ -160,10 +195,21 @@ def apply_edits(reply): return applied def ask(prompt): - r = subprocess.run(["deepseek-axi", "ask", prompt], capture_output=True, - text=True, timeout=900) + """One model, both arms. + + Claude replaces DeepSeek here only because the DeepSeek account ran out of balance + mid-project. The arms still differ in nothing but the retriever, so the paired + comparison stands on its own — but absolute numbers are NOT comparable to the + earlier DeepSeek run and must never be presented as if the published tie moved. + + The model is a parameter because a half-Opus, half-Sonnet set would confound the + aggregate: every instance in one run must be answered by the same model. + """ + r = subprocess.run(["claude", "-p", prompt, "--model", MODEL], + capture_output=True, text=True, timeout=1800) return r.stdout if r.returncode == 0 else "" + done_file = out_root / "done.txt" done = set(done_file.read_text().split()) if done_file.exists() else set() log = open(out_root / "driver.log", "a", buffering=1) @@ -176,14 +222,15 @@ def ask(prompt): try: reset(inst["base_commit"]) sh([f"{R}/reify", "index", "-C", str(repo_dir)], timeout=2400) - picks = {"reify": reify_files(inst["problem_statement"]), - "bm25": bm25_files(inst["problem_statement"])} + reify_pick = reify_regions(inst["problem_statement"]) + bm25_pick = bm25_files(inst["problem_statement"]) stats = {} # Alternate which arm the provider sees first, so cache warmth cannot favour one. order = ["reify", "bm25"] if i % 2 == 0 else ["bm25", "reify"] for arm in order: reset(inst["base_commit"]) - ctx, used = context_block(picks[arm]) + ctx, used = (region_block(reify_pick) if arm == "reify" + else context_block(bm25_pick)) prompt = PROMPT.format(repo=inst["repo"], context=ctx, problem=inst["problem_statement"][:8000]) reply = ask(prompt) @@ -198,9 +245,10 @@ def ask(prompt): diff = sh(["git", "diff"], cwd=repo_dir).stdout with open(out_root / f"preds-{arm}.jsonl", "a") as f: f.write(json.dumps({"instance_id": iid, - "model_name_or_path": f"deepseek+{arm}", + "model_name_or_path": f"{MODEL}+{arm}", "model_patch": diff}) + "\n") - stats[arm] = (len(picks[arm]), used, n, len(diff)) + stats[arm] = (len(reify_pick) if arm == "reify" else len(bm25_pick), + used, n, len(diff)) reset(inst["base_commit"]) with open(done_file, "a") as f: f.write(iid + "\n") diff --git a/crates/reify-bench/src/agent.rs b/crates/reify-bench/src/agent.rs index 17c396b..a6dc923 100644 --- a/crates/reify-bench/src/agent.rs +++ b/crates/reify-bench/src/agent.rs @@ -57,9 +57,14 @@ pub struct AgentOutcome { /// /// Identical across conditions except for the CONTEXT block, so any difference in /// outcome is attributable to the context and not to the wording of the question. -pub fn prompt(task: &Task, context_block: &str) -> String { +/// +/// `repository` comes from the task set rather than from a literal. It was a literal +/// — `ERPNext` — until 2026-08-24, which meant every Medusa, OFBiz and OpenMRS run +/// told the model it was working on a different codebase than the one it was asked +/// about. See the dated notes at the top of the three affected reports. +pub fn prompt(repository: &str, task: &Task, context_block: &str) -> String { format!( - "You are helping a developer change a large existing codebase (ERPNext).\n\ + "You are helping a developer change a large existing codebase ({repository}).\n\ \n\ TASK: {}\n\ \n\ @@ -111,11 +116,12 @@ pub fn oracle_block(task: &Task) -> String { pub fn run( provider: &Provider, root: &Path, + repository: &str, task: &Task, condition: &str, context_block: &str, ) -> AgentOutcome { - let text = prompt(task, context_block); + let text = prompt(repository, task, context_block); let started = std::time::Instant::now(); let mut outcome = AgentOutcome { task: task.id.clone(), @@ -268,8 +274,8 @@ mod tests { #[test] fn the_prompt_differs_between_conditions_only_in_its_context() { - let a = prompt(&task(), "context A"); - let b = prompt(&task(), "context B"); + let a = prompt("erpnext", &task(), "context A"); + let b = prompt("erpnext", &task(), "context B"); let strip = |s: &str| s.replace("context A", "@").replace("context B", "@"); assert_eq!( strip(&a), @@ -278,9 +284,25 @@ mod tests { ); } + #[test] + fn the_prompt_names_the_repository_the_task_came_from() { + // Until 2026-08-24 this was the literal `ERPNext`, so three published + // model-in-the-loop tables asked about one codebase while naming another. + let set = crate::tasks::TaskSet { + repository: ".bench/medusa".into(), + head: "a".repeat(40), + generated_from_commits: 1, + base: None, + tasks: vec![task()], + }; + let text = prompt(set.repository_name(), &task(), "context"); + assert!(text.contains("medusa"), "{text}"); + assert!(!text.contains("ERPNext"), "{text}"); + } + #[test] fn an_empty_context_is_stated_rather_than_left_blank() { - assert!(prompt(&task(), "").contains("(none provided)")); + assert!(prompt("erpnext", &task(), "").contains("(none provided)")); } #[test] diff --git a/crates/reify-bench/src/conditions.rs b/crates/reify-bench/src/conditions.rs index 80906ad..a9af177 100644 --- a/crates/reify-bench/src/conditions.rs +++ b/crates/reify-bench/src/conditions.rs @@ -14,9 +14,13 @@ use std::collections::{BTreeSet, HashMap}; use reify::concepts::meaningful_words; use reify::context::{self, ContextOptions}; -use reify::store::Store; +use reify::model::{EdgeKind, Node}; +use reify::query; +use reify::store::{Direction, Store}; use reify::tokens; +use crate::tasks; + /// A condition's answer: an ordered list of files, and what it cost to produce. #[derive(Debug, Clone, Serialize)] pub struct Answer { @@ -345,6 +349,170 @@ pub fn rank_audit( }) } +// ---- the checker under test ------------------------------------------------- +// +// `reify verify` does not exist. What exists is the graph it would have to stand on, +// and that is what is measured here: *symbols changed by this diff, minus symbols +// present in the diff, where an inbound `CALLS` edge exists*. The query runs through +// the same `impact` machinery `reify impact` uses, so a number measured here is a +// number about the shipped substrate rather than about a checker written to be +// measured. +// +// Only `CALLS` edges at distance 1 count. `impact` also propagates two hops and +// crosses into the data layer; both are legitimate for "what breaks if I change +// this" and neither is what "the patch forgot to update a call site" means. Widening +// the query would raise recall and raise the false-alarm rate with it, which is the +// trade this benchmark exists to measure rather than to pre-empt. + +/// One thing the change touched that has a dependant the change did not touch. +#[derive(Debug, Clone, Serialize)] +pub struct Finding { + /// `path:line` of the dependant, as `reify impact` cites it. + pub location: String, + pub path: String, + /// The dependant's name. + pub what: String, + /// The changed symbol it depends on, in words an engineer can check. + pub reason: String, +} + +/// What the checker produced for one diff, and what it cost. +#[derive(Debug, Clone, Serialize)] +pub struct Findings { + pub findings: Vec, + /// Symbols the diff changes, `path:line`. The minuend of the query, reported so a + /// zero-finding result can be told apart from a diff that resolved to nothing. + pub changed_symbols: Vec, + /// Tokens the findings output itself would cost the agent that reads it. + pub answer_tokens: u32, + /// Wall clock for the query alone. Indexing is a one-off the real feature would + /// not repeat per check, and is timed separately. + pub elapsed_ms: u128, +} + +impl Findings { + /// The findings as an agent would be shown them; the string `answer_tokens` counts. + pub fn render(&self) -> String { + if self.findings.is_empty() { + return "reify verify: nothing in the graph says this patch is incomplete\n".into(); + } + let mut out = format!( + "reify verify: {} not updated by this patch\n", + self.findings.len() + ); + for finding in &self.findings { + out.push_str(&format!( + " {} {} — {}\n", + finding.location, finding.what, finding.reason + )); + } + out + } +} + +/// Does any symbol in `path` **call** a symbol in another file? +/// +/// The ceiling on this whole construction. Every finding is a caller, so a file whose +/// symbols call nothing outside themselves can never be cited, however good the query +/// gets. The direction matters and is easy to get backwards: what is called *into* +/// the file is irrelevant here. +/// +/// Measured rather than assumed, because "the query needs work" and "there is no edge +/// to find" are different conclusions and only one of them is fixable by writing +/// `reify verify`. +pub fn can_be_cited(store: &Store, path: &str) -> Result { + for symbol in store.symbols_in_file(path)? { + for (callee, _, _) in store.neighbors(symbol.id, Direction::Out, &[EdgeKind::Calls])? { + if callee.path.as_deref() != Some(path) { + return Ok(true); + } + } + } + Ok(false) +} + +/// Run the checker over one patch, against an index built at the patch's parent. +pub fn missing_callers(store: &Store, patch: &tasks::Patch) -> Result { + let started = std::time::Instant::now(); + + // Every symbol whose span overlaps a changed line. This is the exclusion set: + // a symbol the patch already edits is not something the patch forgot. + let mut touched: BTreeSet = BTreeSet::new(); + // The innermost symbol at each changed line. These are the origins — the same + // rule `store.symbol_at` applies, batched so a long hunk costs one query. + let mut origins: Vec = Vec::new(); + let mut seen: BTreeSet = BTreeSet::new(); + + for file in &patch.files { + if file.created { + continue; + } + let symbols = store.symbols_in_file(&file.path)?; + if symbols.is_empty() { + continue; + } + for hunk in &file.hunks { + for &line in &hunk.changed_lines { + let mut innermost: Option<&Node> = None; + for symbol in &symbols { + if symbol.line_start > line || symbol.line_end < line { + continue; + } + touched.insert(symbol.location()); + let narrower = innermost.is_none_or(|best| { + symbol.line_end - symbol.line_start < best.line_end - best.line_start + }); + if narrower { + innermost = Some(symbol); + } + } + if let Some(symbol) = innermost { + if seen.insert(symbol.id) { + origins.push(symbol.clone()); + } + } + } + } + } + + let mut findings: Vec = Vec::new(); + let mut cited: BTreeSet = BTreeSet::new(); + for origin in &origins { + let answer = query::impact(store, &origin.location())?; + for affected in answer.affected { + // Distance 1 and a call: the edge the query is defined on. Data coupling + // and callers-of-callers are `impact`'s job, not this checker's. + if affected.distance != 1 || !affected.reason.starts_with("calls ") { + continue; + } + if touched.contains(&affected.location) || !cited.insert(affected.location.clone()) { + continue; + } + let path = affected + .location + .rsplit_once(':') + .map_or(affected.location.as_str(), |(path, _)| path) + .to_string(); + findings.push(Finding { + location: affected.location, + path, + what: affected.what, + reason: affected.reason, + }); + } + } + findings.sort_by(|a, b| a.location.cmp(&b.location)); + + let mut result = Findings { + findings, + changed_symbols: origins.iter().map(|n| n.location()).collect(), + answer_tokens: 0, + elapsed_ms: started.elapsed().as_millis(), + }; + result.answer_tokens = tokens::estimate(&result.render()); + Ok(result) +} + #[cfg(test)] mod tests { use super::*; @@ -371,6 +539,111 @@ mod tests { } } + /// Three symbols: `caller` and `sibling` both call `target`, all in different + /// files, plus one symbol nothing calls. + fn graph() -> Store { + use reify::model::{uid, EdgeKind, NewEdge, NewNode, NodeKind, Status}; + use reify::store::Batch; + + let symbol = |path: &str, name: &str, start: u32, end: u32| { + let mut node = NewNode::new(uid::symbol(path, name), NodeKind::Symbol, name); + node.path = Some(path.to_string()); + node.line_start = start; + node.line_end = end; + node + }; + let mut batch = Batch::default(); + batch.node(symbol("app/pricing.py", "target", 10, 20)); + batch.node(symbol("app/orders.py", "caller", 5, 15)); + batch.node(symbol("app/report.py", "sibling", 30, 40)); + batch.node(symbol("app/lonely.py", "lonely", 1, 4)); + for from in ["app/orders.py#caller", "app/report.py#sibling"] { + let (path, name) = from.split_once('#').unwrap(); + batch.edge(NewEdge::new( + uid::symbol(path, name), + uid::symbol("app/pricing.py", "target"), + EdgeKind::Calls, + Status::Confirmed, + 1.0, + )); + } + let mut store = Store::in_memory().unwrap(); + store.commit(batch).unwrap(); + store + } + + fn patch(files: &[(&str, u32)]) -> tasks::Patch { + tasks::Patch { + files: files + .iter() + .map(|(path, line)| tasks::FilePatch { + path: path.to_string(), + created: false, + hunks: vec![tasks::Hunk { + old_start: *line, + old_len: 1, + changed_lines: vec![*line], + }], + }) + .collect(), + } + } + + #[test] + fn a_caller_the_patch_did_not_touch_is_a_finding() { + let found = missing_callers(&graph(), &patch(&[("app/pricing.py", 12)])).unwrap(); + let cited: Vec<&str> = found.findings.iter().map(|f| f.path.as_str()).collect(); + assert_eq!(cited, vec!["app/orders.py", "app/report.py"]); + assert_eq!(found.changed_symbols, vec!["app/pricing.py:10"]); + } + + #[test] + fn a_caller_the_patch_did_touch_is_not_a_finding() { + // This is the whole subtrahend: a symbol the patch already edits is not + // something the patch forgot. Without it every complete commit would be + // reported as incomplete. + let found = missing_callers( + &graph(), + &patch(&[("app/pricing.py", 12), ("app/orders.py", 7)]), + ) + .unwrap(); + let cited: Vec<&str> = found.findings.iter().map(|f| f.path.as_str()).collect(); + assert_eq!(cited, vec!["app/report.py"]); + } + + #[test] + fn a_complete_change_leaves_nothing_to_report() { + let found = missing_callers( + &graph(), + &patch(&[ + ("app/pricing.py", 12), + ("app/orders.py", 7), + ("app/report.py", 33), + ]), + ) + .unwrap(); + assert!(found.findings.is_empty(), "{:?}", found.findings); + assert!(found.render().contains("nothing")); + } + + #[test] + fn a_diff_that_resolves_to_no_symbol_reports_nothing_rather_than_guessing() { + let found = missing_callers(&graph(), &patch(&[("app/pricing.py", 900)])).unwrap(); + assert!(found.changed_symbols.is_empty()); + assert!(found.findings.is_empty()); + } + + #[test] + fn only_a_file_that_calls_out_of_itself_can_ever_be_cited() { + // The ceiling on the held-out-hunk construction, and the direction is easy to + // get backwards: `pricing.py` is called *by* two files and calls nothing, so no + // caller-based checker can cite it. + let store = graph(); + assert!(can_be_cited(&store, "app/orders.py").unwrap()); + assert!(!can_be_cited(&store, "app/pricing.py").unwrap()); + assert!(!can_be_cited(&store, "app/lonely.py").unwrap()); + } + #[test] fn content_search_prefers_files_matching_more_distinct_terms() { // pricing.py contains all three task terms; huge.py contains two of them many diff --git a/crates/reify-bench/src/main.rs b/crates/reify-bench/src/main.rs index 626ded3..929f79f 100644 --- a/crates/reify-bench/src/main.rs +++ b/crates/reify-bench/src/main.rs @@ -141,6 +141,40 @@ enum Command { #[arg(long)] out: PathBuf, }, + /// Held-out-hunk evaluation: does the graph notice an incomplete patch? + /// + /// Model-free, deterministic, and free to run. It exists to decide whether + /// `reify verify` is worth building, against the pre-registered condition in + /// `metrics::VERIFY_RECALL_FLOOR`. + VerifyEval { + #[arg(long)] + repo: PathBuf, + #[arg(long)] + out: PathBuf, + #[arg(long, default_value_t = 20)] + count: usize, + #[arg(long, default_value_t = 4_000)] + scan: usize, + /// Only take trials from commits after this revision. + #[arg(long)] + after: Option, + /// Only take trials from commits strictly older than this revision. + #[arg(long)] + until: Option, + /// Where parent trees are extracted. Defaults to a temporary directory, which + /// is removed when the run finishes. + #[arg(long)] + work: Option, + }, + /// Render the held-out-hunk report from one or more `verify-eval` result + /// directories. + VerifyReport { + /// Result directories, as `Label=path`, in the order they should appear. + #[arg(long = "results", value_name = "LABEL=DIR", num_args = 1..)] + results: Vec, + #[arg(long)] + out: PathBuf, + }, /// Render a report from raw results. Report { #[arg(long = "in")] @@ -223,6 +257,24 @@ fn run() -> Result<()> { } => audit(&repo, &tasks, budget), Command::Fit { train, out, budget } => fit(&train, &out, budget), Command::Chart { results, out } => charts(&results, &out), + Command::VerifyEval { + repo, + out, + count, + scan, + after, + until, + work, + } => verify_eval( + &repo, + &out, + count, + scan, + after.as_deref(), + until.as_deref(), + work.as_deref(), + ), + Command::VerifyReport { results, out } => verify_report(&results, &out), Command::Report { input, out } => report(&input, &out), } } @@ -555,8 +607,10 @@ fn agent_experiments( ) -> Result<()> { let wanted = |name: &str| arms.is_empty() || arms.iter().any(|a| a == name); let set: tasks::TaskSet = read_json(task_file)?; + let name = set.repository_name().to_string(); + let name = name.as_str(); let provider = agent::provider_or_explain(repo)?; - eprintln!("provider: {}", provider.label); + eprintln!("provider: {} repository: {name}", provider.label); let store_path = repo .join(reify::index::REIFY_DIR) @@ -573,7 +627,7 @@ fn agent_experiments( // E6: memorisation control. No context at all. if wanted("N-no-context") { - outcomes.push(agent::run(&provider, repo, task, "N-no-context", "")); + outcomes.push(agent::run(&provider, repo, name, task, "N-no-context", "")); } // E1: the budget-matched lexical baseline. @@ -582,6 +636,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "B-content-grep", &agent::files_block(&grep), @@ -594,6 +649,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "R-reify", &agent::files_block(&compiled), @@ -612,6 +668,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "R-shuffled", &agent::files_block(&shuffled), @@ -623,6 +680,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "O-oracle", &agent::oracle_block(task), @@ -637,6 +695,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "R-reify-iter3", &agent::files_block(&iterated), @@ -647,6 +706,7 @@ fn agent_experiments( outcomes.push(agent::run( &provider, repo, + name, task, "B-content-grep-x3", &agent::files_block(&grep_wide), @@ -679,6 +739,7 @@ fn agent_experiments( &out.join("agent-environment.json"), &serde_json::json!({ "provider": provider.label, + "repository": name, "tasks": chosen.len(), "budget_tokens": budget, "conditions": names, @@ -818,6 +879,607 @@ fn execute( Ok(()) } +/// Held-out-hunk evaluation: can the graph tell that a patch is incomplete? +/// +/// Model-free and deterministic. For each qualifying merged commit the parent tree is +/// extracted and indexed, the change is fed to the checker twice — once with one file's +/// only hunk withheld, once complete — and the two runs answer two different questions: +/// does a finding cite the withheld hunk, and how many findings does a change that is +/// complete by construction still attract. +fn verify_eval( + repo: &Path, + out: &Path, + count: usize, + scan: usize, + after: Option<&str>, + until: Option<&str>, + work: Option<&Path>, +) -> Result<()> { + let started = std::time::Instant::now(); + let set = tasks::generate_truncated( + repo, + count, + scan, + after, + until, + &std::collections::BTreeSet::new(), + )?; + anyhow::ensure!( + !set.tasks.is_empty(), + "no commit in the scanned history could be truncated; {} candidates were \ + rejected, the commonest reason being `{}`", + set.rejected.len(), + set.rejected + .first() + .map(|(_, why)| why.as_str()) + .unwrap_or("none recorded"), + ); + eprintln!( + "{} trials from {} commits ({} passed every retrieval filter but could not be \ + truncated)", + set.tasks.len(), + set.generated_from_commits, + set.rejected.len() + ); + + let scratch = work.map(Path::to_path_buf).unwrap_or_else(|| { + std::env::temp_dir().join(format!("reify-verify-eval-{}", std::process::id())) + }); + let tree = scratch.join("tree"); + + let mut outcomes: Vec = Vec::new(); + // Taken from the first trial's index and kept: what the repository is written in + // is a property of the repository, not of the label someone passes to the report. + let mut languages: Vec<(String, usize)> = Vec::new(); + for (i, task) in set.tasks.iter().enumerate() { + eprint!("\r trial {}/{} ", i + 1, set.tasks.len()); + let indexing = std::time::Instant::now(); + extract_tree(repo, &task.parent, &tree)?; + let mut store = Store::open( + tree.join(reify::index::REIFY_DIR) + .join(reify::index::STORE_FILE), + )?; + reify::index::index(&mut store, &reify::index::IndexOptions::new(&tree))?; + let index_ms = indexing.elapsed().as_millis(); + if languages.is_empty() { + let mut rows = store.coverage_by_language()?; + rows.sort_by_key(|row| std::cmp::Reverse(row.1)); + languages = rows.into_iter().take(3).map(|(l, n, _)| (l, n)).collect(); + } + + // Resolved at the parent, where the withheld change has not happened yet — the + // same state the checker sees, so a symbol that does not exist there is + // honestly unscorable rather than quietly credited. + let omission_symbol = store + .symbol_at(&task.omission_file, task.omission_line)? + .map(|node| node.location()); + let truncated = conditions::missing_callers(&store, &task.truncated)?; + let complete = conditions::missing_callers(&store, &task.complete)?; + outcomes.push(metrics::score_verify( + task, + omission_symbol, + conditions::can_be_cited(&store, &task.omission_file)?, + &truncated, + &complete, + index_ms, + )); + } + eprintln!(); + let _ = std::fs::remove_dir_all(&scratch); + + let summary = metrics::summarise_verify(&outcomes); + std::fs::create_dir_all(out)?; + write_json(&out.join("verify-outcomes.json"), &outcomes)?; + write_json(&out.join("verify-summary.json"), &summary)?; + write_json(&out.join("verify-tasks.json"), &set)?; + write_json( + &out.join("verify-environment.json"), + &serde_json::json!({ + "reify_version": env!("CARGO_PKG_VERSION"), + "repository": set.repository, + // The local path is wherever the run happened, which is no help to anyone + // reproducing it. The remote is. + "origin": origin(repo), + "head": set.head, + "languages": languages, + // The selection window, so a committed result can be re-run exactly even + // after the branch it was taken from has moved on. + "count": count, + "scan": scan, + "after": after, + "until": until, + "trials": set.tasks.len(), + "candidates_rejected": set.rejected.len(), + "checker": "symbols changed by this diff, minus symbols present in the diff, \ + where an inbound CALLS edge exists at distance 1, via reify::query::impact", + "wall_clock_ms": started.elapsed().as_millis(), + "token_counts": "estimated by reify heuristic-v1", + }), + )?; + + eprintln!("\n{}", render_verify(&summary)); + eprintln!( + "wrote {} ({:.1}s wall clock)", + out.display(), + started.elapsed().as_secs_f32() + ); + Ok(()) +} + +/// The summary as a human reads it, verdict first. +fn render_verify(s: &metrics::VerifySummary) -> String { + let mut text = String::new(); + text.push_str(&format!( + "{:<28} {:.2} (95% CI {:.2}–{:.2}, {}/{} trials)\n", + "omission_recall", + s.omission_recall, + s.omission_recall_ci.0, + s.omission_recall_ci.1, + (s.omission_recall * s.tasks as f32).round() as usize, + s.tasks, + )); + text.push_str(&format!( + "{:<28} {:.2} (95% CI {:.2}–{:.2}) — citations the complete commit does not \ + also produce\n", + " of which attributable", + s.omission_recall_attributable, + s.omission_recall_attributable_ci.0, + s.omission_recall_attributable_ci.1, + )); + match (s.omission_recall_reachable, s.omission_recall_reachable_ci) { + (Some(recall), Some(ci)) => text.push_str(&format!( + "{:<28} {recall:.2} (95% CI {:.2}–{:.2}, {}/{} omitted files call into \ + another file at all — the ceiling on any call-graph checker)\n", + " where citable at all", ci.0, ci.1, s.reachable_omissions, s.tasks + )), + _ => text.push_str(&format!( + "{:<28} — (no omitted file calls into another file; the ceiling on \ + any call-graph checker here is zero)\n", + " where citable at all" + )), + } + match (s.omission_recall_symbol, s.omission_recall_symbol_ci) { + (Some(recall), Some(ci)) => text.push_str(&format!( + "{:<28} {recall:.2} (95% CI {:.2}–{:.2}, {} scorable)\n", + "omission_recall_symbol", ci.0, ci.1, s.symbol_scorable + )), + _ => text.push_str(&format!( + "{:<28} — (no trial's omission fell inside an indexed symbol)\n", + "omission_recall_symbol" + )), + } + text.push_str(&format!( + "{:<28} {:.2} per complete commit ({}/{} commits noisy, 95% CI {:.2}–{:.2})\n", + "false_alarm_rate", + s.false_alarm_rate, + s.commits_with_a_false_alarm, + s.tasks, + s.false_alarm_share_ci.0, + s.false_alarm_share_ci.1, + )); + text.push_str(&format!( + "{:<28} {}\n", + "findings_per_diff (median)", s.median_findings_per_diff + )); + text.push_str(&format!( + "{:<28} {}\n", + "verify_tokens (median)", s.median_verify_tokens + )); + text.push_str(&format!( + "{:<28} {}\n", + "verify_latency_ms (median)", s.median_verify_latency_ms + )); + text.push_str(&format!( + "{:<28} {} (extract + index one parent tree; not part of the query)\n", + "index_ms (median)", s.median_index_ms + )); + if s.diffs_resolving_to_nothing > 0 { + text.push_str(&format!( + "{:<28} {} of {} truncated diffs resolved to no indexed symbol at all\n", + "unresolved", s.diffs_resolving_to_nothing, s.tasks + )); + } + text.push_str(&format!( + "\npre-registered verdict: {}\n {}\n", + match s.verdict() { + metrics::Verdict::Build => "BUILD `reify verify` on this substrate", + metrics::Verdict::DoNotBuild => "DO NOT BUILD `reify verify` on this substrate", + }, + s.why() + )); + text +} + +/// The repository's `origin` remote, so a committed result names something a reader +/// can clone rather than the temporary directory it happened to be run in. +fn origin(repo: &Path) -> Option { + let output = std::process::Command::new("git") + .args(["remote", "get-url", "origin"]) + .current_dir(repo) + .output() + .ok()?; + output + .status + .success() + .then(|| String::from_utf8_lossy(&output.stdout).trim().to_string()) +} + +/// Extract a commit's tree into `into`, replacing whatever was there. +/// +/// `git archive` rather than a worktree: a worktree registers itself in the shared +/// git directory, and this harness must not leave anything behind in the repository +/// it is measuring. The cost is a full index per trial, which is reported. +fn extract_tree(repo: &Path, sha: &str, into: &Path) -> Result<()> { + use std::process::{Command, Stdio}; + if into.exists() { + std::fs::remove_dir_all(into).with_context(|| format!("clearing {}", into.display()))?; + } + std::fs::create_dir_all(into)?; + let mut archive = Command::new("git") + .args(["archive", "--format=tar", sha]) + .current_dir(repo) + .stdout(Stdio::piped()) + .spawn() + .context("running git archive")?; + let stdout = archive + .stdout + .take() + .ok_or_else(|| anyhow::anyhow!("git archive produced no output"))?; + let extracted = Command::new("tar") + .arg("-x") + .arg("-C") + .arg(into) + .stdin(Stdio::from(stdout)) + .status() + .context("running tar to extract a parent tree")?; + let archived = archive.wait()?; + anyhow::ensure!( + archived.success() && extracted.success(), + "cannot extract the tree at {sha}" + ); + Ok(()) +} + +/// Render the held-out-hunk report across every repository that was run. +/// +/// Generated from `verify-summary.json` for the same reason the retrieval report is: +/// a table that can drift from its data is a picture, not a measurement. The per-trial +/// appendix is included so the selection rule's effects are visible rather than +/// described — every omitted file is named. +fn verify_report(results: &[String], out: &Path) -> Result<()> { + struct Run { + label: String, + summary: metrics::VerifySummary, + environment: serde_json::Value, + outcomes: Vec, + } + + let mut runs = Vec::new(); + for spec in results { + let (label, dir) = spec + .split_once('=') + .ok_or_else(|| anyhow::anyhow!("expected LABEL=DIR, got `{spec}`"))?; + let dir = Path::new(dir); + runs.push(Run { + label: label.to_string(), + summary: read_json(&dir.join("verify-summary.json"))?, + environment: read_json(&dir.join("verify-environment.json"))?, + outcomes: read_json(&dir.join("verify-outcomes.json"))?, + }); + } + anyhow::ensure!(!runs.is_empty(), "no results given"); + + let mut md = String::from("# Can the graph tell that a patch is incomplete?\n\n"); + md.push_str( + "Generated by `reify-bench verify-report`. Every number is computed from the \ + `verify-summary.json` files named below; nothing is entered by hand.\n\n\ + This benchmark exists to decide one thing: whether `reify verify` — a \ + post-flight check that reads an agent's diff and reports what the patch \ + missed — is worth building on Reify's call graph. It is model-free, \ + deterministic, and costs nothing per run.\n\n", + ); + + md.push_str("## Construction\n\n"); + md.push_str( + "For each merged commit that passes the retrieval benchmark's filters and \ + touches at least two indexable files:\n\n\ + 1. the parent tree is extracted and indexed, so the change is absent from the \ + index by construction;\n\ + 2. one file's **only** hunk is withheld — the *omission*. Removing it removes \ + that file from the patch entirely, so a citation of it cannot be an echo of \ + a hunk still present. Among the files with exactly one hunk, the last by \ + path order is chosen; the choice is arbitrary, fixed, and made before any \ + checker runs;\n\ + 3. the truncated patch goes to the checker;\n\ + 4. **the same commit goes to the checker complete.** A merged commit is \ + complete by definition, so every finding there is a false positive. This \ + control is not optional: without it the metric would reward a checker that \ + simply shouts.\n\n\ + The checker is not `reify verify`, which does not exist. It is the shipped \ + graph query — *symbols changed by this diff, minus symbols present in the \ + diff, where an inbound `CALLS` edge exists at distance 1* — reached through \ + `reify::query::impact`. That deliberately measures the **substrate**, which is \ + the number the decision needs.\n\n", + ); + + md.push_str("## Pre-registered falsification condition\n\n"); + md.push_str(&format!( + "> If `omission_recall` on this substrate is below **{VERIFY_RECALL_FLOOR:.2}**, \ + or `false_alarm_rate` is above **{VERIFY_FALSE_ALARM_CEILING:.1} per commit**, \ + the `reify verify` feature does not get built on this substrate.\n\n\ + Stated in `crates/reify-bench/src/metrics.rs` before the first run and not \ + moved since. A result that kills the feature is a result.\n\n", + VERIFY_RECALL_FLOOR = metrics::VERIFY_RECALL_FLOOR, + VERIFY_FALSE_ALARM_CEILING = metrics::VERIFY_FALSE_ALARM_CEILING, + )); + + md.push_str("## Results\n\n| Metric |"); + for run in &runs { + md.push_str(&format!(" {} |", run.label)); + } + md.push_str("\n|---|"); + for _ in &runs { + md.push_str("---:|"); + } + md.push('\n'); + let row = |label: &str, f: &dyn Fn(&Run) -> String| { + let mut line = format!("| {label} |"); + for run in &runs { + line.push_str(&format!(" {} |", f(run))); + } + line.push('\n'); + line + }; + md.push_str(&row("Most indexed language", &|r| { + r.environment["languages"][0][0] + .as_str() + .unwrap_or("—") + .to_string() + })); + md.push_str(&row("Trials", &|r| r.summary.tasks.to_string())); + md.push_str(&row("`omission_recall`", &|r| { + format!( + "**{:.2}** ({:.2}–{:.2})", + r.summary.omission_recall, + r.summary.omission_recall_ci.0, + r.summary.omission_recall_ci.1 + ) + })); + md.push_str(&row("…attributable to the omission", &|r| { + format!( + "{:.2} ({:.2}–{:.2})", + r.summary.omission_recall_attributable, + r.summary.omission_recall_attributable_ci.0, + r.summary.omission_recall_attributable_ci.1 + ) + })); + md.push_str(&row("`omission_recall_symbol`", &|r| match ( + r.summary.omission_recall_symbol, + r.summary.omission_recall_symbol_ci, + ) { + (Some(v), Some(ci)) => format!( + "{v:.2} ({:.2}–{:.2}) over {}", + ci.0, ci.1, r.summary.symbol_scorable + ), + _ => "— (0 scorable)".into(), + })); + md.push_str(&row("Omitted files a caller query *could* cite", &|r| { + format!("{}/{}", r.summary.reachable_omissions, r.summary.tasks) + })); + md.push_str(&row("`false_alarm_rate` (per complete commit)", &|r| { + format!("**{:.1}**", r.summary.false_alarm_rate) + })); + md.push_str(&row("Complete commits with ≥1 false alarm", &|r| { + format!( + "{}/{} ({:.2}–{:.2})", + r.summary.commits_with_a_false_alarm, + r.summary.tasks, + r.summary.false_alarm_share_ci.0, + r.summary.false_alarm_share_ci.1 + ) + })); + md.push_str(&row("`findings_per_diff` (median)", &|r| { + r.summary.median_findings_per_diff.to_string() + })); + md.push_str(&row("`verify_tokens` (median)", &|r| { + r.summary.median_verify_tokens.to_string() + })); + md.push_str(&row("`verify_latency_ms` (median)", &|r| { + r.summary.median_verify_latency_ms.to_string() + })); + md.push_str(&row("Index per trial, ms (median)", &|r| { + r.summary.median_index_ms.to_string() + })); + md.push_str(&row("Whole run, wall clock", &|r| { + format!( + "{:.0}s", + r.environment["wall_clock_ms"].as_f64().unwrap_or(0.0) / 1000.0 + ) + })); + md.push_str(&row( + "Pre-registered verdict", + &|r| match r.summary.verdict() { + metrics::Verdict::Build => "build".into(), + metrics::Verdict::DoNotBuild => "**do not build**".into(), + }, + )); + + md.push_str("\n## What the numbers say\n\n"); + for run in &runs { + md.push_str(&format!("**{}** — {}\n\n", run.label, run.summary.why())); + } + let all_fail = runs + .iter() + .all(|r| r.summary.verdict() == metrics::Verdict::DoNotBuild); + md.push_str(if all_fail { + "Every repository fails the pre-registered condition, so **`reify verify` does \ + not get built on this substrate**. The condition was written down before the \ + first run precisely so this outcome could not be argued away afterwards.\n\n" + } else { + "At least one repository clears the pre-registered condition. Read the \ + confidence intervals before treating that as settled.\n\n" + }); + + // Which half of the condition actually fails, counted rather than asserted: the + // interesting question is not "did it fail" but "on what". + let failed_recall = runs + .iter() + .filter(|r| r.summary.omission_recall < metrics::VERIFY_RECALL_FLOOR) + .count(); + let failed_noise = runs + .iter() + .filter(|r| r.summary.false_alarm_rate > metrics::VERIFY_FALSE_ALARM_CEILING) + .count(); + md.push_str(&format!( + "**It fails on noise, not on blindness.** {failed_noise} of {} repositories \ + exceed the false-alarm ceiling; {failed_recall} of {} fall below the recall \ + floor (a repository can fail both). The graph does find the omitted file often enough to be interesting; \ + what it cannot do is stay quiet about a patch that is already complete.\n\n", + runs.len(), + runs.len(), + )); + + md.push_str( + "**The negative control takes most of the headline back.** `omission_recall` \ + counts a citation of the omitted file whether or not the complete commit is \ + cited too. The attributable row counts only citations the complete commit does \ + *not* produce, and it is the smaller number in every repository here. The gap \ + is the checker citing a file it would have cited anyway — which is not \ + detection, however it reads next to the label.\n\n", + ); + + // The ceiling either binds or it does not, and which one decides whether a better + // query could help. Asserting the wrong one would be worse than saying nothing. + let tightest = runs + .iter() + .map(|r| r.summary.reachable_omissions as f32 / r.summary.tasks.max(1) as f32) + .fold(f32::INFINITY, f32::min); + md.push_str(&format!( + "**The ceiling is not what binds.** A finding is a caller, so the omitted file \ + can only be cited if something in it calls out of itself. In the least \ + favourable repository here that holds for {:.0}% of omissions, so the edges \ + mostly exist and `omission_recall` is not capped by their absence. The gap \ + between that row and the recall row is a *ranking* gap, not a coverage one.\n\n", + tightest * 100.0 + )); + + md.push_str( + "**The noise is structural, not marginal.** `false_alarm_rate` is findings per \ + commit that is complete by construction. A `CALLS` edge says a caller exists; \ + it does not say the caller needed changing. Nothing in the graph distinguishes \ + a changed signature from an edit inside a body, so every caller of every \ + touched symbol is a candidate. That is a property of the edge, and no \ + rewriting of the query around the same edge removes it.\n\n", + ); + + md.push_str("## Cost and determinism\n\n"); + md.push_str(&format!( + "No model, no network, no provider key: the whole run is a git extract, an \ + index and a graph query. Total wall clock for everything in this report is \ + **{:.0}s**, dominated by re-indexing one parent tree per trial. The query \ + itself is the `verify_latency_ms` row — single-digit milliseconds.\n\n\ + Each run is deterministic given a fixed `HEAD`: task selection, the omission \ + rule and the query contain no randomness and no tunable threshold. A run \ + against a repository whose history is still moving — this one, for instance — \ + should pin the window with `--until `, or the trial set moves with the \ + branch.\n\n\ + ```bash\n\ + reify-bench verify-eval --repo --out results/verify- --until \n\ + reify-bench verify-report --results \"name=results/verify-\" --out benchmarks/REPORT-verify.md\n\ + ```\n\n", + runs + .iter() + .map(|r| r.environment["wall_clock_ms"].as_f64().unwrap_or(0.0)) + .sum::() + / 1000.0 + )); + + md.push_str("## Limitations\n\n"); + md.push_str( + "1. **Small samples.** The intervals are wide and are printed beside every \ + rate. Where two repositories differ by less than their intervals, they have \ + not been shown to differ.\n\ + 2. **The omission-selection rule has a direction.** \"Last by path order, among \ + files with exactly one hunk\" is arbitrary but not neutral: in a repository \ + laid out as `src/` and `tests/`, path order lands on `tests/`. Counted \ + across every run here, TEST_SHARE omissions sit under a path segment \ + named `test` or `tests`. The rule was fixed before any run and has not been \ + changed since; every omitted file is named in the appendix, so the effect is \ + checkable rather than described.\n\ + 3. **`CALLS` at distance 1 only.** `impact` also propagates two hops and crosses \ + into the data layer. Widening the query would raise recall and raise the \ + false-alarm rate with it — the trade this benchmark measures rather than \ + pre-empts.\n\ + 4. **A checker, not the feature.** `reify verify` could use a signature diff, \ + type information, or the model. This measures the substrate those would all \ + stand on.\n\ + 5. **Parent trees are extracted with `git archive`**, so the indexed tree has no \ + git history and no co-change edges. The checker uses neither; a checker that \ + did would need re-measuring.\n\ + 6. **Ground truth is one commit's hunks.** A change that could correctly have \ + been made elsewhere scores as a miss.\n\ + 7. **`impact`'s own bounds are inherited, not bypassed.** It stops at 60 \ + affected nodes and walks depth-first to two hops, so on a widely-called \ + symbol some direct callers can be crowded out by second-hop ones. That is \ + the shipped query's behaviour and measuring around it would measure \ + something that does not exist.\n\n", + ); + + let (mut in_tests, mut trials) = (0usize, 0usize); + for run in &runs { + for outcome in &run.outcomes { + trials += 1; + if outcome + .omission_file + .split('/') + .any(|part| part == "test" || part == "tests") + { + in_tests += 1; + } + } + } + let md = md.replace("TEST_SHARE", &format!("{in_tests} of {trials}")); + + let mut md = md; + md.push_str("## Appendix: every trial\n\n"); + md.push_str( + "`could cite` is whether the omitted file calls out of itself at all — the \ + ceiling for that trial. `cited` is findings on the truncated patch, `noise` is \ + findings on the same commit complete.\n\n", + ); + for run in &runs { + md.push_str(&format!( + "### {} (`{}`, commit `{}`)\n\n", + run.label, + run.environment["origin"] + .as_str() + .or_else(|| run.environment["repository"].as_str()) + .unwrap_or("?"), + run.environment["head"].as_str().unwrap_or("?"), + )); + md.push_str("| Trial | Omitted file | could cite | hit | attributable | cited | noise |\n"); + md.push_str("|---|---|---|---|---|---:|---:|\n"); + let tick = |yes: bool| if yes { "yes" } else { "no" }; + for outcome in &run.outcomes { + md.push_str(&format!( + "| `{}` | `{}` | {} | {} | {} | {} | {} |\n", + outcome.task, + outcome.omission_file, + tick(outcome.omission_file_reachable), + tick(outcome.file_hit), + tick(outcome.file_hit_attributable), + outcome.findings, + outcome.false_alarms, + )); + } + md.push('\n'); + } + + std::fs::write(out, md).with_context(|| format!("writing {}", out.display()))?; + eprintln!("wrote {}", out.display()); + Ok(()) +} + fn report(input: &Path, out: &Path) -> Result<()> { let outcomes: Vec = read_json(&input.join("outcomes.json"))?; let set: tasks::TaskSet = read_json(&input.join("tasks.json"))?; diff --git a/crates/reify-bench/src/metrics.rs b/crates/reify-bench/src/metrics.rs index b97db0e..885d584 100644 --- a/crates/reify-bench/src/metrics.rs +++ b/crates/reify-bench/src/metrics.rs @@ -6,7 +6,8 @@ use serde::{Deserialize, Serialize}; use std::collections::BTreeSet; -use crate::conditions::Answer; +use crate::agent::wilson_interval; +use crate::conditions::{Answer, Finding, Findings}; /// How one condition performed on one task. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -200,6 +201,260 @@ pub fn expected_tokens(condition: &str, outcomes: &[Outcome], budget: u32) -> f3 total / mine.len() as f32 } +// ---- the held-out-hunk metrics ---------------------------------------------- + +/// Pre-registered falsification condition for `reify verify`, stated before the first +/// run of this harness and not moved since. +/// +/// > If `omission_recall` on this substrate is below **0.25**, or `false_alarm_rate` +/// > is above **0.1 per commit**, the `reify verify` feature does not get built on +/// > this substrate. +/// +/// Both halves matter. Recall alone would be cleared by a checker that reports every +/// caller of everything; the false-alarm ceiling is what stops that, and it is +/// measured against complete merged commits, where a finding cannot be anything but +/// wrong. A result that fails either half is a result, not a failure of the harness: +/// the response is to publish it and not build the feature, never to widen the query +/// until it passes. +pub const VERIFY_RECALL_FLOOR: f32 = 0.25; +/// Findings per complete merged commit, above which the checker is too noisy to ship. +pub const VERIFY_FALSE_ALARM_CEILING: f32 = 0.1; + +/// One held-out-hunk trial. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VerifyOutcome { + pub task: String, + pub commit: String, + /// The file whose only hunk was withheld. + pub omission_file: String, + /// `path:line` of the symbol the withheld hunk falls inside, when it falls inside + /// one. `None` means the change was outside every indexed symbol — an import, a + /// constant, a top-level statement — and the trial is not scorable at symbol + /// granularity. It is excluded there rather than counted as a miss. + pub omission_symbol: Option, + /// Some symbol in the omission's file calls a symbol in another file, at the + /// parent commit. False means no caller-based checker could ever cite this file, + /// whatever query it runs. + pub omission_file_reachable: bool, + /// A finding cites the omission's file. + pub file_hit: bool, + /// A finding cites the omission's file on the truncated diff and **not** on the + /// complete one. A citation the negative control also produces was not caused by + /// the omission, whatever it looks like next to it. + pub file_hit_attributable: bool, + /// A finding cites the omission's symbol. `None` when not scorable. + pub symbol_hit: Option, + pub findings: usize, + /// Findings against the **complete** commit. Complete by construction, so every + /// one of these is a false positive. + pub false_alarms: usize, + /// Symbols the truncated diff resolved to. Zero means the checker had nothing to + /// work from, which is a different failure from having something and missing. + pub changed_symbols: usize, + pub verify_tokens: u32, + pub verify_latency_ms: u128, + /// Wall clock to extract and index the parent tree. The real feature would run + /// against an index that already exists, so this is the harness's cost, not the + /// checker's, and is kept out of `verify_latency_ms`. + pub index_ms: u128, + /// Every location the truncated run cited, and every location the complete run + /// cited. Written out because a rate nobody can check is a claim, not a + /// measurement: these are what `false_alarms` counts, one line each. + pub cited: Vec, + pub cited_on_complete: Vec, +} + +/// Aggregate figures over a set of held-out-hunk trials. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct VerifySummary { + pub tasks: usize, + /// Share of truncated diffs where some finding cites the omitted hunk's file. + pub omission_recall: f32, + pub omission_recall_ci: (f32, f32), + /// The same share, counting only citations the complete commit does **not** also + /// produce. `omission_recall` is the metric as specified; this is the one that + /// says whether the checker responded to the omission or to the file's standing + /// noise. Where the two differ, the gap is the part of the headline that the + /// negative control already explains. + pub omission_recall_attributable: f32, + pub omission_recall_attributable_ci: (f32, f32), + /// Trials whose omitted file calls out of itself at all. This is the ceiling: + /// `omission_recall` cannot exceed `reachable_omissions / tasks` however the query + /// is written, so the two numbers separate a query problem from a substrate + /// problem. + pub reachable_omissions: usize, + /// `omission_recall` restricted to those trials — what the checker managed where + /// there was something to find. + pub omission_recall_reachable: Option, + pub omission_recall_reachable_ci: Option<(f32, f32)>, + /// Trials where the omission falls inside an indexed symbol — the denominator of + /// the symbol-granular figure. + pub symbol_scorable: usize, + /// The same share at symbol granularity, over `symbol_scorable` trials. + pub omission_recall_symbol: Option, + pub omission_recall_symbol_ci: Option<(f32, f32)>, + /// Findings per complete merged commit. A rate over counts, not a proportion, so + /// it carries no Wilson interval; the proportion beside it does. + pub false_alarm_rate: f32, + /// Complete commits producing at least one finding. + pub commits_with_a_false_alarm: usize, + pub false_alarm_share_ci: (f32, f32), + /// Median findings per truncated diff. A checker emitting thirty findings is + /// unusable at any precision, which a mean would hide behind the quiet cases. + pub median_findings_per_diff: usize, + /// Median tokens the findings output would cost the agent that reads it. + pub median_verify_tokens: u32, + /// Median wall clock of the query alone. + pub median_verify_latency_ms: u128, + /// Median wall clock to extract and index one parent tree, reported so the cost of + /// running this in CI is a measured number rather than a promise. + pub median_index_ms: u128, + /// Trials where the truncated diff resolved to no symbol at all. The checker + /// cannot report anything for these; they stay in the denominator, because a + /// substrate that cannot resolve a diff has failed the task. + pub diffs_resolving_to_nothing: usize, +} + +/// Does this substrate clear the pre-registered bar? +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +pub enum Verdict { + Build, + DoNotBuild, +} + +impl VerifySummary { + pub fn verdict(&self) -> Verdict { + if self.omission_recall < VERIFY_RECALL_FLOOR + || self.false_alarm_rate > VERIFY_FALSE_ALARM_CEILING + { + Verdict::DoNotBuild + } else { + Verdict::Build + } + } + + /// Why the verdict came out the way it did, in one line. + pub fn why(&self) -> String { + let mut failed = Vec::new(); + if self.omission_recall < VERIFY_RECALL_FLOOR { + failed.push(format!( + "omission_recall {:.2} < {VERIFY_RECALL_FLOOR:.2}", + self.omission_recall + )); + } + if self.false_alarm_rate > VERIFY_FALSE_ALARM_CEILING { + failed.push(format!( + "false_alarm_rate {:.2} > {VERIFY_FALSE_ALARM_CEILING:.2}", + self.false_alarm_rate + )); + } + if failed.is_empty() { + format!( + "omission_recall {:.2} >= {VERIFY_RECALL_FLOOR:.2} and false_alarm_rate \ + {:.2} <= {VERIFY_FALSE_ALARM_CEILING:.2}", + self.omission_recall, self.false_alarm_rate + ) + } else { + failed.join("; ") + } + } +} + +/// Score one held-out-hunk trial. +pub fn score_verify( + task: &crate::tasks::TruncatedTask, + omission_symbol: Option, + omission_file_reachable: bool, + truncated: &Findings, + complete: &Findings, + index_ms: u128, +) -> VerifyOutcome { + let cites = |predicate: &dyn Fn(&Finding) -> bool| truncated.findings.iter().any(predicate); + VerifyOutcome { + task: task.id.clone(), + commit: task.commit.clone(), + omission_file: task.omission_file.clone(), + omission_file_reachable, + file_hit: cites(&|f| f.path == task.omission_file), + file_hit_attributable: cites(&|f| f.path == task.omission_file) + && !complete + .findings + .iter() + .any(|f| f.path == task.omission_file), + symbol_hit: omission_symbol + .as_ref() + .map(|symbol| cites(&|f| &f.location == symbol)), + omission_symbol, + findings: truncated.findings.len(), + false_alarms: complete.findings.len(), + changed_symbols: truncated.changed_symbols.len(), + verify_tokens: truncated.answer_tokens, + verify_latency_ms: truncated.elapsed_ms, + index_ms, + cited: truncated + .findings + .iter() + .map(|f| f.location.clone()) + .collect(), + cited_on_complete: complete + .findings + .iter() + .map(|f| f.location.clone()) + .collect(), + } +} + +pub fn summarise_verify(outcomes: &[VerifyOutcome]) -> VerifySummary { + let n = outcomes.len(); + let denominator = n.max(1) as f32; + let file_hits = outcomes.iter().filter(|o| o.file_hit).count(); + let attributable = outcomes.iter().filter(|o| o.file_hit_attributable).count(); + let reachable: Vec<&VerifyOutcome> = outcomes + .iter() + .filter(|o| o.omission_file_reachable) + .collect(); + let reachable_hits = reachable.iter().filter(|o| o.file_hit).count(); + let scorable: Vec<&VerifyOutcome> = outcomes + .iter() + .filter(|o| o.omission_symbol.is_some()) + .collect(); + let symbol_hits = scorable + .iter() + .filter(|o| o.symbol_hit == Some(true)) + .count(); + let noisy = outcomes.iter().filter(|o| o.false_alarms > 0).count(); + + VerifySummary { + tasks: n, + omission_recall: file_hits as f32 / denominator, + omission_recall_ci: wilson_interval(file_hits, n), + omission_recall_attributable: attributable as f32 / denominator, + omission_recall_attributable_ci: wilson_interval(attributable, n), + reachable_omissions: reachable.len(), + omission_recall_reachable: (!reachable.is_empty()) + .then(|| reachable_hits as f32 / reachable.len() as f32), + omission_recall_reachable_ci: (!reachable.is_empty()) + .then(|| wilson_interval(reachable_hits, reachable.len())), + symbol_scorable: scorable.len(), + omission_recall_symbol: (!scorable.is_empty()) + .then(|| symbol_hits as f32 / scorable.len() as f32), + omission_recall_symbol_ci: (!scorable.is_empty()) + .then(|| wilson_interval(symbol_hits, scorable.len())), + false_alarm_rate: outcomes.iter().map(|o| o.false_alarms).sum::() as f32 + / denominator, + commits_with_a_false_alarm: noisy, + false_alarm_share_ci: wilson_interval(noisy, n), + median_findings_per_diff: median(outcomes.iter().map(|o| o.findings).collect()) + .unwrap_or(0), + median_verify_tokens: median(outcomes.iter().map(|o| o.verify_tokens).collect()) + .unwrap_or(0), + median_verify_latency_ms: median(outcomes.iter().map(|o| o.verify_latency_ms).collect()) + .unwrap_or(0), + median_index_ms: median(outcomes.iter().map(|o| o.index_ms).collect()).unwrap_or(0), + diffs_resolving_to_nothing: outcomes.iter().filter(|o| o.changed_symbols == 0).count(), + } +} + fn median(mut values: Vec) -> Option { if values.is_empty() { return None; @@ -221,6 +476,141 @@ mod tests { } } + fn trial(omission_file: &str) -> crate::tasks::TruncatedTask { + crate::tasks::TruncatedTask { + id: "v-1".into(), + commit: "a".repeat(40), + parent: "b".repeat(40), + date: "2026-01-01".into(), + prompt: "fix the credit limit check".into(), + complete: crate::tasks::Patch::default(), + truncated: crate::tasks::Patch::default(), + omission_file: omission_file.into(), + omission_line: 10, + } + } + + fn findings(locations: &[&str]) -> Findings { + Findings { + findings: locations + .iter() + .map(|location| Finding { + location: (*location).into(), + path: location + .rsplit_once(':') + .map_or(*location, |(p, _)| p) + .into(), + what: "f".into(), + reason: "calls g".into(), + }) + .collect(), + changed_symbols: vec!["app/x.py:1".into()], + answer_tokens: 40, + elapsed_ms: 1, + } + } + + #[test] + fn a_citation_the_complete_commit_also_produces_is_a_hit_but_not_attributable() { + // The negative control's whole purpose: the checker cited that file whether or + // not anything was withheld, so the omission did not cause the citation. + let outcome = score_verify( + &trial("app/orders.py"), + None, + true, + &findings(&["app/orders.py:5"]), + &findings(&["app/orders.py:5"]), + 100, + ); + assert!(outcome.file_hit); + assert!(!outcome.file_hit_attributable); + assert_eq!(outcome.false_alarms, 1); + } + + #[test] + fn a_citation_only_the_truncated_diff_produces_is_attributable() { + let outcome = score_verify( + &trial("app/orders.py"), + Some("app/orders.py:5".into()), + true, + &findings(&["app/orders.py:5"]), + &findings(&[]), + 100, + ); + assert!(outcome.file_hit_attributable); + assert_eq!(outcome.symbol_hit, Some(true)); + assert_eq!(outcome.false_alarms, 0); + } + + #[test] + fn an_omission_inside_no_symbol_is_unscorable_rather_than_a_miss() { + let outcome = score_verify( + &trial("app/orders.py"), + None, + false, + &findings(&[]), + &findings(&[]), + 100, + ); + assert_eq!( + outcome.symbol_hit, None, + "counting it as a miss would be a lie" + ); + let summary = summarise_verify(&[outcome]); + assert_eq!(summary.symbol_scorable, 0); + assert_eq!(summary.omission_recall_symbol, None); + assert_eq!(summary.reachable_omissions, 0); + assert_eq!(summary.omission_recall_reachable, None); + } + + #[test] + fn the_pre_registered_condition_fails_on_either_half_alone() { + let quiet_but_blind = VerifySummary { + omission_recall: 0.10, + false_alarm_rate: 0.0, + ..summarise_verify(&[]) + }; + assert_eq!(quiet_but_blind.verdict(), Verdict::DoNotBuild); + assert!(quiet_but_blind.why().contains("omission_recall")); + + let sharp_but_noisy = VerifySummary { + omission_recall: 0.90, + false_alarm_rate: 3.0, + ..summarise_verify(&[]) + }; + assert_eq!(sharp_but_noisy.verdict(), Verdict::DoNotBuild); + assert!(sharp_but_noisy.why().contains("false_alarm_rate")); + + let good = VerifySummary { + omission_recall: 0.40, + false_alarm_rate: 0.05, + ..summarise_verify(&[]) + }; + assert_eq!(good.verdict(), Verdict::Build); + } + + #[test] + fn the_false_alarm_rate_counts_findings_not_commits() { + // A checker that shouts thirty times at one commit and stays silent at nine + // others is not a checker with a 10% false-alarm problem. + let outcomes: Vec = (0..10) + .map(|i| { + score_verify( + &trial("app/orders.py"), + None, + true, + &findings(&[]), + &findings(&if i == 0 { vec!["a.py:1"; 30] } else { vec![] }), + 1, + ) + }) + .collect(); + let summary = summarise_verify(&outcomes); + assert_eq!(summary.commits_with_a_false_alarm, 1); + assert!((summary.false_alarm_rate - 3.0).abs() < 1e-6); + assert_eq!(summary.verdict(), Verdict::DoNotBuild); + } + #[test] fn a_perfect_answer_scores_perfectly() { let truth = vec!["a.py".to_string()]; diff --git a/crates/reify-bench/src/tasks.rs b/crates/reify-bench/src/tasks.rs index f27648f..19d5caa 100644 --- a/crates/reify-bench/src/tasks.rs +++ b/crates/reify-bench/src/tasks.rs @@ -45,6 +45,22 @@ pub struct TaskSet { pub tasks: Vec, } +impl TaskSet { + /// The repository's name, for a prompt that has to say what is being worked on. + /// + /// `repository` is a path — `.bench/medusa` — because that is what the generator + /// was pointed at. The final component is the name a developer would use, and a + /// prompt that names the wrong repository is a validity defect rather than a + /// cosmetic one: see the dated note at the top of `benchmarks/REPORT-medusa.md`. + pub fn repository_name(&self) -> &str { + let trimmed = self.repository.trim_end_matches(['/', '\\']); + trimmed + .rsplit(['/', '\\']) + .find(|part| !part.is_empty()) + .unwrap_or(trimmed) + } +} + /// Upper bound on files a task may touch. /// /// A commit touching twenty files is a refactor or a rename; its "ground truth" would @@ -93,67 +109,21 @@ pub fn generate( // walk *starts below it*: only strictly older commits qualify, which is how a // training corpus is kept disjoint from every evaluation window. let base = after.map(|rev| resolve(root, rev)).transpose()?; - let cutoff = base.as_ref().and_then(|sha| { - history - .commits - .iter() - .position(|c| c.sha.starts_with(sha) || sha.starts_with(&c.sha)) - }); - let start = match until.map(|rev| resolve(root, rev)).transpose()? { - None => 0, - // The scanned list is merge-free, so a merge commit named as the boundary is - // legitimately absent from it. When the boundary is HEAD itself, "strictly - // older than HEAD" excludes nothing the list contains. - Some(sha) if sha == head => 0, - Some(sha) => { - match history - .commits - .iter() - .position(|c| c.sha.starts_with(&sha) || sha.starts_with(&c.sha)) - { - Some(position) => position + 1, - // A merge commit is legitimately absent from the merge-free list, so - // the boundary falls back to its timestamp: strictly-older-than holds - // for every commit authored before it. - None => { - let at = commit_time(root, &sha)?; - history - .commits - .iter() - .position(|c| c.timestamp < at) - .with_context(|| { - format!("--until {sha}: nothing older within the scanned history") - })? - } - } - } - }; + let (start, cutoff) = window(root, &history, &head, after, until)?; - let mut tasks = Vec::new(); - for (i, commit) in history.commits.iter().enumerate().skip(start) { - if tasks.len() >= wanted { - break; - } - if cutoff.is_some_and(|stop| i >= stop) { - break; - } - if exclude.contains(&commit.sha) { - continue; - } - let Some(mut task) = candidate(root, commit) else { - continue; - }; + let tasks = take(&history, start, cutoff, exclude, wanted, |commit| { + let mut task = candidate(root, commit)?; // A file created by the change cannot be retrieved from a base that predates // it. Keeping it as ground truth would score every condition zero and measure // nothing, so the task is narrowed to the files that already existed. if let Some(base) = &base { task.ground_truth.retain(|path| exists_at(root, base, path)); if task.ground_truth.is_empty() { - continue; + return None; } } - tasks.push(task); - } + Some(task) + }); Ok(TaskSet { repository: root.display().to_string(), @@ -164,6 +134,88 @@ pub fn generate( }) } +/// The selection loop both generators share: newest first, stopping at the `--after` +/// cutoff, starting below the `--until` boundary, skipping excluded commits, taking +/// the first `wanted` for which `pick` yields something. +/// +/// Shared rather than copied so a filter can never apply to one generator and not the +/// other — which is the way two task sets silently stop being comparable. +fn take( + history: &gitlog::History, + start: usize, + cutoff: Option, + exclude: &BTreeSet, + wanted: usize, + mut pick: impl FnMut(&gitlog::Commit) -> Option, +) -> Vec { + let mut taken = Vec::new(); + for (i, commit) in history.commits.iter().enumerate().skip(start) { + if taken.len() >= wanted { + break; + } + if cutoff.is_some_and(|stop| i >= stop) { + break; + } + if exclude.contains(&commit.sha) { + continue; + } + if let Some(item) = pick(commit) { + taken.push(item); + } + } + taken +} + +/// The slice of history a generator may draw from: `(start, cutoff)` as indices into +/// the merge-free scan, newest first. +/// +/// `--after` sets the cutoff — the walk stops there, so every task describes a change +/// made after the state an index will be built at. `--until` sets the start — the +/// walk begins strictly *below* it, which is how a training corpus stays disjoint +/// from every evaluation window. +fn window( + root: &Path, + history: &gitlog::History, + head: &str, + after: Option<&str>, + until: Option<&str>, +) -> Result<(usize, Option)> { + let position = |sha: &str| { + history + .commits + .iter() + .position(|c| c.sha.starts_with(sha) || sha.starts_with(&c.sha)) + }; + let cutoff = after + .map(|rev| resolve(root, rev)) + .transpose()? + .and_then(|sha| position(&sha)); + let start = match until.map(|rev| resolve(root, rev)).transpose()? { + None => 0, + // The scanned list is merge-free, so a merge commit named as the boundary is + // legitimately absent from it. When the boundary is HEAD itself, "strictly + // older than HEAD" excludes nothing the list contains. + Some(sha) if sha == head => 0, + Some(sha) => match position(&sha) { + Some(position) => position + 1, + // A merge commit is legitimately absent from the merge-free list, so the + // boundary falls back to its timestamp: strictly-older-than holds for + // every commit authored before it. + None => { + let at = commit_time(root, &sha)?; + history + .commits + .iter() + .position(|c| c.timestamp < at) + .with_context(|| { + format!("--until {sha}: nothing older within the scanned history") + })? + } + }, + }; + Ok((start, cutoff)) +} + /// Author timestamp of a commit, for boundary fallback when the commit itself is a /// merge and therefore missing from the merge-free scan. fn commit_time(root: &Path, sha: &str) -> Result { @@ -284,6 +336,341 @@ fn clean_subject(subject: &str) -> String { out.split_whitespace().collect::>().join(" ") } +// ---- the held-out-hunk task set -------------------------------------------- +// +// A retrieval task asks *which files should I open*. A held-out-hunk task asks the +// opposite question: given a patch that is deliberately incomplete, *what did it +// miss*? The construction is model-free — a merged commit is complete by definition, +// so removing one hunk from it manufactures a known omission and leaves the complete +// commit behind as a negative control. Nothing is hand-labelled and nothing is +// judged; the label is the hunk that was taken out. + +/// One hunk of a unified diff, in pre-image coordinates. +/// +/// Pre-image, because the index this is scored against is built at the parent commit: +/// post-image line numbers name lines that do not exist there. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Hunk { + /// First line of the hunk, context included. + pub old_start: u32, + /// Lines of the pre-image the hunk covers, context included. + pub old_len: u32, + /// Pre-image lines the hunk actually *changes*, with context excluded. + /// + /// Separate from the span because context lines routinely reach into the + /// neighbouring function, and resolving a symbol from them would attribute a + /// change to code the patch never touched. + pub changed_lines: Vec, +} + +impl Hunk { + /// The first line this hunk changes, for resolving the symbol it lands in. + pub fn first_changed(&self) -> u32 { + self.changed_lines + .first() + .copied() + .unwrap_or(self.old_start) + } +} + +/// One file's worth of a change. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct FilePatch { + /// Pre-image path, or the post-image path for a file the change creates. + pub path: String, + /// The change creates this file, so it has no pre-image and no indexed symbols. + pub created: bool, + pub hunks: Vec, +} + +/// A change, as the set of files and pre-image lines it touches. +/// +/// Structural rather than textual on purpose: the checker under test reads locations +/// and the graph, never diff text, so carrying the text would invite a checker that +/// greps it. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +pub struct Patch { + pub files: Vec, +} + +impl Patch { + /// The same change with one file left out entirely. + fn without(&self, path: &str) -> Patch { + Patch { + files: self + .files + .iter() + .filter(|f| f.path != path) + .cloned() + .collect(), + } + } +} + +/// One held-out-hunk trial: a truncated change, and the complete one it came from. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct TruncatedTask { + pub id: String, + pub commit: String, + /// The commit an index must be built at. The change is absent there by + /// construction, which is the same guarantee `--after` gives the retrieval set. + pub parent: String, + pub date: String, + /// The developer's own description, kept for tracing a result back to a change. + pub prompt: String, + /// The change as merged. Complete by construction, so every finding against it is + /// a false positive — this is the negative control, not a second data point. + pub complete: Patch, + /// The change with the omission's file removed. + pub truncated: Patch, + /// The file whose only hunk was withheld. + pub omission_file: String, + /// First pre-image line the withheld hunk changes. + pub omission_line: u32, +} + +/// A frozen set of held-out-hunk trials. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct TruncatedSet { + pub repository: String, + pub head: String, + pub generated_from_commits: usize, + /// Commits that passed every retrieval filter but could not be truncated, and why. + /// Reported rather than dropped: a construction that silently discards most of its + /// candidates is measuring the survivors, not the repository. + pub rejected: Vec<(String, String)>, + pub tasks: Vec, +} + +/// Build held-out-hunk trials from a repository's history. +/// +/// Every filter `generate` applies applies here unchanged — the two share `take` and +/// `candidate` — plus two the construction needs: +/// +/// 1. the change must touch **at least two** indexable files that exist at the parent, +/// so removing one leaves a patch behind; +/// 2. one of those files must be touched by **exactly one hunk**, which is the hunk +/// withheld. Removing it removes the file from the patch entirely, so a finding +/// that cites that file cannot be an echo of a hunk still present in it. +/// +/// Among the files with exactly one hunk the **last by path order** is chosen. The +/// choice is arbitrary and fixed; it is made before any checker runs and there is no +/// knob on it. +pub fn generate_truncated( + root: &Path, + wanted: usize, + scan: usize, + after: Option<&str>, + until: Option<&str>, + exclude: &BTreeSet, +) -> Result { + let head = gitlog::head_sha(root).context("reading HEAD")?; + let history = gitlog::history(root, scan)?; + let (start, cutoff) = window(root, &history, &head, after, until)?; + + let mut rejected: Vec<(String, String)> = Vec::new(); + let tasks = take(&history, start, cutoff, exclude, wanted, |commit| { + let task = candidate(root, commit)?; + if task.ground_truth.len() < 2 { + return None; // a one-file change has nothing left after a truncation + } + let parent = match parent_of(root, &commit.sha) { + Some(parent) => parent, + None => { + rejected.push((commit.sha.clone(), "no parent commit".into())); + return None; + } + }; + let patch = match parse_patch(root, &commit.sha) { + Ok(patch) => patch, + Err(_) => { + rejected.push((commit.sha.clone(), "unreadable diff".into())); + return None; + } + }; + + // Only files that exist at the parent and that the indexer treats as code can + // carry a symbol the checker could ever cite. + let indexable: Vec<&FilePatch> = patch + .files + .iter() + .filter(|f| { + !f.created + && !f.hunks.is_empty() + && reify::discover::classify(&f.path).is_code() + && exists_at(root, &parent, &f.path) + }) + .collect(); + if indexable.len() < 2 { + rejected.push((commit.sha.clone(), "fewer than two indexable files".into())); + return None; + } + let Some(omission) = indexable + .iter() + .filter(|f| f.hunks.len() == 1) + .max_by(|a, b| a.path.cmp(&b.path)) + else { + rejected.push(( + commit.sha.clone(), + "no file changed by exactly one hunk".into(), + )); + return None; + }; + let omission_file = omission.path.clone(); + let omission_line = omission.hunks[0].first_changed(); + + let complete = Patch { + files: indexable.iter().map(|f| (*f).clone()).collect(), + }; + Some(TruncatedTask { + id: format!("v-{}", &commit.sha[..8]), + commit: commit.sha.clone(), + parent, + date: commit.date(), + prompt: task.prompt, + truncated: complete.without(&omission_file), + complete, + omission_file, + omission_line, + }) + }); + + Ok(TruncatedSet { + repository: root.display().to_string(), + head, + generated_from_commits: history.commits.len(), + rejected, + tasks, + }) +} + +/// First parent of a commit, or `None` for a root commit. +fn parent_of(root: &Path, sha: &str) -> Option { + let output = Command::new("git") + .args(["rev-parse", &format!("{sha}^")]) + .current_dir(root) + .output() + .ok()?; + if !output.status.success() { + return None; + } + Some(String::from_utf8_lossy(&output.stdout).trim().to_string()) +} + +/// The change a commit made, against its first parent. +fn parse_patch(root: &Path, sha: &str) -> Result { + let output = Command::new("git") + .args([ + "-c", + "core.quotepath=false", + "show", + "--format=", + "--no-color", + // Renames would report a file as changed with no hunks in it, and a + // similarity threshold is a knob this measurement should not have. + "--no-renames", + "--first-parent", + sha, + ]) + .current_dir(root) + .output() + .context("running git show for a patch")?; + anyhow::ensure!(output.status.success(), "cannot read the diff of {sha}"); + Ok(parse_unified(&String::from_utf8_lossy(&output.stdout))) +} + +/// Parse unified diff text into files and pre-image line ranges. +fn parse_unified(text: &str) -> Patch { + let mut patch = Patch::default(); + let mut cursor = 0u32; + for line in text.lines() { + if let Some(rest) = line.strip_prefix("--- ") { + patch.files.push(FilePatch { + path: strip_prefix_path(rest), + created: rest == "/dev/null", + hunks: Vec::new(), + }); + continue; + } + let Some(file) = patch.files.last_mut() else { + continue; + }; + if let Some(rest) = line.strip_prefix("+++ ") { + // A created file has no pre-image path, so the post-image one names it. + if file.created { + file.path = strip_prefix_path(rest); + } + continue; + } + if let Some(header) = line.strip_prefix("@@ ") { + if let Some((old_start, old_len)) = parse_hunk_header(header) { + file.hunks.push(Hunk { + old_start, + old_len, + changed_lines: Vec::new(), + }); + cursor = old_start; + } + continue; + } + let Some(hunk) = file.hunks.last_mut() else { + continue; + }; + match line.chars().next() { + Some(' ') => cursor += 1, + Some('-') => { + hunk.changed_lines.push(cursor); + cursor += 1; + } + // An inserted line has no pre-image number of its own. + // + // When it replaces lines this hunk has already deleted, it needs no number: + // those lines are recorded and the replacement is the same change. When it + // is a genuine insertion it is attributed to the pre-image line it lands + // *before*, not the one it lands after — code appended past the end of a + // file then resolves to no symbol at all, which is the honest answer, where + // attributing it backwards would credit the patch with changing a function + // it only wrote underneath. + Some('+') => { + let replaces = cursor > 0 && hunk.changed_lines.last() == Some(&(cursor - 1)); + if !replaces { + hunk.changed_lines.push(cursor.max(hunk.old_start)); + } + } + // "\ No newline at end of file", or a blank line git emits as empty. + _ => {} + } + } + for file in &mut patch.files { + for hunk in &mut file.hunks { + hunk.changed_lines.sort_unstable(); + hunk.changed_lines.dedup(); + } + } + patch +} + +/// `a/src/x.rs` -> `src/x.rs`; `/dev/null` -> empty. +fn strip_prefix_path(raw: &str) -> String { + let raw = raw.trim_end(); + if raw == "/dev/null" { + return String::new(); + } + raw.split_once('/') + .map_or(raw, |(_, rest)| rest) + .to_string() +} + +/// `-12,7 +12,8 @@ fn something` -> `(12, 7)`. +fn parse_hunk_header(header: &str) -> Option<(u32, u32)> { + let old = header.split_whitespace().next()?.strip_prefix('-')?; + let (start, len) = match old.split_once(',') { + Some((start, len)) => (start, len.parse().ok()?), + None => (old, 1u32), + }; + Some((start.parse().ok()?, len)) +} + #[cfg(test)] mod tests { use super::*; @@ -335,6 +722,101 @@ mod tests { } } + #[test] + fn the_repository_name_is_the_last_path_component() { + let set = |repository: &str| TaskSet { + repository: repository.into(), + head: String::new(), + generated_from_commits: 0, + base: None, + tasks: Vec::new(), + }; + assert_eq!(set(".bench/medusa").repository_name(), "medusa"); + assert_eq!(set("/a/b/openmrs/").repository_name(), "openmrs"); + assert_eq!(set("reify").repository_name(), "reify"); + } + + const SAMPLE_DIFF: &str = "\ +diff --git a/app/pricing.py b/app/pricing.py +index 111..222 100644 +--- a/app/pricing.py ++++ b/app/pricing.py +@@ -10,6 +10,7 @@ class Pricing: + ctx + ctx +- old_line ++ new_line ++ extra_line + ctx + ctx +@@ -40,3 +41,3 @@ def other(): + ctx +- gone ++ added +diff --git a/app/new.py b/app/new.py +new file mode 100644 +--- /dev/null ++++ b/app/new.py +@@ -0,0 +1,2 @@ ++one ++two +"; + + #[test] + fn a_unified_diff_parses_into_pre_image_line_ranges() { + let patch = parse_unified(SAMPLE_DIFF); + assert_eq!(patch.files.len(), 2); + let pricing = &patch.files[0]; + assert_eq!(pricing.path, "app/pricing.py"); + assert!(!pricing.created); + assert_eq!(pricing.hunks.len(), 2); + assert_eq!( + (pricing.hunks[0].old_start, pricing.hunks[0].old_len), + (10, 6) + ); + // Line 12 is the deletion; the insertions replace it, so it is the only + // pre-image line the hunk changes. + assert_eq!(pricing.hunks[0].changed_lines, vec![12]); + assert_eq!(pricing.hunks[1].changed_lines, vec![41]); + assert_eq!(pricing.hunks[1].first_changed(), 41); + } + + #[test] + fn a_created_file_is_marked_and_named_from_its_post_image() { + let patch = parse_unified(SAMPLE_DIFF); + let created = &patch.files[1]; + assert!( + created.created, + "a file with no pre-image has no indexed symbols" + ); + assert_eq!(created.path, "app/new.py"); + } + + #[test] + fn code_appended_past_the_end_of_a_file_resolves_past_the_end() { + // Attributing an append backwards would credit the patch with changing the + // function it was written underneath. It changed no existing line. + let patch = parse_unified( + "--- a/a.py\n+++ b/a.py\n@@ -8,3 +8,5 @@\n ctx\n ctx\n ctx\n+new\n+new\n", + ); + assert_eq!(patch.files[0].hunks[0].changed_lines, vec![11]); + } + + #[test] + fn truncating_removes_the_file_and_leaves_the_rest() { + let patch = parse_unified(SAMPLE_DIFF); + let truncated = patch.without("app/pricing.py"); + assert_eq!(truncated.files.len(), 1); + assert_eq!(truncated.files[0].path, "app/new.py"); + } + + #[test] + fn a_hunk_header_without_a_length_means_one_line() { + assert_eq!(parse_hunk_header("-12 +12,3 @@ fn x"), Some((12, 1))); + assert_eq!(parse_hunk_header("-12,7 +12,8 @@ fn x"), Some((12, 7))); + assert_eq!(parse_hunk_header("nonsense"), None); + } + #[test] fn a_commit_touching_too_many_files_is_not_a_task() { let files: Vec = (0..30).map(|i| format!("f{i}.py")).collect(); diff --git a/crates/reify-cli/src/install.rs b/crates/reify-cli/src/install.rs new file mode 100644 index 0000000..7dc44e8 --- /dev/null +++ b/crates/reify-cli/src/install.rs @@ -0,0 +1,947 @@ +//! `reify install`: detect the agents that are here, wire each the integration +//! `docs/integration/` recommends for it. +//! +//! # Why this installs a shell command and not an MCP server +//! +//! `docs/integration/claude-code.md` ranks the integrations cheapest first and says to +//! start at level 0, the instruction block: "an MCP server's tool schemas are re-sent on +//! every turn of every session. A CLI costs nothing until it is called. For a tool whose +//! entire purpose is reducing context, paying a per-turn tax to deliver it would be +//! self-defeating." +//! +//! That argument holds, and every agent this command can detect can run a shell command, +//! so level 0 is what gets installed by default. `--mcp` is the deliberate opt-in for +//! the client that cannot, and it says what it costs before it writes anything. +//! +//! # What it will not do +//! +//! **Nothing outside the repository.** The home directory is read as *evidence* that an +//! agent exists — `~/.claude` is Claude Code's real config location — but nothing is +//! written there. A machine-wide MCP registration cannot be undone by a per-repository +//! `reify uninit` without breaking every other repository that relies on it, and an +//! integration that cannot be reversed is one nobody should install. So the MCP entries +//! written here are the repository-scoped ones (`.mcp.json`, `.cursor/mcp.json`); for a +//! client whose only MCP config is machine-wide, the plan says so and writes the +//! instruction block instead. +//! +//! **Nothing it cannot parse.** A config that exists but does not parse is reported and +//! skipped. Overwriting it would be the one failure mode that actually costs somebody +//! their afternoon. +//! +//! **Nothing twice.** Every step checks for its own output first, so a second run is a +//! no-op and says so. + +use anyhow::{Context, Result}; +use std::path::{Path, PathBuf}; + +pub const SCHEMA: &str = "reify.install/1"; + +/// The MCP server entry, written as one line so it disturbs a hand-formatted config as +/// little as possible. +const MCP_ENTRY: &str = r#""reify": { "command": "reify", "args": ["serve", "--mcp"] }"#; + +/// The key under which MCP clients list their servers. +const MCP_SERVERS: &str = "mcpServers"; + +/// Our own key inside it. +const MCP_NAME: &str = "reify"; + +/// A marker that identifies our instruction block wherever it was written. +/// +/// The block itself is [`crate::AGENT_INSTRUCTIONS`]; this is the substring used to +/// recognise it, and it matches what `reify init --write-agent-instructions` already +/// looks for so the two commands never double up on the same file. +const INSTRUCTION_MARKER: &str = "reify context"; + +/// An agent Reify knows how to wire, and the evidence it is here. +struct Known { + name: &'static str, + /// Paths in the repository whose presence is evidence this agent is configured here. + repo_markers: &'static [&'static str], + /// Paths under the user's home that are this agent's real config location. + /// + /// Evidence only. Nothing is ever written to any of them. + home_markers: &'static [&'static str], + /// A rules directory. When it exists, a dedicated file goes in it rather than + /// appending to a shared one — a file of our own is cleanly removable. + rules_dir: Option<(&'static str, &'static str)>, + /// Otherwise, the instruction file the block is appended to. + instruction_file: &'static str, + /// This client's *repository-scoped* MCP config, if it has one. + mcp_config: Option<&'static str>, +} + +/// The agents, and what each one reads. +/// +/// `AGENTS.md` is deliberately its own row rather than evidence for Codex or OpenCode: +/// it is a shared convention that a dozen tools read, and claiming a specific agent is +/// installed because a generic file exists is exactly the guess this command must not +/// make. +const KNOWN: &[Known] = &[ + Known { + name: "Claude Code", + repo_markers: &["CLAUDE.md", ".claude"], + home_markers: &[".claude"], + rules_dir: None, + instruction_file: "CLAUDE.md", + mcp_config: Some(".mcp.json"), + }, + Known { + name: "Cursor", + repo_markers: &[".cursor", ".cursorrules"], + home_markers: &[".cursor"], + rules_dir: Some((".cursor/rules", "reify.mdc")), + instruction_file: ".cursorrules", + mcp_config: Some(".cursor/mcp.json"), + }, + Known { + name: "Windsurf", + repo_markers: &[".windsurf", ".windsurfrules"], + home_markers: &[".codeium/windsurf"], + rules_dir: Some((".windsurf/rules", "reify.md")), + instruction_file: ".windsurfrules", + // Windsurf's MCP config is machine-wide only, so there is nothing repository + // scoped to write. The instruction block is the integration here. + mcp_config: None, + }, + Known { + name: "Cline", + repo_markers: &[".clinerules"], + home_markers: &[], + rules_dir: Some((".clinerules", "reify.md")), + instruction_file: ".clinerules", + mcp_config: None, + }, + Known { + name: "GitHub Copilot", + repo_markers: &[".github/copilot-instructions.md"], + home_markers: &[], + rules_dir: None, + instruction_file: ".github/copilot-instructions.md", + mcp_config: None, + }, + Known { + name: "Codex", + repo_markers: &[".codex"], + home_markers: &[".codex"], + rules_dir: None, + instruction_file: "AGENTS.md", + mcp_config: None, + }, + Known { + name: "OpenCode", + repo_markers: &[".opencode"], + home_markers: &[".config/opencode"], + rules_dir: None, + instruction_file: "AGENTS.md", + mcp_config: None, + }, + Known { + name: "Aider", + repo_markers: &["CONVENTIONS.md", ".aider.conf.yml"], + home_markers: &[".aider.conf.yml"], + rules_dir: None, + instruction_file: "CONVENTIONS.md", + mcp_config: None, + }, + Known { + name: "any agent reading AGENTS.md", + repo_markers: &["AGENTS.md"], + home_markers: &[], + rules_dir: None, + instruction_file: "AGENTS.md", + mcp_config: None, + }, +]; + +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Kind { + /// Append the instruction block to a file the agent already reads. + Instructions, + /// Write a dedicated rule file into the agent's rules directory. + RuleFile, + /// Merge a server entry into this client's repository-scoped MCP config. + Mcp, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "snake_case")] +pub enum State { + /// Not there yet; `--yes` will write it. + Planned, + /// Already there. A second run changes nothing. + AlreadyPresent, + /// The file exists and could not be parsed, so it was left alone. + Skipped, +} + +/// One thing `install` would do, and to what. +#[derive(Debug, Clone, serde::Serialize)] +pub struct Step { + /// Repository-relative, always with `/` separators. + pub path: String, + pub kind: Kind, + /// Every agent this one write serves. More than one when two agents read the + /// same file. + pub agents: Vec, + /// What made Reify think those agents are here. + pub evidence: Vec, + pub state: State, + /// Present only when `state` is `skipped`. + pub problem: Option, +} + +#[derive(Debug, Clone, serde::Serialize)] +pub struct Plan { + pub schema: &'static str, + pub root: String, + /// Whether MCP was requested, and therefore whether the per-turn cost was accepted. + pub mcp: bool, + /// Whether the plan was applied, or only shown. + pub applied: bool, + pub steps: Vec, + /// The block to paste by hand, present when no agent was recognised. + pub instruction_block: Option, + /// Agents installed on this machine that nothing in this repository configures. + /// + /// Reported rather than acted on: it explains why an agent the user knows they have + /// is not in the plan, without pretending a home directory says anything about this + /// repository. + pub detected_elsewhere: Vec, +} + +impl Plan { + pub fn has_work(&self) -> bool { + self.steps.iter().any(|s| s.state == State::Planned) + } +} + +/// Build the plan without writing anything. +pub fn plan(root: &Path, mcp: bool) -> Result { + plan_with_home(root, mcp, home_dir().as_deref()) +} + +/// The same, with the home directory supplied. +/// +/// Injected rather than read from the environment so the rule that home evidence never +/// triggers a write can be tested without mutating a process-wide variable that every +/// other test in this binary shares. +pub fn plan_with_home(root: &Path, mcp: bool, home: Option<&Path>) -> Result { + let mut steps: Vec = Vec::new(); + + let mut elsewhere: Vec = Vec::new(); + + for agent in KNOWN { + let mut evidence: Vec = agent + .repo_markers + .iter() + .filter(|m| root.join(m).exists()) + .map(|m| format!("{m} is here")) + .collect(); + let at_home: Vec = home + .into_iter() + .flat_map(|home| { + agent + .home_markers + .iter() + .filter(move |m| home.join(m).exists()) + .map(|m| format!("~/{m} exists")) + }) + .collect(); + + // Repository evidence is required before anything is written. `~/.cursor` + // means this user has Cursor installed, not that this repository is worked on + // with it — creating a `.cursorrules` on that basis is the guess this command + // exists to avoid. Home evidence corroborates; it never triggers. + if evidence.is_empty() { + if !at_home.is_empty() { + elsewhere.push(format!("{} ({})", agent.name, at_home.join(", "))); + } + continue; + } + evidence.extend(at_home); + + // MCP is registered *instead of* the instruction block where the client has a + // repository-scoped config: an agent given both pays for the schemas every turn + // and reads instructions telling it to use the CLI anyway. + let target = match (mcp, agent.mcp_config) { + (true, Some(config)) => (config.to_string(), Kind::Mcp), + _ => match agent.rules_dir { + Some((dir, file)) if root.join(dir).is_dir() => { + (format!("{dir}/{file}"), Kind::RuleFile) + } + _ => (agent.instruction_file.to_string(), Kind::Instructions), + }, + }; + + // Two agents reading one file is one write, credited to both. + match steps.iter_mut().find(|s| s.path == target.0) { + Some(existing) => { + existing.agents.push(agent.name.to_string()); + existing.evidence.extend(evidence); + } + None => { + let (state, problem) = inspect(root, &target.0, target.1); + steps.push(Step { + path: target.0, + kind: target.1, + agents: vec![agent.name.to_string()], + evidence, + state, + problem, + }); + } + } + } + steps.sort_by(|a, b| a.path.cmp(&b.path)); + + Ok(Plan { + schema: SCHEMA, + root: root.display().to_string(), + mcp, + applied: false, + instruction_block: steps + .is_empty() + .then(|| crate::AGENT_INSTRUCTIONS.trim().to_string()), + steps, + detected_elsewhere: elsewhere, + }) +} + +/// Is this step already done, or is its target unusable? +fn inspect(root: &Path, rel: &str, kind: Kind) -> (State, Option) { + let path = root.join(rel); + let Ok(text) = std::fs::read_to_string(&path) else { + // Absent is the normal case: it will be created. + return (State::Planned, None); + }; + match kind { + Kind::Instructions | Kind::RuleFile => { + if text.contains(INSTRUCTION_MARKER) { + (State::AlreadyPresent, None) + } else { + (State::Planned, None) + } + } + Kind::Mcp => match serde_json::from_str::(&text) { + Ok(value) => { + if value + .get(MCP_SERVERS) + .and_then(|s| s.get(MCP_NAME)) + .is_some() + { + (State::AlreadyPresent, None) + } else { + (State::Planned, None) + } + } + // Never overwritten. A config that exists but does not parse is somebody's + // work in progress, and clobbering it is the one outcome worth avoiding + // above all others. + Err(e) => ( + State::Skipped, + Some(format!("{rel} is not valid JSON ({e}); left untouched")), + ), + }, + } +} + +/// Apply every planned step. +pub fn apply(root: &Path, plan: &mut Plan) -> Result<()> { + for step in &mut plan.steps { + if step.state != State::Planned { + continue; + } + let path = root.join(&step.path); + if let Some(parent) = path.parent() { + std::fs::create_dir_all(parent) + .with_context(|| format!("creating {}", parent.display()))?; + } + match step.kind { + Kind::Instructions => { + let mut text = std::fs::read_to_string(&path).unwrap_or_default(); + if !text.is_empty() && !text.ends_with('\n') { + text.push('\n'); + } + text.push_str(crate::AGENT_INSTRUCTIONS); + std::fs::write(&path, text) + .with_context(|| format!("writing {}", path.display()))?; + } + Kind::RuleFile => { + std::fs::write(&path, rule_file_body(&step.path)) + .with_context(|| format!("writing {}", path.display()))?; + } + Kind::Mcp => { + let before = std::fs::read_to_string(&path).unwrap_or_default(); + let after = with_mcp_entry(&before)?; + std::fs::write(&path, after) + .with_context(|| format!("writing {}", path.display()))?; + } + } + step.state = State::AlreadyPresent; + } + plan.applied = true; + Ok(()) +} + +/// The contents of a dedicated rule file. +/// +/// Cursor's `.mdc` rules need frontmatter to apply to every request; without it the +/// file is written and silently never read, which is worse than not writing it. +fn rule_file_body(rel: &str) -> String { + let block = crate::AGENT_INSTRUCTIONS.trim_start(); + if rel.ends_with(".mdc") { + format!("---\ndescription: Reify\nalwaysApply: true\n---\n\n{block}") + } else { + block.to_string() + } +} + +/// Splice our server entry into an MCP config, preserving everything else byte for byte. +/// +/// Textual rather than a `serde_json` round-trip on purpose. Re-serialising sorts the +/// user's keys, collapses their indentation and drops the shape of a file they wrote by +/// hand — this config is theirs, and the only part of it that should change is the part +/// being added. The result is parsed before it is returned, so a splice that would +/// produce broken JSON fails loudly instead of being written. +pub fn with_mcp_entry(text: &str) -> Result { + if text.trim().is_empty() { + return Ok(format!( + "{{\n \"{MCP_SERVERS}\": {{\n {MCP_ENTRY}\n }}\n}}\n" + )); + } + let parsed: serde_json::Value = + serde_json::from_str(text).context("the existing MCP config is not valid JSON")?; + if parsed + .get(MCP_SERVERS) + .and_then(|s| s.get(MCP_NAME)) + .is_some() + { + return Ok(text.to_string()); + } + + let out = match body_start(text, Some(MCP_SERVERS)) { + // `mcpServers` is there: add one member to it. + Some(at) => splice(text, at, MCP_ENTRY, 4), + // It is not: add the whole key to the root object. + None => { + let at = body_start(text, None) + .context("the existing MCP config has no top-level object")?; + splice( + text, + at, + &format!("\"{MCP_SERVERS}\": {{ {MCP_ENTRY} }}"), + 2, + ) + } + }; + serde_json::from_str::(&out) + .context("adding the server entry would have produced invalid JSON; nothing written")?; + Ok(out) +} + +/// Remove our server entry, leaving everything else byte for byte. +/// +/// Returns `None` when there was nothing to remove. +pub fn without_mcp_entry(text: &str) -> Result> { + let parsed: serde_json::Value = + serde_json::from_str(text).context("the MCP config is not valid JSON")?; + if parsed + .get(MCP_SERVERS) + .and_then(|s| s.get(MCP_NAME)) + .is_none() + { + return Ok(None); + } + let Some(span) = member_span(text, MCP_SERVERS, MCP_NAME) else { + return Ok(None); + }; + let mut out = String::with_capacity(text.len()); + out.push_str(&text[..span.0]); + out.push_str(&text[span.1..]); + serde_json::from_str::(&out) + .context("removing the server entry would have produced invalid JSON; nothing written")?; + Ok(Some(out)) +} + +/// Insert `member` just inside an object whose body starts at `at`. +fn splice(text: &str, at: usize, member: &str, indent: usize) -> String { + let rest = &text[at..]; + let empty = rest.trim_start().starts_with('}'); + let pad = " ".repeat(indent); + let mut out = String::with_capacity(text.len() + member.len() + 8); + out.push_str(&text[..at]); + out.push('\n'); + out.push_str(&pad); + out.push_str(member); + if !empty { + out.push(','); + } + // An object that was `{}` gets its closing brace put on its own line; one that + // already had members keeps whatever the author wrote after the brace. + if empty { + out.push('\n'); + out.push_str(&" ".repeat(indent.saturating_sub(2))); + out.push_str(rest.trim_start()); + } else { + out.push_str(rest); + } + out +} + +/// Byte offset just past the `{` opening the root object, or the object at top-level +/// `key`. +/// +/// A small scanner rather than a JSON library: the caller has already parsed the text +/// for validity, and what is needed here is a *position in the original bytes*, which no +/// parse tree carries. +fn body_start(text: &str, key: Option<&str>) -> Option { + let bytes = text.as_bytes(); + let mut depth = 0usize; + let mut in_string = false; + let mut escaped = false; + let mut opened_at = 0usize; + let mut awaiting = false; + + for (i, &c) in bytes.iter().enumerate() { + if in_string { + if escaped { + escaped = false; + } else if c == b'\\' { + escaped = true; + } else if c == b'"' { + in_string = false; + // A key sits at depth 1 — inside the root object — and is followed by + // a colon. Anything else and this was a value that happened to match. + if depth == 1 && key.is_some_and(|k| &text[opened_at + 1..i] == k) { + awaiting = true; + } + } + continue; + } + if awaiting && !c.is_ascii_whitespace() && c != b':' && c != b'{' { + awaiting = false; + } + match c { + b'"' => { + in_string = true; + opened_at = i; + } + b'{' => { + depth += 1; + if awaiting { + return Some(i + 1); + } + if key.is_none() && depth == 1 { + return Some(i + 1); + } + } + b'}' | b']' => depth = depth.saturating_sub(1), + b'[' => depth += 1, + _ => {} + } + } + None +} + +/// The byte range covering `parent.member` and the comma that separates it from its +/// neighbours, so cutting it leaves valid JSON. +fn member_span(text: &str, parent: &str, member: &str) -> Option<(usize, usize)> { + let body = body_start(text, Some(parent))?; + let bytes = text.as_bytes(); + let mut i = body; + let mut depth = 0usize; + let mut in_string = false; + let mut escaped = false; + let mut key_at: Option<(usize, usize)> = None; + let mut quote_at = 0usize; + + while i < bytes.len() { + let c = bytes[i]; + if in_string { + if escaped { + escaped = false; + } else if c == b'\\' { + escaped = true; + } else if c == b'"' { + in_string = false; + if depth == 0 && &text[quote_at + 1..i] == member { + key_at = Some((quote_at, i)); + } + } + i += 1; + continue; + } + match c { + b'"' => { + in_string = true; + quote_at = i; + } + b'{' | b'[' => depth += 1, + b'}' | b']' => { + if depth == 0 { + return None; // end of the parent object, member not found + } + depth -= 1; + // A value that has just closed at the parent's own level ends the + // member we were tracking. + if depth == 0 { + if let Some((start, _)) = key_at { + return Some(widen(text, start, i + 1)); + } + } + } + b',' if depth == 0 => { + if let Some((start, _)) = key_at { + return Some(widen(text, start, i + 1)); + } + } + _ => {} + } + i += 1; + } + None +} + +/// Widen a member's range to swallow one separating comma and the whitespace around it. +/// +/// Without this, removing the last member of an object leaves a trailing comma, which is +/// not valid JSON. +fn widen(text: &str, start: usize, end: usize) -> (usize, usize) { + let bytes = text.as_bytes(); + let mut end = end; + // A comma after the member: take it, plus the newline it sat on. + let mut probe = end; + while probe < bytes.len() && bytes[probe].is_ascii_whitespace() && bytes[probe] != b'\n' { + probe += 1; + } + if probe < bytes.len() && bytes[probe] == b',' { + end = probe + 1; + } else { + // No comma after, so this was the last member: take the one before it instead. + let mut back = start; + while back > 0 && bytes[back - 1].is_ascii_whitespace() { + back -= 1; + } + if back > 0 && bytes[back - 1] == b',' { + return (back - 1, end); + } + } + // Leading whitespace on the member's own line goes with it. + let mut begin = start; + while begin > 0 && (bytes[begin - 1] == b' ' || bytes[begin - 1] == b'\t') { + begin -= 1; + } + if begin > 0 && bytes[begin - 1] == b'\n' { + begin -= 1; + } + (begin, end) +} + +/// Every repository path `install` could ever write to, for `uninit` to undo. +/// +/// Derived from the same table `plan` walks, so a new agent cannot be added to one +/// without appearing in the other. +pub fn removable_targets() -> Vec<(String, Kind)> { + let mut out: Vec<(String, Kind)> = Vec::new(); + for agent in KNOWN { + out.push((agent.instruction_file.to_string(), Kind::Instructions)); + if let Some((dir, file)) = agent.rules_dir { + out.push((format!("{dir}/{file}"), Kind::RuleFile)); + } + if let Some(config) = agent.mcp_config { + out.push((config.to_string(), Kind::Mcp)); + } + } + out.sort_by(|a, b| a.0.cmp(&b.0)); + out.dedup_by(|a, b| a.0 == b.0); + out +} + +fn home_dir() -> Option { + std::env::var_os("HOME") + .or_else(|| std::env::var_os("USERPROFILE")) + .filter(|h| !h.is_empty()) + .map(PathBuf::from) +} + +#[cfg(test)] +mod tests { + use super::*; + use std::fs; + + fn tmp(name: &str) -> PathBuf { + let d = std::env::temp_dir().join(format!("reify-install-{}-{name}", std::process::id())); + let _ = fs::remove_dir_all(&d); + fs::create_dir_all(&d).unwrap(); + d + } + + /// A repository with a `CLAUDE.md` and nothing else Reify recognises. + fn claude_repo(name: &str) -> PathBuf { + let d = tmp(name); + fs::write(d.join("CLAUDE.md"), "# My project\n\nSome house rules.\n").unwrap(); + d + } + + #[test] + fn a_claude_repository_gets_the_shell_integration_not_mcp() { + // The documented position: level 0 first, because MCP schemas cost tokens on + // every turn of every session. + let d = claude_repo("level0"); + let plan = plan_with_home(&d, false, None).unwrap(); + assert_eq!(plan.steps.len(), 1); + assert_eq!(plan.steps[0].path, "CLAUDE.md"); + assert_eq!(plan.steps[0].kind, Kind::Instructions); + assert_eq!(plan.steps[0].agents, vec!["Claude Code"]); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn mcp_is_the_deliberate_opt_in_and_replaces_the_block_rather_than_joining_it() { + let d = claude_repo("mcpoptin"); + let plan = plan_with_home(&d, true, None).unwrap(); + assert_eq!(plan.steps.len(), 1, "one integration per agent, not two"); + assert_eq!(plan.steps[0].path, ".mcp.json"); + assert_eq!(plan.steps[0].kind, Kind::Mcp); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn planning_writes_nothing_and_applying_is_idempotent() { + let d = claude_repo("idempotent"); + let before = fs::read_to_string(d.join("CLAUDE.md")).unwrap(); + + let mut p = plan_with_home(&d, false, None).unwrap(); + assert!(p.has_work()); + assert_eq!( + fs::read_to_string(d.join("CLAUDE.md")).unwrap(), + before, + "a plan must not write" + ); + + apply(&d, &mut p).unwrap(); + let after = fs::read_to_string(d.join("CLAUDE.md")).unwrap(); + assert!(after.starts_with(&before), "the user's content is kept"); + assert!(after.contains("reify context")); + + let again = plan_with_home(&d, false, None).unwrap(); + assert!(!again.has_work(), "a second run has nothing to do"); + let mut again = again; + apply(&d, &mut again).unwrap(); + assert_eq!( + fs::read_to_string(d.join("CLAUDE.md")).unwrap(), + after, + "applying a no-op plan changes nothing" + ); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn a_rules_directory_gets_its_own_file_rather_than_a_shared_one() { + let d = tmp("rulesdir"); + fs::create_dir_all(d.join(".cursor/rules")).unwrap(); + let p = plan_with_home(&d, false, None).unwrap(); + let step = p.steps.iter().find(|s| s.kind == Kind::RuleFile).unwrap(); + assert_eq!(step.path, ".cursor/rules/reify.mdc"); + let mut p = p; + apply(&d, &mut p).unwrap(); + let body = fs::read_to_string(d.join(".cursor/rules/reify.mdc")).unwrap(); + assert!( + body.starts_with("---\n") && body.contains("alwaysApply: true"), + "a Cursor rule without frontmatter is written and never read" + ); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn two_agents_reading_one_file_are_one_write_credited_to_both() { + let d = tmp("shared"); + fs::write(d.join("AGENTS.md"), "# rules\n").unwrap(); + fs::create_dir_all(d.join(".codex")).unwrap(); + let p = plan_with_home(&d, false, None).unwrap(); + let step = p.steps.iter().find(|s| s.path == "AGENTS.md").unwrap(); + assert!(step.agents.len() >= 2, "{:?}", step.agents); + assert_eq!( + p.steps.iter().filter(|s| s.path == "AGENTS.md").count(), + 1, + "one file, one write" + ); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn an_agent_installed_on_this_machine_but_not_in_this_repository_is_never_written_for() { + // `~/.cursor` means the user has Cursor, not that this repository is worked on + // with it. Creating a `.cursorrules` on that evidence is the guess this command + // exists to avoid — so it is reported instead, which also explains the silence. + let d = tmp("homeonly"); + fs::write(d.join("main.rs"), "fn main() {}").unwrap(); + let home = tmp("fakehome"); + fs::create_dir_all(home.join(".cursor")).unwrap(); + + let p = plan_with_home(&d, false, Some(&home)).unwrap(); + assert!(p.steps.is_empty(), "{:?}", p.steps); + assert!(p.detected_elsewhere.iter().any(|a| a.starts_with("Cursor"))); + assert!(p.instruction_block.is_some(), "hand over the block instead"); + assert!(!d.join(".cursorrules").exists()); + let _ = fs::remove_dir_all(&d); + let _ = fs::remove_dir_all(&home); + } + + #[test] + fn home_evidence_corroborates_repository_evidence_rather_than_replacing_it() { + let d = claude_repo("corroborate"); + let home = tmp("fakehome2"); + fs::create_dir_all(home.join(".claude")).unwrap(); + let p = plan_with_home(&d, false, Some(&home)).unwrap(); + assert_eq!(p.steps.len(), 1); + assert_eq!( + p.steps[0].evidence, + vec!["CLAUDE.md is here", "~/.claude exists"], + "both are stated, so the detection can be checked" + ); + let _ = fs::remove_dir_all(&d); + let _ = fs::remove_dir_all(&home); + } + + #[test] + fn nothing_recognised_offers_the_block_to_paste_rather_than_guessing() { + let d = tmp("unknown"); + fs::write(d.join("main.rs"), "fn main() {}").unwrap(); + let p = plan_with_home(&d, false, None).unwrap(); + assert!(p.steps.is_empty()); + assert!(p.instruction_block.unwrap().contains("reify context")); + let _ = fs::remove_dir_all(&d); + } + + // The requirement most likely to break somebody's setup. Everything the user wrote + // must survive, byte for byte, apart from the entry being added. + #[test] + fn an_existing_mcp_config_survives_byte_identical_apart_from_the_added_entry() { + let original = r#"{ + "mcpServers": { + "postgres": { + "command": "npx", + "args": ["-y", "@modelcontextprotocol/server-postgres", "postgres://localhost/db"], + "env": { "PGPASSWORD": "hunter2" } + } + }, + "somethingElse": [1, 2, 3] +} +"#; + let updated = with_mcp_entry(original).unwrap(); + + // The added line, and nothing else. + let removed: Vec<&str> = original + .lines() + .filter(|l| !updated.lines().any(|u| u == *l)) + .collect(); + assert!(removed.is_empty(), "lines disappeared: {removed:?}"); + let added: Vec<&str> = updated + .lines() + .filter(|l| !original.lines().any(|o| o == *l)) + .collect(); + assert_eq!(added.len(), 1, "expected exactly one added line: {added:?}"); + assert!(added[0].contains("\"reify\"")); + + // And the user's own content is intact when read back. + let after: serde_json::Value = serde_json::from_str(&updated).unwrap(); + let before: serde_json::Value = serde_json::from_str(original).unwrap(); + assert_eq!(after["somethingElse"], before["somethingElse"]); + assert_eq!( + after[MCP_SERVERS]["postgres"], before[MCP_SERVERS]["postgres"], + "an unrelated server must survive exactly" + ); + assert!(after[MCP_SERVERS][MCP_NAME]["command"] == "reify"); + + // Removing it puts the file back the way it was. + let restored = without_mcp_entry(&updated).unwrap().unwrap(); + assert_eq!(restored, original, "uninstalling must be a clean reversal"); + } + + #[test] + fn an_empty_or_absent_config_is_created_whole() { + let fresh = with_mcp_entry("").unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&fresh).unwrap(); + assert_eq!(parsed[MCP_SERVERS][MCP_NAME]["command"], "reify"); + + let empty_object = with_mcp_entry("{}\n").unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&empty_object).unwrap(); + assert_eq!(parsed[MCP_SERVERS][MCP_NAME]["args"][0], "serve"); + + let no_servers_key = with_mcp_entry("{\n \"other\": true\n}\n").unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&no_servers_key).unwrap(); + assert_eq!(parsed["other"], true); + assert_eq!(parsed[MCP_SERVERS][MCP_NAME]["command"], "reify"); + + let empty_servers = with_mcp_entry("{\n \"mcpServers\": {}\n}\n").unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&empty_servers).unwrap(); + assert_eq!(parsed[MCP_SERVERS][MCP_NAME]["command"], "reify"); + } + + #[test] + fn adding_an_entry_twice_changes_nothing() { + let once = with_mcp_entry("{\n \"mcpServers\": {}\n}\n").unwrap(); + assert_eq!(with_mcp_entry(&once).unwrap(), once); + } + + #[test] + fn a_config_that_does_not_parse_is_reported_and_left_alone() { + let d = tmp("broken"); + fs::write(d.join("CLAUDE.md"), "# x\n").unwrap(); + let broken = "{ \"mcpServers\": { oops }"; + fs::write(d.join(".mcp.json"), broken).unwrap(); + + let mut p = plan_with_home(&d, true, None).unwrap(); + let step = &p.steps[0]; + assert_eq!(step.state, State::Skipped); + assert!(step.problem.as_ref().unwrap().contains("not valid JSON")); + + apply(&d, &mut p).unwrap(); + assert_eq!( + fs::read_to_string(d.join(".mcp.json")).unwrap(), + broken, + "an unparsable config is never overwritten" + ); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn a_string_that_merely_looks_like_the_servers_key_is_not_mistaken_for_it() { + // `body_start` scans bytes, so it has to tell a key from a value that happens + // to spell the same thing. + let text = "{\n \"note\": \"mcpServers\",\n \"mcpServers\": {\n \"a\": {}\n }\n}\n"; + let updated = with_mcp_entry(text).unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&updated).unwrap(); + assert_eq!(parsed["note"], "mcpServers"); + assert!(parsed[MCP_SERVERS]["a"].is_object()); + assert_eq!(parsed[MCP_SERVERS][MCP_NAME]["command"], "reify"); + } + + #[test] + fn removing_the_only_entry_leaves_valid_json() { + let text = with_mcp_entry("{}").unwrap(); + let stripped = without_mcp_entry(&text).unwrap().unwrap(); + let parsed: serde_json::Value = serde_json::from_str(&stripped).unwrap(); + assert!(parsed[MCP_SERVERS].get(MCP_NAME).is_none()); + assert!(without_mcp_entry(&stripped).unwrap().is_none()); + } + + #[test] + fn every_agent_in_the_table_is_reachable_by_uninit() { + // A new agent added to KNOWN without a removal path would leave orphans. + let targets = removable_targets(); + for agent in KNOWN { + assert!( + targets.iter().any(|(p, _)| p == agent.instruction_file + || agent + .rules_dir + .is_some_and(|(d, f)| *p == format!("{d}/{f}"))), + "{} has no removal path", + agent.name + ); + } + } +} diff --git a/crates/reify-cli/src/main.rs b/crates/reify-cli/src/main.rs index 25a5f5c..a40e04a 100644 --- a/crates/reify-cli/src/main.rs +++ b/crates/reify-cli/src/main.rs @@ -5,6 +5,7 @@ //! no network call at all in this build, which is asserted by a test rather than //! promised in a README. +mod install; mod mcp; mod render; mod selfmanage; @@ -96,7 +97,7 @@ enum Command { /// What breaks if this changes. Impact { - /// A symbol name or a description of the change. + /// A symbol name, a file path, or a description of the change. query: String, }, @@ -141,6 +142,28 @@ enum Command { path: String, }, + /// Detect the agents present here and wire each the integration it should have. + /// + /// Shows the plan and stops, unless `--yes`. Reversible with `reify uninit`. + Install { + /// Actually write it; without this flag, only the plan is shown. + #[arg(long)] + yes: bool, + /// Register the MCP server instead of the shell-command instruction block. + /// + /// `docs/integration/` recommends the instruction block: an MCP server's tool + /// schemas are re-sent every turn of every session, and a CLI costs nothing + /// until it is called. Use this for a client that cannot run a shell command. + #[arg(long)] + mcp: bool, + }, + + /// Should this repository use Reify at all? Answers before you index. + /// + /// Runs against the working tree and `git log`, never the store, so it works + /// before `reify init`. Willing to say no. + Doctor, + /// Model-assistance status and prompt inspection. Llm { #[command(subcommand)] @@ -308,6 +331,14 @@ fn run() -> Result<()> { let store = open_existing(&root)?; render::preflight(&query::preflight(&store, path)?, cli.json) } + Command::Install { yes, mcp } => { + let mut plan = install::plan(&root, *mcp)?; + if *yes && plan.has_work() { + install::apply(&root, &mut plan)?; + } + render::install(&plan, *yes, cli.json) + } + Command::Doctor => render::doctor(&reify::doctor::diagnose(&root)?, cli.json), Command::Llm { action } => match action { LlmAction::Status => render::llm_status(&root, cli.json), LlmAction::Preview { task, budget } => { @@ -550,6 +581,8 @@ mod tests { vec!["reify", "--json", "preflight", "a.py"], vec!["reify", "--json", "llm", "status"], vec!["reify", "--json", "init"], + vec!["reify", "--json", "doctor"], + vec!["reify", "--json", "install"], ] { let cli = Cli::try_parse_from(&args).expect("should parse"); assert!(cli.json, "{args:?}"); diff --git a/crates/reify-cli/src/mcp.rs b/crates/reify-cli/src/mcp.rs index 8f39d05..f29436a 100644 --- a/crates/reify-cli/src/mcp.rs +++ b/crates/reify-cli/src/mcp.rs @@ -1,11 +1,17 @@ //! A minimal MCP server over stdio. //! -//! **Three tools, and that is the whole surface.** An MCP server's tool schemas are +//! **Six tools, and that is the whole surface.** An MCP server's tool schemas are //! re-sent on every turn of every session, so a fifteen-tool server can cost more //! context than the knowledge it retrieves — which would make Reify a counterexample to -//! its own thesis. Anything beyond these three belongs on the command line, where it +//! its own thesis. Anything beyond these six belongs on the command line, where it //! costs nothing until it is used. //! +//! The six are the questions an agent asks *while editing*. The three added after the +//! original three — `explain`, `flow`, `conflicts` — are the capabilities no other +//! retriever offers, and leaving them out meant the distinctive half of the product +//! was unreachable from the integration path most clients actually use. All six +//! together still cost under the 600-token ceiling the original three were held to. +//! //! The CLI remains the primary surface (`docs/integration/`); this exists for clients //! that cannot run a shell command. @@ -72,8 +78,13 @@ fn dispatch(root: &Path, method: &str, params: &Value) -> Result { } } -/// The three tools. Descriptions are written for a model deciding whether to call +/// The exposed tools. Descriptions are written for a model deciding whether to call /// them, not for a human reading documentation. +/// +/// Deliberately a subset of the CLI. Every tool here answers a question an agent asks +/// *while editing*; the commands left out — `index`, `report`, `concepts` — are +/// operator workflows, and offering them would spend the model's tool budget on +/// choices it should never have to make. fn tool_definitions() -> Vec { vec![ json!({ @@ -106,16 +117,53 @@ fn tool_definitions() -> Vec { json!({ "name": "reify_impact", "description": - "List what depends on a symbol, including through shared database tables \ - where no call edge exists. Call this before changing shared logic.", + "List what depends on a symbol or a file — callers, importers, and \ + coupling through shared database tables where no call edge exists. \ + Call this before changing shared logic.", "inputSchema": { "type": "object", "properties": { - "query": {"type": "string", "description": "A symbol name or a described change"} + "query": {"type": "string", "description": "A symbol name, a file path, or a described change"} }, "required": ["query"] } }), + json!({ + "name": "reify_explain", + "description": + "Everything known about one business concept, in every language it appears \ + in: the code that implements it, the documents that define it, and its \ + other names. Call this when a term in the task is not one you recognise.", + "inputSchema": { + "type": "object", + "properties": { + "term": {"type": "string", "description": "A business term, in any indexed language"} + }, + "required": ["term"] + } + }), + json!({ + "name": "reify_flow", + "description": + "The ordered sequence of code that carries out a business process, end to \ + end. Call this when the task spans several steps and you need the path \ + through them rather than one location.", + "inputSchema": { + "type": "object", + "properties": { + "process": {"type": "string", "description": "A described business process"} + }, + "required": ["process"] + } + }), + json!({ + "name": "reify_conflicts", + "description": + "Documentation that disagrees with the implementation. Call this before \ + trusting a document you were given, or when code and a spec seem to \ + describe different behaviour.", + "inputSchema": {"type": "object", "properties": {}} + }), ] } @@ -165,23 +213,13 @@ fn call_tool(root: &Path, params: &Value) -> Result { })); } let payload = match name { - "reify_context" => { - let budget = arguments - .get("budget") - .and_then(Value::as_u64) - .unwrap_or(context::DEFAULT_BUDGET as u64) as u32; - serde_json::to_value(context::compile( - &store, - &string_arg("task")?, - &ContextOptions { - budget, - ..Default::default() - }, - )?)? - } "reify_why" => serde_json::to_value(query::why(&store, root, &string_arg("target")?)?)?, "reify_impact" => serde_json::to_value(query::impact(&store, &string_arg("query")?)?)?, - // Unreachable: the name was validated against the tool list above. + "reify_explain" => serde_json::to_value(query::explain(&store, &string_arg("term")?)?)?, + "reify_flow" => serde_json::to_value(query::flow(&store, &string_arg("process")?)?)?, + "reify_conflicts" => serde_json::to_value(query::conflicts(&store)?)?, + // Unreachable: the name was validated against the tool list above, and + // `reify_context` returned already. other => anyhow::bail!("unknown tool `{other}`"), }; @@ -218,26 +256,49 @@ mod tests { } #[test] - fn the_tool_surface_is_exactly_three_tools() { + fn the_tool_surface_is_exactly_six_tools() { // Load-bearing: schemas are re-sent every turn, so this is a context budget, - // not a style preference. + // not a style preference. Adding a seventh needs a written reason. + // + // The reason for the last three: `explain`, `flow` and `conflicts` are the + // capabilities no other retriever offers, and MCP is how most agents reach + // Reify at all. Exposing only `context`, `why` and `impact` meant the + // distinctive half of the product was unreachable from the main integration + // path. `preflight` was considered and left out — it answers the same question + // as `why` for an agent, and a near-duplicate tool spends the budget twice + // while making the model's choice harder. let tools = tool_definitions(); assert_eq!( tools.len(), - 3, - "adding a fourth tool needs a written reason" + 6, + "adding a seventh tool needs a written reason" ); for tool in &tools { assert!(tool["name"] .as_str() .is_some_and(|n| n.starts_with("reify_"))); assert!(tool["description"].as_str().is_some_and(|d| d.len() > 40)); - assert!(tool["inputSchema"]["required"].is_array()); + assert!(tool["inputSchema"].is_object()); + } + // Every tool that takes an argument must declare which are required, or a + // model discovers the requirement by getting an error back. + for tool in &tools { + let properties = &tool["inputSchema"]["properties"]; + if properties.as_object().is_some_and(|p| !p.is_empty()) { + assert!( + tool["inputSchema"]["required"].is_array(), + "{} takes arguments but declares none required", + tool["name"] + ); + } } } #[test] fn the_tool_schemas_stay_small_enough_to_be_worth_sending() { + // Doubling the tool count did not double the cost: the three added schemas + // take one string argument each. Still a hard ceiling — this is paid on every + // turn of every conversation, whether or not a tool is called. let rendered = serde_json::to_string(&tool_definitions()).unwrap(); let cost = reify::tokens::estimate(&rendered); assert!(cost < 600, "tool schemas cost {cost} tokens every turn"); diff --git a/crates/reify-cli/src/render.rs b/crates/reify-cli/src/render.rs index 6cfdf6f..c5f5607 100644 --- a/crates/reify-cli/src/render.rs +++ b/crates/reify-cli/src/render.rs @@ -4,9 +4,14 @@ //! terminals, no colour when the output is redirected. Agent output is JSON against a //! versioned schema. //! -//! One rule governs both: an epistemic status is always shown next to a claim. A +//! One rule governs both: a claim is never rendered without its epistemic status. A //! renderer that prints an `INFERRED` rule as bare prose is a bug, and there is a test //! that says so. +//! +//! In the human output that status may be stated once for a whole section when every +//! row shares it, rather than repeated down the page. The property being protected is +//! that a reader can always tell a parsed fact from a guess — not that the badge is +//! printed a fixed number of times. Machine output always carries it per item. use anyhow::Result; use owo_colors::OwoColorize; @@ -14,6 +19,9 @@ use serde::Serialize; use reify::context::Context; use reify::discover::Discovery; +use reify::doctor::{self, Diagnosis, Verdict}; + +use crate::install::{Kind as InstallKind, Plan, State as InstallState, Step as InstallStep}; use reify::index::IndexReport; use reify::llm; use reify::model::{Node, Status}; @@ -66,6 +74,34 @@ fn heading(text: &str) { } } +/// The one status a whole section shares, if it shares one. +/// +/// A badge repeated identically down twenty lines carries no information, and an +/// invariant `[confirmed]` invites exactly the wrong reading: it attests that the +/// symbol was *parsed from source*, never that it is the right place to change. +/// Symbols are `CONFIRMED` by construction, so the code section is nearly always +/// uniform — said once on the heading it is a fact, repeated per line it is noise. +/// The machine-readable output is untouched; every item still carries its status. +fn shared_status(items: &[T], status: impl Fn(&T) -> Status) -> Option { + let first = status(items.first()?); + items.iter().all(|i| status(i) == first).then_some(first) +} + +/// A heading that names the status its whole section shares. +fn heading_for(text: &str, shared: Option) { + match shared { + Some(status) => { + let note = format!("all {}", tag(status)); + if colours_wanted() { + println!("\n{} {}", text.bold(), note.dimmed()); + } else { + println!("\n{text} {note}"); + } + } + None => heading(text), + } +} + /// A progress line that overwrites itself, for a stage that runs for a minute. /// /// Written to **stderr** so `reify index` stays pipeable, and suppressed entirely when @@ -237,6 +273,17 @@ pub fn index_report(report: &IndexReport, json: bool) -> Result<()> { if report.history_truncated { println!(" history walk hit its commit limit; older commits were not read"); } + if let Some(reason) = &report.history_unavailable { + // Said plainly, with the cause: everything else indexed, and the user can + // decide whether the missing evidence is worth completing the clone for. + println!(" history could not be read, so `why` and blast radius lose their"); + println!(" commit evidence. Everything else indexed normally."); + println!(" {}", reason.lines().next().unwrap_or(reason)); + if reason.contains("lazy fetching disabled") || reason.contains("promisor") { + println!(" This is a partial clone missing objects it needs. Complete it with:"); + println!(" git fetch --refetch origin"); + } + } if !report.parse_errors.is_empty() { println!( " {} file(s) could not be parsed:", @@ -320,7 +367,8 @@ pub fn context(compiled: &Context, json: bool) -> Result<()> { } } if !compiled.concepts.is_empty() { - heading("Concepts"); + let shared = shared_status(&compiled.concepts, |c| c.status); + heading_for("Concepts", shared); for concept in &compiled.concepts { let labels = concept .labels @@ -332,7 +380,10 @@ pub fn context(compiled: &Context, json: bool) -> Result<()> { .join(" · ") }) .unwrap_or_default(); - println!(" {} {}", tag(concept.status), concept.id); + match shared { + Some(_) => println!(" {}", concept.id), + None => println!(" {} {}", tag(concept.status), concept.id), + } if !labels.is_empty() { println!(" {labels}"); } @@ -353,28 +404,36 @@ pub fn context(compiled: &Context, json: bool) -> Result<()> { } } if !compiled.code.is_empty() { - heading("Code"); + let shared = shared_status(&compiled.code, |i| i.status); + heading_for("Code", shared); for item in &compiled.code { - println!( - " {} {}:{} {}", - tag(item.status), - item.path, - item.lines, - item.symbol - ); + match shared { + Some(_) => println!(" {}:{} {}", item.path, item.lines, item.symbol), + None => println!( + " {} {}:{} {}", + tag(item.status), + item.path, + item.lines, + item.symbol + ), + } println!(" {}", item.why); } } if !compiled.documents.is_empty() { - heading("Documents"); + let shared = shared_status(&compiled.documents, |d| d.status); + heading_for("Documents", shared); for doc in &compiled.documents { let lang = doc.lang.as_deref().unwrap_or("?"); - println!( - " {} {} [{lang}] {}", - tag(doc.status), - doc.location, - doc.document - ); + match shared { + Some(_) => println!(" {} [{lang}] {}", doc.location, doc.document), + None => println!( + " {} {} [{lang}] {}", + tag(doc.status), + doc.location, + doc.document + ), + } println!(" {}", doc.excerpt); } } @@ -476,7 +535,18 @@ pub fn impact(answer: &ImpactAnswer, json: bool) -> Result<()> { } } if !answer.affected.is_empty() { - heading("Affected"); + // A file every module imports has hundreds of dependants. Printing all of them + // spends an agent's budget to say one thing — "a lot" — so the count leads and + // the nearest few are the evidence for it. + let shown = answer.affected.len(); + if answer.affected_total > shown { + heading(&format!( + "Affected {} total, {shown} nearest shown", + answer.affected_total + )); + } else { + heading(&format!("Affected {shown}")); + } for item in &answer.affected { println!( " {} {} {} ({}, {} hop{})", @@ -692,6 +762,294 @@ pub fn preflight(answer: &Preflight, json: bool) -> Result<()> { Ok(()) } +/// `reify install`: what was found, and what will be done about it. +/// +/// Shows the plan and stops unless `--yes`, matching `uninstall`, `uninit` and +/// `upgrade`. A command that changes a repository's agent configuration without showing +/// its work first is one people learn not to run. +pub fn install(plan: &Plan, yes: bool, json: bool) -> Result<()> { + if json { + return emit_json(plan); + } + println!("INSTALL {}", plan.root); + + if plan.steps.is_empty() { + // Guessing which agent is present from a directory name that might mean + // anything is worse than handing over the block and letting a human place it. + println!("\nFound no agent I recognise here."); + println!("Nothing was written. Add this to whatever instruction file your tool reads:\n"); + for line in plan.instruction_block.iter().flat_map(|b| b.lines()) { + println!(" {line}"); + } + return Ok(()); + } + + println!(); + for step in &plan.steps { + println!(" {}", step.agents.join(", ")); + // Detection has to be checkable, so what the claim rests on is printed with it. + println!(" detected because {}", step.evidence.join(", ")); + println!(" {}", install_action(step, plan.applied)); + } + + if !plan.detected_elsewhere.is_empty() { + println!("\n Installed on this machine but not configured in this repository,"); + println!(" so nothing was planned for them:"); + for agent in &plan.detected_elsewhere { + println!(" {agent}"); + } + } + + if plan.mcp { + println!( + "\n {}", + wrap( + "You asked for MCP. Its tool schemas are re-sent on every turn of every \ + session, where the shell-command block costs nothing until it is \ + called — see docs/integration/claude-code.md. Drop --mcp for the \ + cheaper integration.", + WIDTH, + " ", + 2, + ) + ); + } + + if !plan.has_work() && !plan.applied { + println!("\nNothing to do; everything above is already wired."); + return Ok(()); + } + if !yes { + println!("\nNothing was written. Re-run with --yes to apply."); + return Ok(()); + } + println!("\nDone. `reify uninit` removes everything written here."); + Ok(()) +} + +/// What one step will do, has done, or is not doing — as one readable phrase. +fn install_action(step: &InstallStep, applied: bool) -> String { + if step.state == InstallState::Skipped { + return match &step.problem { + Some(problem) => format!("skipping {}: {problem}", step.path), + None => format!("skipping {}", step.path), + }; + } + let what = match step.kind { + InstallKind::Mcp => format!("the MCP server entry in {}", step.path), + InstallKind::RuleFile => format!("the rule file {}", step.path), + InstallKind::Instructions => format!("the instruction block in {}", step.path), + }; + match (step.state, applied) { + (InstallState::Planned, _) => format!("will write {what}"), + (InstallState::AlreadyPresent, true) => format!("wrote {what}"), + (InstallState::AlreadyPresent, false) => format!("already has {what}"), + (InstallState::Skipped, _) => unreachable!("handled above"), + } +} + +/// `reify doctor`: should this repository use Reify at all? +/// +/// Named signals with measured values and a plain-language verdict. Deliberately not a +/// score: `docs/metrics.md` forbids printing a number that cannot be defined, and a +/// weighted blend of four heuristics tuned on four repositories is exactly that. +pub fn doctor(answer: &Diagnosis, json: bool) -> Result<()> { + if json { + return emit_json(answer); + } + println!("DOCTOR {}", answer.root); + println!(); + + let floor = doctor::floor_text(); + signal( + "scale", + &doctor::scale_text(&answer.scale), + if answer.verdict == Verdict::TooSmall { + format!("below the {floor} floor") + } else { + format!("above the {floor} floor") + }, + ); + + // Below the floor the other signals were never computed, and saying why is more + // useful than printing three lines of zeroes. + if answer.verdict == Verdict::TooSmall { + verdict_line(answer); + return Ok(()); + } + + match &answer.vocabulary { + Some(v) => signal( + "vocabulary", + &format!( + "{} of focused commits name a file they changed", + doctor::percent(v.locality) + ), + format!("{} of {} commits", v.commits_local, v.commits_considered), + ), + None => signal( + "vocabulary", + "not measurable without git history", + String::new(), + ), + } + match &answer.history { + Some(h) => { + signal( + "history", + &format!( + "{} commits, {} focused enough to attribute", + h.commits_read, + doctor::percent(h.focus) + ), + format!("median commit changes {} file(s)", h.median_files_changed), + ); + // Said only when it is not the case. On all five repositories this was + // calibrated against it sat at 100%, so printing it always would be noise. + if h.usable_share < 0.9 { + signal( + "", + &format!( + "only {} carry a subject worth reading", + doctor::percent(h.usable_share) + ), + String::new(), + ); + } + } + None => signal("history", "could not be read", String::new()), + } + signal( + "documents", + &format!( + "{} document(s) only Reify can read", + answer.documents.unreadable_by_grep + ), + answer.documents.examples.join(", "), + ); + + verdict_line(answer); + Ok(()) +} + +/// One measured signal: name, value, and the note that puts it in context. +fn signal(name: &str, value: &str, note: String) { + let name = if colours_wanted() { + format!("{:<12}", name.bold()) + } else { + format!("{name:<12}") + }; + if note.is_empty() { + println!(" {name}{value}"); + } else if colours_wanted() { + println!(" {name}{value:<48}{}", note.dimmed()); + } else { + println!(" {name}{value:<48}{note}"); + } +} + +fn verdict_line(answer: &Diagnosis) { + let text = answer.verdict.as_str(); + let painted = if colours_wanted() { + match answer.verdict { + Verdict::LikelyWorthIt => text.green().bold().to_string(), + Verdict::TooSmall | Verdict::UnlikelyToHelp => text.red().bold().to_string(), + Verdict::Marginal => text.yellow().bold().to_string(), + } + } else { + text.to_string() + }; + // The verdict word is part of the first wrapped line, so the wrap has to know how + // wide it is — measured on the unpainted text, since colour codes take no columns. + let lead = format!(" {text} — "); + println!( + "\n {painted} — {}", + wrap(&answer.reason, WIDTH, " ", lead.chars().count()) + ); + + if !answer.what_would_change_it.is_empty() { + println!("\n What would change this:"); + for item in &answer.what_would_change_it { + println!(" - {}", wrap(item, WIDTH, " ", 6)); + } + } + if let Some(c) = doctor::comparable(answer.verdict) { + // Named by role rather than by favourability: for a yes the useful comparison + // is the repository where Reify did worst, and for a no it is the one where it + // did best. Calling both "least favourable" would be wrong half the time. + let role = match answer.verdict { + Verdict::UnlikelyToHelp => "did best", + _ => "did worst", + }; + println!( + "\n {}", + wrap( + &format!( + "For comparison, the measured repository where Reify {role}: {}, \ + where {}. See {}.", + c.name, c.outcome, c.report + ), + WIDTH, + " ", + 2, + ) + ); + } + // The verdict is a heuristic over four repositories, and a reader could otherwise + // mistake it for a measurement of theirs. The cost is stated for the same reason + // the answer is: so nobody has to guess what running it will take. + println!( + "\n {}", + wrap( + &format!( + "This is a heuristic fitted to four measured repositories, not a \ + measurement of this one — `reify-bench` measures this one. Read the \ + working tree and {} in {:.1}s; no index needed, and none was used.", + if answer.git_repository { + "the newest 1000 commits" + } else { + "no history" + }, + answer.elapsed_ms as f64 / 1000.0, + ), + WIDTH, + " ", + 2, + ) + ); +} + +/// Terminal width the doctor output is wrapped to. +/// +/// Fixed rather than read from the terminal: the verdict is the one paragraph that must +/// be readable, and it must read the same in a pipe, a CI log and a screenshot. +const WIDTH: usize = 78; + +/// Wrap `text` to `width`, indenting continuation lines by `indent`. +/// +/// `first_column` is how far into the line the caller has already printed, so a verdict +/// word or a bullet marker is counted against the first line's budget rather than +/// pushing it past the right edge. +fn wrap(text: &str, width: usize, indent: &str, first_column: usize) -> String { + let mut out = String::new(); + let mut column = first_column; + let mut fresh = true; + for word in text.split_whitespace() { + if !fresh && column + 1 + word.chars().count() > width { + out.push('\n'); + out.push_str(indent); + column = indent.chars().count(); + } else if !fresh { + out.push(' '); + column += 1; + } + out.push_str(word); + column += word.chars().count(); + fresh = false; + } + out +} + pub fn concepts(overview: &ConceptOverview, json: bool) -> Result<()> { if json { return emit_json(overview); @@ -858,6 +1216,27 @@ mod tests { } } + #[test] + fn a_sections_status_is_hoisted_only_when_every_row_shares_it() { + // Hoisting a status that is not actually shared would attest to a footing the + // rows do not have, which is the one thing this renderer may never do. + assert_eq!( + shared_status(&[Status::Confirmed, Status::Confirmed], |s| *s), + Some(Status::Confirmed), + "a uniform section states its status once" + ); + assert_eq!( + shared_status(&[Status::Confirmed, Status::Inferred], |s| *s), + None, + "a mixed section must keep its per-row badges" + ); + assert_eq!( + shared_status(&[] as &[Status], |s| *s), + None, + "an empty section has no status to hoist" + ); + } + #[test] fn conflicted_is_visually_distinct_from_confirmed() { assert_ne!(tag(Status::Conflicted), tag(Status::Confirmed)); diff --git a/crates/reify-cli/src/selfmanage.rs b/crates/reify-cli/src/selfmanage.rs index f10f4e4..81b8088 100644 --- a/crates/reify-cli/src/selfmanage.rs +++ b/crates/reify-cli/src/selfmanage.rs @@ -15,6 +15,8 @@ use std::process::Command; const REPO: &str = "lambiengcode/reify"; const CURRENT: &str = env!("CARGO_PKG_VERSION"); +/// The binary's filename inside a release archive. +const BINARY: &str = if cfg!(windows) { "reify.exe" } else { "reify" }; /// The release target this build can upgrade to, or why it cannot. fn release_target() -> Result<&'static str> { @@ -23,6 +25,7 @@ fn release_target() -> Result<&'static str> { ("macos", "x86_64") => Ok("x86_64-apple-darwin"), ("linux", "aarch64") => Ok("aarch64-unknown-linux-gnu"), ("linux", "x86_64") => Ok("x86_64-unknown-linux-gnu"), + ("windows", "x86_64") => Ok("x86_64-pc-windows-msvc"), (os, arch) => bail!( "no prebuilt binary for {os}/{arch}; upgrade from source instead:\n \ cargo install --git https://github.com/{REPO} reify-cli" @@ -164,16 +167,37 @@ fn install_release(tag: &str, target: &str, stage: &Path, exe: &Path) -> Result< .context("running tar")?; anyhow::ensure!(status.success(), "tar could not unpack the release"); - let fresh = stage.join(&name).join("reify"); + let fresh = stage.join(&name).join(BINARY); anyhow::ensure!(fresh.is_file(), "the release archive holds no reify binary"); #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; std::fs::set_permissions(&fresh, std::fs::Permissions::from_mode(0o755))?; } - std::fs::rename(&fresh, exe) - .with_context(|| format!("replacing {} (is it writable?)", exe.display()))?; - Ok(()) + // Windows holds an open handle on the running image, so the new binary cannot be + // renamed over it — but the running one *can* be renamed aside, which frees the + // path. The displaced file is left for the next run to clear: deleting it while + // it is still mapped fails, and failing an upgrade over housekeeping would be + // worse than one stale file. + #[cfg(windows)] + { + let displaced = exe.with_extension("old"); + let _ = std::fs::remove_file(&displaced); + std::fs::rename(exe, &displaced) + .with_context(|| format!("moving {} aside", exe.display()))?; + if let Err(err) = std::fs::rename(&fresh, exe) { + // Put the working binary back rather than leaving the user with none. + let _ = std::fs::rename(&displaced, exe); + return Err(err).with_context(|| format!("replacing {}", exe.display())); + } + return Ok(()); + } + #[cfg(not(windows))] + { + std::fs::rename(&fresh, exe) + .with_context(|| format!("replacing {} (is it writable?)", exe.display()))?; + Ok(()) + } } fn hex(bytes: &[u8]) -> String { @@ -201,25 +225,58 @@ pub fn uninstall(yes: bool) -> Result<()> { Ok(()) } -/// `reify uninit [--yes]`: remove this repository's store and instruction block. +/// `reify uninit [--yes]`: remove this repository's store and everything +/// `reify install` wrote. +/// +/// The removal targets are derived from the same table `install` plans from, so an agent +/// cannot be added to one without appearing in the other — otherwise `install` quietly +/// leaves orphans that only turn up when somebody wonders why their agent still mentions +/// a tool they removed. pub fn uninit(root: &Path, yes: bool) -> Result<()> { let store = root.join(reify::index::REIFY_DIR); let mut planned: Vec = Vec::new(); - if store.is_dir() { - planned.push(format!("remove {}", store.display())); - } - let mut instruction_files: Vec = Vec::new(); - for name in ["AGENTS.md", "CLAUDE.md"] { - let path = root.join(name); - if let Ok(text) = std::fs::read_to_string(&path) { - if text.contains(crate::AGENT_INSTRUCTIONS) { - planned.push(format!("strip the Reify instruction block from {name}")); - instruction_files.push(path); + let mut edits: Vec = Vec::new(); + + for (rel, kind) in crate::install::removable_targets() { + let path = root.join(&rel); + let Ok(text) = std::fs::read_to_string(&path) else { + continue; + }; + match kind { + crate::install::Kind::Mcp => match crate::install::without_mcp_entry(&text) { + Ok(Some(stripped)) => { + planned.push(format!("remove the reify server entry from {rel}")); + edits.push(Removal::Rewrite(path, stripped)); + } + Ok(None) => {} + // An unparsable config is left exactly as it is, on the way out as much + // as on the way in. + Err(e) => planned.push(format!("leave {rel} alone ({e})")), + }, + _ if !text.contains(crate::AGENT_INSTRUCTIONS.trim()) => {} + crate::install::Kind::RuleFile => { + planned.push(format!("remove {rel}")); + edits.push(Removal::Delete(path)); + } + crate::install::Kind::Instructions => { + let stripped = strip_block(&text); + if stripped.trim().is_empty() { + // Nothing but our own block was ever in it. + planned.push(format!("remove {rel}")); + edits.push(Removal::Delete(path)); + } else { + planned.push(format!("strip the Reify instruction block from {rel}")); + edits.push(Removal::Rewrite(path, stripped)); + } } } } + + if store.is_dir() { + planned.push(format!("remove {}", store.display())); + } if planned.is_empty() { - println!("nothing to remove: no `.reify/` store or instruction block here"); + println!("nothing to remove: no `.reify/` store or Reify integration here"); return Ok(()); } for step in &planned { @@ -230,10 +287,13 @@ pub fn uninit(root: &Path, yes: bool) -> Result<()> { println!("Nothing was removed. Re-run with --yes to apply."); return Ok(()); } - for path in instruction_files { - let text = std::fs::read_to_string(&path)?; - std::fs::write(&path, text.replace(crate::AGENT_INSTRUCTIONS, "")) - .with_context(|| format!("rewriting {}", path.display()))?; + for edit in edits { + match edit { + Removal::Rewrite(path, text) => std::fs::write(&path, text) + .with_context(|| format!("rewriting {}", path.display()))?, + Removal::Delete(path) => std::fs::remove_file(&path) + .with_context(|| format!("removing {}", path.display()))?, + } } if store.is_dir() { std::fs::remove_dir_all(&store).with_context(|| format!("removing {}", store.display()))?; @@ -242,6 +302,29 @@ pub fn uninit(root: &Path, yes: bool) -> Result<()> { Ok(()) } +enum Removal { + Rewrite(PathBuf, String), + Delete(PathBuf), +} + +/// Take our block out of a file somebody else also writes to. +/// +/// The full constant is tried first, and it carries the newline that `install` and +/// `init` push in front of it — so removing it restores the file byte for byte rather +/// than leaving the blank lines the block was separated by. The trimmed form is the +/// fallback, for a file where somebody pasted the block by hand. +fn strip_block(text: &str) -> String { + for block in [crate::AGENT_INSTRUCTIONS, crate::AGENT_INSTRUCTIONS.trim()] { + if let Some(at) = text.find(block) { + let mut out = String::with_capacity(text.len()); + out.push_str(&text[..at]); + out.push_str(&text[at + block.len()..]); + return out; + } + } + text.to_string() +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/reify/src/context.rs b/crates/reify/src/context.rs index 9574037..40081e1 100644 --- a/crates/reify/src/context.rs +++ b/crates/reify/src/context.rs @@ -44,6 +44,15 @@ const MIN_SCORE: f32 = 0.02; const LEXICAL_SEEDS: usize = 60; /// Cap on the reading plan. const MAX_NEXT_READS: usize = 6; +/// Symbol slots any one file may claim. +/// +/// Relevance spreads along edges, so every member of a file that matched loosely +/// arrives holding a plausible score. Without this cap a single such file takes the +/// whole symbol budget: measured on Medusa, one HTTP router and one arithmetic helper +/// between them held 13 of 20 slots for a task about discounts, and the promotion +/// service that actually had to change ranked eighteenth. Four is enough to show that +/// several signals agree on a file, and low enough to leave room for five others. +const MAX_SYMBOLS_PER_FILE: usize = 4; /// Rough tokens per line of source, for estimating the cost of a recommended read. const TOKENS_PER_LINE: u32 = 10; @@ -53,6 +62,12 @@ const PATH_AFFINITY_WEIGHT: f32 = 1.0; /// Floor of the question-coverage factor in seed scoring. const COVERAGE_FLOOR: f32 = 0.35; +/// Score a test, fixture or mock gives up to the implementation it exercises. +/// +/// Half, not all: enough to put the source above its own test in every ordering +/// measured, small enough that a task genuinely about a test still reaches it. +const TEST_PATH_PENALTY: f32 = 0.5; + /// Concepts whose other surface forms get searched too. const CONCEPT_EXPANSIONS: usize = 3; @@ -215,6 +230,9 @@ pub struct RankWeights { pub fanout_symbols: usize, /// Share of a file's score its symbols inherit through fan-out. pub file_to_symbol: f32, + /// Score a test, fixture or mock gives up to the implementation it exercises; + /// zero disables the penalty, one hides tests entirely. + pub test_path_penalty: f32, } impl Default for RankWeights { @@ -230,6 +248,7 @@ impl Default for RankWeights { file_fanout: FILE_FANOUT, fanout_symbols: FILE_FANOUT_SYMBOLS, file_to_symbol: FILE_TO_SYMBOL, + test_path_penalty: TEST_PATH_PENALTY, } } } @@ -714,10 +733,16 @@ fn rank(store: &Store, task: &str, weights: &RankWeights) -> Result> let mut out: Vec = scores .into_iter() .filter_map(|(id, (score, reason))| { - nodes.remove(&id).map(|node| Scored { - node, - score, - reason, + nodes.remove(&id).map(|node| { + // Applied here, once, rather than at seed time: a test is reached far + // more often by spreading from the code it exercises than by matching + // the task itself, so penalising only the seeds would miss most of them. + let score = score * test_path_factor(&node, weights.test_path_penalty); + Scored { + node, + score, + reason, + } }) }) .filter(|s| s.score >= MIN_SCORE) @@ -786,6 +811,57 @@ fn stem_match(candidate: &str, asked: &str) -> bool { crate::concepts::same_word(candidate, asked) } +/// How much of its score a node keeps for living in a test, fixture or mock. +/// +/// Tests are excellent evidence about *which* code matters and poor evidence about +/// *what to change*: a test names the domain vocabulary densely, so it scores well, +/// and then tells a reader nothing they can edit. Measured on Medusa, a promotion +/// spec and its fixture both outranked the promotion service they exercise. +/// +/// A penalty rather than an exclusion, deliberately. "Fix the failing test for X" is a +/// real task, and a reproduction is genuinely the right place to start; the test +/// should lose to its implementation, not disappear behind it. +fn test_path_factor(node: &Node, penalty: f32) -> f32 { + let Some(path) = &node.path else { + return 1.0; + }; + if is_test_path(path) { + (1.0 - penalty).clamp(0.0, 1.0) + } else { + 1.0 + } +} + +/// Whether a path is test, fixture or mock material. +/// +/// Segment-wise rather than substring, so `src/contest/` and `src/latest/` are not +/// mistaken for tests. Covers the conventions the indexed languages actually use. +fn is_test_path(path: &str) -> bool { + path.split('/').any(|segment| { + let s = segment.trim_matches('_').to_ascii_lowercase(); + let stem = s.split('.').next().unwrap_or(""); + matches!( + s.as_str(), + "test" + | "tests" + | "testing" + | "spec" + | "specs" + | "fixture" + | "fixtures" + | "mock" + | "mocks" + | "e2e" + | "testdata" + | "integration-tests" + ) || stem.ends_with("_test") + || stem.ends_with("_spec") + || stem.starts_with("test_") + || s.contains(".test.") + || s.contains(".spec.") + }) +} + /// Edge kinds relevance travels along. History edges are excluded from the general /// spread and reached only through the file a selected symbol lives in, because /// "changed in the same commit" is far weaker evidence than "calls". @@ -1026,12 +1102,23 @@ fn seed_weight(kind: NodeKind) -> f32 { } } +/// The file a symbol lives in, for the per-file cap. `None` for everything else, +/// because concepts, rules and documents are already capped by count and by share. +fn symbol_file(item: &Scored) -> Option { + match item.node.kind { + NodeKind::Symbol => item.node.path.clone(), + _ => None, + } +} + /// Fill the budget, best value per token first. /// -/// Two deliberate asymmetries. A `CONFLICTED` node is admitted regardless of budget, +/// Three deliberate asymmetries. A `CONFLICTED` node is admitted regardless of budget, /// because budget pressure may drop useful context but must never drop a known -/// contradiction. And the single best concept and document are reserved, so a -/// symbol-heavy result never crowds out the two things that explain it. +/// contradiction. The single best concept and document are reserved, so a +/// symbol-heavy result never crowds out the two things that explain it. And no single +/// file may claim more than [`MAX_SYMBOLS_PER_FILE`] of the symbol slots, so one +/// loosely-matched file cannot spend the window on its own members. fn select(scored: &[Scored], budget: u32) -> Vec { let mut chosen: Vec = Vec::new(); let mut spent = 0u32; @@ -1072,39 +1159,62 @@ fn select(scored: &[Scored], budget: u32) -> Vec { // Count what the reserved picks already used, so the caps bind on the total. let mut count_by_kind: HashMap = HashMap::new(); let mut tokens_by_kind: HashMap = HashMap::new(); + let mut symbols_by_file: HashMap = HashMap::new(); for item in &chosen { *count_by_kind.entry(item.node.kind).or_insert(0) += 1; *tokens_by_kind.entry(item.node.kind).or_insert(0) += item.node.tokens; + if let Some(path) = symbol_file(item) { + *symbols_by_file.entry(path).or_insert(0) += 1; + } } - // Best value per token, subject to both caps. + // Best value per token, subject to every cap. for item in &remaining { let kind = item.node.kind; let token_cap = (budget as f32 * budget_share(kind)) as u32; let count = *count_by_kind.get(&kind).unwrap_or(&0); let used = *tokens_by_kind.get(&kind).unwrap_or(&0); + let file = symbol_file(item); if spent + item.node.tokens > budget || count >= max_items(kind) || used + item.node.tokens > token_cap + || file.as_ref().is_some_and(|p| { + symbols_by_file + .get(p) + .is_some_and(|n| *n >= MAX_SYMBOLS_PER_FILE) + }) { continue; } *count_by_kind.entry(kind).or_insert(0) += 1; *tokens_by_kind.entry(kind).or_insert(0) += item.node.tokens; + if let Some(path) = file { + *symbols_by_file.entry(path).or_insert(0) += 1; + } admit(item, &mut chosen, &mut spent, &mut taken); } // Spend budget the shares left unused on more code, which is what a change needs - // — but never past the count cap, or the answer becomes a directory listing. + // — but never past the count cap, or the answer becomes a directory listing, and + // never past the per-file cap, or it becomes one file's table of contents. for item in &remaining { let kind = item.node.kind; + let file = symbol_file(item); if taken.contains(&item.node.id) || kind != NodeKind::Symbol || *count_by_kind.get(&kind).unwrap_or(&0) >= max_items(kind) || spent + item.node.tokens > budget + || file.as_ref().is_some_and(|p| { + symbols_by_file + .get(p) + .is_some_and(|n| *n >= MAX_SYMBOLS_PER_FILE) + }) { continue; } *count_by_kind.entry(kind).or_insert(0) += 1; + if let Some(path) = file { + *symbols_by_file.entry(path).or_insert(0) += 1; + } admit(item, &mut chosen, &mut spent, &mut taken); } @@ -1121,6 +1231,12 @@ fn select(scored: &[Scored], budget: u32) -> Vec { /// A span that does not fit is skipped rather than truncated, and a cheaper span /// further down the list may still be taken — a 400-line class must not block six /// precise 20-line methods. +/// +/// Spans are drawn one file at a time in rounds rather than by draining each file in +/// turn. Draining spent the whole plan on the top file: measured on Medusa, two tasks +/// in three produced six entries naming a single file, so the plan pointed at one +/// place and called it a reading list. A round-robin gives every ranked file a span +/// before any file gets a second, which is what makes the plan a *plan*. fn reading_plan(selected: &[Scored], limit: usize, budget: u32, cutoff: f32) -> Vec { // Ordered by *file aggregate* rather than by individual symbol score: three // moderately-scored symbols in one file are stronger evidence about that file @@ -1129,17 +1245,22 @@ fn reading_plan(selected: &[Scored], limit: usize, budget: u32, cutoff: f32) -> let mut plan: Vec = Vec::new(); let mut spent = 0u32; - for (_, items) in &ranked { - for item in items { + let deepest = ranked + .iter() + .map(|(_, items)| items.len()) + .max() + .unwrap_or(0); + for round in 0..deepest { + for (path, items) in &ranked { if plan.len() >= limit { return plan; } + let Some(item) = items.get(round) else { + continue; + }; if item.node.line_start == 0 { continue; } - let Some(path) = &item.node.path else { - continue; - }; let span = item.node.line_end.saturating_sub(item.node.line_start) + 1; let cost = span * TOKENS_PER_LINE; if spent + cost > budget { @@ -1996,4 +2117,138 @@ class DiscountPolicy: assert!(cost < 4_000, "context rendered to {cost} tokens"); let _ = fs::remove_dir_all(&root); } + + /// One symbol per line of a file, scored so that `flood` outranks everything. + /// + /// Built by hand rather than indexed from a fixture: the pathology only appears + /// when one file holds more symbols than the whole cap, which no fixture small + /// enough to keep in the repository would reproduce. + #[cfg(test)] + fn crowded(flood_file: &str, flood: usize, others: usize) -> Vec { + let mut out = Vec::new(); + let mut make = |id: i64, path: &str, score: f32| { + out.push(Scored { + node: crate::model::Node { + id, + uid: format!("{path}#{id}"), + kind: NodeKind::Symbol, + name: format!("member_{id}"), + path: Some(path.into()), + line_start: id as u32 * 10, + line_end: id as u32 * 10 + 5, + lang: None, + status: Status::Confirmed, + confidence: 1.0, + tokens: 10, + data: serde_json::Value::Null, + }, + score, + reason: "test".into(), + }); + }; + for i in 0..flood { + make(i as i64 + 1, flood_file, 1.0 - i as f32 * 0.001); + } + for i in 0..others { + make(1_000 + i as i64, &format!("src/other_{i}.rs"), 0.5); + } + out + } + + #[test] + fn no_single_file_can_claim_every_symbol_slot() { + // Relevance spreads along edges, so every member of a loosely-matched file + // arrives holding a plausible score. Measured on Medusa before this cap, one + // HTTP router held 8 of 20 slots for a task about discounts. + let scored = crowded("src/router.rs", 12, 8); + let chosen = select(&scored, 4_000); + let mut per_file: HashMap<&str, usize> = HashMap::new(); + for item in &chosen { + if let Some(path) = &item.node.path { + *per_file.entry(path.as_str()).or_insert(0) += 1; + } + } + assert_eq!( + per_file.get("src/router.rs").copied().unwrap_or(0), + MAX_SYMBOLS_PER_FILE, + "the flooding file should be held to the cap" + ); + assert!( + per_file.len() > 1, + "capping the flood must leave room for other files, saw {per_file:?}" + ); + } + + #[test] + fn the_reading_plan_visits_distinct_files_before_revisiting_one() { + // A plan whose six entries all name one file points at one place and calls + // itself a reading list. Every ranked file earns a span before any earns a + // second, so the plan is a plan. + let scored = crowded("src/router.rs", 12, 8); + let chosen = select(&scored, 4_000); + let plan = reading_plan(&chosen, MAX_NEXT_READS, 40_000, 0.0); + assert_eq!(plan.len(), MAX_NEXT_READS, "the plan should be full"); + let distinct: BTreeSet<&str> = plan.iter().map(|r| r.path.as_str()).collect(); + assert_eq!( + distinct.len(), + plan.len(), + "plan named {} files across {} entries: {:?}", + distinct.len(), + plan.len(), + plan.iter().map(|r| &r.path).collect::>() + ); + } + + #[test] + fn an_implementation_outranks_the_test_that_exercises_it() { + // A test names the task's vocabulary as densely as the code it exercises, so + // it arrives scoring at least as well, and a reader cannot edit it to change + // the behaviour. It should still be reachable — hence a penalty, not an + // exclusion: at the default weight the test keeps half its score, not none. + let symbol = |path: &str| crate::model::Node { + id: 1, + uid: path.into(), + kind: NodeKind::Symbol, + name: "requires_approval".into(), + path: Some(path.into()), + line_start: 1, + line_end: 2, + lang: None, + status: Status::Confirmed, + confidence: 1.0, + tokens: 10, + data: serde_json::Value::Null, + }; + let penalty = RankWeights::default().test_path_penalty; + let source = test_path_factor(&symbol("app/order.py"), penalty); + let test = test_path_factor(&symbol("app/test_rules.py"), penalty); + assert!( + test < source, + "a test scoring equally with its implementation must lose to it: {test} vs {source}" + ); + assert!(test > 0.0, "a penalty must not become an exclusion"); + } + + #[test] + fn test_detection_matches_path_segments_not_substrings() { + // `contest` and `latest` contain "test"; neither is one. + for path in [ + "app/tests/test_order.py", + "src/order.test.ts", + "pkg/order_test.go", + "integration-tests/__fixtures__/promotion/index.ts", + "spec/models/order_spec.rb", + "src/__mocks__/stripe.ts", + ] { + assert!(is_test_path(path), "{path} should read as test material"); + } + for path in [ + "app/contest/entry.py", + "src/latest/version.ts", + "src/protest/handler.go", + "app/order.py", + ] { + assert!(!is_test_path(path), "{path} is not test material"); + } + } } diff --git a/crates/reify/src/doctor.rs b/crates/reify/src/doctor.rs new file mode 100644 index 0000000..9788dc4 --- /dev/null +++ b/crates/reify/src/doctor.rs @@ -0,0 +1,884 @@ +//! Should this repository use Reify at all? +//! +//! A tool that always recommends itself is worthless, and this project has already +//! published a repository — `benchmarks/REPORT-medusa.md` — where Reify ties grep. The +//! most valuable answer this module can give is a confident *no*, because that is what +//! makes the *yes* worth anything. +//! +//! # Where the signals come from +//! +//! Four repositories were measured end to end (`benchmarks/REPORT*.md`), and two +//! hypotheses were tested against them. Both failed: +//! +//! - **Size.** OFBiz has 1,364 code files and shows the largest margin over grep +//! (70% against 12%); Medusa has 11,821 and shows none (18% against 18%). +//! - **Declared vocabulary.** OFBiz declares almost nothing and still wins. +//! +//! Two signals *do* fit all four outcomes, and each explains a different way the tool +//! fails. Measured over the newest [`MAX_COMMITS`] commits of each repository: +//! +//! | | grep margin | commit focus | subject→path | +//! |---|---:|---:|---:| +//! | OFBiz | +58 | 0.96 | 0.80 | +//! | ERPNext | +48 | 0.98 | 0.85 | +//! | OpenMRS | +9 | 0.98 | **0.48** | +//! | Medusa | 0 | **0.84** | 0.79 | +//! +//! **Commit focus** is the share of commits touching few enough files that their +//! subject says something about them. Medusa is the only measured repository where it +//! falls away, and it is the only one where Reify did not win. That is not a +//! coincidence: Reify attaches a commit's vocabulary to every file it touched, so a +//! history of sweeping squashed merges smears each subject across the tree. The same +//! assumption is already load-bearing in [`crate::gitlog::History::co_changes`], which +//! skips commits touching more than [`FOCUSED_COMMIT_FILES`] files because "a sweeping +//! commit couples everything to everything and tells us nothing". +//! +//! **Subject→path locality** is the share of those focused commits whose subject shares +//! a word with a path it changed — the direct test of whether the words a change is +//! described in point at the code it touches. OpenMRS is the one measured repository +//! where it falls away, and it is the one whose margin over grep was small. +//! +//! Between them the two account for every measured outcome, without either one having +//! to explain a case it does not fit. +//! +//! # What was tried and dropped +//! +//! Corpus-wide overlap between commit vocabulary and path vocabulary — the obvious +//! reading of "history and file naming speak the same vocabulary" — was measured first +//! and **inverts**: Medusa scores 0.43 against OFBiz's 0.38. Pooling every subject into +//! one bag throws away the attribution that makes the signal mean anything, so it is +//! not computed here. Neither is a 0-100 suitability score: `docs/metrics.md` forbids +//! printing a number that cannot be defined, and a weighted blend of heuristics tuned on +//! four repositories is exactly that. +//! +//! # What this is not +//! +//! A heuristic fitted to four repositories, not a measurement of yours. `reify-bench` +//! measures a specific repository; this reads one in about a second. + +use anyhow::Result; +use std::collections::BTreeSet; +use std::path::Path; + +use crate::concepts::{meaningful_words, stem}; +use crate::discover::{self, Discovery}; +use crate::gitlog; +use crate::model::Lang; + +pub const SCHEMA: &str = "reify.doctor/1"; + +/// The line count below which the README's FAQ already says not to bother. +/// +/// Deliberately the number the documentation publishes — "Under roughly 20k LOC Reify +/// buys you nothing a grep and a scroll wheel don't" — rather than a second, quieter +/// threshold that contradicts it. +pub const LINES_FLOOR: u64 = 20_000; + +/// How far back history is read. +/// +/// Bounded because a doctor that takes a minute does not get run. One `git log +/// --name-only` of a thousand commits answers in well under a second even on a +/// repository with a hundred thousand of them. +pub const MAX_COMMITS: usize = 1_000; + +/// A commit touching more files than this tells you nothing about any of them. +/// +/// The same threshold [`crate::gitlog::History::co_changes`] already applies, for the +/// same reason, so the two agree about what a meaningful commit is. +pub const FOCUSED_COMMIT_FILES: usize = 20; + +/// Share of commits that must be focused for history to be usable evidence. +/// +/// The three measured repositories where Reify won all sit at 0.96 or above; Medusa, +/// where it tied, sits at 0.84. +const FOCUS_OK: f32 = 0.90; + +/// Share of focused commits whose subject must name something in a path it changed. +/// +/// OFBiz 0.80, ERPNext 0.85 and Medusa 0.79 clear it; OpenMRS, whose margin over grep +/// was 9 points rather than 48, sits at 0.48. +const LOCALITY_STRONG: f32 = 0.70; + +/// Enough commits that the shares above can be told apart from their thresholds. +/// +/// Not a round number picked for feel. Medusa — the measured repository that fails the +/// focus test — sits at 0.84. A 95% Wilson interval around 0.84 lies entirely below +/// [`FOCUS_OK`] at n = 200 (upper bound 0.88) but straddles it at n = 50 (upper bound +/// 0.92). Below this a `no` would be an artefact of the sample size, so a short history +/// is reported as short rather than condemned. +const MIN_COMMITS: usize = 200; + +/// How much of the repository will be indexed at all. +#[derive(Debug, Clone, serde::Serialize)] +pub struct Scale { + /// Files Reify would index. + pub indexable_files: usize, + /// Of those, files in a language Reify parses as code. + pub code_files: usize, + /// Lines across every indexable file. Includes blanks and comments. + pub lines: u64, +} + +/// Do the words a change is described in point at the code it touches? +#[derive(Debug, Clone, serde::Serialize)] +pub struct Vocabulary { + /// Focused commits with a usable subject — the denominator. + pub commits_considered: usize, + /// Of those, commits whose subject shares a word with a path they changed. + pub commits_local: usize, + /// `commits_local / commits_considered`. + pub locality: f32, +} + +/// Is the history attributable, or is every subject smeared across the tree? +#[derive(Debug, Clone, serde::Serialize)] +pub struct HistorySignal { + /// Commits read, bounded by [`MAX_COMMITS`]. + pub commits_read: usize, + /// Whether the walk stopped at that bound rather than at the root commit. + pub truncated: bool, + /// Commits whose subject carries at least two meaningful words. + pub usable_subjects: usize, + /// `usable_subjects / commits_read`. Sat at 1.0 on all five repositories this was + /// calibrated against, so it discriminates nothing there — but a history of `wip` + /// and version bumps is real, and this is what would catch it. + pub usable_share: f32, + /// Commits touching at most [`FOCUSED_COMMIT_FILES`] files. + pub focused_commits: usize, + /// `focused_commits / commits_read`. + pub focus: f32, + /// Files changed by the median commit. + pub median_files_changed: usize, +} + +/// Documents whose text a grep cannot reach. +#[derive(Debug, Clone, serde::Serialize)] +pub struct Documents { + /// Files in a format Reify converts and an agent cannot read. + pub unreadable_by_grep: usize, + /// A few examples, so the claim can be checked. + pub examples: Vec, +} + +/// The answer. +#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Serialize)] +#[serde(rename_all = "snake_case")] +pub enum Verdict { + /// Under the line floor. Nothing else was worth measuring. + TooSmall, + /// The signals that separated the measured repositories are present here. + LikelyWorthIt, + /// Mixed. Worth measuring rather than guessing. + Marginal, + /// This repository has the shape of the one where Reify tied grep. + UnlikelyToHelp, +} + +impl Verdict { + pub fn as_str(self) -> &'static str { + match self { + Verdict::TooSmall => "TOO SMALL", + Verdict::LikelyWorthIt => "LIKELY WORTH IT", + Verdict::Marginal => "MARGINAL", + Verdict::UnlikelyToHelp => "UNLIKELY TO HELP", + } + } +} + +/// One measured repository, named so a reader can check the comparison. +pub struct Comparable { + pub name: &'static str, + pub outcome: &'static str, + pub report: &'static str, +} + +/// The measured repository where Reify did worst. +/// +/// Pointed at every favourable verdict on purpose: naming a repository where Reify won +/// proves nothing to someone deciding whether to spend an afternoon on it. +pub const MEDUSA: Comparable = Comparable { + name: "Medusa", + outcome: "Reify tied grep — 18% of tasks each", + report: "benchmarks/REPORT-medusa.md", +}; + +/// The measured repository where Reify did best. +pub const OFBIZ: Comparable = Comparable { + name: "OFBiz", + outcome: "Reify reached a changed file on 70% of tasks against grep's 12%", + report: "benchmarks/REPORT-ofbiz.md", +}; + +#[derive(Debug, Clone, serde::Serialize)] +pub struct Diagnosis { + pub schema: &'static str, + pub root: String, + pub scale: Scale, + pub git_repository: bool, + /// Absent below the floor, and when history cannot be read. + pub vocabulary: Option, + /// Absent below the floor, and when history cannot be read. + pub history: Option, + pub documents: Documents, + pub verdict: Verdict, + /// The one sentence carrying the verdict. + pub reason: String, + /// What would change a no or a maybe. Empty for a clear yes. + pub what_would_change_it: Vec, + /// Wall clock of the measurement itself. + pub elapsed_ms: u64, +} + +/// Formats whose text a grep cannot reach. +/// +/// The one categorical advantage: no agent greps a PDF. RTF is nominally text, but its +/// words are broken up by control words, so a grep on it misleads rather than fails — +/// which is worse. +fn unreadable_by_grep(lang: Lang) -> bool { + matches!( + lang, + Lang::Docx | Lang::Doc | Lang::Odt | Lang::Rtf | Lang::Xlsx | Lang::Pptx | Lang::Pdf + ) +} + +/// Does this subject say anything about the change? +/// +/// Two meaningful words is a low bar that "Merge pull request #123 from acme/topic", +/// "Bump version to 4.2.1" and "wip" all fail and that any sentence describing a change +/// passes. Merges are excluded outright: a merge subject names a branch, not a change. +pub fn subject_is_usable(subject: &str) -> bool { + !subject.starts_with("Merge ") && meaningful_words(subject).len() >= 2 +} + +/// Stem-folded words of a string, so `customer` and `customers` are one word. +fn stems(text: &str) -> BTreeSet { + meaningful_words(text) + .iter() + .map(|w| stem(w).to_string()) + .collect() +} + +/// Read the repository and decide. +/// +/// Reads the working tree and `git log`, never the store: the whole point is deciding +/// before committing to the tool, so the answer must not depend on having run it. It +/// also means the answer does not change once `reify index` has run — there is nothing +/// in the store this would rather use. +pub fn diagnose(root: &Path) -> Result { + let started = std::time::Instant::now(); + let found = discover::discover(root)?; + let scale = measure_scale(&found); + let documents = measure_documents(&found); + let git_repository = gitlog::is_repository(root); + + // Below the floor nothing else is worth measuring, and saying so plainly is the + // whole value of the answer. + if scale.lines < LINES_FLOOR { + return Ok(Diagnosis { + schema: SCHEMA, + root: root.display().to_string(), + reason: format!( + "{}. Under roughly {} lines, ripgrep and a scroll wheel do this job. \ + Nothing else here is worth measuring.", + scale_text(&scale), + floor_text() + ), + scale, + git_repository, + vocabulary: None, + history: None, + documents, + verdict: Verdict::TooSmall, + what_would_change_it: Vec::new(), + elapsed_ms: started.elapsed().as_millis() as u64, + }); + } + + // A repository whose history git will not read still gets an answer, with the + // signals honestly absent rather than silently defaulted. + let log = git_repository + .then(|| gitlog::history(root, MAX_COMMITS).ok()) + .flatten(); + let history = log.as_ref().map(measure_history); + let vocabulary = log.as_ref().map(measure_vocabulary); + + let (verdict, reason, what_would_change_it) = + decide(vocabulary.as_ref(), history.as_ref(), &documents); + + Ok(Diagnosis { + schema: SCHEMA, + root: root.display().to_string(), + scale, + git_repository, + vocabulary, + history, + documents, + verdict, + reason, + what_would_change_it, + elapsed_ms: started.elapsed().as_millis() as u64, + }) +} + +fn measure_scale(found: &Discovery) -> Scale { + Scale { + indexable_files: found.files.len(), + code_files: found.files.iter().filter(|f| f.lang.is_code()).count(), + lines: found.files.iter().map(|f| u64::from(f.lines)).sum(), + } +} + +/// Count document formats across everything walked, indexed or not. +/// +/// Both lists are read on purpose: a `.docx` is binary, so discovery records it as +/// skipped even though indexing converts it. Counting only the indexable list would +/// report zero documents for a repository full of them. +fn measure_documents(found: &Discovery) -> Documents { + let mut examples = Vec::new(); + let mut count = 0; + let paths = found + .files + .iter() + .map(|f| f.path.as_str()) + .chain(found.skipped.iter().map(|(p, _)| p.as_str())); + for path in paths { + if unreadable_by_grep(discover::classify(path)) { + count += 1; + if examples.len() < 3 { + examples.push(path.to_string()); + } + } + } + examples.sort(); + Documents { + unreadable_by_grep: count, + examples, + } +} + +fn measure_history(log: &gitlog::History) -> HistorySignal { + let commits_read = log.commits.len(); + let usable = log + .commits + .iter() + .filter(|c| subject_is_usable(&c.subject)) + .count(); + let mut sizes: Vec = log.commits.iter().map(|c| c.files.len()).collect(); + sizes.sort_unstable(); + let focused = sizes + .iter() + .filter(|&&n| (1..=FOCUSED_COMMIT_FILES).contains(&n)) + .count(); + HistorySignal { + commits_read, + truncated: log.truncated, + usable_subjects: usable, + usable_share: share(usable, commits_read), + focused_commits: focused, + focus: share(focused, commits_read), + median_files_changed: sizes.get(sizes.len() / 2).copied().unwrap_or(0), + } +} + +/// How often a commit subject names something in a path that commit changed. +/// +/// Per commit rather than pooled across the repository. The pooled version — every +/// subject word against every path word — was measured on the same four repositories +/// and inverts, because pooling throws away the attribution that makes the question +/// mean anything: it asks whether the words appear *somewhere*, not whether they point +/// at the code that actually changed. +fn measure_vocabulary(log: &gitlog::History) -> Vocabulary { + let mut considered = 0; + let mut local = 0; + for commit in &log.commits { + if !(1..=FOCUSED_COMMIT_FILES).contains(&commit.files.len()) + || !subject_is_usable(&commit.subject) + { + continue; + } + considered += 1; + let subject = stems(&commit.subject); + if commit + .files + .iter() + .any(|path| stems(path).iter().any(|w| subject.contains(w))) + { + local += 1; + } + } + Vocabulary { + commits_considered: considered, + commits_local: local, + locality: share(local, considered), + } +} + +fn share(part: usize, whole: usize) -> f32 { + if whole == 0 { + 0.0 + } else { + part as f32 / whole as f32 + } +} + +/// Below this, a line count is printed exactly rather than abbreviated. +/// +/// Rounding to the nearest thousand is at most a 5% misstatement here and grows worse +/// the smaller the number gets: at 2 lines it is not an abbreviation, it is a wrong +/// answer. In a command whose whole job is honest measurement, that is the one thing it +/// must not do. +const ABBREVIATE_LINES_ABOVE: u64 = 10_000; + +/// A line count, abbreviated only where abbreviating is not misleading. +/// +/// Carries its own `~` when it is approximate, so no caller can mark an exact figure as +/// an estimate or an estimate as exact. +/// +/// Discovery counts lines, not statements: blanks and comments are in there. Above the +/// threshold a figure printed to the unit invites it to be read as a measurement of code +/// size, which it is not. +pub fn lines_text(lines: u64) -> String { + if lines < ABBREVIATE_LINES_ABOVE { + return lines.to_string(); + } + format!("~{}k", (lines as f64 / 1000.0).round() as u64) +} + +/// The line floor, for prose that names it. A threshold, so never marked approximate. +pub fn floor_text() -> String { + format!("{}k", LINES_FLOOR / 1000) +} + +/// `n file` or `n files`. +/// +/// Trivial, and shared rather than inlined so the signal line and the verdict sentence +/// cannot disagree about the same count. +fn count(n: usize, noun: &str) -> String { + if n == 1 { + format!("{n} {noun}") + } else { + format!("{n} {noun}s") + } +} + +/// The scale signal as one phrase, used by both the signal line and the verdict. +/// +/// One function rather than two format strings: they state the same measurement, and the +/// only reason they were ever two was that nobody had noticed the duplication yet. +pub fn scale_text(scale: &Scale) -> String { + format!( + "{}, {} lines", + count(scale.indexable_files, "indexable file"), + lines_text(scale.lines) + ) +} + +/// Pick the verdict, and say what it rests on. +/// +/// Four rules, each traceable to a measured repository: +/// +/// - a history of sweeping commits is the Medusa shape, the one case measured where +/// Reify won nothing; +/// - subjects that do name the code they change is the OFBiz and ERPNext shape, the two +/// large margins; +/// - subjects that do not, over an otherwise focused history, is the OpenMRS shape, +/// where Reify won by 9 points rather than 48; +/// - documents no grep can read are a categorical advantage rather than a comparative +/// one, so they are stated wherever they exist. +fn decide( + vocabulary: Option<&Vocabulary>, + history: Option<&HistorySignal>, + documents: &Documents, +) -> (Verdict, String, Vec) { + let documents_note = format!( + "{} document(s) here hold text no grep can reach, and Reify converts and \ + indexes them", + documents.unreadable_by_grep + ); + + let (Some(vocabulary), Some(history)) = (vocabulary, history) else { + let mut changes = vec![ + "`reify-bench` measures this repository directly, rather than comparing its \ + shape to four others." + .to_string(), + ]; + if documents.unreadable_by_grep == 0 { + changes.push( + "A readable git history. Reify reads commit subjects to connect a change \ + request to code, and without one the strongest signal is missing." + .to_string(), + ); + } + return ( + Verdict::Marginal, + format!( + "No readable git history, so neither measured signal can be computed here.{}", + if documents.unreadable_by_grep > 0 { + format!(" What is clear is that {documents_note}.") + } else { + String::new() + } + ), + changes, + ); + }; + + if history.commits_read < MIN_COMMITS { + return ( + Verdict::Marginal, + format!( + "Only {} commits to read. Both measured signals are shares over commits, \ + and below {MIN_COMMITS} their confidence intervals straddle the \ + thresholds — so a verdict either way would be an artefact of the sample \ + size rather than a reading of this repository.", + history.commits_read + ), + vec![ + format!("More history: at least {MIN_COMMITS} commits."), + "`reify-bench` measures this repository directly, and does not need a \ + long history to do it." + .to_string(), + ], + ); + } + + // The Medusa shape. Reify attaches a commit's vocabulary to every file it touched, + // so a history of sweeping merges spreads each subject across the tree. + if history.focus < FOCUS_OK { + let reason = format!( + "Commits here are sweeping: only {} touch few enough files for their subject \ + to say anything about them, and the median commit changes {} files. Reify \ + attaches a commit's words to every file it touched, so that history is \ + spread too thin to retrieve on. This is the shape of the one measured \ + repository where Reify tied grep.", + percent(history.focus), + history.median_files_changed + ); + if documents.unreadable_by_grep > 0 { + return ( + Verdict::Marginal, + format!("{reason} Against that, {documents_note} — which is an advantage no search tool recovers however well it is used."), + vec![ + "Nothing, for retrieval. The documents are the reason to run it here, \ + not the ranking." + .to_string(), + ], + ); + } + return ( + Verdict::UnlikelyToHelp, + reason, + vec![ + "Smaller commits, whose subject names what they changed. Squashed merges \ + of a hundred files carry no vocabulary any one of them can be found by." + .to_string(), + "Business documents in `.docx`, `.pdf`, `.xlsx` or `.pptx` committed to \ + the tree. Reify reads those and an agent cannot, whatever the history \ + looks like." + .to_string(), + ], + ); + } + + // The OFBiz and ERPNext shape: the words changes are described in name the code. + if vocabulary.locality >= LOCALITY_STRONG { + let mut reason = format!( + "{} of this repository's focused commits have a subject naming something in \ + a path they changed. That agreement between how changes are described and \ + how code is named is what separated the repositories where Reify helped \ + from the one where it did not.", + percent(vocabulary.locality) + ); + if documents.unreadable_by_grep > 0 { + reason.push_str(&format!(" On top of that, {documents_note}.")); + } + return (Verdict::LikelyWorthIt, reason, Vec::new()); + } + + // The OpenMRS shape: an attributable history whose subjects nonetheless describe + // changes in words the code does not use. Measured a modest win, not a large one. + let mut reason = format!( + "History here is attributable, but only {} of its commits have a subject naming \ + something in a path they changed. Of the four measured repositories the one \ + that looked like this beat grep by 9 points rather than 48.", + percent(vocabulary.locality) + ); + if documents.unreadable_by_grep > 0 { + reason.push_str(&format!( + " That said, {documents_note}, which is an advantage no search tool recovers." + )); + return (Verdict::LikelyWorthIt, reason, Vec::new()); + } + ( + Verdict::Marginal, + reason, + vec![ + "Commit subjects that name the thing being changed, in the words the code \ + uses for it. That is the signal Reify's retrieval is built on." + .to_string(), + "A declared glossary — `.reify/glossary.toml` — which bridges the words your \ + team uses to the identifiers the code uses." + .to_string(), + "`reify-bench` measures this repository, rather than comparing its shape to \ + four others." + .to_string(), + ], + ) +} + +pub fn percent(fraction: f32) -> String { + format!("{}%", (fraction * 100.0).round() as i64) +} + +/// The measured repository a verdict should be read against. +pub fn comparable(verdict: Verdict) -> Option { + match verdict { + Verdict::LikelyWorthIt | Verdict::Marginal => Some(MEDUSA), + Verdict::UnlikelyToHelp => Some(OFBIZ), + Verdict::TooSmall => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::fs; + + fn tmp(name: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("reify-doctor-{}-{name}", std::process::id())); + let _ = fs::remove_dir_all(&d); + fs::create_dir_all(&d).unwrap(); + d + } + + fn vocab(locality: f32) -> Vocabulary { + Vocabulary { + commits_considered: 400, + commits_local: (400.0 * locality) as usize, + locality, + } + } + + fn history(focus: f32) -> HistorySignal { + HistorySignal { + commits_read: 500, + truncated: true, + usable_subjects: 500, + usable_share: 1.0, + focused_commits: (500.0 * focus) as usize, + focus, + median_files_changed: if focus < FOCUS_OK { 4 } else { 1 }, + } + } + + fn no_documents() -> Documents { + Documents { + unreadable_by_grep: 0, + examples: Vec::new(), + } + } + + #[test] + fn a_tiny_repository_is_told_not_to_bother_and_nothing_else_is_measured() { + let d = tmp("tiny"); + fs::write(d.join("main.py"), "def f():\n return 1\n").unwrap(); + let answer = diagnose(&d).unwrap(); + assert_eq!(answer.verdict, Verdict::TooSmall); + assert!( + answer.vocabulary.is_none() && answer.history.is_none(), + "below the floor nothing else is worth measuring" + ); + assert!(answer.reason.contains("ripgrep")); + assert!( + answer.reason.starts_with("1 indexable file, 2 lines."), + "a two-line repository is reported as two lines, not as ~1k: {}", + answer.reason + ); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn the_floor_is_the_one_the_documentation_publishes() { + // A second, quieter threshold that contradicts the README would be worse than + // no threshold at all. + assert_eq!(LINES_FLOOR, 20_000); + } + + #[test] + fn the_focus_threshold_agrees_with_what_co_change_already_calls_a_sweeping_commit() { + // Two different numbers for "a commit too broad to learn from" would be two + // different definitions of the same thing. + assert_eq!(FOCUSED_COMMIT_FILES, 20); + } + + // The four measured repositories, as they were measured over their newest 500 + // commits. These are the fit; a threshold change that reclassifies one of them is + // a change of claim, not a tweak. + #[test] + fn each_measured_repository_lands_where_its_benchmark_says_it_should() { + let cases = [ + // repo, focus, locality, expected + ("ofbiz +58", 0.956, 0.803, Verdict::LikelyWorthIt), + ("erpnext +48", 0.982, 0.845, Verdict::LikelyWorthIt), + ("openmrs +9", 0.976, 0.480, Verdict::Marginal), + ("medusa +0", 0.836, 0.792, Verdict::UnlikelyToHelp), + ]; + for (name, focus, locality, expected) in cases { + let (verdict, _, _) = decide( + Some(&vocab(locality)), + Some(&history(focus)), + &no_documents(), + ); + assert_eq!(verdict, expected, "{name}"); + } + } + + #[test] + fn a_sweeping_history_is_a_no_and_says_what_would_change_it() { + let (verdict, reason, changes) = + decide(Some(&vocab(0.79)), Some(&history(0.84)), &no_documents()); + assert_eq!( + verdict, + Verdict::UnlikelyToHelp, + "strong subject vocabulary must not rescue a history it cannot be attributed to" + ); + assert!(reason.contains("tied grep")); + assert!(!changes.is_empty(), "a no must say what would change it"); + } + + #[test] + fn documents_no_grep_can_read_are_stated_wherever_they_exist() { + let documents = Documents { + unreadable_by_grep: 12, + examples: vec!["docs/spec.pdf".into()], + }; + // They lift the OpenMRS shape to a yes... + let (verdict, reason, _) = decide(Some(&vocab(0.48)), Some(&history(0.97)), &documents); + assert_eq!(verdict, Verdict::LikelyWorthIt); + assert!(reason.contains("12 document")); + + // ...and they are the reason to bother even where retrieval looks unpromising, + // but they do not turn a sweeping history into a good one. + let (verdict, reason, _) = decide(Some(&vocab(0.79)), Some(&history(0.84)), &documents); + assert_eq!(verdict, Verdict::Marginal); + assert!(reason.contains("tied grep") && reason.contains("12 document")); + } + + #[test] + fn an_unreadable_history_is_marginal_rather_than_a_guess() { + let (verdict, reason, changes) = decide(None, None, &no_documents()); + assert_eq!(verdict, Verdict::Marginal); + assert!(reason.contains("No readable git history")); + assert!(changes.iter().any(|c| c.contains("reify-bench"))); + } + + #[test] + fn too_little_history_is_admitted_rather_than_measured() { + let mut thin = history(0.5); + thin.commits_read = MIN_COMMITS - 1; + let (verdict, reason, changes) = decide(Some(&vocab(0.9)), Some(&thin), &no_documents()); + assert_eq!( + verdict, + Verdict::Marginal, + "a sweeping-looking history too short to measure is reported as short, not \ + condemned: at this sample size the interval straddles the threshold" + ); + assert!(reason.contains(&format!("{} commits", MIN_COMMITS - 1))); + assert!(changes.iter().any(|c| c.contains("More history"))); + } + + #[test] + fn a_history_of_merges_and_version_bumps_does_not_read_as_usable() { + assert!(!subject_is_usable( + "Merge pull request #123 from acme/topic" + )); + assert!(!subject_is_usable("wip")); + assert!(!subject_is_usable("v1.2.3")); + assert!(subject_is_usable( + "fix: sales order approval ignores the credit limit" + )); + } + + #[test] + fn locality_counts_a_subject_that_names_a_path_it_changed() { + let commit = |subject: &str, files: &[&str]| gitlog::Commit { + sha: "0".repeat(40), + timestamp: 0, + author: "a".into(), + subject: subject.into(), + class: gitlog::classify(subject), + files: files.iter().map(|f| f.to_string()).collect(), + }; + let log = gitlog::History { + commits: vec![ + commit("fix invoice rounding", &["app/invoice.py"]), + commit("tighten the release checklist", &["app/invoice.py"]), + // Excluded: too sweeping to attribute either way. + commit("reformat everything", &vec!["f.py"; 40]), + ], + truncated: false, + }; + let measured = measure_vocabulary(&log); + assert_eq!(measured.commits_considered, 2, "the sweep is not counted"); + assert_eq!(measured.commits_local, 1); + assert_eq!(measured.locality, 0.5); + } + + #[test] + fn a_yes_is_pointed_at_the_least_favourable_measured_repository() { + // Naming a repository where Reify won proves nothing to someone deciding + // whether to spend an afternoon on it. + assert_eq!(comparable(Verdict::LikelyWorthIt).unwrap().name, "Medusa"); + assert_eq!(comparable(Verdict::UnlikelyToHelp).unwrap().name, "OFBiz"); + assert!(comparable(Verdict::TooSmall).is_none()); + } + + #[test] + fn documents_are_counted_even_though_discovery_skips_them_as_binary() { + let d = tmp("docs"); + // Real `.docx` bytes are a zip; what matters here is the NUL that makes + // discovery classify it as binary and skip it. + fs::write(d.join("spec.docx"), [0x50, 0x4b, 0x03, 0x04, 0x00, 0x01]).unwrap(); + fs::write(d.join("a.py"), "x = 1\n").unwrap(); + let found = discover::discover(&d).unwrap(); + assert!( + found.files.iter().all(|f| f.path != "spec.docx"), + "the premise of this test: discovery skips it as binary" + ); + assert_eq!(measure_documents(&found).unreadable_by_grep, 1); + let _ = fs::remove_dir_all(&d); + } + + #[test] + fn a_line_count_is_abbreviated_only_where_abbreviating_is_not_misleading() { + // Large counts are estimates and say so. + assert_eq!(lines_text(612_345), "~612k"); + assert_eq!(lines_text(10_000), "~10k"); + // Small ones are exact. Rounding 2 lines up to "1k" in a command whose job is + // honest measurement is the one thing it must not do. + assert_eq!(lines_text(9_999), "9999"); + assert_eq!(lines_text(4_200), "4200"); + assert_eq!(lines_text(2), "2"); + assert_eq!(lines_text(0), "0"); + // A threshold is never marked approximate. + assert_eq!(floor_text(), "20k"); + } + + #[test] + fn a_count_of_one_reads_as_one() { + let one = Scale { + indexable_files: 1, + code_files: 1, + lines: 2, + }; + assert_eq!(scale_text(&one), "1 indexable file, 2 lines"); + assert_eq!( + scale_text(&Scale { + indexable_files: 4_178, + code_files: 2_974, + lines: 713_087, + }), + "4178 indexable files, ~713k lines" + ); + } +} diff --git a/crates/reify/src/index.rs b/crates/reify/src/index.rs index 7e63e51..63ac7e1 100644 --- a/crates/reify/src/index.rs +++ b/crates/reify/src/index.rs @@ -189,6 +189,12 @@ pub struct IndexReport { pub unresolved_refs: usize, pub history_rebuilt: bool, pub history_truncated: bool, + /// Why history could not be read, when it could not be read at all. + /// + /// Distinct from `history_truncated`, which means the walk stopped early but + /// returned commits. This means there are none, and the answer is poorer for it — + /// so it is reported rather than left for the user to notice a missing section. + pub history_unavailable: Option, pub parse_errors: Vec, /// Wall-clock milliseconds per stage, in the order they ran. /// @@ -615,7 +621,23 @@ pub fn index(store: &mut Store, opts: &IndexOptions) -> Result { stages.begin("reading history"); report.history_rebuilt = true; store.forget_history()?; - let history = gitlog::history(&opts.root, opts.max_commits)?; + // History is evidence, not scaffolding: without it the index is poorer but + // every symbol, document, rule and edge is still there. Aborting the whole + // index because git could not answer throws all of that away over one stage. + // + // This is not hypothetical. A `--filter=blob:none` clone is the ordinary way + // to clone a large repository, and query-time git runs with + // `GIT_NO_LAZY_FETCH=1` so the offline promise covers the whole process tree. + // Together those mean any history walk needing an uncached blob fails — which, + // on a blobless clone at an older commit, is the common case rather than the + // exception. The neighbouring `bodies` call already degrades this way. + let history = match gitlog::history(&opts.root, opts.max_commits) { + Ok(history) => history, + Err(err) => { + report.history_unavailable = Some(format!("{err:#}")); + gitlog::History::default() + } + }; report.history_truncated = history.truncated; store.commit(stage_history(&history, &present))?; // Bodies join the prior behind a structural leakage wall: this stage only diff --git a/crates/reify/src/lib.rs b/crates/reify/src/lib.rs index 87c7b56..5b2d90e 100644 --- a/crates/reify/src/lib.rs +++ b/crates/reify/src/lib.rs @@ -1,6 +1,7 @@ pub mod concepts; pub mod context; pub mod discover; +pub mod doctor; pub mod extract; pub mod gitlog; pub mod index; diff --git a/crates/reify/src/lockfile.rs b/crates/reify/src/lockfile.rs index d27d9c8..152e042 100644 --- a/crates/reify/src/lockfile.rs +++ b/crates/reify/src/lockfile.rs @@ -96,9 +96,11 @@ impl Drop for IndexLock { /// Is a process with this id running? /// -/// `kill(pid, 0)` is the portable POSIX existence check. On other platforms this -/// returns `false`, which errs toward reclaiming a lock rather than deadlocking a -/// repository — the safer failure for an advisory lock. +/// The lock is only as good as this answer. A liveness check that always says "no" +/// does not err on the safe side — it makes every lock look stale, so the lock stops +/// excluding anything and two indexers write the same store. That is what the +/// `not(unix)` stub used to do, and CI on Windows found it by failing to recognise +/// its own process as alive. #[cfg(unix)] fn process_is_alive(pid: u32) -> bool { // SAFETY: `kill` with signal 0 performs no action; it only reports whether the @@ -112,9 +114,57 @@ extern "C" { fn libc_kill(pid: i32, sig: i32) -> i32; } -#[cfg(not(unix))] +/// Win32's answer to `kill(pid, 0)`. +/// +/// Declared by hand rather than pulling in a Windows crate, for one question asked +/// once — the same reason `kill` is declared above rather than taking a libc +/// dependency. +#[cfg(windows)] +mod win32 { + pub type Handle = *mut core::ffi::c_void; + extern "system" { + pub fn OpenProcess(access: u32, inherit: i32, pid: u32) -> Handle; + pub fn WaitForSingleObject(handle: Handle, millis: u32) -> u32; + pub fn CloseHandle(handle: Handle) -> i32; + } +} + +#[cfg(windows)] +fn process_is_alive(pid: u32) -> bool { + /// The narrowest right to ask "does this exist"; granted across integrity levels + /// where `PROCESS_QUERY_INFORMATION` is not. + const QUERY_LIMITED_INFORMATION: u32 = 0x1000; + /// Required to wait on the handle at all. Omitting it does not make the wait + /// stricter — it makes it fail with `WAIT_FAILED`, which reads as "not running" + /// and silently restores the bug this function exists to fix. + const SYNCHRONIZE: u32 = 0x0010_0000; + /// The handle is not signalled, so the process has not exited. + const WAIT_TIMEOUT: u32 = 258; + + // SAFETY: `OpenProcess` returns null rather than an invalid handle on failure, and + // the handle is closed on every path that obtained one. + unsafe { + let handle = win32::OpenProcess(SYNCHRONIZE | QUERY_LIMITED_INFORMATION, 0, pid); + if handle.is_null() { + // No such process, or one this user may not query. Either way, treating + // the lock as reclaimable is the behaviour a dead owner should get. + return false; + } + // Waiting zero milliseconds asks the question without blocking. Preferred over + // `GetExitCodeProcess`, which reports the sentinel 259 for a running process + // and cannot distinguish it from one that genuinely exited with 259. + let state = win32::WaitForSingleObject(handle, 0); + win32::CloseHandle(handle); + state == WAIT_TIMEOUT + } +} + +/// Any other platform. Deliberately pessimistic: without a liveness check the lock +/// cannot be trusted, so it refuses to reclaim rather than silently allowing two +/// indexers to share a store. +#[cfg(not(any(unix, windows)))] fn process_is_alive(_pid: u32) -> bool { - false + true } #[cfg(test)] diff --git a/crates/reify/src/query.rs b/crates/reify/src/query.rs index b32b6e0..0fd6b1d 100644 --- a/crates/reify/src/query.rs +++ b/crates/reify/src/query.rs @@ -229,7 +229,12 @@ pub struct ImpactAnswer { pub schema: &'static str, pub query: String, pub origins: Vec, + /// Everything found to depend on the origins. Truncated for presentation; + /// `affected_total` is what was actually found. pub affected: Vec, + /// How many dependants were found before the list was truncated. Naming sixty of + /// two hundred is a sample, and calling it the answer would be a lie about scope. + pub affected_total: usize, pub tables: Vec, pub co_changing_files: Vec, pub unknowns: Vec, @@ -247,6 +252,7 @@ pub fn impact(store: &Store, query: &str) -> Result { query: query.to_string(), origins: origins.iter().map(citation).collect(), affected: Vec::new(), + affected_total: 0, tables: Vec::new(), co_changing_files: Vec::new(), unknowns: Vec::new(), @@ -258,30 +264,52 @@ pub fn impact(store: &Store, query: &str) -> Result { return Ok(answer); } - let origin_ids: HashSet = origins.iter().map(|n| n.id).collect(); + // A file argument is the common case from an editor hook, and a file is not the + // node dependencies attach to: `CALLS` edges land on symbols, `IMPORTS` on files. + // Seeding a file's symbols alongside the file itself is what makes `impact ` + // agree with `preflight ` instead of contradicting it. + let mut seeds: Vec = origins.clone(); + for origin in &origins { + if origin.kind != NodeKind::File { + continue; + } + if let Some(path) = &origin.path { + seeds.extend(store.symbols_in_file(path)?); + } + } + let origin_ids: HashSet = seeds.iter().map(|n| n.id).collect(); let mut seen: HashSet = origin_ids.clone(); - let mut frontier: Vec<(Node, u32, String)> = origins - .iter() - .cloned() - .map(|n| (n, 0, String::new())) - .collect(); + let mut frontier: Vec<(Node, u32, String)> = + seeds.into_iter().map(|n| (n, 0, String::new())).collect(); while let Some((node, depth, _)) = frontier.pop() { - if depth >= IMPACT_MAX_DEPTH || answer.affected.len() >= IMPACT_MAX_NODES { + if depth >= IMPACT_MAX_DEPTH { continue; } - // Callers depend on this symbol. - for (dependant, _, confidence) in - store.neighbors(node.id, Direction::In, &[EdgeKind::Calls])? - { + // Past the display budget, keep counting but stop widening: the marginal + // second-hop name costs tokens without changing what an engineer decides. + let widen = answer.affected.len() < IMPACT_MAX_NODES; + // Callers depend on this symbol; importers depend on this file. + for (dependant, edge, confidence) in store.neighbors( + node.id, + Direction::In, + &[EdgeKind::Calls, EdgeKind::Imports], + )? { if !seen.insert(dependant.id) { continue; } - let reason = format!("calls {}", node.name); + let verb = if edge == EdgeKind::Imports { + "imports" + } else { + "calls" + }; + let reason = format!("{verb} {}", node.name); answer .affected .push(affected(&dependant, depth + 1, reason, confidence)); - frontier.push((dependant, depth + 1, String::new())); + if widen { + frontier.push((dependant, depth + 1, String::new())); + } } // Data coupling: anything else touching a table this symbol writes. for (table, _, _) in store.neighbors( @@ -332,6 +360,7 @@ pub fn impact(store: &Store, query: &str) -> Result { .then(b.confidence.total_cmp(&a.confidence)) .then(a.location.cmp(&b.location)) }); + answer.affected_total = answer.affected.len(); answer.affected.truncate(IMPACT_MAX_NODES); if answer.affected.is_empty() { @@ -1311,6 +1340,41 @@ class SalesOrder: let _ = fs::remove_dir_all(&root); } + #[test] + fn impact_on_a_file_never_reports_less_than_preflight_on_the_same_file() { + // Two commands contradicting each other about one file is worse than either + // being silent: an agent acts on the answer it was given. `impact` counts + // intra-file callers that `preflight` deliberately excludes, so it may report + // more — never fewer. + let (store, root) = indexed(); + let mut checked = 0; + for file in store.nodes_of_kind(NodeKind::File).unwrap() { + let path = file.path.as_deref().unwrap_or(&file.name); + let expected = preflight(&store, path).unwrap().dependants; + let found = impact(&store, path).unwrap().affected_total; + assert!( + found >= expected, + "impact({path}) found {found} affected, preflight found {expected} dependants" + ); + checked += 1; + } + assert!(checked > 0, "the fixture corpus indexed no files"); + + // The invariant above is satisfiable by file-level imports alone, so pin the + // thing that actually broke: a file argument must reach the symbols that call + // into it, the way a symbol argument does. + let answer = impact(&store, "app/order.py").unwrap(); + assert!( + answer + .affected + .iter() + .any(|a| a.reason.starts_with("calls")), + "a file's callers must be reported, not only its importers: {:?}", + answer.affected + ); + let _ = fs::remove_dir_all(&root); + } + #[test] fn impact_on_an_unknown_query_says_so_rather_than_inventing() { let (store, root) = indexed(); diff --git a/crates/reify/src/rules.rs b/crates/reify/src/rules.rs index 1dd4661..b0c5e94 100644 --- a/crates/reify/src/rules.rs +++ b/crates/reify/src/rules.rs @@ -688,7 +688,10 @@ fn subject_vocabulary(name: &str) -> BTreeSet { fn classify_phrase(phrase: &str) -> Option<(String, Polarity)> { let lowered = phrase.to_lowercase(); for subject in SUBJECTS { - if !subject.terms.iter().any(|t| lowered.contains(t)) { + // Word-boundaried, like the polarity test below. A raw substring match reads + // `must-revalidate` as the `validation` subject, because `validate` sits inside + // `revalidate` — and a cache header then arrives as a business rule at 0.97. + if !subject.terms.iter().any(|t| contains_word(&lowered, t)) { continue; } let signals = |words: &[&str], shared: &[&str]| { @@ -842,6 +845,23 @@ mod tests { assert_eq!(classify_phrase("returns a list of rows"), None); } + #[test] + fn a_subject_term_inside_a_longer_word_is_not_that_subject() { + // `validate` sits inside `revalidate`, and `must` supplies the polarity, so a + // substring subject test mined an HTTP cache header as a validation rule at + // 0.97 confidence. The subject must be word-boundaried like the polarity is. + assert_eq!(classify_phrase("adds a must-revalidate header"), None); + // A genuine claim is untouched. + assert_eq!( + classify_phrase("orders must require approval"), + Some(("approval".into(), Polarity::Require)) + ); + assert_eq!( + classify_phrase("the service rejects an order that fails validation"), + Some(("validation".into(), Polarity::Require)) + ); + } + #[test] fn agglutinative_and_unspaced_scripts_still_match() { // Korean attaches particles to the stem and Thai has no word boundaries, so diff --git a/crates/reify/tests/fixtures.rs b/crates/reify/tests/fixtures.rs index e5bbc62..562b877 100644 --- a/crates/reify/tests/fixtures.rs +++ b/crates/reify/tests/fixtures.rs @@ -365,3 +365,78 @@ fn a_reference_resolves_when_another_file_later_defines_it() { let _ = fs::remove_dir_all(&dir); } + +/// A repository whose history git cannot read must still produce a usable index. +/// +/// This is the blobless-clone case, reproduced without a network: `--filter=blob:none` +/// is the ordinary way to clone a large repository, query-time git runs with +/// `GIT_NO_LAZY_FETCH=1` so the offline promise covers the whole process tree, and +/// together those make any history walk needing an uncached object fail. Before this +/// was handled, that failure took the entire index with it — the user got exit 1 and +/// no store, having lost every symbol, document and edge over one optional stage. +/// +/// Simulated here by deleting the commit object after committing: `.git/HEAD` still +/// resolves, so indexing reaches the history stage, and `git log` then fails exactly +/// as it does when a promisor remote cannot supply an object. +#[test] +fn an_unreadable_history_degrades_instead_of_failing_the_whole_index() { + use std::fs; + let dir = std::env::temp_dir().join(format!( + "reify-nohist-{}-{:?}", + std::process::id(), + std::thread::current().id() + )); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let git = |args: &[&str]| { + std::process::Command::new("git") + .args(args) + .current_dir(&dir) + .env("GIT_AUTHOR_NAME", "t") + .env("GIT_AUTHOR_EMAIL", "t@t") + .env("GIT_COMMITTER_NAME", "t") + .env("GIT_COMMITTER_EMAIL", "t@t") + .output() + .unwrap() + }; + if !git(&["init", "-q"]).status.success() { + return; // no git available; nothing to assert + } + fs::write( + dir.join("order.py"), + "def apply_discount(total):\n return total * 0.9\n", + ) + .unwrap(); + git(&["add", "."]); + git(&["commit", "-q", "-m", "first"]); + + // Remove the commit object HEAD names. `git log` now fails; `.git/HEAD` does not. + let head = String::from_utf8(git(&["rev-parse", "HEAD"]).stdout).unwrap(); + let head = head.trim(); + let object = dir.join(".git/objects").join(&head[..2]).join(&head[2..]); + let _ = fs::remove_file(&object); + let _ = fs::remove_dir_all(dir.join(".git/objects/pack")); + + let mut store = Store::in_memory().expect("in-memory store"); + let report = index(&mut store, &IndexOptions::new(dir.clone())) + .expect("an unreadable history must not fail the index"); + + assert!( + report.history_unavailable.is_some(), + "the failure must be reported, not swallowed" + ); + assert!( + report.files_parsed > 0, + "the code must still be indexed: {report:?}" + ); + + assert!( + !store + .symbols_named("apply_discount") + .expect("lookup") + .is_empty(), + "the symbol must survive a history failure" + ); + + let _ = fs::remove_dir_all(&dir); +} diff --git a/docs/integration/claude-code.md b/docs/integration/claude-code.md index c51b0e2..72210fb 100644 --- a/docs/integration/claude-code.md +++ b/docs/integration/claude-code.md @@ -3,10 +3,15 @@ Four levels, cheapest first. **Start at level 0** — it works today, costs nothing until used, and is what the benchmark measured. +`reify install` picks the right level for whichever agents this repository is actually +configured for and shows its plan before writing anything; `reify install --yes` applies +it. It installs level 0 by default, for the reason below. `reify uninit` removes +everything it wrote. + ## Level 0 — a shell command (recommended) -`reify init` finds your `AGENTS.md` or `CLAUDE.md` and tells you what to add. -`reify init --write-agent-instructions` appends it for you: +`reify install` writes it into whatever files the agents here already read. +`reify init --write-agent-instructions` appends it to one file: ```markdown ## Before changing code in this repo @@ -25,12 +30,19 @@ is reducing context, paying a per-turn tax to deliver it would be self-defeating ## Level 1 — MCP, if your client cannot run a shell command ```bash -reify serve --mcp +reify install --mcp --yes # merges the server entry into .mcp.json +reify serve --mcp # or register it by hand ``` -Three tools — `reify_context`, `reify_why`, `reify_impact` — and that is the whole -surface, deliberately. `mcp::tests::the_tool_schemas_stay_small_enough_to_be_worth_sending` -asserts the schemas cost under 600 tokens. +`--mcp` is an opt-in to the per-turn cost above, and `install` says so before writing. +It merges into an existing `.mcp.json` rather than replacing it: unrelated servers, their +environment blocks and the file's formatting survive untouched, and a config that does +not parse is reported and skipped rather than overwritten. + +Six tools — `reify_context`, `reify_why`, `reify_impact`, `reify_explain`, +`reify_flow`, `reify_conflicts` — and that is the whole surface, deliberately. +`mcp::tests::the_tool_schemas_stay_small_enough_to_be_worth_sending` asserts the +schemas cost under 600 tokens, which all six still fit inside. ## Level 2 — a preflight hook diff --git a/docs/integration/generic-cli.md b/docs/integration/generic-cli.md index a75c6a4..ce4c780 100644 --- a/docs/integration/generic-cli.md +++ b/docs/integration/generic-cli.md @@ -38,7 +38,21 @@ Claims marked INFERRED are leads to verify, not facts. If `conflicts` is non-empty, resolve the disagreement before changing behaviour. ``` -## Codex, Cursor, OpenCode, Aider, Pi +## Codex, Cursor, Windsurf, Cline, Copilot, OpenCode, Aider, Pi -No adapter needed. Put the block above in whatever instruction file the tool reads -(`AGENTS.md`, `.cursorrules`, `CONVENTIONS.md`). The CLI is the interface. +No adapter needed. `reify install` finds which of these this repository is configured for +and writes the block into each one's own file — a dedicated rule file where the tool has +a rules directory (`.cursor/rules/`, `.windsurf/rules/`, `.clinerules/`), an append where +it reads a single file (`AGENTS.md`, `.cursorrules`, `.github/copilot-instructions.md`, +`CONVENTIONS.md`). It shows the plan and stops unless `--yes`. + +Detection needs evidence **in the repository**. A tool installed on your machine but not +configured here is listed and left alone: `~/.cursor` says you have Cursor, not that this +repository is worked on with it, and creating a `.cursorrules` on that basis would be a +guess. Where nothing is recognised, `install` prints the block for you to place. + +Everything it writes is inside the repository, so `reify uninit` reverses all of it. That +is also why no machine-wide MCP config is touched: a per-repository uninstall cannot +safely undo a machine-wide registration. + +The CLI is the interface; put the block above anywhere yourself if you prefer. diff --git a/docs/json-schema/README.md b/docs/json-schema/README.md index 3aa2714..c2fd8cc 100644 --- a/docs/json-schema/README.md +++ b/docs/json-schema/README.md @@ -197,3 +197,86 @@ version. "suggested_command": "string" } ``` + +## `reify doctor --json` + +Answers before there is an index, so it reads the working tree and `git log` rather than +the store. `verdict` is one of `too_small`, `likely_worth_it`, `marginal`, +`unlikely_to_help`. `vocabulary` and `history` are `null` below the line floor and when +git history cannot be read — absent rather than defaulted, so a consumer cannot mistake +"not measured" for "measured zero". Metric definitions: [`../metrics.md`](../metrics.md). + +```json +{ + "schema": "string", + "root": "string", + "scale": { + "indexable_files": "integer", + "code_files": "integer", + "lines": "integer" + }, + "git_repository": "boolean", + "vocabulary": { + "commits_considered": "integer", + "commits_local": "integer", + "locality": "number" + }, + "history": { + "commits_read": "integer", + "truncated": "boolean", + "usable_subjects": "integer", + "usable_share": "number", + "focused_commits": "integer", + "focus": "number", + "median_files_changed": "integer" + }, + "documents": { + "unreadable_by_grep": "integer", + "examples": [ + "string" + ] + }, + "verdict": "string", + "reason": "string", + "what_would_change_it": [ + "string" + ], + "elapsed_ms": "integer" +} +``` + +## `reify install --json` + +The plan, whether or not it was applied. `applied` is false unless `--yes` was passed. +`kind` is one of `instructions`, `rule_file`, `mcp`; `state` is one of `planned`, +`already_present`, `skipped`. `problem` is non-null only when `state` is `skipped`, and +says why the file was left alone. `evidence` is what each detection rests on, so a +consumer can check the claim rather than trust it. `instruction_block` is non-null only +when no agent was recognised — it is the text to paste by hand. + +```json +{ + "schema": "string", + "root": "string", + "mcp": "boolean", + "applied": "boolean", + "steps": [ + { + "path": "string", + "kind": "string", + "agents": [ + "string" + ], + "evidence": [ + "string" + ], + "state": "string", + "problem": "null" + } + ], + "instruction_block": "null", + "detected_elsewhere": [ + "string" + ] +} +``` diff --git a/docs/json-schema/regenerate.sh b/docs/json-schema/regenerate.sh index a10635e..76e8914 100755 --- a/docs/json-schema/regenerate.sh +++ b/docs/json-schema/regenerate.sh @@ -9,4 +9,8 @@ reify -C "$REPO" --json context "approval for corporate orders" > /tmp/reify-ctx reify -C "$REPO" --json why "SalesOrder.requires_approval" > /tmp/reify-why.json reify -C "$REPO" --json impact "requires_approval" > /tmp/reify-impact.json reify -C "$REPO" --json preflight "app/order.py" > /tmp/reify-pre.json +# doctor needs no index; point it at a repository with real history instead. +reify -C "$REPO" --json doctor > /tmp/reify-doctor.json +# install without --yes writes nothing, so this is safe to run anywhere. +reify -C "$REPO" --json install > /tmp/reify-install.json echo "Now run the shape extractor in docs/json-schema/ to rebuild README.md" diff --git a/docs/metrics.md b/docs/metrics.md index c5f23a0..ff1a5e0 100644 --- a/docs/metrics.md +++ b/docs/metrics.md @@ -20,6 +20,29 @@ liability, not marketing. | **Documented symbols** | Symbols with a docstring or leading comment, over all symbols. | | **Knowledge coverage** | Symbols reachable from at least one concept or document section, over all symbols. Measures how much of the code the knowledge layer can say anything about. | +## `reify doctor` + +Every figure is measured over the working tree and the newest 1000 commits. No index is +read, so these do not change once `reify index` has run. + +| Metric | Definition | +|---|---| +| **Indexable files** | Files `reify index` would parse. The same walk `reify init` reports, so the two agree by construction. | +| **Lines** | Lines across those files, printed rounded to the nearest thousand. Includes blanks and comments — it is a size estimate, not a measure of code. | +| **Commit focus** | Commits touching between 1 and 20 files, over all commits read. 20 is the threshold `gitlog::History::co_changes` already uses to discard a commit as too sweeping to learn from. | +| **Subject→path locality** | Focused commits whose subject shares a stem-folded meaningful word with a path that commit changed, over all focused commits with a usable subject. Per commit, never pooled: the pooled version was measured on the same four repositories and inverts. | +| **Usable subject** | A subject that is not a merge and carries at least two meaningful words. Excludes "Merge pull request #123 from…", "Bump version to 4.2.1" and "wip". | +| **Documents only Reify can read** | Files classified `.docx`, `.doc`, `.odt`, `.rtf`, `.xlsx`, `.pptx` or `.pdf`. Counted across everything walked, indexable or not, because these are binary and discovery records them as skipped. | + +The **verdict** is not a metric and is deliberately not a score. It is a rule over the +figures above, fitted to the four repositories in `benchmarks/REPORT*.md`, and the +command says so in its own output. A 0-100 "suitability" number blended from four +heuristics tuned on four repositories is exactly the false precision this page exists to +forbid. + +Thresholds and their evidence are in the module documentation of +`crates/reify/src/doctor.rs`, next to the code that applies them. + ## Benchmark | Metric | Definition | @@ -32,6 +55,28 @@ liability, not marketing. | **Expected tokens** | Mean tokens to reach a changed file, charging a miss the full budget. The comparable single number: a condition cannot improve it by failing more often. | | **Head to head** | Median tokens over only the tasks *both* conditions solved. Removes the difficulty bias in the per-condition median. | +## Held-out-hunk benchmark + +`reify-bench verify-eval`. Each trial takes a merged commit, withholds one file's only +hunk — the *omission* — and asks the graph what the truncated patch missed. The same +commit is then run complete; a merged commit is complete by construction, so every +finding there is a false positive. + +| Metric | Definition | +|---|---| +| **`omission_recall`** | Truncated diffs where some finding cites the omitted hunk's file, over all trials. | +| **…attributable** | The same, counting only citations the *complete* commit does not also produce. A citation the negative control produces too was not caused by the omission. | +| **`omission_recall_symbol`** | The same at symbol granularity, over the trials whose omission falls inside an indexed symbol. Trials where it does not are excluded, never counted as misses. | +| **Omitted files a caller query could cite** | Trials whose omitted file has an outbound cross-file `CALLS` edge at the parent commit. Every finding is a caller, so `omission_recall` cannot exceed this share however the query is written. | +| **`false_alarm_rate`** | Findings per *complete* merged commit. A rate over counts, not a proportion, so it carries no Wilson interval; the share of commits with at least one false alarm is reported beside it and does. | +| **`findings_per_diff`** | Median findings per truncated diff. Median rather than mean because a checker emitting thirty findings once is not a checker with a small problem everywhere. | +| **`verify_tokens`** | Median `heuristic-v1` estimate of the findings output itself — what an agent pays to read the answer. Excludes the diff and the files it would then open. | +| **`verify_latency_ms`** | Median wall clock of the graph query alone. Extracting and indexing the parent tree is reported separately, because the real feature would run against an index that already exists. | + +The checker under test is not `reify verify`, which does not exist. It is the shipped +graph query — *symbols changed by this diff, minus symbols present in the diff, where +an inbound `CALLS` edge exists at distance 1* — reached through `reify::query::impact`. + ## Token estimation `reify` estimates tokens with `heuristic-v1`: Latin script at four bytes per token, CJK diff --git a/install.sh b/install.sh index b6bfaf5..e4a032e 100755 --- a/install.sh +++ b/install.sh @@ -12,6 +12,7 @@ BIN_DIR="${REIFY_INSTALL_DIR:-$HOME/.local/bin}" os=$(uname -s) arch=$(uname -m) target="" +exe="" case "$os" in Darwin) case "$arch" in @@ -23,6 +24,11 @@ case "$os" in aarch64 | arm64) target="aarch64-unknown-linux-gnu" ;; x86_64) target="x86_64-unknown-linux-gnu" ;; esac ;; + # Git Bash, MSYS2 and Cygwin are how a Windows developer runs `curl | sh`. + MINGW* | MSYS* | CYGWIN*) + case "$arch" in + x86_64) target="x86_64-pc-windows-msvc"; exe=".exe" ;; + esac ;; esac if [ -z "$target" ]; then echo "install.sh: no prebuilt binary for $os/$arch." >&2 @@ -44,11 +50,32 @@ trap 'rm -rf "$tmp"' EXIT echo "downloading $url" curl -fsSL "$url" -o "$tmp/$name.tar.gz" + +# Verify before unpacking, the same check `reify upgrade` makes. Every release +# publishes `.sha256`; a missing or mismatched one stops the install rather +# than running an unverified binary. +curl -fsSL "$url.sha256" -o "$tmp/$name.tar.gz.sha256" +expected=$(awk '{print $1}' "$tmp/$name.tar.gz.sha256") +if command -v shasum >/dev/null 2>&1; then + actual=$(shasum -a 256 "$tmp/$name.tar.gz" | awk '{print $1}') +elif command -v sha256sum >/dev/null 2>&1; then + actual=$(sha256sum "$tmp/$name.tar.gz" | awk '{print $1}') +else + echo "install.sh: no shasum or sha256sum available to verify the download" >&2 + exit 1 +fi +if [ -z "$expected" ] || [ "$expected" != "$actual" ]; then + echo "install.sh: checksum mismatch for $name.tar.gz" >&2 + echo " published: $expected" >&2 + echo " downloaded: $actual" >&2 + exit 1 +fi + tar -xzf "$tmp/$name.tar.gz" -C "$tmp" mkdir -p "$BIN_DIR" -install -m 0755 "$tmp/$name/reify" "$BIN_DIR/reify" +install -m 0755 "$tmp/$name/reify$exe" "$BIN_DIR/reify$exe" -echo "installed reify $tag to $BIN_DIR/reify" +echo "installed reify $tag to $BIN_DIR/reify$exe" case ":$PATH:" in *":$BIN_DIR:"*) ;; *) echo "note: $BIN_DIR is not on your PATH" ;; diff --git a/site/docs.html b/site/docs.html index 449770b..6ff273b 100644 --- a/site/docs.html +++ b/site/docs.html @@ -43,7 +43,14 @@

Install

-

Prebuilt binaries for macOS (Apple Silicon and Intel) and Linux (x86_64 and aarch64).

+

+ Prebuilt binaries for macOS (Apple Silicon and Intel), Linux (x86_64 and aarch64) + and Windows (x86_64). On Windows the command above works in Git Bash, MSYS2 or + WSL; from PowerShell, take the x86_64-pc-windows-msvc archive from + the latest release, verify its .sha256, and put + reify.exe on your PATH. Every platform runs the full + test suite in CI. + 

❯ curl -fsSL https://raw.githubusercontent.com/lambiengcode/reify/main/install.sh | sh
@@ -139,8 +146,9 @@

reify context

MCP server

- Three tools, and that is the whole surface. An MCP server's schemas are paid for on - every turn, so a large one is a tax on every message an agent sends. + Six tools, and that is the whole surface. An MCP server's schemas are paid for on + every turn, so a large one is a tax on every message an agent sends — all six + together still cost under 600 tokens, which a test asserts.

@@ -148,6 +156,9 @@

MCP server

+ + +
reify_contextThe minimum context for a task, under a budget
reify_whyProvenance for one file and line
reify_impactThe blast radius of a symbol
reify_explainOne business concept, in every language it appears in
reify_flowThe ordered code that carries out a business process
reify_conflictsDocumentation that disagrees with the implementation
diff --git a/site/index.html b/site/index.html index 4dc27bc..69a61a8 100644 --- a/site/index.html +++ b/site/index.html @@ -6,7 +6,7 @@ Reify — a local knowledge engine for AI coding agents - + @@ -60,21 +60,22 @@

Measured on someone else's benchmark

-
84.6%Reify offered a file the fix touched
+
87.0%Reify offered a file the fix touched
6.6%grep, same budget
12 / 12repositories where Reify wins
0network calls, asserted in CI
-
reify84.6%
+
reify87.0%
grep6.6%

Retrieval is not the same as resolution, and the README says so with equal - prominence: end to end, Reify and a BM25 baseline currently resolve the same - number of issues. The full write-up + prominence: end to end, Reify resolves 73.3% against a BM25 baseline's 67.3% — + ahead, but at 101 instances not yet a statistically significant lead. + The full write-up shows both, including the arms that failed.

@@ -109,7 +110,7 @@

reify conflicts

reify serve

-

The Model Context Protocol over stdio, three tools, for agents that speak it.

+

The Model Context Protocol over stdio, six tools, for agents that speak it.