-
Notifications
You must be signed in to change notification settings - Fork 0
Comparing changes
Open a pull request
base repository: lambiengcode/reify
base: v0.2.2
head repository: lambiengcode/reify
compare: v0.2.3
- 17 commits
- 56 files changed
- 2 contributors
Commits on Aug 24, 2026
-
feat(context): stop one file from spending the whole window
Retrieval on SWE-bench Verified goes 84.6% -> 87.0% at three rounds and 66.0% -> 72.6% at one. Both grep baselines came back byte-identical across the two runs, which is the control: nothing changed but the retriever. Three things were wrong with how the window was filled, all found by running `reify context` against Medusa and reading the output rather than the code. Relevance spreads along edges, so every member of a file that matched the task loosely arrived holding a plausible score, and nothing bounded how many were admitted. Asked to prevent a discount being applied twice, the answer spent 13 of 20 slots on one HTTP router and one arithmetic helper, and the promotion service that actually had to change ranked eighteenth. No file may now claim more than four symbol slots. The reading plan drained each ranked file in turn, so the top file consumed every slot: two of three sampled tasks produced six entries naming a single file. Spans are now drawn a file at a time in rounds, so every ranked file earns one before any earns a second. `next.dedup()` had looked like it covered this; it removes only consecutive duplicates, and was deduping on symbol rather than path. Tests and fixtures outranked the code they exercise — a promotion spec and its fixture both beat the promotion service. A test names the domain vocabulary as densely as its implementation and a reader cannot edit it to change behaviour, so it now gives up half its score. A penalty rather than an exclusion: "fix the failing test for X" is a real task, and the reproduction is genuinely where to start. Tunable as `test_path_penalty`. The cost is real and worth stating: mean reciprocal rank falls, 0.45 -> 0.43 at three rounds. Spreading the top slots across more files means that when the right file was already first, it now shares the window. That is the intended direction, because a file never offered cannot be used while a file at rank six is still in the context, and it is exactly the failure the end-to-end run diagnosed — retrieval was already far ahead and the model still did no better. Six of twelve repositories improved and none regressed. Every mechanism has a test confirmed to FAIL when the mechanism is removed; two earlier attempts passed with the feature disabled and were thrown away rather than counted.
Configuration menu - View commit details
-
Copy full SHA for 8c9e91d - Browse repository at this point
Copy the full SHA 8c9e91dView commit details -
fix(index): a repository whose history git cannot read still indexes
`reify index` exited 1 and left no usable store on a `--filter=blob:none` clone, which is the ordinary way to clone a large repository. Reproduced on a blobless clone of Flask at an older commit: every symbol, document, rule and edge was thrown away because one optional stage could not run. Two correct behaviours combined into a broken one. v0.2.2 gave query-time git `GIT_NO_LAZY_FETCH=1` so the offline promise covers the whole process tree — right, and it means any history walk needing an uncached object now fails instead of silently fetching it. On a blobless clone at an older commit that is the common case, not the exception. The caller then treated that failure as fatal. History is evidence, not scaffolding. It now degrades: the index keeps everything else and records why history is missing, the same shape the neighbouring `bodies` call has always had six lines below. The test reproduces the failure without a network by deleting the commit object after committing, so `.git/HEAD` still resolves, indexing reaches the history stage, and `git log` fails exactly as it does when a promisor remote cannot supply an object. Confirmed to FAIL against the old fail-fast code. Found while re-running the benchmark, where it stopped three of twelve repositories from indexing at all — so it was blocking measurement as well as users.
Configuration menu - View commit details
-
Copy full SHA for 9af59e4 - Browse repository at this point
Copy the full SHA 9af59e4View commit details -
feat(render): say a status once when a whole section shares it
Every code row of every answer read `[confirmed]`. Symbols are CONFIRMED by construction — they were parsed from source — so the badge was constant down twenty lines, carried no information, and invited precisely the wrong reading: it attests that a symbol exists, never that it is the right place to change. A reader scanning twenty identical green tags takes them as twenty endorsements. A section whose rows all share a status now states it once on the heading. A section with mixed statuses still badges every row, and the machine-readable output is untouched: every item carries its own status in JSON and TOON, which is where the safety property actually lives. The module contract is reworded to match — a claim is never rendered without its status, which is not the same as the badge appearing a fixed number of times — and there is a test that the hoisting happens only when the status is genuinely shared. Also prints why history is missing when the index could not read it, with the `git fetch --refetch` that completes a partial clone. An index that quietly drops history leaves `why` and blast radius poorer with nothing on screen to explain it.
Configuration menu - View commit details
-
Copy full SHA for 641bc88 - Browse repository at this point
Copy the full SHA 641bc88View commit details -
feat(mcp): expose explain, flow and conflicts
The server offered `context`, `why` and `impact`. Those are good tools and they are also the three any retriever could plausibly provide. `explain`, `flow` and `conflicts` are the ones nothing else ships — a concept across every language it appears in, the ordered code that carries out a process, and documentation that disagrees with the implementation — and MCP is how most clients reach Reify at all. The distinctive half of the product was unreachable from the integration path people actually use. The three-tool cap was load-bearing rather than stylistic: schemas are re-sent every turn of every session, so a large surface taxes every message whether or not a tool is called. That constraint is kept, not relaxed. All six schemas together still cost under the 600-token ceiling the original three were held to, because each addition takes one string argument, and the test asserting it is unchanged. `preflight` was considered and left out. For an agent it answers the same question as `why`, and a near-duplicate tool spends the budget twice while making the model's choice harder — which is the cost the cap exists to prevent. Also removes an unreachable `reify_context` arm that could never run: the TOON path returns before the match.
Configuration menu - View commit details
-
Copy full SHA for 36bedda - Browse repository at this point
Copy the full SHA 36beddaView commit details -
feat(dist): Windows binaries, and verify the checksum the installer p…
…romised Releases shipped macOS and Linux only, silently excluding Windows — where a large share of the mature business systems this tool exists for are maintained. `reify upgrade` needed real work to follow, not just a target triple. Windows holds an open handle on the running image, so the new binary cannot be renamed over it; the running one is renamed aside to free the path, and put back if the replacement then fails, so a failed upgrade never leaves the user with no binary at all. The displaced file is left for the next run to clear — deleting it while still mapped fails, and losing an upgrade over housekeeping would be worse than one stale file. `install.sh` recognises Git Bash, MSYS2 and Cygwin, which is how a Windows developer runs `curl | sh`. The installer also now verifies the SHA-256 it was already advertising. The line under the quickstart promised a checksum verified before installing; that was true of `reify upgrade` and not of the `curl | sh` path everyone uses first. A mismatch installs nothing. `documentation` pointed at docs.rs/reify, which belongs to an unrelated crate published in 2021 — readers following it got somebody else's software. It now points at this project's own docs.
Configuration menu - View commit details
-
Copy full SHA for 4e33974 - Browse repository at this point
Copy the full SHA 4e33974View commit details -
docs: the measured numbers, and a driver that reproduces them
Every SWE-bench figure re-measured against the new ranking and updated in all three READMEs, the badge and the site. Retrieval, 500 instances: one round 66.0% -> 72.6%, three rounds 84.6% -> 87.0%, and offering *every* touched file 77.0% -> 81.4%. Paired against content-grep, three rounds now win 406 and lose 4. Both grep arms came back byte-identical to the previous run, which is what makes the comparison a measurement rather than an anecdote. End to end, re-run through the official harness on 101 instances rather than 63: Reify 73.3% against BM25's 67.3%, paired 12-6, exact McNemar p = 0.24. The section leads with that p-value and calls the result *ahead and not yet proven*, because six issues and eighteen disagreements do not establish a lead at this sample size. It was a loss, then a tie, now this. Those absolute rates are not comparable to the 23.8% published before: that run used DeepSeek, whose account ran dry mid-project, and this one uses Claude Sonnet. A stronger model lifts both arms. What survives a model change is the paired comparison, since both arms always answer the same instance with the same model, and the caveat is written into the README, this file and the results JSON so it cannot drift loose from the numbers it qualifies. The committed stage-2 driver was not the script that produced the published figures. `stage2c.py` fed the model whole files; the real run used `--for-edit` regions, which is the entire reason the earlier loss became a tie. Anyone following the repository's own instructions would have failed to reproduce its own results. Replaced with the driver that produced the numbers above, model as a parameter. Two claims that had quietly become false are corrected rather than left: the hero line said a BM25 baseline resolves *more* issues than Reify, and the site said the two resolve the same number. The Roadmap and Status sections are gone from all three READMEs — hand-maintained prose that needed editing every release to stay true, and Status had already drifted, still quoting `reify why` at 205 ms three versions after that measurement changed.
Configuration menu - View commit details
-
Copy full SHA for 870905b - Browse repository at this point
Copy the full SHA 870905bView commit details -
fix(impact): a file argument reaches the symbols that depend on it
`impact <file>` seeded its traversal from the File node and walked inbound CALLS. Call edges attach to symbols, never to files, so a file argument could only ever return "no dependants found" — while `preflight` on the same file, using `symbols_in_file`, correctly reported 199. Two commands contradicting each other about one file is worse than either being silent, because an agent acts on the answer it was given. Seed a file's symbols alongside the file itself, and follow inbound IMPORTS so file-level importers are reported next to symbol-level callers. Symbol arguments are untouched. A file every module imports has hundreds of dependants, so the renderer leads with the count and shows the nearest few as evidence rather than spending an agent's budget to say "a lot". The corpus-wide `impact >= preflight` invariant is satisfiable by IMPORTS edges alone, so the test also pins what actually broke: a file's affected list must contain a caller, which fails without this change.
Configuration menu - View commit details
-
Copy full SHA for 3a95c15 - Browse repository at this point
Copy the full SHA 3a95c15View commit details -
fix(rules): match a rule's subject on whole words, as its polarity al…
…ready does `classify_phrase` tested the subject with a raw substring match while the polarity test on the next line used the word-boundaried `contains_word`. The asymmetry read `must-revalidate` as the `validation` subject — `validate` sits inside `revalidate` — with `must` supplying the polarity, so an HTTP cache header was mined as a business rule at 0.97 confidence. `contains_word` already falls back to substring for multi-word needles like "credit limit" and for non-ASCII ones, so the multilingual corpus is unaffected. Django's mined rules go 180 -> 152. All 30 losses were inspected: every one is a test method name, a mangled documentation snippet, or the same bug in French via premise/remise. Two genuine rules are newly gained. Known and deliberately out of scope: the `validation` subject lacks the inflections `approval` has, so prose like "the service validates the order" is now missed where the loose match caught it incidentally. Restoring those inflections would also restore the test-name noise this removes; that is a tuning decision about the subject table, not part of this bug.
Configuration menu - View commit details
-
Copy full SHA for 6aa58ba - Browse repository at this point
Copy the full SHA 6aa58baView commit details -
fix(bench): the prompt names the repository the task came from
`prompt()` hardcoded "(ERPNext)" with no parameter, so every model-in-the-loop run against Medusa, OFBiz and OpenMRS told the model it was working on ERPNext while asking about a different codebase in a different language. It does not obviously favour one arm — every condition shared the same wrong name, and the prompt-differs-only-in-context test still holds — but it is a validity defect in three published tables. The name now comes from `TaskSet::repository`. The three affected reports carry a dated note stating the defect and its scope rather than being silently regenerated from runs that were not redone.
Configuration menu - View commit details
-
Copy full SHA for a9b32f7 - Browse repository at this point
Copy the full SHA a9b32f7View commit details -
feat(bench): a model-free benchmark for detecting an incomplete patch
`reify verify` — a post-flight check that reads an agent's diff and reports what the patch missed — is only worth building if the call graph can support it. This measures that before the feature is written. For each qualifying merged commit the parent tree is indexed, one file's only hunk is withheld, and the truncated patch goes to the checker. The same commit then goes through complete, where every finding is a false positive by construction. That negative control is what stops the metric rewarding a checker that simply shouts. The checker is the shipped graph query reached through `query::impact`, not a new one, so this measures the substrate the decision needs. The pre-registered condition — recall below 0.25 or false alarms above 0.1 per commit means do not build — was written into metrics.rs before the first run. It fires on all three repositories. Recall is 0.50 on Rust, 0.10 on Python and 0.40 on Go, but false alarms run 4.4 to 23.5 per already-complete commit. It fails on noise, not blindness: a CALLS edge says a caller exists, not that the caller needed changing, and nothing in the graph separates a changed signature from an edit inside a body. No rewrite around that edge removes it. `reify verify` is therefore not built. 116s, no model, no network. AGENTS.md records the verdict so it is not rebuilt without beating these numbers first.
Configuration menu - View commit details
-
Copy full SHA for c011eb8 - Browse repository at this point
Copy the full SHA c011eb8View commit details -
docs(readme): remove duplication, fix three self-contradictions
The README disagreed with itself in ways that undercut the measurement discipline it is arguing for. - Three different answers for its own index timings. The install section said 4.6s/0.7s where the measured table says 4.2s/0.49s. Unified to the measured numbers. - It said `init` appends "a six-line block", then showed a four-line block and called it "the same four lines". Neither matched what the tool writes. It now shows the actual constant, including the two lines the paraphrase dropped. - Shell completions were documented twice, forty lines apart. - The quickstart and Install repeated the same commands and the same uninstall explanation; Install now carries only what the quickstart does not. - The optimisation war stories are compressed to a paragraph pointing at the changelog, keeping the `git log -L` lazy-fetch story because it is load-bearing for the privacy claim rather than for speed. Adds the verify-benchmark section, and records that `impact` now takes a file. Net 48 lines shorter. Every measurement section — Medusa's no-win row, "where it doesn't work", the p-value section — is untouched.
Configuration menu - View commit details
-
Copy full SHA for d1c9e17 - Browse repository at this point
Copy the full SHA d1c9e17View commit details -
feat(doctor): tell people not to use this, when they should not
Nothing told anyone whether Reify would help before they invested in it. The README FAQ answered it informally and `benchmarks/REPORT-medusa.md` already shows the advantage can be zero on a real repository. `reify doctor` makes that measured instead of anecdotal, and runs before there is an index — the whole point is deciding before committing to the tool. The signals were fitted against the four repositories in `benchmarks/`, not invented. Two hypotheses were tested and dropped: size inverts (OFBiz wins biggest at 1,364 code files, Medusa ties at 11,821), and so does the obvious reading of "history and file naming share vocabulary" — pooled corpus-wide, Medusa scores 0.43 against OFBiz's 0.38. Two signals do fit all four outcomes, each explaining a different failure: repo grep margin commit focus subject->path OFBiz +58 0.96 0.80 ERPNext +48 0.98 0.85 OpenMRS +9 0.98 0.48 Medusa 0 0.84 0.79 Commit focus is the share of commits touching few enough files that their subject says anything about them; Medusa is the only measured repository where it falls away and the only one where Reify won nothing. The mechanism is already load-bearing elsewhere: `History::co_changes` discards commits touching more than 20 files for the same reason, so `doctor` uses the same threshold rather than a second definition of a sweeping commit. No suitability score. `docs/metrics.md` forbids printing a number that cannot be defined, and a weighted blend of heuristics tuned on four repositories is exactly that. Named signals, measured values, a plain verdict, and — where the answer is no or marginal — what would change it. The verdict floor is 200 commits, which is where a 95% Wilson interval around Medusa's 0.84 lies wholly below the threshold; at 50 it straddles it, so a shorter history is reported as short rather than condemned. Verified on all four benchmarked repositories, which land where their reports say they should, plus django (yes) and a two-commit scratch repository (too small). No new dependency; the offline guarantee is untouched.
Configuration menu - View commit details
-
Copy full SHA for e32f567 - Browse repository at this point
Copy the full SHA e32f567View commit details -
feat(install): wire the integration each agent here should have
`reify init --write-agent-instructions` handled one file, chosen from a list four filenames long. `reify install` detects which agents this repository is actually configured for and wires each one in its own format and location, shows its plan before writing, and is fully reversed by `reify uninit`. It installs the shell-command instruction block, not MCP. That is the existing reasoned position in docs/integration/claude-code.md — an MCP server's tool schemas are re-sent on every turn of every session, and for a tool whose purpose is reducing context, a per-turn tax to deliver it is self-defeating — and nothing here overturns it. Every agent this can detect runs shell commands, so level 0 is right for all of them. `--mcp` is the deliberate opt-in and says what it costs before it writes anything. Two rules do most of the work: Detection requires evidence *in the repository*. `~/.cursor` says the user has Cursor installed, not that this repository is worked on with it; creating a `.cursorrules` on that basis is exactly the guess the command must not make. Home directories are read, reported as corroboration, and never acted on alone — an agent seen only there is listed so its absence from the plan is explained. Every detection prints what it rests on. Everything written stays inside the repository. A machine-wide MCP registration cannot be undone by a per-repository `uninit` without breaking every other repository relying on it, so the MCP entries written are the repository-scoped ones and a client with no such config gets the block instead, with the reason stated. Config merging is textual, not a serde round-trip: re-serialising sorts a user's keys and drops their formatting, and the file is theirs. Adding the entry to a hand-written `.mcp.json` changes exactly one line, an unrelated server's env block survives byte for byte, removal restores the original exactly, and the result is parsed before it is written so a bad splice fails loudly. A config that exists but does not parse is reported and skipped — never overwritten. `uninit` derives its removal targets from the same table `install` plans from, so a new agent cannot be added to one without appearing in the other, and a test asserts it. The MCP surface stays at six tools: these are operator commands and belong to the CLI.
Configuration menu - View commit details
-
Copy full SHA for 75eb0ff - Browse repository at this point
Copy the full SHA 75eb0ffView commit details -
docs(readme): lead with doctor and install
The quickstart still taught `init --write-agent-instructions`, the hand-wiring path, and skipped the question `doctor` exists to answer. It now runs the two commands in the order someone actually needs them: should you use this, then wire it. The Install section gains the same pair, with what `install` detects, that an agent present on the machine but not configured in the repository is reported rather than written to, and that nothing outside the repository is touched. Both blocks were run as written against a scratch repository before being documented.
Configuration menu - View commit details
-
Copy full SHA for 6ff262d - Browse repository at this point
Copy the full SHA 6ff262dView commit details
Commits on Aug 25, 2026
-
Merge pull request #2 from lambiengcode/feat/doctor-install-and-verif…
…y-benchmark Two new commands, three fixes, and a benchmark that killed a feature
Configuration menu - View commit details
-
Copy full SHA for de84695 - Browse repository at this point
Copy the full SHA de84695View commit details -
ci: test on the platforms releases actually ship (#3)
* ci: test on the platforms releases actually ship Releases ship Windows and macOS binaries. Neither had ever been executed by a job: CI ran on Linux only, so a portability break would have reached users before it reached us, and the Windows binary added last week was shipped on the strength of a type-check alone. A new `platform` job runs the suite and an end-to-end CLI smoke on windows-latest and macos-latest. `check` stays on Linux and keeps everything that does not vary by platform — formatting, clippy, and the network-egress gate, which needs iptables. Duplicating those three times would cost minutes per run and catch nothing. Two platform details are handled rather than discovered later. Windows runners convert LF to CRLF on checkout, which would test fixtures no user's repository actually contains and shift every byte offset the index records. `core.autocrlf false` is set in a step *before* checkout, because configuring it afterwards is too late. The smoke step runs under bash rather than the Windows default. PowerShell propagates only the last command's exit code, so a failing `init` followed by a passing `context` would have left the step green — a CI step that cannot fail is worse than no step. `fail-fast: false`, so a Windows failure does not hide a macOS one. * fix(lockfile): the index lock did nothing on Windows `process_is_alive` was implemented for unix and stubbed to `false` everywhere else. The comment called that the safer failure — erring toward reclaiming a lock rather than deadlocking a repository — but for the *current* process it means every lock looks stale, so the lock reclaims itself and stops excluding anything. Two `reify index` runs on Windows would both proceed against the same store. Windows CI found it in its first run, by failing to recognise its own process as alive. The Windows binary shipped last week has carried this since; it was added on the strength of a type-check, which is exactly what a type-check cannot catch. Implemented with `OpenProcess` + `WaitForSingleObject(handle, 0)`, declared by hand for one question asked once, the same way `kill` already is rather than taking a libc dependency. `WaitForSingleObject` is used in preference to `GetExitCodeProcess`, which reports the sentinel 259 for a running process and cannot distinguish it from one that genuinely exited with 259. `QUERY_LIMITED_INFORMATION` is the narrowest right that answers the question and is granted where `PROCESS_QUERY_INFORMATION` is not. The remaining `not(any(unix, windows))` arm now returns `true` rather than `false`. Without a liveness check the lock cannot be trusted, and refusing to reclaim a lock is a worse outcome for one user than letting two indexers share a store is for everyone. * fix(lockfile): request SYNCHRONIZE, or the Windows wait cannot run The first attempt opened the process with `QUERY_LIMITED_INFORMATION` alone. `WaitForSingleObject` requires `SYNCHRONIZE`, so the wait did not return `WAIT_TIMEOUT` for a running process — it returned `WAIT_FAILED`, which the comparison read as "not running", restoring exactly the bug the function was written to fix. Windows CI caught it a second time, on the same assertion. Worth the comment it now carries: omitting an access right does not make the check stricter, it makes the call fail, and a failed liveness check that reads as "dead" is indistinguishable from the stub this replaced. * docs: say Windows is supported, now that it is tested The install section listed a Windows binary; nothing else did. The two translated READMEs never mentioned Windows at all — they still offered macOS and Linux only — and the docs site said the same. A reader on Windows had no way to tell the tool was for them. All three READMEs and the site now carry a platform badge and name Windows alongside macOS and Linux, with what to do from PowerShell, where a `curl | sh` line is no help: take the msvc archive, verify its checksum, put `reify.exe` on PATH. The claim is only made because it is now true. Every listed platform runs the full suite in CI and has the CLI exercised end to end — the sentence saying so is in the README because it is the difference between this and the previous release, which published a Windows binary whose index lock did nothing.
Configuration menu - View commit details
-
Copy full SHA for fcd52cc - Browse repository at this point
Copy the full SHA fcd52ccView commit details -
release: v0.2.3 — Windows works, and is now tested
For Windows users this is a correctness fix rather than a feature. The index lock in 0.2.2 did nothing there: `process_is_alive` was implemented for unix and stubbed to `false` elsewhere, so every lock looked stale and reclaimed itself, and two `reify index` runs could write the same store. No configuration avoids it; upgrading is the fix. It was found by testing on the platforms releases actually ship for. CI ran on Linux only, so the Windows binary added in 0.2.2 went out on the strength of a type-check — which is exactly what a type-check cannot catch. Windows and macOS now run the full suite and an end-to-end CLI smoke; the job found the lock bug in its first run, by failing to recognise its own process as alive. Also in this release: retrieval on SWE-bench Verified 84.6% -> 87.0% at three rounds, from capping how much of the window one file may claim; the end-to-end arm ahead of BM25 for the first time, 73.3% against 67.3%, though not significantly so at p = 0.24; three more MCP tools within the same token ceiling; and an index that no longer fails outright when git cannot read history on a partial clone.
Configuration menu - View commit details
-
Copy full SHA for d22bbc2 - Browse repository at this point
Copy the full SHA d22bbc2View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v0.2.2...v0.2.3