docs: SWE-Bench Qwen3-Coder-7B example (32%→76.4%, 4, one night) - #9
Closed
Protocol-zero-0 wants to merge 3 commits into
Closed
docs: SWE-Bench Qwen3-Coder-7B example (32%→76.4%, 4, one night)#9Protocol-zero-0 wants to merge 3 commits into
Protocol-zero-0 wants to merge 3 commits into
Conversation
- Replace test coverage example with game AI evolution (35→72% win rate) — more visceral, shows self-correction on rejection, closer to AlphaEvolve spirit - Add progress bar visual (before/after) and round-by-round terminal output - Add callout on why Round 3 rejection matters (memory / no repeated mistakes) - Update architecture Mermaid with emoji labels and loop subgraph label - Full rewrite of README.zh.md as faithful Chinese translation (was stale v0) - Fix zh-CN badge URLs and remove Token-Ignition/legacy references Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
$34, one overnight run, frozen 7B model goes from 32.4% to 76.4% on SWE-Bench Verified — matching Mistral Medium 3.5 and Qwen3.6-27B. Shows 5 generations of reasoned hypothesis: voting → ACI editor → ast-grep dep scan → diagnose-then-retry → TDD mode. Gen 09 is the tell: LLM reads the ledger, diagnoses multi-file dep failures, reaches for ast-grep unprompted. Chinese README updated to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Updates the project’s documentation to replace the prior introductory example content with a SWE-Bench Verified benchmark scenario, and refreshes both English and Chinese READMEs to better explain the closed-loop “evolution” workflow, ledger artifacts, and CLI usage.
Changes:
- Reworks the main example narrative to a SWE-Bench Verified improvement run (Qwen3-Coder-7B 32.4% → 76.4%) and updates the Quick Start accordingly.
- Expands/modernizes the README structure: loop steps table, ledger/audit explanation, architecture diagram, CLI/install/test sections.
- Applies corresponding updates to
README.zh.mdto keep it aligned with the English README.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
| README.md | Replaces the main example with a SWE-Bench scenario and restructures sections describing the loop, ledger, and CLI usage. |
| README.zh.md | Mirrors the README restructure and SWE-Bench example content in Chinese. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| [gen 21] STOP — 4 generations with no significant improvement | ||
|
|
||
| {"halted": true, "reason": "max_consecutive_failures", "iterations": 21, |
|
|
||
| [gen 21] STOP — 连续 4 代无显著改进 | ||
|
|
||
| {"halted": true, "reason": "max_consecutive_failures", "iterations": 21, |
| ``` | ||
| ledger/ | ||
| .evolution_state.json # persisted counters (iterations, usd, tokens) — survives restarts | ||
| .evolution_state.json ← budget counters; survives restarts |
Comment on lines
175
to
+181
| 0001/ | ||
| config.json # full snapshot of your evolution.yml | ||
| observation.json # raw output of your evidence_sources commands | ||
| plan.json # LLM plan: summary, steps, expected_improvement | ||
| patch.diff # exact diff the executor applied | ||
| candidate_commit.txt # git SHA of the sandbox commit | ||
| evaluation.json # verdict + metrics + cost_usd + tokens_used | ||
| decision.json # accept / reject + reason | ||
| reflection.json # one-line summary injected into the next round's history | ||
| 0002/ | ||
| ... | ||
| config.json ← full snapshot of your evolution.yml | ||
| observation.json ← raw output of your evidence_sources commands | ||
| plan.json ← LLM plan: summary · steps · expected_improvement | ||
| patch.diff ← exact diff the executor applied | ||
| candidate_commit.txt ← git SHA of the sandbox commit | ||
| evaluation.json ← verdict + metrics + cost_usd + tokens_used |
| --evaluator python3 /path/to/evaluator.py | ||
| ``` | ||
| ledger/ | ||
| .evolution_state.json ← 预算计数器，进程重启后依然有效 |
Comment on lines
+177
to
+180
| observation.json ← evidence_sources 命令的原始输出 | ||
| plan.json ← LLM 方案：摘要 · 步骤 · 预期改进 | ||
| patch.diff ← 执行器实际应用的 diff | ||
| candidate_commit.txt ← 沙箱 commit 的 git SHA |
| evolution-kernel --config evolution.yml --repo /path/to/repo --ledger /tmp/ledger | ||
|
|
||
| # Reset hard-stop counters to start a fresh session | ||
| # Reset budget counters after a halt |
| --config examples/evolution.yml \ | ||
| --repo examples/demo_target \ | ||
| --ledger /tmp/ek-ledger | ||
| # 触发 halt 后重置预算计数器 |
Comment on lines
+245
to
+246
| # Required — what "better" means for your project | ||
| mission: "Evolve the game AI to win at least 60% of games" |
Comment on lines
+245
to
+246
| # 必填——"更好"对你的项目意味着什么 | ||
| mission: "让游戏 AI 对内置对手的胜率达到 60% 以上" |
- Halted JSON: match actual CLI output format and reason string
("max_consecutive_failures reached (4)", not bare key + stats)
- .evolution_state.json: label as "hard-stop state" not "budget counters"
(also tracks iterations and consecutive_failures)
- Ledger file list: add planner_input.json, executor_input.json/output,
evaluator_input.json; clarify halted/ contains full run stats
- --reset CLI comment: "Reset all hard-stop state" not "budget counters"
- Config reference mission: align with SWE-Bench example, drop game-AI mission
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Owner
Author
|
Closing — superseded by docs/readme-final which cherry-picks cleanly onto current main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replace the game AI example with a more compelling real-world AI benchmark scenario:
$34. One overnight run. A frozen 7B model from 32.4% to 76.4% on SWE-Bench Verified — matching 30B closed models. Zero weight changes.
The example shows 5 generations of hypothesis-driven improvement:
Both
README.mdandREADME.zh.mdupdated with matching content.Documentation-only — no code changed.
🤖 Generated with Claude Code