From d44bb5304e04ded52714538e987937ce3686b9c4 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 03:49:10 +0800 Subject: [PATCH 01/10] =?UTF-8?q?docs:=20the=20three=20Windows=20toolchain?= =?UTF-8?q?=20axes=20=E2=80=94=20a=20design,=20and=20two=20corrections?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The architecture review found where the problems are. This is what to do about them, written after a round of questions that overturned two shapes in my own proposal. Both corrections are recorded in §0.2, because the wrong version looked equally reasonable and will otherwise be proposed again: - generalising `@system` to gcc/llvm is backwards. xlings is a user-space OS and mcpp minimises host dependence; `msvc@system` is a Windows concession, not a capability three families are missing. - a `windows_sdk = "..."` manifest key should not exist. The model already reserves the slot (`runtime_binding.cppm:26` documents `ucrt@...` and nothing populates it), and a version key would sit next to `_WIN32_WINNT` looking interchangeable while controlling a different thing — which is worse than not having it. The design separates three axes that are currently entangled: where the compiler came from, which SDK is used, and what the artifact ships. They take values independently — a managed toolset with the machine's SDK is what xrgui's CI does today — so any design that fuses them into one switch is wrong. It also records the finding that makes `mcpp pack` more than a missing feature: `ldd_parse` computes the dependency closure by RUNNING the binary (LD_TRACE_LOADED_OBJECTS), so it cannot cross an OS or an architecture by construction. Reading imports statically is what makes cross-packaging fall out rather than be added. The acceptance criteria are written so that none of them can be met by CI going green — two of them explicitly require a machine POORER than the CI runner, because "the verification environment is richer than the target" is the shape that produced most of this round's eleven defect layers. --- ...026-08-16-toolchain-architecture-review.md | 6 + ...-16-windows-toolchain-three-axes-design.md | 324 ++++++++++++++++++ 2 files changed, 330 insertions(+) create mode 100644 .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md diff --git a/.agents/docs/2026-08-16-toolchain-architecture-review.md b/.agents/docs/2026-08-16-toolchain-architecture-review.md index 6ed89763..300a49cb 100644 --- a/.agents/docs/2026-08-16-toolchain-architecture-review.md +++ b/.agents/docs/2026-08-16-toolchain-architecture-review.md @@ -8,6 +8,12 @@ > 三处"不可能失败的测试"。下面每一条都指得出具体的 `file:line`, > 不是风格偏好。 +> ⚠️ **§1 / §2 / §3 / §3d 已被方案取代**: +> `2026-08-16-windows-toolchain-three-axes-design.md`。 +> 那份追问推翻了这里两处形状 —— `gcc@system` 的推广方向、以及新加 +> `windows_sdk` manifest 键 —— 两处都在方案的 §0.2 里写明了为什么错。 +> 本文其余各节(§3b / §3c / §4 / §5 / §6 / §7)已落地,保留作记录。 + --- ## 0. 结论先行 diff --git a/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md new file mode 100644 index 00000000..735f25e6 --- /dev/null +++ b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md @@ -0,0 +1,324 @@ +# Windows 工具链的三条轴:来源、SDK、运行时(2026-08-16) + +> 取代 `2026-08-16-toolchain-architecture-review.md` 的 §1 / §2 / §3 / §3d。 +> 那份是**发现**(问题在哪、file:line),这份是**方案**(怎么改、怎么算通过)。 +> review 的 §3b / §3c / §4 / §5 / §6 / §7 已落地,不在此文范围。 +> +> 起因是 review 之后的一轮追问,它推翻了我原方案里的两处形状错误 —— +> 两处都记在 §0.2,因为**错的那版看起来同样合理**,不写下来会被重新提出来。 + +--- + +## 0. 先把三样东西分开 + +### 0.1 今天被混在一起的三层 + +Windows 上一次构建牵涉三个**互相独立**的问题,而现在它们纠缠在一起: + +| 轴 | 问的是什么 | 今天在哪 | 状态 | +|---|---|---|---| +| **来源** | 编译器**哪来的**:机器上的 VS,还是 xim 装的 payload | `is_system_toolchain()` `registry.cppm:428` | ✅ 已建模,但代价摊在 26 处 | +| **SDK** | 用**哪套**头/导入库/工具 | `find_windows_sdk()` `msvc.cppm:628` | ❌ **靠搜**,没有身份 | +| **运行时分发** | 产物**带不带** `vcruntime140.dll` | `distribution.cppm:432` | ❌ 对 MSVC **一律拒绝** | + +**判据:三个问题的答案可以两两独立取值。** 受管 toolset 可以配系统 SDK(今天 xrgui CI 就是), +系统 VS 也带自己的 Redist。任何把它们绑成一个开关的设计都是错的。 + +### 0.2 两处已被推翻的形状(不要重新提出来) + +**(a) 不要把 `@system` 推广到所有族。** + +初稿把 `is_system_toolchain()` 里的 `Family::Msvc` 读成"不对称,应该补齐",提议 +支持 `gcc@system`。**方向反了**:xlings 是用户态 OS,mcpp 建立在它之上,整条设计 +就是**把 host 依赖降到最低**。`msvc@system` 是 **Windows 的让步** +(Visual Studio 常常已经装了、又不总能重新分发),不是一种缺失的通用能力。 + +顺带记一笔:真正提供 host 依赖的那一处,拼写是**不带族的 `system`** +(`prepare.cppm` 里 `*tcSpec == "system"` 那条 `// Explicit user opt-in to system +PATH compiler — kept as escape hatch`)。别再把它当成"缺失的能力"。 + +**(b) 不要新加 `windows_sdk = "..."` 这个 manifest 键。** + +理由有两层: + +1. **模型里已经有位置了。** `runtime_binding.cppm:26` 的 `runtimeId` 注释原文就是 + *"Provider-native runtime identity (`glibc@...`, `macos_sdk@...`, `ucrt@...`)"* + —— `ucrt@...` **早就写着**,只是全仓库没有一处填它,而消费者硬编码 `glibc@` 前缀 + (`runtime_validation.cppm:387`)。 +2. **它和 `_WIN32_WINNT` 会变成两个看起来能互相替代、实际管不同事的旋钮**,见 §2.4。 + +--- + +## 1. 轴一:来源 —— 收拢特例,不是推广它 + +### 1.1 问题 + +`is_system_toolchain()` 的判据本身没问题。问题是**这个平台特例的代价被摊开了**: +lifecycle.cppm **13 处**、prepare.cppm **13 处** msvc 专有分支,而 gcc / llvm / mingw +在 lifecycle.cppm 里**一处都没有**(mingw 更是零分支 —— 它整个被表达成一个 +*target*,`x86_64-windows-gnu`,那才是这套设计想要的形状)。 + +另外三处是**同一条规则的重复拼写**,与特例本身无关: + +1. `xim_tool()` + `installation_at()` + 错误信息 —— prepare 与 lifecycle 几乎逐字重复 +2. sysroot 依赖规则两份且**不等价**(一份只看 musl,一份 musl + PE + 宿主), + 而注释声称它们互相对应 +3. 解析链的注释说 4 步,实际有 **9 个输入** + +### 1.2 方案 + +把"来源"变成一个**解析一次**的值,而不是每个子命令自己再问一遍: + +```cpp +enum class Origin { Managed, SystemMsvc }; // 只有两种,而且只会有两种 +``` + +- `prepare` / `lifecycle` 各在入口解析一次,之后按它分发 +- `parse_toolchain_spec` 对**非 msvc 的 `@system`** 显式报错并指出替代写法 + (今天是"未实现",于是失败发生在别处、消息说的是别的事) + +配套消掉上面三处重复:提 `resolve_managed_msvc(env, pkg)`;合并 sysroot 谓词; +把解析顺序变成一张**数据表**,让注释无处可撒谎。 + +### 1.3 边界 + +- **裸 `gcc` 的语义不能变**(那会改现有 manifest 的行为)。只有显式 `@system` 是新语义 +- 裸 `system`(不带族)那条逃生口**保留**,它是有意的 + +--- + +## 2. 轴二:SDK —— 绑定,而不是搜索 + +### 2.1 问题 + +编译器拿到了版本轴,**SDK 没有**。`find_windows_sdk()` 按顺序扫: +`WindowsSdkDir` → 兄弟 xpkgs → 写死的 `C:\Program Files (x86)\Windows Kits\10`。 + +但对 `msvc@`,SDK 是 recipe 里**声明过的依赖** +(`xim:windows-sdk@10.0.26100`),位置是确定的。把确定的东西拿去搜,三个后果: + +1. **环境能悄悄改写钉住的选择** —— 一个游离的 vcvars 设了 `WindowsSdkDir`, + manifest 钉的 SDK 就失效了 +2. **今天那个 LNK1104 就是搜出来的** —— 半装的 payload 因为版本号更高, + 排在机器自己那套完整 SDK 前面,每个 TU 都编过,最后炸在链接, + **日志里没有一行提到 SDK** +3. **两台机器仍可能用不同 SDK 编同一份源码** —— 正是版本轴当初要解决的那个问题, + 只是换到了下一层 + +### 2.2 方案:按来源分流 + +``` +msvc@ → SDK 从同一条 xim 解析拿(和编译器同一个机制) +msvc@system → 维持现在的搜索链(机器上的东西只能靠找) +``` + +受管那条不再"搜到就用",而是**问 toolset 声明了哪个 SDK 依赖**。搜索链降级为 +`msvc@system` 专用。 + +**`WindowsSdkDir` 的地位要一起想清楚。** 它是"有人明确指定",按"声明压过探测" +应当优先;但对受管 toolset 它又会破坏可复现性。建议: + +- `msvc@system`:`WindowsSdkDir` 优先(维持今天) +- `msvc@`:**忽略它,但要说出来** —— 一行 `note:`,别静默 + +理由是这条轴的全部意义就是"声明什么就用什么",而静默覆盖会让它失效且不可见。 + +### 2.3 运行时身份:填那个留了很久的槽 + +`runtimeId = "ucrt@10.0.26100"`,从 toolset 的 SDK 依赖投影而来,进 +`runtimeContractHash`。 + +**但它和 glibc 不完全同构,这个差别必须写进代码注释**: + +| | Linux | Windows | +|---|---|---| +| `glibc@2.39` 绑的是 | 一个 **payload**:头 + `.so` 都在里面 | — | +| 能不能真绑上去 | ✅ patchelf 让产物**真的跑在那份 glibc 上** | ❌ `ucrtbase.dll` 在系统里,**换不掉也不该换** | +| 于是标识的含义 | **运行时绑定** | **兼容性下限声明** | + +我们的 `windows-sdk` 包**只装了 ucrt 的一半**(头 + 导入库), +**没有** `Universal CRT Redistributable`,因为 Win10 起 `ucrtbase.dll` 是系统组件、 +应用不该自带。这是**有意的**,要写进注释,否则下一个人会以为能像 glibc 那样 +把 ucrt 打进产物。 + +配套:`runtime_validation.cppm:387` 与 `post_install.cppm` 里硬编码的 `glibc@` 前缀 +要从"只认 glibc"改成"按 provider 分派",否则填了也没人认。 + +### 2.4 **不要**加 manifest 的 SDK 版本键 + +一般 Windows 开发者怎么处理 SDK: + +| 做法 | 普遍程度 | 后果 | +|---|---|---| +| VS 装一个"最新",`10.0`(字面意思就是"本机最新") | 绝大多数 | **同一份源码在两台机器上链到不同 SDK,且无提示** | +| 钉死 `10.0.22621.0` | 有纪律的团队 | 声明路线 | +| xwin / msvc-wine 自己下 payload | Rust 圈 / 交叉编译 | 本质就是我们做的事 | + +**默认(跟着 toolset 依赖走)已经强过主流做法** —— 它至少保证每台机器一致, +而主流做法连这一点都不保证。 + +差别到底在哪,分两半: + +| | 影响 | 为什么 | +|---|---|---| +| **ucrt 那一半** | **很小** | Win10 起 `ucrtbase.dll` 是系统组件且向后兼容 | +| **um(Win32 API)那一半** | **会咬人** | 新 API 只存在于新 SDK 的头/导入库 | + +两种失败难度天差地别: + +- 链到**老** SDK 用了新 API → **编译期**失败,一眼看出 ✅ +- 链到**新** SDK 但跑在老 Windows → **运行期**失败(加载失败 / `GetProcAddress` 返回 null)❌ + +**而真正声明"我承诺跑在哪个 Windows"的是 `_WIN32_WINNT` / `WINVER` 宏,不是 SDK 版本。** +SDK 版本决定的是"我能看见哪些 API"。加一个 `windows_sdk =` 键会让用户以为它就是 +兼容性下限旋钮 —— **两个旋钮看起来能互相替代,实际管不同事**,这是比不做更糟的结果。 + +结论:**这一条不做**。真有少数场景要钉,先把 `_WIN32_WINNT` 的表达方式想清楚, +再决定要不要第二个旋钮。 + +--- + +## 3. 轴三:运行时分发 —— `toolchain-coupled` 对 MSVC 是成立的 + +### 3.1 问题 + +`distribution.cppm:432` 说: + +> `cxx_runtime = "toolchain-coupled"` has no meaning for the MSVC runtime +> (it ships with the OS/redistributable, **not with the toolchain**) + +这句话对 `msvc@system` 是对的,对 `msvc@` 是**错的** —— 受管 payload 里就有 +`VC/Redist/MSVC///Microsoft.VC143.CRT/{vcruntime140,msvcp140,...}.dll`, +而且 VS 自身安装也带同一个目录,所以**两条来源都能支持**。 + +于是分发矩阵上有个洞: + +| | 自包含 | 宿主耦合 | 工具链耦合 | +|---|---|---|---| +| gcc / libstdc++ | 静态链接 | 用系统 libstdc++ | **拷 libstdc++ 到产物旁** ✅ | +| MSVC | `/MT` ✅ | `/MD` ✅ | **拒绝** ❌ | + +### 3.2 已落地的一半 + +`vc_redist_dir()` 已经把 toolset 自带的那份 CRT 放进 `linkRuntimeDirs`, +Windows 上它就是 PATH —— 所以 **`mcpp run` 已经能在只有受管 toolset 的干净机器上 +跑起来**(在此之前默认 `/MD` 产物会以"找不到 vcruntime140.dll"失败, +而 CI 看不到,因为 runner 上装着 VS)。 + +### 3.3 还差的一半 + +**把那些 DLL 拷到产物旁**,即 PE 上真正实现 `toolchain-coupled` —— 和 gcc 拷 +libstdc++ 是同一件事、同一个语义。 + +⚠️ **只取 `/Microsoft.VC*.CRT/`,不能取 `debug_nonredist/`** —— +后者(`vcruntime140d.dll` 等)**不可再分发**。这条要有测试钉住, +`vc_redist_dir()` 已经这么做了,拷贝那一步要沿用同一个判据。 + +### 3.4 三层的最终形状 + +``` +ucrt → SDK 给导入库,运行时由 OS 提供 → 下限声明(runtimeId) +vcruntime → toolset 自带 Redist,可随产物走 → cxx_runtime(§3) +um/shared → SDK 给,纯链接期 → 无运行期对应物 +``` + +--- + +## 4. 打包:读二进制,别运行它 + +### 4.1 问题比"没做 Windows"更深 + +`pack.cppm:323` 的 `ldd_parse` 求依赖闭包的方式是: + +``` +LD_TRACE_LOADED_OBJECTS=1 '' +``` + +**它要把目标二进制跑起来。** 所以这不是"缺一个 PE 分支": + +- **跨不了 OS**(Linux 上跑不了 PE) +- **跨不了架构**(x86_64 上跑不了 aarch64 产物,即便同是 Linux) + +`pack.cppm:638` 那条 `#if defined(_WIN32)` 的拒绝只是结果,不是原因。 + +### 4.2 分层 + +| 层 | 职责 | 平台相关? | +|---|---|---| +| **契约** `distribution.cppm` | 决定**该带什么** | ❌ **已存在** | +| **闭包** | 静态读导入表:ELF `DT_NEEDED` / PE 导入表 / Mach-O `LC_LOAD_DYLIB` | ✅ 纯解析 | +| **系统库判据** | ELF 现有那张表 / PE 的 kernel32、user32、ucrtbase… / Mach-O `/usr/lib`、`/System` | ✅ | +| **重定位** | ELF: patchelf `$ORIGIN/../lib` / **PE: 无操作**(DLL 放 exe 旁就是规则) / Mach-O: `install_name_tool` + `@loader_path` | ✅ | +| **打包** | tar.gz / zip | ✅ | + +**换成静态解析之后,跨 OS 打包不是额外功能,是自然结果** —— 没有任何一步需要执行产物。 + +### 4.3 还有一处断链 + +**`pack` 从不读 Contract。** `MechanismInput.msvcStaticCrt` 只影响编译链接**旗标**; +`pack` 决定真拷哪些文件,却不认识 `cxx_runtime`。所以那个契约在打包这一步 +**没有执行者** —— ELF 上靠 `ldd` 闭包歪打正着,PE 上完全没有。 + +把第一层接到第二、四层,才是"一套系统"而不是"两套"。矛盾时要报出来 +(`pack` 已有"带自己 libc 的包不能消费宿主库"这类拒绝的先例,照着长)。 + +--- + +## 5. 兼容性与迁移 + +| 改动 | 是否改用户可见格式 | 迁移 | +|---|---|---| +| §1 Origin 收拢 | ❌ 纯内部 | 无 | +| §1 拒绝 `gcc@system` | ⚠️ 之前"未实现"、现在显式报错 | 消息要给替代写法 | +| §2 SDK 绑定 | ❌ 行为变更,非格式 | 受管 toolset 上 `WindowsSdkDir` 从"生效"变"忽略+提示" | +| §2 `ucrt@` 身份 | ⚠️ 进 `runtimeContractHash` → **缓存键变** | 一次性全量重建,和别的 contract 变更同类 | +| §3 PE toolchain-coupled | ❌ 新增能力 | 之前是 degraded,现在能满足 | +| §4 PE pack | ❌ 新增能力 | 之前是硬错误 | + +**唯一需要留意的是 §2 的 hash 变更** —— 它会让 Windows 上已有的构建缓存失效一次。 + +--- + +## 6. 验收判据(每条怎么算通过) + +**没有一条可以靠"CI 绿了"算过。** 这一轮十一层缺陷,每一层出现时 CI 都是绿的。 + +| # | 判据 | 为什么它不能被自我满足 | +|---|---|---| +| §1 | 26 处分支的数量**下降**,且 `gcc@system` 报错里出现替代写法 | 数量可数;消息内容有断言 | +| §2 | 受管 toolset 下,**设一个指向别处的 `WindowsSdkDir`,构建仍用 payload 的 SDK**,且打印了 note | 环境能覆盖=没绑定,这条直接证伪 | +| §2 | `runtimeContractHash` 在 Windows 上随 SDK 版本变化 | 不变就是没进 hash | +| §3 | **在一台没有 VS、没装过 redist 的干净 Windows 上**,`/MD` 产物拷到别处能跑起来 | CI runner 有 VS,**必须用干净机器或容器**,否则这条测不出东西 | +| §3 | 产物旁**没有** `*d.dll` | debug CRT 不可再分发 | +| §4 | **在 Linux 上**为 Windows 产物打出 zip,内含正确 DLL 闭包 | 跨 OS 是这条的全部意义;同 OS 打包证明不了 | +| §4 | `cxx_runtime` 与 `--mode` 矛盾时**报出来** | 静默通过=契约无执行者 | + +> §3 和 §4 的判据都要求**比 CI 环境更贫瘠的机器**。这一轮反复出现的形状是 +> "验收环境比目标环境富裕",这两条是直接针对它写的。 + +--- + +## 7. 落地顺序 + +| # | 项 | 规模 | 依赖 | +|---|---|---|---| +| 1 | §3.3 PE `toolchain-coupled`(拷 DLL) | 中 | 无。`vc_redist_dir()` 已在 | +| 2 | §2.2 SDK 按来源分流 | 中 | 无 | +| 3 | §2.3 `ucrt@` 身份 + 解开 `glibc@` 硬编码 | 中 | 建议紧跟 2 | +| 4 | §1 Origin 收拢 + 消三处重复 | 大 | 建议在 2/3 之后 —— 都动同一片代码 | +| 5 | §4 PE pack | 大 | 排在 1 之后:先解决"能跑",再解决"能分发" | + +**§2.4(manifest SDK 版本键)不做**,理由见该节。 + +前三项互不依赖,可并行。§1 和 §5 各自单独一轮。 + +--- + +## 8. 一句话 + +这三条轴今天被搅在一起,是因为**只有编译器那一条被显式建模过**。 +SDK 靠搜、运行时靠拒绝、打包靠一个要跑起来才work的原语 —— +每一处都在"环境恰好合适"时正常工作,而在贫瘠环境里失败, +**且失败信息不指向真正的原因**。 + +把三条轴各自变成一个**声明出来的、解析一次的值**,是这份方案唯一在做的事。 From a557310725fe88f1945fc0fc9b441df367a6f668 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 04:26:17 +0800 Subject: [PATCH 02/10] feat(windows): give the SDK and the runtime the axes the compiler already had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements §1 / §2 / §3 of .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md. Three questions were entangled because only one of them had ever been modelled. The compiler got a version axis in the last round; the headers it compiles against and the runtime the artifact loads did not. §2 THE SDK IS BOUND, NOT SEARCHED. `find_windows_sdk()` scanned — WindowsSdkDir, then the sibling store, then the conventional roots — for BOTH origins. So a pinned `msvc@` was a pin the environment could overwrite, and two machines could build one manifest against two SDKs with nothing in the log naming either. It is now resolved by origin: a managed toolset takes the SDK payload from its own store and ignores WindowsSdkDir/WindowsSdkVersion *out loud*; `msvc@system` keeps today's chain, because a machine's things can only be found by looking. A managed toolset with no SDK payload beside it still falls back to the machine's — working beats failing — and says so, because that build is no longer reproducible and only that line records it. §2.3 `ucrt@` FILLS A SLOT THAT HAS BEEN RESERVED SINCE THE FIELD EXISTED. `RuntimeBinding::runtimeId`'s comment has documented it from the start and nothing ever wrote one, so the SDK version never reached `runtimeContractHash` and two SDKs shared one build cache. It is NOT isomorphic to `glibc@`, and the comment says so where it will be read: glibc@ binds a payload (headers + .so, patchelf makes the artifact run on that copy), ucrt@ declares a floor (ucrtbase.dll is an OS component from Win10 on; mcpp's windows-sdk payload deliberately carries only half of ucrt and no redistributable). It is therefore not projected into `libc`. The `glibc@`-prefix gates become `runtime_provider()` dispatch, so another provider reads as "no rules here" rather than "no identity". §3.3 `toolchain-coupled` NOW MEANS SOMETHING ON PE. The refusal said the MSVC runtime "ships with the OS/redistributable, not with the toolchain" — true of ucrtbase.dll, false of vcruntime140.dll/msvcp140.dll, which sit in VC\Redist\MSVC\ inside every toolset. That is the relationship gcc has to libstdc++.so, so it takes the same contract; PE has no rpath, so the mechanism is a copy beside the artifact rather than a search path. /MT stays a degradation, and a genuine one: a static CRT leaves no DLL to couple to. The DLL set comes from `vc_redist_dir()` — the single criterion that excludes `debug_nonredist\`, which may not be redistributed. A second, name-shaped rule here could disagree with it, and disagreeing about that is a licensing defect rather than a bug. §1 THE ORIGIN AXIS IS CONTAINED, NOT GENERALISED. - `gcc@system` / `llvm@system` are refused where they are read, naming both things the user might have meant. They used to parse and then fail elsewhere as `xim:gcc@system` → "no such package", sending the reader after a version that was never going to exist. `msvc@system` is a concession to one platform, not a capability the other families lack; the family-less `system` escape hatch is untouched. - `resolve_managed_msvc()` replaces two hand-written copies of "where does a managed toolset live, and why is the fetcher's `root` wrong for it" — the reason existed in only one of them. - `needs_linux_sysroot_payloads()` replaces two spellings of one rule whose comment claimed they mirrored each other. They did not: the PE term was missing from one. Unreachable today, which is how it survived. - The toolchain resolution order was documented twice, as "3 steps" and "4 steps", naming five of the nine inputs and disagreeing about two. One table now, keyed to `TcOrigin` enumerators so it cannot quietly stop matching. `dist::Format` is derived from the target triple before falling back to the host, which only ADDS answers — and makes a Windows contract assertable on the Linux runner that reviews most of this. Tests: 22 new. The SDK-override criterion is the design doc's §6 acceptance test as a unit test (point WindowsSdkDir elsewhere; the payload SDK must still win, and the note must say the variable was ignored); the deploy tests assert reachability twice over, since a copy edge nothing asks for never runs under explicit ninja goals. --- src/build/distribution.cppm | 53 ++++++- src/build/flags.cppm | 95 ++++++++++++ src/build/ninja_backend.cppm | 22 ++- src/build/prepare.cppm | 104 ++++++++++--- src/build/runtime_validation.cppm | 6 +- src/platform/elf_runtime.cppm | 7 +- src/platform/runtime_binding.cppm | 70 +++++++++ src/toolchain/lifecycle.cppm | 53 +++---- src/toolchain/model.cppm | 49 ++++++ src/toolchain/msvc.cppm | 201 ++++++++++++++++++++----- src/toolchain/post_install.cppm | 18 ++- src/toolchain/registry.cppm | 102 +++++++++++++ tests/unit/test_distribution.cpp | 80 ++++++++++ tests/unit/test_ninja_backend.cpp | 136 +++++++++++++++++ tests/unit/test_runtime_contract.cpp | 77 ++++++++++ tests/unit/test_toolchain_msvc.cpp | 167 ++++++++++++++++++++ tests/unit/test_toolchain_registry.cpp | 71 +++++++++ 17 files changed, 1204 insertions(+), 107 deletions(-) diff --git a/src/build/distribution.cppm b/src/build/distribution.cppm index 5f797342..b18bcf24 100644 --- a/src/build/distribution.cppm +++ b/src/build/distribution.cppm @@ -268,6 +268,16 @@ struct Mechanism { // global constructor that touches std::cout runs before the streams // exist. Asks the backend for the ordering shim. See issue #336. bool streamInitShim = false; + // PE + MSVC runtime + toolchain-coupled: the toolset's own redistributable + // CRT DLLs must be STAGED BESIDE the artifact. + // + // On ELF, `toolchain-coupled` needs no files copied — the artifact carries + // an rpath into the toolchain's lib directory and the loader follows it. + // PE has no rpath: a DLL is resolved from the directory of the executable + // (and then PATH), so on this format the mechanism IS the copy. Same + // contract, same meaning, different mechanism — which is exactly the split + // this module's three layers exist to express. + bool deployToolchainRuntime = false; }; namespace detail { @@ -426,12 +436,43 @@ Mechanism resolve(const MechanismInput& in) { "it everywhere; using host-coupled here" : ""; } else if (in.requested == Contract::ToolchainCoupled) { - // Only reachable from an explicit request: it is never a default. - m.degraded = true; - m.diagnostic = std::format( - "cxx_runtime = \"toolchain-coupled\" has no meaning for the " - "MSVC runtime (it ships with the OS/redistributable, not with " - "the toolchain); using {}", to_string(m.effective)); + // THIS USED TO BE A FLAT REFUSAL, and the sentence it refused + // with was half true: + // + // "…has no meaning for the MSVC runtime (it ships with the + // OS/redistributable, not with the toolchain)" + // + // True of `ucrtbase.dll`, which IS an OS component since + // Win10. NOT true of `vcruntime140.dll` / `msvcp140.dll`, + // which are the toolset's own and sit inside every MSVC + // toolset ever shipped: + // + // VC\Redist\MSVC\\\Microsoft.VC.CRT\*.dll + // + // That is the same relationship gcc has to libstdc++.so, so it + // takes the same contract — and refusing it left a hole in the + // matrix that had a real cost: the default `/MD` artifact + // depends on DLLs a machine with only a managed toolset does + // not have, and there was no spelling that made them travel. + // + // `/MT` is the one case that stays a degradation, and it is a + // genuine contradiction rather than a missing mechanism: a + // static CRT leaves NO DLL to couple to. Say which one won. + if (in.msvcStaticCrt) { + m.effective = Contract::SelfContained; + m.degraded = true; + m.diagnostic = + "cxx_runtime = \"toolchain-coupled\" cannot apply to a " + "project compiled with the static CRT (/MT): there is " + "no vcruntime140.dll/msvcp140.dll dependency left to " + "couple to. Drop linkage = \"static\" (or the " + "project-wide self-contained contract) if the toolset's " + "CRT should travel beside the artifact instead; using " + "self-contained"; + } else { + m.effective = Contract::ToolchainCoupled; + m.deployToolchainRuntime = true; + } } return m; } diff --git a/src/build/flags.cppm b/src/build/flags.cppm index d69895a8..2058e12d 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -77,6 +77,28 @@ struct CompileFlags { // macOS + self-contained: link units need the initializer-ordering shim // object prepended to their inputs (issue #336). bool needsStreamInitShim = false; + // PE + `toolchain-coupled`: the toolset's own CRT DLLs, to be staged + // beside the artifact. Resolved HERE rather than in the emitter because + // "which files does this contract imply" is a contract question; the + // backend only knows how to spell a copy edge. + // + // A whole-BUILD list, not a per-role one, and that is a property of the + // format rather than a simplification: a PE artifact resolves a DLL from + // its own directory, so one directory holds one answer and two roles in + // one output tree cannot disagree about it. Any built role asking for the + // contract is enough to populate it. + // + // The DIRECTORY comes from `msvc::vc_redist_dir()` via + // `Toolchain::linkRuntimeDirs`, which is what keeps `debug_nonredist\` + // (vcruntime140d.dll & friends — NOT redistributable) out of the list. The + // criterion lives in exactly one place on purpose: a second name-shaped + // rule here could disagree with it, and a copy step that disagrees about + // what may be redistributed is a licensing defect, not a bug. + // + // Already deduped against the plan's own deploy files, so the emitter can + // append without deciding anything: a name the manifest already claims + // stays the manifest's and the conflict is reported through `diagnostics`. + std::vector toolchainRuntimeDeploy; // Non-empty when a requested contract could not be honored. The caller // MUST surface these — a silent downgrade is the failure mode this whole // model exists to prevent. Emitted once by the backend, not here, because @@ -762,6 +784,22 @@ CompileFlags compute_flags(const BuildPlan& plan) { // produces a PE and must take the PE answer. const dist::Format format = [&] { if (isMingwTc) return dist::Format::Pe; + // The TARGET's own word, when it has one. `isMingwTc` was the only + // cross case this knew about, so every other question about the + // output format was answered by asking the HOST — which is right + // whenever they agree and unaskable in a test that does not run on + // the platform it is about. A triple that names its OS is a fact; + // the host is a stand-in for one. + // + // Only ADDS answers: a triple that says neither falls through to + // exactly the previous derivation, so no existing build changes. + const auto& t = plan.toolchain.targetTriple; + if (t.find("windows") != std::string::npos + || t.find("mingw") != std::string::npos) + return dist::Format::Pe; + if (t.find("apple") != std::string::npos + || t.find("darwin") != std::string::npos) + return dist::Format::MachO; if constexpr (mcpp::platform::needs_explicit_libcxx) return dist::Format::MachO; else if constexpr (mcpp::platform::is_windows) @@ -896,6 +934,8 @@ CompileFlags compute_flags(const BuildPlan& plan) { }); }; + bool wantsToolchainRuntime = false; + for (auto [role, requested, wasAsked] : { std::tuple{dist::Role::Distributable, base, explicitBase}, std::tuple{dist::Role::Test, testsContract, explicitTests}, @@ -910,10 +950,65 @@ CompileFlags compute_flags(const BuildPlan& plan) { f.ldStdlibCByRole[i] = r.unitFlagsC; f.contractByRole[i] = r.effective; if (r.streamInitShim) f.needsStreamInitShim = true; + // Only a role this build actually HAS may pull DLLs into the + // output tree. The contract is resolved for every role because + // `ldStdlibByRole` must be total; staging files is a side effect + // on disk, and a project with no test binaries should not get a + // CRT copied beside nothing. + if (r.deployToolchainRuntime && role_is_built(role)) + wantsToolchainRuntime = true; if (!r.diagnostic.empty() && role_is_built(role)) f.diagnostics.push_back(std::format( "{} target: {}", dist::to_string(role), r.diagnostic)); } + if (wantsToolchainRuntime) { + // `linkRuntimeDirs` is the toolset's own redistributable CRT + // directory and nothing else on this toolchain — `enrich_toolchain + // _from_cl` puts exactly `vc_redist_dir()` there. Guarded on the + // compiler anyway: the field means "the toolchain's private + // runtime" for every provider, and on gcc it holds libstdc++'s + // directory, which has no business being copied into a PE tree. + if (plan.toolchain.compiler == mcpp::toolchain::CompilerId::MSVC) { + std::vector sources; + std::error_code ec; + for (auto const& dir : plan.toolchain.linkRuntimeDirs) { + for (auto const& e : + std::filesystem::directory_iterator(dir, ec)) { + if (!e.is_regular_file(ec)) continue; + auto ext = e.path().extension().string(); + std::ranges::transform(ext, ext.begin(), + [](unsigned char c) { return std::tolower(c); }); + if (ext != ".dll") continue; + sources.push_back(e.path()); + } + } + // Directory order is not a stable input: this list reaches + // build.ninja, and a graph that differs between two runs of + // the same build re-runs edges for no reason. + std::ranges::sort(sources); + for (auto const& src : sources) { + auto dest = std::filesystem::path("bin") / src.filename(); + // An explicit `[runtime] deploy_files` naming the same DLL + // WINS, and says so. A human wrote that one down; this list + // is derived. Silently overwriting a vendored redist with + // the toolset's copy is a different program than the one + // the manifest describes. + auto clash = std::ranges::find_if(plan.runtimeDeployFiles, + [&](auto const& d) { return d.dest == dest; }); + if (clash != plan.runtimeDeployFiles.end()) { + if (clash->source.lexically_normal() + != src.lexically_normal()) + f.diagnostics.push_back(std::format( + "toolchain-coupled would stage '{}' beside the " + "artifact, but this project already deploys " + "'{}' there; keeping the project's file", + src.string(), clash->source.string())); + continue; + } + f.toolchainRuntimeDeploy.push_back({src, dest}); + } + } + } // Two roles usually share a contract, so they usually share a // complaint; report each distinct one once. std::ranges::sort(f.diagnostics); diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index a6acad5e..d4dcea6f 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -449,6 +449,18 @@ std::string emit_ninja_string(const BuildPlan& plan) { // All compile/link flags are computed once via flags.cppm. auto flags = compute_flags(plan); + // Everything that has to sit beside the artifact, from both producers: + // the manifest's `[runtime] deploy_files` (already in the plan) and the + // C++ runtime contract's own answer (`toolchain-coupled` on PE — see + // mcpp.build.distribution). Merged ONCE, here, because three places below + // consume the list — the implicit dependency of each executable, the copy + // edges, and `default` — and a list that is complete in two of them is a + // graph where the DLL is copied only when something else happens to ask. + auto deployFiles = plan.runtimeDeployFiles; + deployFiles.insert(deployFiles.end(), + flags.toolchainRuntimeDeploy.begin(), + flags.toolchainRuntimeDeploy.end()); + bool need_c_rule = false, need_asm_rule = false, need_nasm_rule = false; for (auto& cu : plan.compileUnits) { if (is_c_source(cu)) need_c_rule = true; @@ -1770,7 +1782,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { // beside the .exe before the build is considered done. Empty on RPATH // platforms (no *.dll deps), so other targets are unaffected. if (lu.kind == LinkUnit::Binary || lu.kind == LinkUnit::TestBinary) { - for (auto const& d : plan.runtimeDeployFiles) + for (auto const& d : deployFiles) implicit += " " + escape_ninja_path(d.dest); } @@ -1826,13 +1838,13 @@ std::string emit_ninja_string(const BuildPlan& plan) { // — which also means a DLL still loaded by a running program from a // previous `mcpp run` gets the skip-if-equivalent treatment instead of a // hard "cannot copy" failure. - // Inert on RPATH platforms where runtimeDeployFiles is empty. - for (auto const& d : plan.runtimeDeployFiles) { + // Inert on RPATH platforms where the merged deploy list is empty. + for (auto const& d : deployFiles) { append(std::format("build {} : stage_file {}\n", escape_ninja_path(d.dest), escape_ninja_path(d.source))); } - if (!plan.runtimeDeployFiles.empty()) + if (!deployFiles.empty()) append("\n"); // ── Declared build-graph nodes (`mcpp:action=`) ───────────────────────── @@ -1899,7 +1911,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { defaults += " " + escape_ninja_path(alias); } } - for (auto const& d : plan.runtimeDeployFiles) { + for (auto const& d : deployFiles) { defaults += " " + escape_ninja_path(d.dest); } defaults += actionDefaults; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 2284b0c5..ea425a80 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -921,10 +921,46 @@ prepare_build(bool print_fingerprint, } // ─── Toolchain resolution (docs/21) ──────────────────────────────── - // Priority chain: - // 1. mcpp.toml [toolchain]. → resolve_xpkg_path → abs path - // 2. $CXX env var - // 3. PATH g++ (with warning) + // + // THE WHOLE CHAIN, in the order it is applied. It was documented twice, as + // "3 steps" here and "4 steps" further down, and neither list had been + // true for a long time — between them they named five of the nine inputs + // below and disagreed about two. A comment that undercounts the inputs to + // a decision is worse than none: it tells the next reader they have seen + // the whole thing. + // + // The WHAT (which spec) is settled first, then the HOW (which binary). + // Anything that WRITES `tcSpec` also writes `tcOrigin`, and that is the + // invariant this table rests on — the enumerator names below are real, so + // this comment cannot quietly stop matching the code. + // + // WHICH SPEC tcOrigin + // 1. mcpp.toml [toolchain]. / .default ManifestToolchain + // 2. global config.toml [toolchain] default GlobalDefault + // 3. mcpp.toml [target.].toolchain TargetSection + // (--target / [build] target / config default + // select the section; 3 outranks 1 and 2) + // 4. the target vocabulary's pin (triple.cppm) TargetPin + // — a convention, and it stands down when a + // REMEMBERED target would overrule a spec the + // user wrote down + // 5. the platform's first-run default, installed FirstRun + // and persisted by this very invocation + // + // WHICH BINARY, from the spec settled above + // 6. `msvc@system` → probe the machine (no xim package exists) + // 7. `@` → xim payload; msvc resolves through + // resolve_managed_msvc, everything else through + // the bin/-shaped frontend lookup + // 8. bare `system` → the PATH compiler. A deliberate escape hatch, and + // the ONLY host-compiler route: there is no + // `gcc@system` (see parse_toolchain_spec) + // 9. offline / MCPP_NO_AUTO_INSTALL → hard error rather than a silent + // ~800 MB download + // + // AND ONE REVISION, after 1-9 have produced a toolchain: a spec targeting + // the MSVC ABI on a machine with no usable MSVC is switched to MinGW-w64 + // — but only when `tc_origin_is_user_explicit` says mcpp chose it itself. std::filesystem::path explicit_compiler; std::optional cfg_opt; bool bootstrap_checked = false; @@ -1000,11 +1036,10 @@ prepare_build(bool print_fingerprint, constexpr std::string_view kCurrentPlatform = mcpp::platform::name; - // M5.5: toolchain resolution priority: - // 0. --target X / --static, looked up in [target.] - // 1. project mcpp.toml [toolchain]. or .default - // 2. global ~/.mcpp/config.toml [toolchain].default - // 3. hard error (no system fallback) + // Toolchain resolution priority: see the table at the top of this + // function. Stated once, where `tcOrigin` is introduced — this used to be + // a second, shorter and differently-wrong list of the same thing. + // // Resolve the build profile, overlaid by any [profile.] from the // manifest → buildConfig. `effectiveProfile` outlives the block: the // build.mcpp env contract exposes it as MCPP_PROFILE. @@ -1331,20 +1366,13 @@ prepare_build(bool print_fingerprint, // (cl.exe is four levels deeper) and the ELF post-install fixup // (there is nothing to patchelf on a PE toolchain). if (spec->family == mcpp::toolchain::Family::Msvc) { - // Not `payload->root`: that field is the fetcher's guess at where - // the useful tree starts, and it descends into a lone - // subdirectory when the version dir has no bin/ include/ lib/. - // An msvc payload's only entry is `VC/`, so the guess lands one - // level too deep. (store, name, version) is known — use it. - auto verDir = mcpp::xlings::paths::xim_tool( - mcpp::config::make_xlings_env(**cfg), pkg.ximName, pkg.ximVersion); - auto inst = mcpp::toolchain::msvc::installation_at( - verDir, pkg.ximVersion); - if (!inst) { - return std::unexpected(std::format( - "msvc payload at '{}' has no cl.exe under VC/Tools/MSVC/{}", - verDir.string(), pkg.ximVersion)); - } + // One rule, one place: where a managed toolset lives and why the + // fetcher's `root` must not be used for it (mcpp.toolchain. + // registry). Install and build asked the same question and each + // answered it in its own words. + auto inst = mcpp::toolchain::resolve_managed_msvc( + mcpp::config::make_xlings_env(**cfg), pkg); + if (!inst) return std::unexpected(inst.error()); explicit_compiler = inst->clPath; mcpp::ui::info("Resolved", std::format( "{} → msvc {} ({})", spec->display(), @@ -1484,8 +1512,9 @@ prepare_build(bool print_fingerprint, mcpp::fetcher::InstallProgressHandler progress; // The glibc default toolchain needs the sysroot payloads (C library + - // kernel headers). The musl default is self-contained, so skip them. - if (!mcpp::platform::is_macos && !mcpp::platform::is_windows && !muslDefault) { + // kernel headers). One derivation, shared with `toolchain install` — + // see registry.cppm for what the two spellings used to disagree about. + if (mcpp::toolchain::needs_linux_sysroot_payloads(defaultParsed->target)) { for (auto dep : {"xim:glibc", "xim:linux-headers"}) { (void)fetcher.resolve_xpkg_path(dep, /*autoInstall=*/true, &progress); } @@ -1553,6 +1582,31 @@ prepare_build(bool print_fingerprint, explicit_compiler, runtimePayload, runtimeBindingSnapshot.contractHash); if (!tc) return std::unexpected(tc.error().message); + // Something about the resolution the user has to be told, but which is + // not a failure. Today's only producer is the Windows SDK axis: a managed + // toolset binds the SDK it was installed with, so a `WindowsSdkDir` in + // the environment does not apply — and an override that is ignored + // SILENTLY is indistinguishable from one that was never set. + if (!tc->resolutionNote.empty()) + mcpp::ui::info("note", tc->resolutionNote); + + // The Windows runtime identity, flowing BACK into the contract. + // + // Everything else about the runtime is known before a toolchain is + // resolved, and deliberately so (see the RuntimeBinding block above). The + // Windows SDK is the exception: it is a property of the toolchain, and + // until it reached the contract hash the version axis simply did not + // exist one layer below the compiler — two SDKs produced one cache key. + // + // `ucrt@` is a COMPATIBILITY FLOOR, not a payload binding like + // `glibc@`: ucrtbase.dll is an OS component and mcpp ships no + // redistributable for it. See mcpp.platform.runtime_binding. + if (!tc->windowsSdkVersion.empty()) { + mcpp::platform::runtime::bind_windows_ucrt( + runtimeBindingSnapshot, tc->windowsSdkVersion); + tc->runtimeContractHash = runtimeBindingSnapshot.contractHash; + } + // ── Targeting the MSVC ABI without a usable MSVC ───────────────────── // // One judgement, one place. This used to be two separate concerns and diff --git a/src/build/runtime_validation.cppm b/src/build/runtime_validation.cppm index 88a743c0..f80a171a 100644 --- a/src/build/runtime_validation.cppm +++ b/src/build/runtime_validation.cppm @@ -383,8 +383,12 @@ ValidationReport validate_changed_artifacts( const ArtifactSnapshot& before) { ValidationReport report; if constexpr (!mcpp::platform::is_linux) return report; + // Provider dispatch (see mcpp.platform.runtime_binding): what follows is + // ELF/glibc physics, and an identity from another provider — `ucrt@…` on + // Windows — has no rules here rather than a missing glibc. if (plan.runtimeBinding.platform != "linux" - || !plan.runtimeBinding.runtimeId.starts_with("glibc@")) + || mcpp::platform::runtime::runtime_provider( + plan.runtimeBinding.runtimeId) != "glibc") return report; auto doc = read_cache(plan.outputDir); diff --git a/src/platform/elf_runtime.cppm b/src/platform/elf_runtime.cppm index 1be390cb..d4af87a9 100644 --- a/src/platform/elf_runtime.cppm +++ b/src/platform/elf_runtime.cppm @@ -697,7 +697,12 @@ RuntimeVerdict validate_runtime_artifact( const RuntimeResolution& resolution, bool hostLibsAllowed) { RuntimeVerdict verdict; - const bool isGlibc = binding.runtimeId.starts_with("glibc@"); + // Provider dispatch, not a prefix match. These are ELF/glibc physics — + // PT_INTERP, DT_RUNPATH, a private loader — and a `ucrt@…` identity is + // not a glibc that is missing, it is a different provider whose runtime + // this validator has no rules for. + const bool isGlibc = + mcpp::platform::runtime::runtime_provider(binding.runtimeId) == "glibc"; if constexpr (!mcpp::platform::is_linux) { verdict.diagnostics.push_back( "runtime physics: non-Linux platform; ELF/glibc rules are not applicable"); diff --git a/src/platform/runtime_binding.cppm b/src/platform/runtime_binding.cppm index 94550ed2..c727cb74 100644 --- a/src/platform/runtime_binding.cppm +++ b/src/platform/runtime_binding.cppm @@ -23,6 +23,26 @@ struct RuntimeBinding { std::string arch; // Provider-native runtime identity (`glibc@...`, `macos_sdk@...`, // `ucrt@...`). Only Linux additionally projects this into `libc`. + // + // THE PROVIDERS ARE NOT ISOMORPHIC, and treating them as such is the + // mistake this note exists to prevent: + // + // glibc@2.39 binds a PAYLOAD. The headers and the .so are both in it, + // and patchelf makes the artifact actually run on that + // copy. It is a runtime BINDING. + // ucrt@10.0… declares a FLOOR. `ucrtbase.dll` is an OS component from + // Win10 on; it cannot be swapped and should not be + // shipped, so mcpp's `windows-sdk` payload deliberately + // carries only half of ucrt (headers + import libraries) + // and no `Universal CRT Redistributable`. The identity + // says which API surface was compiled against, not which + // binary will be loaded. + // + // Both belong in this field — they are the same QUESTION ("which C + // runtime is this artifact built for") answered by different providers — + // but a consumer that assumes the glibc shape will try to resolve a + // payload that was never supposed to exist. Ask `runtime_provider()` + // before acting on the value. std::string runtimeId; std::string contractHash; std::optional loader; @@ -75,6 +95,42 @@ struct RuntimeBinding { bool hermetic() const { return loader.has_value(); } }; +// The PROVIDER half of a runtime identity: "glibc" from "glibc@2.39", "ucrt" +// from "ucrt@10.0.26100.0". Empty when there is no identity at all. +// +// Exists so consumers DISPATCH instead of pattern-matching. Every reader of +// `runtimeId` used to spell `starts_with("glibc@")`, which reads as "is this +// glibc" and behaves as "is this the only provider I have ever seen" — so the +// day a second provider appeared, each of those sites silently classified it +// as "no runtime identity" rather than "an identity with no rules here". The +// two are not the same thing, and only one of them is worth reporting. +std::string_view runtime_provider(std::string_view runtimeId) { + auto at = runtimeId.find('@'); + if (at == std::string_view::npos) return {}; + return runtimeId.substr(0, at); +} + +// Attach the Windows C runtime identity to an already-resolved snapshot, and +// re-derive the contract hash so it takes effect. +// +// WHY IT IS A SECOND STEP rather than part of resolve_runtime_binding(): the +// SDK version is a property of the TOOLCHAIN, and the binding is deliberately +// resolved BEFORE any toolchain — the post-install fixup is itself a consumer +// of the binding, so resolving a toolchain first would let directory order +// choose a libc and only then discover what the project selected (#392). +// Windows is the one platform where a fact flows the other way, so it flows +// back explicitly instead of reordering the two. +// +// WHAT IT BUYS: the SDK version enters `contractHash`, which is part of the +// toolchain fingerprint, which keys the build cache. Two SDKs produce two +// caches. Without this the version axis simply stopped existing one layer +// below the compiler — the exact defect this identity was added to close. +// +// Idempotent, and a no-op for an empty version: a Windows box with no SDK +// found still builds (selection UX must work there), it just has nothing to +// declare. +void bind_windows_ucrt(RuntimeBinding& binding, std::string_view sdkVersion); + namespace detail { std::string hash_contract(std::string_view data) { @@ -398,6 +454,20 @@ resolve_runtime_binding( return out; } +void bind_windows_ucrt(RuntimeBinding& binding, std::string_view sdkVersion) { + if (sdkVersion.empty()) return; + auto identity = std::format("ucrt@{}", sdkVersion); + if (binding.runtimeId == identity) return; // idempotent + binding.runtimeId = std::move(identity); + // Deliberately NOT projected into `libc`: that field is the Linux-only + // private-libc payload, read by the loader/patchelf machinery, and there + // is no ucrt payload for it to name. `runtime_binding.cppm`'s own comment + // on `libc` says "Only Linux additionally projects this" — this is that + // sentence being true rather than merely written down. + binding.contractHash = + detail::hash_contract(detail::canonical_contract(binding)); +} + std::string serialize_runtime_binding(const RuntimeBinding& binding) { nlohmann::json j; j["schema"] = binding.schema; diff --git a/src/toolchain/lifecycle.cppm b/src/toolchain/lifecycle.cppm index a982503d..e5bac541 100644 --- a/src/toolchain/lifecycle.cppm +++ b/src/toolchain/lifecycle.cppm @@ -382,10 +382,15 @@ first_undeletable(const std::filesystem::path& root) { } void msvc_warn_if_sdk_missing(const mcpp::toolchain::msvc::MsvcInstallation& inst) { - auto roots = mcpp::toolchain::msvc::sibling_sdk_roots(inst.clPath); - if (auto sdk = mcpp::toolchain::msvc::find_windows_sdk(roots)) { + // The SAME resolution the build will perform, not a second one shaped + // like it. Install-time and build-time disagreeing about which SDK this + // toolset uses is worse than not reporting at all: the line printed here + // is what the user will believe. + auto choice = mcpp::toolchain::msvc::resolve_sdk_for(inst.clPath); + if (choice.sdk) { std::println(" windows sdk: {} ({})", - sdk->version, sdk->root.string()); + choice.sdk->version, choice.sdk->root.string()); + if (!choice.note.empty()) mcpp::ui::info("note", choice.note); return; } mcpp::ui::warning( @@ -756,14 +761,12 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, mcpp::fetcher::Fetcher fetcher(cfg); mcpp::fetcher::InstallProgressHandler progress; - // Ensure sysroot dependencies (glibc, linux-headers) are installed. - // These are required for C library + kernel headers during compilation. - // Decided by the TARGET, not the payload name: musl targets are - // self-contained; PE targets (native mingw AND the Linux-hosted - // cross) bring their own CRT; Windows/macOS hosts never need the - // Linux sysroot. Mirrors the guard on prepare.cppm's first-run install. - if (!spec->target.is_musl() && !spec->target.is_pe() - && !mcpp::platform::is_windows && !mcpp::platform::is_macos) { + // Ensure sysroot dependencies (glibc, linux-headers) are installed: + // the C library and kernel headers a glibc-targeting compile needs. + // The rule itself lives in registry.cppm — prepare's first-run + // install asks the same question and used to answer it in its own + // words, with one term missing. + if (mcpp::toolchain::needs_linux_sysroot_payloads(spec->target)) { for (auto dep : {"xim:glibc", "xim:linux-headers"}) { mcpp::log::verbose("toolchain", std::format("installing dep: {}", dep)); auto depPayload = fetcher.resolve_xpkg_path(dep, /*autoInstall=*/true, &progress); @@ -799,26 +802,18 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, // VC/Tools/MSVC//bin/Hostx64/x64, and there is nothing to // patchelf on a PE toolchain. if (spec->family == mcpp::toolchain::Family::Msvc) { - // NOT `payload->root` — that is a GUESS, and it guesses wrong - // here. resolve_xpkg_path calls the version directory the root - // only when it directly contains bin/ include/ lib/; otherwise it - // descends into a lone subdirectory. An installed msvc payload - // has exactly one entry, `VC/`, so the "root" comes back as - // …/14.44.35207/VC and the toolset then looks like it is missing. - // - // The location is not something to infer: it is (store, name, - // version), and all three are known here. resolve_xpkg_path above - // is what INSTALLS; this is what says where. - auto verDir = mcpp::xlings::paths::xim_tool( - mcpp::config::make_xlings_env(cfg), pkg.ximName, pkg.ximVersion); - auto inst = mcpp::toolchain::msvc::installation_at( - verDir, pkg.ximVersion); + // Where the payload IS — not `payload->root`, which is the + // fetcher's guess and guesses wrong here. That rule and its + // reasons live in `resolve_managed_msvc`, because a build needs + // exactly the same one and the two used to spell it out + // separately. `resolve_xpkg_path` above is what INSTALLS; this is + // what says where. + auto inst = mcpp::toolchain::resolve_managed_msvc( + mcpp::config::make_xlings_env(cfg), pkg); if (!inst) { mcpp::ui::error(std::format( - "msvc payload installed at '{}', but no cl.exe under " - "VC/Tools/MSVC/{} — the payload is not what this version " - "claims to be", - verDir.string(), pkg.ximVersion)); + "{} — the payload is not what this version claims to be", + inst.error())); return 1; } msvc_print_detected(*inst, "Installed"); diff --git a/src/toolchain/model.cppm b/src/toolchain/model.cppm index aa8da0a6..e6660064 100644 --- a/src/toolchain/model.cppm +++ b/src/toolchain/model.cppm @@ -9,6 +9,34 @@ export namespace mcpp::toolchain { enum class CompilerId { Unknown, GCC, Clang, MSVC }; +// WHERE A COMPILER CAME FROM. Two values, and there will only ever be two. +// +// `Managed` an xlings payload the manifest named. The answer is in the +// manifest; the machine only decides whether it has been +// downloaded yet. +// `SystemMsvc` probed on this machine. The answer depends on what happens to +// be installed here. +// +// THE SECOND ONE IS NOT A GENERAL CAPABILITY, and reading it as one is a +// mistake this comment exists to prevent. mcpp is built on xlings, a +// user-space OS, and the whole design is to drive host dependencies to a +// minimum — there is no `gcc@system`, deliberately. `msvc@system` is a +// concession to ONE platform: Visual Studio is very often already installed +// and cannot always be redistributed, so refusing to use it would cost more +// than it buys. (The bare, family-less `system` spec is a different thing +// again: a deliberate escape hatch to the PATH compiler.) +// +// Lives in the data model rather than in msvc.cppm so the toolchain-spec +// side (registry) and the located-compiler side (msvc) name the SAME axis. +// They used to answer it independently, in 26 scattered branches, which is +// how "is this managed" and "is this a system MSVC" came to be asked with +// different predicates in the same build. +enum class Origin { Managed, SystemMsvc }; + +inline std::string_view origin_name(Origin o) { + return o == Origin::SystemMsvc ? "system" : "managed"; +} + // Fine-grained sysroot paths derived from xpkgs payloads. // When populated, flags are assembled from these paths instead of --sysroot. // One environment variable a toolchain needs at tool-invocation time. @@ -73,6 +101,27 @@ struct Toolchain { // GCC/libc++ answer 20; MSVC answers 20 from cl 19.38 (VS 2022 17.8, // microsoft/STL#3977) and 23 below that. int importStdMinLevel = 0; + // The Windows SDK this toolchain compiles and links against + // ("10.0.26100.0"), once resolved. Empty everywhere else — and empty on + // Windows too when no SDK was found, which detection tolerates so that + // toolchain SELECTION still works on an SDK-less box. + // + // Recorded rather than re-derived because two different questions read it + // and they must not be able to disagree: the compile environment + // (INCLUDE/LIB) and the runtime identity (`ucrt@`, which enters + // the runtime contract hash). Deriving the second from a second search is + // how the SDK came to have no identity in the first place. + std::string windowsSdkVersion; + // Something about HOW this toolchain was resolved that the user has to be + // told, but which is not a failure. Non-empty ⇒ the caller MUST surface it. + // + // Today's only producer is the Windows SDK axis: a managed toolset binds + // the SDK that came with it, so a `WindowsSdkDir` in the environment is + // ignored — and an override that is ignored SILENTLY is indistinguishable + // from one that did not exist. That is the failure shape this whole round + // kept finding: "it did not happen" and "it succeeded" producing identical + // output. + std::string resolutionNote; std::string label() const { return std::format("{} {} ({})", compiler_name(), version, targetTriple); diff --git a/src/toolchain/msvc.cppm b/src/toolchain/msvc.cppm index 60369e1f..c34b4e0d 100644 --- a/src/toolchain/msvc.cppm +++ b/src/toolchain/msvc.cppm @@ -165,6 +165,44 @@ std::optional find_windows_sdk( std::vector sibling_sdk_roots(const std::filesystem::path& clPath); +// Which origin produced this cl.exe, answered by where the binary lives. +// +// Not a guess: an xlings store is a specific directory layout that mcpp +// itself created, and `xpkgs_from_compiler` recognises it or does not. A +// compiler outside every store came from the machine. +Origin origin_of(const std::filesystem::path& clPath); + +// The Windows SDK for a located cl.exe, chosen BY ORIGIN. +// +// This is the second of the three Windows axes (see +// .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §2), and +// until now it had no identity at all: `find_windows_sdk()` scanned, and +// whatever the scan reached first won — for both origins. +// +// MANAGED (`msvc@`) the SDK is a DECLARED dependency of the +// toolset, installed into the same store. It is +// bound, not searched: `WindowsSdkDir` / +// `WindowsSdkVersion` do not participate, +// because a pin that the environment can +// overwrite is not a pin. Two machines building +// the same manifest must see the same headers. +// SYSTEM (`msvc@system`) the machine's own SDK, and a machine's things +// can only be found by looking. Unchanged: the +// declared `WindowsSdkDir` still outranks the +// scan, exactly as it does for VSINSTALLDIR. +// +// `note` is non-empty when something the user could have expected to matter +// did not, and the caller MUST surface it. There are exactly two: +// - a managed toolset ignoring a `WindowsSdkDir` that was set +// - a managed toolset with NO SDK payload beside it, falling back to the +// machine's — which works, and is not reproducible, so it says so +struct SdkChoice { + std::optional sdk; + Origin origin = Origin::Managed; + std::string note; +}; +SdkChoice resolve_sdk_for(const std::filesystem::path& clPath); + // True only when BOTH halves of a usable MSVC C++ setup are present: the // STL's std module source AND the Windows SDK. // @@ -625,6 +663,46 @@ constexpr std::string_view sdk_lib_arch = "arm64"; constexpr std::string_view sdk_lib_arch = "x64"; #endif +namespace { + +// Highest version dir under `root/Include` that actually carries the UCRT +// headers; `want` (from WindowsSdkVersion) wins if it is one of them. +std::optional pick_sdk_in(const std::filesystem::path& root, + std::string_view want) { + std::error_code ec; + auto inc = root / "Include"; + if (!std::filesystem::is_directory(inc, ec)) return std::nullopt; + auto usable = [&](const std::filesystem::path& verDir, const std::string& v) { + return std::filesystem::exists(verDir / "ucrt" / "corecrt.h", ec) + && std::filesystem::exists( + root / "Lib" / v / "um" / sdk_lib_arch / "kernel32.lib", ec); + }; + std::string best; + for (auto& e : std::filesystem::directory_iterator(inc, ec)) { + if (!e.is_directory(ec)) continue; + auto v = e.path().filename().string(); + if (!usable(e.path(), v)) continue; + if (!want.empty() && v == want) return WindowsSdk{root, v}; + if (v > best) best = v; + } + if (best.empty()) return std::nullopt; + return WindowsSdk{root, best}; +} + +// `WindowsSdkVersion` without its trailing backslash (vcvars exports one; it +// is not part of the directory name). Empty when unset. +std::string declared_sdk_version() { + std::string want; + if (auto* v = std::getenv("WindowsSdkVersion"); v && *v) { + want = v; + while (!want.empty() && (want.back() == '\\' || want.back() == '/')) + want.pop_back(); + } + return want; +} + +} // namespace + std::optional find_windows_sdk( std::span extraRoots) { // Highest version dir under `root/Include` that actually carries the UCRT @@ -648,54 +726,97 @@ std::optional find_windows_sdk( // // kernel32.lib is the right sentinel: every link needs it, and unlike the // ucrt libs it is not spread across the SDK's optional pieces. - auto pick = [](const std::filesystem::path& root, - std::string_view want) -> std::optional { - std::error_code ec; - auto inc = root / "Include"; - if (!std::filesystem::is_directory(inc, ec)) return std::nullopt; - auto usable = [&](const std::filesystem::path& verDir, - const std::string& v) { - return std::filesystem::exists(verDir / "ucrt" / "corecrt.h", ec) - && std::filesystem::exists( - root / "Lib" / v / "um" / sdk_lib_arch / "kernel32.lib", ec); - }; - std::string best; - for (auto& e : std::filesystem::directory_iterator(inc, ec)) { - if (!e.is_directory(ec)) continue; - auto v = e.path().filename().string(); - if (!usable(e.path(), v)) continue; - if (!want.empty() && v == want) return WindowsSdk{root, v}; - if (v > best) best = v; - } - if (best.empty()) return std::nullopt; - return WindowsSdk{root, best}; - }; + // + // The version-selection rule itself lives in `pick_sdk_in`, because the + // MANAGED origin needs the same rule applied to a different (and much + // shorter) list of roots — see `resolve_sdk_for`. // 1. Declared: WindowsSdkDir (+ WindowsSdkVersion). vcvars exports both; // WindowsSdkVersion carries a trailing backslash there, which is not // part of the directory name. - std::string want; - if (auto* v = std::getenv("WindowsSdkVersion"); v && *v) { - want = v; - while (!want.empty() && (want.back() == '\\' || want.back() == '/')) - want.pop_back(); - } + const std::string want = declared_sdk_version(); if (auto* dir = std::getenv("WindowsSdkDir"); dir && *dir) { - if (auto s = pick(std::filesystem::path{dir}, want)) return s; + if (auto s = pick_sdk_in(std::filesystem::path{dir}, want)) return s; } // 2. Roots the caller knows about (managed toolset's own store). for (const auto& root : extraRoots) - if (auto s = pick(root, want)) return s; + if (auto s = pick_sdk_in(root, want)) return s; // 3. The conventional absolute install roots. for (const char* base : {"C:\\Program Files (x86)\\Windows Kits\\10", "C:\\Program Files\\Windows Kits\\10"}) { - if (auto s = pick(std::filesystem::path{base}, want)) return s; + if (auto s = pick_sdk_in(std::filesystem::path{base}, want)) return s; } return std::nullopt; } +Origin origin_of(const std::filesystem::path& clPath) { + return mcpp::xlings::paths::xpkgs_from_compiler(clPath) + ? Origin::Managed : Origin::SystemMsvc; +} + +SdkChoice resolve_sdk_for(const std::filesystem::path& clPath) { + SdkChoice out; + out.origin = origin_of(clPath); + auto siblings = sibling_sdk_roots(clPath); + + if (out.origin == Origin::SystemMsvc) { + // A machine's things can only be found by looking, and a declared + // WindowsSdkDir is still the most specific answer available. + // `siblings` is empty here by construction (a system cl.exe is in no + // store); passed through so the two origins share one call shape. + out.sdk = find_windows_sdk(siblings); + return out; + } + + // MANAGED. The SDK arrived as a declared dependency of this toolset and + // sits in the same store — so it is looked up, not searched for, and + // nothing in the environment gets a vote. See the design doc's §2.1 for + // what searching cost: a half-unpacked payload outranked the machine's own + // complete SDK because its version number was higher, every TU compiled, + // and the build died at LNK1104 with nothing in the log naming the SDK. + // + // `sibling_sdk_roots` returns newest first, so the first hit is also the + // one the highest-version rule would have chosen. + for (auto const& root : siblings) { + // `want` deliberately empty: WindowsSdkVersion is the same declaration + // channel as WindowsSdkDir, and letting it pick among payloads is the + // same override wearing a smaller hat. + if ((out.sdk = pick_sdk_in(root, {}))) break; + } + + if (out.sdk) { + const char* dir = std::getenv("WindowsSdkDir"); + const char* ver = std::getenv("WindowsSdkVersion"); + if ((dir && *dir) || (ver && *ver)) { + out.note = std::format( + "WindowsSdkDir/WindowsSdkVersion in the environment " + "({}) is ignored: this build pins a managed MSVC toolset, and " + "the Windows SDK it was installed with ({} at {}) is part of " + "that pin. Use msvc@system if the machine's SDK is what you " + "want.", + dir && *dir ? dir : ver, + out.sdk->version, out.sdk->root.string()); + } + return out; + } + + // No SDK payload beside the toolset. Falling back keeps the build working + // — but quietly falling back is precisely how the version axis stopped + // meaning anything one layer down, so it is reported. + out.sdk = find_windows_sdk({}); + out.note = out.sdk + ? std::format( + "no windows-sdk payload was found beside this managed MSVC " + "toolset, so the machine's SDK ({} at {}) is being used. That " + "makes the build depend on this machine — reinstall the toolset " + "(`mcpp toolchain install msvc `) to pull its own SDK.", + out.sdk->version, out.sdk->root.string()) + : std::string{}; + return out; +} + bool has_usable_msvc() { #if defined(_WIN32) // Both, deliberately — see the declaration for why either half alone is @@ -928,13 +1049,19 @@ std::expected enrich_toolchain_from_cl(Toolchain& tc) { // detection working (selection UX on SDK-less boxes); the build path // errors with guidance when envOverrides is empty. // - // A managed toolset carries its own SDK as an xlings dependency, and the - // compiler's own path says which store to look in — so the two halves of - // a pinned toolchain stay together without anything being configured. - auto extraSdkRoots = sibling_sdk_roots(tc.binaryPath); - if (auto sdk = find_windows_sdk(extraSdkRoots)) { - tc.envOverrides = build_env_for_cl(tc.binaryPath, parsed->second, *sdk); + // BY ORIGIN, not by search. A managed toolset carries its own SDK as an + // xlings dependency and the compiler's own path says which store it is in, + // so the two halves of a pinned toolchain stay together — and stay + // together even when the environment says otherwise, which is the part a + // search could not give. `msvc@system` keeps the search chain: the + // machine's things can only be found by looking. + auto choice = resolve_sdk_for(tc.binaryPath); + if (choice.sdk) { + tc.envOverrides = build_env_for_cl(tc.binaryPath, parsed->second, + *choice.sdk); + tc.windowsSdkVersion = choice.sdk->version; } + if (!choice.note.empty()) tc.resolutionNote = choice.note; // The toolset's own redistributable CRT, in the same field gcc uses for // libstdc++ — so `mcpp run` puts it on PATH exactly the way it puts a diff --git a/src/toolchain/post_install.cppm b/src/toolchain/post_install.cppm index 394755c2..6004c87b 100644 --- a/src/toolchain/post_install.cppm +++ b/src/toolchain/post_install.cppm @@ -16,6 +16,7 @@ import mcpp.config; import mcpp.libs.json; import mcpp.log; import mcpp.platform; +import mcpp.platform.runtime_binding; import mcpp.toolchain.linkmodel; import mcpp.toolchain.registry; import mcpp.ui; @@ -384,9 +385,20 @@ select_glibc_payload_lib(const std::filesystem::path& glibcRoot, std::string_view runtimeId) { constexpr std::string_view prefix = "glibc@"; if (!runtimeId.starts_with(prefix)) { - return std::unexpected(std::format( - "selected RuntimeBinding '{}' is not a glibc payload identity " - "(expected glibc@)", runtimeId)); + // Name the PROVIDER that was found, not just the one that was + // wanted. `ucrt@10.0.26100.0` is a perfectly valid runtime identity + // that this fixup has nothing to do with (there is no ucrt payload to + // bind to — it is an OS component), and "not a glibc payload + // identity" reads as "malformed" for it. + auto provider = mcpp::platform::runtime::runtime_provider(runtimeId); + return std::unexpected(provider.empty() + ? std::format( + "selected RuntimeBinding '{}' is not a glibc payload identity " + "(expected glibc@)", runtimeId) + : std::format( + "selected RuntimeBinding '{}' names the '{}' runtime " + "provider; the glibc payload fixup does not apply to it", + runtimeId, provider)); } auto version = runtimeId.substr(prefix.size()); if (version.empty() || version == "." || version == ".." diff --git a/src/toolchain/registry.cppm b/src/toolchain/registry.cppm index 0dd21fb9..dc23e0af 100644 --- a/src/toolchain/registry.cppm +++ b/src/toolchain/registry.cppm @@ -17,6 +17,7 @@ export module mcpp.toolchain.registry; import std; import mcpp.platform; +import mcpp.platform.xlings; import mcpp.toolchain.clang; import mcpp.toolchain.compat; import mcpp.toolchain.gcc; @@ -148,6 +149,50 @@ bool spec_matches_payload(const ToolchainSpec& def, // and older mechanism.) bool is_system_toolchain(const ToolchainSpec& spec); +// The same question as `is_system_toolchain`, asked as the AXIS it belongs to +// (mcpp.toolchain.model). Both spellings exist because the predicate reads +// better at a site that is deciding one thing, and the enum reads better at a +// site that dispatches — but there is one derivation, so they cannot drift. +Origin origin_of(const ToolchainSpec& spec); + +// Resolve a MANAGED msvc payload to its installation record. +// +// `prepare` (a build) and `lifecycle` (install / default / remove) both need +// this, and both used to spell it out: derive the version directory from +// (store, name, version) rather than trusting the fetcher's `root` guess — +// which descends into a lone subdirectory and therefore lands one level too +// deep for an msvc payload, whose only entry is `VC/` — then call +// `installation_at`, then format the same error. Three copies of one rule, +// and the comment explaining WHY the fetcher's guess is wrong existed in only +// one of them. +// +// `identifyVersion = false` skips running cl.exe for its banner; callers that +// only need to know whether a toolset is there should pass false. +std::expected +resolve_managed_msvc(const mcpp::xlings::Env& env, + const XimToolchainPackage& pkg, + bool identifyVersion = true); + +// Does installing a toolchain FOR THIS TARGET additionally need the Linux +// sysroot payloads (`xim:glibc` + `xim:linux-headers`)? +// +// THE SINGLE DERIVATION. It was two, and they were not equivalent while a +// comment on one of them said "mirrors the guard on the other": +// +// lifecycle !musl && !pe && !windows-host && !macos-host +// prepare !macos-host && !windows-host && !musl +// +// The PE term was missing from the second. It happens to be unreachable today +// (first-run never selects a PE target on Linux), which is what let the +// divergence sit there — a latent difference between two spellings of one +// rule is exactly the state that becomes a bug the moment either side moves. +// +// Decided by the TARGET, not the payload name: musl targets are +// self-contained, PE targets (native MinGW and the Linux-hosted cross alike) +// bring their own CRT, and a non-Linux host never needs a Linux sysroot at +// all. +bool needs_linux_sysroot_payloads(const triple::Triple& target); + // Can THIS host serve that target — is there an installable payload for the // (host, target) pair? Empty target = host target, always serviceable. // @@ -235,10 +280,62 @@ parse_toolchain_spec(std::string compilerArg, else spec.family = Family::Gcc; spec.version = std::move(norm->version); spec.target = std::move(norm->target); + + // `@system` IS NOT A GENERAL SPELLING, and refusing it here is the point. + // + // mcpp is built on xlings, a user-space OS, and the whole design drives + // host dependencies to a minimum: a toolchain comes from a payload the + // manifest names, so every machine compiles with the same compiler. + // `msvc@system` is a concession to ONE platform — Visual Studio is very + // often already installed and cannot always be redistributed — not a + // capability the other families are missing. + // + // It used to parse and then fail somewhere else entirely, as + // `xim:gcc@system` → "no such package", which sends the reader looking + // for a version that was never going to exist. A spec that cannot mean + // anything should be rejected where it is read, by name, with the two + // things it might have meant spelled out. + if (spec.version == "system" && spec.family != Family::Msvc) { + return std::unexpected(std::format( + "'{}@system' is not a toolchain spelling: only msvc has a system " + "origin, because Visual Studio is often already installed and " + "cannot always be redistributed.\n" + " mcpp installs every other toolchain itself, so that each " + "machine builds with the same one:\n" + " {}@ pin a payload (mcpp toolchain list --available {})\n" + " system the PATH compiler, whatever it is — an escape " + "hatch, and it takes no family", + family_name(spec.family), family_name(spec.family), + family_name(spec.family))); + } if (norm->changed) spec.compatHint = std::move(norm->hint); return spec; } +Origin origin_of(const ToolchainSpec& spec) { + return is_system_toolchain(spec) ? Origin::SystemMsvc : Origin::Managed; +} + +std::expected +resolve_managed_msvc(const mcpp::xlings::Env& env, + const XimToolchainPackage& pkg, + bool identifyVersion) { + // NOT the fetcher's `root`: that field is its guess at where the useful + // tree starts, and it descends into a lone subdirectory when the version + // dir has no bin/ include/ lib/. An msvc payload's only entry is `VC/`, + // so the guess lands exactly one level too deep. (store, name, version) + // is known — compose it. + auto verDir = mcpp::xlings::paths::xim_tool(env, pkg.ximName, pkg.ximVersion); + // The package version IS the toolset directory name, so cl.exe is derived + // rather than searched for: nothing here can silently pick a different + // toolset, which is the whole reason the version axis exists. + if (auto inst = msvc::installation_at(verDir, pkg.ximVersion, identifyVersion)) + return *inst; + return std::unexpected(std::format( + "msvc payload at '{}' has no cl.exe under VC/Tools/MSVC/{}", + verDir.string(), pkg.ximVersion)); +} + void print_compat_hint(const ToolchainSpec& spec) { if (spec.compatHint.empty()) return; compat::print_hint_once(spec.compatHint); @@ -437,6 +534,11 @@ bool is_system_toolchain(const ToolchainSpec& spec) { && (spec.version.empty() || spec.version == "system"); } +bool needs_linux_sysroot_payloads(const triple::Triple& target) { + if constexpr (!mcpp::platform::is_linux) return false; + return !target.is_musl() && !target.is_pe(); +} + bool host_can_serve(const triple::Triple& target) { if (target.empty()) return true; // host target diff --git a/tests/unit/test_distribution.cpp b/tests/unit/test_distribution.cpp index bbb089a3..2953f406 100644 --- a/tests/unit/test_distribution.cpp +++ b/tests/unit/test_distribution.cpp @@ -273,6 +273,86 @@ TEST(Distribution, MsvcCrtModelIsWholeProjectAndReportedAsSuch) { EXPECT_EQ(quiet.effective, dist::Contract::HostCoupled); } +// `toolchain-coupled` on the MSVC runtime used to be a flat refusal, and the +// sentence it refused with conflated two different DLLs: +// +// ucrtbase.dll an OS component since Win10 — the refusal was right +// vcruntime140.dll the TOOLSET's own, sitting in VC\Redist\MSVC\… inside +// every toolset mcpp installs — the refusal was wrong +// +// The second one is exactly the relationship gcc has to libstdc++.so, so it +// takes the same contract. What differs is the MECHANISM: PE has no rpath, so +// the DLL travels by being copied beside the artifact. +TEST(Distribution, MsvcToolchainCoupledStagesTheToolsetCrt) { + dist::MechanismInput in; + in.format = dist::Format::Pe; + in.stdlibId = "msvc"; + in.explicitRequest = true; + in.requested = dist::Contract::ToolchainCoupled; + + // /MD: the artifact HAS a vcruntime140.dll dependency, so the contract is + // deliverable — and delivering it means staging files, not adding flags. + in.msvcStaticCrt = false; + auto coupled = dist::resolve(in); + EXPECT_EQ(coupled.effective, dist::Contract::ToolchainCoupled); + EXPECT_FALSE(coupled.degraded); + EXPECT_TRUE(coupled.diagnostic.empty()); + EXPECT_TRUE(coupled.deployToolchainRuntime); + // The CRT model is a compile flag on every TU; nothing goes on the link line. + EXPECT_TRUE(coupled.unitFlags.empty()); + + // /MT is the one case that stays a degradation, and it is a genuine + // contradiction rather than a missing mechanism: a static CRT leaves no + // DLL to couple to. The message has to say which one won. + in.msvcStaticCrt = true; + auto contradiction = dist::resolve(in); + EXPECT_EQ(contradiction.effective, dist::Contract::SelfContained); + EXPECT_TRUE(contradiction.degraded); + EXPECT_FALSE(contradiction.deployToolchainRuntime) + << "a /MT build has no CRT DLL dependency; staging one is dead weight"; + EXPECT_NE(contradiction.diagnostic.find("/MT"), std::string::npos) + << contradiction.diagnostic; + EXPECT_NE(contradiction.diagnostic.find("self-contained"), std::string::npos) + << contradiction.diagnostic; +} + +// Nothing but PE+MSVC+toolchain-coupled may ask for files to be staged. The +// flag reaches a copy step, so a stray `true` puts DLLs in an output tree on a +// platform that has no such thing. +TEST(Distribution, NothingElseAsksForStagedRuntimeFiles) { + const dist::Contract contracts[] = {dist::Contract::SelfContained, + dist::Contract::ToolchainCoupled, + dist::Contract::HostCoupled}; + const dist::Format formats[] = {dist::Format::Elf, dist::Format::MachO, + dist::Format::Pe}; + const std::string_view stdlibs[] = {"libstdc++", "libc++", "msvc", "surprise"}; + for (auto fmt : formats) + for (auto sl : stdlibs) + for (auto c : contracts) + for (bool mt : {false, true}) + for (bool explicitly : {false, true}) { + dist::MechanismInput in; + in.format = fmt; + in.stdlibId = sl; + in.requested = c; + in.msvcStaticCrt = mt; + in.explicitRequest = explicitly; + in.mingw = (fmt == dist::Format::Pe + && sl == "libstdc++"); + auto m = dist::resolve(in); + if (!m.deployToolchainRuntime) continue; + EXPECT_EQ(fmt, dist::Format::Pe); + EXPECT_NE(sl, std::string_view("libstdc++")); + EXPECT_EQ(c, dist::Contract::ToolchainCoupled); + EXPECT_FALSE(mt); + // Staging files is a promise KEPT. A degraded cell did + // not deliver the contract, so it must not act as if + // it had. + EXPECT_FALSE(m.degraded); + EXPECT_EQ(m.effective, dist::Contract::ToolchainCoupled); + } +} + // --------------------------------------------------------------------------- // INV-1, stated as a property rather than a list: the table is TOTAL, and // every cell that does not deliver what was asked explains itself. A future diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 7af42d59..d7614274 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -1430,3 +1430,139 @@ TEST(Plan, ExpandManifestIncludeEntryNativeSpelling) { fs::remove_all(root); } + +// ── PE `toolchain-coupled`: the toolset's CRT travels with the artifact ───── +// +// On ELF this contract needs no files copied — the artifact carries an rpath +// into the toolchain's lib directory. PE has no rpath: a DLL is resolved from +// the directory of the executable, so on this format the mechanism IS the +// copy, and a graph that resolves the contract but stages nothing delivers +// exactly what the old flat refusal did. +// +// Runs on every host. The format now comes from the TARGET TRIPLE rather than +// from `mcpp::platform::is_windows`, which is what makes a Windows contract +// assertable on the Linux runner that reviews most of these changes. +namespace { + +struct FakeRedistDir { + std::filesystem::path path; + FakeRedistDir() { + path = std::filesystem::temp_directory_path() + / std::format("mcpp-deploy-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(path); + for (auto name : {"vcruntime140.dll", "msvcp140.dll", + "vcruntime140_1.dll"}) + std::ofstream{path / name} << "MZ"; + // Not a DLL: the redist directory also carries a manifest, and copying + // it beside the artifact would be cargo. + std::ofstream{path / "Microsoft.VC143.CRT.manifest"} << ""; + } + ~FakeRedistDir() { std::error_code ec; std::filesystem::remove_all(path, ec); } + FakeRedistDir(const FakeRedistDir&) = delete; + FakeRedistDir& operator=(const FakeRedistDir&) = delete; +}; + +BuildPlan msvc_plan_with_redist(const FakeRedistDir& redist, + std::string_view cxxRuntime) { + auto plan = minimal_plan(); + plan.toolchain.compiler = mcpp::toolchain::CompilerId::MSVC; + plan.toolchain.binaryPath = "cl.exe"; + plan.toolchain.targetTriple = "x86_64-pc-windows-msvc"; + plan.toolchain.linkRuntimeDirs = {redist.path}; + plan.manifest.buildConfig.cxxRuntime = std::string(cxxRuntime); + plan.linkUnits.push_back({ + .targetName = "app", + .kind = mcpp::build::LinkUnit::Binary, + .objects = {"obj/main.o"}, + .output = "bin/app.exe", + .entryMain = "src/main.cpp", + }); + return plan; +} + +} // namespace + +TEST(NinjaBackendPeRuntime, ToolchainCoupledStagesTheToolsetCrtBesideTheExe) { + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + + auto flags = compute_flags(plan); + ASSERT_EQ(flags.toolchainRuntimeDeploy.size(), 3u) + << "expected the three .dll and not the .manifest beside them"; + for (auto const& d : flags.toolchainRuntimeDeploy) { + EXPECT_EQ(d.dest.parent_path(), std::filesystem::path("bin")) + << "a DLL must land in the same directory as the .exe: " + << d.dest.string(); + EXPECT_EQ(d.source.extension(), ".dll") << d.source.string(); + } + + auto ninja = emit_ninja_string(plan); + for (auto name : {"vcruntime140.dll", "msvcp140.dll", "vcruntime140_1.dll"}) { + EXPECT_NE(ninja.find(std::format("build bin/{} : stage_file", name)), + std::string::npos) + << name << " has no copy edge\n" << ninja; + } + EXPECT_EQ(ninja.find("Microsoft.VC143.CRT.manifest"), std::string::npos) + << "copied something that is not a runtime DLL\n" << ninja; + + // Reachability, twice over. A copy edge nothing asks for is never run + // under explicit ninja goals, and an .exe that does not depend on its DLLs + // can be reported "up to date" while they are missing. + auto exeAt = ninja.find("build bin/app.exe :"); + ASSERT_NE(exeAt, std::string::npos) << ninja; + auto exeStanza = ninja.substr(exeAt, ninja.find("\nbuild ", exeAt + 1) - exeAt); + EXPECT_NE(exeStanza.find("bin/vcruntime140.dll"), std::string::npos) + << "the executable does not depend on the CRT staged beside it\n" + << exeStanza; + auto defaultAt = ninja.rfind("\ndefault "); + ASSERT_NE(defaultAt, std::string::npos) << ninja; + EXPECT_NE(ninja.find("bin/vcruntime140.dll", defaultAt), std::string::npos) + << "the staged CRT is not a default goal\n" << ninja.substr(defaultAt); +} + +TEST(NinjaBackendPeRuntime, HostCoupledStagesNothing) { + // The DEFAULT Windows build. Staging DLLs unasked would change what every + // existing project ships, and `host-coupled` is a promise that the machine + // provides them — keeping a copy beside the artifact contradicts it. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "host-coupled"); + EXPECT_TRUE(compute_flags(plan).toolchainRuntimeDeploy.empty()); + + auto bare = msvc_plan_with_redist(redist, ""); + EXPECT_TRUE(compute_flags(bare).toolchainRuntimeDeploy.empty()) + << "a project that never mentioned cxx_runtime gained staged DLLs"; +} + +TEST(NinjaBackendPeRuntime, AProjectsOwnDeployFileOutranksTheToolsets) { + // A vendored redist named in `[runtime] deploy_files` is a human's + // statement about which build of msvcp140.dll this program ships. Silently + // replacing it with the toolset's copy produces a different program than + // the manifest describes, so the explicit one wins — out loud. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + plan.runtimeDeployFiles.push_back( + {"/vendor/msvcp140.dll", std::filesystem::path("bin") / "msvcp140.dll"}); + + auto flags = compute_flags(plan); + for (auto const& d : flags.toolchainRuntimeDeploy) + EXPECT_NE(d.dest.filename(), std::filesystem::path("msvcp140.dll")) + << "overwrote the project's own deploy file"; + EXPECT_EQ(flags.toolchainRuntimeDeploy.size(), 2u); + EXPECT_TRUE(std::ranges::any_of(flags.diagnostics, [](auto const& d) { + return d.find("msvcp140.dll") != std::string::npos; + })) << "the collision was resolved silently"; +} + +TEST(NinjaBackendPeRuntime, AnElfToolchainNeverStagesItsRuntimeDirs) { + // `linkRuntimeDirs` means "the toolchain's private runtime" for every + // provider — on gcc it is libstdc++'s directory. That is reached through + // an rpath, and copying its contents into an output tree would be a + // different contract wearing the same name. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + plan.toolchain.compiler = mcpp::toolchain::CompilerId::GCC; + plan.toolchain.binaryPath = "/usr/bin/g++"; + plan.toolchain.targetTriple = "x86_64-linux-gnu"; + EXPECT_TRUE(compute_flags(plan).toolchainRuntimeDeploy.empty()); +} diff --git a/tests/unit/test_runtime_contract.cpp b/tests/unit/test_runtime_contract.cpp index 5664652f..2da1734c 100644 --- a/tests/unit/test_runtime_contract.cpp +++ b/tests/unit/test_runtime_contract.cpp @@ -370,3 +370,80 @@ TEST(RuntimeSearchClosure, UndeclaredBindingStillGetsItsOwnFarm) { EXPECT_EQ(farm_entries(build::runtime_search_closure( plan_for("aarch64-linux-gnu"), b)), 0u); } + +// ─── the Windows runtime identity ──────────────────────────────────────── +// +// `runtimeId`'s own comment has documented `ucrt@…` since the field was +// introduced, and nothing in the repository ever wrote one. The compiler had +// a version axis and the C runtime it compiles against did not, so two +// Windows SDKs produced one cache key — the version axis simply stopped +// existing one layer down. + +TEST(RuntimeIdentity, ProviderIsDispatchedOnNotPatternMatched) { + namespace rt = mcpp::platform::runtime; + EXPECT_EQ(rt::runtime_provider("glibc@2.39"), "glibc"); + EXPECT_EQ(rt::runtime_provider("ucrt@10.0.26100.0"), "ucrt"); + EXPECT_EQ(rt::runtime_provider("macos_sdk@14.0"), "macos_sdk"); + // No identity at all is a DIFFERENT answer from "an identity belonging to + // some other provider", and every consumer that spelled + // `starts_with("glibc@")` collapsed the two. + EXPECT_TRUE(rt::runtime_provider("").empty()); + EXPECT_TRUE(rt::runtime_provider("nonsense").empty()); +} + +TEST(RuntimeIdentity, TheSdkVersionReachesTheContractHash) { + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding a; + a.platform = "windows"; + a.contractHash = "unset"; + + rt::RuntimeBinding b = a; + rt::bind_windows_ucrt(a, "10.0.22621.0"); + rt::bind_windows_ucrt(b, "10.0.26100.0"); + + EXPECT_EQ(a.runtimeId, "ucrt@10.0.22621.0"); + EXPECT_EQ(b.runtimeId, "ucrt@10.0.26100.0"); + EXPECT_NE(a.contractHash, b.contractHash) + << "two SDKs produced one contract hash, so they produce one build " + "cache — which is the defect this identity closes"; + EXPECT_NE(a.contractHash, "unset") << "the hash was not re-derived"; +} + +TEST(RuntimeIdentity, BindingUcrtIsIdempotentAndSkipsAnEmptyVersion) { + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding b; + b.platform = "windows"; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + auto once = b.contractHash; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + EXPECT_EQ(b.contractHash, once); + + // A Windows box with no SDK still builds — toolchain SELECTION has to + // work there — so there is simply nothing to declare. + rt::RuntimeBinding empty; + empty.contractHash = "keep"; + rt::bind_windows_ucrt(empty, ""); + EXPECT_TRUE(empty.runtimeId.empty()); + EXPECT_EQ(empty.contractHash, "keep"); +} + +TEST(RuntimeIdentity, UcrtIsAFloorDeclarationNotAPrivatePayload) { + // The asymmetry with glibc, pinned so it cannot be "tidied up" later. + // + // `glibc@2.39` binds a payload: the headers and the .so are both in it and + // patchelf makes the artifact run on that exact copy, so it is ALSO + // projected into `libc`, which the loader machinery reads. `ucrtbase.dll` + // is an OS component from Win10 on — it cannot be swapped and must not be + // shipped — so mcpp's windows-sdk payload deliberately carries only half + // of ucrt (headers + import libraries). Projecting it into `libc` would + // send the private-libc machinery looking for a payload that was never + // supposed to exist. + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding b; + b.platform = "windows"; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + EXPECT_FALSE(b.libc.has_value()) + << "ucrt was projected into the private-libc field"; + EXPECT_FALSE(b.loader.has_value()); + EXPECT_TRUE(b.libraryDirs.empty()); +} diff --git a/tests/unit/test_toolchain_msvc.cpp b/tests/unit/test_toolchain_msvc.cpp index 034a9c0e..a4968023 100644 --- a/tests/unit/test_toolchain_msvc.cpp +++ b/tests/unit/test_toolchain_msvc.cpp @@ -657,3 +657,170 @@ TEST(ToolchainMsvc, BothKeysSelectTheStaticCrt) { EXPECT_FALSE(msvc_wants_static_crt("", "host-coupled")); EXPECT_FALSE(msvc_wants_static_crt("", "toolchain-coupled")); } + +// ─── the SDK axis: bound for a managed toolset, searched for a system one ── +// +// Until now the Windows SDK had no identity: `find_windows_sdk()` scanned, +// and whatever the scan reached first won, for BOTH origins. That is the +// defect .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §2 +// is about — the compiler got a version axis and the headers it compiles +// against did not, so two machines could build one manifest against two SDKs +// with nothing in the log naming either. + +namespace { + +// An xlings store, laid out the way the real one is: `xpkgs_from_compiler` +// finds the store by walking up for a directory literally named `xpkgs`, so +// that name is load-bearing and not decoration. +struct FakeStore { + std::filesystem::path root; // …/xpkgs + + FakeStore() { + root = std::filesystem::temp_directory_path() + / std::format("mcpp-store-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()) + / "data" / "xpkgs"; + std::filesystem::create_directories(root); + } + ~FakeStore() { + std::error_code ec; + std::filesystem::remove_all(root.parent_path().parent_path(), ec); + } + FakeStore(const FakeStore&) = delete; + FakeStore& operator=(const FakeStore&) = delete; + + std::filesystem::path add_toolset(std::string_view version) { + auto tools = root / "xim-x-msvc" / std::string(version) + / "VC" / "Tools" / "MSVC" / std::string(version); + auto bin = tools / "bin" / "Hostx64" / "x64"; + std::filesystem::create_directories(bin); + std::ofstream{bin / "cl.exe"} << "not a compiler"; + std::filesystem::create_directories(tools / "modules"); + std::ofstream{tools / "modules" / "std.ixx"} << "export module std;"; + return bin / "cl.exe"; + } + // A complete SDK payload — headers AND import libraries, for both + // architectures so the fixture does not care which host runs it. + std::filesystem::path add_sdk(std::string_view version) { + auto sdk = root / "xim-x-windows-sdk" / std::string(version); + auto inc = sdk / "Include" / std::string(version) / "ucrt"; + std::filesystem::create_directories(inc); + std::ofstream{inc / "corecrt.h"} << "#pragma once"; + for (auto arch : {"x64", "arm64"}) { + auto lib = sdk / "Lib" / std::string(version) / "um" / arch; + std::filesystem::create_directories(lib); + std::ofstream{lib / "kernel32.lib"} << "not a library"; + } + return sdk; + } +}; + +} // namespace + +TEST(MsvcSdkOrigin, ACompilerInAStoreIsManagedAndOneOutsideIsNot) { + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + EXPECT_EQ(msvc::origin_of(cl), Origin::Managed); + EXPECT_EQ(msvc::origin_of( + "C:/Program Files/Microsoft Visual Studio/18/Enterprise/VC/Tools/" + "MSVC/14.51.36231/bin/Hostx64/x64/cl.exe"), Origin::SystemMsvc); +} + +TEST(MsvcSdkOrigin, AManagedToolsetTakesTheSdkFromItsOwnStore) { + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.26100.0"); + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()) + << "the toolset's own SDK payload was not found"; + EXPECT_EQ(choice.origin, Origin::Managed); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0"); + EXPECT_TRUE(choice.note.empty()) << choice.note; +} + +// THE CRITERION THIS AXIS EXISTS FOR. An environment that can overwrite the +// choice means the manifest did not pin anything — it only expressed a +// preference that the machine gets to overrule, silently. +// +// This is the design doc's §6 acceptance test, as a unit test: point +// WindowsSdkDir somewhere else entirely and the build must still use the +// payload's SDK, and must SAY that the variable was ignored. +TEST(MsvcSdkOrigin, WindowsSdkDirCannotOverrideAPinnedToolsetsSdk) { + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.26100.0"); + + FakeToolset elsewhere{"sdk-elsewhere"}; + elsewhere.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", elsewhere.root.string()}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0") + << "the environment overrode a pinned toolset's SDK; the pin is not one"; + EXPECT_NE(choice.sdk->root, elsewhere.root); + EXPECT_NE(choice.note.find("WindowsSdkDir"), std::string::npos) + << "an ignored override that says nothing is indistinguishable from " + "one that was never set: " << choice.note; +} + +TEST(MsvcSdkOrigin, WindowsSdkVersionDoesNotPickAmongPayloadsEither) { + // Same override wearing a smaller hat: naming a version is still the + // environment choosing which headers a pinned build compiles against. + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.22621.0"); + store.add_sdk("10.0.26100.0"); + ScopedEnv ver{"WindowsSdkVersion", "10.0.22621.0\\"}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0") + << "WindowsSdkVersion selected among the store's payloads"; + EXPECT_FALSE(choice.note.empty()); +} + +TEST(MsvcSdkOrigin, AManagedToolsetWithNoSdkPayloadFallsBackAndSaysSo) { + // Working beats failing here — the machine may well have a complete SDK, + // and refusing to build would be a regression for anyone whose toolset + // predates the SDK dependency. What must not happen is the fallback being + // invisible: the build is no longer reproducible, and only this line says + // so. + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); // no add_sdk + + FakeToolset machine{"sdk-machine"}; + machine.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", machine.root.string()}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->root, machine.root); + EXPECT_NE(choice.note.find("machine"), std::string::npos) << choice.note; +} + +TEST(MsvcSdkOrigin, ASystemToolsetKeepsTheDeclaredSearchChain) { + // The other origin is unchanged, and must be: a machine's things can only + // be found by looking, and WindowsSdkDir is the most specific answer + // available there — the same precedence VSINSTALLDIR has over vswhere. + NoSdkEnv clean; + FakeToolset vs{"sdk-system-vs"}; + vs.add_toolset("14.51.36231"); + FakeToolset declared{"sdk-system-declared"}; + declared.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", declared.root.string()}; + + auto cl = vs.root / "VC" / "Tools" / "MSVC" / "14.51.36231" + / "bin" / "Hostx64" / "x64" / "cl.exe"; + auto choice = msvc::resolve_sdk_for(cl); + EXPECT_EQ(choice.origin, Origin::SystemMsvc); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->root, declared.root); + EXPECT_TRUE(choice.note.empty()) + << "nothing was ignored, so there is nothing to report: " << choice.note; +} diff --git a/tests/unit/test_toolchain_registry.cpp b/tests/unit/test_toolchain_registry.cpp index 8e961ae7..2437d968 100644 --- a/tests/unit/test_toolchain_registry.cpp +++ b/tests/unit/test_toolchain_registry.cpp @@ -2,6 +2,7 @@ import std; import mcpp.platform; +import mcpp.toolchain.model; import mcpp.toolchain.registry; import mcpp.toolchain.triple; @@ -224,3 +225,73 @@ TEST(ToolchainRegistry, NativeGccPayloadFollowsWhatTheArchActuallyPublishes) { // Non-Linux hosts are out of scope: macOS uses llvm, Windows mingw/msvc. EXPECT_FALSE(gcc_native_payload_is_musl("aarch64", false, none)); } + +// ─── the origin axis ───────────────────────────────────────────────────── +// +// `@system` is not a general spelling and must not become one. mcpp is built +// on xlings, a user-space OS, and the design drives host dependencies to a +// minimum: a toolchain comes from a payload the manifest names, so every +// machine compiles with the same compiler. `msvc@system` is a concession to +// ONE platform — Visual Studio is very often already installed and cannot +// always be redistributed. + +TEST(ToolchainOrigin, MsvcIsTheOnlyFamilyWithASystemSpelling) { + auto msvcSystem = parse_toolchain_spec("msvc@system"); + ASSERT_TRUE(msvcSystem.has_value()) << msvcSystem.error(); + EXPECT_TRUE(is_system_toolchain(*msvcSystem)); + EXPECT_EQ(origin_of(*msvcSystem), Origin::SystemMsvc); + + // A VERSIONED msvc spec is the other origin — that split is the whole + // point of the version axis. + auto pinned = parse_toolchain_spec("msvc@14.44.35207"); + ASSERT_TRUE(pinned.has_value()); + EXPECT_EQ(origin_of(*pinned), Origin::Managed); + + auto gcc = parse_toolchain_spec("gcc@16.1.0"); + ASSERT_TRUE(gcc.has_value()); + EXPECT_EQ(origin_of(*gcc), Origin::Managed); +} + +TEST(ToolchainOrigin, NonMsvcSystemIsRejectedWhereItIsReadAndOffersTheAlternatives) { + // It used to parse, and then fail somewhere else entirely as + // `xim:gcc@system` → "no such package" — which sends the reader looking + // for a version that was never going to exist. The error has to name both + // things the user might have meant. + for (auto spec : {"gcc@system", "llvm@system"}) { + auto r = parse_toolchain_spec(spec); + ASSERT_FALSE(r.has_value()) << spec << " was accepted"; + EXPECT_NE(r.error().find("only msvc"), std::string::npos) << r.error(); + EXPECT_NE(r.error().find("@"), std::string::npos) + << "the pin alternative is not offered: " << r.error(); + // The family-less PATH escape hatch is the OTHER thing they might + // have wanted, and it is a different mechanism. + EXPECT_NE(r.error().find("PATH compiler"), std::string::npos) + << "the escape hatch is not offered: " << r.error(); + EXPECT_EQ(r.error().find("xim:"), std::string::npos) + << "still leaking the package spelling that cannot exist: " + << r.error(); + } +} + +// The Linux sysroot payloads (`xim:glibc` + `xim:linux-headers`) had two +// derivations, and a comment on one claimed it mirrored the other. It did +// not: the PE term was missing from the second. +TEST(ToolchainSysrootDeps, OneDerivationForTheGlibcSysrootPayloads) { + mcpp::toolchain::triple::Triple host{}; // empty = host + mcpp::toolchain::triple::Triple musl{std::string(mcpp::platform::host_arch), "linux", "musl"}; + mcpp::toolchain::triple::Triple mingw{"x86_64", "windows", "gnu"}; + + if constexpr (mcpp::platform::is_linux) { + EXPECT_TRUE(needs_linux_sysroot_payloads(host)); + // Self-contained: a musl payload carries its own C library. + EXPECT_FALSE(needs_linux_sysroot_payloads(musl)); + // THE TERM THAT WAS MISSING. A PE target brings its own CRT, whether + // it is a native MinGW or the Linux-hosted cross, so a Linux sysroot + // is not part of installing one. + EXPECT_FALSE(needs_linux_sysroot_payloads(mingw)); + } else { + // No Linux sysroot exists to want. + for (auto const& t : {host, musl, mingw}) + EXPECT_FALSE(needs_linux_sysroot_payloads(t)); + } +} From ad2e446cf2acac65f10bb861296d3fcf1b21f816 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 04:46:23 +0800 Subject: [PATCH 03/10] =?UTF-8?q?feat(pack):=20read=20the=20import=20table?= =?UTF-8?q?=20instead=20of=20running=20the=20binary=20(=C2=A74)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `mcpp pack` refused Windows with `#if defined(_WIN32)`, and the reason given was that the tools were POSIX-only. That was the symptom. The cause is one layer down: the dependency closure comes from LD_TRACE_LOADED_OBJECTS=1 '' which RUNS the artifact — so it can cross neither an OS (a Linux box cannot execute a PE) nor an ARCHITECTURE (an x86_64 box cannot execute an aarch64 ELF, same OS or not). Porting `tar` would not have helped, and every tool the 2026-05-19 design proposed — dumpbin, ImageNtHeader, Compress-Archive — would have reintroduced the obstacle one layer down, because each exists only on the platform where the problem had already gone away. mcpp.pack.binfmt reads it out of the file instead: ELF DT_NEEDED through the segment table, PE imports AND delay-imports (a missing delay-load does not fail at startup — it fails at the first call through it, which is strictly worse to debug). Cross-OS packaging is then not a feature that had to be added; it is what remains once nothing has to be executed. mcpp.pack.zip writes the archive, for the same reason: no zip tool exists on every host (GNU tar cannot write zip, `zip(1)` is often absent, Compress-Archive is Windows-only). Entries are STORED, which is a real size cost and the honest trade — a DEFLATE encoder is the one part that could produce an archive that unpacks WRONG rather than failing loudly, and mcpp has no zlib to borrow one from. Deterministic by construction: no timestamps are read, so a published checksum means something. THE CONTRACT NOW REACHES PACKAGING (§4.3). `cxx_runtime` used to stop at the compile and link flags, so the step that decides which files actually travel could not see what had been promised — on ELF the `ldd` closure agreed with it by luck, on PE nothing did. It is now an input: toolchain-coupled the toolchain's runtime directory joins the search set host-coupled it stays OUT, so a vcruntime140.dll in the toolset is not silently swept into a package that promised the host would provide it --mode system/static + toolchain-coupled → refused, naming the way out PE layout is flat, and that is the relocation mechanism rather than a style: the Win32 loader resolves a DLL from the directory of the executable, and there is no rpath to point elsewhere. Windows' own DLLs are never bundled — two of something that must be unique is a broken program, not a heavier one — but `force_bundle` overrides that, as it always did on ELF. Verified on a Linux host against a real cross-built PE, not only synthesised fixtures: e2e 240 builds a mingw target, drops a stand-in for a DLL the EXE imports, packs, and has PYTHON verify the archive. The msvcrt.dll assertion is the positive half (a parser that read nothing could not have produced it) and the kernel32.dll assertion the negative half; together they are decisive. It runs in the mingw-cross job because running it on Windows would prove nothing. Docs: the Windows layout, the cross-host story and both size/determinism consequences in 02-pack-and-release (en+zh); the MSVC half of `cxx_runtime` in 05-mcpp-toml (en+zh), replacing a claim about /MT that stopped being true; SDK-by-origin and the `@system` rule in 03-toolchains (en; the zh MSVC section was rewritten — it still described msvc as a system-only toolchain and `msvc@19.44` as a pin-verify). 2026-05-19-pack-windows-design.md is marked superseded with what it got wrong and why, since the mistake is instructive. --- .../docs/2026-05-19-pack-windows-design.md | 32 +- .github/workflows/cross-build-test.yml | 12 + docs/02-pack-and-release.md | 66 +++ docs/03-toolchains.md | 40 +- docs/05-mcpp-toml.md | 37 +- docs/zh/02-pack-and-release.md | 59 +++ docs/zh/03-toolchains.md | 127 ++++- docs/zh/05-mcpp-toml.md | 32 +- src/pack/binfmt.cppm | 480 ++++++++++++++++++ src/pack/pack.cppm | 333 ++++++++++-- src/pack/pipeline.cppm | 27 +- src/pack/zip.cppm | 213 ++++++++ tests/e2e/240_pack_pe_zip_cross.sh | 161 ++++++ tests/unit/test_pack_binfmt.cpp | 365 +++++++++++++ 14 files changed, 1908 insertions(+), 76 deletions(-) create mode 100644 src/pack/binfmt.cppm create mode 100644 src/pack/zip.cppm create mode 100755 tests/e2e/240_pack_pe_zip_cross.sh create mode 100644 tests/unit/test_pack_binfmt.cpp diff --git a/.agents/docs/2026-05-19-pack-windows-design.md b/.agents/docs/2026-05-19-pack-windows-design.md index 14544d03..093f31a5 100644 --- a/.agents/docs/2026-05-19-pack-windows-design.md +++ b/.agents/docs/2026-05-19-pack-windows-design.md @@ -1,7 +1,37 @@ # Windows Pack Design **Date:** 2026-05-19 -**Status:** Planned (stub guard in place, implementation not yet started) +**Status:** SUPERSEDED and implemented, 2026-08-17. See +`2026-08-16-windows-toolchain-three-axes-design.md` §4 for the design that +shipped, and `src/pack/binfmt.cppm` / `src/pack/zip.cppm` for the code. + +> ## What this document got wrong, and it is worth keeping +> +> Everything below assumes **pack runs on Windows**. That assumption is +> visible in every proposal: `dumpbin /dependents`, `ImageNtHeader` from +> ``, PowerShell's `Compress-Archive`, and an implementation +> "under `#if defined(_WIN32)`". +> +> The assumption came from the guard it was trying to remove. `pack` refused +> Windows, so the problem looked like "pack has no Windows branch". It was +> not: the ELF closure is derived by RUNNING the artifact +> (`LD_TRACE_LOADED_OBJECTS=1 ''`), so it can cross neither an OS nor +> an ARCHITECTURE — a Linux box cannot trace a PE, and an x86_64 box cannot +> trace an aarch64 ELF either. The `#if defined(_WIN32)` was that limitation +> surfacing at the nearest place a user would hit it. +> +> Reading the import table statically removes both limits at once, and then +> cross-OS packaging is not a feature that had to be added — it is what +> remains when the obstacle is gone. Each Windows-only tool above would have +> reintroduced the obstacle one layer down. +> +> What did survive from here: the `.zip` output, the flat +> DLLs-beside-the-`.exe` layout, no wrapper script, and the skip-list concept +> (with one correction — `vcruntime*.dll` is listed below as a system DLL, and +> it is not: it belongs to the TOOLSET, and whether it travels is +> `cxx_runtime`'s decision). + +--- ## Current state diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 68a072ff..379def22 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -308,6 +308,18 @@ jobs: export MCPP_VENDORED_XLINGS="$XLINGS_BIN" bash tests/e2e/198_windows_resources_cross.sh + # Packaging a Windows program FROM LINUX — and this job is the only + # place that can happen, for the same reason as the two above. + # + # Running it on a Windows runner would prove nothing: the point of + # reading the import table instead of executing the artifact + # (mcpp.pack.binfmt) is precisely that the packaging host need not be + # the target. A same-OS pack cannot tell the two implementations apart. + - name: "e2e: pack a PE from Linux (zip + DLL closure)" + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + bash tests/e2e/240_pack_pe_zip_cross.sh + # ── windows → linux ─────────────────────────────────────────────────────── # The mirror of mingw-cross-wine. Two jobs because a Windows runner cannot # execute the ELF it produces; the artefact is handed to a Linux job and diff --git a/docs/02-pack-and-release.md b/docs/02-pack-and-release.md index 589785f5..195d7b27 100644 --- a/docs/02-pack-and-release.md +++ b/docs/02-pack-and-release.md @@ -236,6 +236,72 @@ own resolution, say — use `--mode vendored` instead. It repoints `PT_INTERP` at the host loader, at the cost of requiring the host's glibc to be at least as new as the one the artifact was built against. +### Windows (PE) — a `.zip`, and the DLLs sit beside the `.exe` + +A Windows target produces a **`.zip`**, not a `.tar.gz`, and the layout is +flat: + +``` +target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip +└── myapp-0.1.0-x86_64-pc-windows-msvc/ + ├── myapp.exe + ├── vcruntime140.dll ← only under cxx_runtime = "toolchain-coupled" + ├── mydep.dll ← third-party dependencies + ├── README.md + └── LICENSE +``` + +There is no `bin/` + `lib/` split and no entry-point wrapper, and neither is a +style choice. The Win32 loader resolves a DLL from **the directory of the +executable**; PE has no `RUNPATH` to point anywhere else, so "next to the +`.exe`" *is* the mechanism that `$ORIGIN/../lib` provides on ELF. + +**Windows' own DLLs are never bundled** — `kernel32.dll`, `ntdll.dll`, +`ucrtbase.dll`, the `api-ms-win-*` API sets. Shipping a private copy of an OS +component is a broken program rather than a heavier one (the process ends up +with two of something that must be unique), and Microsoft's redistribution +terms say the same thing from the other side. `[pack.bundle-project] +force_bundle` still overrides this, as it does the ELF skip list. + +`vcruntime140.dll` and `msvcp140.dll` are **not** Windows' own: they belong to +the MSVC toolset, exactly as `libstdc++.so` belongs to gcc. Whether they +travel is decided by `cxx_runtime` (see `docs/05-mcpp-toml.md`), not by this +list — and `mcpp pack` refuses a combination that cannot deliver what the +contract promised: + +``` +$ mcpp pack --mode system # with cxx_runtime = "toolchain-coupled" +error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other. +``` + +#### Packing a Windows program from Linux or macOS + +This works, and it is not a special mode — just build for a Windows target and +pack: + +```bash +mcpp pack --target x86_64-windows-gnu # from a Linux host +``` + +`mcpp pack` used to refuse Windows outright. The reason was not the archiver: +the ELF dependency closure is obtained by **running the artifact** under +`LD_TRACE_LOADED_OBJECTS`, which cannot cross an OS *or* an architecture. A PE +closure is read out of the file's import table instead, so nothing has to be +executed and the packaging host is free. The archive is written by mcpp itself +for the same reason — there is no zip tool present on every host. + +Two consequences worth knowing: + +- Entries are **stored, not deflated**, so a Windows package is roughly the + size of its contents. Compression is a size optimization, not a correctness + one, and it is not implemented yet. +- The archive is **deterministic**: no timestamps are read, so two packs of + the same tree are byte-identical and a published checksum means something. + +The reverse direction — packing a Linux or macOS artifact *from* Windows — +still does not work, and for the original reason: that closure is resolved by +the target's own dynamic linker, which a Windows host has no way to run. + ## Configuration Packaging behavior is configured via the `[pack]` section in `mcpp.toml`. The diff --git a/docs/03-toolchains.md b/docs/03-toolchains.md index c6be373c..1ebc8b69 100644 --- a/docs/03-toolchains.md +++ b/docs/03-toolchains.md @@ -228,6 +228,17 @@ questions. `msvc@system` asks *"use what this developer already has"*; `msvc@14.44.35207` asks *"build this project with exactly this compiler"*. Pinned toolsets coexist with each other and with a system Visual Studio. +> **`@system` is an MSVC-only spelling.** There is no `gcc@system` or +> `llvm@system`, and that is deliberate rather than an omission: mcpp is built +> on xlings, a user-space OS, and the design drives host dependencies to a +> minimum — a toolchain comes from a payload the manifest names, so every +> machine builds with the same compiler. Windows is the one place where +> refusing to use what is already installed would cost more than it buys: +> Visual Studio is very often present and cannot always be redistributed. +> `@system` for any other family is an error that names both things +> you might have meant. (The family-less `[toolchain] … = "system"` — the PATH +> compiler — is a separate and deliberate escape hatch, and is unaffected.) + ### `msvc@system` — the machine's own Visual Studio mcpp locates and identifies an installed Visual Studio / Build Tools; it never @@ -304,11 +315,23 @@ environment from the VC tools + Windows SDK (no `vcvarsall` involved), stages `/interface /TP /ifcOutput`, scans with `/scanDependencies`, and links with `link.exe`/`lib.exe` through response files. -The Windows SDK is located in this order: **`WindowsSdkDir`** (+ -`WindowsSdkVersion`) if declared, then the `xim:windows-sdk` payload beside a -pinned toolset in mcpp's store, then `C:\Program Files (x86)\Windows Kits\10`. -A missing SDK fails the build with guidance (`mcpp self doctor` reports SDK -status). +**The Windows SDK follows the origin**, because the two origins answer +different questions and so must the SDK: + +| origin | how the SDK is chosen | +|---|---| +| `msvc@` | the `xim:windows-sdk` payload installed **with that toolset**, in mcpp's own store. `WindowsSdkDir` / `WindowsSdkVersion` in the environment are **ignored**, and mcpp prints a `note:` saying so. | +| `msvc@system` | **`WindowsSdkDir`** (+ `WindowsSdkVersion`) if declared, then `C:\Program Files (x86)\Windows Kits\10`. | + +The asymmetry is the point. A pinned toolset is a promise that two machines +compile the same source against the same headers; an environment variable that +can quietly redirect it turns the pin into a preference. A machine's own SDK, +on the other hand, can only be found by looking, and there a declared answer +outranks a scan — the same precedence `VSINSTALLDIR` has over `vswhere`. + +If a pinned toolset has no SDK payload beside it (an older install, say), mcpp +falls back to the machine's SDK rather than failing — and says so, because that +build is no longer reproducible and nothing else would record it. A root only counts as an SDK when it has **both** halves — `Include\\ucrt\ corecrt.h` *and* `Lib\\um\\kernel32.lib`. A root with headers and no @@ -316,6 +339,13 @@ import libraries is skipped rather than selected, so a partially unpacked payload cannot outrank the machine's complete SDK and turn into `LNK1104: cannot open file 'kernel32.lib'` at the very end of a build. +The resolved SDK version is part of the build's **runtime identity** +(`ucrt@10.0.26100.0`) and therefore of the fingerprint that keys the build +cache: changing SDK changes the cache key, exactly as changing compiler does. +It is a **compatibility floor declaration**, not a payload binding like +`glibc@2.39` on Linux — `ucrtbase.dll` is a Windows component and mcpp neither +ships nor substitutes it. + **CRT model.** `/MD` (host-coupled) by default; `/MT` when either ```toml diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index 9e025c6f..1d1b6a82 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -395,10 +395,39 @@ default applies only when nobody said anything. `self-contained`, `false` means `host-coupled`. An explicit `cxx_runtime` wins. **A contract that cannot be honored is reported, never silently downgraded.** If a -toolchain ships no `libc++.a`, or a contract has no mechanism on that platform -(`self-contained` under the MSVC runtime would need `/MT`, which mcpp does not emit -yet), the build prints what it fell back to instead of quietly producing a -different artifact than the manifest asked for. +toolchain ships no `libc++.a`, or a contract has no mechanism on that platform, +the build prints what it fell back to instead of quietly producing a different +artifact than the manifest asked for. + +#### On the MSVC runtime + +The CRT model is the mechanism here, and it is a **whole-project** switch: cl +bakes `_MSVC_MT`/`_MSVC_MD` into the one `std` module a project builds, so a +per-role contract that disagrees with the project's cannot be honoured and is +reported rather than ignored. + +| value | what it is on MSVC | +|---|---| +| `self-contained` | `/MT` — the static CRT. `linkage = "static"` selects the same thing from the libc axis. | +| `host-coupled` (default under `/MD`) | the target provides `vcruntime140.dll` / `msvcp140.dll` — i.e. Visual Studio or the redistributable is installed there. | +| `toolchain-coupled` | the toolset's **own** copy of those DLLs travels with the artifact. | + +`toolchain-coupled` is worth spelling out, because the obvious reading is +wrong. `ucrtbase.dll` *is* a Windows component (since Windows 10) and mcpp +never ships it. `vcruntime140.dll` and `msvcp140.dll` are **not**: every MSVC +toolset carries them under `VC\Redist\MSVC\\\`, exactly the +way a gcc payload carries `libstdc++.so`. Under this contract mcpp stages them +beside the artifact — which is what makes a default `/MD` build runnable on a +machine that has only the pinned toolset and no Visual Studio at all. + +The debug CRT (`vcruntime140d.dll` and friends, under `debug_nonredist\`) is +never staged: it may not be redistributed. + +Combining it with `/MT` is a contradiction rather than a missing feature — a +static CRT leaves no DLL to couple to — so it is reported and resolved to +`self-contained`. `mcpp pack` enforces the other half: a mode that bundles +nothing (`--mode system`, `--mode static`) cannot deliver `toolchain-coupled` +and refuses. **Scope.** The contract governs the C++ runtime only. Static **libc** is a separate axis (`linkage = "static"` / `--static`, e.g. a musl target), and the deployment diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index e6eaff0a..66adde31 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -172,6 +172,65 @@ if (!base) { 它把 `PT_INTERP` 重指到宿主 loader,`/proc/self/exe` 正常,代价是要求宿主 glibc 不低于构建时所用的那份。 +### Windows(PE)—— 产物是 `.zip`,DLL 就放在 `.exe` 旁边 + +Windows 目标产出的是 **`.zip`** 而不是 `.tar.gz`,并且是扁平布局: + +``` +target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip +└── myapp-0.1.0-x86_64-pc-windows-msvc/ + ├── myapp.exe + ├── vcruntime140.dll ← 仅在 cxx_runtime = "toolchain-coupled" 时 + ├── mydep.dll ← 第三方依赖 + ├── README.md + └── LICENSE +``` + +没有 `bin/` + `lib/` 的分层,也没有入口 wrapper —— 这两点都不是风格选择。 +Win32 loader 解析 DLL 的第一顺位就是**可执行文件所在目录**,而 PE 没有 +`RUNPATH` 可以指向别处,所以"放在 `.exe` 旁边"**就是** ELF 上 +`$ORIGIN/../lib` 所提供的那个机制。 + +**Windows 自己的 DLL 永远不会被打进包里** —— `kernel32.dll`、`ntdll.dll`、 +`ucrtbase.dll`、`api-ms-win-*` API set。带一份系统组件的私有拷贝不是"包变大 +了",而是**程序坏了**(进程里出现了两份本应唯一的东西),而 Microsoft 的再分发 +条款也从另一侧说了同一件事。`[pack.bundle-project] force_bundle` 仍然可以覆盖 +这条,和它覆盖 ELF 跳过表一样。 + +`vcruntime140.dll` / `msvcp140.dll` **不是** Windows 自己的:它们属于 MSVC +toolset,就像 `libstdc++.so` 属于 gcc。它们要不要跟着产物走,由 `cxx_runtime` +决定(见 `docs/zh/05-mcpp-toml.md`),不由这张表决定 —— 而 `mcpp pack` 会拒绝 +那些无法兑现契约的组合: + +``` +$ mcpp pack --mode system # 且 cxx_runtime = "toolchain-coupled" +error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other. +``` + +#### 在 Linux / macOS 上给 Windows 打包 + +这是可以的,而且不是什么特殊模式 —— 指定 Windows 目标构建,然后打包即可: + +```bash +mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上 +``` + +`mcpp pack` 以前直接拒绝 Windows。真正的原因不是打包工具:ELF 的依赖闭包是靠 +**把产物跑起来**(`LD_TRACE_LOADED_OBJECTS`)求出来的,所以它既跨不了 OS 也跨 +不了架构。PE 的闭包改为从文件的导入表里**读**出来,于是不需要执行任何东西, +打包宿主也就自由了。压缩包本身也由 mcpp 自己写,理由相同 —— 没有哪个 zip 工具 +在每个宿主上都存在。 + +有两点值得知道: + +- 条目是 **stored(不压缩)** 的,所以 Windows 包的体积约等于其内容之和。压缩 + 是体积优化而不是正确性问题,目前尚未实现。 +- 压缩包是**确定性**的:不读取任何时间戳,同一棵树打两次字节一致,公布的校验和 + 才有意义。 + +反方向 —— 在 Windows 上给 Linux / macOS 产物打包 —— 仍然不支持,原因还是最初 +那个:那条闭包要由目标自己的动态链接器解析,而 Windows 宿主没有办法运行它。 + ## 配置项 打包行为通过 `mcpp.toml` 中的 `[pack]` 节配置,常用字段如下: diff --git a/docs/zh/03-toolchains.md b/docs/zh/03-toolchains.md index 20912eb4..ae97488a 100644 --- a/docs/zh/03-toolchains.md +++ b/docs/zh/03-toolchains.md @@ -203,19 +203,46 @@ payload 是按宿主 arch 构建的。**macOS 宿主则完全没有面向 Linux Targets 一栏里没有的,就是这台机器确实服务不了(实现见 `toolchain::host_can_serve`)。 -## MSVC(系统工具链,Windows) +## MSVC(Windows) -MSVC 与 mcpp 管理的其它工具链都不同:它是一条**系统工具链**。mcpp 只负责 -定位并识别已安装的 Visual Studio / Build Tools —— **从不**安装、升级或卸载 -MSVC 本身。 +一个 MSVC toolset 有两条路径进入构建,由 **spec 的版本轴**决定是哪一条: + +| Spec | 来源 | 你拿到的是哪个编译器 | +|---|---|---| +| `msvc@system`(或裸 `msvc`) | 这台机器自己的 Visual Studio | 这里恰好装了什么就是什么 | +| `msvc@`(如 `msvc@14.44.35207`) | mcpp 安装的 xlings payload | 你点名的那一个,在每台机器上都一样 | + +它们不是"二选一",而是回答了不同的问题。`msvc@system` 问的是*"用这位开发者 +已经有的东西"*;`msvc@14.44.35207` 问的是*"用恰好这个编译器构建本工程"*。 +pinned toolset 之间、以及与系统 Visual Studio 之间都可以共存。 + +> **`@system` 是 MSVC 独有的拼写。** 没有 `gcc@system`,也没有 `llvm@system`, +> 这是有意的而不是漏了:mcpp 建立在用户态 OS xlings 之上,整条设计就是**把 host +> 依赖降到最低** —— 工具链来自 manifest 点名的 payload,于是每台机器用同一个 +> 编译器。Windows 是唯一一处"拒绝使用已装好的东西"代价大于收益的地方: +> Visual Studio 常常已经装了,又不总能重新分发。其它族写 `@system` 会 +> 直接报错,并同时给出你可能想要的两种写法。(不带族的 +> `[toolchain] … = "system"` —— 即 PATH 上的编译器 —— 是另一套、也是有意保留的 +> 逃生口,不受影响。) + +### `msvc@system` —— 机器自己的 Visual Studio + +mcpp 只负责定位并识别已安装的 Visual Studio / Build Tools,**从不**安装、 +升级或卸载它。 ```bash mcpp toolchain default msvc ``` -在装有 MSVC 的机器上,mcpp 会自动定位(依次尝试 `vswhere.exe`、 -`VSINSTALLDIR`/`VS*COMNTOOLS`、标准安装路径),识别涉及的各个版本, -并持久化为稳定 spec `msvc@system`: +定位顺序: + +1. **`VSINSTALLDIR`** —— 由开发者命令提示符或跑过 `vcvarsall` 的 CI 步骤设置。 + 这是一个**回答**而不是猜测,所以排在下面几种探测之前。 +2. `vswhere.exe`(含 prerelease / Insiders 实例) +3. `VS*COMNTOOLS` +4. 标准的 `Program Files\Microsoft Visual Studio\\` 路径 + +随后识别涉及的各个版本,并持久化为稳定 spec `msvc@system`: ``` Detected msvc 19.44.35211 (VS 2022 BuildTools) (VC tools 14.44.35207) @@ -224,29 +251,87 @@ Detected msvc 19.44.35211 (VS 2022 BuildTools) (VC tools 14.44.35207) Default set to msvc@system (was: llvm@20.1.7) ``` -若机器上**没有** MSVC,mcpp 打印安装指引(Visual Studio Installer 勾选 -*Desktop development with C++* 负载,或 `winget install -Microsoft.VisualStudio.2022.BuildTools`)并以非零码退出 —— 需要你自己装好, -再重跑该命令。 +若机器上没有 Visual Studio,mcpp 会说出来并同时给出两条路:一个它可以替你装的 +pinned toolset,或者 Visual Studio Installer / +`winget install Microsoft.VisualStudio.2022.BuildTools`。 `mcpp toolchain list` 会把检测到的 MSVC 列在单独的 `System:` 分区, -`mcpp self doctor` 在 Windows 上会报告它的状态。manifest 里可按平台 pin: +`mcpp self doctor` 在 Windows 上会报告它的状态。manifest 里: ```toml [toolchain] windows = "msvc@system" ``` -`msvc@<前缀>`(如 `msvc@19.44`)是一个 **pin-verify**:mcpp 仍然使用已安装 -的最新 VC tools,但检测到的版本与前缀不符时报错。 +### `msvc@` —— mcpp 安装并 pin 的 toolset + +```bash +mcpp toolchain list --available msvc # 可以 pin 哪些 +mcpp toolchain install msvc 14.44.35207 +``` + +这在每个方面都和 `gcc@16.1.0` 一样:payload 下载进 mcpp 自己的 store,多个 +toolset 共存,`mcpp toolchain remove msvc@` 卸载其中一个,manifest 里 +点名的那个会在首次构建时自动安装。 + +```toml +[toolchain] +windows = "msvc@14.44.35207" +``` + +**这里的版本是 toolset 目录名**(`14.44.35207` —— 即 `VC\Tools\MSVC\` 下的 +目录名、也是 `-vcvars_ver` 接受的值),**不是** cl banner 版本 +(`19.44.35211`),也不是产品年份。机器上不需要预装任何东西:payload 带来编译器、 +STL,以及通过它的 `xim:windows-sdk` 依赖带来 ucrt/um 的头文件与导入库。 + +> **已变更:** `msvc@19.44` 过去的含义是"用系统 MSVC,并校验其 banner 以 19.44 +> 开头" —— 这条只有 `mcpp toolchain default` 会查,构建则静默忽略。现在版本轴 +> 在所有地方都指 toolset。写成 `19.x` 会得到一条同时给出两种替代写法的错误 +> —— `msvc@system`,或那台机器实际拥有的 toolset 版本。 + +### 原生 cl.exe 构建 + +自 0.0.90 起两条来源都可用:mcpp 从 VC tools + Windows SDK 合成 INCLUDE/LIB +环境(不经 `vcvarsall`),把 `std.ixx`/`std.compat.ixx` staging 成 `.ifc` BMI, +用 `/interface /TP /ifcOutput` 编译 `.cppm` 模块单元,用 `/scanDependencies` +扫描,并通过 response file 调 `link.exe`/`lib.exe` 链接。 + +**Windows SDK 跟着来源走**,因为两条来源回答的是不同的问题,SDK 也必须如此: + +| 来源 | SDK 怎么选 | +|---|---| +| `msvc@` | **随该 toolset 一起装进 mcpp store 的** `xim:windows-sdk` payload。环境里的 `WindowsSdkDir` / `WindowsSdkVersion` 会被**忽略**,并且 mcpp 会打印一行 `note:` 说明。 | +| `msvc@system` | 先 **`WindowsSdkDir`**(+ `WindowsSdkVersion`),再 `C:\Program Files (x86)\Windows Kits\10`。 | + +这种不对称正是要点。pin 一个 toolset 是在承诺"两台机器用同一套头文件编同一份 +源码";一个能悄悄改写它的环境变量会把这条承诺降格成偏好。反过来,机器自己的 +SDK 只能靠找,而在那里"明确声明"应当压过"扫描" —— 和 `VSINSTALLDIR` 压过 +`vswhere` 是同一条优先级。 + +如果一个 pinned toolset 旁边没有 SDK payload(比如较老的安装),mcpp 会退回用 +机器上的 SDK 而不是失败 —— 并且会说出来,因为那次构建已经不可复现,而除此之外 +没有任何东西会记录这件事。 + +一个根目录只有**两半都在**才算 SDK —— `Include\\ucrt\corecrt.h` **且** +`Lib\\um\\kernel32.lib`。只有头文件、没有导入库的根会被跳过而不是 +被选中,于是一个只解包了一半的 payload 不会压过机器上完整的 SDK,也就不会在 +构建的最后一刻变成 `LNK1104: cannot open file 'kernel32.lib'`。 + +解析出的 SDK 版本是这次构建的**运行时身份**(`ucrt@10.0.26100.0`)的一部分, +因而也进入了给构建缓存做 key 的那个指纹:换 SDK 就换缓存键,和换编译器一样。 +它是一条**兼容性下限声明**,而不是 Linux 上 `glibc@2.39` 那种 payload 绑定 —— +`ucrtbase.dll` 是 Windows 组件,mcpp 既不分发也不替换它。 + +**CRT 模型。** 默认 `/MD`(host-coupled);下面两者任一都会选 `/MT`: + +```toml +[build] +linkage = "static" # libc 那根轴 +cxx_runtime = "self-contained" # C++ 运行时那根轴 +``` -自 0.0.90 起,**原生 cl.exe 构建可用**:mcpp 从检测到的 VC tools + Windows -SDK 合成 INCLUDE/LIB 环境(不经 `vcvarsall`),把 `std.ixx`/`std.compat.ixx` -staging 成 `.ifc` BMI,用 `/interface /TP /ifcOutput` 编译 `.cppm` 模块单元, -用 `/scanDependencies` 扫描,并通过 response file 调 `link.exe`/`lib.exe` -链接。选择 `/MT` CRT 用 `[target.x86_64-windows-msvc] linkage = "static"` -(或 `mcpp build --static`)—— 不是 `[build] linkage`,那个键不存在。 -缺 Windows SDK 会让构建失败并给出安装指引(`mcpp self doctor` 会报告 SDK 状态)。 +toolset 自带的那份可再分发 CRT(`vcruntime140.dll` / `msvcp140.dll`)可以跟着 +产物走 —— 见 `docs/zh/05-mcpp-toml.md` 的 `cxx_runtime = "toolchain-coupled"`。 ## 项目级版本锁定 diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 01bdf315..b6af2965 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -349,9 +349,35 @@ C++ 运行时的进程。 等价于 `host-coupled`。显式写了 `cxx_runtime` 时以后者为准。 **兑现不了的契约会被报出来,绝不静默降级。** 若工具链不带 `libc++.a`,或某个 -契约在该平台上没有对应机制(MSVC 运行时的 `self-contained` 需要 `/MT`,mcpp -目前不发射),构建会打印实际退到了哪一档,而不是悄悄交付一个与 manifest 所述 -不同的产物。 +契约在该平台上没有对应机制,构建会打印实际退到了哪一档,而不是悄悄交付一个与 +manifest 所述不同的产物。 + +#### 在 MSVC 运行时上 + +这里的机制就是 CRT 模型,而它是**整个工程级**的开关:cl 会把 `_MSVC_MT` / +`_MSVC_MD` 烘进工程唯一的那份 `std` 模块,所以与工程不一致的按角色契约无法兑现, +会被报出来而不是被忽略。 + +| 取值 | 在 MSVC 上是什么 | +|---|---| +| `self-contained` | `/MT`,静态 CRT。`linkage = "static"` 从 libc 那根轴选中的是同一件事。 | +| `host-coupled`(`/MD` 下的默认) | 由目标机器提供 `vcruntime140.dll` / `msvcp140.dll` —— 即那台机器装了 Visual Studio 或 redistributable。 | +| `toolchain-coupled` | toolset **自带**的那份 DLL 跟着产物走。 | + +`toolchain-coupled` 值得说清楚,因为直觉上的理解是错的。`ucrtbase.dll` **是** +Windows 组件(Win10 起),mcpp 从不分发它;而 `vcruntime140.dll` / +`msvcp140.dll` **不是**:每个 MSVC toolset 都在 +`VC\Redist\MSVC\\\` 下带着它们,和 gcc payload 带着 +`libstdc++.so` 是同一件事。在这个契约下 mcpp 会把它们放到产物旁边 —— 这正是让 +默认的 `/MD` 产物能在"只装了 pinned toolset、根本没有 Visual Studio"的机器上跑 +起来的原因。 + +调试版 CRT(`debug_nonredist\` 下的 `vcruntime140d.dll` 等)永远不会被放进去: +它不可再分发。 + +把它和 `/MT` 一起用是**矛盾**而不是缺功能 —— 静态 CRT 根本没有 DLL 可以耦合 —— +所以会被报出来并落到 `self-contained`。另一半由 `mcpp pack` 兜底:什么都不打包的 +模式(`--mode system`、`--mode static`)兑现不了 `toolchain-coupled`,会直接拒绝。 **边界。** 该契约只管 C++ 运行时。静态 **libc** 是另一根轴(`linkage = "static"` / `--static`,如 musl 目标),部署下限是第三根轴(`macos_deployment_target`)。 diff --git a/src/pack/binfmt.cppm b/src/pack/binfmt.cppm new file mode 100644 index 00000000..c1dddb47 --- /dev/null +++ b/src/pack/binfmt.cppm @@ -0,0 +1,480 @@ +// mcpp.pack.binfmt — what a binary depends on, read from the FILE. +// +// WHY THIS MODULE EXISTS +// +// `pack` used to derive an artifact's dependency closure like this: +// +// LD_TRACE_LOADED_OBJECTS=1 '' +// +// That is the dynamic linker being asked to resolve the closure, which is +// accurate and complete — and requires RUNNING THE BINARY. So it is not +// "pack has no Windows branch": it cannot cross an OS (a Linux box cannot +// execute a PE) and it cannot cross an ARCHITECTURE (an x86_64 box cannot +// execute an aarch64 ELF, same OS or not). The `#if defined(_WIN32)` refusal +// at the top of `pack::run()` was the symptom; this was the cause. +// +// Reading the file instead removes both limits at once, because nothing here +// executes anything. Cross-OS packaging then is not a feature that had to be +// added — it is what remains when the obstacle is gone. +// +// See .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §4. +// +// WHAT THIS MODULE DOES NOT DO. It answers "which names does this object +// ask for", not "which files will the loader hand it". Resolution is the +// caller's, because the search rules are the loader's policy — and on ELF +// they are genuinely intricate (DT_RPATH before LD_LIBRARY_PATH before +// DT_RUNPATH before ld.so.cache, `$ORIGIN` expansion, hwcaps subdirectories). +// mcpp does not reimplement that: on the host's own format it still asks the +// loader, and this reader serves the cases the loader cannot be asked about. + +export module mcpp.pack.binfmt; + +import std; + +export namespace mcpp::pack::binfmt { + +enum class Format { Unknown, Elf, Pe, MachO }; + +// The identity a packaging decision needs: can THIS machine run that file? +// Both halves matter and only one of them used to be considered. +struct Ident { + Format format = Format::Unknown; + // The object's own machine, in mcpp's canonical arch spelling + // ("x86_64", "aarch64", "i686", …). Empty when unrecognised — a new + // machine type is not a corrupt file, and reporting it as one would be + // worse than saying nothing. + std::string arch; + bool is64 = false; +}; + +std::string_view format_name(Format f) { + switch (f) { + case Format::Elf: return "ELF"; + case Format::Pe: return "PE"; + case Format::MachO: return "Mach-O"; + case Format::Unknown: break; + } + return "unknown"; +} + +// Identify by magic. Never throws, never runs anything; an unreadable or +// truncated file simply comes back Unknown. +Ident identify(const std::filesystem::path& binary); + +// The DIRECT dependency names an object declares: +// ELF DT_NEEDED entries ("libstdc++.so.6") +// PE import + delay-import DLL names ("vcruntime140.dll") +// +// Names, in the file's own spelling and order, deduplicated. An object with +// no dynamic dependencies yields an empty list — which is a valid answer, not +// a failure, and the caller must not confuse the two (a fully static binary +// and an unparseable one had better not look alike). +std::expected, std::string> +needed_names(const std::filesystem::path& binary); + +// Is `name` provided by the target OS itself — i.e. must NOT be bundled? +// +// On ELF this is the manylinux allow-list, which `pack` already had. +// +// On PE the list is short and the reason is different: shipping a copy of a +// Windows component is not "extra weight", it is a broken program. The OS +// loader resolves `kernel32.dll` from the system directory whatever sits +// beside the executable — except when a DLL that sits beside it is loaded +// FIRST by name, at which point the process has two of something that must be +// unique. Microsoft's redistribution terms say the same thing from the other +// side. +// +// Note what is NOT here: `vcruntime140.dll` and `msvcp140.dll`. Those belong +// to the TOOLSET, not to Windows (see mcpp.build.distribution's PE branch), +// and whether they travel is a contract decision — the one thing this +// predicate must not quietly make for it. +bool is_system_lib(Format f, std::string_view name); + +} // namespace mcpp::pack::binfmt + +namespace mcpp::pack::binfmt { + +namespace detail { + +// A bounded, whole-file read. Executables are tens of megabytes at worst and +// the alternative is a seek-heavy parser that has to re-validate every offset +// against the file length anyway; with the bytes in hand, one bounds-checked +// accessor covers every read in this file. +std::optional slurp(const std::filesystem::path& p) { + std::error_code ec; + auto size = std::filesystem::file_size(p, ec); + if (ec) return std::nullopt; + // 512 MiB. A larger "executable" is not one, and refusing is better than + // allocating whatever a caller was handed. + if (size > (512ull << 20)) return std::nullopt; + std::ifstream is(p, std::ios::binary); + if (!is) return std::nullopt; + std::string buf(static_cast(size), '\0'); + is.read(buf.data(), static_cast(size)); + buf.resize(static_cast(is.gcount())); + return buf; +} + +// Little-endian integer at `off`, or nullopt if it would run off the end. +// EVERY read in this module goes through here: a malformed file is ordinary +// input (a truncated download, a text file named `.exe`), and the parser has +// to be total over it rather than trusting a length field it just read. +template +std::optional le(std::string_view b, std::size_t off) { + if (off + sizeof(T) > b.size()) return std::nullopt; + T v = 0; + for (std::size_t i = 0; i < sizeof(T); ++i) + v |= static_cast(static_cast(b[off + i])) + << (8 * i); + return v; +} + +// NUL-terminated string at `off`, bounded by the file end. +std::optional cstr(std::string_view b, std::size_t off) { + if (off >= b.size()) return std::nullopt; + auto end = b.find('\0', off); + if (end == std::string_view::npos) return std::nullopt; + return std::string(b.substr(off, end - off)); +} + +std::string elf_arch(std::uint16_t machine) { + switch (machine) { + case 0x03: return "i686"; // EM_386 + case 0x28: return "arm"; // EM_ARM + case 0x3E: return "x86_64"; // EM_X86_64 + case 0xB7: return "aarch64"; // EM_AARCH64 + case 0xF3: return "riscv64"; // EM_RISCV + case 0x15: return "ppc64"; // EM_PPC64 + case 0x08: return "mips"; // EM_MIPS + case 0x16: return "s390x"; // EM_S390 + case 0x102: return "loongarch64"; + default: return {}; + } +} + +std::string pe_arch(std::uint16_t machine) { + switch (machine) { + case 0x014c: return "i686"; // IMAGE_FILE_MACHINE_I386 + case 0x8664: return "x86_64"; // AMD64 + case 0xAA64: return "aarch64"; // ARM64 + case 0x01c4: return "arm"; // ARMNT + default: return {}; + } +} + +// ─── ELF ──────────────────────────────────────────────────────────────── +// +// DT_NEEDED lives in the .dynamic section, whose entries are (tag, value) +// pairs; the NEEDED value is a byte offset into the string table named by +// DT_STRTAB. DT_STRTAB is a VIRTUAL ADDRESS, not a file offset, so the PT_LOAD +// segments have to be walked to translate it — which is the one step that +// makes this more than "read a list". +std::expected, std::string> +elf_needed(std::string_view b) { + auto cls = le(b, 4); + auto dat = le(b, 5); + if (!cls || !dat) return std::unexpected("ELF header is truncated"); + if (*dat != 1) + return std::unexpected("big-endian ELF objects are not supported"); + const bool is64 = (*cls == 2); + if (*cls != 1 && *cls != 2) + return std::unexpected("ELF header declares an unknown class"); + + const std::size_t phoffAt = is64 ? 0x20 : 0x1C; + std::uint64_t phoff = 0; + if (is64) { + auto v = le(b, phoffAt); + if (!v) return std::unexpected("ELF program header offset is truncated"); + phoff = *v; + } else { + auto v = le(b, phoffAt); + if (!v) return std::unexpected("ELF program header offset is truncated"); + phoff = *v; + } + auto phentsize = le(b, is64 ? 0x36 : 0x2A); + auto phnum = le(b, is64 ? 0x38 : 0x2C); + if (!phentsize || !phnum) + return std::unexpected("ELF program header table is truncated"); + + struct Load { std::uint64_t vaddr, offset, filesz; }; + std::vector loads; + std::uint64_t dynOff = 0, dynSize = 0; + for (std::uint16_t i = 0; i < *phnum; ++i) { + const std::size_t ph = static_cast(phoff) + + static_cast(i) * *phentsize; + auto type = le(b, ph); + if (!type) break; + auto rd = [&](std::size_t off64, std::size_t off32) + -> std::optional { + if (is64) return le(b, ph + off64); + if (auto v = le(b, ph + off32)) return *v; + return std::nullopt; + }; + auto offset = rd(0x08, 0x04); + auto vaddr = rd(0x10, 0x08); + auto filesz = rd(0x20, 0x10); + if (!offset || !vaddr || !filesz) continue; + if (*type == 1) loads.push_back({*vaddr, *offset, *filesz}); // PT_LOAD + if (*type == 2) { dynOff = *offset; dynSize = *filesz; } // PT_DYNAMIC + } + // No PT_DYNAMIC = a static executable. Zero dependencies is the answer, + // and it is a different answer from "could not be read". + if (dynSize == 0) return std::vector{}; + + auto vaddr_to_off = [&](std::uint64_t va) -> std::optional { + for (auto const& l : loads) + if (va >= l.vaddr && va - l.vaddr < l.filesz) + return l.offset + (va - l.vaddr); + return std::nullopt; + }; + + const std::size_t entSize = is64 ? 16 : 8; + std::vector neededOffsets; + std::optional strtabVa; + for (std::uint64_t at = dynOff; at + entSize <= dynOff + dynSize; + at += entSize) { + std::uint64_t tag = 0, val = 0; + if (is64) { + auto t = le(b, static_cast(at)); + auto v = le(b, static_cast(at) + 8); + if (!t || !v) break; + tag = *t; val = *v; + } else { + auto t = le(b, static_cast(at)); + auto v = le(b, static_cast(at) + 4); + if (!t || !v) break; + tag = *t; val = *v; + } + if (tag == 0) break; // DT_NULL + if (tag == 1) neededOffsets.push_back(val); // DT_NEEDED + if (tag == 5) strtabVa = val; // DT_STRTAB + } + if (neededOffsets.empty()) return std::vector{}; + if (!strtabVa) + return std::unexpected("ELF .dynamic has DT_NEEDED but no DT_STRTAB"); + auto strOff = vaddr_to_off(*strtabVa); + if (!strOff) + return std::unexpected("ELF DT_STRTAB address is in no PT_LOAD segment"); + + std::vector out; + for (auto n : neededOffsets) { + if (auto s = cstr(b, static_cast(*strOff + n)); + s && !s->empty()) + out.push_back(std::move(*s)); + } + return out; +} + +// ─── PE ───────────────────────────────────────────────────────────────── +// +// Two directories carry DLL names and BOTH are dependencies: the ordinary +// import table (directory 1), resolved by the loader before the process +// starts, and the delay-load table (directory 13), resolved at the first call +// through it. A delay-loaded DLL that is missing does not fail at startup — +// it fails later, somewhere in the program, which is strictly worse to debug. +// Leaving it out of the closure would produce exactly that. +std::expected, std::string> +pe_needed(std::string_view b) { + auto lfanew = le(b, 0x3C); + if (!lfanew) return std::unexpected("PE: no e_lfanew"); + const std::size_t nt = *lfanew; + if (b.substr(nt, 4) != std::string_view("PE\0\0", 4)) + return std::unexpected("PE: no PE\\0\\0 signature at e_lfanew"); + + auto numSections = le(b, nt + 6); + auto optSize = le(b, nt + 20); + auto magic = le(b, nt + 24); + if (!numSections || !optSize || !magic) + return std::unexpected("PE: headers are truncated"); + // 0x10b PE32, 0x20b PE32+. They differ only in where the data directories + // start — the extra 16 bytes are the 64-bit ImageBase and friends. + std::size_t dirsAt = 0; + if (*magic == 0x10b) dirsAt = nt + 24 + 96; + else if (*magic == 0x20b) dirsAt = nt + 24 + 112; + else return std::unexpected("PE: optional header magic is neither PE32 nor PE32+"); + auto numDirs = le(b, dirsAt - 4); + if (!numDirs) return std::unexpected("PE: data directory count is truncated"); + + struct Section { std::uint32_t va, vsize, raw, rawSize; }; + std::vector
sections; + const std::size_t secAt = nt + 24 + *optSize; + for (std::uint16_t i = 0; i < *numSections; ++i) { + const std::size_t s = secAt + static_cast(i) * 40; + auto vsize = le(b, s + 8); + auto va = le(b, s + 12); + auto rawSz = le(b, s + 16); + auto raw = le(b, s + 20); + if (!vsize || !va || !rawSz || !raw) break; + sections.push_back({*va, *vsize, *raw, *rawSz}); + } + + auto rva_to_off = [&](std::uint32_t rva) -> std::optional { + for (auto const& s : sections) { + // A section's mapped size is VirtualSize, but a section whose + // VirtualSize is 0 (some linkers) still maps SizeOfRawData. + auto span = s.vsize ? s.vsize : s.rawSize; + if (rva >= s.va && rva - s.va < span) { + std::size_t off = s.raw + (rva - s.va); + if (off < b.size()) return off; + return std::nullopt; + } + } + return std::nullopt; + }; + + std::vector out; + auto push = [&](std::optional name) { + if (!name || name->empty()) return; + if (std::ranges::find(out, *name) == out.end()) + out.push_back(std::move(*name)); + }; + + // Directory 1 — imports. 20-byte descriptors, terminated by an all-zero + // one; `Name` (offset 12) is an RVA to the DLL's ASCIIZ name. + if (*numDirs > 1) { + auto rva = le(b, dirsAt + 1 * 8); + if (rva && *rva) { + if (auto at = rva_to_off(*rva)) { + for (std::size_t d = *at; ; d += 20) { + auto nameRva = le(b, d + 12); + auto oft = le(b, d); + auto ft = le(b, d + 16); + if (!nameRva || !oft || !ft) break; + if (*nameRva == 0 && *oft == 0 && *ft == 0) break; + if (auto o = rva_to_off(*nameRva)) push(cstr(b, *o)); + } + } + } + } + + // Directory 13 — delay imports. 32-byte descriptors. `grAttrs` bit 0 + // (`dlattrRva`) says the fields are RVAs; the pre-VC7 form stored virtual + // ADDRESSES instead, and nothing produced in this century does that, so a + // descriptor without the bit is skipped rather than guessed at. + if (*numDirs > 13) { + auto rva = le(b, dirsAt + 13 * 8); + if (rva && *rva) { + if (auto at = rva_to_off(*rva)) { + for (std::size_t d = *at; ; d += 32) { + auto attrs = le(b, d); + auto nameRva = le(b, d + 4); + if (!attrs || !nameRva) break; + if (*attrs == 0 && *nameRva == 0) break; + if ((*attrs & 1u) == 0) continue; + if (auto o = rva_to_off(*nameRva)) push(cstr(b, *o)); + } + } + } + } + return out; +} + +// PEP 600 / manylinux2014: assumed present on any target Linux glibc system. +constexpr std::array kElfSystem = std::to_array({ + "libc.so", "libm.so", "libdl.so", "libpthread.so", "librt.so", + "libutil.so", "libnsl.so", "libresolv.so", "libcrypt.so", + "libstdc++.so", "libgcc_s.so", "linux-vdso.so", "ld-linux", "libld-linux", +}); + +// Windows' own. Deliberately NOT including vcruntime140/msvcp140: those +// belong to the toolset, and whether they travel is `cxx_runtime`'s decision. +constexpr std::array kPeSystem = std::to_array({ + "ntdll.dll", "kernel32.dll", "kernelbase.dll", "user32.dll", "gdi32.dll", + "gdi32full.dll", "advapi32.dll", "shell32.dll", "shlwapi.dll", + "ole32.dll", "oleaut32.dll", "combase.dll", "comdlg32.dll", + "comctl32.dll", "rpcrt4.dll", "sechost.dll", "setupapi.dll", + "cfgmgr32.dll", "version.dll", "winmm.dll", "imm32.dll", "uxtheme.dll", + "dwmapi.dll", "userenv.dll", "psapi.dll", "iphlpapi.dll", "netapi32.dll", + "ws2_32.dll", "wsock32.dll", "mswsock.dll", "crypt32.dll", "bcrypt.dll", + "bcryptprimitives.dll", "ncrypt.dll", "secur32.dll", "wintrust.dll", + "powrprof.dll", "winspool.drv", "msimg32.dll", "opengl32.dll", + "glu32.dll", "dxgi.dll", "d3d9.dll", "d3d11.dll", "d3d12.dll", + "dbghelp.dll", "hid.dll", "avrt.dll", "mfplat.dll", "dnsapi.dll", + "profapi.dll", "ucrtbase.dll", "msvcrt.dll", "win32u.dll", +}); + +} // namespace detail + +Ident identify(const std::filesystem::path& binary) { + Ident id; + auto buf = detail::slurp(binary); + if (!buf || buf->size() < 8) return id; + std::string_view b{*buf}; + + if (b.starts_with(std::string_view("\x7f" "ELF", 4))) { + id.format = Format::Elf; + auto cls = detail::le(b, 4); + id.is64 = cls && *cls == 2; + if (auto m = detail::le(b, 18)) id.arch = detail::elf_arch(*m); + return id; + } + if (b.starts_with("MZ")) { + // MZ alone is a DOS stub; a PE needs the signature e_lfanew points at. + // Saying "PE" for a file that has none would send the caller into a + // parser that cannot succeed. + if (auto lfanew = detail::le(b, 0x3C)) { + if (b.substr(*lfanew, 4) == std::string_view("PE\0\0", 4)) { + id.format = Format::Pe; + if (auto m = detail::le(b, *lfanew + 4)) + id.arch = detail::pe_arch(*m); + if (auto magic = detail::le(b, *lfanew + 24)) + id.is64 = (*magic == 0x20b); + return id; + } + } + return id; + } + // Mach-O, both endiannesses and the fat wrapper. Recognised but not + // parsed: macOS packaging still asks the loader, and a caller that lands + // here deserves a message naming the format rather than "unknown". + for (auto magic : {std::string_view("\xcf\xfa\xed\xfe", 4), + std::string_view("\xce\xfa\xed\xfe", 4), + std::string_view("\xfe\xed\xfa\xcf", 4), + std::string_view("\xfe\xed\xfa\xce", 4), + std::string_view("\xca\xfe\xba\xbe", 4)}) { + if (b.starts_with(magic)) { id.format = Format::MachO; return id; } + } + return id; +} + +std::expected, std::string> +needed_names(const std::filesystem::path& binary) { + auto buf = detail::slurp(binary); + if (!buf) + return std::unexpected(std::format("cannot read '{}'", binary.string())); + std::string_view b{*buf}; + switch (identify(binary).format) { + case Format::Elf: return detail::elf_needed(b); + case Format::Pe: return detail::pe_needed(b); + case Format::MachO: + return std::unexpected( + "Mach-O dependency reading is not implemented; macOS packaging " + "resolves the closure through the loader instead"); + case Format::Unknown: break; + } + return std::unexpected(std::format( + "'{}' is not an ELF, PE or Mach-O object", binary.string())); +} + +bool is_system_lib(Format f, std::string_view name) { + std::string lower(name); + std::ranges::transform(lower, lower.begin(), + [](unsigned char c) { return std::tolower(c); }); + if (f == Format::Pe) { + // API sets: `api-ms-win-crt-runtime-l1-1-0.dll` and friends are + // forwarders resolved by the loader against the OS. They have no file + // to copy on most systems and copying one would be wrong anyway. + if (lower.starts_with("api-ms-") || lower.starts_with("ext-ms-")) + return true; + return std::ranges::find(detail::kPeSystem, lower) + != detail::kPeSystem.end(); + } + if (f == Format::Elf) { + for (auto prefix : detail::kElfSystem) + if (lower.starts_with(prefix)) return true; + return false; + } + return false; +} + +} // namespace mcpp::pack::binfmt diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index d93d5623..f06c7759 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -1,4 +1,19 @@ -// mcpp.pack — bundle a built binary into a self-contained release tarball. +// mcpp.pack — bundle a built binary into a self-contained release archive. +// +// TWO OUTPUT FAMILIES, ONE PIPELINE. An ELF/Mach-O artifact becomes a +// `.tar.gz` whose libraries live in `lib/` and are reached through a rewritten +// RUNPATH; a PE artifact becomes a `.zip` whose DLLs sit BESIDE the .exe, +// because that is where the Win32 loader looks and PE has no rpath to rewrite. +// Same contract, same modes, different mechanism — see +// .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §4. +// +// The PE path runs on ANY host. That is not a portability nicety: the ELF +// path derives its closure by running the artifact under +// LD_TRACE_LOADED_OBJECTS, which is why it can cross neither an OS nor an +// architecture, and the `#if defined(_WIN32)` refusal that used to sit at the +// top of `run()` was that limitation surfacing rather than a missing branch. +// mcpp.pack.binfmt reads the import table instead, so a Linux box packaging a +// Windows build is simply what happens when nothing has to be executed. // // See docs/35-pack-design.md for the full design. Three modes: // Static full musl static, no PT_INTERP / RUNPATH @@ -20,9 +35,12 @@ module; export module mcpp.pack; import std; +import mcpp.build.distribution; import mcpp.build.loader_contract; import mcpp.config; +import mcpp.pack.binfmt; import mcpp.pack.host_requirements; +import mcpp.pack.zip; import mcpp.platform; import mcpp.platform.xlings; import mcpp.manifest; @@ -38,6 +56,27 @@ struct Options { Format format = Format::Tar; std::filesystem::path output; // empty = derive from manifest std::string targetTriple; // empty = host + // Where a dependency NAME may be resolved to a file. + // + // Only used where the closure is read STATICALLY (PE): on ELF the loader + // hands back resolved paths and no search is performed here. Deliberately + // never the target's own system directories — a DLL that resolves only + // there is the target's to provide, and copying one is a broken program + // rather than a heavier one. + std::vector depSearchDirs; + // The TOOLCHAIN's own runtime directory: libstdc++'s for gcc, the MSVC + // toolset's `VC\Redist\MSVC\\\Microsoft.VC*.CRT\` for cl. + // Searched ONLY under the toolchain-coupled contract — see make_plan. + std::vector toolchainRuntimeDirs; + // What the artifact promises about the machine that runs it, as resolved + // by mcpp.build.distribution for the distributable role. + // + // `pack` used to be unable to see this at all (design §4.3): the contract + // reached compile and link FLAGS and stopped there, so the step that + // decides which files actually travel had no idea what had been promised. + // On ELF the `ldd` closure happened to agree with it; on PE nothing did. + mcpp::build::dist::Contract cxxRuntime = + mcpp::build::dist::Contract::SelfContained; }; // Resolved plan — all paths absolute, all decisions baked in. @@ -47,7 +86,7 @@ struct Plan { std::filesystem::path builtBinary; // mcpp build artefact std::string binaryName; // basename(builtBinary) std::filesystem::path stagingRoot; // target/dist// - std::filesystem::path tarballPath; // staging_root parent / .tar.gz + std::filesystem::path archivePath; // …/.tar.gz | .zip std::string packageName; std::string packageVersion; std::string triple; // e.g. "x86_64-linux-musl" @@ -59,6 +98,12 @@ struct Plan { // What the TARGET machine must provide. Derived once, in make_plan, from // the same predicate `mcpp publish` uses — see mcpp.pack.host_requirements. std::vector hostRequirements; + // Is the artifact a PE? Read from the FILE, not inferred from the triple — + // the file is the thing being packaged, and a triple is a request. + bool targetIsPe = false; + // The search set the PE closure resolves names against, after the + // contract has had its say (see make_plan). + std::vector searchDirs; }; struct Error { std::string message; }; @@ -138,29 +183,35 @@ namespace detail { // errors. Naming the namespace gives every helper module linkage and // sidesteps the rule entirely. -// Default tarball name: `--[-].tar.gz`. +// Default archive name: `--[-].`. // Mode suffix only for non-default modes so adjacent builds of different // modes don't stomp each other in target/dist/. -std::string default_tarball_name(std::string_view name, std::string_view version, - std::string_view triple, Mode mode) +// +// The EXTENSION follows the artifact, not the host: a Windows package is a +// `.zip` whoever built it, and a `.tar.gz` full of DLLs is a package most +// Windows users cannot open without installing something first. +std::string default_archive_name(std::string_view name, std::string_view version, + std::string_view triple, Mode mode, bool pe) { + std::string_view ext = pe ? ".zip" : ".tar.gz"; auto sfx = mode_tarball_suffix(mode); if (sfx.empty()) - return std::format("{}-{}-{}.tar.gz", name, version, triple); - return std::format("{}-{}-{}-{}.tar.gz", name, version, triple, sfx); + return std::format("{}-{}-{}{}", name, version, triple, ext); + return std::format("{}-{}-{}-{}{}", name, version, triple, sfx, ext); } -// Strip the `.tar.gz` (or `.tgz`) suffix from a tarball filename to get -// the canonical wrapper-directory name. The result names both the disk -// staging dir and the top-level entry inside the archive — keeping the -// two in lock-step makes click-to-extract behave the way users expect. -std::string wrapper_dirname_from_tarball(const std::filesystem::path& tarball) { - auto name = tarball.filename().string(); - for (auto suffix : {std::string_view{".tar.gz"}, std::string_view{".tgz"}}) { +// Strip the archive suffix to get the canonical wrapper-directory name. The +// result names both the disk staging dir and the top-level entry inside the +// archive — keeping the two in lock-step makes click-to-extract behave the +// way users expect (and on Windows, Explorer's "extract here" too). +std::string wrapper_dirname_from_archive(const std::filesystem::path& archive) { + auto name = archive.filename().string(); + for (auto suffix : {std::string_view{".tar.gz"}, std::string_view{".tgz"}, + std::string_view{".zip"}}) { if (name.ends_with(suffix)) return name.substr(0, name.size() - suffix.size()); } // No recognised compression suffix — fall back to the bare stem. - return tarball.stem().string(); + return archive.stem().string(); } } // namespace detail @@ -228,23 +279,75 @@ make_plan(const mcpp::manifest::Manifest& manifest, mode_cli_name(opts.mode), names)}); } + // WHAT THE FILE IS, not what the triple asked for. `--target` states an + // intention; the artifact on disk is the thing being packaged, and when + // the two disagree it is the file that has to be believed. (They can + // disagree for real: `[pack] default_mode = "static"` re-prepares the + // build with a different target after the first one has already run.) + p.targetIsPe = + mcpp::pack::binfmt::identify(builtBinary).format + == mcpp::pack::binfmt::Format::Pe; + + // THE CONTRACT REACHES PACKAGING. Until now it stopped at the compile and + // link flags, so the step that decides which files travel could not see + // what the artifact had promised (design §4.3). + // + // toolchain-coupled the toolchain's own runtime travels WITH the + // artifact — so its directory joins the search set. + // host-coupled the target provides it — so that directory stays + // OUT, and a vcruntime140.dll sitting in the + // toolset is not silently swept into the package. + // self-contained there is nothing to carry. + // + // A mode that bundles NOTHING cannot honour a contract that requires + // files to travel. `pack` already refuses the mirror-image contradiction + // (a bundle carrying its own libc cannot consume a host capability), and + // this grows from the same root: a contract with no executor is a promise + // the build prints and the package quietly drops. + using Contract = mcpp::build::dist::Contract; + const bool modeBundlesNothing = + opts.mode == Mode::None || opts.mode == Mode::Static; + if (opts.cxxRuntime == Contract::ToolchainCoupled && modeBundlesNothing) { + return std::unexpected(Error{std::format( + "cxx_runtime = \"toolchain-coupled\" and --mode {} contradict each " + "other.\n" + " The contract says the toolchain's C++ runtime travels WITH this " + "artifact; --mode {}\n" + " bundles nothing, so it would ship a program that cannot start " + "anywhere the toolchain\n" + " is not already installed.\n" + " use: --mode vendored (carry it), or cxx_runtime = " + "\"host-coupled\" / \"self-contained\"\n" + " in [build] if the target is expected to provide the runtime " + "itself.", + mode_cli_name(opts.mode), mode_cli_name(opts.mode))}); + } + + // The artifact's own directory is always searched: whatever the build + // staged beside it (the toolchain-coupled CRT, a dependency's DLL) is by + // definition part of what it runs with. + p.searchDirs.push_back(builtBinary.parent_path()); + for (auto const& d : opts.depSearchDirs) p.searchDirs.push_back(d); + if (opts.cxxRuntime == Contract::ToolchainCoupled) + for (auto const& d : opts.toolchainRuntimeDirs) p.searchDirs.push_back(d); + auto distDir = projectRoot / "target" / "dist"; if (opts.output.empty()) { - p.tarballPath = distDir / detail::default_tarball_name( - p.packageName, p.packageVersion, p.triple, opts.mode); + p.archivePath = distDir / detail::default_archive_name( + p.packageName, p.packageVersion, p.triple, opts.mode, p.targetIsPe); } else if (!opts.output.has_parent_path()) { // `-o name.tar.gz` (bare filename) → place in target/dist/. // `-o ./name.tar.gz` or `-o sub/name.tar.gz` → use as-is. // `-o /abs/path.tar.gz` → use as-is. - p.tarballPath = distDir / opts.output; + p.archivePath = distDir / opts.output; } else { - p.tarballPath = opts.output; + p.archivePath = opts.output; } - // Derive the staging dir from the tarball stem so the in-archive + // Derive the staging dir from the archive stem so the in-archive // wrapper directory and the on-disk staging dir share one name — // matches what GUI extractors create on click and what `tar -xzf` // produces on the CLI. - p.stagingRoot = distDir / detail::wrapper_dirname_from_tarball(p.tarballPath); + p.stagingRoot = distDir / detail::wrapper_dirname_from_archive(p.archivePath); p.includeGlobs = manifest.packConfig.include; p.excludeGlobs = manifest.packConfig.exclude; @@ -608,6 +711,63 @@ void copy_if_exists(const std::filesystem::path& src, std::filesystem::copy_options::overwrite_existing, ec); } +// ─── PE: the closure, read rather than executed ───────────────────────── +// +// BFS over the import tables, resolving each name against `searchDirs`. A +// name that resolves NOWHERE is deliberately not an error: on a Linux host +// `kernel32.dll` has no file to find, and on a Windows host it would resolve +// only in the system directory, which is not searched. Both are the same +// answer — the target provides it — and both are correct. +std::vector +pe_closure(const std::filesystem::path& binary, + std::span searchDirs, + const std::vector& forceBundle) +{ + namespace bf = mcpp::pack::binfmt; + std::vector out; + std::set seen; // lowercased: PE names are not + std::vector queue{binary}; + + auto lower = [](std::string_view s) { + std::string l(s); + std::ranges::transform(l, l.begin(), + [](unsigned char c) { return std::tolower(c); }); + return l; + }; + + while (!queue.empty()) { + auto current = queue.back(); + queue.pop_back(); + auto names = bf::needed_names(current); + if (!names) continue; // unreadable: nothing to add + for (auto const& name : *names) { + auto key = lower(name); + if (!seen.insert(key).second) continue; + // `[pack] force_bundle` is the escape hatch, and it has to reach + // the SYSTEM list too — on ELF it always did. Shipping a Windows + // component is normally a broken program rather than a heavier + // one, so this is a decision a human has to make explicitly; when + // they have, mcpp does not know better than them. + if (bf::is_system_lib(bf::Format::Pe, name) + && !soname_matches(name, forceBundle)) + continue; + std::error_code ec; + for (auto const& dir : searchDirs) { + auto cand = dir / name; + if (!std::filesystem::is_regular_file(cand, ec)) continue; + out.push_back({name, cand}); + // Transitive: a bundled DLL brings its own imports, and a + // closure that stops at depth one ships a package that starts + // failing one link further in. + queue.push_back(cand); + break; + } + } + } + std::ranges::sort(out, {}, &ResolvedDep::soname); + return out; +} + std::expected make_tarball(const std::filesystem::path& stagingRoot, const std::filesystem::path& tarballPath) @@ -631,30 +791,121 @@ make_tarball(const std::filesystem::path& stagingRoot, } // namespace detail +namespace detail { + +// The PE half of `run`. Flat layout, deliberately: the Win32 loader resolves +// a DLL from the directory of the executable, so `bin/` + `lib/` would need a +// mechanism PE does not have. Extract-and-double-click is also what a Windows +// user expects, and it is what the earlier design already specified +// (.agents/docs/2026-05-19-pack-windows-design.md). +// +// No patchelf step and no wrapper script: "put the DLLs next to the exe" IS +// the relocation rule on this format, which is why the row for it in the +// design's layering table reads "no operation". +std::expected +run_pe(const Plan& plan) +{ + std::error_code ec; + std::filesystem::remove_all(plan.stagingRoot, ec); + std::filesystem::create_directories(plan.stagingRoot, ec); + if (ec) return std::unexpected(Error{std::format( + "cannot create staging '{}': {}", plan.stagingRoot.string(), ec.message())}); + + auto stagedExe = plan.stagingRoot / plan.binaryName; + std::filesystem::copy_file(plan.builtBinary, stagedExe, + std::filesystem::copy_options::overwrite_existing, ec); + if (ec) return std::unexpected(Error{std::format( + "copy binary failed: {}", ec.message())}); + + copy_if_exists(plan.projectRoot / "README.md", plan.stagingRoot); + copy_if_exists(plan.projectRoot / "LICENSE", plan.stagingRoot); + if (!plan.hostRequirements.empty()) { + std::ofstream out(plan.stagingRoot / std::filesystem::path(kFileName)); + if (!out) return std::unexpected(Error{std::format( + "cannot write {} into the bundle", kFileName)}); + out << render(plan.hostRequirements); + } + + std::vector deps; + if (plan.opts.mode != Mode::None && plan.opts.mode != Mode::Static) { + // `vendored` and `self-contained` collect the same set here, and that + // is a property of the PLATFORM rather than a simplification. + // `self-contained` on ELF means "ship the loader too"; on PE there is + // no loader to ship and kernel32.dll and friends may not be + // redistributed — a process that loaded a private copy of one would + // have two of something that must be unique. So the ceiling on + // "everything" is the same for both modes: every dependency mcpp is + // ALLOWED to carry. + deps = pe_closure(stagedExe, plan.searchDirs, plan.forceBundleLibs); + for (auto const& d : deps) { + if (soname_matches(d.soname, plan.alsoSkipLibs) + && !soname_matches(d.soname, plan.forceBundleLibs)) + continue; + auto dst = plan.stagingRoot / d.soname; + // The build may already have staged it beside the artifact, in + // which case source and destination are the same file. + std::error_code cec; + if (std::filesystem::equivalent(d.path, dst, cec)) continue; + std::filesystem::copy_file(d.path, dst, + std::filesystem::copy_options::overwrite_existing, cec); + if (cec) return std::unexpected(Error{std::format( + "failed to copy {} → {}: {}", + d.path.string(), dst.string(), cec.message())}); + } + } + + if (plan.opts.format != Format::Tar) return {}; + + std::vector entries; + const auto wrapper = plan.stagingRoot.filename().string(); + for (auto const& e : + std::filesystem::recursive_directory_iterator(plan.stagingRoot, ec)) { + if (!e.is_regular_file(ec)) continue; + auto rel = std::filesystem::relative(e.path(), plan.stagingRoot, ec); + if (ec) continue; + entries.push_back({ + wrapper + "/" + rel.generic_string(), + e.path(), + e.path().filename() == plan.builtBinary.filename(), + }); + } + // Deterministic order: a directory iteration order that leaks into an + // archive is how two identical builds get two different checksums. + std::ranges::sort(entries, {}, &mcpp::pack::zip::Entry::name); + if (auto r = mcpp::pack::zip::write(plan.archivePath, entries); !r) + return std::unexpected(Error{r.error()}); + return {}; +} + +} // namespace detail + std::expected run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) { + // A PE package is produced the same way on every host, because nothing in + // that path executes the artifact. This is the branch the `#if + // defined(_WIN32)` refusal used to occupy — and it was never really about + // the host: `LD_TRACE_LOADED_OBJECTS` cannot trace a PE from Linux either. + if (plan.targetIsPe) return detail::run_pe(plan); + #if defined(_WIN32) - // `mcpp pack` is not yet supported on Windows. - // - // The current implementation relies on POSIX-only tools: - // - LD_TRACE_LOADED_OBJECTS=1 (ELF dynamic linker trick; no equivalent - // on Windows PE/COFF) - // - ldd / patchelf (Linux ELF tools; not available on Windows) - // - tar -czf (GNU tar; not universally present on Windows) - // - // For CI-produced Windows zip packages, use the ci-windows.yml workflow - // which zips the MSVC/Clang build output directly. - // - // Windows PE packaging (DLL collection + zip) is planned. - // See .agents/docs/2026-05-19-pack-windows-design.md for the design. - (void)plan; + // A NON-PE artifact on a Windows host: a cross build to Linux or macOS. + // The closure below asks the dynamic linker by running the binary, which + // this machine cannot do — so say that, rather than reporting a platform + // limitation that no longer exists for the case a Windows user is + // actually likely to hit. (void)cfg; - return std::unexpected(Error{ - "error: `mcpp pack` is not yet supported on Windows.\n" - " Use the CI workflow (ci-windows.yml) to produce Windows zip packages.\n" - " Windows PE packaging (DLL collection + zip) is planned." - }); + return std::unexpected(Error{std::format( + "cannot package a {} artifact from a Windows host.\n" + " The dependency closure for that format is resolved by running " + "the artifact under\n" + " the target's own dynamic linker, which this machine has no way " + "to do. Package it\n" + " on the target OS, or build for Windows (`--target " + "x86_64-pc-windows-msvc`), which\n" + " needs no such step.", + std::string(mcpp::pack::binfmt::format_name( + mcpp::pack::binfmt::identify(plan.builtBinary).format)))}); #else using namespace detail; std::error_code ec; @@ -829,7 +1080,7 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // 5. Output. if (plan.opts.format == Format::Tar) { - if (auto r = make_tarball(plan.stagingRoot, plan.tarballPath); !r) + if (auto r = make_tarball(plan.stagingRoot, plan.archivePath); !r) return r; } return {}; diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 5596cc2f..d5f1306e 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -11,6 +11,8 @@ export module mcpp.pack.pipeline; import std; import mcpp.build.prepare; import mcpp.build.backend; +import mcpp.build.distribution; +import mcpp.build.flags; import mcpp.build.ninja; import mcpp.build.plan; import mcpp.config; @@ -100,6 +102,29 @@ export int build_and_pack(Options opts, bool modeFromUser) { mcpp::fetcher::make_bootstrap_progress_callback()); if (!cfg) { mcpp::ui::error(cfg.error().message); return 4; } + // ─── What the build promised, and where its runtime lives ──────── + // + // The C++ runtime contract has been resolved since the flags were + // computed; `pack` simply had no way to see it (design §4.3), so on PE + // nothing enforced it and on ELF the `ldd` closure agreed with it by + // luck. Reading the RESOLVED value rather than the manifest string is the + // point: a request that was downgraded (a per-role self-contained on + // /MD, say) must not make the package behave as though it had been + // honoured. + { + const auto flags = mcpp::build::compute_flags(ctx->plan); + opts.cxxRuntime = flags.contractByRole[ + static_cast(mcpp::build::dist::Role::Distributable)]; + opts.toolchainRuntimeDirs = ctx->plan.toolchain.linkRuntimeDirs; + // Where a third-party dependency's shared library may be found. Both + // channels, because they answer for different things: the runtime + // library dirs are what `mcpp run` puts on the loader's path, and the + // link intent's search dirs are what a dependency package declared. + opts.depSearchDirs = ctx->plan.runtimeLibraryDirs; + for (auto const& d : ctx->plan.linkIntent.runtimeSearchDirs) + opts.depSearchDirs.push_back(d); + } + // ─── Build the plan + run ──────────────────────────────────────── auto plan = mcpp::pack::make_plan(ctx->manifest, *cfg, opts, mainBinary, ctx->projectRoot, ctx->tc.targetTriple, @@ -121,7 +146,7 @@ export int build_and_pack(Options opts, bool modeFromUser) { auto pathCtx = mcpp::fetcher::make_path_ctx(&*cfg, ctx->projectRoot); auto outPath = (opts.format == mcpp::pack::Format::Tar) - ? plan->tarballPath : plan->stagingRoot; + ? plan->archivePath : plan->stagingRoot; mcpp::ui::status("Packed", mcpp::ui::shorten_path(outPath, pathCtx)); return 0; } diff --git a/src/pack/zip.cppm b/src/pack/zip.cppm new file mode 100644 index 00000000..ac9047fe --- /dev/null +++ b/src/pack/zip.cppm @@ -0,0 +1,213 @@ +// mcpp.pack.zip — write a .zip, from any host, with no external tool. +// +// WHY NOT SHELL OUT. The rest of `pack` produces `.tar.gz` by running `tar`, +// and the obvious symmetry would be to run something for zip too. There is no +// such something that exists everywhere: GNU tar (Linux) cannot write zip, +// `zip(1)` is frequently absent, PowerShell's `Compress-Archive` is +// Windows-only, and bsdtar is Windows-and-macOS-mostly. The earlier design +// (.agents/docs/2026-05-19-pack-windows-design.md) proposed `Compress-Archive` +// precisely because it assumed pack ran ON Windows — the assumption that +// mcpp.pack.binfmt exists to remove. A packer that only works on the target's +// own OS is the thing being fixed; picking an archiver that only exists there +// would reintroduce it one layer down. +// +// WHY STORED, NOT DEFLATED. Entries are written with method 0 (stored). That +// is a real cost — a release zip is roughly the size of its contents — and it +// is the honest trade for now: a DEFLATE encoder is the only part of this +// file that could produce an archive which UNPACKS WRONG rather than failing +// loudly, and mcpp has no zlib to borrow one from. Adding compression later +// changes only `write()`; the archive layout below does not move. +// +// Deterministic by construction: no timestamps are read (every entry gets a +// fixed DOS date), and entries are written in the order given. Two packs of +// the same tree produce byte-identical archives, which is what makes a +// published checksum mean anything. + +export module mcpp.pack.zip; + +import std; + +export namespace mcpp::pack::zip { + +struct Entry { + // Path INSIDE the archive, always with forward slashes — the ZIP spec + // says so (4.4.17.1), and a backslash here is what makes an archive + // written on Windows extract into one flat file with a strange name + // everywhere else. + std::string name; + std::filesystem::path source; + // Only meaningful to POSIX extractors; Windows ignores it. Set for the + // executable so `unzip` on a Linux box does not hand back a non-runnable + // file — which matters because a Windows artifact is routinely inspected + // (and cross-tested under wine) from Linux. + bool executable = false; +}; + +// Write `entries` to `out`, creating parent directories as needed. +std::expected +write(const std::filesystem::path& out, std::span entries); + +// CRC-32 (IEEE), exposed for tests: an archive whose CRCs are wrong extracts +// with a warning on some tools and silently on others, so this is worth +// pinning against known vectors rather than against itself. +std::uint32_t crc32(std::string_view data); + +} // namespace mcpp::pack::zip + +namespace mcpp::pack::zip { + +namespace detail { + +const std::array& crc_table() { + static const auto table = [] { + std::array t{}; + for (std::uint32_t i = 0; i < 256; ++i) { + std::uint32_t c = i; + for (int k = 0; k < 8; ++k) + c = (c & 1) ? (0xEDB88320u ^ (c >> 1)) : (c >> 1); + t[i] = c; + } + return t; + }(); + return table; +} + +void put16(std::string& out, std::uint16_t v) { + out.push_back(static_cast(v & 0xFF)); + out.push_back(static_cast((v >> 8) & 0xFF)); +} + +void put32(std::string& out, std::uint32_t v) { + for (int i = 0; i < 4; ++i) + out.push_back(static_cast((v >> (8 * i)) & 0xFF)); +} + +// A fixed DOS timestamp: 1980-01-01 00:00:00, the earliest the format can +// express. Reading the file's mtime instead would make the archive depend on +// when it was built, which is the usual reason two identical builds produce +// two different checksums. +constexpr std::uint16_t kDosTime = 0; +constexpr std::uint16_t kDosDate = 0x0021; // (1980-1980)<<9 | 1<<5 | 1 + +// External attributes. The high 16 bits are the Unix mode when the +// version-made-by byte says Unix (3); the low byte is the DOS attribute set. +std::uint32_t external_attrs(bool executable) { + std::uint32_t mode = executable ? 0100755u : 0100644u; + return mode << 16; +} + +std::optional slurp(const std::filesystem::path& p) { + std::ifstream is(p, std::ios::binary); + if (!is) return std::nullopt; + std::ostringstream ss; + ss << is.rdbuf(); + return ss.str(); +} + +} // namespace detail + +std::uint32_t crc32(std::string_view data) { + auto const& table = detail::crc_table(); + std::uint32_t c = 0xFFFFFFFFu; + for (unsigned char b : data) + c = table[(c ^ b) & 0xFF] ^ (c >> 8); + return c ^ 0xFFFFFFFFu; +} + +std::expected +write(const std::filesystem::path& out, std::span entries) { + std::error_code ec; + if (out.has_parent_path()) + std::filesystem::create_directories(out.parent_path(), ec); + + struct Central { + std::string name; + std::uint32_t crc = 0, size = 0, offset = 0; + bool executable = false; + }; + std::vector central; + central.reserve(entries.size()); + + std::string blob; + for (auto const& e : entries) { + auto data = detail::slurp(e.source); + if (!data) + return std::unexpected(std::format( + "cannot read '{}' for the archive", e.source.string())); + // 4 GiB is the point where ZIP64 becomes mandatory. mcpp does not + // write ZIP64, so refuse rather than emit an archive whose sizes have + // silently wrapped — a truncated field here produces an archive that + // extracts to garbage instead of failing. + if (data->size() > 0xFFFFFFFFull || blob.size() > 0xFFFFFFFFull) + return std::unexpected(std::format( + "'{}' exceeds the 4 GiB limit of a non-ZIP64 archive", + e.source.string())); + + Central c; + c.name = e.name; + c.crc = crc32(*data); + c.size = static_cast(data->size()); + c.offset = static_cast(blob.size()); + c.executable = e.executable; + + detail::put32(blob, 0x04034b50); // local file header + detail::put16(blob, 20); // version needed (2.0) + detail::put16(blob, 0); // flags — no data descriptor + detail::put16(blob, 0); // method 0: stored + detail::put16(blob, detail::kDosTime); + detail::put16(blob, detail::kDosDate); + detail::put32(blob, c.crc); + detail::put32(blob, c.size); // compressed == uncompressed + detail::put32(blob, c.size); + detail::put16(blob, static_cast(c.name.size())); + detail::put16(blob, 0); // extra field length + blob += c.name; + blob += *data; + + central.push_back(std::move(c)); + } + + const auto centralAt = static_cast(blob.size()); + for (auto const& c : central) { + detail::put32(blob, 0x02014b50); // central directory header + // version made by: 3 (Unix) << 8 | 20. The Unix half is what makes the + // mode bits in `external attributes` meaningful to an extractor. + detail::put16(blob, (3 << 8) | 20); + detail::put16(blob, 20); // version needed + detail::put16(blob, 0); + detail::put16(blob, 0); // stored + detail::put16(blob, detail::kDosTime); + detail::put16(blob, detail::kDosDate); + detail::put32(blob, c.crc); + detail::put32(blob, c.size); + detail::put32(blob, c.size); + detail::put16(blob, static_cast(c.name.size())); + detail::put16(blob, 0); // extra + detail::put16(blob, 0); // comment + detail::put16(blob, 0); // disk number + detail::put16(blob, 0); // internal attrs + detail::put32(blob, detail::external_attrs(c.executable)); + detail::put32(blob, c.offset); + blob += c.name; + } + const auto centralSize = static_cast(blob.size()) - centralAt; + + detail::put32(blob, 0x06054b50); // end of central directory + detail::put16(blob, 0); + detail::put16(blob, 0); + detail::put16(blob, static_cast(central.size())); + detail::put16(blob, static_cast(central.size())); + detail::put32(blob, centralSize); + detail::put32(blob, centralAt); + detail::put16(blob, 0); // comment length + + std::ofstream os(out, std::ios::binary | std::ios::trunc); + if (!os) return std::unexpected(std::format( + "cannot write '{}'", out.string())); + os.write(blob.data(), static_cast(blob.size())); + if (!os) return std::unexpected(std::format( + "failed writing '{}'", out.string())); + return {}; +} + +} // namespace mcpp::pack::zip diff --git a/tests/e2e/240_pack_pe_zip_cross.sh b/tests/e2e/240_pack_pe_zip_cross.sh new file mode 100755 index 00000000..17684c96 --- /dev/null +++ b/tests/e2e/240_pack_pe_zip_cross.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# requires: mingw-cross python3 +# 240_pack_pe_zip_cross.sh — packaging a Windows program FROM LINUX. +# +# This is the acceptance criterion for §4 of +# .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md, and it is +# written so that running it on Windows would prove nothing: +# +# "在 Linux 上为 Windows 产物打出 zip,内含正确 DLL 闭包 +# — 跨 OS 是这条的全部意义;同 OS 打包证明不了" +# +# `mcpp pack` used to refuse Windows with `#if defined(_WIN32)`, and the +# reason given was that the tools were POSIX-only. The real reason was one +# layer down: the dependency closure came from +# +# LD_TRACE_LOADED_OBJECTS=1 '' +# +# which RUNS the artifact — so it could cross neither an OS nor an +# architecture, and no amount of porting `tar` would have helped. Reading the +# import table instead (mcpp.pack.binfmt) removes both limits at once, and +# this test is what says that actually happened rather than that the guard was +# deleted. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +"$MCPP" new winpack > /dev/null +cd winpack +cat > src/main.cpp <<'EOF' +#include +int main() { std::printf("winpack ok\n"); return 0; } +EOF +rm -f src/winpack.cppm + +cat > mcpp.toml <<'EOF' +[package] +name = "winpack" +version = "0.1.0" + +# `force_bundle` reaching a SYSTEM name is what makes the next assertion +# positive rather than vacuous — see the comment at the msvcrt.dll check. +[pack.bundle-project] +force_bundle = ["msvcrt.dll"] +EOF + +"$MCPP" build --target x86_64-windows-gnu > build.log 2>&1 || { cat build.log; exit 1; } +EXE="$(find target/x86_64-windows-gnu -name 'winpack.exe' | head -1)" +[[ -n "$EXE" ]] || { echo "FAIL: no PE was produced"; exit 1; } + +# The artifact really is a PE, checked here rather than assumed from the +# triple: everything below is about reading THAT file. +python3 - "$EXE" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +assert d[:2] == b'MZ', "not an MZ image" +lfanew, = struct.unpack_from(' "$BINDIR/msvcrt.dll" + +"$MCPP" pack --target x86_64-windows-gnu > pack.log 2>&1 || { cat pack.log; exit 1; } + +ZIP="$(find target/dist -name '*.zip' | head -1)" +[[ -n "$ZIP" ]] || { + echo "FAIL: no .zip produced — a Windows package is a zip whoever built it" + ls -R target/dist || true + cat pack.log + exit 1 +} +# And NOT a tarball: the extension follows the artifact, not the host. +[[ -z "$(find target/dist -name '*.tar.gz' | head -1)" ]] || { + echo "FAIL: produced a .tar.gz for a Windows target"; exit 1; } + +python3 - "$ZIP" <<'PY' +import sys, zipfile +z = zipfile.ZipFile(sys.argv[1]) + +# An INDEPENDENT reader. mcpp writes this archive itself (no host has a zip +# tool that exists everywhere), so "our writer agrees with our reader" would +# be worth nothing. +bad = z.testzip() +assert bad is None, f"corrupt entry: {bad}" + +names = z.namelist() +tops = {n.split('/')[0] for n in names} +assert len(tops) == 1, f"archive has no single wrapper directory: {tops}" +wrapper = tops.pop() + +exe = f"{wrapper}/winpack.exe" +assert exe in names, f"the executable is missing: {names}" + +# FLAT, beside the .exe. On PE that is not a layout preference — it is the +# relocation mechanism: the Win32 loader resolves a DLL from the directory of +# the executable, and there is no rpath to point anywhere else. +dll = f"{wrapper}/msvcrt.dll" +assert dll in names, ( + "msvcrt.dll was not bundled. It is named in the EXE's import table and in " + f"[pack] force_bundle, so this is the closure reader failing: {names}") +assert z.read(dll) == b'MZ-not-a-real-dll', "bundled the wrong file" + +# The other half, which the assertion above cannot give: the closure must +# EXCLUDE Windows' own. A parser that read nothing at all would also produce +# an archive with no kernel32.dll in it — but it could not have produced the +# msvcrt.dll above, so the two together are decisive. +lower = [n.lower() for n in names] +for sysdll in ("kernel32.dll", "ntdll.dll", "ucrtbase.dll"): + assert not any(n.endswith(sysdll) for n in lower), ( + f"{sysdll} was bundled — shipping a private copy of a Windows " + "component is a broken program, not a heavier one") + +# The .exe keeps its executable bit, which only matters because a Windows +# package is routinely unpacked and inspected (and wine-tested) from Linux. +info = z.getinfo(exe) +assert (info.external_attr >> 16) & 0o111, "the executable bit was lost" +print("OK: zip verified by an independent reader") +PY + +# `--mode system` promises the target provides everything. A DLL beside the +# .exe would contradict that, and the modes have to mean the same thing on +# both formats or the flag is decoration. +rm -rf target/dist +"$MCPP" pack --target x86_64-windows-gnu --mode system > pack2.log 2>&1 || { + cat pack2.log; exit 1; } +python3 - "$(find target/dist -name '*.zip' | head -1)" <<'PY' +import sys, zipfile +names = zipfile.ZipFile(sys.argv[1]).namelist() +assert not any(n.lower().endswith('.dll') for n in names), \ + f"--mode system bundled a DLL: {names}" +PY + +# The contract reaches packaging (design §4.3). `toolchain-coupled` says the +# toolchain's runtime travels WITH the artifact; a mode that bundles nothing +# cannot deliver that, and a contract with no executor is a promise the build +# prints and the package quietly drops. +cat >> mcpp.toml <<'EOF' + +[build] +cxx_runtime = "toolchain-coupled" +EOF +rm -rf target/dist +if "$MCPP" pack --target x86_64-windows-gnu --mode system > pack3.log 2>&1; then + echo "FAIL: cxx_runtime = toolchain-coupled + --mode system was accepted" + cat pack3.log + exit 1 +fi +grep -q 'toolchain-coupled' pack3.log || { + echo "FAIL: the refusal does not name the contract"; cat pack3.log; exit 1; } +grep -q 'vendored' pack3.log || { + echo "FAIL: the refusal offers no way out"; cat pack3.log; exit 1; } + +echo "OK" diff --git a/tests/unit/test_pack_binfmt.cpp b/tests/unit/test_pack_binfmt.cpp new file mode 100644 index 00000000..3a8986a1 --- /dev/null +++ b/tests/unit/test_pack_binfmt.cpp @@ -0,0 +1,365 @@ +#include + +import std; +import mcpp.pack.binfmt; +import mcpp.pack.zip; + +namespace bf = mcpp::pack::binfmt; + +// WHY THE FIXTURES ARE SYNTHESIZED RATHER THAN CHECKED IN. +// +// This module's whole reason to exist is that a dependency closure must be +// derivable WITHOUT running the binary — so that a Linux runner can package a +// Windows build. A test that needed a real PE would need a Windows toolchain +// to produce one, which is the same dependency one layer up: the test would +// only run where the feature was already unnecessary. +// +// Hand-built headers also make the assertions exact. "Found two DLLs" is a +// much weaker statement than "found the two names I wrote at these RVAs, and +// not the terminator". + +namespace { + +void put(std::string& b, std::size_t at, std::uint64_t v, std::size_t width) { + if (b.size() < at + width) b.resize(at + width, '\0'); + for (std::size_t i = 0; i < width; ++i) + b[at + i] = static_cast((v >> (8 * i)) & 0xFF); +} + +std::filesystem::path write_temp(std::string_view tag, std::string_view bytes) { + auto p = std::filesystem::temp_directory_path() + / std::format("mcpp-binfmt-{}-{}", tag, + std::chrono::steady_clock::now().time_since_epoch().count()); + std::ofstream os(p, std::ios::binary); + os.write(bytes.data(), static_cast(bytes.size())); + return p; +} + +// ─── a minimal ELF64 shared object ────────────────────────────────────── +// +// One PT_LOAD covering the whole file at vaddr 0 (so vaddr == file offset, +// which keeps the fixture readable without making the parser's translation +// step a no-op — DT_STRTAB is still resolved THROUGH the segment table), plus +// a PT_DYNAMIC carrying two DT_NEEDED and a DT_STRTAB. +std::string elf_with_needed(std::span needed) { + std::string b; + b.resize(64, '\0'); + b[0] = 0x7f; b[1] = 'E'; b[2] = 'L'; b[3] = 'F'; + b[4] = 2; // ELFCLASS64 + b[5] = 1; // ELFDATA2LSB + b[6] = 1; // EV_CURRENT + put(b, 16, 3, 2); // e_type = ET_DYN + put(b, 18, 0x3E, 2); // e_machine = EM_X86_64 + put(b, 20, 1, 4); // e_version + put(b, 32, 64, 8); // e_phoff + put(b, 52, 64, 2); // e_ehsize + put(b, 54, 56, 2); // e_phentsize + put(b, 56, 2, 2); // e_phnum + + const std::size_t dynAt = 64 + 2 * 56; + const std::size_t dynEntries = needed.size() + 2; // + DT_STRTAB + DT_NULL + const std::size_t dynSize = dynEntries * 16; + const std::size_t strAt = dynAt + dynSize; + + std::string strtab; + strtab.push_back('\0'); // index 0 is empty + std::vector offsets; + for (auto n : needed) { + offsets.push_back(strtab.size()); + strtab.append(n); + strtab.push_back('\0'); + } + const std::size_t total = strAt + strtab.size(); + + // PT_LOAD: the whole file, vaddr 0. + put(b, 64 + 0, 1, 4); // p_type + put(b, 64 + 4, 5, 4); // p_flags + put(b, 64 + 8, 0, 8); // p_offset + put(b, 64 + 16, 0, 8); // p_vaddr + put(b, 64 + 32, total, 8); // p_filesz + put(b, 64 + 40, total, 8); // p_memsz + + // PT_DYNAMIC + put(b, 120 + 0, 2, 4); + put(b, 120 + 8, dynAt, 8); + put(b, 120 + 16, dynAt, 8); + put(b, 120 + 32, dynSize, 8); + + std::size_t at = dynAt; + for (auto off : offsets) { + put(b, at, 1, 8); // DT_NEEDED + put(b, at + 8, off, 8); + at += 16; + } + put(b, at, 5, 8); // DT_STRTAB (a VIRTUAL ADDRESS) + put(b, at + 8, strAt, 8); + at += 16; + put(b, at, 0, 8); // DT_NULL + put(b, at + 8, 0, 8); + + b.resize(total, '\0'); + std::copy(strtab.begin(), strtab.end(), b.begin() + static_cast(strAt)); + return b; +} + +// ─── a minimal PE32+ image ────────────────────────────────────────────── +// +// One section mapping RVA 0x1000 → file 0x200, an import directory and a +// delay-import directory inside it. +std::string pe_with_imports(std::span imports, + std::span delayImports) { + constexpr std::size_t kNt = 0x40; + constexpr std::size_t kOptSize = 0xF0; // 112 + 16 directories * 8 + constexpr std::size_t kSecAt = kNt + 24 + kOptSize; + constexpr std::size_t kRawAt = 0x400; + constexpr std::uint32_t kSecVa = 0x1000; + + std::string b; + b.resize(kRawAt, '\0'); + b[0] = 'M'; b[1] = 'Z'; + put(b, 0x3C, kNt, 4); + b[kNt] = 'P'; b[kNt + 1] = 'E'; // "PE\0\0" + put(b, kNt + 4, 0x8664, 2); // Machine = AMD64 + put(b, kNt + 6, 1, 2); // NumberOfSections + put(b, kNt + 20, kOptSize, 2); // SizeOfOptionalHeader + put(b, kNt + 24, 0x20b, 2); // PE32+ + const std::size_t dirsAt = kNt + 24 + 112; + put(b, dirsAt - 4, 16, 4); // NumberOfRvaAndSizes + + // Section header: name, VirtualSize, VirtualAddress, SizeOfRawData, + // PointerToRawData. + const std::string_view secName = ".rdata"; + for (std::size_t i = 0; i < secName.size(); ++i) b[kSecAt + i] = secName[i]; + put(b, kSecAt + 8, 0x1000, 4); + put(b, kSecAt + 12, kSecVa, 4); + put(b, kSecAt + 16, 0x1000, 4); + put(b, kSecAt + 20, kRawAt, 4); + + auto rva_of = [&](std::size_t fileOff) { + return static_cast(kSecVa + (fileOff - kRawAt)); + }; + + // Names first, so the descriptors can point at them. + std::vector importNameRvas, delayNameRvas; + std::size_t cursor = kRawAt; + auto emit_name = [&](std::string_view n) { + auto rva = rva_of(cursor); + for (char c : n) put(b, cursor++, static_cast(c), 1); + put(b, cursor++, 0, 1); + return rva; + }; + for (auto n : imports) importNameRvas.push_back(emit_name(n)); + for (auto n : delayImports) delayNameRvas.push_back(emit_name(n)); + + // Import descriptors (20 bytes each) + an all-zero terminator. + cursor = (cursor + 15) & ~std::size_t{15}; + const std::size_t importAt = cursor; + for (auto rva : importNameRvas) { + put(b, cursor + 0, 0x9000, 4); // OriginalFirstThunk (nonzero) + put(b, cursor + 12, rva, 4); // Name + put(b, cursor + 16, 0x9100, 4); // FirstThunk (nonzero) + cursor += 20; + } + for (int i = 0; i < 20; ++i) put(b, cursor++, 0, 1); + + // Delay-import descriptors (32 bytes each). grAttrs bit 0 = the fields + // are RVAs; without it a descriptor is the pre-VC7 address form and must + // be skipped rather than misread. + cursor = (cursor + 15) & ~std::size_t{15}; + const std::size_t delayAt = cursor; + for (auto rva : delayNameRvas) { + put(b, cursor + 0, 1, 4); // grAttrs = dlattrRva + put(b, cursor + 4, rva, 4); // rvaDLLName + cursor += 32; + } + for (int i = 0; i < 32; ++i) put(b, cursor++, 0, 1); + + if (!imports.empty()) { + put(b, dirsAt + 1 * 8, rva_of(importAt), 4); + put(b, dirsAt + 1 * 8 + 4, 20 * (imports.size() + 1), 4); + } + if (!delayImports.empty()) { + put(b, dirsAt + 13 * 8, rva_of(delayAt), 4); + put(b, dirsAt + 13 * 8 + 4, 32 * (delayImports.size() + 1), 4); + } + return b; +} + +struct TempFile { + std::filesystem::path path; + TempFile(std::string_view tag, std::string_view bytes) + : path(write_temp(tag, bytes)) {} + ~TempFile() { std::error_code ec; std::filesystem::remove(path, ec); } + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; +}; + +} // namespace + +TEST(PackBinfmt, IdentifiesElfWithoutRunningIt) { + std::array needed{"libc.so.6"}; + TempFile f{"elf", elf_with_needed(needed)}; + auto id = bf::identify(f.path); + EXPECT_EQ(id.format, bf::Format::Elf); + EXPECT_EQ(id.arch, "x86_64"); + EXPECT_TRUE(id.is64); +} + +TEST(PackBinfmt, ReadsElfDtNeededThroughTheSegmentTable) { + // DT_STRTAB is a virtual address, so the parser has to translate it via + // PT_LOAD. Two entries, in file order. + std::array needed{"libstdc++.so.6", "libcustom.so.1"}; + TempFile f{"elfneeded", elf_with_needed(needed)}; + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + ASSERT_EQ(names->size(), 2u); + EXPECT_EQ((*names)[0], "libstdc++.so.6"); + EXPECT_EQ((*names)[1], "libcustom.so.1"); +} + +TEST(PackBinfmt, AnElfWithNoDynamicSectionHasZeroDepsAndIsNotAnError) { + // A fully static binary. "No dependencies" and "could not be read" must + // not look alike — one is a packageable artifact and the other is a bug. + auto bytes = elf_with_needed({}); + // Strip PT_DYNAMIC by zeroing its type field. + put(bytes, 120, 0, 4); + TempFile f{"elfstatic", bytes}; + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + EXPECT_TRUE(names->empty()); +} + +TEST(PackBinfmt, IdentifiesPeAndReadsBothImportDirectories) { + std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; + std::array delayed{"dbghelp.dll"}; + TempFile f{"pe", pe_with_imports(imports, delayed)}; + + auto id = bf::identify(f.path); + EXPECT_EQ(id.format, bf::Format::Pe); + EXPECT_EQ(id.arch, "x86_64"); + EXPECT_TRUE(id.is64); + + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + // The DELAY-loaded one is a dependency too, and leaving it out is worse + // than leaving out an ordinary import: a missing delay-load does not fail + // at startup, it fails at the first call through it. + EXPECT_EQ(names->size(), 3u); + for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) + EXPECT_NE(std::ranges::find(*names, want), names->end()) + << want << " missing from the closure"; +} + +TEST(PackBinfmt, ADosStubWithoutAPeSignatureIsNotAPe) { + // "MZ" alone is a DOS executable. Calling it PE would send the caller + // into a parser that cannot succeed, and the error would describe the + // wrong thing. + std::string b(0x100, '\0'); + b[0] = 'M'; b[1] = 'Z'; + put(b, 0x3C, 0x40, 4); // points at zeros, not "PE\0\0" + TempFile f{"dos", b}; + EXPECT_EQ(bf::identify(f.path).format, bf::Format::Unknown); + EXPECT_FALSE(bf::needed_names(f.path).has_value()); +} + +TEST(PackBinfmt, TruncatedInputIsRejectedRatherThanRead) { + // Malformed input is ordinary: a half-downloaded file, a text file named + // `.exe`. Every read is bounds-checked, so the parser is total over it. + for (std::size_t keep : {0u, 4u, 0x40u, 0x80u}) { + std::array imports{"KERNEL32.dll"}; + auto bytes = pe_with_imports(imports, {}); + bytes.resize(keep); + TempFile f{"trunc", bytes}; + auto names = bf::needed_names(f.path); + if (names) EXPECT_TRUE(names->empty()) << "keep=" << keep; + } +} + +TEST(PackBinfmt, TheSystemPredicateKnowsWindowsFromTheToolset) { + // Windows' own: shipping a private copy is a broken program, not a + // heavier one. + for (auto n : {"KERNEL32.dll", "kernel32.dll", "ucrtbase.dll", "ntdll.dll", + "api-ms-win-crt-runtime-l1-1-0.dll"}) + EXPECT_TRUE(bf::is_system_lib(bf::Format::Pe, n)) << n; + + // NOT Windows' own. These belong to the TOOLSET, and whether they travel + // is `cxx_runtime`'s decision — the one call this predicate must not make + // on its behalf. + for (auto n : {"vcruntime140.dll", "vcruntime140_1.dll", "msvcp140.dll", + "libwinpthread-1.dll", "libstdc++-6.dll"}) + EXPECT_FALSE(bf::is_system_lib(bf::Format::Pe, n)) << n; + + // ELF keeps the manylinux allow-list it always had. + EXPECT_TRUE(bf::is_system_lib(bf::Format::Elf, "libstdc++.so.6")); + EXPECT_TRUE(bf::is_system_lib(bf::Format::Elf, "ld-linux-x86-64.so.2")); + EXPECT_FALSE(bf::is_system_lib(bf::Format::Elf, "libcurl.so.4")); +} + +// ─── the zip writer ────────────────────────────────────────────────────── + +TEST(PackZip, Crc32MatchesTheKnownVectors) { + // Self-consistency proves nothing here: an archive whose CRCs are wrong + // extracts with a warning on some tools and silently on others. + EXPECT_EQ(mcpp::pack::zip::crc32(""), 0u); + EXPECT_EQ(mcpp::pack::zip::crc32("123456789"), 0xCBF43926u); + EXPECT_EQ(mcpp::pack::zip::crc32( + "The quick brown fox jumps over the lazy dog"), 0x414FA339u); +} + +TEST(PackZip, WritesAReadableArchiveWithForwardSlashNames) { + auto dir = std::filesystem::temp_directory_path() + / std::format("mcpp-zip-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(dir); + std::ofstream{dir / "a.txt"} << "hello"; + std::ofstream{dir / "b.dll"} << "MZ-not-really"; + + std::vector entries{ + {"pkg/a.txt", dir / "a.txt", false}, + {"pkg/b.dll", dir / "b.dll", false}, + }; + auto out = dir / "pkg.zip"; + auto r = mcpp::pack::zip::write(out, entries); + ASSERT_TRUE(r.has_value()) << r.error(); + + std::ifstream is(out, std::ios::binary); + std::string bytes((std::istreambuf_iterator(is)), {}); + ASSERT_GE(bytes.size(), 22u); + EXPECT_EQ(bytes.substr(0, 4), std::string("PK\x03\x04", 4)) + << "no local file header signature"; + EXPECT_NE(bytes.find(std::string("PK\x01\x02", 4)), std::string::npos) + << "no central directory"; + EXPECT_EQ(bytes.substr(bytes.size() - 22, 4), std::string("PK\x05\x06", 4)) + << "end-of-central-directory is not last (a trailing comment?)"; + // Names are stored verbatim; a backslash here is what makes an archive + // written on Windows extract to one oddly-named file everywhere else. + EXPECT_NE(bytes.find("pkg/a.txt"), std::string::npos); + EXPECT_EQ(bytes.find("pkg\\a.txt"), std::string::npos); + // Stored, not deflated: the payload appears literally. + EXPECT_NE(bytes.find("hello"), std::string::npos); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +TEST(PackZip, TwoWritesOfTheSameInputAreByteIdentical) { + // A published checksum only means something if the archive is a function + // of its contents. Reading mtimes would make it a function of when. + auto dir = std::filesystem::temp_directory_path() + / std::format("mcpp-zip-det-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(dir); + std::ofstream{dir / "x.bin"} << "payload"; + std::vector entries{{"p/x.bin", dir / "x.bin", true}}; + + auto read = [](const std::filesystem::path& p) { + std::ifstream is(p, std::ios::binary); + return std::string((std::istreambuf_iterator(is)), {}); + }; + ASSERT_TRUE(mcpp::pack::zip::write(dir / "one.zip", entries).has_value()); + ASSERT_TRUE(mcpp::pack::zip::write(dir / "two.zip", entries).has_value()); + EXPECT_EQ(read(dir / "one.zip"), read(dir / "two.zip")); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} From 92562604f22de1dfeb828f8f3b14d2b0b4b37d82 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 04:51:36 +0800 Subject: [PATCH 04/10] =?UTF-8?q?chore:=20bump=20version=20to=202026.8.17.?= =?UTF-8?q?1,=20and=20record=20what=20=C2=A71=E2=80=93=C2=A74=20actually?= =?UTF-8?q?=20became?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The design doc gains a status section per item, including the two places the PLAN was corrected by the implementation and the one half of §4 that was deliberately not done: - `dist::Format` now reads the target triple before falling back to the host. Everything but MinGW used to ask the host, which made a Windows contract unassertable on the Linux runner where most of this gets reviewed. It only ADDS answers, so no existing build changes. - `force_bundle` had to reach the PE system list too. ELF always worked that way; making the PE exclusion unconditional would have turned an explicitly written decision into decoration. - The ELF closure still runs the artifact. `ldd` hands back RESOLVED PATHS while `DT_NEEDED` gives only names, and turning names into paths means reimplementing the loader's search order ($ORIGIN, DT_RPATH before LD_LIBRARY_PATH before DT_RUNPATH before ld.so.cache, hwcaps). Rewriting that under a correct, e2e-covered path is more risk than it buys — and the cost is stated rather than left to be discovered: cross-ARCHITECTURE ELF packing is still unsupported, which is the second limit §4.1 names. `mcpp self doctor` reports the Windows SDK PER ORIGIN. One unlabelled line was the same "one question, two answerers" shape this axis exists to close: a user reading it would believe it applied to their pinned build, and it did not. --- ...-16-windows-toolchain-three-axes-design.md | 42 +++++++++++++++++++ mcpp.toml | 2 +- src/build/prepare.cppm | 42 +++++++++++++------ src/doctor.cppm | 40 ++++++++++++++++-- src/version.cppm | 2 +- 5 files changed, 110 insertions(+), 18 deletions(-) diff --git a/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md index 735f25e6..bbccc394 100644 --- a/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md +++ b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md @@ -298,6 +298,48 @@ LD_TRACE_LOADED_OBJECTS=1 '' --- +## 6.5 落地状态(2026-08-17,mcpp 2026.8.17.1) + +**§1 / §2 / §3 / §4 全部实现,§2.4 明确不做。** 逐条对应: + +| 条目 | 状态 | 落点 | +|---|---|---| +| §1 Origin 建模 | ✅ | `enum class Origin`(`model.cppm`)—— 放在数据模型里,让 spec 侧(`registry`)与已定位编译器侧(`msvc`)指的是**同一根轴** | +| §1 拒绝 `gcc@system` | ✅ | `parse_toolchain_spec`,错误里同时给出 pin 与 PATH 逃生口两种写法 | +| §1 解析一次 | ✅ | `prepare` 原先把同一个字符串**解析两遍**、各自下结论;现在一次,`origin_of()` 分发 | +| §1 消重复:`resolve_managed_msvc` | ✅ | `registry.cppm`。"payload 在哪、为什么不能用 fetcher 的 `root`"两份手写实现,而理由只写在其中一份里 | +| §1 消重复:sysroot 谓词 | ✅ | `needs_linux_sysroot_payloads()`。两份**不等价**,而其中一份的注释声称它们互为镜像 —— 少的是 PE 那一项 | +| §1 解析链注释 | ✅ | 一张按 `TcOrigin` 枚举名写的表。原先两处、分别声称 3 步和 4 步,合起来点到 9 个输入里的 5 个,还互相矛盾 | +| §2.2 SDK 按来源分流 | ✅ | `msvc::resolve_sdk_for()`;受管来源忽略 `WindowsSdkDir`/`WindowsSdkVersion` 并打印 `note:` | +| §2.2 无 SDK payload 时 | ✅ | 退回机器 SDK 并**说出来**(可用 > 失败,但不可复现这件事必须留痕) | +| §2.3 `ucrt@` 身份 | ✅ | `bind_windows_ucrt()`,进 `runtimeContractHash`;`runtime_provider()` 取代各处 `starts_with("glibc@")` | +| §2.4 manifest SDK 键 | ⛔ **不做** | 理由见该节 | +| §3.3 PE `toolchain-coupled` | ✅ | `dist::Mechanism::deployToolchainRuntime` → `CompileFlags::toolchainRuntimeDeploy` → ninja `stage_file` 边 | +| §3.3 排除 debug CRT | ✅ | 判据只有一处(`vc_redist_dir()`),拷贝那步复用它而不是另立一条按名字的规则 | +| §4 静态读闭包 | ✅ | `mcpp.pack.binfmt`:ELF `DT_NEEDED`、PE 导入表 **+ 延迟导入表** | +| §4 跨 OS 打包 | ✅ | Linux 上给 Windows 产物打 zip;`mcpp.pack.zip` 自己写压缩包(没有哪个 zip 工具在每个宿主上都存在) | +| §4.3 `pack` 读 Contract | ✅ | `cxx_runtime` 决定 toolchain runtime 目录进不进搜索集;与 `--mode` 矛盾时拒绝 | + +**两处对方案本身的修正**,来自实现过程: + +1. **`dist::Format` 改为先看 target triple、再退回宿主。** 原先除 MinGW 外一律 + 问宿主,于是"Windows 上的契约"在 Linux runner 上**根本无法断言** —— 而这轮 + 大部分 review 就发生在 Linux runner 上。只**新增**答案:说不出 OS 的 triple + 仍走原来那条推导,现有构建一个都不变。 +2. **`force_bundle` 在 PE 上也要能覆盖系统表。** ELF 上一直如此;PE 上第一版把 + 系统排除写成了无条件的,那会让一个明确写下来的决定变成装饰。 + +**§4 未做的一半,以及理由。** ELF 闭包**仍然**通过运行产物取得 +(`LD_TRACE_LOADED_OBJECTS`),`binfmt` 的 `DT_NEEDED` 读取只用于识别与诊断。 +原因不是懒:`ldd` 交回的是**已解析的路径**,而 `DT_NEEDED` 只有名字,把名字变 +成路径要重新实现 loader 的搜索规则(`DT_RPATH` → `LD_LIBRARY_PATH` → +`DT_RUNPATH` → `ld.so.cache` → 默认目录,加上 `$ORIGIN` 展开与 hwcaps 子目录)。 +在一条**已经正确、且被 e2e 覆盖**的路径上重写这个,风险远大于收益。 +**代价要说清楚:跨架构的 ELF 打包(x86_64 上给 aarch64 产物打包)仍然不支持**, +而这正是 §4.1 指出的第二个限制。PE 那条没有既有实现,所以它从一开始就是静态的。 + +--- + ## 7. 落地顺序 | # | 项 | 规模 | 依赖 | diff --git a/mcpp.toml b/mcpp.toml index e1646824..e1f18217 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.16.3" +version = "2026.8.17.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index ea425a80..a4f91dd7 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -1294,18 +1294,36 @@ prepare_build(bool print_fingerprint, // resolution / fingerprinting. fold_build_defines_into_flags(m->buildConfig); - // msvc@system: located on the machine, never resolved through xim - // packages. mcpp does not install the machine's Visual Studio. + // ORIGIN, RESOLVED ONCE. // - // A VERSIONED msvc spec is a different origin and takes the xim path - // below — which is the whole point: what the manifest says is what gets + // The spec used to be parsed TWICE from the same string a dozen lines + // apart — once to ask "is this msvc@system", once to get the package — + // and each call site drew its own conclusions from the result. Two parses + // of one string is two places for the answer to differ, which is the shape + // §1 of the three-axes design is about: a platform special case whose cost + // is paid at every site that has to know about it. + // + // `Origin::SystemMsvc` is located on the machine and never resolved + // through an xim package — mcpp does not install the machine's Visual + // Studio. `Origin::Managed` is everything else, including a VERSIONED + // msvc spec, and that is the point: what the manifest says is what gets // used, on every machine, instead of whatever this one happens to have. - bool tcSpecIsMsvc = false; - if (tcSpec.has_value()) { - if (auto s = mcpp::toolchain::parse_toolchain_spec(*tcSpec); - s && mcpp::toolchain::is_system_toolchain(*s)) - tcSpecIsMsvc = true; + std::optional parsedSpec; + auto tcOriginAxis = mcpp::toolchain::Origin::Managed; + if (tcSpec.has_value() && *tcSpec != "system") { + // A parse FAILURE is not the same as an unparseable spec being + // absent: `gcc@system` now fails here by name (see + // parse_toolchain_spec), and swallowing that would put the error back + // where it used to happen — somewhere else, saying something else. + auto s = mcpp::toolchain::parse_toolchain_spec(*tcSpec); + if (!s) return std::unexpected(std::format( + "[toolchain].{} = '{}': {}", kCurrentPlatform, *tcSpec, s.error())); + parsedSpec = std::move(*s); + tcOriginAxis = mcpp::toolchain::origin_of(*parsedSpec); } + const bool tcSpecIsMsvc = + parsedSpec && tcOriginAxis == mcpp::toolchain::Origin::SystemMsvc; + if (tcSpecIsMsvc) { if (!mcpp::platform::is_windows) { return std::unexpected(std::format( @@ -1319,9 +1337,9 @@ prepare_build(bool print_fingerprint, mcpp::ui::info("Resolved", std::format( "msvc@system → msvc {} ({})", inst->display_version(), inst->clPath.string())); - } else if (tcSpec.has_value() && *tcSpec != "system") { - auto spec = mcpp::toolchain::parse_toolchain_spec(*tcSpec); - if (!spec || spec->version.empty()) { + } else if (parsedSpec) { + auto spec = parsedSpec; + if (spec->version.empty()) { return std::unexpected(std::format( "[toolchain].{} = '{}' is invalid; expected '@'", kCurrentPlatform, *tcSpec)); diff --git a/src/doctor.cppm b/src/doctor.cppm index ece85bb4..96e62d81 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -133,13 +133,45 @@ export int doctor_report() { warn("msvc not detected — run `mcpp toolchain default msvc` for " "setup guidance (mcpp does not install MSVC)"); } - // Windows SDK (native cl.exe builds need its UCRT/um headers). + // Windows SDK. REPORTED PER ORIGIN, because it is chosen per origin + // (design §2.2): `msvc@system` searches the machine, a pinned toolset + // takes the payload installed with it. One unlabelled line here would + // be the same "one question, two answerers" shape the SDK axis exists + // to close — a user reading it would believe it applied to their + // pinned build, and it does not. if (auto sdk = mcpp::toolchain::msvc::find_windows_sdk()) { - ok(std::format("Windows SDK {} at {}", sdk->version, + ok(std::format("Windows SDK (msvc@system) {} at {}", sdk->version, sdk->root.string())); } else { - warn("no Windows SDK found — native msvc builds will fail " - "(install the 'Windows 11 SDK' VS component)"); + warn("no Windows SDK found for msvc@system — native builds with " + "the machine's Visual Studio will fail (install the " + "'Windows 11 SDK' VS component)"); + } + { + std::error_code sdkEc; + auto msvcRoot = mcpp::home::root() + / "registry" / "data" / "xpkgs" / "xim-x-msvc"; + for (auto& v : std::filesystem::directory_iterator(msvcRoot, sdkEc)) { + if (!v.is_directory(sdkEc)) continue; + auto ver = v.path().filename().string(); + auto inst = mcpp::toolchain::msvc::installation_at( + v.path(), ver, /*identifyVersion=*/false); + if (!inst) continue; + // The SAME resolution a build performs, not a second one + // shaped like it: what doctor prints is what the user will + // believe, so the two must not be able to disagree. + auto choice = mcpp::toolchain::msvc::resolve_sdk_for(inst->clPath); + if (choice.sdk) { + ok(std::format("Windows SDK (msvc@{}) {} at {}", + ver, choice.sdk->version, + choice.sdk->root.string())); + } else { + warn(std::format( + "msvc@{} has no usable Windows SDK — reinstall it to " + "pull its `xim:windows-sdk` dependency", ver)); + } + if (!choice.note.empty()) warn(choice.note); + } } mcpp::ui::status("Checking", "mingw (xim:mingw-gcc)"); diff --git a/src/version.cppm b/src/version.cppm index 76e7199c..13ca3351 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.16.3"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.17.1"; } // namespace mcpp From 44f58bda45f68d8682546e58a5e1527fdf903fb0 Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:06:51 +0800 Subject: [PATCH 05/10] fix(pack): stop using the module-boundary shapes clang miscompiles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every Windows job and the macOS job went red on the same new code, in two different ways, while gcc was green everywhere: Windows clang 20.1.7 (MSVC ABI) segfaulted COMPILING mcpp.pack — 0xC0000005, no diagnostic, five jobs at once macOS test_pack_binfmt died with SIGSEGV at RUN time, inside the PE import-table test The parser is not the problem, and that was measured rather than assumed: the same code is clean under ASan+UBSan with clang 22.1.8 + libc++, and correct when compiled AS A CLANG MODULE on x86_64 Linux at both -O0 and -O2. What is left is the compiler, on the two targets neither of those probes covers. This codebase has been here before. hostflags.cppm exists because adding an UNUSED helper to a module's anonymous namespace miscompiled a NEIGHBOURING function under clang + C++20 modules + -O2, and its verdict was "mechanism unknown, reproduction solid; the cheap response is to not grow that namespace". Same response here — remove the shapes, keep the behaviour: mcpp.pack a scoped enum from another module as a defaulted member of an EXPORTED struct (`dist::Contract` in `Options`) → a plain bool, since only one of the three values changes anything here; a ranges projection over an imported type's member → std::sort with a comparator mcpp.pack.zip `std::span` across the boundary → the vector by const reference (one caller) mcpp.pack.binfmt `template le(...)` in the module purview → four concrete le8/le16/le32/le64; `constexpr std::array` via `std::to_array<>` → plain arrays; ranges algorithms over them → loops WHICH ONE IT WAS IS NOT ESTABLISHED, and the comments say so rather than inventing a finding — they were removed together because each CI round costs minutes and none of the replacements is worse than what it replaced. Every one is also simpler, so nothing is being paid for the avoidance. Behaviour is unchanged: the same 10 unit tests and e2e 240 pass. --- src/pack/binfmt.cppm | 139 +++++++++++++++++++++++++---------------- src/pack/pack.cppm | 60 ++++++++++++++---- src/pack/pipeline.cppm | 6 +- src/pack/zip.cppm | 10 ++- 4 files changed, 147 insertions(+), 68 deletions(-) diff --git a/src/pack/binfmt.cppm b/src/pack/binfmt.cppm index c1dddb47..3bd03e3c 100644 --- a/src/pack/binfmt.cppm +++ b/src/pack/binfmt.cppm @@ -116,16 +116,47 @@ std::optional slurp(const std::filesystem::path& p) { } // Little-endian integer at `off`, or nullopt if it would run off the end. -// EVERY read in this module goes through here: a malformed file is ordinary +// EVERY read in this module goes through these: a malformed file is ordinary // input (a truncated download, a text file named `.exe`), and the parser has // to be total over it rather than trusting a length field it just read. -template -std::optional le(std::string_view b, std::size_t off) { - if (off + sizeof(T) > b.size()) return std::nullopt; - T v = 0; - for (std::size_t i = 0; i < sizeof(T); ++i) - v |= static_cast(static_cast(b[off + i])) - << (8 * i); +// +// FOUR CONCRETE FUNCTIONS, NOT ONE TEMPLATE, and the difference is not style. +// A function template in a module interface's non-exported namespace is a +// shape this codebase has been bitten by before (see hostflags.cppm's opening +// comment: an unused helper added to a module's anonymous namespace +// miscompiled a NEIGHBOURING function under clang + C++20 modules). The first +// version of this file used `template le(...)`, and clang +// misbehaved on BOTH of its targets at once while gcc was fine — a frontend +// segfault on Windows and a runtime segfault on macOS ARM64, from code that +// is clean under ASan+UBSan and correct as a clang module on x86_64 Linux. +// Whether the template was the cause is NOT established; what is established +// is that the failures are compiler-side, and that four short functions cost +// nothing. +std::optional le8(std::string_view b, std::size_t off) { + if (off >= b.size()) return std::nullopt; + return static_cast(static_cast(b[off])); +} + +std::optional le16(std::string_view b, std::size_t off) { + if (off + 2 > b.size()) return std::nullopt; + return static_cast( + static_cast(b[off]) + | (static_cast(static_cast(b[off + 1])) << 8)); +} + +std::optional le32(std::string_view b, std::size_t off) { + if (off + 4 > b.size()) return std::nullopt; + std::uint32_t v = 0; + for (int i = 3; i >= 0; --i) + v = (v << 8) | static_cast(b[off + static_cast(i)]); + return v; +} + +std::optional le64(std::string_view b, std::size_t off) { + if (off + 8 > b.size()) return std::nullopt; + std::uint64_t v = 0; + for (int i = 7; i >= 0; --i) + v = (v << 8) | static_cast(b[off + static_cast(i)]); return v; } @@ -171,8 +202,8 @@ std::string pe_arch(std::uint16_t machine) { // makes this more than "read a list". std::expected, std::string> elf_needed(std::string_view b) { - auto cls = le(b, 4); - auto dat = le(b, 5); + auto cls = le8(b, 4); + auto dat = le8(b, 5); if (!cls || !dat) return std::unexpected("ELF header is truncated"); if (*dat != 1) return std::unexpected("big-endian ELF objects are not supported"); @@ -183,16 +214,16 @@ elf_needed(std::string_view b) { const std::size_t phoffAt = is64 ? 0x20 : 0x1C; std::uint64_t phoff = 0; if (is64) { - auto v = le(b, phoffAt); + auto v = le64(b, phoffAt); if (!v) return std::unexpected("ELF program header offset is truncated"); phoff = *v; } else { - auto v = le(b, phoffAt); + auto v = le32(b, phoffAt); if (!v) return std::unexpected("ELF program header offset is truncated"); phoff = *v; } - auto phentsize = le(b, is64 ? 0x36 : 0x2A); - auto phnum = le(b, is64 ? 0x38 : 0x2C); + auto phentsize = le16(b, is64 ? 0x36 : 0x2A); + auto phnum = le16(b, is64 ? 0x38 : 0x2C); if (!phentsize || !phnum) return std::unexpected("ELF program header table is truncated"); @@ -202,12 +233,12 @@ elf_needed(std::string_view b) { for (std::uint16_t i = 0; i < *phnum; ++i) { const std::size_t ph = static_cast(phoff) + static_cast(i) * *phentsize; - auto type = le(b, ph); + auto type = le32(b, ph); if (!type) break; auto rd = [&](std::size_t off64, std::size_t off32) -> std::optional { - if (is64) return le(b, ph + off64); - if (auto v = le(b, ph + off32)) return *v; + if (is64) return le64(b, ph + off64); + if (auto v = le32(b, ph + off32)) return *v; return std::nullopt; }; auto offset = rd(0x08, 0x04); @@ -235,13 +266,13 @@ elf_needed(std::string_view b) { at += entSize) { std::uint64_t tag = 0, val = 0; if (is64) { - auto t = le(b, static_cast(at)); - auto v = le(b, static_cast(at) + 8); + auto t = le64(b, static_cast(at)); + auto v = le64(b, static_cast(at) + 8); if (!t || !v) break; tag = *t; val = *v; } else { - auto t = le(b, static_cast(at)); - auto v = le(b, static_cast(at) + 4); + auto t = le32(b, static_cast(at)); + auto v = le32(b, static_cast(at) + 4); if (!t || !v) break; tag = *t; val = *v; } @@ -275,15 +306,15 @@ elf_needed(std::string_view b) { // Leaving it out of the closure would produce exactly that. std::expected, std::string> pe_needed(std::string_view b) { - auto lfanew = le(b, 0x3C); + auto lfanew = le32(b, 0x3C); if (!lfanew) return std::unexpected("PE: no e_lfanew"); const std::size_t nt = *lfanew; if (b.substr(nt, 4) != std::string_view("PE\0\0", 4)) return std::unexpected("PE: no PE\\0\\0 signature at e_lfanew"); - auto numSections = le(b, nt + 6); - auto optSize = le(b, nt + 20); - auto magic = le(b, nt + 24); + auto numSections = le16(b, nt + 6); + auto optSize = le16(b, nt + 20); + auto magic = le16(b, nt + 24); if (!numSections || !optSize || !magic) return std::unexpected("PE: headers are truncated"); // 0x10b PE32, 0x20b PE32+. They differ only in where the data directories @@ -292,7 +323,7 @@ pe_needed(std::string_view b) { if (*magic == 0x10b) dirsAt = nt + 24 + 96; else if (*magic == 0x20b) dirsAt = nt + 24 + 112; else return std::unexpected("PE: optional header magic is neither PE32 nor PE32+"); - auto numDirs = le(b, dirsAt - 4); + auto numDirs = le32(b, dirsAt - 4); if (!numDirs) return std::unexpected("PE: data directory count is truncated"); struct Section { std::uint32_t va, vsize, raw, rawSize; }; @@ -300,10 +331,10 @@ pe_needed(std::string_view b) { const std::size_t secAt = nt + 24 + *optSize; for (std::uint16_t i = 0; i < *numSections; ++i) { const std::size_t s = secAt + static_cast(i) * 40; - auto vsize = le(b, s + 8); - auto va = le(b, s + 12); - auto rawSz = le(b, s + 16); - auto raw = le(b, s + 20); + auto vsize = le32(b, s + 8); + auto va = le32(b, s + 12); + auto rawSz = le32(b, s + 16); + auto raw = le32(b, s + 20); if (!vsize || !va || !rawSz || !raw) break; sections.push_back({*va, *vsize, *raw, *rawSz}); } @@ -323,22 +354,23 @@ pe_needed(std::string_view b) { }; std::vector out; - auto push = [&](std::optional name) { + auto push = [&out](const std::optional& name) { if (!name || name->empty()) return; - if (std::ranges::find(out, *name) == out.end()) - out.push_back(std::move(*name)); + for (auto const& seen : out) + if (seen == *name) return; + out.push_back(*name); }; // Directory 1 — imports. 20-byte descriptors, terminated by an all-zero // one; `Name` (offset 12) is an RVA to the DLL's ASCIIZ name. if (*numDirs > 1) { - auto rva = le(b, dirsAt + 1 * 8); + auto rva = le32(b, dirsAt + 1 * 8); if (rva && *rva) { if (auto at = rva_to_off(*rva)) { for (std::size_t d = *at; ; d += 20) { - auto nameRva = le(b, d + 12); - auto oft = le(b, d); - auto ft = le(b, d + 16); + auto nameRva = le32(b, d + 12); + auto oft = le32(b, d); + auto ft = le32(b, d + 16); if (!nameRva || !oft || !ft) break; if (*nameRva == 0 && *oft == 0 && *ft == 0) break; if (auto o = rva_to_off(*nameRva)) push(cstr(b, *o)); @@ -352,12 +384,12 @@ pe_needed(std::string_view b) { // ADDRESSES instead, and nothing produced in this century does that, so a // descriptor without the bit is skipped rather than guessed at. if (*numDirs > 13) { - auto rva = le(b, dirsAt + 13 * 8); + auto rva = le32(b, dirsAt + 13 * 8); if (rva && *rva) { if (auto at = rva_to_off(*rva)) { for (std::size_t d = *at; ; d += 32) { - auto attrs = le(b, d); - auto nameRva = le(b, d + 4); + auto attrs = le32(b, d); + auto nameRva = le32(b, d + 4); if (!attrs || !nameRva) break; if (*attrs == 0 && *nameRva == 0) break; if ((*attrs & 1u) == 0) continue; @@ -370,15 +402,15 @@ pe_needed(std::string_view b) { } // PEP 600 / manylinux2014: assumed present on any target Linux glibc system. -constexpr std::array kElfSystem = std::to_array({ +constexpr std::string_view kElfSystem[] = { "libc.so", "libm.so", "libdl.so", "libpthread.so", "librt.so", "libutil.so", "libnsl.so", "libresolv.so", "libcrypt.so", "libstdc++.so", "libgcc_s.so", "linux-vdso.so", "ld-linux", "libld-linux", -}); +}; // Windows' own. Deliberately NOT including vcruntime140/msvcp140: those // belong to the toolset, and whether they travel is `cxx_runtime`'s decision. -constexpr std::array kPeSystem = std::to_array({ +constexpr std::string_view kPeSystem[] = { "ntdll.dll", "kernel32.dll", "kernelbase.dll", "user32.dll", "gdi32.dll", "gdi32full.dll", "advapi32.dll", "shell32.dll", "shlwapi.dll", "ole32.dll", "oleaut32.dll", "combase.dll", "comdlg32.dll", @@ -391,7 +423,7 @@ constexpr std::array kPeSystem = std::to_array({ "glu32.dll", "dxgi.dll", "d3d9.dll", "d3d11.dll", "d3d12.dll", "dbghelp.dll", "hid.dll", "avrt.dll", "mfplat.dll", "dnsapi.dll", "profapi.dll", "ucrtbase.dll", "msvcrt.dll", "win32u.dll", -}); +}; } // namespace detail @@ -403,21 +435,21 @@ Ident identify(const std::filesystem::path& binary) { if (b.starts_with(std::string_view("\x7f" "ELF", 4))) { id.format = Format::Elf; - auto cls = detail::le(b, 4); + auto cls = detail::le8(b, 4); id.is64 = cls && *cls == 2; - if (auto m = detail::le(b, 18)) id.arch = detail::elf_arch(*m); + if (auto m = detail::le16(b, 18)) id.arch = detail::elf_arch(*m); return id; } if (b.starts_with("MZ")) { // MZ alone is a DOS stub; a PE needs the signature e_lfanew points at. // Saying "PE" for a file that has none would send the caller into a // parser that cannot succeed. - if (auto lfanew = detail::le(b, 0x3C)) { + if (auto lfanew = detail::le32(b, 0x3C)) { if (b.substr(*lfanew, 4) == std::string_view("PE\0\0", 4)) { id.format = Format::Pe; - if (auto m = detail::le(b, *lfanew + 4)) + if (auto m = detail::le16(b, *lfanew + 4)) id.arch = detail::pe_arch(*m); - if (auto magic = detail::le(b, *lfanew + 24)) + if (auto magic = detail::le16(b, *lfanew + 24)) id.is64 = (*magic == 0x20b); return id; } @@ -458,16 +490,17 @@ needed_names(const std::filesystem::path& binary) { bool is_system_lib(Format f, std::string_view name) { std::string lower(name); - std::ranges::transform(lower, lower.begin(), - [](unsigned char c) { return std::tolower(c); }); + for (auto& c : lower) + c = static_cast(std::tolower(static_cast(c))); if (f == Format::Pe) { // API sets: `api-ms-win-crt-runtime-l1-1-0.dll` and friends are // forwarders resolved by the loader against the OS. They have no file // to copy on most systems and copying one would be wrong anyway. if (lower.starts_with("api-ms-") || lower.starts_with("ext-ms-")) return true; - return std::ranges::find(detail::kPeSystem, lower) - != detail::kPeSystem.end(); + for (auto known : detail::kPeSystem) + if (lower == known) return true; + return false; } if (f == Format::Elf) { for (auto prefix : detail::kElfSystem) diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index f06c7759..765558d8 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -35,7 +35,6 @@ module; export module mcpp.pack; import std; -import mcpp.build.distribution; import mcpp.build.loader_contract; import mcpp.config; import mcpp.pack.binfmt; @@ -68,15 +67,22 @@ struct Options { // toolset's `VC\Redist\MSVC\\\Microsoft.VC*.CRT\` for cl. // Searched ONLY under the toolchain-coupled contract — see make_plan. std::vector toolchainRuntimeDirs; - // What the artifact promises about the machine that runs it, as resolved - // by mcpp.build.distribution for the distributable role. + // Does the RESOLVED C++ runtime contract require the toolchain's own + // runtime to travel WITH the artifact — i.e. `cxx_runtime = + // "toolchain-coupled"`? // - // `pack` used to be unable to see this at all (design §4.3): the contract + // `pack` used to be unable to see the contract at all (design §4.3): it // reached compile and link FLAGS and stopped there, so the step that // decides which files actually travel had no idea what had been promised. // On ELF the `ldd` closure happened to agree with it; on PE nothing did. - mcpp::build::dist::Contract cxxRuntime = - mcpp::build::dist::Contract::SelfContained; + // + // A BOOL rather than `dist::Contract`, deliberately. Only one of the + // three values changes anything here, so the enum would be three states + // where the decision has two — and carrying it across this module + // boundary crashed the clang 20.1.7 frontend outright (see run_pe). The + // caller resolves the contract; this is the one bit of it that packaging + // acts on. + bool carryToolchainRuntime = false; }; // Resolved plan — all paths absolute, all decisions baked in. @@ -304,10 +310,9 @@ make_plan(const mcpp::manifest::Manifest& manifest, // (a bundle carrying its own libc cannot consume a host capability), and // this grows from the same root: a contract with no executor is a promise // the build prints and the package quietly drops. - using Contract = mcpp::build::dist::Contract; const bool modeBundlesNothing = opts.mode == Mode::None || opts.mode == Mode::Static; - if (opts.cxxRuntime == Contract::ToolchainCoupled && modeBundlesNothing) { + if (opts.carryToolchainRuntime && modeBundlesNothing) { return std::unexpected(Error{std::format( "cxx_runtime = \"toolchain-coupled\" and --mode {} contradict each " "other.\n" @@ -328,7 +333,7 @@ make_plan(const mcpp::manifest::Manifest& manifest, // definition part of what it runs with. p.searchDirs.push_back(builtBinary.parent_path()); for (auto const& d : opts.depSearchDirs) p.searchDirs.push_back(d); - if (opts.cxxRuntime == Contract::ToolchainCoupled) + if (opts.carryToolchainRuntime) for (auto const& d : opts.toolchainRuntimeDirs) p.searchDirs.push_back(d); auto distDir = projectRoot / "target" / "dist"; @@ -764,7 +769,10 @@ pe_closure(const std::filesystem::path& binary, } } } - std::ranges::sort(out, {}, &ResolvedDep::soname); + std::sort(out.begin(), out.end(), + [](const ResolvedDep& a, const ResolvedDep& b) { + return a.soname < b.soname; + }); return out; } @@ -802,6 +810,27 @@ namespace detail { // No patchelf step and no wrapper script: "put the DLLs next to the exe" IS // the relocation rule on this format, which is why the row for it in the // design's layering table reads "no operation". +// ⚠️ THIS FUNCTION ONCE CRASHED THE COMPILER, and the shape it crashed on is +// worth not reintroducing. +// +// clang 20.1.7 targeting the MSVC ABI — the pinned Windows toolchain — +// segfaulted (0xC0000005) while compiling this module interface, with no +// diagnostic, on every Windows job at once. The first version carried three +// module-boundary constructs that the rest of this file does not: +// +// 1. a scoped enum from ANOTHER module as a defaulted member of an +// EXPORTED struct (`dist::Contract cxxRuntime = …` in `Options`) +// 2. a ranges projection naming a member of an IMPORTED type +// (`std::ranges::sort(entries, {}, &zip::Entry::name)`) +// 3. `std::span` across the module boundary into zip::write +// +// All three were removed together, so WHICH one it was is not established — +// stating otherwise would be a guess dressed as a finding. What is +// established: the crash is reproducible only on that toolchain, and this +// file has a documented history of the same class (see the `detail` +// namespace comment above, and hostflags.cppm on a neighbouring function +// being miscompiled by an unrelated addition). Each replacement is also +// simpler than what it replaced, so nothing is being paid for the avoidance. std::expected run_pe(const Plan& plan) { @@ -871,7 +900,16 @@ run_pe(const Plan& plan) } // Deterministic order: a directory iteration order that leaks into an // archive is how two identical builds get two different checksums. - std::ranges::sort(entries, {}, &mcpp::pack::zip::Entry::name); + // + // Plain `std::sort` with an explicit comparator, not + // `std::ranges::sort(entries, {}, &Entry::name)`. A ranges projection + // naming a member of an IMPORTED type is one of three module-boundary + // shapes this function used to carry, and together they crashed the clang + // 20.1.7 frontend (0xC0000005, no diagnostic) on every Windows job. See + // the note at the top of run_pe. + std::sort(entries.begin(), entries.end(), + [](const mcpp::pack::zip::Entry& a, + const mcpp::pack::zip::Entry& b) { return a.name < b.name; }); if (auto r = mcpp::pack::zip::write(plan.archivePath, entries); !r) return std::unexpected(Error{r.error()}); return {}; diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index d5f1306e..9767b720 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -113,8 +113,10 @@ export int build_and_pack(Options opts, bool modeFromUser) { // honoured. { const auto flags = mcpp::build::compute_flags(ctx->plan); - opts.cxxRuntime = flags.contractByRole[ - static_cast(mcpp::build::dist::Role::Distributable)]; + opts.carryToolchainRuntime = + flags.contractByRole[static_cast( + mcpp::build::dist::Role::Distributable)] + == mcpp::build::dist::Contract::ToolchainCoupled; opts.toolchainRuntimeDirs = ctx->plan.toolchain.linkRuntimeDirs; // Where a third-party dependency's shared library may be found. Both // channels, because they answer for different things: the runtime diff --git a/src/pack/zip.cppm b/src/pack/zip.cppm index ac9047fe..0460a580 100644 --- a/src/pack/zip.cppm +++ b/src/pack/zip.cppm @@ -44,8 +44,14 @@ struct Entry { }; // Write `entries` to `out`, creating parent directories as needed. +// +// Takes the vector by reference rather than `std::span`. The +// span is the better interface and it is not worth a compiler crash: passing +// one across this module boundary was one of three constructs present when +// clang 20.1.7 (MSVC ABI) segfaulted compiling the consumer — see the note at +// mcpp.pack's `run_pe`. There is exactly one caller. std::expected -write(const std::filesystem::path& out, std::span entries); +write(const std::filesystem::path& out, const std::vector& entries); // CRC-32 (IEEE), exposed for tests: an archive whose CRCs are wrong extracts // with a warning on some tools and silently on others, so this is worth @@ -115,7 +121,7 @@ std::uint32_t crc32(std::string_view data) { } std::expected -write(const std::filesystem::path& out, std::span entries) { +write(const std::filesystem::path& out, const std::vector& entries) { std::error_code ec; if (out.has_parent_path()) std::filesystem::create_directories(out.parent_path(), ec); From e395f084ed6d4883569920f10ea0e78d11c940cf Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:17:50 +0800 Subject: [PATCH 06/10] test(pack): split the PE case so it can localise its own crash MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The combined test died with SIGSEGV on the macOS ARM64 runner and nowhere else — not under ASan+UBSan with clang 22 + libc++, not as a clang module on x86_64 Linux, not under gcc. A single test that builds a fixture, identifies it and parses it cannot say which of the three it was, so each hypothesis costs a CI round. Three tests now: the fixture is well-formed, identify(), and needed_names(). --- tests/unit/test_pack_binfmt.cpp | 36 ++++++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 5 deletions(-) diff --git a/tests/unit/test_pack_binfmt.cpp b/tests/unit/test_pack_binfmt.cpp index 3a8986a1..eb8742ea 100644 --- a/tests/unit/test_pack_binfmt.cpp +++ b/tests/unit/test_pack_binfmt.cpp @@ -229,15 +229,39 @@ TEST(PackBinfmt, AnElfWithNoDynamicSectionHasZeroDepsAndIsNotAnError) { EXPECT_TRUE(names->empty()); } -TEST(PackBinfmt, IdentifiesPeAndReadsBothImportDirectories) { +// SPLIT INTO THREE ON PURPOSE. The combined version died with SIGSEGV on the +// macOS ARM64 runner and nowhere else, and a single test that builds a +// fixture, identifies it and parses it cannot say WHICH of the three it was. +// A test that cannot localise its own failure is a test that costs a CI round +// per hypothesis. +TEST(PackBinfmt, ThePeFixtureItselfIsWellFormed) { std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; std::array delayed{"dbghelp.dll"}; - TempFile f{"pe", pe_with_imports(imports, delayed)}; + auto bytes = pe_with_imports(imports, delayed); + ASSERT_GT(bytes.size(), 0x400u); + EXPECT_EQ(bytes.substr(0, 2), "MZ"); + EXPECT_EQ(bytes.substr(0x40, 4), std::string("PE\0\0", 4)); + // The names have to be IN the image, or every assertion below is about + // the fixture rather than about the reader. + for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) + EXPECT_NE(bytes.find(want), std::string::npos) << want; +} + +TEST(PackBinfmt, IdentifiesPe) { + std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; + std::array delayed{"dbghelp.dll"}; + TempFile f{"peid", pe_with_imports(imports, delayed)}; auto id = bf::identify(f.path); EXPECT_EQ(id.format, bf::Format::Pe); EXPECT_EQ(id.arch, "x86_64"); EXPECT_TRUE(id.is64); +} + +TEST(PackBinfmt, ReadsBothPeImportDirectories) { + std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; + std::array delayed{"dbghelp.dll"}; + TempFile f{"peimp", pe_with_imports(imports, delayed)}; auto names = bf::needed_names(f.path); ASSERT_TRUE(names.has_value()) << names.error(); @@ -245,9 +269,11 @@ TEST(PackBinfmt, IdentifiesPeAndReadsBothImportDirectories) { // than leaving out an ordinary import: a missing delay-load does not fail // at startup, it fails at the first call through it. EXPECT_EQ(names->size(), 3u); - for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) - EXPECT_NE(std::ranges::find(*names, want), names->end()) - << want << " missing from the closure"; + for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) { + bool found = false; + for (auto const& n : *names) found = found || n == want; + EXPECT_TRUE(found) << want << " missing from the closure"; + } } TEST(PackBinfmt, ADosStubWithoutAPeSignatureIsNotAPe) { From a3b05a3c894d3175fcf95c3570eedeb33731fcae Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:19:49 +0800 Subject: [PATCH 07/10] docs: 03-toolchains contradicted itself about `[build] linkage` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The MinGW section says the key does not exist and is silently ignored; the MSVC section two hundred lines down showed exactly that form as the way to select `/MT`. Found by writing it out and watching mcpp print "unsupported key 'linkage' (ignored)" — which is also what a user following the page would have got, without the page telling them why nothing changed. `linkage` is exact-triple only (`[target.]`, or `--static`). Both language editions now show that, and say which section each key lives in — the zh page had it right before this round and lost it when the MSVC section was rewritten from the English one. --- docs/03-toolchains.md | 9 +++++++-- docs/zh/03-toolchains.md | 8 +++++++- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/03-toolchains.md b/docs/03-toolchains.md index 1ebc8b69..289aa58f 100644 --- a/docs/03-toolchains.md +++ b/docs/03-toolchains.md @@ -349,12 +349,17 @@ ships nor substitutes it. **CRT model.** `/MD` (host-coupled) by default; `/MT` when either ```toml +[target.x86_64-windows-msvc] +linkage = "static" # the libc axis — TARGET section, or `--static` + [build] -linkage = "static" # the libc axis cxx_runtime = "self-contained" # the C++ runtime axis ``` -is written down. On the MSVC ABI these are one physical switch — `/MT` links +is written down. Note which section each one lives in: `linkage` is +exact-triple only and **there is no `[build] linkage` key** — writing one gets +an "unsupported key (ignored)" warning and no static CRT. (This page said +exactly that a few sections up, and then showed the wrong form here.) On the MSVC ABI these are one physical switch — `/MT` links the C and C++ runtimes out of the same library — so both spellings select it and mean the same thing. It is a **whole-project** property: one `std` module is built per project and cl bakes `_MSVC_MT`/`_MSVC_MD` into it, so a diff --git a/docs/zh/03-toolchains.md b/docs/zh/03-toolchains.md index ae97488a..08cd9105 100644 --- a/docs/zh/03-toolchains.md +++ b/docs/zh/03-toolchains.md @@ -325,11 +325,17 @@ SDK 只能靠找,而在那里"明确声明"应当压过"扫描" —— 和 `VSIN **CRT 模型。** 默认 `/MD`(host-coupled);下面两者任一都会选 `/MT`: ```toml +[target.x86_64-windows-msvc] +linkage = "static" # libc 那根轴 —— TARGET 段,或 `--static` + [build] -linkage = "static" # libc 那根轴 cxx_runtime = "self-contained" # C++ 运行时那根轴 ``` +注意这两个键分别属于哪个段:`linkage` 只认精确 triple,**没有 `[build] linkage` +这个键** —— 写了会得到一条 "unsupported key (ignored)" 警告,而且不会切到静态 +CRT。 + toolset 自带的那份可再分发 CRT(`vcruntime140.dll` / `msvcp140.dll`)可以跟着 产物走 —— 见 `docs/zh/05-mcpp-toml.md` 的 `cxx_runtime = "toolchain-coupled"`。 From ea8b1be1bf7c3435b171ddbf9365200b43b8c82c Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:27:15 +0800 Subject: [PATCH 08/10] test(pack): rewrite the PE fixture plainly, and make it say where it dies MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Splitting the test localised the macOS ARM64 SIGSEGV to ThePeFixtureItselfIsWellFormed — fixture code that calls no module at all. So it is not mcpp.pack.binfmt, and the three probes that came back clean (ASan + UBSan under clang 22 + libc++, the same module compiled by clang on x86_64 Linux, gcc everywhere) were looking in the right place for the wrong thing. The fixture used two `std::span` parameters and two lambdas that mutated a captured string through a captured cursor. It now takes vectors, indexes explicitly, and captures nothing — and traces each phase to stderr, so if it moves again the log names the step instead of costing another CI round. `#include ` is not redundant next to `import std;`: `stderr` is a macro, and a module cannot export one. --- tests/unit/test_pack_binfmt.cpp | 122 ++++++++++++++++++++------------ 1 file changed, 75 insertions(+), 47 deletions(-) diff --git a/tests/unit/test_pack_binfmt.cpp b/tests/unit/test_pack_binfmt.cpp index eb8742ea..9893921a 100644 --- a/tests/unit/test_pack_binfmt.cpp +++ b/tests/unit/test_pack_binfmt.cpp @@ -1,4 +1,5 @@ #include +#include // stderr is a MACRO — `import std;` cannot export it import std; import mcpp.pack.binfmt; @@ -104,18 +105,33 @@ std::string elf_with_needed(std::span needed) { // ─── a minimal PE32+ image ────────────────────────────────────────────── // -// One section mapping RVA 0x1000 → file 0x200, an import directory and a +// One section mapping RVA 0x1000 → file 0x400, an import directory and a // delay-import directory inside it. -std::string pe_with_imports(std::span imports, - std::span delayImports) { - constexpr std::size_t kNt = 0x40; - constexpr std::size_t kOptSize = 0xF0; // 112 + 16 directories * 8 - constexpr std::size_t kSecAt = kNt + 24 + kOptSize; - constexpr std::size_t kRawAt = 0x400; - constexpr std::uint32_t kSecVa = 0x1000; +// +// WRITTEN AS PLAINLY AS POSSIBLE, and the breadcrumbs are not decoration. +// The first version used two `std::span` parameters and two lambdas that +// mutated a captured string through a captured cursor; it segfaulted on the +// macOS ARM64 runner and NOWHERE else — not under ASan+UBSan with clang 22 + +// libc++, not as a clang module on x86_64 Linux, not under gcc. Splitting the +// test proved the crash is HERE, in fixture code that touches no module at +// all, so the shapes went and the trace stayed: if it moves again, the log +// says which step. +void trace(const char* step, const std::string& b) { + std::fprintf(stderr, "[pe-fixture] %-14s size=%zu\n", step, b.size()); + std::fflush(stderr); +} - std::string b; - b.resize(kRawAt, '\0'); +std::string pe_with_imports(const std::vector& imports, + const std::vector& delayImports) { + const std::size_t kNt = 0x40; + const std::size_t kOptSize = 0xF0; // 112 + 16 directories * 8 + const std::size_t kSecAt = kNt + 24 + kOptSize; + const std::size_t kRawAt = 0x400; + const std::uint32_t kSecVa = 0x1000; + const std::size_t dirsAt = kNt + 24 + 112; + + std::string b(kRawAt, '\0'); + trace("start", b); b[0] = 'M'; b[1] = 'Z'; put(b, 0x3C, kNt, 4); b[kNt] = 'P'; b[kNt + 1] = 'E'; // "PE\0\0" @@ -123,68 +139,87 @@ std::string pe_with_imports(std::span imports, put(b, kNt + 6, 1, 2); // NumberOfSections put(b, kNt + 20, kOptSize, 2); // SizeOfOptionalHeader put(b, kNt + 24, 0x20b, 2); // PE32+ - const std::size_t dirsAt = kNt + 24 + 112; put(b, dirsAt - 4, 16, 4); // NumberOfRvaAndSizes + trace("headers", b); // Section header: name, VirtualSize, VirtualAddress, SizeOfRawData, // PointerToRawData. - const std::string_view secName = ".rdata"; + const std::string secName = ".rdata"; for (std::size_t i = 0; i < secName.size(); ++i) b[kSecAt + i] = secName[i]; put(b, kSecAt + 8, 0x1000, 4); put(b, kSecAt + 12, kSecVa, 4); put(b, kSecAt + 16, 0x1000, 4); put(b, kSecAt + 20, kRawAt, 4); - - auto rva_of = [&](std::size_t fileOff) { - return static_cast(kSecVa + (fileOff - kRawAt)); - }; + trace("section", b); // Names first, so the descriptors can point at them. - std::vector importNameRvas, delayNameRvas; + std::vector importNameRvas; + std::vector delayNameRvas; std::size_t cursor = kRawAt; - auto emit_name = [&](std::string_view n) { - auto rva = rva_of(cursor); - for (char c : n) put(b, cursor++, static_cast(c), 1); - put(b, cursor++, 0, 1); - return rva; - }; - for (auto n : imports) importNameRvas.push_back(emit_name(n)); - for (auto n : delayImports) delayNameRvas.push_back(emit_name(n)); + for (std::size_t which = 0; which < 2; ++which) { + const std::vector& names = which == 0 ? imports : delayImports; + for (std::size_t k = 0; k < names.size(); ++k) { + const std::uint32_t rva = + static_cast(kSecVa + (cursor - kRawAt)); + const std::string& n = names[k]; + for (std::size_t i = 0; i < n.size(); ++i) { + put(b, cursor, static_cast(n[i]), 1); + ++cursor; + } + put(b, cursor, 0, 1); + ++cursor; + if (which == 0) importNameRvas.push_back(rva); + else delayNameRvas.push_back(rva); + } + } + trace("names", b); // Import descriptors (20 bytes each) + an all-zero terminator. - cursor = (cursor + 15) & ~std::size_t{15}; + cursor = (cursor + 15) & ~static_cast(15); const std::size_t importAt = cursor; - for (auto rva : importNameRvas) { + for (std::size_t k = 0; k < importNameRvas.size(); ++k) { put(b, cursor + 0, 0x9000, 4); // OriginalFirstThunk (nonzero) - put(b, cursor + 12, rva, 4); // Name + put(b, cursor + 12, importNameRvas[k], 4); // Name put(b, cursor + 16, 0x9100, 4); // FirstThunk (nonzero) cursor += 20; } - for (int i = 0; i < 20; ++i) put(b, cursor++, 0, 1); + for (std::size_t i = 0; i < 20; ++i) { put(b, cursor, 0, 1); ++cursor; } + trace("imports", b); // Delay-import descriptors (32 bytes each). grAttrs bit 0 = the fields // are RVAs; without it a descriptor is the pre-VC7 address form and must // be skipped rather than misread. - cursor = (cursor + 15) & ~std::size_t{15}; + cursor = (cursor + 15) & ~static_cast(15); const std::size_t delayAt = cursor; - for (auto rva : delayNameRvas) { - put(b, cursor + 0, 1, 4); // grAttrs = dlattrRva - put(b, cursor + 4, rva, 4); // rvaDLLName + for (std::size_t k = 0; k < delayNameRvas.size(); ++k) { + put(b, cursor + 0, 1, 4); // grAttrs = dlattrRva + put(b, cursor + 4, delayNameRvas[k], 4); // rvaDLLName cursor += 32; } - for (int i = 0; i < 32; ++i) put(b, cursor++, 0, 1); + for (std::size_t i = 0; i < 32; ++i) { put(b, cursor, 0, 1); ++cursor; } + trace("delay", b); if (!imports.empty()) { - put(b, dirsAt + 1 * 8, rva_of(importAt), 4); + put(b, dirsAt + 1 * 8, + static_cast(kSecVa + (importAt - kRawAt)), 4); put(b, dirsAt + 1 * 8 + 4, 20 * (imports.size() + 1), 4); } if (!delayImports.empty()) { - put(b, dirsAt + 13 * 8, rva_of(delayAt), 4); + put(b, dirsAt + 13 * 8, + static_cast(kSecVa + (delayAt - kRawAt)), 4); put(b, dirsAt + 13 * 8 + 4, 32 * (delayImports.size() + 1), 4); } + trace("directories", b); return b; } +// The three names every PE test below builds an image around. A function, not +// a namespace-scope constant: a `std::vector` at namespace scope +// in a test TU is a static initializer, and this file is already investigating +// one platform-specific crash. +std::vector pe_imports() { return {"KERNEL32.dll", "vcruntime140.dll"}; } +std::vector pe_delayed() { return {"dbghelp.dll"}; } + struct TempFile { std::filesystem::path path; TempFile(std::string_view tag, std::string_view bytes) @@ -235,9 +270,7 @@ TEST(PackBinfmt, AnElfWithNoDynamicSectionHasZeroDepsAndIsNotAnError) { // A test that cannot localise its own failure is a test that costs a CI round // per hypothesis. TEST(PackBinfmt, ThePeFixtureItselfIsWellFormed) { - std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; - std::array delayed{"dbghelp.dll"}; - auto bytes = pe_with_imports(imports, delayed); + auto bytes = pe_with_imports(pe_imports(), pe_delayed()); ASSERT_GT(bytes.size(), 0x400u); EXPECT_EQ(bytes.substr(0, 2), "MZ"); EXPECT_EQ(bytes.substr(0x40, 4), std::string("PE\0\0", 4)); @@ -248,9 +281,7 @@ TEST(PackBinfmt, ThePeFixtureItselfIsWellFormed) { } TEST(PackBinfmt, IdentifiesPe) { - std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; - std::array delayed{"dbghelp.dll"}; - TempFile f{"peid", pe_with_imports(imports, delayed)}; + TempFile f{"peid", pe_with_imports(pe_imports(), pe_delayed())}; auto id = bf::identify(f.path); EXPECT_EQ(id.format, bf::Format::Pe); @@ -259,9 +290,7 @@ TEST(PackBinfmt, IdentifiesPe) { } TEST(PackBinfmt, ReadsBothPeImportDirectories) { - std::array imports{"KERNEL32.dll", "vcruntime140.dll"}; - std::array delayed{"dbghelp.dll"}; - TempFile f{"peimp", pe_with_imports(imports, delayed)}; + TempFile f{"peimp", pe_with_imports(pe_imports(), pe_delayed())}; auto names = bf::needed_names(f.path); ASSERT_TRUE(names.has_value()) << names.error(); @@ -292,8 +321,7 @@ TEST(PackBinfmt, TruncatedInputIsRejectedRatherThanRead) { // Malformed input is ordinary: a half-downloaded file, a text file named // `.exe`. Every read is bounds-checked, so the parser is total over it. for (std::size_t keep : {0u, 4u, 0x40u, 0x80u}) { - std::array imports{"KERNEL32.dll"}; - auto bytes = pe_with_imports(imports, {}); + auto bytes = pe_with_imports({"KERNEL32.dll"}, {}); bytes.resize(keep); TempFile f{"trunc", bytes}; auto names = bf::needed_names(f.path); From b8a9676e0c7d63ec2731f8719e0301d874d5041a Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:28:54 +0800 Subject: [PATCH 09/10] fix(pack): `--mode static` must not override a target the user asked for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `--mode static` on its own has always meant "the musl-static ELF", and that is unchanged. But the re-prepare that enforces it ignored `opts.targetTriple`, so `--mode static --target x86_64-windows-gnu` silently produced a LINUX build. That was invisible while PE packaging did not exist — there was no Windows package to notice was missing — and it is a wrong answer now that there is. Measured both ways: with an explicit target the output is `…-x86_64-w64-mingw32-static.zip`; without one it is still `…-x86_64-linux-musl-static.tar.gz`. --- src/pack/pipeline.cppm | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 9767b720..8dad8f23 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -57,7 +57,15 @@ export int build_and_pack(Options opts, bool modeFromUser) { // Re-derive target triple: if mode is Static we force the musl // triple even when the manifest's [pack].default_mode bumped us // here after `prepare_build` ran with the host toolchain. - if (opts.mode == mcpp::pack::Mode::Static && ctx->tc.targetTriple.find("-musl") == std::string::npos) { + // + // ...but NOT over a target the user asked for. `--mode static` on its own + // has always meant "the musl-static ELF", and that stays; `--mode static + // --target x86_64-windows-gnu` used to silently become a Linux build, + // which was invisible while PE packaging did not exist and is a wrong + // answer now that it does. An explicit `--target` is an instruction. + if (opts.mode == mcpp::pack::Mode::Static + && opts.targetTriple.empty() + && ctx->tc.targetTriple.find("-musl") == std::string::npos) { // Need to re-prepare the build with the musl target. mcpp::build::BuildOverrides ov2; ov2.target_triple = "x86_64-linux-musl"; From 9edb15ebd8d95ec27688d62d1c5fbabef30649dc Mon Sep 17 00:00:00 2001 From: speak-agent <248744407+speak-agent@users.noreply.github.com> Date: Mon, 17 Aug 2026 05:39:52 +0800 Subject: [PATCH 10/10] test(pack): the PE fixture trace is opt-in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It did its job — the plain rewrite is green on macOS ARM64 — and 24 lines of stderr in every CI run forever is a poor trade for a crash that is currently fixed. `MCPP_TEST_TRACE=1` brings it back, which keeps a recurrence one CI round to localise instead of the four this one cost. --- tests/unit/test_pack_binfmt.cpp | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/tests/unit/test_pack_binfmt.cpp b/tests/unit/test_pack_binfmt.cpp index 9893921a..da365372 100644 --- a/tests/unit/test_pack_binfmt.cpp +++ b/tests/unit/test_pack_binfmt.cpp @@ -1,5 +1,6 @@ #include #include // stderr is a MACRO — `import std;` cannot export it +#include // getenv import std; import mcpp.pack.binfmt; @@ -116,7 +117,13 @@ std::string elf_with_needed(std::span needed) { // test proved the crash is HERE, in fixture code that touches no module at // all, so the shapes went and the trace stayed: if it moves again, the log // says which step. +// Off by default — 24 lines of stderr in every CI run forever is a poor +// trade for a crash that is currently fixed. `MCPP_TEST_TRACE=1` brings it +// back, which is what makes a recurrence one CI round to localise instead of +// the four this one cost. void trace(const char* step, const std::string& b) { + static const bool on = std::getenv("MCPP_TEST_TRACE") != nullptr; + if (!on) return; std::fprintf(stderr, "[pe-fixture] %-14s size=%zu\n", step, b.size()); std::fflush(stderr); }