diff --git a/.agents/docs/2026-05-19-pack-windows-design.md b/.agents/docs/2026-05-19-pack-windows-design.md index 14544d03..093f31a5 100644 --- a/.agents/docs/2026-05-19-pack-windows-design.md +++ b/.agents/docs/2026-05-19-pack-windows-design.md @@ -1,7 +1,37 @@ # Windows Pack Design **Date:** 2026-05-19 -**Status:** Planned (stub guard in place, implementation not yet started) +**Status:** SUPERSEDED and implemented, 2026-08-17. See +`2026-08-16-windows-toolchain-three-axes-design.md` §4 for the design that +shipped, and `src/pack/binfmt.cppm` / `src/pack/zip.cppm` for the code. + +> ## What this document got wrong, and it is worth keeping +> +> Everything below assumes **pack runs on Windows**. That assumption is +> visible in every proposal: `dumpbin /dependents`, `ImageNtHeader` from +> ``, PowerShell's `Compress-Archive`, and an implementation +> "under `#if defined(_WIN32)`". +> +> The assumption came from the guard it was trying to remove. `pack` refused +> Windows, so the problem looked like "pack has no Windows branch". It was +> not: the ELF closure is derived by RUNNING the artifact +> (`LD_TRACE_LOADED_OBJECTS=1 ''`), so it can cross neither an OS nor +> an ARCHITECTURE — a Linux box cannot trace a PE, and an x86_64 box cannot +> trace an aarch64 ELF either. The `#if defined(_WIN32)` was that limitation +> surfacing at the nearest place a user would hit it. +> +> Reading the import table statically removes both limits at once, and then +> cross-OS packaging is not a feature that had to be added — it is what +> remains when the obstacle is gone. Each Windows-only tool above would have +> reintroduced the obstacle one layer down. +> +> What did survive from here: the `.zip` output, the flat +> DLLs-beside-the-`.exe` layout, no wrapper script, and the skip-list concept +> (with one correction — `vcruntime*.dll` is listed below as a system DLL, and +> it is not: it belongs to the TOOLSET, and whether it travels is +> `cxx_runtime`'s decision). + +--- ## Current state diff --git a/.agents/docs/2026-08-16-toolchain-architecture-review.md b/.agents/docs/2026-08-16-toolchain-architecture-review.md index 6ed89763..300a49cb 100644 --- a/.agents/docs/2026-08-16-toolchain-architecture-review.md +++ b/.agents/docs/2026-08-16-toolchain-architecture-review.md @@ -8,6 +8,12 @@ > 三处"不可能失败的测试"。下面每一条都指得出具体的 `file:line`, > 不是风格偏好。 +> ⚠️ **§1 / §2 / §3 / §3d 已被方案取代**: +> `2026-08-16-windows-toolchain-three-axes-design.md`。 +> 那份追问推翻了这里两处形状 —— `gcc@system` 的推广方向、以及新加 +> `windows_sdk` manifest 键 —— 两处都在方案的 §0.2 里写明了为什么错。 +> 本文其余各节(§3b / §3c / §4 / §5 / §6 / §7)已落地,保留作记录。 + --- ## 0. 结论先行 diff --git a/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md new file mode 100644 index 00000000..bbccc394 --- /dev/null +++ b/.agents/docs/2026-08-16-windows-toolchain-three-axes-design.md @@ -0,0 +1,366 @@ +# Windows 工具链的三条轴:来源、SDK、运行时(2026-08-16) + +> 取代 `2026-08-16-toolchain-architecture-review.md` 的 §1 / §2 / §3 / §3d。 +> 那份是**发现**(问题在哪、file:line),这份是**方案**(怎么改、怎么算通过)。 +> review 的 §3b / §3c / §4 / §5 / §6 / §7 已落地,不在此文范围。 +> +> 起因是 review 之后的一轮追问,它推翻了我原方案里的两处形状错误 —— +> 两处都记在 §0.2,因为**错的那版看起来同样合理**,不写下来会被重新提出来。 + +--- + +## 0. 先把三样东西分开 + +### 0.1 今天被混在一起的三层 + +Windows 上一次构建牵涉三个**互相独立**的问题,而现在它们纠缠在一起: + +| 轴 | 问的是什么 | 今天在哪 | 状态 | +|---|---|---|---| +| **来源** | 编译器**哪来的**:机器上的 VS,还是 xim 装的 payload | `is_system_toolchain()` `registry.cppm:428` | ✅ 已建模,但代价摊在 26 处 | +| **SDK** | 用**哪套**头/导入库/工具 | `find_windows_sdk()` `msvc.cppm:628` | ❌ **靠搜**,没有身份 | +| **运行时分发** | 产物**带不带** `vcruntime140.dll` | `distribution.cppm:432` | ❌ 对 MSVC **一律拒绝** | + +**判据:三个问题的答案可以两两独立取值。** 受管 toolset 可以配系统 SDK(今天 xrgui CI 就是), +系统 VS 也带自己的 Redist。任何把它们绑成一个开关的设计都是错的。 + +### 0.2 两处已被推翻的形状(不要重新提出来) + +**(a) 不要把 `@system` 推广到所有族。** + +初稿把 `is_system_toolchain()` 里的 `Family::Msvc` 读成"不对称,应该补齐",提议 +支持 `gcc@system`。**方向反了**:xlings 是用户态 OS,mcpp 建立在它之上,整条设计 +就是**把 host 依赖降到最低**。`msvc@system` 是 **Windows 的让步** +(Visual Studio 常常已经装了、又不总能重新分发),不是一种缺失的通用能力。 + +顺带记一笔:真正提供 host 依赖的那一处,拼写是**不带族的 `system`** +(`prepare.cppm` 里 `*tcSpec == "system"` 那条 `// Explicit user opt-in to system +PATH compiler — kept as escape hatch`)。别再把它当成"缺失的能力"。 + +**(b) 不要新加 `windows_sdk = "..."` 这个 manifest 键。** + +理由有两层: + +1. **模型里已经有位置了。** `runtime_binding.cppm:26` 的 `runtimeId` 注释原文就是 + *"Provider-native runtime identity (`glibc@...`, `macos_sdk@...`, `ucrt@...`)"* + —— `ucrt@...` **早就写着**,只是全仓库没有一处填它,而消费者硬编码 `glibc@` 前缀 + (`runtime_validation.cppm:387`)。 +2. **它和 `_WIN32_WINNT` 会变成两个看起来能互相替代、实际管不同事的旋钮**,见 §2.4。 + +--- + +## 1. 轴一:来源 —— 收拢特例,不是推广它 + +### 1.1 问题 + +`is_system_toolchain()` 的判据本身没问题。问题是**这个平台特例的代价被摊开了**: +lifecycle.cppm **13 处**、prepare.cppm **13 处** msvc 专有分支,而 gcc / llvm / mingw +在 lifecycle.cppm 里**一处都没有**(mingw 更是零分支 —— 它整个被表达成一个 +*target*,`x86_64-windows-gnu`,那才是这套设计想要的形状)。 + +另外三处是**同一条规则的重复拼写**,与特例本身无关: + +1. `xim_tool()` + `installation_at()` + 错误信息 —— prepare 与 lifecycle 几乎逐字重复 +2. sysroot 依赖规则两份且**不等价**(一份只看 musl,一份 musl + PE + 宿主), + 而注释声称它们互相对应 +3. 解析链的注释说 4 步,实际有 **9 个输入** + +### 1.2 方案 + +把"来源"变成一个**解析一次**的值,而不是每个子命令自己再问一遍: + +```cpp +enum class Origin { Managed, SystemMsvc }; // 只有两种,而且只会有两种 +``` + +- `prepare` / `lifecycle` 各在入口解析一次,之后按它分发 +- `parse_toolchain_spec` 对**非 msvc 的 `@system`** 显式报错并指出替代写法 + (今天是"未实现",于是失败发生在别处、消息说的是别的事) + +配套消掉上面三处重复:提 `resolve_managed_msvc(env, pkg)`;合并 sysroot 谓词; +把解析顺序变成一张**数据表**,让注释无处可撒谎。 + +### 1.3 边界 + +- **裸 `gcc` 的语义不能变**(那会改现有 manifest 的行为)。只有显式 `@system` 是新语义 +- 裸 `system`(不带族)那条逃生口**保留**,它是有意的 + +--- + +## 2. 轴二:SDK —— 绑定,而不是搜索 + +### 2.1 问题 + +编译器拿到了版本轴,**SDK 没有**。`find_windows_sdk()` 按顺序扫: +`WindowsSdkDir` → 兄弟 xpkgs → 写死的 `C:\Program Files (x86)\Windows Kits\10`。 + +但对 `msvc@`,SDK 是 recipe 里**声明过的依赖** +(`xim:windows-sdk@10.0.26100`),位置是确定的。把确定的东西拿去搜,三个后果: + +1. **环境能悄悄改写钉住的选择** —— 一个游离的 vcvars 设了 `WindowsSdkDir`, + manifest 钉的 SDK 就失效了 +2. **今天那个 LNK1104 就是搜出来的** —— 半装的 payload 因为版本号更高, + 排在机器自己那套完整 SDK 前面,每个 TU 都编过,最后炸在链接, + **日志里没有一行提到 SDK** +3. **两台机器仍可能用不同 SDK 编同一份源码** —— 正是版本轴当初要解决的那个问题, + 只是换到了下一层 + +### 2.2 方案:按来源分流 + +``` +msvc@ → SDK 从同一条 xim 解析拿(和编译器同一个机制) +msvc@system → 维持现在的搜索链(机器上的东西只能靠找) +``` + +受管那条不再"搜到就用",而是**问 toolset 声明了哪个 SDK 依赖**。搜索链降级为 +`msvc@system` 专用。 + +**`WindowsSdkDir` 的地位要一起想清楚。** 它是"有人明确指定",按"声明压过探测" +应当优先;但对受管 toolset 它又会破坏可复现性。建议: + +- `msvc@system`:`WindowsSdkDir` 优先(维持今天) +- `msvc@`:**忽略它,但要说出来** —— 一行 `note:`,别静默 + +理由是这条轴的全部意义就是"声明什么就用什么",而静默覆盖会让它失效且不可见。 + +### 2.3 运行时身份:填那个留了很久的槽 + +`runtimeId = "ucrt@10.0.26100"`,从 toolset 的 SDK 依赖投影而来,进 +`runtimeContractHash`。 + +**但它和 glibc 不完全同构,这个差别必须写进代码注释**: + +| | Linux | Windows | +|---|---|---| +| `glibc@2.39` 绑的是 | 一个 **payload**:头 + `.so` 都在里面 | — | +| 能不能真绑上去 | ✅ patchelf 让产物**真的跑在那份 glibc 上** | ❌ `ucrtbase.dll` 在系统里,**换不掉也不该换** | +| 于是标识的含义 | **运行时绑定** | **兼容性下限声明** | + +我们的 `windows-sdk` 包**只装了 ucrt 的一半**(头 + 导入库), +**没有** `Universal CRT Redistributable`,因为 Win10 起 `ucrtbase.dll` 是系统组件、 +应用不该自带。这是**有意的**,要写进注释,否则下一个人会以为能像 glibc 那样 +把 ucrt 打进产物。 + +配套:`runtime_validation.cppm:387` 与 `post_install.cppm` 里硬编码的 `glibc@` 前缀 +要从"只认 glibc"改成"按 provider 分派",否则填了也没人认。 + +### 2.4 **不要**加 manifest 的 SDK 版本键 + +一般 Windows 开发者怎么处理 SDK: + +| 做法 | 普遍程度 | 后果 | +|---|---|---| +| VS 装一个"最新",`10.0`(字面意思就是"本机最新") | 绝大多数 | **同一份源码在两台机器上链到不同 SDK,且无提示** | +| 钉死 `10.0.22621.0` | 有纪律的团队 | 声明路线 | +| xwin / msvc-wine 自己下 payload | Rust 圈 / 交叉编译 | 本质就是我们做的事 | + +**默认(跟着 toolset 依赖走)已经强过主流做法** —— 它至少保证每台机器一致, +而主流做法连这一点都不保证。 + +差别到底在哪,分两半: + +| | 影响 | 为什么 | +|---|---|---| +| **ucrt 那一半** | **很小** | Win10 起 `ucrtbase.dll` 是系统组件且向后兼容 | +| **um(Win32 API)那一半** | **会咬人** | 新 API 只存在于新 SDK 的头/导入库 | + +两种失败难度天差地别: + +- 链到**老** SDK 用了新 API → **编译期**失败,一眼看出 ✅ +- 链到**新** SDK 但跑在老 Windows → **运行期**失败(加载失败 / `GetProcAddress` 返回 null)❌ + +**而真正声明"我承诺跑在哪个 Windows"的是 `_WIN32_WINNT` / `WINVER` 宏,不是 SDK 版本。** +SDK 版本决定的是"我能看见哪些 API"。加一个 `windows_sdk =` 键会让用户以为它就是 +兼容性下限旋钮 —— **两个旋钮看起来能互相替代,实际管不同事**,这是比不做更糟的结果。 + +结论:**这一条不做**。真有少数场景要钉,先把 `_WIN32_WINNT` 的表达方式想清楚, +再决定要不要第二个旋钮。 + +--- + +## 3. 轴三:运行时分发 —— `toolchain-coupled` 对 MSVC 是成立的 + +### 3.1 问题 + +`distribution.cppm:432` 说: + +> `cxx_runtime = "toolchain-coupled"` has no meaning for the MSVC runtime +> (it ships with the OS/redistributable, **not with the toolchain**) + +这句话对 `msvc@system` 是对的,对 `msvc@` 是**错的** —— 受管 payload 里就有 +`VC/Redist/MSVC///Microsoft.VC143.CRT/{vcruntime140,msvcp140,...}.dll`, +而且 VS 自身安装也带同一个目录,所以**两条来源都能支持**。 + +于是分发矩阵上有个洞: + +| | 自包含 | 宿主耦合 | 工具链耦合 | +|---|---|---|---| +| gcc / libstdc++ | 静态链接 | 用系统 libstdc++ | **拷 libstdc++ 到产物旁** ✅ | +| MSVC | `/MT` ✅ | `/MD` ✅ | **拒绝** ❌ | + +### 3.2 已落地的一半 + +`vc_redist_dir()` 已经把 toolset 自带的那份 CRT 放进 `linkRuntimeDirs`, +Windows 上它就是 PATH —— 所以 **`mcpp run` 已经能在只有受管 toolset 的干净机器上 +跑起来**(在此之前默认 `/MD` 产物会以"找不到 vcruntime140.dll"失败, +而 CI 看不到,因为 runner 上装着 VS)。 + +### 3.3 还差的一半 + +**把那些 DLL 拷到产物旁**,即 PE 上真正实现 `toolchain-coupled` —— 和 gcc 拷 +libstdc++ 是同一件事、同一个语义。 + +⚠️ **只取 `/Microsoft.VC*.CRT/`,不能取 `debug_nonredist/`** —— +后者(`vcruntime140d.dll` 等)**不可再分发**。这条要有测试钉住, +`vc_redist_dir()` 已经这么做了,拷贝那一步要沿用同一个判据。 + +### 3.4 三层的最终形状 + +``` +ucrt → SDK 给导入库,运行时由 OS 提供 → 下限声明(runtimeId) +vcruntime → toolset 自带 Redist,可随产物走 → cxx_runtime(§3) +um/shared → SDK 给,纯链接期 → 无运行期对应物 +``` + +--- + +## 4. 打包:读二进制,别运行它 + +### 4.1 问题比"没做 Windows"更深 + +`pack.cppm:323` 的 `ldd_parse` 求依赖闭包的方式是: + +``` +LD_TRACE_LOADED_OBJECTS=1 '' +``` + +**它要把目标二进制跑起来。** 所以这不是"缺一个 PE 分支": + +- **跨不了 OS**(Linux 上跑不了 PE) +- **跨不了架构**(x86_64 上跑不了 aarch64 产物,即便同是 Linux) + +`pack.cppm:638` 那条 `#if defined(_WIN32)` 的拒绝只是结果,不是原因。 + +### 4.2 分层 + +| 层 | 职责 | 平台相关? | +|---|---|---| +| **契约** `distribution.cppm` | 决定**该带什么** | ❌ **已存在** | +| **闭包** | 静态读导入表:ELF `DT_NEEDED` / PE 导入表 / Mach-O `LC_LOAD_DYLIB` | ✅ 纯解析 | +| **系统库判据** | ELF 现有那张表 / PE 的 kernel32、user32、ucrtbase… / Mach-O `/usr/lib`、`/System` | ✅ | +| **重定位** | ELF: patchelf `$ORIGIN/../lib` / **PE: 无操作**(DLL 放 exe 旁就是规则) / Mach-O: `install_name_tool` + `@loader_path` | ✅ | +| **打包** | tar.gz / zip | ✅ | + +**换成静态解析之后,跨 OS 打包不是额外功能,是自然结果** —— 没有任何一步需要执行产物。 + +### 4.3 还有一处断链 + +**`pack` 从不读 Contract。** `MechanismInput.msvcStaticCrt` 只影响编译链接**旗标**; +`pack` 决定真拷哪些文件,却不认识 `cxx_runtime`。所以那个契约在打包这一步 +**没有执行者** —— ELF 上靠 `ldd` 闭包歪打正着,PE 上完全没有。 + +把第一层接到第二、四层,才是"一套系统"而不是"两套"。矛盾时要报出来 +(`pack` 已有"带自己 libc 的包不能消费宿主库"这类拒绝的先例,照着长)。 + +--- + +## 5. 兼容性与迁移 + +| 改动 | 是否改用户可见格式 | 迁移 | +|---|---|---| +| §1 Origin 收拢 | ❌ 纯内部 | 无 | +| §1 拒绝 `gcc@system` | ⚠️ 之前"未实现"、现在显式报错 | 消息要给替代写法 | +| §2 SDK 绑定 | ❌ 行为变更,非格式 | 受管 toolset 上 `WindowsSdkDir` 从"生效"变"忽略+提示" | +| §2 `ucrt@` 身份 | ⚠️ 进 `runtimeContractHash` → **缓存键变** | 一次性全量重建,和别的 contract 变更同类 | +| §3 PE toolchain-coupled | ❌ 新增能力 | 之前是 degraded,现在能满足 | +| §4 PE pack | ❌ 新增能力 | 之前是硬错误 | + +**唯一需要留意的是 §2 的 hash 变更** —— 它会让 Windows 上已有的构建缓存失效一次。 + +--- + +## 6. 验收判据(每条怎么算通过) + +**没有一条可以靠"CI 绿了"算过。** 这一轮十一层缺陷,每一层出现时 CI 都是绿的。 + +| # | 判据 | 为什么它不能被自我满足 | +|---|---|---| +| §1 | 26 处分支的数量**下降**,且 `gcc@system` 报错里出现替代写法 | 数量可数;消息内容有断言 | +| §2 | 受管 toolset 下,**设一个指向别处的 `WindowsSdkDir`,构建仍用 payload 的 SDK**,且打印了 note | 环境能覆盖=没绑定,这条直接证伪 | +| §2 | `runtimeContractHash` 在 Windows 上随 SDK 版本变化 | 不变就是没进 hash | +| §3 | **在一台没有 VS、没装过 redist 的干净 Windows 上**,`/MD` 产物拷到别处能跑起来 | CI runner 有 VS,**必须用干净机器或容器**,否则这条测不出东西 | +| §3 | 产物旁**没有** `*d.dll` | debug CRT 不可再分发 | +| §4 | **在 Linux 上**为 Windows 产物打出 zip,内含正确 DLL 闭包 | 跨 OS 是这条的全部意义;同 OS 打包证明不了 | +| §4 | `cxx_runtime` 与 `--mode` 矛盾时**报出来** | 静默通过=契约无执行者 | + +> §3 和 §4 的判据都要求**比 CI 环境更贫瘠的机器**。这一轮反复出现的形状是 +> "验收环境比目标环境富裕",这两条是直接针对它写的。 + +--- + +## 6.5 落地状态(2026-08-17,mcpp 2026.8.17.1) + +**§1 / §2 / §3 / §4 全部实现,§2.4 明确不做。** 逐条对应: + +| 条目 | 状态 | 落点 | +|---|---|---| +| §1 Origin 建模 | ✅ | `enum class Origin`(`model.cppm`)—— 放在数据模型里,让 spec 侧(`registry`)与已定位编译器侧(`msvc`)指的是**同一根轴** | +| §1 拒绝 `gcc@system` | ✅ | `parse_toolchain_spec`,错误里同时给出 pin 与 PATH 逃生口两种写法 | +| §1 解析一次 | ✅ | `prepare` 原先把同一个字符串**解析两遍**、各自下结论;现在一次,`origin_of()` 分发 | +| §1 消重复:`resolve_managed_msvc` | ✅ | `registry.cppm`。"payload 在哪、为什么不能用 fetcher 的 `root`"两份手写实现,而理由只写在其中一份里 | +| §1 消重复:sysroot 谓词 | ✅ | `needs_linux_sysroot_payloads()`。两份**不等价**,而其中一份的注释声称它们互为镜像 —— 少的是 PE 那一项 | +| §1 解析链注释 | ✅ | 一张按 `TcOrigin` 枚举名写的表。原先两处、分别声称 3 步和 4 步,合起来点到 9 个输入里的 5 个,还互相矛盾 | +| §2.2 SDK 按来源分流 | ✅ | `msvc::resolve_sdk_for()`;受管来源忽略 `WindowsSdkDir`/`WindowsSdkVersion` 并打印 `note:` | +| §2.2 无 SDK payload 时 | ✅ | 退回机器 SDK 并**说出来**(可用 > 失败,但不可复现这件事必须留痕) | +| §2.3 `ucrt@` 身份 | ✅ | `bind_windows_ucrt()`,进 `runtimeContractHash`;`runtime_provider()` 取代各处 `starts_with("glibc@")` | +| §2.4 manifest SDK 键 | ⛔ **不做** | 理由见该节 | +| §3.3 PE `toolchain-coupled` | ✅ | `dist::Mechanism::deployToolchainRuntime` → `CompileFlags::toolchainRuntimeDeploy` → ninja `stage_file` 边 | +| §3.3 排除 debug CRT | ✅ | 判据只有一处(`vc_redist_dir()`),拷贝那步复用它而不是另立一条按名字的规则 | +| §4 静态读闭包 | ✅ | `mcpp.pack.binfmt`:ELF `DT_NEEDED`、PE 导入表 **+ 延迟导入表** | +| §4 跨 OS 打包 | ✅ | Linux 上给 Windows 产物打 zip;`mcpp.pack.zip` 自己写压缩包(没有哪个 zip 工具在每个宿主上都存在) | +| §4.3 `pack` 读 Contract | ✅ | `cxx_runtime` 决定 toolchain runtime 目录进不进搜索集;与 `--mode` 矛盾时拒绝 | + +**两处对方案本身的修正**,来自实现过程: + +1. **`dist::Format` 改为先看 target triple、再退回宿主。** 原先除 MinGW 外一律 + 问宿主,于是"Windows 上的契约"在 Linux runner 上**根本无法断言** —— 而这轮 + 大部分 review 就发生在 Linux runner 上。只**新增**答案:说不出 OS 的 triple + 仍走原来那条推导,现有构建一个都不变。 +2. **`force_bundle` 在 PE 上也要能覆盖系统表。** ELF 上一直如此;PE 上第一版把 + 系统排除写成了无条件的,那会让一个明确写下来的决定变成装饰。 + +**§4 未做的一半,以及理由。** ELF 闭包**仍然**通过运行产物取得 +(`LD_TRACE_LOADED_OBJECTS`),`binfmt` 的 `DT_NEEDED` 读取只用于识别与诊断。 +原因不是懒:`ldd` 交回的是**已解析的路径**,而 `DT_NEEDED` 只有名字,把名字变 +成路径要重新实现 loader 的搜索规则(`DT_RPATH` → `LD_LIBRARY_PATH` → +`DT_RUNPATH` → `ld.so.cache` → 默认目录,加上 `$ORIGIN` 展开与 hwcaps 子目录)。 +在一条**已经正确、且被 e2e 覆盖**的路径上重写这个,风险远大于收益。 +**代价要说清楚:跨架构的 ELF 打包(x86_64 上给 aarch64 产物打包)仍然不支持**, +而这正是 §4.1 指出的第二个限制。PE 那条没有既有实现,所以它从一开始就是静态的。 + +--- + +## 7. 落地顺序 + +| # | 项 | 规模 | 依赖 | +|---|---|---|---| +| 1 | §3.3 PE `toolchain-coupled`(拷 DLL) | 中 | 无。`vc_redist_dir()` 已在 | +| 2 | §2.2 SDK 按来源分流 | 中 | 无 | +| 3 | §2.3 `ucrt@` 身份 + 解开 `glibc@` 硬编码 | 中 | 建议紧跟 2 | +| 4 | §1 Origin 收拢 + 消三处重复 | 大 | 建议在 2/3 之后 —— 都动同一片代码 | +| 5 | §4 PE pack | 大 | 排在 1 之后:先解决"能跑",再解决"能分发" | + +**§2.4(manifest SDK 版本键)不做**,理由见该节。 + +前三项互不依赖,可并行。§1 和 §5 各自单独一轮。 + +--- + +## 8. 一句话 + +这三条轴今天被搅在一起,是因为**只有编译器那一条被显式建模过**。 +SDK 靠搜、运行时靠拒绝、打包靠一个要跑起来才work的原语 —— +每一处都在"环境恰好合适"时正常工作,而在贫瘠环境里失败, +**且失败信息不指向真正的原因**。 + +把三条轴各自变成一个**声明出来的、解析一次的值**,是这份方案唯一在做的事。 diff --git a/.github/workflows/cross-build-test.yml b/.github/workflows/cross-build-test.yml index 68a072ff..379def22 100644 --- a/.github/workflows/cross-build-test.yml +++ b/.github/workflows/cross-build-test.yml @@ -308,6 +308,18 @@ jobs: export MCPP_VENDORED_XLINGS="$XLINGS_BIN" bash tests/e2e/198_windows_resources_cross.sh + # Packaging a Windows program FROM LINUX — and this job is the only + # place that can happen, for the same reason as the two above. + # + # Running it on a Windows runner would prove nothing: the point of + # reading the import table instead of executing the artifact + # (mcpp.pack.binfmt) is precisely that the packaging host need not be + # the target. A same-OS pack cannot tell the two implementations apart. + - name: "e2e: pack a PE from Linux (zip + DLL closure)" + run: | + export MCPP_VENDORED_XLINGS="$XLINGS_BIN" + bash tests/e2e/240_pack_pe_zip_cross.sh + # ── windows → linux ─────────────────────────────────────────────────────── # The mirror of mingw-cross-wine. Two jobs because a Windows runner cannot # execute the ELF it produces; the artefact is handed to a Linux job and diff --git a/docs/02-pack-and-release.md b/docs/02-pack-and-release.md index 589785f5..195d7b27 100644 --- a/docs/02-pack-and-release.md +++ b/docs/02-pack-and-release.md @@ -236,6 +236,72 @@ own resolution, say — use `--mode vendored` instead. It repoints `PT_INTERP` at the host loader, at the cost of requiring the host's glibc to be at least as new as the one the artifact was built against. +### Windows (PE) — a `.zip`, and the DLLs sit beside the `.exe` + +A Windows target produces a **`.zip`**, not a `.tar.gz`, and the layout is +flat: + +``` +target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip +└── myapp-0.1.0-x86_64-pc-windows-msvc/ + ├── myapp.exe + ├── vcruntime140.dll ← only under cxx_runtime = "toolchain-coupled" + ├── mydep.dll ← third-party dependencies + ├── README.md + └── LICENSE +``` + +There is no `bin/` + `lib/` split and no entry-point wrapper, and neither is a +style choice. The Win32 loader resolves a DLL from **the directory of the +executable**; PE has no `RUNPATH` to point anywhere else, so "next to the +`.exe`" *is* the mechanism that `$ORIGIN/../lib` provides on ELF. + +**Windows' own DLLs are never bundled** — `kernel32.dll`, `ntdll.dll`, +`ucrtbase.dll`, the `api-ms-win-*` API sets. Shipping a private copy of an OS +component is a broken program rather than a heavier one (the process ends up +with two of something that must be unique), and Microsoft's redistribution +terms say the same thing from the other side. `[pack.bundle-project] +force_bundle` still overrides this, as it does the ELF skip list. + +`vcruntime140.dll` and `msvcp140.dll` are **not** Windows' own: they belong to +the MSVC toolset, exactly as `libstdc++.so` belongs to gcc. Whether they +travel is decided by `cxx_runtime` (see `docs/05-mcpp-toml.md`), not by this +list — and `mcpp pack` refuses a combination that cannot deliver what the +contract promised: + +``` +$ mcpp pack --mode system # with cxx_runtime = "toolchain-coupled" +error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other. +``` + +#### Packing a Windows program from Linux or macOS + +This works, and it is not a special mode — just build for a Windows target and +pack: + +```bash +mcpp pack --target x86_64-windows-gnu # from a Linux host +``` + +`mcpp pack` used to refuse Windows outright. The reason was not the archiver: +the ELF dependency closure is obtained by **running the artifact** under +`LD_TRACE_LOADED_OBJECTS`, which cannot cross an OS *or* an architecture. A PE +closure is read out of the file's import table instead, so nothing has to be +executed and the packaging host is free. The archive is written by mcpp itself +for the same reason — there is no zip tool present on every host. + +Two consequences worth knowing: + +- Entries are **stored, not deflated**, so a Windows package is roughly the + size of its contents. Compression is a size optimization, not a correctness + one, and it is not implemented yet. +- The archive is **deterministic**: no timestamps are read, so two packs of + the same tree are byte-identical and a published checksum means something. + +The reverse direction — packing a Linux or macOS artifact *from* Windows — +still does not work, and for the original reason: that closure is resolved by +the target's own dynamic linker, which a Windows host has no way to run. + ## Configuration Packaging behavior is configured via the `[pack]` section in `mcpp.toml`. The diff --git a/docs/03-toolchains.md b/docs/03-toolchains.md index c6be373c..289aa58f 100644 --- a/docs/03-toolchains.md +++ b/docs/03-toolchains.md @@ -228,6 +228,17 @@ questions. `msvc@system` asks *"use what this developer already has"*; `msvc@14.44.35207` asks *"build this project with exactly this compiler"*. Pinned toolsets coexist with each other and with a system Visual Studio. +> **`@system` is an MSVC-only spelling.** There is no `gcc@system` or +> `llvm@system`, and that is deliberate rather than an omission: mcpp is built +> on xlings, a user-space OS, and the design drives host dependencies to a +> minimum — a toolchain comes from a payload the manifest names, so every +> machine builds with the same compiler. Windows is the one place where +> refusing to use what is already installed would cost more than it buys: +> Visual Studio is very often present and cannot always be redistributed. +> `@system` for any other family is an error that names both things +> you might have meant. (The family-less `[toolchain] … = "system"` — the PATH +> compiler — is a separate and deliberate escape hatch, and is unaffected.) + ### `msvc@system` — the machine's own Visual Studio mcpp locates and identifies an installed Visual Studio / Build Tools; it never @@ -304,11 +315,23 @@ environment from the VC tools + Windows SDK (no `vcvarsall` involved), stages `/interface /TP /ifcOutput`, scans with `/scanDependencies`, and links with `link.exe`/`lib.exe` through response files. -The Windows SDK is located in this order: **`WindowsSdkDir`** (+ -`WindowsSdkVersion`) if declared, then the `xim:windows-sdk` payload beside a -pinned toolset in mcpp's store, then `C:\Program Files (x86)\Windows Kits\10`. -A missing SDK fails the build with guidance (`mcpp self doctor` reports SDK -status). +**The Windows SDK follows the origin**, because the two origins answer +different questions and so must the SDK: + +| origin | how the SDK is chosen | +|---|---| +| `msvc@` | the `xim:windows-sdk` payload installed **with that toolset**, in mcpp's own store. `WindowsSdkDir` / `WindowsSdkVersion` in the environment are **ignored**, and mcpp prints a `note:` saying so. | +| `msvc@system` | **`WindowsSdkDir`** (+ `WindowsSdkVersion`) if declared, then `C:\Program Files (x86)\Windows Kits\10`. | + +The asymmetry is the point. A pinned toolset is a promise that two machines +compile the same source against the same headers; an environment variable that +can quietly redirect it turns the pin into a preference. A machine's own SDK, +on the other hand, can only be found by looking, and there a declared answer +outranks a scan — the same precedence `VSINSTALLDIR` has over `vswhere`. + +If a pinned toolset has no SDK payload beside it (an older install, say), mcpp +falls back to the machine's SDK rather than failing — and says so, because that +build is no longer reproducible and nothing else would record it. A root only counts as an SDK when it has **both** halves — `Include\\ucrt\ corecrt.h` *and* `Lib\\um\\kernel32.lib`. A root with headers and no @@ -316,15 +339,27 @@ import libraries is skipped rather than selected, so a partially unpacked payload cannot outrank the machine's complete SDK and turn into `LNK1104: cannot open file 'kernel32.lib'` at the very end of a build. +The resolved SDK version is part of the build's **runtime identity** +(`ucrt@10.0.26100.0`) and therefore of the fingerprint that keys the build +cache: changing SDK changes the cache key, exactly as changing compiler does. +It is a **compatibility floor declaration**, not a payload binding like +`glibc@2.39` on Linux — `ucrtbase.dll` is a Windows component and mcpp neither +ships nor substitutes it. + **CRT model.** `/MD` (host-coupled) by default; `/MT` when either ```toml +[target.x86_64-windows-msvc] +linkage = "static" # the libc axis — TARGET section, or `--static` + [build] -linkage = "static" # the libc axis cxx_runtime = "self-contained" # the C++ runtime axis ``` -is written down. On the MSVC ABI these are one physical switch — `/MT` links +is written down. Note which section each one lives in: `linkage` is +exact-triple only and **there is no `[build] linkage` key** — writing one gets +an "unsupported key (ignored)" warning and no static CRT. (This page said +exactly that a few sections up, and then showed the wrong form here.) On the MSVC ABI these are one physical switch — `/MT` links the C and C++ runtimes out of the same library — so both spellings select it and mean the same thing. It is a **whole-project** property: one `std` module is built per project and cl bakes `_MSVC_MT`/`_MSVC_MD` into it, so a diff --git a/docs/05-mcpp-toml.md b/docs/05-mcpp-toml.md index 9e025c6f..1d1b6a82 100644 --- a/docs/05-mcpp-toml.md +++ b/docs/05-mcpp-toml.md @@ -395,10 +395,39 @@ default applies only when nobody said anything. `self-contained`, `false` means `host-coupled`. An explicit `cxx_runtime` wins. **A contract that cannot be honored is reported, never silently downgraded.** If a -toolchain ships no `libc++.a`, or a contract has no mechanism on that platform -(`self-contained` under the MSVC runtime would need `/MT`, which mcpp does not emit -yet), the build prints what it fell back to instead of quietly producing a -different artifact than the manifest asked for. +toolchain ships no `libc++.a`, or a contract has no mechanism on that platform, +the build prints what it fell back to instead of quietly producing a different +artifact than the manifest asked for. + +#### On the MSVC runtime + +The CRT model is the mechanism here, and it is a **whole-project** switch: cl +bakes `_MSVC_MT`/`_MSVC_MD` into the one `std` module a project builds, so a +per-role contract that disagrees with the project's cannot be honoured and is +reported rather than ignored. + +| value | what it is on MSVC | +|---|---| +| `self-contained` | `/MT` — the static CRT. `linkage = "static"` selects the same thing from the libc axis. | +| `host-coupled` (default under `/MD`) | the target provides `vcruntime140.dll` / `msvcp140.dll` — i.e. Visual Studio or the redistributable is installed there. | +| `toolchain-coupled` | the toolset's **own** copy of those DLLs travels with the artifact. | + +`toolchain-coupled` is worth spelling out, because the obvious reading is +wrong. `ucrtbase.dll` *is* a Windows component (since Windows 10) and mcpp +never ships it. `vcruntime140.dll` and `msvcp140.dll` are **not**: every MSVC +toolset carries them under `VC\Redist\MSVC\\\`, exactly the +way a gcc payload carries `libstdc++.so`. Under this contract mcpp stages them +beside the artifact — which is what makes a default `/MD` build runnable on a +machine that has only the pinned toolset and no Visual Studio at all. + +The debug CRT (`vcruntime140d.dll` and friends, under `debug_nonredist\`) is +never staged: it may not be redistributed. + +Combining it with `/MT` is a contradiction rather than a missing feature — a +static CRT leaves no DLL to couple to — so it is reported and resolved to +`self-contained`. `mcpp pack` enforces the other half: a mode that bundles +nothing (`--mode system`, `--mode static`) cannot deliver `toolchain-coupled` +and refuses. **Scope.** The contract governs the C++ runtime only. Static **libc** is a separate axis (`linkage = "static"` / `--static`, e.g. a musl target), and the deployment diff --git a/docs/zh/02-pack-and-release.md b/docs/zh/02-pack-and-release.md index e6eaff0a..66adde31 100644 --- a/docs/zh/02-pack-and-release.md +++ b/docs/zh/02-pack-and-release.md @@ -172,6 +172,65 @@ if (!base) { 它把 `PT_INTERP` 重指到宿主 loader,`/proc/self/exe` 正常,代价是要求宿主 glibc 不低于构建时所用的那份。 +### Windows(PE)—— 产物是 `.zip`,DLL 就放在 `.exe` 旁边 + +Windows 目标产出的是 **`.zip`** 而不是 `.tar.gz`,并且是扁平布局: + +``` +target/dist/myapp-0.1.0-x86_64-pc-windows-msvc.zip +└── myapp-0.1.0-x86_64-pc-windows-msvc/ + ├── myapp.exe + ├── vcruntime140.dll ← 仅在 cxx_runtime = "toolchain-coupled" 时 + ├── mydep.dll ← 第三方依赖 + ├── README.md + └── LICENSE +``` + +没有 `bin/` + `lib/` 的分层,也没有入口 wrapper —— 这两点都不是风格选择。 +Win32 loader 解析 DLL 的第一顺位就是**可执行文件所在目录**,而 PE 没有 +`RUNPATH` 可以指向别处,所以"放在 `.exe` 旁边"**就是** ELF 上 +`$ORIGIN/../lib` 所提供的那个机制。 + +**Windows 自己的 DLL 永远不会被打进包里** —— `kernel32.dll`、`ntdll.dll`、 +`ucrtbase.dll`、`api-ms-win-*` API set。带一份系统组件的私有拷贝不是"包变大 +了",而是**程序坏了**(进程里出现了两份本应唯一的东西),而 Microsoft 的再分发 +条款也从另一侧说了同一件事。`[pack.bundle-project] force_bundle` 仍然可以覆盖 +这条,和它覆盖 ELF 跳过表一样。 + +`vcruntime140.dll` / `msvcp140.dll` **不是** Windows 自己的:它们属于 MSVC +toolset,就像 `libstdc++.so` 属于 gcc。它们要不要跟着产物走,由 `cxx_runtime` +决定(见 `docs/zh/05-mcpp-toml.md`),不由这张表决定 —— 而 `mcpp pack` 会拒绝 +那些无法兑现契约的组合: + +``` +$ mcpp pack --mode system # 且 cxx_runtime = "toolchain-coupled" +error: cxx_runtime = "toolchain-coupled" and --mode system contradict each other. +``` + +#### 在 Linux / macOS 上给 Windows 打包 + +这是可以的,而且不是什么特殊模式 —— 指定 Windows 目标构建,然后打包即可: + +```bash +mcpp pack --target x86_64-windows-gnu # 在 Linux 宿主上 +``` + +`mcpp pack` 以前直接拒绝 Windows。真正的原因不是打包工具:ELF 的依赖闭包是靠 +**把产物跑起来**(`LD_TRACE_LOADED_OBJECTS`)求出来的,所以它既跨不了 OS 也跨 +不了架构。PE 的闭包改为从文件的导入表里**读**出来,于是不需要执行任何东西, +打包宿主也就自由了。压缩包本身也由 mcpp 自己写,理由相同 —— 没有哪个 zip 工具 +在每个宿主上都存在。 + +有两点值得知道: + +- 条目是 **stored(不压缩)** 的,所以 Windows 包的体积约等于其内容之和。压缩 + 是体积优化而不是正确性问题,目前尚未实现。 +- 压缩包是**确定性**的:不读取任何时间戳,同一棵树打两次字节一致,公布的校验和 + 才有意义。 + +反方向 —— 在 Windows 上给 Linux / macOS 产物打包 —— 仍然不支持,原因还是最初 +那个:那条闭包要由目标自己的动态链接器解析,而 Windows 宿主没有办法运行它。 + ## 配置项 打包行为通过 `mcpp.toml` 中的 `[pack]` 节配置,常用字段如下: diff --git a/docs/zh/03-toolchains.md b/docs/zh/03-toolchains.md index 20912eb4..08cd9105 100644 --- a/docs/zh/03-toolchains.md +++ b/docs/zh/03-toolchains.md @@ -203,19 +203,46 @@ payload 是按宿主 arch 构建的。**macOS 宿主则完全没有面向 Linux Targets 一栏里没有的,就是这台机器确实服务不了(实现见 `toolchain::host_can_serve`)。 -## MSVC(系统工具链,Windows) +## MSVC(Windows) -MSVC 与 mcpp 管理的其它工具链都不同:它是一条**系统工具链**。mcpp 只负责 -定位并识别已安装的 Visual Studio / Build Tools —— **从不**安装、升级或卸载 -MSVC 本身。 +一个 MSVC toolset 有两条路径进入构建,由 **spec 的版本轴**决定是哪一条: + +| Spec | 来源 | 你拿到的是哪个编译器 | +|---|---|---| +| `msvc@system`(或裸 `msvc`) | 这台机器自己的 Visual Studio | 这里恰好装了什么就是什么 | +| `msvc@`(如 `msvc@14.44.35207`) | mcpp 安装的 xlings payload | 你点名的那一个,在每台机器上都一样 | + +它们不是"二选一",而是回答了不同的问题。`msvc@system` 问的是*"用这位开发者 +已经有的东西"*;`msvc@14.44.35207` 问的是*"用恰好这个编译器构建本工程"*。 +pinned toolset 之间、以及与系统 Visual Studio 之间都可以共存。 + +> **`@system` 是 MSVC 独有的拼写。** 没有 `gcc@system`,也没有 `llvm@system`, +> 这是有意的而不是漏了:mcpp 建立在用户态 OS xlings 之上,整条设计就是**把 host +> 依赖降到最低** —— 工具链来自 manifest 点名的 payload,于是每台机器用同一个 +> 编译器。Windows 是唯一一处"拒绝使用已装好的东西"代价大于收益的地方: +> Visual Studio 常常已经装了,又不总能重新分发。其它族写 `@system` 会 +> 直接报错,并同时给出你可能想要的两种写法。(不带族的 +> `[toolchain] … = "system"` —— 即 PATH 上的编译器 —— 是另一套、也是有意保留的 +> 逃生口,不受影响。) + +### `msvc@system` —— 机器自己的 Visual Studio + +mcpp 只负责定位并识别已安装的 Visual Studio / Build Tools,**从不**安装、 +升级或卸载它。 ```bash mcpp toolchain default msvc ``` -在装有 MSVC 的机器上,mcpp 会自动定位(依次尝试 `vswhere.exe`、 -`VSINSTALLDIR`/`VS*COMNTOOLS`、标准安装路径),识别涉及的各个版本, -并持久化为稳定 spec `msvc@system`: +定位顺序: + +1. **`VSINSTALLDIR`** —— 由开发者命令提示符或跑过 `vcvarsall` 的 CI 步骤设置。 + 这是一个**回答**而不是猜测,所以排在下面几种探测之前。 +2. `vswhere.exe`(含 prerelease / Insiders 实例) +3. `VS*COMNTOOLS` +4. 标准的 `Program Files\Microsoft Visual Studio\\` 路径 + +随后识别涉及的各个版本,并持久化为稳定 spec `msvc@system`: ``` Detected msvc 19.44.35211 (VS 2022 BuildTools) (VC tools 14.44.35207) @@ -224,29 +251,93 @@ Detected msvc 19.44.35211 (VS 2022 BuildTools) (VC tools 14.44.35207) Default set to msvc@system (was: llvm@20.1.7) ``` -若机器上**没有** MSVC,mcpp 打印安装指引(Visual Studio Installer 勾选 -*Desktop development with C++* 负载,或 `winget install -Microsoft.VisualStudio.2022.BuildTools`)并以非零码退出 —— 需要你自己装好, -再重跑该命令。 +若机器上没有 Visual Studio,mcpp 会说出来并同时给出两条路:一个它可以替你装的 +pinned toolset,或者 Visual Studio Installer / +`winget install Microsoft.VisualStudio.2022.BuildTools`。 `mcpp toolchain list` 会把检测到的 MSVC 列在单独的 `System:` 分区, -`mcpp self doctor` 在 Windows 上会报告它的状态。manifest 里可按平台 pin: +`mcpp self doctor` 在 Windows 上会报告它的状态。manifest 里: ```toml [toolchain] windows = "msvc@system" ``` -`msvc@<前缀>`(如 `msvc@19.44`)是一个 **pin-verify**:mcpp 仍然使用已安装 -的最新 VC tools,但检测到的版本与前缀不符时报错。 +### `msvc@` —— mcpp 安装并 pin 的 toolset + +```bash +mcpp toolchain list --available msvc # 可以 pin 哪些 +mcpp toolchain install msvc 14.44.35207 +``` + +这在每个方面都和 `gcc@16.1.0` 一样:payload 下载进 mcpp 自己的 store,多个 +toolset 共存,`mcpp toolchain remove msvc@` 卸载其中一个,manifest 里 +点名的那个会在首次构建时自动安装。 + +```toml +[toolchain] +windows = "msvc@14.44.35207" +``` + +**这里的版本是 toolset 目录名**(`14.44.35207` —— 即 `VC\Tools\MSVC\` 下的 +目录名、也是 `-vcvars_ver` 接受的值),**不是** cl banner 版本 +(`19.44.35211`),也不是产品年份。机器上不需要预装任何东西:payload 带来编译器、 +STL,以及通过它的 `xim:windows-sdk` 依赖带来 ucrt/um 的头文件与导入库。 + +> **已变更:** `msvc@19.44` 过去的含义是"用系统 MSVC,并校验其 banner 以 19.44 +> 开头" —— 这条只有 `mcpp toolchain default` 会查,构建则静默忽略。现在版本轴 +> 在所有地方都指 toolset。写成 `19.x` 会得到一条同时给出两种替代写法的错误 +> —— `msvc@system`,或那台机器实际拥有的 toolset 版本。 + +### 原生 cl.exe 构建 + +自 0.0.90 起两条来源都可用:mcpp 从 VC tools + Windows SDK 合成 INCLUDE/LIB +环境(不经 `vcvarsall`),把 `std.ixx`/`std.compat.ixx` staging 成 `.ifc` BMI, +用 `/interface /TP /ifcOutput` 编译 `.cppm` 模块单元,用 `/scanDependencies` +扫描,并通过 response file 调 `link.exe`/`lib.exe` 链接。 + +**Windows SDK 跟着来源走**,因为两条来源回答的是不同的问题,SDK 也必须如此: + +| 来源 | SDK 怎么选 | +|---|---| +| `msvc@` | **随该 toolset 一起装进 mcpp store 的** `xim:windows-sdk` payload。环境里的 `WindowsSdkDir` / `WindowsSdkVersion` 会被**忽略**,并且 mcpp 会打印一行 `note:` 说明。 | +| `msvc@system` | 先 **`WindowsSdkDir`**(+ `WindowsSdkVersion`),再 `C:\Program Files (x86)\Windows Kits\10`。 | + +这种不对称正是要点。pin 一个 toolset 是在承诺"两台机器用同一套头文件编同一份 +源码";一个能悄悄改写它的环境变量会把这条承诺降格成偏好。反过来,机器自己的 +SDK 只能靠找,而在那里"明确声明"应当压过"扫描" —— 和 `VSINSTALLDIR` 压过 +`vswhere` 是同一条优先级。 + +如果一个 pinned toolset 旁边没有 SDK payload(比如较老的安装),mcpp 会退回用 +机器上的 SDK 而不是失败 —— 并且会说出来,因为那次构建已经不可复现,而除此之外 +没有任何东西会记录这件事。 + +一个根目录只有**两半都在**才算 SDK —— `Include\\ucrt\corecrt.h` **且** +`Lib\\um\\kernel32.lib`。只有头文件、没有导入库的根会被跳过而不是 +被选中,于是一个只解包了一半的 payload 不会压过机器上完整的 SDK,也就不会在 +构建的最后一刻变成 `LNK1104: cannot open file 'kernel32.lib'`。 + +解析出的 SDK 版本是这次构建的**运行时身份**(`ucrt@10.0.26100.0`)的一部分, +因而也进入了给构建缓存做 key 的那个指纹:换 SDK 就换缓存键,和换编译器一样。 +它是一条**兼容性下限声明**,而不是 Linux 上 `glibc@2.39` 那种 payload 绑定 —— +`ucrtbase.dll` 是 Windows 组件,mcpp 既不分发也不替换它。 + +**CRT 模型。** 默认 `/MD`(host-coupled);下面两者任一都会选 `/MT`: + +```toml +[target.x86_64-windows-msvc] +linkage = "static" # libc 那根轴 —— TARGET 段,或 `--static` + +[build] +cxx_runtime = "self-contained" # C++ 运行时那根轴 +``` + +注意这两个键分别属于哪个段:`linkage` 只认精确 triple,**没有 `[build] linkage` +这个键** —— 写了会得到一条 "unsupported key (ignored)" 警告,而且不会切到静态 +CRT。 -自 0.0.90 起,**原生 cl.exe 构建可用**:mcpp 从检测到的 VC tools + Windows -SDK 合成 INCLUDE/LIB 环境(不经 `vcvarsall`),把 `std.ixx`/`std.compat.ixx` -staging 成 `.ifc` BMI,用 `/interface /TP /ifcOutput` 编译 `.cppm` 模块单元, -用 `/scanDependencies` 扫描,并通过 response file 调 `link.exe`/`lib.exe` -链接。选择 `/MT` CRT 用 `[target.x86_64-windows-msvc] linkage = "static"` -(或 `mcpp build --static`)—— 不是 `[build] linkage`,那个键不存在。 -缺 Windows SDK 会让构建失败并给出安装指引(`mcpp self doctor` 会报告 SDK 状态)。 +toolset 自带的那份可再分发 CRT(`vcruntime140.dll` / `msvcp140.dll`)可以跟着 +产物走 —— 见 `docs/zh/05-mcpp-toml.md` 的 `cxx_runtime = "toolchain-coupled"`。 ## 项目级版本锁定 diff --git a/docs/zh/05-mcpp-toml.md b/docs/zh/05-mcpp-toml.md index 01bdf315..b6af2965 100644 --- a/docs/zh/05-mcpp-toml.md +++ b/docs/zh/05-mcpp-toml.md @@ -349,9 +349,35 @@ C++ 运行时的进程。 等价于 `host-coupled`。显式写了 `cxx_runtime` 时以后者为准。 **兑现不了的契约会被报出来,绝不静默降级。** 若工具链不带 `libc++.a`,或某个 -契约在该平台上没有对应机制(MSVC 运行时的 `self-contained` 需要 `/MT`,mcpp -目前不发射),构建会打印实际退到了哪一档,而不是悄悄交付一个与 manifest 所述 -不同的产物。 +契约在该平台上没有对应机制,构建会打印实际退到了哪一档,而不是悄悄交付一个与 +manifest 所述不同的产物。 + +#### 在 MSVC 运行时上 + +这里的机制就是 CRT 模型,而它是**整个工程级**的开关:cl 会把 `_MSVC_MT` / +`_MSVC_MD` 烘进工程唯一的那份 `std` 模块,所以与工程不一致的按角色契约无法兑现, +会被报出来而不是被忽略。 + +| 取值 | 在 MSVC 上是什么 | +|---|---| +| `self-contained` | `/MT`,静态 CRT。`linkage = "static"` 从 libc 那根轴选中的是同一件事。 | +| `host-coupled`(`/MD` 下的默认) | 由目标机器提供 `vcruntime140.dll` / `msvcp140.dll` —— 即那台机器装了 Visual Studio 或 redistributable。 | +| `toolchain-coupled` | toolset **自带**的那份 DLL 跟着产物走。 | + +`toolchain-coupled` 值得说清楚,因为直觉上的理解是错的。`ucrtbase.dll` **是** +Windows 组件(Win10 起),mcpp 从不分发它;而 `vcruntime140.dll` / +`msvcp140.dll` **不是**:每个 MSVC toolset 都在 +`VC\Redist\MSVC\\\` 下带着它们,和 gcc payload 带着 +`libstdc++.so` 是同一件事。在这个契约下 mcpp 会把它们放到产物旁边 —— 这正是让 +默认的 `/MD` 产物能在"只装了 pinned toolset、根本没有 Visual Studio"的机器上跑 +起来的原因。 + +调试版 CRT(`debug_nonredist\` 下的 `vcruntime140d.dll` 等)永远不会被放进去: +它不可再分发。 + +把它和 `/MT` 一起用是**矛盾**而不是缺功能 —— 静态 CRT 根本没有 DLL 可以耦合 —— +所以会被报出来并落到 `self-contained`。另一半由 `mcpp pack` 兜底:什么都不打包的 +模式(`--mode system`、`--mode static`)兑现不了 `toolchain-coupled`,会直接拒绝。 **边界。** 该契约只管 C++ 运行时。静态 **libc** 是另一根轴(`linkage = "static"` / `--static`,如 musl 目标),部署下限是第三根轴(`macos_deployment_target`)。 diff --git a/mcpp.toml b/mcpp.toml index e1646824..e1f18217 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "mcpp" -version = "2026.8.16.3" +version = "2026.8.17.1" description = "Modern C++ build & package management tool" license = "Apache-2.0" authors = ["mcpp-community"] diff --git a/src/build/distribution.cppm b/src/build/distribution.cppm index 5f797342..b18bcf24 100644 --- a/src/build/distribution.cppm +++ b/src/build/distribution.cppm @@ -268,6 +268,16 @@ struct Mechanism { // global constructor that touches std::cout runs before the streams // exist. Asks the backend for the ordering shim. See issue #336. bool streamInitShim = false; + // PE + MSVC runtime + toolchain-coupled: the toolset's own redistributable + // CRT DLLs must be STAGED BESIDE the artifact. + // + // On ELF, `toolchain-coupled` needs no files copied — the artifact carries + // an rpath into the toolchain's lib directory and the loader follows it. + // PE has no rpath: a DLL is resolved from the directory of the executable + // (and then PATH), so on this format the mechanism IS the copy. Same + // contract, same meaning, different mechanism — which is exactly the split + // this module's three layers exist to express. + bool deployToolchainRuntime = false; }; namespace detail { @@ -426,12 +436,43 @@ Mechanism resolve(const MechanismInput& in) { "it everywhere; using host-coupled here" : ""; } else if (in.requested == Contract::ToolchainCoupled) { - // Only reachable from an explicit request: it is never a default. - m.degraded = true; - m.diagnostic = std::format( - "cxx_runtime = \"toolchain-coupled\" has no meaning for the " - "MSVC runtime (it ships with the OS/redistributable, not with " - "the toolchain); using {}", to_string(m.effective)); + // THIS USED TO BE A FLAT REFUSAL, and the sentence it refused + // with was half true: + // + // "…has no meaning for the MSVC runtime (it ships with the + // OS/redistributable, not with the toolchain)" + // + // True of `ucrtbase.dll`, which IS an OS component since + // Win10. NOT true of `vcruntime140.dll` / `msvcp140.dll`, + // which are the toolset's own and sit inside every MSVC + // toolset ever shipped: + // + // VC\Redist\MSVC\\\Microsoft.VC.CRT\*.dll + // + // That is the same relationship gcc has to libstdc++.so, so it + // takes the same contract — and refusing it left a hole in the + // matrix that had a real cost: the default `/MD` artifact + // depends on DLLs a machine with only a managed toolset does + // not have, and there was no spelling that made them travel. + // + // `/MT` is the one case that stays a degradation, and it is a + // genuine contradiction rather than a missing mechanism: a + // static CRT leaves NO DLL to couple to. Say which one won. + if (in.msvcStaticCrt) { + m.effective = Contract::SelfContained; + m.degraded = true; + m.diagnostic = + "cxx_runtime = \"toolchain-coupled\" cannot apply to a " + "project compiled with the static CRT (/MT): there is " + "no vcruntime140.dll/msvcp140.dll dependency left to " + "couple to. Drop linkage = \"static\" (or the " + "project-wide self-contained contract) if the toolset's " + "CRT should travel beside the artifact instead; using " + "self-contained"; + } else { + m.effective = Contract::ToolchainCoupled; + m.deployToolchainRuntime = true; + } } return m; } diff --git a/src/build/flags.cppm b/src/build/flags.cppm index d69895a8..2058e12d 100644 --- a/src/build/flags.cppm +++ b/src/build/flags.cppm @@ -77,6 +77,28 @@ struct CompileFlags { // macOS + self-contained: link units need the initializer-ordering shim // object prepended to their inputs (issue #336). bool needsStreamInitShim = false; + // PE + `toolchain-coupled`: the toolset's own CRT DLLs, to be staged + // beside the artifact. Resolved HERE rather than in the emitter because + // "which files does this contract imply" is a contract question; the + // backend only knows how to spell a copy edge. + // + // A whole-BUILD list, not a per-role one, and that is a property of the + // format rather than a simplification: a PE artifact resolves a DLL from + // its own directory, so one directory holds one answer and two roles in + // one output tree cannot disagree about it. Any built role asking for the + // contract is enough to populate it. + // + // The DIRECTORY comes from `msvc::vc_redist_dir()` via + // `Toolchain::linkRuntimeDirs`, which is what keeps `debug_nonredist\` + // (vcruntime140d.dll & friends — NOT redistributable) out of the list. The + // criterion lives in exactly one place on purpose: a second name-shaped + // rule here could disagree with it, and a copy step that disagrees about + // what may be redistributed is a licensing defect, not a bug. + // + // Already deduped against the plan's own deploy files, so the emitter can + // append without deciding anything: a name the manifest already claims + // stays the manifest's and the conflict is reported through `diagnostics`. + std::vector toolchainRuntimeDeploy; // Non-empty when a requested contract could not be honored. The caller // MUST surface these — a silent downgrade is the failure mode this whole // model exists to prevent. Emitted once by the backend, not here, because @@ -762,6 +784,22 @@ CompileFlags compute_flags(const BuildPlan& plan) { // produces a PE and must take the PE answer. const dist::Format format = [&] { if (isMingwTc) return dist::Format::Pe; + // The TARGET's own word, when it has one. `isMingwTc` was the only + // cross case this knew about, so every other question about the + // output format was answered by asking the HOST — which is right + // whenever they agree and unaskable in a test that does not run on + // the platform it is about. A triple that names its OS is a fact; + // the host is a stand-in for one. + // + // Only ADDS answers: a triple that says neither falls through to + // exactly the previous derivation, so no existing build changes. + const auto& t = plan.toolchain.targetTriple; + if (t.find("windows") != std::string::npos + || t.find("mingw") != std::string::npos) + return dist::Format::Pe; + if (t.find("apple") != std::string::npos + || t.find("darwin") != std::string::npos) + return dist::Format::MachO; if constexpr (mcpp::platform::needs_explicit_libcxx) return dist::Format::MachO; else if constexpr (mcpp::platform::is_windows) @@ -896,6 +934,8 @@ CompileFlags compute_flags(const BuildPlan& plan) { }); }; + bool wantsToolchainRuntime = false; + for (auto [role, requested, wasAsked] : { std::tuple{dist::Role::Distributable, base, explicitBase}, std::tuple{dist::Role::Test, testsContract, explicitTests}, @@ -910,10 +950,65 @@ CompileFlags compute_flags(const BuildPlan& plan) { f.ldStdlibCByRole[i] = r.unitFlagsC; f.contractByRole[i] = r.effective; if (r.streamInitShim) f.needsStreamInitShim = true; + // Only a role this build actually HAS may pull DLLs into the + // output tree. The contract is resolved for every role because + // `ldStdlibByRole` must be total; staging files is a side effect + // on disk, and a project with no test binaries should not get a + // CRT copied beside nothing. + if (r.deployToolchainRuntime && role_is_built(role)) + wantsToolchainRuntime = true; if (!r.diagnostic.empty() && role_is_built(role)) f.diagnostics.push_back(std::format( "{} target: {}", dist::to_string(role), r.diagnostic)); } + if (wantsToolchainRuntime) { + // `linkRuntimeDirs` is the toolset's own redistributable CRT + // directory and nothing else on this toolchain — `enrich_toolchain + // _from_cl` puts exactly `vc_redist_dir()` there. Guarded on the + // compiler anyway: the field means "the toolchain's private + // runtime" for every provider, and on gcc it holds libstdc++'s + // directory, which has no business being copied into a PE tree. + if (plan.toolchain.compiler == mcpp::toolchain::CompilerId::MSVC) { + std::vector sources; + std::error_code ec; + for (auto const& dir : plan.toolchain.linkRuntimeDirs) { + for (auto const& e : + std::filesystem::directory_iterator(dir, ec)) { + if (!e.is_regular_file(ec)) continue; + auto ext = e.path().extension().string(); + std::ranges::transform(ext, ext.begin(), + [](unsigned char c) { return std::tolower(c); }); + if (ext != ".dll") continue; + sources.push_back(e.path()); + } + } + // Directory order is not a stable input: this list reaches + // build.ninja, and a graph that differs between two runs of + // the same build re-runs edges for no reason. + std::ranges::sort(sources); + for (auto const& src : sources) { + auto dest = std::filesystem::path("bin") / src.filename(); + // An explicit `[runtime] deploy_files` naming the same DLL + // WINS, and says so. A human wrote that one down; this list + // is derived. Silently overwriting a vendored redist with + // the toolset's copy is a different program than the one + // the manifest describes. + auto clash = std::ranges::find_if(plan.runtimeDeployFiles, + [&](auto const& d) { return d.dest == dest; }); + if (clash != plan.runtimeDeployFiles.end()) { + if (clash->source.lexically_normal() + != src.lexically_normal()) + f.diagnostics.push_back(std::format( + "toolchain-coupled would stage '{}' beside the " + "artifact, but this project already deploys " + "'{}' there; keeping the project's file", + src.string(), clash->source.string())); + continue; + } + f.toolchainRuntimeDeploy.push_back({src, dest}); + } + } + } // Two roles usually share a contract, so they usually share a // complaint; report each distinct one once. std::ranges::sort(f.diagnostics); diff --git a/src/build/ninja_backend.cppm b/src/build/ninja_backend.cppm index a6acad5e..d4dcea6f 100644 --- a/src/build/ninja_backend.cppm +++ b/src/build/ninja_backend.cppm @@ -449,6 +449,18 @@ std::string emit_ninja_string(const BuildPlan& plan) { // All compile/link flags are computed once via flags.cppm. auto flags = compute_flags(plan); + // Everything that has to sit beside the artifact, from both producers: + // the manifest's `[runtime] deploy_files` (already in the plan) and the + // C++ runtime contract's own answer (`toolchain-coupled` on PE — see + // mcpp.build.distribution). Merged ONCE, here, because three places below + // consume the list — the implicit dependency of each executable, the copy + // edges, and `default` — and a list that is complete in two of them is a + // graph where the DLL is copied only when something else happens to ask. + auto deployFiles = plan.runtimeDeployFiles; + deployFiles.insert(deployFiles.end(), + flags.toolchainRuntimeDeploy.begin(), + flags.toolchainRuntimeDeploy.end()); + bool need_c_rule = false, need_asm_rule = false, need_nasm_rule = false; for (auto& cu : plan.compileUnits) { if (is_c_source(cu)) need_c_rule = true; @@ -1770,7 +1782,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { // beside the .exe before the build is considered done. Empty on RPATH // platforms (no *.dll deps), so other targets are unaffected. if (lu.kind == LinkUnit::Binary || lu.kind == LinkUnit::TestBinary) { - for (auto const& d : plan.runtimeDeployFiles) + for (auto const& d : deployFiles) implicit += " " + escape_ninja_path(d.dest); } @@ -1826,13 +1838,13 @@ std::string emit_ninja_string(const BuildPlan& plan) { // — which also means a DLL still loaded by a running program from a // previous `mcpp run` gets the skip-if-equivalent treatment instead of a // hard "cannot copy" failure. - // Inert on RPATH platforms where runtimeDeployFiles is empty. - for (auto const& d : plan.runtimeDeployFiles) { + // Inert on RPATH platforms where the merged deploy list is empty. + for (auto const& d : deployFiles) { append(std::format("build {} : stage_file {}\n", escape_ninja_path(d.dest), escape_ninja_path(d.source))); } - if (!plan.runtimeDeployFiles.empty()) + if (!deployFiles.empty()) append("\n"); // ── Declared build-graph nodes (`mcpp:action=`) ───────────────────────── @@ -1899,7 +1911,7 @@ std::string emit_ninja_string(const BuildPlan& plan) { defaults += " " + escape_ninja_path(alias); } } - for (auto const& d : plan.runtimeDeployFiles) { + for (auto const& d : deployFiles) { defaults += " " + escape_ninja_path(d.dest); } defaults += actionDefaults; diff --git a/src/build/prepare.cppm b/src/build/prepare.cppm index 2284b0c5..a4f91dd7 100644 --- a/src/build/prepare.cppm +++ b/src/build/prepare.cppm @@ -921,10 +921,46 @@ prepare_build(bool print_fingerprint, } // ─── Toolchain resolution (docs/21) ──────────────────────────────── - // Priority chain: - // 1. mcpp.toml [toolchain]. → resolve_xpkg_path → abs path - // 2. $CXX env var - // 3. PATH g++ (with warning) + // + // THE WHOLE CHAIN, in the order it is applied. It was documented twice, as + // "3 steps" here and "4 steps" further down, and neither list had been + // true for a long time — between them they named five of the nine inputs + // below and disagreed about two. A comment that undercounts the inputs to + // a decision is worse than none: it tells the next reader they have seen + // the whole thing. + // + // The WHAT (which spec) is settled first, then the HOW (which binary). + // Anything that WRITES `tcSpec` also writes `tcOrigin`, and that is the + // invariant this table rests on — the enumerator names below are real, so + // this comment cannot quietly stop matching the code. + // + // WHICH SPEC tcOrigin + // 1. mcpp.toml [toolchain]. / .default ManifestToolchain + // 2. global config.toml [toolchain] default GlobalDefault + // 3. mcpp.toml [target.].toolchain TargetSection + // (--target / [build] target / config default + // select the section; 3 outranks 1 and 2) + // 4. the target vocabulary's pin (triple.cppm) TargetPin + // — a convention, and it stands down when a + // REMEMBERED target would overrule a spec the + // user wrote down + // 5. the platform's first-run default, installed FirstRun + // and persisted by this very invocation + // + // WHICH BINARY, from the spec settled above + // 6. `msvc@system` → probe the machine (no xim package exists) + // 7. `@` → xim payload; msvc resolves through + // resolve_managed_msvc, everything else through + // the bin/-shaped frontend lookup + // 8. bare `system` → the PATH compiler. A deliberate escape hatch, and + // the ONLY host-compiler route: there is no + // `gcc@system` (see parse_toolchain_spec) + // 9. offline / MCPP_NO_AUTO_INSTALL → hard error rather than a silent + // ~800 MB download + // + // AND ONE REVISION, after 1-9 have produced a toolchain: a spec targeting + // the MSVC ABI on a machine with no usable MSVC is switched to MinGW-w64 + // — but only when `tc_origin_is_user_explicit` says mcpp chose it itself. std::filesystem::path explicit_compiler; std::optional cfg_opt; bool bootstrap_checked = false; @@ -1000,11 +1036,10 @@ prepare_build(bool print_fingerprint, constexpr std::string_view kCurrentPlatform = mcpp::platform::name; - // M5.5: toolchain resolution priority: - // 0. --target X / --static, looked up in [target.] - // 1. project mcpp.toml [toolchain]. or .default - // 2. global ~/.mcpp/config.toml [toolchain].default - // 3. hard error (no system fallback) + // Toolchain resolution priority: see the table at the top of this + // function. Stated once, where `tcOrigin` is introduced — this used to be + // a second, shorter and differently-wrong list of the same thing. + // // Resolve the build profile, overlaid by any [profile.] from the // manifest → buildConfig. `effectiveProfile` outlives the block: the // build.mcpp env contract exposes it as MCPP_PROFILE. @@ -1259,18 +1294,36 @@ prepare_build(bool print_fingerprint, // resolution / fingerprinting. fold_build_defines_into_flags(m->buildConfig); - // msvc@system: located on the machine, never resolved through xim - // packages. mcpp does not install the machine's Visual Studio. + // ORIGIN, RESOLVED ONCE. + // + // The spec used to be parsed TWICE from the same string a dozen lines + // apart — once to ask "is this msvc@system", once to get the package — + // and each call site drew its own conclusions from the result. Two parses + // of one string is two places for the answer to differ, which is the shape + // §1 of the three-axes design is about: a platform special case whose cost + // is paid at every site that has to know about it. // - // A VERSIONED msvc spec is a different origin and takes the xim path - // below — which is the whole point: what the manifest says is what gets + // `Origin::SystemMsvc` is located on the machine and never resolved + // through an xim package — mcpp does not install the machine's Visual + // Studio. `Origin::Managed` is everything else, including a VERSIONED + // msvc spec, and that is the point: what the manifest says is what gets // used, on every machine, instead of whatever this one happens to have. - bool tcSpecIsMsvc = false; - if (tcSpec.has_value()) { - if (auto s = mcpp::toolchain::parse_toolchain_spec(*tcSpec); - s && mcpp::toolchain::is_system_toolchain(*s)) - tcSpecIsMsvc = true; + std::optional parsedSpec; + auto tcOriginAxis = mcpp::toolchain::Origin::Managed; + if (tcSpec.has_value() && *tcSpec != "system") { + // A parse FAILURE is not the same as an unparseable spec being + // absent: `gcc@system` now fails here by name (see + // parse_toolchain_spec), and swallowing that would put the error back + // where it used to happen — somewhere else, saying something else. + auto s = mcpp::toolchain::parse_toolchain_spec(*tcSpec); + if (!s) return std::unexpected(std::format( + "[toolchain].{} = '{}': {}", kCurrentPlatform, *tcSpec, s.error())); + parsedSpec = std::move(*s); + tcOriginAxis = mcpp::toolchain::origin_of(*parsedSpec); } + const bool tcSpecIsMsvc = + parsedSpec && tcOriginAxis == mcpp::toolchain::Origin::SystemMsvc; + if (tcSpecIsMsvc) { if (!mcpp::platform::is_windows) { return std::unexpected(std::format( @@ -1284,9 +1337,9 @@ prepare_build(bool print_fingerprint, mcpp::ui::info("Resolved", std::format( "msvc@system → msvc {} ({})", inst->display_version(), inst->clPath.string())); - } else if (tcSpec.has_value() && *tcSpec != "system") { - auto spec = mcpp::toolchain::parse_toolchain_spec(*tcSpec); - if (!spec || spec->version.empty()) { + } else if (parsedSpec) { + auto spec = parsedSpec; + if (spec->version.empty()) { return std::unexpected(std::format( "[toolchain].{} = '{}' is invalid; expected '@'", kCurrentPlatform, *tcSpec)); @@ -1331,20 +1384,13 @@ prepare_build(bool print_fingerprint, // (cl.exe is four levels deeper) and the ELF post-install fixup // (there is nothing to patchelf on a PE toolchain). if (spec->family == mcpp::toolchain::Family::Msvc) { - // Not `payload->root`: that field is the fetcher's guess at where - // the useful tree starts, and it descends into a lone - // subdirectory when the version dir has no bin/ include/ lib/. - // An msvc payload's only entry is `VC/`, so the guess lands one - // level too deep. (store, name, version) is known — use it. - auto verDir = mcpp::xlings::paths::xim_tool( - mcpp::config::make_xlings_env(**cfg), pkg.ximName, pkg.ximVersion); - auto inst = mcpp::toolchain::msvc::installation_at( - verDir, pkg.ximVersion); - if (!inst) { - return std::unexpected(std::format( - "msvc payload at '{}' has no cl.exe under VC/Tools/MSVC/{}", - verDir.string(), pkg.ximVersion)); - } + // One rule, one place: where a managed toolset lives and why the + // fetcher's `root` must not be used for it (mcpp.toolchain. + // registry). Install and build asked the same question and each + // answered it in its own words. + auto inst = mcpp::toolchain::resolve_managed_msvc( + mcpp::config::make_xlings_env(**cfg), pkg); + if (!inst) return std::unexpected(inst.error()); explicit_compiler = inst->clPath; mcpp::ui::info("Resolved", std::format( "{} → msvc {} ({})", spec->display(), @@ -1484,8 +1530,9 @@ prepare_build(bool print_fingerprint, mcpp::fetcher::InstallProgressHandler progress; // The glibc default toolchain needs the sysroot payloads (C library + - // kernel headers). The musl default is self-contained, so skip them. - if (!mcpp::platform::is_macos && !mcpp::platform::is_windows && !muslDefault) { + // kernel headers). One derivation, shared with `toolchain install` — + // see registry.cppm for what the two spellings used to disagree about. + if (mcpp::toolchain::needs_linux_sysroot_payloads(defaultParsed->target)) { for (auto dep : {"xim:glibc", "xim:linux-headers"}) { (void)fetcher.resolve_xpkg_path(dep, /*autoInstall=*/true, &progress); } @@ -1553,6 +1600,31 @@ prepare_build(bool print_fingerprint, explicit_compiler, runtimePayload, runtimeBindingSnapshot.contractHash); if (!tc) return std::unexpected(tc.error().message); + // Something about the resolution the user has to be told, but which is + // not a failure. Today's only producer is the Windows SDK axis: a managed + // toolset binds the SDK it was installed with, so a `WindowsSdkDir` in + // the environment does not apply — and an override that is ignored + // SILENTLY is indistinguishable from one that was never set. + if (!tc->resolutionNote.empty()) + mcpp::ui::info("note", tc->resolutionNote); + + // The Windows runtime identity, flowing BACK into the contract. + // + // Everything else about the runtime is known before a toolchain is + // resolved, and deliberately so (see the RuntimeBinding block above). The + // Windows SDK is the exception: it is a property of the toolchain, and + // until it reached the contract hash the version axis simply did not + // exist one layer below the compiler — two SDKs produced one cache key. + // + // `ucrt@` is a COMPATIBILITY FLOOR, not a payload binding like + // `glibc@`: ucrtbase.dll is an OS component and mcpp ships no + // redistributable for it. See mcpp.platform.runtime_binding. + if (!tc->windowsSdkVersion.empty()) { + mcpp::platform::runtime::bind_windows_ucrt( + runtimeBindingSnapshot, tc->windowsSdkVersion); + tc->runtimeContractHash = runtimeBindingSnapshot.contractHash; + } + // ── Targeting the MSVC ABI without a usable MSVC ───────────────────── // // One judgement, one place. This used to be two separate concerns and diff --git a/src/build/runtime_validation.cppm b/src/build/runtime_validation.cppm index 88a743c0..f80a171a 100644 --- a/src/build/runtime_validation.cppm +++ b/src/build/runtime_validation.cppm @@ -383,8 +383,12 @@ ValidationReport validate_changed_artifacts( const ArtifactSnapshot& before) { ValidationReport report; if constexpr (!mcpp::platform::is_linux) return report; + // Provider dispatch (see mcpp.platform.runtime_binding): what follows is + // ELF/glibc physics, and an identity from another provider — `ucrt@…` on + // Windows — has no rules here rather than a missing glibc. if (plan.runtimeBinding.platform != "linux" - || !plan.runtimeBinding.runtimeId.starts_with("glibc@")) + || mcpp::platform::runtime::runtime_provider( + plan.runtimeBinding.runtimeId) != "glibc") return report; auto doc = read_cache(plan.outputDir); diff --git a/src/doctor.cppm b/src/doctor.cppm index ece85bb4..96e62d81 100644 --- a/src/doctor.cppm +++ b/src/doctor.cppm @@ -133,13 +133,45 @@ export int doctor_report() { warn("msvc not detected — run `mcpp toolchain default msvc` for " "setup guidance (mcpp does not install MSVC)"); } - // Windows SDK (native cl.exe builds need its UCRT/um headers). + // Windows SDK. REPORTED PER ORIGIN, because it is chosen per origin + // (design §2.2): `msvc@system` searches the machine, a pinned toolset + // takes the payload installed with it. One unlabelled line here would + // be the same "one question, two answerers" shape the SDK axis exists + // to close — a user reading it would believe it applied to their + // pinned build, and it does not. if (auto sdk = mcpp::toolchain::msvc::find_windows_sdk()) { - ok(std::format("Windows SDK {} at {}", sdk->version, + ok(std::format("Windows SDK (msvc@system) {} at {}", sdk->version, sdk->root.string())); } else { - warn("no Windows SDK found — native msvc builds will fail " - "(install the 'Windows 11 SDK' VS component)"); + warn("no Windows SDK found for msvc@system — native builds with " + "the machine's Visual Studio will fail (install the " + "'Windows 11 SDK' VS component)"); + } + { + std::error_code sdkEc; + auto msvcRoot = mcpp::home::root() + / "registry" / "data" / "xpkgs" / "xim-x-msvc"; + for (auto& v : std::filesystem::directory_iterator(msvcRoot, sdkEc)) { + if (!v.is_directory(sdkEc)) continue; + auto ver = v.path().filename().string(); + auto inst = mcpp::toolchain::msvc::installation_at( + v.path(), ver, /*identifyVersion=*/false); + if (!inst) continue; + // The SAME resolution a build performs, not a second one + // shaped like it: what doctor prints is what the user will + // believe, so the two must not be able to disagree. + auto choice = mcpp::toolchain::msvc::resolve_sdk_for(inst->clPath); + if (choice.sdk) { + ok(std::format("Windows SDK (msvc@{}) {} at {}", + ver, choice.sdk->version, + choice.sdk->root.string())); + } else { + warn(std::format( + "msvc@{} has no usable Windows SDK — reinstall it to " + "pull its `xim:windows-sdk` dependency", ver)); + } + if (!choice.note.empty()) warn(choice.note); + } } mcpp::ui::status("Checking", "mingw (xim:mingw-gcc)"); diff --git a/src/pack/binfmt.cppm b/src/pack/binfmt.cppm new file mode 100644 index 00000000..3bd03e3c --- /dev/null +++ b/src/pack/binfmt.cppm @@ -0,0 +1,513 @@ +// mcpp.pack.binfmt — what a binary depends on, read from the FILE. +// +// WHY THIS MODULE EXISTS +// +// `pack` used to derive an artifact's dependency closure like this: +// +// LD_TRACE_LOADED_OBJECTS=1 '' +// +// That is the dynamic linker being asked to resolve the closure, which is +// accurate and complete — and requires RUNNING THE BINARY. So it is not +// "pack has no Windows branch": it cannot cross an OS (a Linux box cannot +// execute a PE) and it cannot cross an ARCHITECTURE (an x86_64 box cannot +// execute an aarch64 ELF, same OS or not). The `#if defined(_WIN32)` refusal +// at the top of `pack::run()` was the symptom; this was the cause. +// +// Reading the file instead removes both limits at once, because nothing here +// executes anything. Cross-OS packaging then is not a feature that had to be +// added — it is what remains when the obstacle is gone. +// +// See .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §4. +// +// WHAT THIS MODULE DOES NOT DO. It answers "which names does this object +// ask for", not "which files will the loader hand it". Resolution is the +// caller's, because the search rules are the loader's policy — and on ELF +// they are genuinely intricate (DT_RPATH before LD_LIBRARY_PATH before +// DT_RUNPATH before ld.so.cache, `$ORIGIN` expansion, hwcaps subdirectories). +// mcpp does not reimplement that: on the host's own format it still asks the +// loader, and this reader serves the cases the loader cannot be asked about. + +export module mcpp.pack.binfmt; + +import std; + +export namespace mcpp::pack::binfmt { + +enum class Format { Unknown, Elf, Pe, MachO }; + +// The identity a packaging decision needs: can THIS machine run that file? +// Both halves matter and only one of them used to be considered. +struct Ident { + Format format = Format::Unknown; + // The object's own machine, in mcpp's canonical arch spelling + // ("x86_64", "aarch64", "i686", …). Empty when unrecognised — a new + // machine type is not a corrupt file, and reporting it as one would be + // worse than saying nothing. + std::string arch; + bool is64 = false; +}; + +std::string_view format_name(Format f) { + switch (f) { + case Format::Elf: return "ELF"; + case Format::Pe: return "PE"; + case Format::MachO: return "Mach-O"; + case Format::Unknown: break; + } + return "unknown"; +} + +// Identify by magic. Never throws, never runs anything; an unreadable or +// truncated file simply comes back Unknown. +Ident identify(const std::filesystem::path& binary); + +// The DIRECT dependency names an object declares: +// ELF DT_NEEDED entries ("libstdc++.so.6") +// PE import + delay-import DLL names ("vcruntime140.dll") +// +// Names, in the file's own spelling and order, deduplicated. An object with +// no dynamic dependencies yields an empty list — which is a valid answer, not +// a failure, and the caller must not confuse the two (a fully static binary +// and an unparseable one had better not look alike). +std::expected, std::string> +needed_names(const std::filesystem::path& binary); + +// Is `name` provided by the target OS itself — i.e. must NOT be bundled? +// +// On ELF this is the manylinux allow-list, which `pack` already had. +// +// On PE the list is short and the reason is different: shipping a copy of a +// Windows component is not "extra weight", it is a broken program. The OS +// loader resolves `kernel32.dll` from the system directory whatever sits +// beside the executable — except when a DLL that sits beside it is loaded +// FIRST by name, at which point the process has two of something that must be +// unique. Microsoft's redistribution terms say the same thing from the other +// side. +// +// Note what is NOT here: `vcruntime140.dll` and `msvcp140.dll`. Those belong +// to the TOOLSET, not to Windows (see mcpp.build.distribution's PE branch), +// and whether they travel is a contract decision — the one thing this +// predicate must not quietly make for it. +bool is_system_lib(Format f, std::string_view name); + +} // namespace mcpp::pack::binfmt + +namespace mcpp::pack::binfmt { + +namespace detail { + +// A bounded, whole-file read. Executables are tens of megabytes at worst and +// the alternative is a seek-heavy parser that has to re-validate every offset +// against the file length anyway; with the bytes in hand, one bounds-checked +// accessor covers every read in this file. +std::optional slurp(const std::filesystem::path& p) { + std::error_code ec; + auto size = std::filesystem::file_size(p, ec); + if (ec) return std::nullopt; + // 512 MiB. A larger "executable" is not one, and refusing is better than + // allocating whatever a caller was handed. + if (size > (512ull << 20)) return std::nullopt; + std::ifstream is(p, std::ios::binary); + if (!is) return std::nullopt; + std::string buf(static_cast(size), '\0'); + is.read(buf.data(), static_cast(size)); + buf.resize(static_cast(is.gcount())); + return buf; +} + +// Little-endian integer at `off`, or nullopt if it would run off the end. +// EVERY read in this module goes through these: a malformed file is ordinary +// input (a truncated download, a text file named `.exe`), and the parser has +// to be total over it rather than trusting a length field it just read. +// +// FOUR CONCRETE FUNCTIONS, NOT ONE TEMPLATE, and the difference is not style. +// A function template in a module interface's non-exported namespace is a +// shape this codebase has been bitten by before (see hostflags.cppm's opening +// comment: an unused helper added to a module's anonymous namespace +// miscompiled a NEIGHBOURING function under clang + C++20 modules). The first +// version of this file used `template le(...)`, and clang +// misbehaved on BOTH of its targets at once while gcc was fine — a frontend +// segfault on Windows and a runtime segfault on macOS ARM64, from code that +// is clean under ASan+UBSan and correct as a clang module on x86_64 Linux. +// Whether the template was the cause is NOT established; what is established +// is that the failures are compiler-side, and that four short functions cost +// nothing. +std::optional le8(std::string_view b, std::size_t off) { + if (off >= b.size()) return std::nullopt; + return static_cast(static_cast(b[off])); +} + +std::optional le16(std::string_view b, std::size_t off) { + if (off + 2 > b.size()) return std::nullopt; + return static_cast( + static_cast(b[off]) + | (static_cast(static_cast(b[off + 1])) << 8)); +} + +std::optional le32(std::string_view b, std::size_t off) { + if (off + 4 > b.size()) return std::nullopt; + std::uint32_t v = 0; + for (int i = 3; i >= 0; --i) + v = (v << 8) | static_cast(b[off + static_cast(i)]); + return v; +} + +std::optional le64(std::string_view b, std::size_t off) { + if (off + 8 > b.size()) return std::nullopt; + std::uint64_t v = 0; + for (int i = 7; i >= 0; --i) + v = (v << 8) | static_cast(b[off + static_cast(i)]); + return v; +} + +// NUL-terminated string at `off`, bounded by the file end. +std::optional cstr(std::string_view b, std::size_t off) { + if (off >= b.size()) return std::nullopt; + auto end = b.find('\0', off); + if (end == std::string_view::npos) return std::nullopt; + return std::string(b.substr(off, end - off)); +} + +std::string elf_arch(std::uint16_t machine) { + switch (machine) { + case 0x03: return "i686"; // EM_386 + case 0x28: return "arm"; // EM_ARM + case 0x3E: return "x86_64"; // EM_X86_64 + case 0xB7: return "aarch64"; // EM_AARCH64 + case 0xF3: return "riscv64"; // EM_RISCV + case 0x15: return "ppc64"; // EM_PPC64 + case 0x08: return "mips"; // EM_MIPS + case 0x16: return "s390x"; // EM_S390 + case 0x102: return "loongarch64"; + default: return {}; + } +} + +std::string pe_arch(std::uint16_t machine) { + switch (machine) { + case 0x014c: return "i686"; // IMAGE_FILE_MACHINE_I386 + case 0x8664: return "x86_64"; // AMD64 + case 0xAA64: return "aarch64"; // ARM64 + case 0x01c4: return "arm"; // ARMNT + default: return {}; + } +} + +// ─── ELF ──────────────────────────────────────────────────────────────── +// +// DT_NEEDED lives in the .dynamic section, whose entries are (tag, value) +// pairs; the NEEDED value is a byte offset into the string table named by +// DT_STRTAB. DT_STRTAB is a VIRTUAL ADDRESS, not a file offset, so the PT_LOAD +// segments have to be walked to translate it — which is the one step that +// makes this more than "read a list". +std::expected, std::string> +elf_needed(std::string_view b) { + auto cls = le8(b, 4); + auto dat = le8(b, 5); + if (!cls || !dat) return std::unexpected("ELF header is truncated"); + if (*dat != 1) + return std::unexpected("big-endian ELF objects are not supported"); + const bool is64 = (*cls == 2); + if (*cls != 1 && *cls != 2) + return std::unexpected("ELF header declares an unknown class"); + + const std::size_t phoffAt = is64 ? 0x20 : 0x1C; + std::uint64_t phoff = 0; + if (is64) { + auto v = le64(b, phoffAt); + if (!v) return std::unexpected("ELF program header offset is truncated"); + phoff = *v; + } else { + auto v = le32(b, phoffAt); + if (!v) return std::unexpected("ELF program header offset is truncated"); + phoff = *v; + } + auto phentsize = le16(b, is64 ? 0x36 : 0x2A); + auto phnum = le16(b, is64 ? 0x38 : 0x2C); + if (!phentsize || !phnum) + return std::unexpected("ELF program header table is truncated"); + + struct Load { std::uint64_t vaddr, offset, filesz; }; + std::vector loads; + std::uint64_t dynOff = 0, dynSize = 0; + for (std::uint16_t i = 0; i < *phnum; ++i) { + const std::size_t ph = static_cast(phoff) + + static_cast(i) * *phentsize; + auto type = le32(b, ph); + if (!type) break; + auto rd = [&](std::size_t off64, std::size_t off32) + -> std::optional { + if (is64) return le64(b, ph + off64); + if (auto v = le32(b, ph + off32)) return *v; + return std::nullopt; + }; + auto offset = rd(0x08, 0x04); + auto vaddr = rd(0x10, 0x08); + auto filesz = rd(0x20, 0x10); + if (!offset || !vaddr || !filesz) continue; + if (*type == 1) loads.push_back({*vaddr, *offset, *filesz}); // PT_LOAD + if (*type == 2) { dynOff = *offset; dynSize = *filesz; } // PT_DYNAMIC + } + // No PT_DYNAMIC = a static executable. Zero dependencies is the answer, + // and it is a different answer from "could not be read". + if (dynSize == 0) return std::vector{}; + + auto vaddr_to_off = [&](std::uint64_t va) -> std::optional { + for (auto const& l : loads) + if (va >= l.vaddr && va - l.vaddr < l.filesz) + return l.offset + (va - l.vaddr); + return std::nullopt; + }; + + const std::size_t entSize = is64 ? 16 : 8; + std::vector neededOffsets; + std::optional strtabVa; + for (std::uint64_t at = dynOff; at + entSize <= dynOff + dynSize; + at += entSize) { + std::uint64_t tag = 0, val = 0; + if (is64) { + auto t = le64(b, static_cast(at)); + auto v = le64(b, static_cast(at) + 8); + if (!t || !v) break; + tag = *t; val = *v; + } else { + auto t = le32(b, static_cast(at)); + auto v = le32(b, static_cast(at) + 4); + if (!t || !v) break; + tag = *t; val = *v; + } + if (tag == 0) break; // DT_NULL + if (tag == 1) neededOffsets.push_back(val); // DT_NEEDED + if (tag == 5) strtabVa = val; // DT_STRTAB + } + if (neededOffsets.empty()) return std::vector{}; + if (!strtabVa) + return std::unexpected("ELF .dynamic has DT_NEEDED but no DT_STRTAB"); + auto strOff = vaddr_to_off(*strtabVa); + if (!strOff) + return std::unexpected("ELF DT_STRTAB address is in no PT_LOAD segment"); + + std::vector out; + for (auto n : neededOffsets) { + if (auto s = cstr(b, static_cast(*strOff + n)); + s && !s->empty()) + out.push_back(std::move(*s)); + } + return out; +} + +// ─── PE ───────────────────────────────────────────────────────────────── +// +// Two directories carry DLL names and BOTH are dependencies: the ordinary +// import table (directory 1), resolved by the loader before the process +// starts, and the delay-load table (directory 13), resolved at the first call +// through it. A delay-loaded DLL that is missing does not fail at startup — +// it fails later, somewhere in the program, which is strictly worse to debug. +// Leaving it out of the closure would produce exactly that. +std::expected, std::string> +pe_needed(std::string_view b) { + auto lfanew = le32(b, 0x3C); + if (!lfanew) return std::unexpected("PE: no e_lfanew"); + const std::size_t nt = *lfanew; + if (b.substr(nt, 4) != std::string_view("PE\0\0", 4)) + return std::unexpected("PE: no PE\\0\\0 signature at e_lfanew"); + + auto numSections = le16(b, nt + 6); + auto optSize = le16(b, nt + 20); + auto magic = le16(b, nt + 24); + if (!numSections || !optSize || !magic) + return std::unexpected("PE: headers are truncated"); + // 0x10b PE32, 0x20b PE32+. They differ only in where the data directories + // start — the extra 16 bytes are the 64-bit ImageBase and friends. + std::size_t dirsAt = 0; + if (*magic == 0x10b) dirsAt = nt + 24 + 96; + else if (*magic == 0x20b) dirsAt = nt + 24 + 112; + else return std::unexpected("PE: optional header magic is neither PE32 nor PE32+"); + auto numDirs = le32(b, dirsAt - 4); + if (!numDirs) return std::unexpected("PE: data directory count is truncated"); + + struct Section { std::uint32_t va, vsize, raw, rawSize; }; + std::vector
sections; + const std::size_t secAt = nt + 24 + *optSize; + for (std::uint16_t i = 0; i < *numSections; ++i) { + const std::size_t s = secAt + static_cast(i) * 40; + auto vsize = le32(b, s + 8); + auto va = le32(b, s + 12); + auto rawSz = le32(b, s + 16); + auto raw = le32(b, s + 20); + if (!vsize || !va || !rawSz || !raw) break; + sections.push_back({*va, *vsize, *raw, *rawSz}); + } + + auto rva_to_off = [&](std::uint32_t rva) -> std::optional { + for (auto const& s : sections) { + // A section's mapped size is VirtualSize, but a section whose + // VirtualSize is 0 (some linkers) still maps SizeOfRawData. + auto span = s.vsize ? s.vsize : s.rawSize; + if (rva >= s.va && rva - s.va < span) { + std::size_t off = s.raw + (rva - s.va); + if (off < b.size()) return off; + return std::nullopt; + } + } + return std::nullopt; + }; + + std::vector out; + auto push = [&out](const std::optional& name) { + if (!name || name->empty()) return; + for (auto const& seen : out) + if (seen == *name) return; + out.push_back(*name); + }; + + // Directory 1 — imports. 20-byte descriptors, terminated by an all-zero + // one; `Name` (offset 12) is an RVA to the DLL's ASCIIZ name. + if (*numDirs > 1) { + auto rva = le32(b, dirsAt + 1 * 8); + if (rva && *rva) { + if (auto at = rva_to_off(*rva)) { + for (std::size_t d = *at; ; d += 20) { + auto nameRva = le32(b, d + 12); + auto oft = le32(b, d); + auto ft = le32(b, d + 16); + if (!nameRva || !oft || !ft) break; + if (*nameRva == 0 && *oft == 0 && *ft == 0) break; + if (auto o = rva_to_off(*nameRva)) push(cstr(b, *o)); + } + } + } + } + + // Directory 13 — delay imports. 32-byte descriptors. `grAttrs` bit 0 + // (`dlattrRva`) says the fields are RVAs; the pre-VC7 form stored virtual + // ADDRESSES instead, and nothing produced in this century does that, so a + // descriptor without the bit is skipped rather than guessed at. + if (*numDirs > 13) { + auto rva = le32(b, dirsAt + 13 * 8); + if (rva && *rva) { + if (auto at = rva_to_off(*rva)) { + for (std::size_t d = *at; ; d += 32) { + auto attrs = le32(b, d); + auto nameRva = le32(b, d + 4); + if (!attrs || !nameRva) break; + if (*attrs == 0 && *nameRva == 0) break; + if ((*attrs & 1u) == 0) continue; + if (auto o = rva_to_off(*nameRva)) push(cstr(b, *o)); + } + } + } + } + return out; +} + +// PEP 600 / manylinux2014: assumed present on any target Linux glibc system. +constexpr std::string_view kElfSystem[] = { + "libc.so", "libm.so", "libdl.so", "libpthread.so", "librt.so", + "libutil.so", "libnsl.so", "libresolv.so", "libcrypt.so", + "libstdc++.so", "libgcc_s.so", "linux-vdso.so", "ld-linux", "libld-linux", +}; + +// Windows' own. Deliberately NOT including vcruntime140/msvcp140: those +// belong to the toolset, and whether they travel is `cxx_runtime`'s decision. +constexpr std::string_view kPeSystem[] = { + "ntdll.dll", "kernel32.dll", "kernelbase.dll", "user32.dll", "gdi32.dll", + "gdi32full.dll", "advapi32.dll", "shell32.dll", "shlwapi.dll", + "ole32.dll", "oleaut32.dll", "combase.dll", "comdlg32.dll", + "comctl32.dll", "rpcrt4.dll", "sechost.dll", "setupapi.dll", + "cfgmgr32.dll", "version.dll", "winmm.dll", "imm32.dll", "uxtheme.dll", + "dwmapi.dll", "userenv.dll", "psapi.dll", "iphlpapi.dll", "netapi32.dll", + "ws2_32.dll", "wsock32.dll", "mswsock.dll", "crypt32.dll", "bcrypt.dll", + "bcryptprimitives.dll", "ncrypt.dll", "secur32.dll", "wintrust.dll", + "powrprof.dll", "winspool.drv", "msimg32.dll", "opengl32.dll", + "glu32.dll", "dxgi.dll", "d3d9.dll", "d3d11.dll", "d3d12.dll", + "dbghelp.dll", "hid.dll", "avrt.dll", "mfplat.dll", "dnsapi.dll", + "profapi.dll", "ucrtbase.dll", "msvcrt.dll", "win32u.dll", +}; + +} // namespace detail + +Ident identify(const std::filesystem::path& binary) { + Ident id; + auto buf = detail::slurp(binary); + if (!buf || buf->size() < 8) return id; + std::string_view b{*buf}; + + if (b.starts_with(std::string_view("\x7f" "ELF", 4))) { + id.format = Format::Elf; + auto cls = detail::le8(b, 4); + id.is64 = cls && *cls == 2; + if (auto m = detail::le16(b, 18)) id.arch = detail::elf_arch(*m); + return id; + } + if (b.starts_with("MZ")) { + // MZ alone is a DOS stub; a PE needs the signature e_lfanew points at. + // Saying "PE" for a file that has none would send the caller into a + // parser that cannot succeed. + if (auto lfanew = detail::le32(b, 0x3C)) { + if (b.substr(*lfanew, 4) == std::string_view("PE\0\0", 4)) { + id.format = Format::Pe; + if (auto m = detail::le16(b, *lfanew + 4)) + id.arch = detail::pe_arch(*m); + if (auto magic = detail::le16(b, *lfanew + 24)) + id.is64 = (*magic == 0x20b); + return id; + } + } + return id; + } + // Mach-O, both endiannesses and the fat wrapper. Recognised but not + // parsed: macOS packaging still asks the loader, and a caller that lands + // here deserves a message naming the format rather than "unknown". + for (auto magic : {std::string_view("\xcf\xfa\xed\xfe", 4), + std::string_view("\xce\xfa\xed\xfe", 4), + std::string_view("\xfe\xed\xfa\xcf", 4), + std::string_view("\xfe\xed\xfa\xce", 4), + std::string_view("\xca\xfe\xba\xbe", 4)}) { + if (b.starts_with(magic)) { id.format = Format::MachO; return id; } + } + return id; +} + +std::expected, std::string> +needed_names(const std::filesystem::path& binary) { + auto buf = detail::slurp(binary); + if (!buf) + return std::unexpected(std::format("cannot read '{}'", binary.string())); + std::string_view b{*buf}; + switch (identify(binary).format) { + case Format::Elf: return detail::elf_needed(b); + case Format::Pe: return detail::pe_needed(b); + case Format::MachO: + return std::unexpected( + "Mach-O dependency reading is not implemented; macOS packaging " + "resolves the closure through the loader instead"); + case Format::Unknown: break; + } + return std::unexpected(std::format( + "'{}' is not an ELF, PE or Mach-O object", binary.string())); +} + +bool is_system_lib(Format f, std::string_view name) { + std::string lower(name); + for (auto& c : lower) + c = static_cast(std::tolower(static_cast(c))); + if (f == Format::Pe) { + // API sets: `api-ms-win-crt-runtime-l1-1-0.dll` and friends are + // forwarders resolved by the loader against the OS. They have no file + // to copy on most systems and copying one would be wrong anyway. + if (lower.starts_with("api-ms-") || lower.starts_with("ext-ms-")) + return true; + for (auto known : detail::kPeSystem) + if (lower == known) return true; + return false; + } + if (f == Format::Elf) { + for (auto prefix : detail::kElfSystem) + if (lower.starts_with(prefix)) return true; + return false; + } + return false; +} + +} // namespace mcpp::pack::binfmt diff --git a/src/pack/pack.cppm b/src/pack/pack.cppm index d93d5623..765558d8 100644 --- a/src/pack/pack.cppm +++ b/src/pack/pack.cppm @@ -1,4 +1,19 @@ -// mcpp.pack — bundle a built binary into a self-contained release tarball. +// mcpp.pack — bundle a built binary into a self-contained release archive. +// +// TWO OUTPUT FAMILIES, ONE PIPELINE. An ELF/Mach-O artifact becomes a +// `.tar.gz` whose libraries live in `lib/` and are reached through a rewritten +// RUNPATH; a PE artifact becomes a `.zip` whose DLLs sit BESIDE the .exe, +// because that is where the Win32 loader looks and PE has no rpath to rewrite. +// Same contract, same modes, different mechanism — see +// .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §4. +// +// The PE path runs on ANY host. That is not a portability nicety: the ELF +// path derives its closure by running the artifact under +// LD_TRACE_LOADED_OBJECTS, which is why it can cross neither an OS nor an +// architecture, and the `#if defined(_WIN32)` refusal that used to sit at the +// top of `run()` was that limitation surfacing rather than a missing branch. +// mcpp.pack.binfmt reads the import table instead, so a Linux box packaging a +// Windows build is simply what happens when nothing has to be executed. // // See docs/35-pack-design.md for the full design. Three modes: // Static full musl static, no PT_INTERP / RUNPATH @@ -22,7 +37,9 @@ export module mcpp.pack; import std; import mcpp.build.loader_contract; import mcpp.config; +import mcpp.pack.binfmt; import mcpp.pack.host_requirements; +import mcpp.pack.zip; import mcpp.platform; import mcpp.platform.xlings; import mcpp.manifest; @@ -38,6 +55,34 @@ struct Options { Format format = Format::Tar; std::filesystem::path output; // empty = derive from manifest std::string targetTriple; // empty = host + // Where a dependency NAME may be resolved to a file. + // + // Only used where the closure is read STATICALLY (PE): on ELF the loader + // hands back resolved paths and no search is performed here. Deliberately + // never the target's own system directories — a DLL that resolves only + // there is the target's to provide, and copying one is a broken program + // rather than a heavier one. + std::vector depSearchDirs; + // The TOOLCHAIN's own runtime directory: libstdc++'s for gcc, the MSVC + // toolset's `VC\Redist\MSVC\\\Microsoft.VC*.CRT\` for cl. + // Searched ONLY under the toolchain-coupled contract — see make_plan. + std::vector toolchainRuntimeDirs; + // Does the RESOLVED C++ runtime contract require the toolchain's own + // runtime to travel WITH the artifact — i.e. `cxx_runtime = + // "toolchain-coupled"`? + // + // `pack` used to be unable to see the contract at all (design §4.3): it + // reached compile and link FLAGS and stopped there, so the step that + // decides which files actually travel had no idea what had been promised. + // On ELF the `ldd` closure happened to agree with it; on PE nothing did. + // + // A BOOL rather than `dist::Contract`, deliberately. Only one of the + // three values changes anything here, so the enum would be three states + // where the decision has two — and carrying it across this module + // boundary crashed the clang 20.1.7 frontend outright (see run_pe). The + // caller resolves the contract; this is the one bit of it that packaging + // acts on. + bool carryToolchainRuntime = false; }; // Resolved plan — all paths absolute, all decisions baked in. @@ -47,7 +92,7 @@ struct Plan { std::filesystem::path builtBinary; // mcpp build artefact std::string binaryName; // basename(builtBinary) std::filesystem::path stagingRoot; // target/dist// - std::filesystem::path tarballPath; // staging_root parent / .tar.gz + std::filesystem::path archivePath; // …/.tar.gz | .zip std::string packageName; std::string packageVersion; std::string triple; // e.g. "x86_64-linux-musl" @@ -59,6 +104,12 @@ struct Plan { // What the TARGET machine must provide. Derived once, in make_plan, from // the same predicate `mcpp publish` uses — see mcpp.pack.host_requirements. std::vector hostRequirements; + // Is the artifact a PE? Read from the FILE, not inferred from the triple — + // the file is the thing being packaged, and a triple is a request. + bool targetIsPe = false; + // The search set the PE closure resolves names against, after the + // contract has had its say (see make_plan). + std::vector searchDirs; }; struct Error { std::string message; }; @@ -138,29 +189,35 @@ namespace detail { // errors. Naming the namespace gives every helper module linkage and // sidesteps the rule entirely. -// Default tarball name: `--[-].tar.gz`. +// Default archive name: `--[-].`. // Mode suffix only for non-default modes so adjacent builds of different // modes don't stomp each other in target/dist/. -std::string default_tarball_name(std::string_view name, std::string_view version, - std::string_view triple, Mode mode) +// +// The EXTENSION follows the artifact, not the host: a Windows package is a +// `.zip` whoever built it, and a `.tar.gz` full of DLLs is a package most +// Windows users cannot open without installing something first. +std::string default_archive_name(std::string_view name, std::string_view version, + std::string_view triple, Mode mode, bool pe) { + std::string_view ext = pe ? ".zip" : ".tar.gz"; auto sfx = mode_tarball_suffix(mode); if (sfx.empty()) - return std::format("{}-{}-{}.tar.gz", name, version, triple); - return std::format("{}-{}-{}-{}.tar.gz", name, version, triple, sfx); + return std::format("{}-{}-{}{}", name, version, triple, ext); + return std::format("{}-{}-{}-{}{}", name, version, triple, sfx, ext); } -// Strip the `.tar.gz` (or `.tgz`) suffix from a tarball filename to get -// the canonical wrapper-directory name. The result names both the disk -// staging dir and the top-level entry inside the archive — keeping the -// two in lock-step makes click-to-extract behave the way users expect. -std::string wrapper_dirname_from_tarball(const std::filesystem::path& tarball) { - auto name = tarball.filename().string(); - for (auto suffix : {std::string_view{".tar.gz"}, std::string_view{".tgz"}}) { +// Strip the archive suffix to get the canonical wrapper-directory name. The +// result names both the disk staging dir and the top-level entry inside the +// archive — keeping the two in lock-step makes click-to-extract behave the +// way users expect (and on Windows, Explorer's "extract here" too). +std::string wrapper_dirname_from_archive(const std::filesystem::path& archive) { + auto name = archive.filename().string(); + for (auto suffix : {std::string_view{".tar.gz"}, std::string_view{".tgz"}, + std::string_view{".zip"}}) { if (name.ends_with(suffix)) return name.substr(0, name.size() - suffix.size()); } // No recognised compression suffix — fall back to the bare stem. - return tarball.stem().string(); + return archive.stem().string(); } } // namespace detail @@ -228,23 +285,74 @@ make_plan(const mcpp::manifest::Manifest& manifest, mode_cli_name(opts.mode), names)}); } + // WHAT THE FILE IS, not what the triple asked for. `--target` states an + // intention; the artifact on disk is the thing being packaged, and when + // the two disagree it is the file that has to be believed. (They can + // disagree for real: `[pack] default_mode = "static"` re-prepares the + // build with a different target after the first one has already run.) + p.targetIsPe = + mcpp::pack::binfmt::identify(builtBinary).format + == mcpp::pack::binfmt::Format::Pe; + + // THE CONTRACT REACHES PACKAGING. Until now it stopped at the compile and + // link flags, so the step that decides which files travel could not see + // what the artifact had promised (design §4.3). + // + // toolchain-coupled the toolchain's own runtime travels WITH the + // artifact — so its directory joins the search set. + // host-coupled the target provides it — so that directory stays + // OUT, and a vcruntime140.dll sitting in the + // toolset is not silently swept into the package. + // self-contained there is nothing to carry. + // + // A mode that bundles NOTHING cannot honour a contract that requires + // files to travel. `pack` already refuses the mirror-image contradiction + // (a bundle carrying its own libc cannot consume a host capability), and + // this grows from the same root: a contract with no executor is a promise + // the build prints and the package quietly drops. + const bool modeBundlesNothing = + opts.mode == Mode::None || opts.mode == Mode::Static; + if (opts.carryToolchainRuntime && modeBundlesNothing) { + return std::unexpected(Error{std::format( + "cxx_runtime = \"toolchain-coupled\" and --mode {} contradict each " + "other.\n" + " The contract says the toolchain's C++ runtime travels WITH this " + "artifact; --mode {}\n" + " bundles nothing, so it would ship a program that cannot start " + "anywhere the toolchain\n" + " is not already installed.\n" + " use: --mode vendored (carry it), or cxx_runtime = " + "\"host-coupled\" / \"self-contained\"\n" + " in [build] if the target is expected to provide the runtime " + "itself.", + mode_cli_name(opts.mode), mode_cli_name(opts.mode))}); + } + + // The artifact's own directory is always searched: whatever the build + // staged beside it (the toolchain-coupled CRT, a dependency's DLL) is by + // definition part of what it runs with. + p.searchDirs.push_back(builtBinary.parent_path()); + for (auto const& d : opts.depSearchDirs) p.searchDirs.push_back(d); + if (opts.carryToolchainRuntime) + for (auto const& d : opts.toolchainRuntimeDirs) p.searchDirs.push_back(d); + auto distDir = projectRoot / "target" / "dist"; if (opts.output.empty()) { - p.tarballPath = distDir / detail::default_tarball_name( - p.packageName, p.packageVersion, p.triple, opts.mode); + p.archivePath = distDir / detail::default_archive_name( + p.packageName, p.packageVersion, p.triple, opts.mode, p.targetIsPe); } else if (!opts.output.has_parent_path()) { // `-o name.tar.gz` (bare filename) → place in target/dist/. // `-o ./name.tar.gz` or `-o sub/name.tar.gz` → use as-is. // `-o /abs/path.tar.gz` → use as-is. - p.tarballPath = distDir / opts.output; + p.archivePath = distDir / opts.output; } else { - p.tarballPath = opts.output; + p.archivePath = opts.output; } - // Derive the staging dir from the tarball stem so the in-archive + // Derive the staging dir from the archive stem so the in-archive // wrapper directory and the on-disk staging dir share one name — // matches what GUI extractors create on click and what `tar -xzf` // produces on the CLI. - p.stagingRoot = distDir / detail::wrapper_dirname_from_tarball(p.tarballPath); + p.stagingRoot = distDir / detail::wrapper_dirname_from_archive(p.archivePath); p.includeGlobs = manifest.packConfig.include; p.excludeGlobs = manifest.packConfig.exclude; @@ -608,6 +716,66 @@ void copy_if_exists(const std::filesystem::path& src, std::filesystem::copy_options::overwrite_existing, ec); } +// ─── PE: the closure, read rather than executed ───────────────────────── +// +// BFS over the import tables, resolving each name against `searchDirs`. A +// name that resolves NOWHERE is deliberately not an error: on a Linux host +// `kernel32.dll` has no file to find, and on a Windows host it would resolve +// only in the system directory, which is not searched. Both are the same +// answer — the target provides it — and both are correct. +std::vector +pe_closure(const std::filesystem::path& binary, + std::span searchDirs, + const std::vector& forceBundle) +{ + namespace bf = mcpp::pack::binfmt; + std::vector out; + std::set seen; // lowercased: PE names are not + std::vector queue{binary}; + + auto lower = [](std::string_view s) { + std::string l(s); + std::ranges::transform(l, l.begin(), + [](unsigned char c) { return std::tolower(c); }); + return l; + }; + + while (!queue.empty()) { + auto current = queue.back(); + queue.pop_back(); + auto names = bf::needed_names(current); + if (!names) continue; // unreadable: nothing to add + for (auto const& name : *names) { + auto key = lower(name); + if (!seen.insert(key).second) continue; + // `[pack] force_bundle` is the escape hatch, and it has to reach + // the SYSTEM list too — on ELF it always did. Shipping a Windows + // component is normally a broken program rather than a heavier + // one, so this is a decision a human has to make explicitly; when + // they have, mcpp does not know better than them. + if (bf::is_system_lib(bf::Format::Pe, name) + && !soname_matches(name, forceBundle)) + continue; + std::error_code ec; + for (auto const& dir : searchDirs) { + auto cand = dir / name; + if (!std::filesystem::is_regular_file(cand, ec)) continue; + out.push_back({name, cand}); + // Transitive: a bundled DLL brings its own imports, and a + // closure that stops at depth one ships a package that starts + // failing one link further in. + queue.push_back(cand); + break; + } + } + } + std::sort(out.begin(), out.end(), + [](const ResolvedDep& a, const ResolvedDep& b) { + return a.soname < b.soname; + }); + return out; +} + std::expected make_tarball(const std::filesystem::path& stagingRoot, const std::filesystem::path& tarballPath) @@ -631,30 +799,151 @@ make_tarball(const std::filesystem::path& stagingRoot, } // namespace detail +namespace detail { + +// The PE half of `run`. Flat layout, deliberately: the Win32 loader resolves +// a DLL from the directory of the executable, so `bin/` + `lib/` would need a +// mechanism PE does not have. Extract-and-double-click is also what a Windows +// user expects, and it is what the earlier design already specified +// (.agents/docs/2026-05-19-pack-windows-design.md). +// +// No patchelf step and no wrapper script: "put the DLLs next to the exe" IS +// the relocation rule on this format, which is why the row for it in the +// design's layering table reads "no operation". +// ⚠️ THIS FUNCTION ONCE CRASHED THE COMPILER, and the shape it crashed on is +// worth not reintroducing. +// +// clang 20.1.7 targeting the MSVC ABI — the pinned Windows toolchain — +// segfaulted (0xC0000005) while compiling this module interface, with no +// diagnostic, on every Windows job at once. The first version carried three +// module-boundary constructs that the rest of this file does not: +// +// 1. a scoped enum from ANOTHER module as a defaulted member of an +// EXPORTED struct (`dist::Contract cxxRuntime = …` in `Options`) +// 2. a ranges projection naming a member of an IMPORTED type +// (`std::ranges::sort(entries, {}, &zip::Entry::name)`) +// 3. `std::span` across the module boundary into zip::write +// +// All three were removed together, so WHICH one it was is not established — +// stating otherwise would be a guess dressed as a finding. What is +// established: the crash is reproducible only on that toolchain, and this +// file has a documented history of the same class (see the `detail` +// namespace comment above, and hostflags.cppm on a neighbouring function +// being miscompiled by an unrelated addition). Each replacement is also +// simpler than what it replaced, so nothing is being paid for the avoidance. +std::expected +run_pe(const Plan& plan) +{ + std::error_code ec; + std::filesystem::remove_all(plan.stagingRoot, ec); + std::filesystem::create_directories(plan.stagingRoot, ec); + if (ec) return std::unexpected(Error{std::format( + "cannot create staging '{}': {}", plan.stagingRoot.string(), ec.message())}); + + auto stagedExe = plan.stagingRoot / plan.binaryName; + std::filesystem::copy_file(plan.builtBinary, stagedExe, + std::filesystem::copy_options::overwrite_existing, ec); + if (ec) return std::unexpected(Error{std::format( + "copy binary failed: {}", ec.message())}); + + copy_if_exists(plan.projectRoot / "README.md", plan.stagingRoot); + copy_if_exists(plan.projectRoot / "LICENSE", plan.stagingRoot); + if (!plan.hostRequirements.empty()) { + std::ofstream out(plan.stagingRoot / std::filesystem::path(kFileName)); + if (!out) return std::unexpected(Error{std::format( + "cannot write {} into the bundle", kFileName)}); + out << render(plan.hostRequirements); + } + + std::vector deps; + if (plan.opts.mode != Mode::None && plan.opts.mode != Mode::Static) { + // `vendored` and `self-contained` collect the same set here, and that + // is a property of the PLATFORM rather than a simplification. + // `self-contained` on ELF means "ship the loader too"; on PE there is + // no loader to ship and kernel32.dll and friends may not be + // redistributed — a process that loaded a private copy of one would + // have two of something that must be unique. So the ceiling on + // "everything" is the same for both modes: every dependency mcpp is + // ALLOWED to carry. + deps = pe_closure(stagedExe, plan.searchDirs, plan.forceBundleLibs); + for (auto const& d : deps) { + if (soname_matches(d.soname, plan.alsoSkipLibs) + && !soname_matches(d.soname, plan.forceBundleLibs)) + continue; + auto dst = plan.stagingRoot / d.soname; + // The build may already have staged it beside the artifact, in + // which case source and destination are the same file. + std::error_code cec; + if (std::filesystem::equivalent(d.path, dst, cec)) continue; + std::filesystem::copy_file(d.path, dst, + std::filesystem::copy_options::overwrite_existing, cec); + if (cec) return std::unexpected(Error{std::format( + "failed to copy {} → {}: {}", + d.path.string(), dst.string(), cec.message())}); + } + } + + if (plan.opts.format != Format::Tar) return {}; + + std::vector entries; + const auto wrapper = plan.stagingRoot.filename().string(); + for (auto const& e : + std::filesystem::recursive_directory_iterator(plan.stagingRoot, ec)) { + if (!e.is_regular_file(ec)) continue; + auto rel = std::filesystem::relative(e.path(), plan.stagingRoot, ec); + if (ec) continue; + entries.push_back({ + wrapper + "/" + rel.generic_string(), + e.path(), + e.path().filename() == plan.builtBinary.filename(), + }); + } + // Deterministic order: a directory iteration order that leaks into an + // archive is how two identical builds get two different checksums. + // + // Plain `std::sort` with an explicit comparator, not + // `std::ranges::sort(entries, {}, &Entry::name)`. A ranges projection + // naming a member of an IMPORTED type is one of three module-boundary + // shapes this function used to carry, and together they crashed the clang + // 20.1.7 frontend (0xC0000005, no diagnostic) on every Windows job. See + // the note at the top of run_pe. + std::sort(entries.begin(), entries.end(), + [](const mcpp::pack::zip::Entry& a, + const mcpp::pack::zip::Entry& b) { return a.name < b.name; }); + if (auto r = mcpp::pack::zip::write(plan.archivePath, entries); !r) + return std::unexpected(Error{r.error()}); + return {}; +} + +} // namespace detail + std::expected run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) { + // A PE package is produced the same way on every host, because nothing in + // that path executes the artifact. This is the branch the `#if + // defined(_WIN32)` refusal used to occupy — and it was never really about + // the host: `LD_TRACE_LOADED_OBJECTS` cannot trace a PE from Linux either. + if (plan.targetIsPe) return detail::run_pe(plan); + #if defined(_WIN32) - // `mcpp pack` is not yet supported on Windows. - // - // The current implementation relies on POSIX-only tools: - // - LD_TRACE_LOADED_OBJECTS=1 (ELF dynamic linker trick; no equivalent - // on Windows PE/COFF) - // - ldd / patchelf (Linux ELF tools; not available on Windows) - // - tar -czf (GNU tar; not universally present on Windows) - // - // For CI-produced Windows zip packages, use the ci-windows.yml workflow - // which zips the MSVC/Clang build output directly. - // - // Windows PE packaging (DLL collection + zip) is planned. - // See .agents/docs/2026-05-19-pack-windows-design.md for the design. - (void)plan; + // A NON-PE artifact on a Windows host: a cross build to Linux or macOS. + // The closure below asks the dynamic linker by running the binary, which + // this machine cannot do — so say that, rather than reporting a platform + // limitation that no longer exists for the case a Windows user is + // actually likely to hit. (void)cfg; - return std::unexpected(Error{ - "error: `mcpp pack` is not yet supported on Windows.\n" - " Use the CI workflow (ci-windows.yml) to produce Windows zip packages.\n" - " Windows PE packaging (DLL collection + zip) is planned." - }); + return std::unexpected(Error{std::format( + "cannot package a {} artifact from a Windows host.\n" + " The dependency closure for that format is resolved by running " + "the artifact under\n" + " the target's own dynamic linker, which this machine has no way " + "to do. Package it\n" + " on the target OS, or build for Windows (`--target " + "x86_64-pc-windows-msvc`), which\n" + " needs no such step.", + std::string(mcpp::pack::binfmt::format_name( + mcpp::pack::binfmt::identify(plan.builtBinary).format)))}); #else using namespace detail; std::error_code ec; @@ -829,7 +1118,7 @@ run(const Plan& plan, const mcpp::config::GlobalConfig& cfg) // 5. Output. if (plan.opts.format == Format::Tar) { - if (auto r = make_tarball(plan.stagingRoot, plan.tarballPath); !r) + if (auto r = make_tarball(plan.stagingRoot, plan.archivePath); !r) return r; } return {}; diff --git a/src/pack/pipeline.cppm b/src/pack/pipeline.cppm index 5596cc2f..8dad8f23 100644 --- a/src/pack/pipeline.cppm +++ b/src/pack/pipeline.cppm @@ -11,6 +11,8 @@ export module mcpp.pack.pipeline; import std; import mcpp.build.prepare; import mcpp.build.backend; +import mcpp.build.distribution; +import mcpp.build.flags; import mcpp.build.ninja; import mcpp.build.plan; import mcpp.config; @@ -55,7 +57,15 @@ export int build_and_pack(Options opts, bool modeFromUser) { // Re-derive target triple: if mode is Static we force the musl // triple even when the manifest's [pack].default_mode bumped us // here after `prepare_build` ran with the host toolchain. - if (opts.mode == mcpp::pack::Mode::Static && ctx->tc.targetTriple.find("-musl") == std::string::npos) { + // + // ...but NOT over a target the user asked for. `--mode static` on its own + // has always meant "the musl-static ELF", and that stays; `--mode static + // --target x86_64-windows-gnu` used to silently become a Linux build, + // which was invisible while PE packaging did not exist and is a wrong + // answer now that it does. An explicit `--target` is an instruction. + if (opts.mode == mcpp::pack::Mode::Static + && opts.targetTriple.empty() + && ctx->tc.targetTriple.find("-musl") == std::string::npos) { // Need to re-prepare the build with the musl target. mcpp::build::BuildOverrides ov2; ov2.target_triple = "x86_64-linux-musl"; @@ -100,6 +110,31 @@ export int build_and_pack(Options opts, bool modeFromUser) { mcpp::fetcher::make_bootstrap_progress_callback()); if (!cfg) { mcpp::ui::error(cfg.error().message); return 4; } + // ─── What the build promised, and where its runtime lives ──────── + // + // The C++ runtime contract has been resolved since the flags were + // computed; `pack` simply had no way to see it (design §4.3), so on PE + // nothing enforced it and on ELF the `ldd` closure agreed with it by + // luck. Reading the RESOLVED value rather than the manifest string is the + // point: a request that was downgraded (a per-role self-contained on + // /MD, say) must not make the package behave as though it had been + // honoured. + { + const auto flags = mcpp::build::compute_flags(ctx->plan); + opts.carryToolchainRuntime = + flags.contractByRole[static_cast( + mcpp::build::dist::Role::Distributable)] + == mcpp::build::dist::Contract::ToolchainCoupled; + opts.toolchainRuntimeDirs = ctx->plan.toolchain.linkRuntimeDirs; + // Where a third-party dependency's shared library may be found. Both + // channels, because they answer for different things: the runtime + // library dirs are what `mcpp run` puts on the loader's path, and the + // link intent's search dirs are what a dependency package declared. + opts.depSearchDirs = ctx->plan.runtimeLibraryDirs; + for (auto const& d : ctx->plan.linkIntent.runtimeSearchDirs) + opts.depSearchDirs.push_back(d); + } + // ─── Build the plan + run ──────────────────────────────────────── auto plan = mcpp::pack::make_plan(ctx->manifest, *cfg, opts, mainBinary, ctx->projectRoot, ctx->tc.targetTriple, @@ -121,7 +156,7 @@ export int build_and_pack(Options opts, bool modeFromUser) { auto pathCtx = mcpp::fetcher::make_path_ctx(&*cfg, ctx->projectRoot); auto outPath = (opts.format == mcpp::pack::Format::Tar) - ? plan->tarballPath : plan->stagingRoot; + ? plan->archivePath : plan->stagingRoot; mcpp::ui::status("Packed", mcpp::ui::shorten_path(outPath, pathCtx)); return 0; } diff --git a/src/pack/zip.cppm b/src/pack/zip.cppm new file mode 100644 index 00000000..0460a580 --- /dev/null +++ b/src/pack/zip.cppm @@ -0,0 +1,219 @@ +// mcpp.pack.zip — write a .zip, from any host, with no external tool. +// +// WHY NOT SHELL OUT. The rest of `pack` produces `.tar.gz` by running `tar`, +// and the obvious symmetry would be to run something for zip too. There is no +// such something that exists everywhere: GNU tar (Linux) cannot write zip, +// `zip(1)` is frequently absent, PowerShell's `Compress-Archive` is +// Windows-only, and bsdtar is Windows-and-macOS-mostly. The earlier design +// (.agents/docs/2026-05-19-pack-windows-design.md) proposed `Compress-Archive` +// precisely because it assumed pack ran ON Windows — the assumption that +// mcpp.pack.binfmt exists to remove. A packer that only works on the target's +// own OS is the thing being fixed; picking an archiver that only exists there +// would reintroduce it one layer down. +// +// WHY STORED, NOT DEFLATED. Entries are written with method 0 (stored). That +// is a real cost — a release zip is roughly the size of its contents — and it +// is the honest trade for now: a DEFLATE encoder is the only part of this +// file that could produce an archive which UNPACKS WRONG rather than failing +// loudly, and mcpp has no zlib to borrow one from. Adding compression later +// changes only `write()`; the archive layout below does not move. +// +// Deterministic by construction: no timestamps are read (every entry gets a +// fixed DOS date), and entries are written in the order given. Two packs of +// the same tree produce byte-identical archives, which is what makes a +// published checksum mean anything. + +export module mcpp.pack.zip; + +import std; + +export namespace mcpp::pack::zip { + +struct Entry { + // Path INSIDE the archive, always with forward slashes — the ZIP spec + // says so (4.4.17.1), and a backslash here is what makes an archive + // written on Windows extract into one flat file with a strange name + // everywhere else. + std::string name; + std::filesystem::path source; + // Only meaningful to POSIX extractors; Windows ignores it. Set for the + // executable so `unzip` on a Linux box does not hand back a non-runnable + // file — which matters because a Windows artifact is routinely inspected + // (and cross-tested under wine) from Linux. + bool executable = false; +}; + +// Write `entries` to `out`, creating parent directories as needed. +// +// Takes the vector by reference rather than `std::span`. The +// span is the better interface and it is not worth a compiler crash: passing +// one across this module boundary was one of three constructs present when +// clang 20.1.7 (MSVC ABI) segfaulted compiling the consumer — see the note at +// mcpp.pack's `run_pe`. There is exactly one caller. +std::expected +write(const std::filesystem::path& out, const std::vector& entries); + +// CRC-32 (IEEE), exposed for tests: an archive whose CRCs are wrong extracts +// with a warning on some tools and silently on others, so this is worth +// pinning against known vectors rather than against itself. +std::uint32_t crc32(std::string_view data); + +} // namespace mcpp::pack::zip + +namespace mcpp::pack::zip { + +namespace detail { + +const std::array& crc_table() { + static const auto table = [] { + std::array t{}; + for (std::uint32_t i = 0; i < 256; ++i) { + std::uint32_t c = i; + for (int k = 0; k < 8; ++k) + c = (c & 1) ? (0xEDB88320u ^ (c >> 1)) : (c >> 1); + t[i] = c; + } + return t; + }(); + return table; +} + +void put16(std::string& out, std::uint16_t v) { + out.push_back(static_cast(v & 0xFF)); + out.push_back(static_cast((v >> 8) & 0xFF)); +} + +void put32(std::string& out, std::uint32_t v) { + for (int i = 0; i < 4; ++i) + out.push_back(static_cast((v >> (8 * i)) & 0xFF)); +} + +// A fixed DOS timestamp: 1980-01-01 00:00:00, the earliest the format can +// express. Reading the file's mtime instead would make the archive depend on +// when it was built, which is the usual reason two identical builds produce +// two different checksums. +constexpr std::uint16_t kDosTime = 0; +constexpr std::uint16_t kDosDate = 0x0021; // (1980-1980)<<9 | 1<<5 | 1 + +// External attributes. The high 16 bits are the Unix mode when the +// version-made-by byte says Unix (3); the low byte is the DOS attribute set. +std::uint32_t external_attrs(bool executable) { + std::uint32_t mode = executable ? 0100755u : 0100644u; + return mode << 16; +} + +std::optional slurp(const std::filesystem::path& p) { + std::ifstream is(p, std::ios::binary); + if (!is) return std::nullopt; + std::ostringstream ss; + ss << is.rdbuf(); + return ss.str(); +} + +} // namespace detail + +std::uint32_t crc32(std::string_view data) { + auto const& table = detail::crc_table(); + std::uint32_t c = 0xFFFFFFFFu; + for (unsigned char b : data) + c = table[(c ^ b) & 0xFF] ^ (c >> 8); + return c ^ 0xFFFFFFFFu; +} + +std::expected +write(const std::filesystem::path& out, const std::vector& entries) { + std::error_code ec; + if (out.has_parent_path()) + std::filesystem::create_directories(out.parent_path(), ec); + + struct Central { + std::string name; + std::uint32_t crc = 0, size = 0, offset = 0; + bool executable = false; + }; + std::vector central; + central.reserve(entries.size()); + + std::string blob; + for (auto const& e : entries) { + auto data = detail::slurp(e.source); + if (!data) + return std::unexpected(std::format( + "cannot read '{}' for the archive", e.source.string())); + // 4 GiB is the point where ZIP64 becomes mandatory. mcpp does not + // write ZIP64, so refuse rather than emit an archive whose sizes have + // silently wrapped — a truncated field here produces an archive that + // extracts to garbage instead of failing. + if (data->size() > 0xFFFFFFFFull || blob.size() > 0xFFFFFFFFull) + return std::unexpected(std::format( + "'{}' exceeds the 4 GiB limit of a non-ZIP64 archive", + e.source.string())); + + Central c; + c.name = e.name; + c.crc = crc32(*data); + c.size = static_cast(data->size()); + c.offset = static_cast(blob.size()); + c.executable = e.executable; + + detail::put32(blob, 0x04034b50); // local file header + detail::put16(blob, 20); // version needed (2.0) + detail::put16(blob, 0); // flags — no data descriptor + detail::put16(blob, 0); // method 0: stored + detail::put16(blob, detail::kDosTime); + detail::put16(blob, detail::kDosDate); + detail::put32(blob, c.crc); + detail::put32(blob, c.size); // compressed == uncompressed + detail::put32(blob, c.size); + detail::put16(blob, static_cast(c.name.size())); + detail::put16(blob, 0); // extra field length + blob += c.name; + blob += *data; + + central.push_back(std::move(c)); + } + + const auto centralAt = static_cast(blob.size()); + for (auto const& c : central) { + detail::put32(blob, 0x02014b50); // central directory header + // version made by: 3 (Unix) << 8 | 20. The Unix half is what makes the + // mode bits in `external attributes` meaningful to an extractor. + detail::put16(blob, (3 << 8) | 20); + detail::put16(blob, 20); // version needed + detail::put16(blob, 0); + detail::put16(blob, 0); // stored + detail::put16(blob, detail::kDosTime); + detail::put16(blob, detail::kDosDate); + detail::put32(blob, c.crc); + detail::put32(blob, c.size); + detail::put32(blob, c.size); + detail::put16(blob, static_cast(c.name.size())); + detail::put16(blob, 0); // extra + detail::put16(blob, 0); // comment + detail::put16(blob, 0); // disk number + detail::put16(blob, 0); // internal attrs + detail::put32(blob, detail::external_attrs(c.executable)); + detail::put32(blob, c.offset); + blob += c.name; + } + const auto centralSize = static_cast(blob.size()) - centralAt; + + detail::put32(blob, 0x06054b50); // end of central directory + detail::put16(blob, 0); + detail::put16(blob, 0); + detail::put16(blob, static_cast(central.size())); + detail::put16(blob, static_cast(central.size())); + detail::put32(blob, centralSize); + detail::put32(blob, centralAt); + detail::put16(blob, 0); // comment length + + std::ofstream os(out, std::ios::binary | std::ios::trunc); + if (!os) return std::unexpected(std::format( + "cannot write '{}'", out.string())); + os.write(blob.data(), static_cast(blob.size())); + if (!os) return std::unexpected(std::format( + "failed writing '{}'", out.string())); + return {}; +} + +} // namespace mcpp::pack::zip diff --git a/src/platform/elf_runtime.cppm b/src/platform/elf_runtime.cppm index 1be390cb..d4af87a9 100644 --- a/src/platform/elf_runtime.cppm +++ b/src/platform/elf_runtime.cppm @@ -697,7 +697,12 @@ RuntimeVerdict validate_runtime_artifact( const RuntimeResolution& resolution, bool hostLibsAllowed) { RuntimeVerdict verdict; - const bool isGlibc = binding.runtimeId.starts_with("glibc@"); + // Provider dispatch, not a prefix match. These are ELF/glibc physics — + // PT_INTERP, DT_RUNPATH, a private loader — and a `ucrt@…` identity is + // not a glibc that is missing, it is a different provider whose runtime + // this validator has no rules for. + const bool isGlibc = + mcpp::platform::runtime::runtime_provider(binding.runtimeId) == "glibc"; if constexpr (!mcpp::platform::is_linux) { verdict.diagnostics.push_back( "runtime physics: non-Linux platform; ELF/glibc rules are not applicable"); diff --git a/src/platform/runtime_binding.cppm b/src/platform/runtime_binding.cppm index 94550ed2..c727cb74 100644 --- a/src/platform/runtime_binding.cppm +++ b/src/platform/runtime_binding.cppm @@ -23,6 +23,26 @@ struct RuntimeBinding { std::string arch; // Provider-native runtime identity (`glibc@...`, `macos_sdk@...`, // `ucrt@...`). Only Linux additionally projects this into `libc`. + // + // THE PROVIDERS ARE NOT ISOMORPHIC, and treating them as such is the + // mistake this note exists to prevent: + // + // glibc@2.39 binds a PAYLOAD. The headers and the .so are both in it, + // and patchelf makes the artifact actually run on that + // copy. It is a runtime BINDING. + // ucrt@10.0… declares a FLOOR. `ucrtbase.dll` is an OS component from + // Win10 on; it cannot be swapped and should not be + // shipped, so mcpp's `windows-sdk` payload deliberately + // carries only half of ucrt (headers + import libraries) + // and no `Universal CRT Redistributable`. The identity + // says which API surface was compiled against, not which + // binary will be loaded. + // + // Both belong in this field — they are the same QUESTION ("which C + // runtime is this artifact built for") answered by different providers — + // but a consumer that assumes the glibc shape will try to resolve a + // payload that was never supposed to exist. Ask `runtime_provider()` + // before acting on the value. std::string runtimeId; std::string contractHash; std::optional loader; @@ -75,6 +95,42 @@ struct RuntimeBinding { bool hermetic() const { return loader.has_value(); } }; +// The PROVIDER half of a runtime identity: "glibc" from "glibc@2.39", "ucrt" +// from "ucrt@10.0.26100.0". Empty when there is no identity at all. +// +// Exists so consumers DISPATCH instead of pattern-matching. Every reader of +// `runtimeId` used to spell `starts_with("glibc@")`, which reads as "is this +// glibc" and behaves as "is this the only provider I have ever seen" — so the +// day a second provider appeared, each of those sites silently classified it +// as "no runtime identity" rather than "an identity with no rules here". The +// two are not the same thing, and only one of them is worth reporting. +std::string_view runtime_provider(std::string_view runtimeId) { + auto at = runtimeId.find('@'); + if (at == std::string_view::npos) return {}; + return runtimeId.substr(0, at); +} + +// Attach the Windows C runtime identity to an already-resolved snapshot, and +// re-derive the contract hash so it takes effect. +// +// WHY IT IS A SECOND STEP rather than part of resolve_runtime_binding(): the +// SDK version is a property of the TOOLCHAIN, and the binding is deliberately +// resolved BEFORE any toolchain — the post-install fixup is itself a consumer +// of the binding, so resolving a toolchain first would let directory order +// choose a libc and only then discover what the project selected (#392). +// Windows is the one platform where a fact flows the other way, so it flows +// back explicitly instead of reordering the two. +// +// WHAT IT BUYS: the SDK version enters `contractHash`, which is part of the +// toolchain fingerprint, which keys the build cache. Two SDKs produce two +// caches. Without this the version axis simply stopped existing one layer +// below the compiler — the exact defect this identity was added to close. +// +// Idempotent, and a no-op for an empty version: a Windows box with no SDK +// found still builds (selection UX must work there), it just has nothing to +// declare. +void bind_windows_ucrt(RuntimeBinding& binding, std::string_view sdkVersion); + namespace detail { std::string hash_contract(std::string_view data) { @@ -398,6 +454,20 @@ resolve_runtime_binding( return out; } +void bind_windows_ucrt(RuntimeBinding& binding, std::string_view sdkVersion) { + if (sdkVersion.empty()) return; + auto identity = std::format("ucrt@{}", sdkVersion); + if (binding.runtimeId == identity) return; // idempotent + binding.runtimeId = std::move(identity); + // Deliberately NOT projected into `libc`: that field is the Linux-only + // private-libc payload, read by the loader/patchelf machinery, and there + // is no ucrt payload for it to name. `runtime_binding.cppm`'s own comment + // on `libc` says "Only Linux additionally projects this" — this is that + // sentence being true rather than merely written down. + binding.contractHash = + detail::hash_contract(detail::canonical_contract(binding)); +} + std::string serialize_runtime_binding(const RuntimeBinding& binding) { nlohmann::json j; j["schema"] = binding.schema; diff --git a/src/toolchain/lifecycle.cppm b/src/toolchain/lifecycle.cppm index a982503d..e5bac541 100644 --- a/src/toolchain/lifecycle.cppm +++ b/src/toolchain/lifecycle.cppm @@ -382,10 +382,15 @@ first_undeletable(const std::filesystem::path& root) { } void msvc_warn_if_sdk_missing(const mcpp::toolchain::msvc::MsvcInstallation& inst) { - auto roots = mcpp::toolchain::msvc::sibling_sdk_roots(inst.clPath); - if (auto sdk = mcpp::toolchain::msvc::find_windows_sdk(roots)) { + // The SAME resolution the build will perform, not a second one shaped + // like it. Install-time and build-time disagreeing about which SDK this + // toolset uses is worse than not reporting at all: the line printed here + // is what the user will believe. + auto choice = mcpp::toolchain::msvc::resolve_sdk_for(inst.clPath); + if (choice.sdk) { std::println(" windows sdk: {} ({})", - sdk->version, sdk->root.string()); + choice.sdk->version, choice.sdk->root.string()); + if (!choice.note.empty()) mcpp::ui::info("note", choice.note); return; } mcpp::ui::warning( @@ -756,14 +761,12 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, mcpp::fetcher::Fetcher fetcher(cfg); mcpp::fetcher::InstallProgressHandler progress; - // Ensure sysroot dependencies (glibc, linux-headers) are installed. - // These are required for C library + kernel headers during compilation. - // Decided by the TARGET, not the payload name: musl targets are - // self-contained; PE targets (native mingw AND the Linux-hosted - // cross) bring their own CRT; Windows/macOS hosts never need the - // Linux sysroot. Mirrors the guard on prepare.cppm's first-run install. - if (!spec->target.is_musl() && !spec->target.is_pe() - && !mcpp::platform::is_windows && !mcpp::platform::is_macos) { + // Ensure sysroot dependencies (glibc, linux-headers) are installed: + // the C library and kernel headers a glibc-targeting compile needs. + // The rule itself lives in registry.cppm — prepare's first-run + // install asks the same question and used to answer it in its own + // words, with one term missing. + if (mcpp::toolchain::needs_linux_sysroot_payloads(spec->target)) { for (auto dep : {"xim:glibc", "xim:linux-headers"}) { mcpp::log::verbose("toolchain", std::format("installing dep: {}", dep)); auto depPayload = fetcher.resolve_xpkg_path(dep, /*autoInstall=*/true, &progress); @@ -799,26 +802,18 @@ export int toolchain_install(const mcpp::config::GlobalConfig& cfg, // VC/Tools/MSVC//bin/Hostx64/x64, and there is nothing to // patchelf on a PE toolchain. if (spec->family == mcpp::toolchain::Family::Msvc) { - // NOT `payload->root` — that is a GUESS, and it guesses wrong - // here. resolve_xpkg_path calls the version directory the root - // only when it directly contains bin/ include/ lib/; otherwise it - // descends into a lone subdirectory. An installed msvc payload - // has exactly one entry, `VC/`, so the "root" comes back as - // …/14.44.35207/VC and the toolset then looks like it is missing. - // - // The location is not something to infer: it is (store, name, - // version), and all three are known here. resolve_xpkg_path above - // is what INSTALLS; this is what says where. - auto verDir = mcpp::xlings::paths::xim_tool( - mcpp::config::make_xlings_env(cfg), pkg.ximName, pkg.ximVersion); - auto inst = mcpp::toolchain::msvc::installation_at( - verDir, pkg.ximVersion); + // Where the payload IS — not `payload->root`, which is the + // fetcher's guess and guesses wrong here. That rule and its + // reasons live in `resolve_managed_msvc`, because a build needs + // exactly the same one and the two used to spell it out + // separately. `resolve_xpkg_path` above is what INSTALLS; this is + // what says where. + auto inst = mcpp::toolchain::resolve_managed_msvc( + mcpp::config::make_xlings_env(cfg), pkg); if (!inst) { mcpp::ui::error(std::format( - "msvc payload installed at '{}', but no cl.exe under " - "VC/Tools/MSVC/{} — the payload is not what this version " - "claims to be", - verDir.string(), pkg.ximVersion)); + "{} — the payload is not what this version claims to be", + inst.error())); return 1; } msvc_print_detected(*inst, "Installed"); diff --git a/src/toolchain/model.cppm b/src/toolchain/model.cppm index aa8da0a6..e6660064 100644 --- a/src/toolchain/model.cppm +++ b/src/toolchain/model.cppm @@ -9,6 +9,34 @@ export namespace mcpp::toolchain { enum class CompilerId { Unknown, GCC, Clang, MSVC }; +// WHERE A COMPILER CAME FROM. Two values, and there will only ever be two. +// +// `Managed` an xlings payload the manifest named. The answer is in the +// manifest; the machine only decides whether it has been +// downloaded yet. +// `SystemMsvc` probed on this machine. The answer depends on what happens to +// be installed here. +// +// THE SECOND ONE IS NOT A GENERAL CAPABILITY, and reading it as one is a +// mistake this comment exists to prevent. mcpp is built on xlings, a +// user-space OS, and the whole design is to drive host dependencies to a +// minimum — there is no `gcc@system`, deliberately. `msvc@system` is a +// concession to ONE platform: Visual Studio is very often already installed +// and cannot always be redistributed, so refusing to use it would cost more +// than it buys. (The bare, family-less `system` spec is a different thing +// again: a deliberate escape hatch to the PATH compiler.) +// +// Lives in the data model rather than in msvc.cppm so the toolchain-spec +// side (registry) and the located-compiler side (msvc) name the SAME axis. +// They used to answer it independently, in 26 scattered branches, which is +// how "is this managed" and "is this a system MSVC" came to be asked with +// different predicates in the same build. +enum class Origin { Managed, SystemMsvc }; + +inline std::string_view origin_name(Origin o) { + return o == Origin::SystemMsvc ? "system" : "managed"; +} + // Fine-grained sysroot paths derived from xpkgs payloads. // When populated, flags are assembled from these paths instead of --sysroot. // One environment variable a toolchain needs at tool-invocation time. @@ -73,6 +101,27 @@ struct Toolchain { // GCC/libc++ answer 20; MSVC answers 20 from cl 19.38 (VS 2022 17.8, // microsoft/STL#3977) and 23 below that. int importStdMinLevel = 0; + // The Windows SDK this toolchain compiles and links against + // ("10.0.26100.0"), once resolved. Empty everywhere else — and empty on + // Windows too when no SDK was found, which detection tolerates so that + // toolchain SELECTION still works on an SDK-less box. + // + // Recorded rather than re-derived because two different questions read it + // and they must not be able to disagree: the compile environment + // (INCLUDE/LIB) and the runtime identity (`ucrt@`, which enters + // the runtime contract hash). Deriving the second from a second search is + // how the SDK came to have no identity in the first place. + std::string windowsSdkVersion; + // Something about HOW this toolchain was resolved that the user has to be + // told, but which is not a failure. Non-empty ⇒ the caller MUST surface it. + // + // Today's only producer is the Windows SDK axis: a managed toolset binds + // the SDK that came with it, so a `WindowsSdkDir` in the environment is + // ignored — and an override that is ignored SILENTLY is indistinguishable + // from one that did not exist. That is the failure shape this whole round + // kept finding: "it did not happen" and "it succeeded" producing identical + // output. + std::string resolutionNote; std::string label() const { return std::format("{} {} ({})", compiler_name(), version, targetTriple); diff --git a/src/toolchain/msvc.cppm b/src/toolchain/msvc.cppm index 60369e1f..c34b4e0d 100644 --- a/src/toolchain/msvc.cppm +++ b/src/toolchain/msvc.cppm @@ -165,6 +165,44 @@ std::optional find_windows_sdk( std::vector sibling_sdk_roots(const std::filesystem::path& clPath); +// Which origin produced this cl.exe, answered by where the binary lives. +// +// Not a guess: an xlings store is a specific directory layout that mcpp +// itself created, and `xpkgs_from_compiler` recognises it or does not. A +// compiler outside every store came from the machine. +Origin origin_of(const std::filesystem::path& clPath); + +// The Windows SDK for a located cl.exe, chosen BY ORIGIN. +// +// This is the second of the three Windows axes (see +// .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §2), and +// until now it had no identity at all: `find_windows_sdk()` scanned, and +// whatever the scan reached first won — for both origins. +// +// MANAGED (`msvc@`) the SDK is a DECLARED dependency of the +// toolset, installed into the same store. It is +// bound, not searched: `WindowsSdkDir` / +// `WindowsSdkVersion` do not participate, +// because a pin that the environment can +// overwrite is not a pin. Two machines building +// the same manifest must see the same headers. +// SYSTEM (`msvc@system`) the machine's own SDK, and a machine's things +// can only be found by looking. Unchanged: the +// declared `WindowsSdkDir` still outranks the +// scan, exactly as it does for VSINSTALLDIR. +// +// `note` is non-empty when something the user could have expected to matter +// did not, and the caller MUST surface it. There are exactly two: +// - a managed toolset ignoring a `WindowsSdkDir` that was set +// - a managed toolset with NO SDK payload beside it, falling back to the +// machine's — which works, and is not reproducible, so it says so +struct SdkChoice { + std::optional sdk; + Origin origin = Origin::Managed; + std::string note; +}; +SdkChoice resolve_sdk_for(const std::filesystem::path& clPath); + // True only when BOTH halves of a usable MSVC C++ setup are present: the // STL's std module source AND the Windows SDK. // @@ -625,6 +663,46 @@ constexpr std::string_view sdk_lib_arch = "arm64"; constexpr std::string_view sdk_lib_arch = "x64"; #endif +namespace { + +// Highest version dir under `root/Include` that actually carries the UCRT +// headers; `want` (from WindowsSdkVersion) wins if it is one of them. +std::optional pick_sdk_in(const std::filesystem::path& root, + std::string_view want) { + std::error_code ec; + auto inc = root / "Include"; + if (!std::filesystem::is_directory(inc, ec)) return std::nullopt; + auto usable = [&](const std::filesystem::path& verDir, const std::string& v) { + return std::filesystem::exists(verDir / "ucrt" / "corecrt.h", ec) + && std::filesystem::exists( + root / "Lib" / v / "um" / sdk_lib_arch / "kernel32.lib", ec); + }; + std::string best; + for (auto& e : std::filesystem::directory_iterator(inc, ec)) { + if (!e.is_directory(ec)) continue; + auto v = e.path().filename().string(); + if (!usable(e.path(), v)) continue; + if (!want.empty() && v == want) return WindowsSdk{root, v}; + if (v > best) best = v; + } + if (best.empty()) return std::nullopt; + return WindowsSdk{root, best}; +} + +// `WindowsSdkVersion` without its trailing backslash (vcvars exports one; it +// is not part of the directory name). Empty when unset. +std::string declared_sdk_version() { + std::string want; + if (auto* v = std::getenv("WindowsSdkVersion"); v && *v) { + want = v; + while (!want.empty() && (want.back() == '\\' || want.back() == '/')) + want.pop_back(); + } + return want; +} + +} // namespace + std::optional find_windows_sdk( std::span extraRoots) { // Highest version dir under `root/Include` that actually carries the UCRT @@ -648,54 +726,97 @@ std::optional find_windows_sdk( // // kernel32.lib is the right sentinel: every link needs it, and unlike the // ucrt libs it is not spread across the SDK's optional pieces. - auto pick = [](const std::filesystem::path& root, - std::string_view want) -> std::optional { - std::error_code ec; - auto inc = root / "Include"; - if (!std::filesystem::is_directory(inc, ec)) return std::nullopt; - auto usable = [&](const std::filesystem::path& verDir, - const std::string& v) { - return std::filesystem::exists(verDir / "ucrt" / "corecrt.h", ec) - && std::filesystem::exists( - root / "Lib" / v / "um" / sdk_lib_arch / "kernel32.lib", ec); - }; - std::string best; - for (auto& e : std::filesystem::directory_iterator(inc, ec)) { - if (!e.is_directory(ec)) continue; - auto v = e.path().filename().string(); - if (!usable(e.path(), v)) continue; - if (!want.empty() && v == want) return WindowsSdk{root, v}; - if (v > best) best = v; - } - if (best.empty()) return std::nullopt; - return WindowsSdk{root, best}; - }; + // + // The version-selection rule itself lives in `pick_sdk_in`, because the + // MANAGED origin needs the same rule applied to a different (and much + // shorter) list of roots — see `resolve_sdk_for`. // 1. Declared: WindowsSdkDir (+ WindowsSdkVersion). vcvars exports both; // WindowsSdkVersion carries a trailing backslash there, which is not // part of the directory name. - std::string want; - if (auto* v = std::getenv("WindowsSdkVersion"); v && *v) { - want = v; - while (!want.empty() && (want.back() == '\\' || want.back() == '/')) - want.pop_back(); - } + const std::string want = declared_sdk_version(); if (auto* dir = std::getenv("WindowsSdkDir"); dir && *dir) { - if (auto s = pick(std::filesystem::path{dir}, want)) return s; + if (auto s = pick_sdk_in(std::filesystem::path{dir}, want)) return s; } // 2. Roots the caller knows about (managed toolset's own store). for (const auto& root : extraRoots) - if (auto s = pick(root, want)) return s; + if (auto s = pick_sdk_in(root, want)) return s; // 3. The conventional absolute install roots. for (const char* base : {"C:\\Program Files (x86)\\Windows Kits\\10", "C:\\Program Files\\Windows Kits\\10"}) { - if (auto s = pick(std::filesystem::path{base}, want)) return s; + if (auto s = pick_sdk_in(std::filesystem::path{base}, want)) return s; } return std::nullopt; } +Origin origin_of(const std::filesystem::path& clPath) { + return mcpp::xlings::paths::xpkgs_from_compiler(clPath) + ? Origin::Managed : Origin::SystemMsvc; +} + +SdkChoice resolve_sdk_for(const std::filesystem::path& clPath) { + SdkChoice out; + out.origin = origin_of(clPath); + auto siblings = sibling_sdk_roots(clPath); + + if (out.origin == Origin::SystemMsvc) { + // A machine's things can only be found by looking, and a declared + // WindowsSdkDir is still the most specific answer available. + // `siblings` is empty here by construction (a system cl.exe is in no + // store); passed through so the two origins share one call shape. + out.sdk = find_windows_sdk(siblings); + return out; + } + + // MANAGED. The SDK arrived as a declared dependency of this toolset and + // sits in the same store — so it is looked up, not searched for, and + // nothing in the environment gets a vote. See the design doc's §2.1 for + // what searching cost: a half-unpacked payload outranked the machine's own + // complete SDK because its version number was higher, every TU compiled, + // and the build died at LNK1104 with nothing in the log naming the SDK. + // + // `sibling_sdk_roots` returns newest first, so the first hit is also the + // one the highest-version rule would have chosen. + for (auto const& root : siblings) { + // `want` deliberately empty: WindowsSdkVersion is the same declaration + // channel as WindowsSdkDir, and letting it pick among payloads is the + // same override wearing a smaller hat. + if ((out.sdk = pick_sdk_in(root, {}))) break; + } + + if (out.sdk) { + const char* dir = std::getenv("WindowsSdkDir"); + const char* ver = std::getenv("WindowsSdkVersion"); + if ((dir && *dir) || (ver && *ver)) { + out.note = std::format( + "WindowsSdkDir/WindowsSdkVersion in the environment " + "({}) is ignored: this build pins a managed MSVC toolset, and " + "the Windows SDK it was installed with ({} at {}) is part of " + "that pin. Use msvc@system if the machine's SDK is what you " + "want.", + dir && *dir ? dir : ver, + out.sdk->version, out.sdk->root.string()); + } + return out; + } + + // No SDK payload beside the toolset. Falling back keeps the build working + // — but quietly falling back is precisely how the version axis stopped + // meaning anything one layer down, so it is reported. + out.sdk = find_windows_sdk({}); + out.note = out.sdk + ? std::format( + "no windows-sdk payload was found beside this managed MSVC " + "toolset, so the machine's SDK ({} at {}) is being used. That " + "makes the build depend on this machine — reinstall the toolset " + "(`mcpp toolchain install msvc `) to pull its own SDK.", + out.sdk->version, out.sdk->root.string()) + : std::string{}; + return out; +} + bool has_usable_msvc() { #if defined(_WIN32) // Both, deliberately — see the declaration for why either half alone is @@ -928,13 +1049,19 @@ std::expected enrich_toolchain_from_cl(Toolchain& tc) { // detection working (selection UX on SDK-less boxes); the build path // errors with guidance when envOverrides is empty. // - // A managed toolset carries its own SDK as an xlings dependency, and the - // compiler's own path says which store to look in — so the two halves of - // a pinned toolchain stay together without anything being configured. - auto extraSdkRoots = sibling_sdk_roots(tc.binaryPath); - if (auto sdk = find_windows_sdk(extraSdkRoots)) { - tc.envOverrides = build_env_for_cl(tc.binaryPath, parsed->second, *sdk); + // BY ORIGIN, not by search. A managed toolset carries its own SDK as an + // xlings dependency and the compiler's own path says which store it is in, + // so the two halves of a pinned toolchain stay together — and stay + // together even when the environment says otherwise, which is the part a + // search could not give. `msvc@system` keeps the search chain: the + // machine's things can only be found by looking. + auto choice = resolve_sdk_for(tc.binaryPath); + if (choice.sdk) { + tc.envOverrides = build_env_for_cl(tc.binaryPath, parsed->second, + *choice.sdk); + tc.windowsSdkVersion = choice.sdk->version; } + if (!choice.note.empty()) tc.resolutionNote = choice.note; // The toolset's own redistributable CRT, in the same field gcc uses for // libstdc++ — so `mcpp run` puts it on PATH exactly the way it puts a diff --git a/src/toolchain/post_install.cppm b/src/toolchain/post_install.cppm index 394755c2..6004c87b 100644 --- a/src/toolchain/post_install.cppm +++ b/src/toolchain/post_install.cppm @@ -16,6 +16,7 @@ import mcpp.config; import mcpp.libs.json; import mcpp.log; import mcpp.platform; +import mcpp.platform.runtime_binding; import mcpp.toolchain.linkmodel; import mcpp.toolchain.registry; import mcpp.ui; @@ -384,9 +385,20 @@ select_glibc_payload_lib(const std::filesystem::path& glibcRoot, std::string_view runtimeId) { constexpr std::string_view prefix = "glibc@"; if (!runtimeId.starts_with(prefix)) { - return std::unexpected(std::format( - "selected RuntimeBinding '{}' is not a glibc payload identity " - "(expected glibc@)", runtimeId)); + // Name the PROVIDER that was found, not just the one that was + // wanted. `ucrt@10.0.26100.0` is a perfectly valid runtime identity + // that this fixup has nothing to do with (there is no ucrt payload to + // bind to — it is an OS component), and "not a glibc payload + // identity" reads as "malformed" for it. + auto provider = mcpp::platform::runtime::runtime_provider(runtimeId); + return std::unexpected(provider.empty() + ? std::format( + "selected RuntimeBinding '{}' is not a glibc payload identity " + "(expected glibc@)", runtimeId) + : std::format( + "selected RuntimeBinding '{}' names the '{}' runtime " + "provider; the glibc payload fixup does not apply to it", + runtimeId, provider)); } auto version = runtimeId.substr(prefix.size()); if (version.empty() || version == "." || version == ".." diff --git a/src/toolchain/registry.cppm b/src/toolchain/registry.cppm index 0dd21fb9..dc23e0af 100644 --- a/src/toolchain/registry.cppm +++ b/src/toolchain/registry.cppm @@ -17,6 +17,7 @@ export module mcpp.toolchain.registry; import std; import mcpp.platform; +import mcpp.platform.xlings; import mcpp.toolchain.clang; import mcpp.toolchain.compat; import mcpp.toolchain.gcc; @@ -148,6 +149,50 @@ bool spec_matches_payload(const ToolchainSpec& def, // and older mechanism.) bool is_system_toolchain(const ToolchainSpec& spec); +// The same question as `is_system_toolchain`, asked as the AXIS it belongs to +// (mcpp.toolchain.model). Both spellings exist because the predicate reads +// better at a site that is deciding one thing, and the enum reads better at a +// site that dispatches — but there is one derivation, so they cannot drift. +Origin origin_of(const ToolchainSpec& spec); + +// Resolve a MANAGED msvc payload to its installation record. +// +// `prepare` (a build) and `lifecycle` (install / default / remove) both need +// this, and both used to spell it out: derive the version directory from +// (store, name, version) rather than trusting the fetcher's `root` guess — +// which descends into a lone subdirectory and therefore lands one level too +// deep for an msvc payload, whose only entry is `VC/` — then call +// `installation_at`, then format the same error. Three copies of one rule, +// and the comment explaining WHY the fetcher's guess is wrong existed in only +// one of them. +// +// `identifyVersion = false` skips running cl.exe for its banner; callers that +// only need to know whether a toolset is there should pass false. +std::expected +resolve_managed_msvc(const mcpp::xlings::Env& env, + const XimToolchainPackage& pkg, + bool identifyVersion = true); + +// Does installing a toolchain FOR THIS TARGET additionally need the Linux +// sysroot payloads (`xim:glibc` + `xim:linux-headers`)? +// +// THE SINGLE DERIVATION. It was two, and they were not equivalent while a +// comment on one of them said "mirrors the guard on the other": +// +// lifecycle !musl && !pe && !windows-host && !macos-host +// prepare !macos-host && !windows-host && !musl +// +// The PE term was missing from the second. It happens to be unreachable today +// (first-run never selects a PE target on Linux), which is what let the +// divergence sit there — a latent difference between two spellings of one +// rule is exactly the state that becomes a bug the moment either side moves. +// +// Decided by the TARGET, not the payload name: musl targets are +// self-contained, PE targets (native MinGW and the Linux-hosted cross alike) +// bring their own CRT, and a non-Linux host never needs a Linux sysroot at +// all. +bool needs_linux_sysroot_payloads(const triple::Triple& target); + // Can THIS host serve that target — is there an installable payload for the // (host, target) pair? Empty target = host target, always serviceable. // @@ -235,10 +280,62 @@ parse_toolchain_spec(std::string compilerArg, else spec.family = Family::Gcc; spec.version = std::move(norm->version); spec.target = std::move(norm->target); + + // `@system` IS NOT A GENERAL SPELLING, and refusing it here is the point. + // + // mcpp is built on xlings, a user-space OS, and the whole design drives + // host dependencies to a minimum: a toolchain comes from a payload the + // manifest names, so every machine compiles with the same compiler. + // `msvc@system` is a concession to ONE platform — Visual Studio is very + // often already installed and cannot always be redistributed — not a + // capability the other families are missing. + // + // It used to parse and then fail somewhere else entirely, as + // `xim:gcc@system` → "no such package", which sends the reader looking + // for a version that was never going to exist. A spec that cannot mean + // anything should be rejected where it is read, by name, with the two + // things it might have meant spelled out. + if (spec.version == "system" && spec.family != Family::Msvc) { + return std::unexpected(std::format( + "'{}@system' is not a toolchain spelling: only msvc has a system " + "origin, because Visual Studio is often already installed and " + "cannot always be redistributed.\n" + " mcpp installs every other toolchain itself, so that each " + "machine builds with the same one:\n" + " {}@ pin a payload (mcpp toolchain list --available {})\n" + " system the PATH compiler, whatever it is — an escape " + "hatch, and it takes no family", + family_name(spec.family), family_name(spec.family), + family_name(spec.family))); + } if (norm->changed) spec.compatHint = std::move(norm->hint); return spec; } +Origin origin_of(const ToolchainSpec& spec) { + return is_system_toolchain(spec) ? Origin::SystemMsvc : Origin::Managed; +} + +std::expected +resolve_managed_msvc(const mcpp::xlings::Env& env, + const XimToolchainPackage& pkg, + bool identifyVersion) { + // NOT the fetcher's `root`: that field is its guess at where the useful + // tree starts, and it descends into a lone subdirectory when the version + // dir has no bin/ include/ lib/. An msvc payload's only entry is `VC/`, + // so the guess lands exactly one level too deep. (store, name, version) + // is known — compose it. + auto verDir = mcpp::xlings::paths::xim_tool(env, pkg.ximName, pkg.ximVersion); + // The package version IS the toolset directory name, so cl.exe is derived + // rather than searched for: nothing here can silently pick a different + // toolset, which is the whole reason the version axis exists. + if (auto inst = msvc::installation_at(verDir, pkg.ximVersion, identifyVersion)) + return *inst; + return std::unexpected(std::format( + "msvc payload at '{}' has no cl.exe under VC/Tools/MSVC/{}", + verDir.string(), pkg.ximVersion)); +} + void print_compat_hint(const ToolchainSpec& spec) { if (spec.compatHint.empty()) return; compat::print_hint_once(spec.compatHint); @@ -437,6 +534,11 @@ bool is_system_toolchain(const ToolchainSpec& spec) { && (spec.version.empty() || spec.version == "system"); } +bool needs_linux_sysroot_payloads(const triple::Triple& target) { + if constexpr (!mcpp::platform::is_linux) return false; + return !target.is_musl() && !target.is_pe(); +} + bool host_can_serve(const triple::Triple& target) { if (target.empty()) return true; // host target diff --git a/src/version.cppm b/src/version.cppm index 76e7199c..13ca3351 100644 --- a/src/version.cppm +++ b/src/version.cppm @@ -31,6 +31,6 @@ import std; export namespace mcpp { -inline constexpr std::string_view MCPP_VERSION = "2026.8.16.3"; +inline constexpr std::string_view MCPP_VERSION = "2026.8.17.1"; } // namespace mcpp diff --git a/tests/e2e/240_pack_pe_zip_cross.sh b/tests/e2e/240_pack_pe_zip_cross.sh new file mode 100755 index 00000000..17684c96 --- /dev/null +++ b/tests/e2e/240_pack_pe_zip_cross.sh @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# requires: mingw-cross python3 +# 240_pack_pe_zip_cross.sh — packaging a Windows program FROM LINUX. +# +# This is the acceptance criterion for §4 of +# .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md, and it is +# written so that running it on Windows would prove nothing: +# +# "在 Linux 上为 Windows 产物打出 zip,内含正确 DLL 闭包 +# — 跨 OS 是这条的全部意义;同 OS 打包证明不了" +# +# `mcpp pack` used to refuse Windows with `#if defined(_WIN32)`, and the +# reason given was that the tools were POSIX-only. The real reason was one +# layer down: the dependency closure came from +# +# LD_TRACE_LOADED_OBJECTS=1 '' +# +# which RUNS the artifact — so it could cross neither an OS nor an +# architecture, and no amount of porting `tar` would have helped. Reading the +# import table instead (mcpp.pack.binfmt) removes both limits at once, and +# this test is what says that actually happened rather than that the guard was +# deleted. +set -e + +TMP=$(mktemp -d) +trap "rm -rf $TMP" EXIT +cd "$TMP" + +"$MCPP" new winpack > /dev/null +cd winpack +cat > src/main.cpp <<'EOF' +#include +int main() { std::printf("winpack ok\n"); return 0; } +EOF +rm -f src/winpack.cppm + +cat > mcpp.toml <<'EOF' +[package] +name = "winpack" +version = "0.1.0" + +# `force_bundle` reaching a SYSTEM name is what makes the next assertion +# positive rather than vacuous — see the comment at the msvcrt.dll check. +[pack.bundle-project] +force_bundle = ["msvcrt.dll"] +EOF + +"$MCPP" build --target x86_64-windows-gnu > build.log 2>&1 || { cat build.log; exit 1; } +EXE="$(find target/x86_64-windows-gnu -name 'winpack.exe' | head -1)" +[[ -n "$EXE" ]] || { echo "FAIL: no PE was produced"; exit 1; } + +# The artifact really is a PE, checked here rather than assumed from the +# triple: everything below is about reading THAT file. +python3 - "$EXE" <<'PY' +import struct, sys +d = open(sys.argv[1], 'rb').read() +assert d[:2] == b'MZ', "not an MZ image" +lfanew, = struct.unpack_from(' "$BINDIR/msvcrt.dll" + +"$MCPP" pack --target x86_64-windows-gnu > pack.log 2>&1 || { cat pack.log; exit 1; } + +ZIP="$(find target/dist -name '*.zip' | head -1)" +[[ -n "$ZIP" ]] || { + echo "FAIL: no .zip produced — a Windows package is a zip whoever built it" + ls -R target/dist || true + cat pack.log + exit 1 +} +# And NOT a tarball: the extension follows the artifact, not the host. +[[ -z "$(find target/dist -name '*.tar.gz' | head -1)" ]] || { + echo "FAIL: produced a .tar.gz for a Windows target"; exit 1; } + +python3 - "$ZIP" <<'PY' +import sys, zipfile +z = zipfile.ZipFile(sys.argv[1]) + +# An INDEPENDENT reader. mcpp writes this archive itself (no host has a zip +# tool that exists everywhere), so "our writer agrees with our reader" would +# be worth nothing. +bad = z.testzip() +assert bad is None, f"corrupt entry: {bad}" + +names = z.namelist() +tops = {n.split('/')[0] for n in names} +assert len(tops) == 1, f"archive has no single wrapper directory: {tops}" +wrapper = tops.pop() + +exe = f"{wrapper}/winpack.exe" +assert exe in names, f"the executable is missing: {names}" + +# FLAT, beside the .exe. On PE that is not a layout preference — it is the +# relocation mechanism: the Win32 loader resolves a DLL from the directory of +# the executable, and there is no rpath to point anywhere else. +dll = f"{wrapper}/msvcrt.dll" +assert dll in names, ( + "msvcrt.dll was not bundled. It is named in the EXE's import table and in " + f"[pack] force_bundle, so this is the closure reader failing: {names}") +assert z.read(dll) == b'MZ-not-a-real-dll', "bundled the wrong file" + +# The other half, which the assertion above cannot give: the closure must +# EXCLUDE Windows' own. A parser that read nothing at all would also produce +# an archive with no kernel32.dll in it — but it could not have produced the +# msvcrt.dll above, so the two together are decisive. +lower = [n.lower() for n in names] +for sysdll in ("kernel32.dll", "ntdll.dll", "ucrtbase.dll"): + assert not any(n.endswith(sysdll) for n in lower), ( + f"{sysdll} was bundled — shipping a private copy of a Windows " + "component is a broken program, not a heavier one") + +# The .exe keeps its executable bit, which only matters because a Windows +# package is routinely unpacked and inspected (and wine-tested) from Linux. +info = z.getinfo(exe) +assert (info.external_attr >> 16) & 0o111, "the executable bit was lost" +print("OK: zip verified by an independent reader") +PY + +# `--mode system` promises the target provides everything. A DLL beside the +# .exe would contradict that, and the modes have to mean the same thing on +# both formats or the flag is decoration. +rm -rf target/dist +"$MCPP" pack --target x86_64-windows-gnu --mode system > pack2.log 2>&1 || { + cat pack2.log; exit 1; } +python3 - "$(find target/dist -name '*.zip' | head -1)" <<'PY' +import sys, zipfile +names = zipfile.ZipFile(sys.argv[1]).namelist() +assert not any(n.lower().endswith('.dll') for n in names), \ + f"--mode system bundled a DLL: {names}" +PY + +# The contract reaches packaging (design §4.3). `toolchain-coupled` says the +# toolchain's runtime travels WITH the artifact; a mode that bundles nothing +# cannot deliver that, and a contract with no executor is a promise the build +# prints and the package quietly drops. +cat >> mcpp.toml <<'EOF' + +[build] +cxx_runtime = "toolchain-coupled" +EOF +rm -rf target/dist +if "$MCPP" pack --target x86_64-windows-gnu --mode system > pack3.log 2>&1; then + echo "FAIL: cxx_runtime = toolchain-coupled + --mode system was accepted" + cat pack3.log + exit 1 +fi +grep -q 'toolchain-coupled' pack3.log || { + echo "FAIL: the refusal does not name the contract"; cat pack3.log; exit 1; } +grep -q 'vendored' pack3.log || { + echo "FAIL: the refusal offers no way out"; cat pack3.log; exit 1; } + +echo "OK" diff --git a/tests/unit/test_distribution.cpp b/tests/unit/test_distribution.cpp index bbb089a3..2953f406 100644 --- a/tests/unit/test_distribution.cpp +++ b/tests/unit/test_distribution.cpp @@ -273,6 +273,86 @@ TEST(Distribution, MsvcCrtModelIsWholeProjectAndReportedAsSuch) { EXPECT_EQ(quiet.effective, dist::Contract::HostCoupled); } +// `toolchain-coupled` on the MSVC runtime used to be a flat refusal, and the +// sentence it refused with conflated two different DLLs: +// +// ucrtbase.dll an OS component since Win10 — the refusal was right +// vcruntime140.dll the TOOLSET's own, sitting in VC\Redist\MSVC\… inside +// every toolset mcpp installs — the refusal was wrong +// +// The second one is exactly the relationship gcc has to libstdc++.so, so it +// takes the same contract. What differs is the MECHANISM: PE has no rpath, so +// the DLL travels by being copied beside the artifact. +TEST(Distribution, MsvcToolchainCoupledStagesTheToolsetCrt) { + dist::MechanismInput in; + in.format = dist::Format::Pe; + in.stdlibId = "msvc"; + in.explicitRequest = true; + in.requested = dist::Contract::ToolchainCoupled; + + // /MD: the artifact HAS a vcruntime140.dll dependency, so the contract is + // deliverable — and delivering it means staging files, not adding flags. + in.msvcStaticCrt = false; + auto coupled = dist::resolve(in); + EXPECT_EQ(coupled.effective, dist::Contract::ToolchainCoupled); + EXPECT_FALSE(coupled.degraded); + EXPECT_TRUE(coupled.diagnostic.empty()); + EXPECT_TRUE(coupled.deployToolchainRuntime); + // The CRT model is a compile flag on every TU; nothing goes on the link line. + EXPECT_TRUE(coupled.unitFlags.empty()); + + // /MT is the one case that stays a degradation, and it is a genuine + // contradiction rather than a missing mechanism: a static CRT leaves no + // DLL to couple to. The message has to say which one won. + in.msvcStaticCrt = true; + auto contradiction = dist::resolve(in); + EXPECT_EQ(contradiction.effective, dist::Contract::SelfContained); + EXPECT_TRUE(contradiction.degraded); + EXPECT_FALSE(contradiction.deployToolchainRuntime) + << "a /MT build has no CRT DLL dependency; staging one is dead weight"; + EXPECT_NE(contradiction.diagnostic.find("/MT"), std::string::npos) + << contradiction.diagnostic; + EXPECT_NE(contradiction.diagnostic.find("self-contained"), std::string::npos) + << contradiction.diagnostic; +} + +// Nothing but PE+MSVC+toolchain-coupled may ask for files to be staged. The +// flag reaches a copy step, so a stray `true` puts DLLs in an output tree on a +// platform that has no such thing. +TEST(Distribution, NothingElseAsksForStagedRuntimeFiles) { + const dist::Contract contracts[] = {dist::Contract::SelfContained, + dist::Contract::ToolchainCoupled, + dist::Contract::HostCoupled}; + const dist::Format formats[] = {dist::Format::Elf, dist::Format::MachO, + dist::Format::Pe}; + const std::string_view stdlibs[] = {"libstdc++", "libc++", "msvc", "surprise"}; + for (auto fmt : formats) + for (auto sl : stdlibs) + for (auto c : contracts) + for (bool mt : {false, true}) + for (bool explicitly : {false, true}) { + dist::MechanismInput in; + in.format = fmt; + in.stdlibId = sl; + in.requested = c; + in.msvcStaticCrt = mt; + in.explicitRequest = explicitly; + in.mingw = (fmt == dist::Format::Pe + && sl == "libstdc++"); + auto m = dist::resolve(in); + if (!m.deployToolchainRuntime) continue; + EXPECT_EQ(fmt, dist::Format::Pe); + EXPECT_NE(sl, std::string_view("libstdc++")); + EXPECT_EQ(c, dist::Contract::ToolchainCoupled); + EXPECT_FALSE(mt); + // Staging files is a promise KEPT. A degraded cell did + // not deliver the contract, so it must not act as if + // it had. + EXPECT_FALSE(m.degraded); + EXPECT_EQ(m.effective, dist::Contract::ToolchainCoupled); + } +} + // --------------------------------------------------------------------------- // INV-1, stated as a property rather than a list: the table is TOTAL, and // every cell that does not deliver what was asked explains itself. A future diff --git a/tests/unit/test_ninja_backend.cpp b/tests/unit/test_ninja_backend.cpp index 7af42d59..d7614274 100644 --- a/tests/unit/test_ninja_backend.cpp +++ b/tests/unit/test_ninja_backend.cpp @@ -1430,3 +1430,139 @@ TEST(Plan, ExpandManifestIncludeEntryNativeSpelling) { fs::remove_all(root); } + +// ── PE `toolchain-coupled`: the toolset's CRT travels with the artifact ───── +// +// On ELF this contract needs no files copied — the artifact carries an rpath +// into the toolchain's lib directory. PE has no rpath: a DLL is resolved from +// the directory of the executable, so on this format the mechanism IS the +// copy, and a graph that resolves the contract but stages nothing delivers +// exactly what the old flat refusal did. +// +// Runs on every host. The format now comes from the TARGET TRIPLE rather than +// from `mcpp::platform::is_windows`, which is what makes a Windows contract +// assertable on the Linux runner that reviews most of these changes. +namespace { + +struct FakeRedistDir { + std::filesystem::path path; + FakeRedistDir() { + path = std::filesystem::temp_directory_path() + / std::format("mcpp-deploy-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(path); + for (auto name : {"vcruntime140.dll", "msvcp140.dll", + "vcruntime140_1.dll"}) + std::ofstream{path / name} << "MZ"; + // Not a DLL: the redist directory also carries a manifest, and copying + // it beside the artifact would be cargo. + std::ofstream{path / "Microsoft.VC143.CRT.manifest"} << ""; + } + ~FakeRedistDir() { std::error_code ec; std::filesystem::remove_all(path, ec); } + FakeRedistDir(const FakeRedistDir&) = delete; + FakeRedistDir& operator=(const FakeRedistDir&) = delete; +}; + +BuildPlan msvc_plan_with_redist(const FakeRedistDir& redist, + std::string_view cxxRuntime) { + auto plan = minimal_plan(); + plan.toolchain.compiler = mcpp::toolchain::CompilerId::MSVC; + plan.toolchain.binaryPath = "cl.exe"; + plan.toolchain.targetTriple = "x86_64-pc-windows-msvc"; + plan.toolchain.linkRuntimeDirs = {redist.path}; + plan.manifest.buildConfig.cxxRuntime = std::string(cxxRuntime); + plan.linkUnits.push_back({ + .targetName = "app", + .kind = mcpp::build::LinkUnit::Binary, + .objects = {"obj/main.o"}, + .output = "bin/app.exe", + .entryMain = "src/main.cpp", + }); + return plan; +} + +} // namespace + +TEST(NinjaBackendPeRuntime, ToolchainCoupledStagesTheToolsetCrtBesideTheExe) { + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + + auto flags = compute_flags(plan); + ASSERT_EQ(flags.toolchainRuntimeDeploy.size(), 3u) + << "expected the three .dll and not the .manifest beside them"; + for (auto const& d : flags.toolchainRuntimeDeploy) { + EXPECT_EQ(d.dest.parent_path(), std::filesystem::path("bin")) + << "a DLL must land in the same directory as the .exe: " + << d.dest.string(); + EXPECT_EQ(d.source.extension(), ".dll") << d.source.string(); + } + + auto ninja = emit_ninja_string(plan); + for (auto name : {"vcruntime140.dll", "msvcp140.dll", "vcruntime140_1.dll"}) { + EXPECT_NE(ninja.find(std::format("build bin/{} : stage_file", name)), + std::string::npos) + << name << " has no copy edge\n" << ninja; + } + EXPECT_EQ(ninja.find("Microsoft.VC143.CRT.manifest"), std::string::npos) + << "copied something that is not a runtime DLL\n" << ninja; + + // Reachability, twice over. A copy edge nothing asks for is never run + // under explicit ninja goals, and an .exe that does not depend on its DLLs + // can be reported "up to date" while they are missing. + auto exeAt = ninja.find("build bin/app.exe :"); + ASSERT_NE(exeAt, std::string::npos) << ninja; + auto exeStanza = ninja.substr(exeAt, ninja.find("\nbuild ", exeAt + 1) - exeAt); + EXPECT_NE(exeStanza.find("bin/vcruntime140.dll"), std::string::npos) + << "the executable does not depend on the CRT staged beside it\n" + << exeStanza; + auto defaultAt = ninja.rfind("\ndefault "); + ASSERT_NE(defaultAt, std::string::npos) << ninja; + EXPECT_NE(ninja.find("bin/vcruntime140.dll", defaultAt), std::string::npos) + << "the staged CRT is not a default goal\n" << ninja.substr(defaultAt); +} + +TEST(NinjaBackendPeRuntime, HostCoupledStagesNothing) { + // The DEFAULT Windows build. Staging DLLs unasked would change what every + // existing project ships, and `host-coupled` is a promise that the machine + // provides them — keeping a copy beside the artifact contradicts it. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "host-coupled"); + EXPECT_TRUE(compute_flags(plan).toolchainRuntimeDeploy.empty()); + + auto bare = msvc_plan_with_redist(redist, ""); + EXPECT_TRUE(compute_flags(bare).toolchainRuntimeDeploy.empty()) + << "a project that never mentioned cxx_runtime gained staged DLLs"; +} + +TEST(NinjaBackendPeRuntime, AProjectsOwnDeployFileOutranksTheToolsets) { + // A vendored redist named in `[runtime] deploy_files` is a human's + // statement about which build of msvcp140.dll this program ships. Silently + // replacing it with the toolset's copy produces a different program than + // the manifest describes, so the explicit one wins — out loud. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + plan.runtimeDeployFiles.push_back( + {"/vendor/msvcp140.dll", std::filesystem::path("bin") / "msvcp140.dll"}); + + auto flags = compute_flags(plan); + for (auto const& d : flags.toolchainRuntimeDeploy) + EXPECT_NE(d.dest.filename(), std::filesystem::path("msvcp140.dll")) + << "overwrote the project's own deploy file"; + EXPECT_EQ(flags.toolchainRuntimeDeploy.size(), 2u); + EXPECT_TRUE(std::ranges::any_of(flags.diagnostics, [](auto const& d) { + return d.find("msvcp140.dll") != std::string::npos; + })) << "the collision was resolved silently"; +} + +TEST(NinjaBackendPeRuntime, AnElfToolchainNeverStagesItsRuntimeDirs) { + // `linkRuntimeDirs` means "the toolchain's private runtime" for every + // provider — on gcc it is libstdc++'s directory. That is reached through + // an rpath, and copying its contents into an output tree would be a + // different contract wearing the same name. + FakeRedistDir redist; + auto plan = msvc_plan_with_redist(redist, "toolchain-coupled"); + plan.toolchain.compiler = mcpp::toolchain::CompilerId::GCC; + plan.toolchain.binaryPath = "/usr/bin/g++"; + plan.toolchain.targetTriple = "x86_64-linux-gnu"; + EXPECT_TRUE(compute_flags(plan).toolchainRuntimeDeploy.empty()); +} diff --git a/tests/unit/test_pack_binfmt.cpp b/tests/unit/test_pack_binfmt.cpp new file mode 100644 index 00000000..da365372 --- /dev/null +++ b/tests/unit/test_pack_binfmt.cpp @@ -0,0 +1,426 @@ +#include +#include // stderr is a MACRO — `import std;` cannot export it +#include // getenv + +import std; +import mcpp.pack.binfmt; +import mcpp.pack.zip; + +namespace bf = mcpp::pack::binfmt; + +// WHY THE FIXTURES ARE SYNTHESIZED RATHER THAN CHECKED IN. +// +// This module's whole reason to exist is that a dependency closure must be +// derivable WITHOUT running the binary — so that a Linux runner can package a +// Windows build. A test that needed a real PE would need a Windows toolchain +// to produce one, which is the same dependency one layer up: the test would +// only run where the feature was already unnecessary. +// +// Hand-built headers also make the assertions exact. "Found two DLLs" is a +// much weaker statement than "found the two names I wrote at these RVAs, and +// not the terminator". + +namespace { + +void put(std::string& b, std::size_t at, std::uint64_t v, std::size_t width) { + if (b.size() < at + width) b.resize(at + width, '\0'); + for (std::size_t i = 0; i < width; ++i) + b[at + i] = static_cast((v >> (8 * i)) & 0xFF); +} + +std::filesystem::path write_temp(std::string_view tag, std::string_view bytes) { + auto p = std::filesystem::temp_directory_path() + / std::format("mcpp-binfmt-{}-{}", tag, + std::chrono::steady_clock::now().time_since_epoch().count()); + std::ofstream os(p, std::ios::binary); + os.write(bytes.data(), static_cast(bytes.size())); + return p; +} + +// ─── a minimal ELF64 shared object ────────────────────────────────────── +// +// One PT_LOAD covering the whole file at vaddr 0 (so vaddr == file offset, +// which keeps the fixture readable without making the parser's translation +// step a no-op — DT_STRTAB is still resolved THROUGH the segment table), plus +// a PT_DYNAMIC carrying two DT_NEEDED and a DT_STRTAB. +std::string elf_with_needed(std::span needed) { + std::string b; + b.resize(64, '\0'); + b[0] = 0x7f; b[1] = 'E'; b[2] = 'L'; b[3] = 'F'; + b[4] = 2; // ELFCLASS64 + b[5] = 1; // ELFDATA2LSB + b[6] = 1; // EV_CURRENT + put(b, 16, 3, 2); // e_type = ET_DYN + put(b, 18, 0x3E, 2); // e_machine = EM_X86_64 + put(b, 20, 1, 4); // e_version + put(b, 32, 64, 8); // e_phoff + put(b, 52, 64, 2); // e_ehsize + put(b, 54, 56, 2); // e_phentsize + put(b, 56, 2, 2); // e_phnum + + const std::size_t dynAt = 64 + 2 * 56; + const std::size_t dynEntries = needed.size() + 2; // + DT_STRTAB + DT_NULL + const std::size_t dynSize = dynEntries * 16; + const std::size_t strAt = dynAt + dynSize; + + std::string strtab; + strtab.push_back('\0'); // index 0 is empty + std::vector offsets; + for (auto n : needed) { + offsets.push_back(strtab.size()); + strtab.append(n); + strtab.push_back('\0'); + } + const std::size_t total = strAt + strtab.size(); + + // PT_LOAD: the whole file, vaddr 0. + put(b, 64 + 0, 1, 4); // p_type + put(b, 64 + 4, 5, 4); // p_flags + put(b, 64 + 8, 0, 8); // p_offset + put(b, 64 + 16, 0, 8); // p_vaddr + put(b, 64 + 32, total, 8); // p_filesz + put(b, 64 + 40, total, 8); // p_memsz + + // PT_DYNAMIC + put(b, 120 + 0, 2, 4); + put(b, 120 + 8, dynAt, 8); + put(b, 120 + 16, dynAt, 8); + put(b, 120 + 32, dynSize, 8); + + std::size_t at = dynAt; + for (auto off : offsets) { + put(b, at, 1, 8); // DT_NEEDED + put(b, at + 8, off, 8); + at += 16; + } + put(b, at, 5, 8); // DT_STRTAB (a VIRTUAL ADDRESS) + put(b, at + 8, strAt, 8); + at += 16; + put(b, at, 0, 8); // DT_NULL + put(b, at + 8, 0, 8); + + b.resize(total, '\0'); + std::copy(strtab.begin(), strtab.end(), b.begin() + static_cast(strAt)); + return b; +} + +// ─── a minimal PE32+ image ────────────────────────────────────────────── +// +// One section mapping RVA 0x1000 → file 0x400, an import directory and a +// delay-import directory inside it. +// +// WRITTEN AS PLAINLY AS POSSIBLE, and the breadcrumbs are not decoration. +// The first version used two `std::span` parameters and two lambdas that +// mutated a captured string through a captured cursor; it segfaulted on the +// macOS ARM64 runner and NOWHERE else — not under ASan+UBSan with clang 22 + +// libc++, not as a clang module on x86_64 Linux, not under gcc. Splitting the +// test proved the crash is HERE, in fixture code that touches no module at +// all, so the shapes went and the trace stayed: if it moves again, the log +// says which step. +// Off by default — 24 lines of stderr in every CI run forever is a poor +// trade for a crash that is currently fixed. `MCPP_TEST_TRACE=1` brings it +// back, which is what makes a recurrence one CI round to localise instead of +// the four this one cost. +void trace(const char* step, const std::string& b) { + static const bool on = std::getenv("MCPP_TEST_TRACE") != nullptr; + if (!on) return; + std::fprintf(stderr, "[pe-fixture] %-14s size=%zu\n", step, b.size()); + std::fflush(stderr); +} + +std::string pe_with_imports(const std::vector& imports, + const std::vector& delayImports) { + const std::size_t kNt = 0x40; + const std::size_t kOptSize = 0xF0; // 112 + 16 directories * 8 + const std::size_t kSecAt = kNt + 24 + kOptSize; + const std::size_t kRawAt = 0x400; + const std::uint32_t kSecVa = 0x1000; + const std::size_t dirsAt = kNt + 24 + 112; + + std::string b(kRawAt, '\0'); + trace("start", b); + b[0] = 'M'; b[1] = 'Z'; + put(b, 0x3C, kNt, 4); + b[kNt] = 'P'; b[kNt + 1] = 'E'; // "PE\0\0" + put(b, kNt + 4, 0x8664, 2); // Machine = AMD64 + put(b, kNt + 6, 1, 2); // NumberOfSections + put(b, kNt + 20, kOptSize, 2); // SizeOfOptionalHeader + put(b, kNt + 24, 0x20b, 2); // PE32+ + put(b, dirsAt - 4, 16, 4); // NumberOfRvaAndSizes + trace("headers", b); + + // Section header: name, VirtualSize, VirtualAddress, SizeOfRawData, + // PointerToRawData. + const std::string secName = ".rdata"; + for (std::size_t i = 0; i < secName.size(); ++i) b[kSecAt + i] = secName[i]; + put(b, kSecAt + 8, 0x1000, 4); + put(b, kSecAt + 12, kSecVa, 4); + put(b, kSecAt + 16, 0x1000, 4); + put(b, kSecAt + 20, kRawAt, 4); + trace("section", b); + + // Names first, so the descriptors can point at them. + std::vector importNameRvas; + std::vector delayNameRvas; + std::size_t cursor = kRawAt; + for (std::size_t which = 0; which < 2; ++which) { + const std::vector& names = which == 0 ? imports : delayImports; + for (std::size_t k = 0; k < names.size(); ++k) { + const std::uint32_t rva = + static_cast(kSecVa + (cursor - kRawAt)); + const std::string& n = names[k]; + for (std::size_t i = 0; i < n.size(); ++i) { + put(b, cursor, static_cast(n[i]), 1); + ++cursor; + } + put(b, cursor, 0, 1); + ++cursor; + if (which == 0) importNameRvas.push_back(rva); + else delayNameRvas.push_back(rva); + } + } + trace("names", b); + + // Import descriptors (20 bytes each) + an all-zero terminator. + cursor = (cursor + 15) & ~static_cast(15); + const std::size_t importAt = cursor; + for (std::size_t k = 0; k < importNameRvas.size(); ++k) { + put(b, cursor + 0, 0x9000, 4); // OriginalFirstThunk (nonzero) + put(b, cursor + 12, importNameRvas[k], 4); // Name + put(b, cursor + 16, 0x9100, 4); // FirstThunk (nonzero) + cursor += 20; + } + for (std::size_t i = 0; i < 20; ++i) { put(b, cursor, 0, 1); ++cursor; } + trace("imports", b); + + // Delay-import descriptors (32 bytes each). grAttrs bit 0 = the fields + // are RVAs; without it a descriptor is the pre-VC7 address form and must + // be skipped rather than misread. + cursor = (cursor + 15) & ~static_cast(15); + const std::size_t delayAt = cursor; + for (std::size_t k = 0; k < delayNameRvas.size(); ++k) { + put(b, cursor + 0, 1, 4); // grAttrs = dlattrRva + put(b, cursor + 4, delayNameRvas[k], 4); // rvaDLLName + cursor += 32; + } + for (std::size_t i = 0; i < 32; ++i) { put(b, cursor, 0, 1); ++cursor; } + trace("delay", b); + + if (!imports.empty()) { + put(b, dirsAt + 1 * 8, + static_cast(kSecVa + (importAt - kRawAt)), 4); + put(b, dirsAt + 1 * 8 + 4, 20 * (imports.size() + 1), 4); + } + if (!delayImports.empty()) { + put(b, dirsAt + 13 * 8, + static_cast(kSecVa + (delayAt - kRawAt)), 4); + put(b, dirsAt + 13 * 8 + 4, 32 * (delayImports.size() + 1), 4); + } + trace("directories", b); + return b; +} + +// The three names every PE test below builds an image around. A function, not +// a namespace-scope constant: a `std::vector` at namespace scope +// in a test TU is a static initializer, and this file is already investigating +// one platform-specific crash. +std::vector pe_imports() { return {"KERNEL32.dll", "vcruntime140.dll"}; } +std::vector pe_delayed() { return {"dbghelp.dll"}; } + +struct TempFile { + std::filesystem::path path; + TempFile(std::string_view tag, std::string_view bytes) + : path(write_temp(tag, bytes)) {} + ~TempFile() { std::error_code ec; std::filesystem::remove(path, ec); } + TempFile(const TempFile&) = delete; + TempFile& operator=(const TempFile&) = delete; +}; + +} // namespace + +TEST(PackBinfmt, IdentifiesElfWithoutRunningIt) { + std::array needed{"libc.so.6"}; + TempFile f{"elf", elf_with_needed(needed)}; + auto id = bf::identify(f.path); + EXPECT_EQ(id.format, bf::Format::Elf); + EXPECT_EQ(id.arch, "x86_64"); + EXPECT_TRUE(id.is64); +} + +TEST(PackBinfmt, ReadsElfDtNeededThroughTheSegmentTable) { + // DT_STRTAB is a virtual address, so the parser has to translate it via + // PT_LOAD. Two entries, in file order. + std::array needed{"libstdc++.so.6", "libcustom.so.1"}; + TempFile f{"elfneeded", elf_with_needed(needed)}; + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + ASSERT_EQ(names->size(), 2u); + EXPECT_EQ((*names)[0], "libstdc++.so.6"); + EXPECT_EQ((*names)[1], "libcustom.so.1"); +} + +TEST(PackBinfmt, AnElfWithNoDynamicSectionHasZeroDepsAndIsNotAnError) { + // A fully static binary. "No dependencies" and "could not be read" must + // not look alike — one is a packageable artifact and the other is a bug. + auto bytes = elf_with_needed({}); + // Strip PT_DYNAMIC by zeroing its type field. + put(bytes, 120, 0, 4); + TempFile f{"elfstatic", bytes}; + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + EXPECT_TRUE(names->empty()); +} + +// SPLIT INTO THREE ON PURPOSE. The combined version died with SIGSEGV on the +// macOS ARM64 runner and nowhere else, and a single test that builds a +// fixture, identifies it and parses it cannot say WHICH of the three it was. +// A test that cannot localise its own failure is a test that costs a CI round +// per hypothesis. +TEST(PackBinfmt, ThePeFixtureItselfIsWellFormed) { + auto bytes = pe_with_imports(pe_imports(), pe_delayed()); + ASSERT_GT(bytes.size(), 0x400u); + EXPECT_EQ(bytes.substr(0, 2), "MZ"); + EXPECT_EQ(bytes.substr(0x40, 4), std::string("PE\0\0", 4)); + // The names have to be IN the image, or every assertion below is about + // the fixture rather than about the reader. + for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) + EXPECT_NE(bytes.find(want), std::string::npos) << want; +} + +TEST(PackBinfmt, IdentifiesPe) { + TempFile f{"peid", pe_with_imports(pe_imports(), pe_delayed())}; + + auto id = bf::identify(f.path); + EXPECT_EQ(id.format, bf::Format::Pe); + EXPECT_EQ(id.arch, "x86_64"); + EXPECT_TRUE(id.is64); +} + +TEST(PackBinfmt, ReadsBothPeImportDirectories) { + TempFile f{"peimp", pe_with_imports(pe_imports(), pe_delayed())}; + + auto names = bf::needed_names(f.path); + ASSERT_TRUE(names.has_value()) << names.error(); + // The DELAY-loaded one is a dependency too, and leaving it out is worse + // than leaving out an ordinary import: a missing delay-load does not fail + // at startup, it fails at the first call through it. + EXPECT_EQ(names->size(), 3u); + for (auto want : {"KERNEL32.dll", "vcruntime140.dll", "dbghelp.dll"}) { + bool found = false; + for (auto const& n : *names) found = found || n == want; + EXPECT_TRUE(found) << want << " missing from the closure"; + } +} + +TEST(PackBinfmt, ADosStubWithoutAPeSignatureIsNotAPe) { + // "MZ" alone is a DOS executable. Calling it PE would send the caller + // into a parser that cannot succeed, and the error would describe the + // wrong thing. + std::string b(0x100, '\0'); + b[0] = 'M'; b[1] = 'Z'; + put(b, 0x3C, 0x40, 4); // points at zeros, not "PE\0\0" + TempFile f{"dos", b}; + EXPECT_EQ(bf::identify(f.path).format, bf::Format::Unknown); + EXPECT_FALSE(bf::needed_names(f.path).has_value()); +} + +TEST(PackBinfmt, TruncatedInputIsRejectedRatherThanRead) { + // Malformed input is ordinary: a half-downloaded file, a text file named + // `.exe`. Every read is bounds-checked, so the parser is total over it. + for (std::size_t keep : {0u, 4u, 0x40u, 0x80u}) { + auto bytes = pe_with_imports({"KERNEL32.dll"}, {}); + bytes.resize(keep); + TempFile f{"trunc", bytes}; + auto names = bf::needed_names(f.path); + if (names) EXPECT_TRUE(names->empty()) << "keep=" << keep; + } +} + +TEST(PackBinfmt, TheSystemPredicateKnowsWindowsFromTheToolset) { + // Windows' own: shipping a private copy is a broken program, not a + // heavier one. + for (auto n : {"KERNEL32.dll", "kernel32.dll", "ucrtbase.dll", "ntdll.dll", + "api-ms-win-crt-runtime-l1-1-0.dll"}) + EXPECT_TRUE(bf::is_system_lib(bf::Format::Pe, n)) << n; + + // NOT Windows' own. These belong to the TOOLSET, and whether they travel + // is `cxx_runtime`'s decision — the one call this predicate must not make + // on its behalf. + for (auto n : {"vcruntime140.dll", "vcruntime140_1.dll", "msvcp140.dll", + "libwinpthread-1.dll", "libstdc++-6.dll"}) + EXPECT_FALSE(bf::is_system_lib(bf::Format::Pe, n)) << n; + + // ELF keeps the manylinux allow-list it always had. + EXPECT_TRUE(bf::is_system_lib(bf::Format::Elf, "libstdc++.so.6")); + EXPECT_TRUE(bf::is_system_lib(bf::Format::Elf, "ld-linux-x86-64.so.2")); + EXPECT_FALSE(bf::is_system_lib(bf::Format::Elf, "libcurl.so.4")); +} + +// ─── the zip writer ────────────────────────────────────────────────────── + +TEST(PackZip, Crc32MatchesTheKnownVectors) { + // Self-consistency proves nothing here: an archive whose CRCs are wrong + // extracts with a warning on some tools and silently on others. + EXPECT_EQ(mcpp::pack::zip::crc32(""), 0u); + EXPECT_EQ(mcpp::pack::zip::crc32("123456789"), 0xCBF43926u); + EXPECT_EQ(mcpp::pack::zip::crc32( + "The quick brown fox jumps over the lazy dog"), 0x414FA339u); +} + +TEST(PackZip, WritesAReadableArchiveWithForwardSlashNames) { + auto dir = std::filesystem::temp_directory_path() + / std::format("mcpp-zip-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(dir); + std::ofstream{dir / "a.txt"} << "hello"; + std::ofstream{dir / "b.dll"} << "MZ-not-really"; + + std::vector entries{ + {"pkg/a.txt", dir / "a.txt", false}, + {"pkg/b.dll", dir / "b.dll", false}, + }; + auto out = dir / "pkg.zip"; + auto r = mcpp::pack::zip::write(out, entries); + ASSERT_TRUE(r.has_value()) << r.error(); + + std::ifstream is(out, std::ios::binary); + std::string bytes((std::istreambuf_iterator(is)), {}); + ASSERT_GE(bytes.size(), 22u); + EXPECT_EQ(bytes.substr(0, 4), std::string("PK\x03\x04", 4)) + << "no local file header signature"; + EXPECT_NE(bytes.find(std::string("PK\x01\x02", 4)), std::string::npos) + << "no central directory"; + EXPECT_EQ(bytes.substr(bytes.size() - 22, 4), std::string("PK\x05\x06", 4)) + << "end-of-central-directory is not last (a trailing comment?)"; + // Names are stored verbatim; a backslash here is what makes an archive + // written on Windows extract to one oddly-named file everywhere else. + EXPECT_NE(bytes.find("pkg/a.txt"), std::string::npos); + EXPECT_EQ(bytes.find("pkg\\a.txt"), std::string::npos); + // Stored, not deflated: the payload appears literally. + EXPECT_NE(bytes.find("hello"), std::string::npos); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} + +TEST(PackZip, TwoWritesOfTheSameInputAreByteIdentical) { + // A published checksum only means something if the archive is a function + // of its contents. Reading mtimes would make it a function of when. + auto dir = std::filesystem::temp_directory_path() + / std::format("mcpp-zip-det-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()); + std::filesystem::create_directories(dir); + std::ofstream{dir / "x.bin"} << "payload"; + std::vector entries{{"p/x.bin", dir / "x.bin", true}}; + + auto read = [](const std::filesystem::path& p) { + std::ifstream is(p, std::ios::binary); + return std::string((std::istreambuf_iterator(is)), {}); + }; + ASSERT_TRUE(mcpp::pack::zip::write(dir / "one.zip", entries).has_value()); + ASSERT_TRUE(mcpp::pack::zip::write(dir / "two.zip", entries).has_value()); + EXPECT_EQ(read(dir / "one.zip"), read(dir / "two.zip")); + + std::error_code ec; + std::filesystem::remove_all(dir, ec); +} diff --git a/tests/unit/test_runtime_contract.cpp b/tests/unit/test_runtime_contract.cpp index 5664652f..2da1734c 100644 --- a/tests/unit/test_runtime_contract.cpp +++ b/tests/unit/test_runtime_contract.cpp @@ -370,3 +370,80 @@ TEST(RuntimeSearchClosure, UndeclaredBindingStillGetsItsOwnFarm) { EXPECT_EQ(farm_entries(build::runtime_search_closure( plan_for("aarch64-linux-gnu"), b)), 0u); } + +// ─── the Windows runtime identity ──────────────────────────────────────── +// +// `runtimeId`'s own comment has documented `ucrt@…` since the field was +// introduced, and nothing in the repository ever wrote one. The compiler had +// a version axis and the C runtime it compiles against did not, so two +// Windows SDKs produced one cache key — the version axis simply stopped +// existing one layer down. + +TEST(RuntimeIdentity, ProviderIsDispatchedOnNotPatternMatched) { + namespace rt = mcpp::platform::runtime; + EXPECT_EQ(rt::runtime_provider("glibc@2.39"), "glibc"); + EXPECT_EQ(rt::runtime_provider("ucrt@10.0.26100.0"), "ucrt"); + EXPECT_EQ(rt::runtime_provider("macos_sdk@14.0"), "macos_sdk"); + // No identity at all is a DIFFERENT answer from "an identity belonging to + // some other provider", and every consumer that spelled + // `starts_with("glibc@")` collapsed the two. + EXPECT_TRUE(rt::runtime_provider("").empty()); + EXPECT_TRUE(rt::runtime_provider("nonsense").empty()); +} + +TEST(RuntimeIdentity, TheSdkVersionReachesTheContractHash) { + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding a; + a.platform = "windows"; + a.contractHash = "unset"; + + rt::RuntimeBinding b = a; + rt::bind_windows_ucrt(a, "10.0.22621.0"); + rt::bind_windows_ucrt(b, "10.0.26100.0"); + + EXPECT_EQ(a.runtimeId, "ucrt@10.0.22621.0"); + EXPECT_EQ(b.runtimeId, "ucrt@10.0.26100.0"); + EXPECT_NE(a.contractHash, b.contractHash) + << "two SDKs produced one contract hash, so they produce one build " + "cache — which is the defect this identity closes"; + EXPECT_NE(a.contractHash, "unset") << "the hash was not re-derived"; +} + +TEST(RuntimeIdentity, BindingUcrtIsIdempotentAndSkipsAnEmptyVersion) { + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding b; + b.platform = "windows"; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + auto once = b.contractHash; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + EXPECT_EQ(b.contractHash, once); + + // A Windows box with no SDK still builds — toolchain SELECTION has to + // work there — so there is simply nothing to declare. + rt::RuntimeBinding empty; + empty.contractHash = "keep"; + rt::bind_windows_ucrt(empty, ""); + EXPECT_TRUE(empty.runtimeId.empty()); + EXPECT_EQ(empty.contractHash, "keep"); +} + +TEST(RuntimeIdentity, UcrtIsAFloorDeclarationNotAPrivatePayload) { + // The asymmetry with glibc, pinned so it cannot be "tidied up" later. + // + // `glibc@2.39` binds a payload: the headers and the .so are both in it and + // patchelf makes the artifact run on that exact copy, so it is ALSO + // projected into `libc`, which the loader machinery reads. `ucrtbase.dll` + // is an OS component from Win10 on — it cannot be swapped and must not be + // shipped — so mcpp's windows-sdk payload deliberately carries only half + // of ucrt (headers + import libraries). Projecting it into `libc` would + // send the private-libc machinery looking for a payload that was never + // supposed to exist. + namespace rt = mcpp::platform::runtime; + rt::RuntimeBinding b; + b.platform = "windows"; + rt::bind_windows_ucrt(b, "10.0.26100.0"); + EXPECT_FALSE(b.libc.has_value()) + << "ucrt was projected into the private-libc field"; + EXPECT_FALSE(b.loader.has_value()); + EXPECT_TRUE(b.libraryDirs.empty()); +} diff --git a/tests/unit/test_toolchain_msvc.cpp b/tests/unit/test_toolchain_msvc.cpp index 034a9c0e..a4968023 100644 --- a/tests/unit/test_toolchain_msvc.cpp +++ b/tests/unit/test_toolchain_msvc.cpp @@ -657,3 +657,170 @@ TEST(ToolchainMsvc, BothKeysSelectTheStaticCrt) { EXPECT_FALSE(msvc_wants_static_crt("", "host-coupled")); EXPECT_FALSE(msvc_wants_static_crt("", "toolchain-coupled")); } + +// ─── the SDK axis: bound for a managed toolset, searched for a system one ── +// +// Until now the Windows SDK had no identity: `find_windows_sdk()` scanned, +// and whatever the scan reached first won, for BOTH origins. That is the +// defect .agents/docs/2026-08-16-windows-toolchain-three-axes-design.md §2 +// is about — the compiler got a version axis and the headers it compiles +// against did not, so two machines could build one manifest against two SDKs +// with nothing in the log naming either. + +namespace { + +// An xlings store, laid out the way the real one is: `xpkgs_from_compiler` +// finds the store by walking up for a directory literally named `xpkgs`, so +// that name is load-bearing and not decoration. +struct FakeStore { + std::filesystem::path root; // …/xpkgs + + FakeStore() { + root = std::filesystem::temp_directory_path() + / std::format("mcpp-store-{}", std::chrono::steady_clock::now() + .time_since_epoch().count()) + / "data" / "xpkgs"; + std::filesystem::create_directories(root); + } + ~FakeStore() { + std::error_code ec; + std::filesystem::remove_all(root.parent_path().parent_path(), ec); + } + FakeStore(const FakeStore&) = delete; + FakeStore& operator=(const FakeStore&) = delete; + + std::filesystem::path add_toolset(std::string_view version) { + auto tools = root / "xim-x-msvc" / std::string(version) + / "VC" / "Tools" / "MSVC" / std::string(version); + auto bin = tools / "bin" / "Hostx64" / "x64"; + std::filesystem::create_directories(bin); + std::ofstream{bin / "cl.exe"} << "not a compiler"; + std::filesystem::create_directories(tools / "modules"); + std::ofstream{tools / "modules" / "std.ixx"} << "export module std;"; + return bin / "cl.exe"; + } + // A complete SDK payload — headers AND import libraries, for both + // architectures so the fixture does not care which host runs it. + std::filesystem::path add_sdk(std::string_view version) { + auto sdk = root / "xim-x-windows-sdk" / std::string(version); + auto inc = sdk / "Include" / std::string(version) / "ucrt"; + std::filesystem::create_directories(inc); + std::ofstream{inc / "corecrt.h"} << "#pragma once"; + for (auto arch : {"x64", "arm64"}) { + auto lib = sdk / "Lib" / std::string(version) / "um" / arch; + std::filesystem::create_directories(lib); + std::ofstream{lib / "kernel32.lib"} << "not a library"; + } + return sdk; + } +}; + +} // namespace + +TEST(MsvcSdkOrigin, ACompilerInAStoreIsManagedAndOneOutsideIsNot) { + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + EXPECT_EQ(msvc::origin_of(cl), Origin::Managed); + EXPECT_EQ(msvc::origin_of( + "C:/Program Files/Microsoft Visual Studio/18/Enterprise/VC/Tools/" + "MSVC/14.51.36231/bin/Hostx64/x64/cl.exe"), Origin::SystemMsvc); +} + +TEST(MsvcSdkOrigin, AManagedToolsetTakesTheSdkFromItsOwnStore) { + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.26100.0"); + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()) + << "the toolset's own SDK payload was not found"; + EXPECT_EQ(choice.origin, Origin::Managed); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0"); + EXPECT_TRUE(choice.note.empty()) << choice.note; +} + +// THE CRITERION THIS AXIS EXISTS FOR. An environment that can overwrite the +// choice means the manifest did not pin anything — it only expressed a +// preference that the machine gets to overrule, silently. +// +// This is the design doc's §6 acceptance test, as a unit test: point +// WindowsSdkDir somewhere else entirely and the build must still use the +// payload's SDK, and must SAY that the variable was ignored. +TEST(MsvcSdkOrigin, WindowsSdkDirCannotOverrideAPinnedToolsetsSdk) { + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.26100.0"); + + FakeToolset elsewhere{"sdk-elsewhere"}; + elsewhere.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", elsewhere.root.string()}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0") + << "the environment overrode a pinned toolset's SDK; the pin is not one"; + EXPECT_NE(choice.sdk->root, elsewhere.root); + EXPECT_NE(choice.note.find("WindowsSdkDir"), std::string::npos) + << "an ignored override that says nothing is indistinguishable from " + "one that was never set: " << choice.note; +} + +TEST(MsvcSdkOrigin, WindowsSdkVersionDoesNotPickAmongPayloadsEither) { + // Same override wearing a smaller hat: naming a version is still the + // environment choosing which headers a pinned build compiles against. + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); + store.add_sdk("10.0.22621.0"); + store.add_sdk("10.0.26100.0"); + ScopedEnv ver{"WindowsSdkVersion", "10.0.22621.0\\"}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->version, "10.0.26100.0") + << "WindowsSdkVersion selected among the store's payloads"; + EXPECT_FALSE(choice.note.empty()); +} + +TEST(MsvcSdkOrigin, AManagedToolsetWithNoSdkPayloadFallsBackAndSaysSo) { + // Working beats failing here — the machine may well have a complete SDK, + // and refusing to build would be a regression for anyone whose toolset + // predates the SDK dependency. What must not happen is the fallback being + // invisible: the build is no longer reproducible, and only this line says + // so. + NoSdkEnv clean; + FakeStore store; + auto cl = store.add_toolset("14.44.35207"); // no add_sdk + + FakeToolset machine{"sdk-machine"}; + machine.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", machine.root.string()}; + + auto choice = msvc::resolve_sdk_for(cl); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->root, machine.root); + EXPECT_NE(choice.note.find("machine"), std::string::npos) << choice.note; +} + +TEST(MsvcSdkOrigin, ASystemToolsetKeepsTheDeclaredSearchChain) { + // The other origin is unchanged, and must be: a machine's things can only + // be found by looking, and WindowsSdkDir is the most specific answer + // available there — the same precedence VSINSTALLDIR has over vswhere. + NoSdkEnv clean; + FakeToolset vs{"sdk-system-vs"}; + vs.add_toolset("14.51.36231"); + FakeToolset declared{"sdk-system-declared"}; + declared.add_sdk("10.0.22621.0"); + ScopedEnv dir{"WindowsSdkDir", declared.root.string()}; + + auto cl = vs.root / "VC" / "Tools" / "MSVC" / "14.51.36231" + / "bin" / "Hostx64" / "x64" / "cl.exe"; + auto choice = msvc::resolve_sdk_for(cl); + EXPECT_EQ(choice.origin, Origin::SystemMsvc); + ASSERT_TRUE(choice.sdk.has_value()); + EXPECT_EQ(choice.sdk->root, declared.root); + EXPECT_TRUE(choice.note.empty()) + << "nothing was ignored, so there is nothing to report: " << choice.note; +} diff --git a/tests/unit/test_toolchain_registry.cpp b/tests/unit/test_toolchain_registry.cpp index 8e961ae7..2437d968 100644 --- a/tests/unit/test_toolchain_registry.cpp +++ b/tests/unit/test_toolchain_registry.cpp @@ -2,6 +2,7 @@ import std; import mcpp.platform; +import mcpp.toolchain.model; import mcpp.toolchain.registry; import mcpp.toolchain.triple; @@ -224,3 +225,73 @@ TEST(ToolchainRegistry, NativeGccPayloadFollowsWhatTheArchActuallyPublishes) { // Non-Linux hosts are out of scope: macOS uses llvm, Windows mingw/msvc. EXPECT_FALSE(gcc_native_payload_is_musl("aarch64", false, none)); } + +// ─── the origin axis ───────────────────────────────────────────────────── +// +// `@system` is not a general spelling and must not become one. mcpp is built +// on xlings, a user-space OS, and the design drives host dependencies to a +// minimum: a toolchain comes from a payload the manifest names, so every +// machine compiles with the same compiler. `msvc@system` is a concession to +// ONE platform — Visual Studio is very often already installed and cannot +// always be redistributed. + +TEST(ToolchainOrigin, MsvcIsTheOnlyFamilyWithASystemSpelling) { + auto msvcSystem = parse_toolchain_spec("msvc@system"); + ASSERT_TRUE(msvcSystem.has_value()) << msvcSystem.error(); + EXPECT_TRUE(is_system_toolchain(*msvcSystem)); + EXPECT_EQ(origin_of(*msvcSystem), Origin::SystemMsvc); + + // A VERSIONED msvc spec is the other origin — that split is the whole + // point of the version axis. + auto pinned = parse_toolchain_spec("msvc@14.44.35207"); + ASSERT_TRUE(pinned.has_value()); + EXPECT_EQ(origin_of(*pinned), Origin::Managed); + + auto gcc = parse_toolchain_spec("gcc@16.1.0"); + ASSERT_TRUE(gcc.has_value()); + EXPECT_EQ(origin_of(*gcc), Origin::Managed); +} + +TEST(ToolchainOrigin, NonMsvcSystemIsRejectedWhereItIsReadAndOffersTheAlternatives) { + // It used to parse, and then fail somewhere else entirely as + // `xim:gcc@system` → "no such package" — which sends the reader looking + // for a version that was never going to exist. The error has to name both + // things the user might have meant. + for (auto spec : {"gcc@system", "llvm@system"}) { + auto r = parse_toolchain_spec(spec); + ASSERT_FALSE(r.has_value()) << spec << " was accepted"; + EXPECT_NE(r.error().find("only msvc"), std::string::npos) << r.error(); + EXPECT_NE(r.error().find("@"), std::string::npos) + << "the pin alternative is not offered: " << r.error(); + // The family-less PATH escape hatch is the OTHER thing they might + // have wanted, and it is a different mechanism. + EXPECT_NE(r.error().find("PATH compiler"), std::string::npos) + << "the escape hatch is not offered: " << r.error(); + EXPECT_EQ(r.error().find("xim:"), std::string::npos) + << "still leaking the package spelling that cannot exist: " + << r.error(); + } +} + +// The Linux sysroot payloads (`xim:glibc` + `xim:linux-headers`) had two +// derivations, and a comment on one claimed it mirrored the other. It did +// not: the PE term was missing from the second. +TEST(ToolchainSysrootDeps, OneDerivationForTheGlibcSysrootPayloads) { + mcpp::toolchain::triple::Triple host{}; // empty = host + mcpp::toolchain::triple::Triple musl{std::string(mcpp::platform::host_arch), "linux", "musl"}; + mcpp::toolchain::triple::Triple mingw{"x86_64", "windows", "gnu"}; + + if constexpr (mcpp::platform::is_linux) { + EXPECT_TRUE(needs_linux_sysroot_payloads(host)); + // Self-contained: a musl payload carries its own C library. + EXPECT_FALSE(needs_linux_sysroot_payloads(musl)); + // THE TERM THAT WAS MISSING. A PE target brings its own CRT, whether + // it is a native MinGW or the Linux-hosted cross, so a Linux sysroot + // is not part of installing one. + EXPECT_FALSE(needs_linux_sysroot_payloads(mingw)); + } else { + // No Linux sysroot exists to want. + for (auto const& t : {host, musl, mingw}) + EXPECT_FALSE(needs_linux_sysroot_payloads(t)); + } +}