Skip to content

test: verify sanitizer build configuration - #1716

Open
astandrik wants to merge 11 commits into
DeusData:mainfrom
astandrik:codex/fix-1648-sanitized-build-config
Open

astandrik wants to merge 11 commits into
DeusData:mainfrom
astandrik:codex/fix-1648-sanitized-build-config

Conversation

@astandrik

@astandrik astandrik commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #1648 by checking what each test runner was actually compiled with.

test-runner --build-config exits before suite or test-cache setup and prints one exact line: sanitized=<0|1> test_seams=<0|1>. scripts/test.sh checks that line immediately after building the runner in its iteration, full, and TSan paths. scripts/msan.sh performs the same check before its suites start. If a stale or misconfigured runner disagrees with the lane, the script now stops before any suite can run with the wrong timing budget.

The patch only changes tests/test_main.c, scripts/test.sh, and scripts/msan.sh. It does not touch the Makefile, workflows, dependencies, or production code.

Current CI

On head e3c0010267e783b4270e0ac7c8ee6332b61aeabb, PR CI has finished: 29 checks passed, 2 failed, 1 skipped. DCO, lint, memory analysis, CodeQL, Linux/macOS tests, diagnostic sanitizers, MSan, LSan, all three TSan lanes, Windows guards, the second Windows shard and shard completeness passed.

The first Windows CLANG64 shard reported 4,106 passed, 1 failed, 12 skipped. The failure is version_cohort_crash_releases_process_lifetime_lease at tests/test_version_cohort.c:950, ASSERT(ready); the aggregate ci-ok consequently failed. This is not a build-config mismatch.

The same named test and assertion also failed in PR #1817's Windows run on 24 August, whose head 87576413 does not contain this PR's build-config probe. The failing test, cohort implementation and subprocess implementation are unchanged from the current base. This establishes a prior occurrence outside this patch; it does not establish the underlying Windows failure mechanism. Native Windows ARM64 remains unverified.

Original implementation checks

  • Plain runner: sanitized=0 test_seams=1
  • Default and explicit UBSan runners: sanitized=1 test_seams=1
  • Empty and whitespace-only SANITIZE paths
  • Reused plain build followed by a default sanitized iteration, which failed at the new check before the suite
  • Full scripts/test.sh on Linux x86-64
  • bash -n scripts/test.sh scripts/msan.sh
  • git diff HEAD^ HEAD --check
  • Commit signature and DCO checks

Apple Silicon ASan and TSan fail in the sanitizer runtime before main, so they do not provide evidence about this patch. The remote MSan path is also blocked before its build command because the Compose service is pinned to linux/arm64 and the x86-64 builder has no binfmt/QEMU registration. These are historical local-environment limits; current CI coverage and the remaining Windows failure are recorded above.

The original full local lint reached existing diagnostics in unchanged source files; none of the three files in this patch appeared in that output. Current CI lint and memory analysis have passed.

Checklist

  • Every commit is signed off (git commit -s): CI requires DCO.
  • Required CI is fully green: the first Windows shard and aggregate ci-ok remain red as described above.
  • CI lint and memory analysis pass on the current head; historical local baseline limitations are disclosed above.
  • New behavior is covered by the test entrypoints and focused positive and stale-build checks.

@github-actions

Copy link
Copy Markdown

Thanks for opening this — it has been seen, and it is queued.

This note is automated, but it is not a brush-off: it exists so you know where your PR stands instead of having to guess from silence.

Current review status: working through a backlog. 0.9.1-rc.1 is out, so the release freeze that held reviews is over — but it left a large queue of open pull requests behind it, and we are reading through them oldest-first. The background is in discussion #1144.

What that means for this PR, concretely:

  • It will not be closed for inactivity. No stale bot touches pull requests here.
  • It may still sit a while before a human reads it. That is on us, not on you.
  • Older PRs are read first, so a recent one is not being skipped — it is behind a queue.

Things that will genuinely speed it up whenever review does happen:

  • Keep it rebased on main — the tree is moving quickly right now, and a conflicting branch cannot be reviewed as the diff you intended.
  • Get CI green, or say which failures you believe are pre-existing.
  • Keep the change to one claim. Bundled features and refactors get split before they get merged, which costs you a round trip.
  • Every commit needs a sign-off (git commit -s) — CI enforces DCO.

If this fixes a bug, a reproduction we can run is worth more than a description of the symptom.

Thanks for contributing, and sorry in advance for the wait.

Signed-off-by: astandrik <astandrik@yandex-team.ru>
Signed-off-by: astandrik <astandrik@yandex-team.ru>
@astandrik
astandrik force-pushed the codex/fix-1648-sanitized-build-config branch from 4619986 to ba0c499 Compare August 18, 2026 18:49
Signed-off-by: astandrik <astandrik@yandex-team.ru>
@astandrik
astandrik marked this pull request as ready for review August 19, 2026 12:19
@astandrik
astandrik requested a review from DeusData as a code owner August 19, 2026 12:19
Copilot AI lite review requested due to automatic review settings August 19, 2026 12:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an early “compiled build configuration” probe to the C test-runner binary and enforces it from the test entrypoint scripts, preventing suites from running when a stale/misconfigured runner doesn’t match the intended sanitizer lane.

Changes:

  • Add test-runner --build-config (early-exit) that prints sanitized=<0|1> test_seams=<0|1>.
  • In scripts/test.sh, assert the freshly-built runner reports the expected sanitized value before running any suites (iteration / full / TSan paths).
  • In scripts/msan.sh, assert the MSan runner reports the expected config before running its suites.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tests/test_main.c Adds --build-config output based on compile-time macros (CBM_SANITIZED, CBM_ENABLE_TEST_SEAMS).
scripts/test.sh Adds build-config assertion gate after building runners in each mode.
scripts/msan.sh Adds the same build-config assertion gate for the MSan runner before suites.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/test_main.c Outdated
Signed-off-by: astandrik <astandrik@yandex-team.ru>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Comment thread tests/test_main.c Outdated
Signed-off-by: astandrik <astandrik@yandex-team.ru>
@DeusData DeusData added maintainer-notes Internal observations and improvement notes priority/normal Standard review queue; useful PR with ordinary maintainer urgency. labels Sep 3, 2026
@DeusData

DeusData commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Thank you for this contribution, and sorry that it has had only the automated acknowledgement since August 18.

A sanitizer-configuration assertion changes the confidence we place in several test venues, so it needs a deliberate maintainer review even though the diff is test-only. I have now routed it in the maintainer test queue with normal priority.

The queue is quite full and review may still take some time, but the PR is no longer unclassified. No changes are requested from you at this triage stage.

@DeusData DeusData added the github_actions Pull requests that update GitHub Actions code label Sep 5, 2026
DeusData and others added 6 commits September 5, 2026 20:27
Conflict in tests/test_main.c: main added tf_maybe_run_windows_memory_limit_probe
after the CBM_TEST_BUILD_FINGERPRINT setenv, while this branch hoisted the
blocking-git probe and --version above that setenv to host the early
--build-config probe. Resolved by keeping main's probe order
(memory-limit -> blocking-git -> --version) inside the hoisted early block,
with --build-config last before the setenv.
Conflict in tests/test_main.c: main (DeusData#2195) added the --userns-secure-probe
block after --version, while this branch keeps --build-config and the
CBM_TEST_BUILD_FINGERPRINT setenv there. Resolved by leaving the userns probe
where main runs it relative to the setenv: after it, immediately before
tf_maybe_run_mcp_idxfailclosed_probe.
@DeusData

Copy link
Copy Markdown
Owner

Here is the deliberate review this was promised on 2026-09-03, and I am sorry it took another seventeen days on top of the wait you had already had. You were told it needed careful attention and then heard nothing, which is the worst combination.

The review verdict: this is a good change and it closes a real class of false green.

A sanitizer lane that runs a runner built without the sanitizer asserts coverage it does not have, and nothing in the output says so — the suite passes, the badge is green, and the protection was never there. That failure mode is not hypothetical here. In the last week alone this repository produced a green suite from a stale object file that did not contain the fix under test, and a green lane running a binary whose build flags nobody checked. Refusing to run suites when the runner disagrees with the lane that produced it is the right shape of fix: it fails closed, before any suite can report a misleading pass.

Details I want to credit specifically, because they are the parts that make it hold up:

Exact-match comparison. [ "$actual" != "$expected"$'\n' ] rather than a substring or a grep means a missing field, a duplicated field, or a future field nobody handled yet all fail closed. A looser check would have silently weakened the gate the first time someone added a third flag.

The marker trick. captured="$("$runner" --build-config && printf '%s' "$marker")" distinguishes "the probe failed" from "the probe printed nothing", which a bare $(...) cannot. That is the difference between a gate and a gate-shaped thing.

Windows binary mode. Setting _setmode(..., _O_BINARY) before printing means the runner's \n does not become \r\n and silently fail an exact comparison on one platform only. That is precisely the kind of cross-platform detail that otherwise surfaces months later as an unexplainable Windows-only red.

I checked one thing before raising it, and it was my misreading, not your scope creep. The diff moves the CBM_TEST_BUILD_FINGERPRINT block, and at first glance it looked like an unadvertised behaviour change riding along. It is a relocation — the block exists on main today at tests/test_main.c:928, and it has to move below the --build-config early exit or the flag would do environment setup before printing. No behaviour change, and your description was accurate.

Both red checks are ours, not yours:

  • === VENDORED INTEGRITY CHECK FAILED === — your diff touches no vendored file. I ran the check against current main just now and it passes cleanly, so whatever that was, it is already resolved upstream of you.
  • MSan daemon_bootstrap_fails_fast_when_daemon_dies_at_publication (test_daemon_bootstrap.c:1116) — the daemon family has a documented readiness-flake history on constrained runners, and this is that neighbourhood rather than anything your change reaches.

What I need from you is a rebase, and nothing else. The CI on this PR is from 2026-09-13 and main has moved a long way since — including #2196 and #2197 today, which also land on scripts/test.sh. A rebase gives us a current verdict on a tree where those two reds should not recur, and then I will merge it.

Thank you for your patience, and for writing the gate to fail closed rather than to be convenient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

github_actions Pull requests that update GitHub Actions code maintainer-notes Internal observations and improvement notes priority/normal Standard review queue; useful PR with ordinary maintainer urgency.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nothing verifies that a sanitized lane actually compiled with CBM_SANITIZED_BUILD (UBSan and trap-UBSan have no probe)

3 participants