t: detect errors outside of test cases#2270
Open
pks-gitlab wants to merge 13 commits intogit:masterfrom
Open
Conversation
Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v5: - Allow opting in via `GIT_TEST_USE_SET_E=yes` and enable this option for Linux CI jobs. - EDITME: use bulletpoints and terse descriptions. - Link to v4: https://patch.msgid.link/20260417-b4-pks-tests-with-set-e-v4-0-44d43efdafb1@pks.im Changes in v4: - Simplify how we read a multi-line variable value. - Link to v3: https://patch.msgid.link/20260416-b4-pks-tests-with-set-e-v3-0-7a90e5dccadd@pks.im Changes in v3: - Adapt `linux-TEST-vars` job to use Bash instead of Dash. Ubuntu packet mirrors seem to be having problems, so I wasn't able to get past installing dependencies in any jobs. All to say that I couldn't verify that this works as expected :/ - Link to v2: https://patch.msgid.link/20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` consistently instead of `|| true`. - Fix up a couple of tests that fail on FreeBSD 15. The test suite is now passing on this system, too. - Only enable `set -e` on Bash 5 and newer. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> Cc: Jeff King <peff@peff.net> Cc: SZEDER Gábor <szeder.dev@gmail.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 5, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ], "v2": [ "20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im" ], "v3": [ "20260416-b4-pks-tests-with-set-e-v3-0-7a90e5dccadd@pks.im" ], "v4": [ "20260417-b4-pks-tests-with-set-e-v4-0-44d43efdafb1@pks.im" ] } } }
|
There is an issue in commit 642490e:
|
pks-gitlab
pushed a commit
to pks-gitlab/git
that referenced
this pull request
Apr 14, 2026
Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` instead of `|| true`. - Fix handling ot DATE_PARSER prerequisite in t9501. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 2, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ] } } }
defca84 to
f0c9cde
Compare
|
There is an issue in commit 153c5d4:
|
f0c9cde to
9ff7caa
Compare
|
There is an issue in commit 153c5d4:
|
pks-gitlab
pushed a commit
to pks-gitlab/git
that referenced
this pull request
Apr 14, 2026
Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` instead of `|| true`. - Fix handling ot DATE_PARSER prerequisite in t9501. - Fix up a couple of tests that fail on FreeBSD 15. The test suite is now passing on this system, too. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 2, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ] } } }
9ff7caa to
edf3b15
Compare
|
There is an issue in commit 9cf1860:
|
edf3b15 to
387cbfd
Compare
|
There is an issue in commit 9cf1860:
|
|
There are issues in commit 387cbfd:
|
387cbfd to
3bc33df
Compare
|
There is an issue in commit 9cf1860:
|
pks-gitlab
pushed a commit
to pks-gitlab/git
that referenced
this pull request
Apr 15, 2026
Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` instead of `|| true`. - Fix up a couple of tests that fail on FreeBSD 15. The test suite is now passing on this system, too. - Only enable `set -e` on Bash 5 and newer. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 2, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ] } } }
622bd75 to
ed0e7ca
Compare
pks-gitlab
pushed a commit
to pks-gitlab/git
that referenced
this pull request
Apr 16, 2026
- EDITME: describe what is new in this series revision. - EDITME: use bulletpoints and terse descriptions. - Link to v2: https://patch.msgid.link/20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im t: detect errors outside of test cases Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` consistently instead of `|| true`. - Fix up a couple of tests that fail on FreeBSD 15. The test suite is now passing on this system, too. - Only enable `set -e` on Bash 5 and newer. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> Cc: Jeff King <peff@peff.net> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 3, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ], "v2": [ "20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im" ] } } }
ed0e7ca to
222eb4e
Compare
|
There are issues in commit 3e34f60:
|
We have a couple of calls to `test_match_signal ()` where we execute a Git command and expect it to die with a specific signal. These calls will essentially execute the process in a subshell via `foo; echo $?`, but as we expect `foo` to fail this will cause the overall subshell to fail once we `set -e`. Fix this issue by using `foo && echo 0 || echo $?` instead. Signed-off-by: Patrick Steinhardt <ps@pks.im>
The helper function `test_must_fail ()` executes a specific Git command that may or may not fail in a specific way. This is done by executing the command in question and then comparing its exit code against a set of conditions. This works, but once we run our test suite with `set -e` we may bail out of `test_must_fail ()` early in case the command actually fails, even though we expect it to fail. Prepare for this change by handling the failed case with `||`. Signed-off-by: Patrick Steinhardt <ps@pks.im>
We have a couple of calls to `stop_git_daemon ()` outside of specific
test cases that will kill a backgrounded git-daemon(1) process and
expect the process with a specific error code. While these function
calls do end up killing git-daemon(1), the error handling we have in
those contexts is basically ineffective. So while we expect the process
to exit with a specific error code, we will just continue with any error
in case it doesn't.
This will change once we enable `set -e` in a subsequent commit. There's
two issues though that will make this _always_ fail:
- Our call to `wait` is expected to fail, but because it's not part of
a condition it will cause us to bail out immediately with `set -e`.
- We try to kill git-daemon(1) a second time via the pidfile. We can
generally expect that this is the same PID though as we had in the
"GIT_DAEMON_PID" environment variable, and thus it's more likely
than not that we have already killed it, and the call to kill will
fail.
Prepare for this change by handling the failure of `wait` with `||` and
by silencing failures of the second call to `kill`.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
We have a couple of calls to `git config --unset` that ultimately end up as no-ops as the configuration variables aren't set (anymore) in the first place. These calls are mostly intended to recover unconditionally from tests that may have executed only partially, but they'll ultimately fail during a normal test run. This hasn't been a problem until now as we aren't running tests with `set -e`. This is about to change though, so let's silence the case where we cannot unset the config keys. Signed-off-by: Patrick Steinhardt <ps@pks.im>
We have some test in our test suite where we use the pattern of `test ... && test_expect_succeess` to conditionally execute a test. The problem is that when we decide to not execute the test, we'll indeed skip the test, but the overall statement will also be unsuccessful. This will become a problem once we enable `set -e`. Prepare for this future by turning this into a proper conditional, which is also a bit easier to read overall. Signed-off-by: Patrick Steinhardt <ps@pks.im>
Several of our tests verify whether a certain binary can be executed, potentially skipping tests in case we cannot, for example because the binary doesn't exist. In those cases we often run the binary outside of any conditionally. This will start to fail once we enable `set -e`, as that will cause us to bail out the test immediately. Improve these tests by executing them inside of a conditional instead. Signed-off-by: Patrick Steinhardt <ps@pks.im>
Both `test_when_finished ()` and `test_atexit ()` build up a chain of
cleanup commands by prepending each new command to the existing cleanup
string. To preserve the exit code of the test body across cleanup
execution, we append the following logic:
} && (exit "$eval_ret"); eval_ret=$?; ...
The intent of this is to run the cleanup block and then unconditionally
restore `eval_ret`. The original behaviour of this is is:
+------------------+---------+------------------------------------+
|test body │ cleanup │ old behaviour │
+------------------+---------+------------------------------------+
│pass (eval_ret=0) | pass │ && taken -> (exit 0) -> eval_ret=0 |
+------------------+---------+------------------------------------+
│pass (eval_ret=0) | fail │ && not taken -> eval_ret=$? |
+------------------+---------+------------------------------------+
│fail (eval_ret=1) | pass │ && taken -> (exit 1) -> eval_ret=1 |
+------------------+---------+------------------------------------+
│fail (eval_ret=1) | fail | && not taken -> eval_ret=$? |
+------------------+---------+------------------------------------+
This logic will start to fail once we enable `set -e`. When `$eval_ret`
is non-zero, the subshell we create will fail, and with `set -e` we'll
thus bail out without evaluating the logic after the semicolon.
Fix this issue by instead using `|| eval_ret=\$?; ...`. Besides being
a bit simpler, it also retains the original behaviour:
+------------------+---------+------------------------------------+
|test body │ cleanup │ old behaviour │
+------------------+---------+------------------------------------+
│pass (eval_ret=0) | pass │ || not taken -> eval_ret unchanged |
+------------------+---------+------------------------------------+
│pass (eval_ret=0) | fail │ || taken -> eval_ret=$? |
+------------------+---------+------------------------------------+
│fail (eval_ret=1) | pass │ || not taken -> eval_ret unchanged |
+------------------+---------+------------------------------------+
│fail (eval_ret=1) | fail | || taken -> eval_ret=$? |
+------------------+---------+------------------------------------+
Signed-off-by: Patrick Steinhardt <ps@pks.im>
In t0008 we use `grep -v` in a subshell, but expect that this command will sometimes not match anything. This would cause grep(1) to return an error code, but given that we don't run with `set -e` we swallow this error. We're about to enable `set -e`. Prepare for this by ignoring any errors. Signed-off-by: Patrick Steinhardt <ps@pks.im>
In t1301 we're trying to remove any potentially-existing default ACLs that might exist on the transh directory by executing setfacl(1). According to 8ed0a74 (t1301-shared-repo.sh: don't let a default ACL interfere with the test, 2008-10-16), this is done because we play around with permissions and umasks in this test suite. The setfacl(1) binary may not exist on some systems though, even though tests ultimately still pass. This doesn't matter currently, but will cause the test to fail once we start running with `set -e`. Silence such failures by ignoring failures here. Signed-off-by: Patrick Steinhardt <ps@pks.im>
In `test_bisection_diff ()` we use `expr` to perform some math. This command has some gotchas though in that it will only return success when the result is neither null nor zero. In some of our cases though it actually _is_ zero, and that will cause the expressions to fail once we enable `set -e`. Prepare for this change by instead using `$(( ))`, which doesn't have the same issue. While at it, modernize the function a tiny bit. Signed-off-by: Patrick Steinhardt <ps@pks.im>
In t9902 we're using the `read` builtin to read some values into a variable. This is done by using `-d ""`, which cause us to read until the end of the heredoc. As the read is terminated by EOF, the command will end up returning a non-zero error code. This hasn't been an issue until now as we didn't run with `set -e`, but that'll change in a subsequent commit. Prepare for this change by not using read at all, as we can simply store the multi-line value directly. Suggested-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Patrick Steinhardt <ps@pks.im>
pks-gitlab
pushed a commit
to pks-gitlab/git
that referenced
this pull request
Apr 20, 2026
- EDITME: describe what is new in this series revision. - EDITME: use bulletpoints and terse descriptions. - Link to v4: https://patch.msgid.link/20260417-b4-pks-tests-with-set-e-v4-0-44d43efdafb1@pks.im t: detect errors outside of test cases Hi, this is a follow-up to the recent discussion we had around `set -e` to make our tests more robust and basically supersedes Junio's [1]. I've tested the patches with both Bash and Dash, and all tests are passing on my machine with both of them. CI seems to be happy, as well. But I would expect that this change probably has some fallout, even though I hope that it's generally going to be small and contained. This series is based on 8c9303b (Merge branch 'jc/no-writev-does-not-work', 2026-04-10). I've created an MR with GitLab [2] and a PR with GitHub [3] to verify that these changes work on both platforms. Changes in v4: - Simplify how we read a multi-line variable value. - Link to v3: https://patch.msgid.link/20260416-b4-pks-tests-with-set-e-v3-0-7a90e5dccadd@pks.im Changes in v3: - Adapt `linux-TEST-vars` job to use Bash instead of Dash. Ubuntu packet mirrors seem to be having problems, so I wasn't able to get past installing dependencies in any jobs. All to say that I couldn't verify that this works as expected :/ - Link to v2: https://patch.msgid.link/20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im Changes in v2: - Use `ret=0; $command || ret=$?` pattern. - Restore `echo 0` in SIGPIPE tests. - Fix "lib-git-svn.sh" to gracefully handle the case where SVN Perl modules aren't installed. - Use `|| :` consistently instead of `|| true`. - Fix up a couple of tests that fail on FreeBSD 15. The test suite is now passing on this system, too. - Only enable `set -e` on Bash 5 and newer. - Link to v1: https://patch.msgid.link/20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im Thanks! Patrick [1]: <20260325062114.2067946-1-gitster@pobox.com> [2]: https://gitlab.com/gitlab-org/git/-/merge_requests/541 [3]: git#2270 To: git@vger.kernel.org Cc: Junio C Hamano <gitster@pobox.com> Cc: Jeff King <peff@peff.net> Cc: SZEDER Gábor <szeder.dev@gmail.com> --- b4-submit-tracking --- # This section is used internally by b4 prep for tracking purposes. { "series": { "revision": 5, "change-id": "20260410-b4-pks-tests-with-set-e-3ae479b24b51", "prefixes": [], "presubject": "", "history": { "v1": [ "20260413-b4-pks-tests-with-set-e-v1-0-5b83763a0e84@pks.im" ], "v2": [ "20260415-b4-pks-tests-with-set-e-v2-0-4e4904a96f15@pks.im" ], "v3": [ "20260416-b4-pks-tests-with-set-e-v3-0-7a90e5dccadd@pks.im" ], "v4": [ "20260417-b4-pks-tests-with-set-e-v4-0-44d43efdafb1@pks.im" ] } } }
222eb4e to
5c5c82e
Compare
|
There are issues in commit c60f44d:
|
5c5c82e to
ba7238f
Compare
|
There is an issue in commit 04b69f5:
|
We have recently merged a patch series that had a simple misspelling of `test_expect_success`. Instead of making our tests fail though, this typo went completely undetected and all of our tests passed, which is of course unfortunate. This is a more general issue with our test suite: all commands that run outside of a specific test case can fail, and if we don't explicitly check for such failure then this failure will be silently ignored. Improve the status quo by enabling the errexit option so that any such unchecked failures will cause us to abort immediately. Note that for now, we only enable this option for Bash 5 and newer. This is because other shells have wildly different behaviour, and older versions of Bash (especially on macOS) are buggy. The list of enabled shells may be extended going forward. Helped-by: Jeff King <peff@peff.net> Signed-off-by: Patrick Steinhardt <ps@pks.im>
ba7238f to
6bf6753
Compare
|
There is an issue in commit 04b69f5:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR verifies that CI is succeeding with those changes.