Skip to content

Allow injection of TUFMetadataDir in tests - #10478

Merged
williammartin merged 1 commit into
trunkfrom
wm/fix-flaky-tuf-client-race
Feb 21, 2025
Merged

Allow injection of TUFMetadataDir in tests#10478
williammartin merged 1 commit into
trunkfrom
wm/fix-flaky-tuf-client-race

Conversation

@williammartin

Copy link
Copy Markdown
Member

Description

Fixes #10390 (hopefully)

As described in #10390 (comment) the fundamental issue here is that there is a third party client (TUF) that is usually instantiated once on a gh attestation invocation but gets instantiated once per test. It downloads and stores json metadata in a shared location. It does this by writing a temp file and then using os.Rename to move it to the right place…godoc says:

Even within the same directory, on non-Unix platforms Rename is not an atomic operation.

This PR allows for injection of this location.

Reviewer Notes

I'm not really sure whether TUF is something that is supposed to leak out of the verification package, so this may be leaky. I chose to use an Option type to signify the possible absence of this because the alternative without larger refactoring is to pass around bare strings and rely on the zero value, which is pretty uninformative to the reader.

An alternative might be to have a struct that can be passed to the DefaultOptionsWithCacheSetting and GitHubTUFOptions functions but that seems overblown right now.

I'm not wedded to either of these things, just wanted to get something out that fixed the flaky tests, and it seems to have:

PS C:\Users\williammartin\workspace\cli> 1..10 | % { go test -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast }
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.849s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        22.185s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.424s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.310s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.837s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.663s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.917s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.998s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.919s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.572s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.875s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        22.144s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.236s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.297s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.907s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.929s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.854s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.692s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.439s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.337s

This avoids multiple tests using the same dir for metadata, which was causing flakes

@malancas malancas left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@williammartin
williammartin merged commit 7fa4882 into trunk Feb 21, 2025
@williammartin
williammartin deleted the wm/fix-flaky-tuf-client-race branch February 21, 2025 10:58
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Mar 6, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [cli/cli](https://github.com/cli/cli) | minor | `v2.67.0` -> `v2.68.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>cli/cli (cli/cli)</summary>

### [`v2.68.0`](https://github.com/cli/cli/releases/tag/v2.68.0): GitHub CLI 2.68.0

[Compare Source](cli/cli@v2.67.0...v2.68.0)

#### What's Changed

##### ✨ Features

-   \[gh repo view] Improve error message for forked repo by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10334
-   Add signer-digest, source-ref, and source-digest options for `gh attestation verify` by [@&#8203;malancas](https://github.com/malancas) in cli/cli#10308
-   \[gh pr checkout] Add --no-tags option to git fetch commands in checkout by [@&#8203;latzskim](https://github.com/latzskim) in cli/cli#10479
-   \[`gh issue/pr comment`] Add `--create-if-none` and prompts to create a comment if no comment already exists  by [@&#8203;latzskim](https://github.com/latzskim) in cli/cli#10427
-   \[gh cache delete --all] Add `--succeed-on-no-caches` flag to return exit code 0 by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10327
-   \[gh release create] Fail when there are no new commits since the last release by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10398
-   update default upstream when forking repo during MR creation by [@&#8203;daviddl9](https://github.com/daviddl9) in cli/cli#10458

##### 🐛 Fixes

-   Refactor `GetLocalAttestations` and clean up custom registry transport by [@&#8203;malancas](https://github.com/malancas) in cli/cli#10382
-   Check `GH_REPO` too in addition to `--repo` for disambiguation by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10539
    -   (Fixes `gh secret` subcommands not working outside of a repository)
-   Fix unhandled panic in FindWorkflow and add tests by [@&#8203;jtmcg](https://github.com/jtmcg) in cli/cli#10521
-   Fix checkout when URL arg is from fork and cwd is upstream by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10512
-   \[gh api] Escape package name (URL encoding) for packages endpoint by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10384
-   Fix `remoteResolver` caching issue by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10456
-   Fix gh project item-edit to allow --number 0 as a valid value by [@&#8203;aryanbhosale](https://github.com/aryanbhosale) in cli/cli#10417
-   Add mutex to fix race in attestation test client by [@&#8203;codysoyland](https://github.com/codysoyland) in cli/cli#10439
-   Base64 decode GPG passphrase in deployment workflow by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10546

##### 📚 Docs & Chores

-   Deep Dive Document Release Process by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10503
-   Inconsistent format of examples in help text by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10508
-   Inconsistent format of description of flags (starting with lowercase letter) by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10507
-   Update Go version to 1.23 in CONTRIBUTING.md by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10504
-   Fix minor auth login help typo by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10501
-   docs: document how to revoke `gh` OAuth tokens in `auth logout`'s help by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10490
-   chore: update codespaces Go version by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10491
-   Allow injection of TUFMetadataDir in tests by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10478
-   refactor: use a more straightforward return value by [@&#8203;beforetech](https://github.com/beforetech) in cli/cli#10489
-   Use subtests in attestation verification integration tests by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10463
-   Fix typo in README by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10445
-   Update usage to lower-kebab-case by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10447
-   Standardize URLs by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10429
-   Remove trailing whitespace by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10430

##### :dependabot: Dependencies

-   Bump actions/attest-build-provenance from 2.2.0 to 2.2.2 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10518
-   Bump github.com/go-jose/go-jose/v4 from 4.0.2 to 4.0.5 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10499
-   Bump github.com/spf13/pflag from 1.0.5 to 1.0.6 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10338

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xODYuMCIsInVwZGF0ZWRJblZlciI6IjM5LjE4Ni4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestLiveSigstoreVerifier/with_2/3_verified_attestations is flaky

2 participants