Add SigstoreVerifier unit tests with mocked `sigstore-go/SignedEntityVerifier` by malancas · Pull Request #10465 · cli/cli

malancas · 2025-02-18T21:05:37Z

This adds unit tests for the SigstoreVerfier that mock the sigstore-go/SignedEntityVerifier, which is the actual struct required for integration tests. I am adding these unit tests because I would like to debug the root cause of the ongoing flaky SigstoreVerifier integration tests noted in #10390.

If these new unit tests consistently pass, then we know the root cause lies with how this code uses the SignedEntityVerifier instead of the surrounding Sigstore verification logic implemented in this package.

Signed-off-by: Meredith Lancaster <malancas@github.com>

…ncas/cli into attestation-verification-unit-tests

Signed-off-by: Meredith Lancaster <malancas@github.com>

malancas · 2025-02-19T01:40:52Z

 }

-func (v *LiveSigstoreVerifier) chooseVerifier(issuer string) (*verify.SignedEntityVerifier, error) {
+func chooseLiveVerifier(noPublicGood bool, issuer, trustedRoot, trustDomain string) (SignedEntityVerifier, error) {


I'd like to refactor this function further in a follow up.

williammartin · 2025-02-19T15:12:02Z

@iamazeem can you please run these tests on your windows machine and see whether the flaky behaviour still exists?

iamazeem · 2025-02-19T17:40:47Z

@williammartin: Here are the tests:

PS C:\Users\azeem\Downloads> gh repo clone cli/cli
PS C:\Users\azeem\Downloads> cd cli
PS C:\Users\azeem\Downloads\cli> gh pr checkout 10465

PS C:\Users\azeem\Downloads\cli> 1..10 | % { go test -tags=integration .\pkg\cmd\attestation\... -count 5 -failfast }
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.070s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.099s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.235s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.088s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.241s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.491s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.207s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.290s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  49.931s
--- FAIL: TestVerifyAttestations (3.13s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (3.13s)
        attestation_integration_test.go:53:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                Error:          "[0xc000a78f10]" should have 2 item(s), but has 1
                Test:           TestVerifyAttestations/all_attestations_pass_verification
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        3.258s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.759s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.109s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.272s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.111s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.339s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.666s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.237s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.338s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  42.959s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        172.967s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.424s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.113s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.290s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.105s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.289s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.528s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.213s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.309s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  41.962s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        172.919s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.112s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.108s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.319s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.476s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.386s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.757s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.268s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.313s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  39.508s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        170.555s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.317s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.112s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.270s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.095s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.273s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.499s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.229s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.284s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  44.468s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        168.539s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.247s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.101s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.235s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.091s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.251s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.487s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.222s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.282s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  41.270s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        172.357s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.459s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.108s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.234s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.098s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.243s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.534s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.219s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.311s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  41.903s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        175.436s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.121s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.097s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.250s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.087s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.239s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.479s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.202s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.251s
--- FAIL: TestLiveSigstoreVerifier (5.30s)
    --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (1.18s)
        sigstore_integration_test.go:80:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:80
                Error:          "[0xc0003ee880]" should have 2 item(s), but has 1
                Test:           TestLiveSigstoreVerifier/with_2/3_verified_attestations
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.455s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        184.865s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.411s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.110s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.287s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.111s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.297s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.533s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.213s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.328s
--- FAIL: TestLiveSigstoreVerifier (2.01s)
    --- FAIL: TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles (0.01s)
        sigstore_integration_test.go:62:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:62
                Error:          Received unexpected error:
                                failed to find recognized issuer from bundle content: failed to create TUF client: rename C:\Users\azeem\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\tuf_tmp1852533849 C:\Users\azeem\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\root.json: Access is denied.
                Test:           TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  2.169s
--- FAIL: TestVerifyIntegration (21.47s)
    --- FAIL: TestVerifyIntegration/with_invalid_SAN (15.74s)
        verify_integration_test.go:94:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verify/verify_integration_test.go:94
                Error:          Error "failed to find recognized issuer from bundle content: failed to create TUF client: tuf refresh failed: Get \"https://tuf-repo-cdn.sigstore.dev/timestamp.json\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" does not contain "verifying with issuer \"sigstore.dev\""
                Test:           TestVerifyIntegration/with_invalid_SAN
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        140.706s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.382s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.104s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.263s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.106s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.270s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.542s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.226s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.303s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  42.627s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        181.641s

iamazeem · 2025-02-19T18:06:06Z

@williammartin: Here's another run:

PS C:\Users\azeem\Downloads\cli> 1..10 | % { go test -tags=integration .\pkg\cmd\attestation\... -count 2 -failfast }
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.448s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.241s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.309s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.182s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.437s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      1.436s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.307s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.337s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  16.332s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        54.035s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   5.353s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.243s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.384s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.359s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.501s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.712s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.524s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.468s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  13.090s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        51.985s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   7.974s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.228s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.327s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.159s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.536s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      1.016s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.288s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.674s
--- FAIL: TestLiveSigstoreVerifier (3.15s)
    --- FAIL: TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles (1.44s)
        sigstore_integration_test.go:63:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:63
                Error:          Not equal:
                                expected: 2
                                actual  : 1
                Test:           TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  3.555s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        57.556s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.989s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.130s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.436s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.187s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.656s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.816s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.390s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.300s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  13.140s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        56.448s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.265s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.103s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.354s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.103s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.340s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.430s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.202s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.258s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  12.096s
--- FAIL: TestVerifyAttestations (2.60s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (2.60s)
        attestation_integration_test.go:53:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                Error:          "[0xc000591a30]" should have 2 item(s), but has 1
                Test:           TestVerifyAttestations/all_attestations_pass_verification
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        2.746s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.024s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.107s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.271s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.100s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.271s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.306s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.149s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.190s
--- FAIL: TestLiveSigstoreVerifier (5.57s)
    --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.81s)
        sigstore_integration_test.go:80:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:80
                Error:          "[0xc0005f5650]" should have 2 item(s), but has 1
                Test:           TestLiveSigstoreVerifier/with_2/3_verified_attestations
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.718s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        52.851s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.038s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.103s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.245s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.097s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.282s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.338s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.201s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.270s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  13.223s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        56.099s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.009s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.105s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.249s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.090s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.268s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.325s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.165s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.200s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  11.562s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        54.091s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.041s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.099s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.281s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.093s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.315s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.418s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.201s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.190s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  15.260s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        61.373s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.029s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.116s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.263s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.092s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.252s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.311s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.146s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.205s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  14.652s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        51.824s

Signed-off-by: Meredith Lancaster <malancas@github.com>

malancas · 2025-02-19T21:06:07Z

Thanks for posting your results @iamazeem. Taking a look based on what you shared

iamazeem · 2025-02-20T05:33:48Z

Ran tests on Windows 11 to make sure it wasn't related to Windows 10:

PS C:\Users\azeem\Downloads\cli> 1..10 | % { go test -tags=integration .\pkg\cmd\attestation\... -count 2 -failfast }
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.332s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.130s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.355s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.132s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.445s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.399s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.182s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.231s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  17.696s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        54.873s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.507s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.212s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.340s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.123s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.346s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.427s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.197s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.307s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  14.179s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        55.913s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.626s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.132s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.448s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.146s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.425s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.385s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.236s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.274s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  12.041s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        52.293s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.435s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.102s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.291s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.106s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.334s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.316s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.191s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.250s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  11.638s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        59.830s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.087s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.098s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.261s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.087s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.261s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.335s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.141s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.196s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  12.604s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        54.050s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.050s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.101s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.252s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.106s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.296s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.316s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.139s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.213s
--- FAIL: TestLiveSigstoreVerifier (4.71s)
    --- FAIL: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.85s)
        sigstore_integration_test.go:80:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:80
                Error:          "[0xc000483d40]" should have 2 item(s), but has 1
                Test:           TestLiveSigstoreVerifier/with_2/3_verified_attestations
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.859s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        56.042s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.092s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.096s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.255s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.092s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.254s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.281s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.207s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.197s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  11.922s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        59.608s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   3.195s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.100s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.257s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.109s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.282s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.296s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.185s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.177s
--- FAIL: TestLiveSigstoreVerifier (2.01s)
    --- FAIL: TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles (0.11s)
        sigstore_integration_test.go:62:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verification/sigstore_integration_test.go:62
                Error:          Received unexpected error:
                                failed to find recognized issuer from bundle content: failed to create TUF client: rename C:\Users\azeem\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\tuf_tmp865874090 C:\Users\azeem\AppData\Local\GitHub CLI\.sigstore\root\tuf-repo-cdn.sigstore.dev\root.json: Access is denied.
                Test:           TestLiveSigstoreVerifier/with_valid_artifact_and_JSON_lines_file_containing_multiple_Sigstore_bundles
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verification  2.164s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        59.067s
FAIL
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   5.266s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.181s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.369s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.125s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.514s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.519s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.560s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.488s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  17.458s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        85.564s
?       github.com/cli/cli/v2/pkg/cmd/attestation       [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/api   4.258s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact      0.163s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/digest       0.368s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/artifact/oci  0.117s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/auth  0.526s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/download      0.540s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/inspect       0.242s
?       github.com/cli/cli/v2/pkg/cmd/attestation/io    [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test  [no test files]
?       github.com/cli/cli/v2/pkg/cmd/attestation/test/data     [no test files]
ok      github.com/cli/cli/v2/pkg/cmd/attestation/trustedroot   0.241s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  29.501s
--- FAIL: TestVerifyAttestations (4.25s)
    --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.01s)
        attestation_integration_test.go:63:
                Error Trace:    C:/Users/azeem/Downloads/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:63
                Error:          Received unexpected error:
                                failed to get bundle issuer: unsupported bundle version: application/vnd.dev.sigstore.bundle+json;version=0.1
                Test:           TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        4.436s
FAIL

williammartin · 2025-02-20T11:38:55Z

Thanks a lot @iamazeem. Frustrating that I can't reproduce this on my parallels windows machine.

williammartin · 2025-02-20T12:49:50Z

I bumped the specs on my VM and it started failing immediately 🥳

williammartin · 2025-02-20T13:45:54Z

For future reference, since testing only the verification package didn't reproduce this flake, I used this LLM created powershell script to pairwise the failing package with other packages:

# Set UTF-8 encoding to avoid garbled characters
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8

# Define the test package
$testPkg = "./pkg/cmd/attestation/verification"

# Get the current working directory (project root)
$cwd = Get-Location

# Get all directories under pkg/cmd/attestation that contain Go files
$allPkgs = Get-ChildItem -Path "pkg/cmd/attestation" -Recurse -Directory | `
    Where-Object { (Get-ChildItem -Path $_.FullName -Filter "*.go").Count -gt 0 } | `
    ForEach-Object { 
        # Convert absolute path to relative path
        $relativePath = $_.FullName.Replace($cwd, "").TrimStart("\", "/")
        "./$relativePath" -replace "\\", "/"  # Ensure Unix-style paths for Go compatibility
    }

# Remove the test package itself from the list
$allPkgs = $allPkgs | Where-Object { $_ -ne $testPkg }

# Create an array to store failing package pairs
$failingPairs = @()

# Iterate over each package and test it alongside the verification package
foreach ($pkg in $allPkgs) {
    Write-Host "`nTesting $testPkg with $pkg..."

    # Construct the Go test command
    $cmd = "go test -tags=integration $testPkg $pkg -count 2 -failfast"

    # Print the command being run
    Write-Host "Running: $cmd"

    # Execute the command and capture both output and exit code
    $output = Invoke-Expression $cmd 2>&1
    $exitCode = $LASTEXITCODE

    # Print the output of the command
    Write-Host "Output:"
    Write-Host $output

    if ($exitCode -ne 0) {
        Write-Host "Test failed for pair: $testPkg + $pkg"
        $failingPairs += "$testPkg + $pkg"

        # Save the failure output to a log file for further debugging
        $output | Out-File -Append -Encoding utf8 "test_failures.log"
    } else {
        Write-Host "Test passed for pair: $testPkg + $pkg"
    }
}

# Output summary
Write-Host "`n=== Test Run Summary ==="
if ($failingPairs.Count -eq 0) {
    Write-Host "No failing pairs detected."
} else {
    Write-Host "Failing pairs detected:"
    $failingPairs | ForEach-Object { Write-Host $_ }
}

Write-Host "`nCheck 'test_failures.log' for failure details."

I absolutely have not read this code and validated it is good 😅 , but it did produce:

=== Test Run Summary ===
Failing pairs detected:
./pkg/cmd/attestation/verification + ./pkg/cmd/attestation/verify

Which makes some logical sense. Indeed, running these two packages together failed:

PS C:\Users\williammartin\workspace\cli> 1..10 | % { go test -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast }
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.018s
--- FAIL: TestVerifyAttestations (1.04s)
    --- FAIL: TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification (0.42s)
        attestation_integration_test.go:65:
                Error Trace:    C:/Users/williammartin/workspace/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:65
                Error:          "[0x4000902af0]" should have 2 item(s), but has 1
                Test:           TestVerifyAttestations/passes_verification_with_2/3_attestations_passing_Sigstore_verification
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        1.129s
FAIL

So hopefully that helps us narrow this down.

williammartin · 2025-02-20T13:49:23Z

In my experience when there are flakes on Windows and not on MacOS/Linux it's frequently because there is a missing file close somewhere.

williammartin · 2025-02-20T14:22:39Z

FWIW this branch contains some modifications that I believe only run 2 tests, and it fails very frequently:

PS C:\Users\williammartin\workspace\cli> go test -v -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast
=== RUN   TestLiveSigstoreVerifier
=== RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
--- PASS: TestLiveSigstoreVerifier (0.78s)
    --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.78s)
=== RUN   TestLiveSigstoreVerifier
=== RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
--- PASS: TestLiveSigstoreVerifier (0.59s)
    --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.59s)
PASS
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  1.499s
=== RUN   TestVerifyAttestations
=== RUN   TestVerifyAttestations/all_attestations_pass_verification
    attestation_integration_test.go:53:
                Error Trace:    C:/Users/williammartin/workspace/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                Error:          "[0x400045ae30]" should have 2 item(s), but has 1
                Test:           TestVerifyAttestations/all_attestations_pass_verification
--- FAIL: TestVerifyAttestations (0.43s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.43s)
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        0.548s
FAIL

malancas · 2025-02-20T15:09:41Z

FWIW this branch contains some modifications that I believe only run 2 tests, and it fails very frequently:

PS C:\Users\williammartin\workspace\cli> go test -v -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast
=== RUN   TestLiveSigstoreVerifier
=== RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
--- PASS: TestLiveSigstoreVerifier (0.78s)
    --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.78s)
=== RUN   TestLiveSigstoreVerifier
=== RUN   TestLiveSigstoreVerifier/with_2/3_verified_attestations
--- PASS: TestLiveSigstoreVerifier (0.59s)
    --- PASS: TestLiveSigstoreVerifier/with_2/3_verified_attestations (0.59s)
PASS
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  1.499s
=== RUN   TestVerifyAttestations
=== RUN   TestVerifyAttestations/all_attestations_pass_verification
    attestation_integration_test.go:53:
                Error Trace:    C:/Users/williammartin/workspace/cli/pkg/cmd/attestation/verify/attestation_integration_test.go:53
                Error:          "[0x400045ae30]" should have 2 item(s), but has 1
                Test:           TestVerifyAttestations/all_attestations_pass_verification
--- FAIL: TestVerifyAttestations (0.43s)
    --- FAIL: TestVerifyAttestations/all_attestations_pass_verification (0.43s)
FAIL
FAIL    github.com/cli/cli/v2/pkg/cmd/attestation/verify        0.548s
FAIL

Thanks, going to keep debugging with this.

williammartin · 2025-02-21T10:59:07Z

Closing in favour of #10478, but if there's stuff here you want to maintain, please reopen!

malancas added 5 commits February 18, 2025 10:02

add unit tests for verification logic

306554d

Signed-off-by: Meredith Lancaster <malancas@github.com>

create SignedEntityVerifier interface for better testing

e36b604

Signed-off-by: Meredith Lancaster <malancas@github.com>

add test verifier implementation with mock signed entity verifier

d48c6e0

Signed-off-by: Meredith Lancaster <malancas@github.com>

add sigstore verifier unit tests, move common test helpers

2737902

Signed-off-by: Meredith Lancaster <malancas@github.com>

comment

e46ab29

Signed-off-by: Meredith Lancaster <malancas@github.com>

malancas changed the title ~~Add SigstoreVerifier unit tests with mocked signed entity verifier~~ Add SigstoreVerifier unit tests with mocked sigstore-go/SignedEntityVerifier Feb 18, 2025

malancas marked this pull request as ready for review February 18, 2025 21:20

malancas requested a review from a team as a code owner February 18, 2025 21:20

cliAutomation added the external pull request originating outside of the CLI core team label Feb 18, 2025

malancas marked this pull request as draft February 18, 2025 22:00

malancas and others added 6 commits February 18, 2025 15:50

use func field to control signed entity verification choice

578ccd8

Signed-off-by: Meredith Lancaster <malancas@github.com>

clean up test fixtures

8ad81d4

Signed-off-by: Meredith Lancaster <malancas@github.com>

add unit test that mirrors the integration test

b175841

Signed-off-by: Meredith Lancaster <malancas@github.com>

clean up fields

147336a

Signed-off-by: Meredith Lancaster <malancas@github.com>

comment

0065bcb

Signed-off-by: Meredith Lancaster <malancas@github.com>

Merge branch 'trunk' into attestation-verification-unit-tests

3bfb435

malancas marked this pull request as ready for review February 18, 2025 23:23

malancas added 3 commits February 18, 2025 16:28

add mutex

5fb4148

Signed-off-by: Meredith Lancaster <malancas@github.com>

Merge branch 'attestation-verification-unit-tests' of github.com:mala…

c0ea31d

…ncas/cli into attestation-verification-unit-tests

only use require

0e9eef6

Signed-off-by: Meredith Lancaster <malancas@github.com>

malancas commented Feb 19, 2025

View reviewed changes

remove unneeded fields

9abf7eb

Signed-off-by: Meredith Lancaster <malancas@github.com>

williammartin closed this Feb 21, 2025

malancas deleted the attestation-verification-unit-tests branch May 28, 2025 22:56

Navigation Menu

Uh oh!

Add SigstoreVerifier unit tests with mocked `sigstore-go/SignedEntityVerifier` - #10465

Add SigstoreVerifier unit tests with mocked `sigstore-go/SignedEntityVerifier`#10465
malancas wants to merge 15 commits into
cli:trunkfrom
malancas:attestation-verification-unit-tests

malancas commented Feb 18, 2025 •

edited

Loading

Uh oh!

malancas Feb 19, 2025

Uh oh!

williammartin commented Feb 19, 2025

Uh oh!

iamazeem commented Feb 19, 2025 •

edited

Loading

Uh oh!

iamazeem commented Feb 19, 2025

Uh oh!

malancas commented Feb 19, 2025

Uh oh!

iamazeem commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

malancas commented Feb 20, 2025

Uh oh!

williammartin commented Feb 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

malancas commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

malancas Feb 19, 2025

Choose a reason for hiding this comment

Uh oh!

williammartin commented Feb 19, 2025

Uh oh!

iamazeem commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

iamazeem commented Feb 19, 2025

Uh oh!

malancas commented Feb 19, 2025

Uh oh!

iamazeem commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

williammartin commented Feb 20, 2025

Uh oh!

malancas commented Feb 20, 2025

Uh oh!

williammartin commented Feb 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

malancas commented Feb 18, 2025 •

edited

Loading

iamazeem commented Feb 19, 2025 •

edited

Loading