Skip to content

fix gh run download <run-id> not getting > 100 artifacts - #5799

Merged
mislav merged 3 commits into
cli:trunkfrom
faubion-hbo:tf-page-run-artifacts
Jul 12, 2022
Merged

fix gh run download <run-id> not getting > 100 artifacts#5799
mislav merged 3 commits into
cli:trunkfrom
faubion-hbo:tf-page-run-artifacts

Conversation

@faubion-hbo

Copy link
Copy Markdown
Contributor

Fixes #5790

Testing

Used this workflow file to create 201 individual artifact uploads to the run, and was able to reproduce the bug not getting all of them when running gh run download <run-id>.

---
name: upload a bunch of run artifacts
on:
  workflow_dispatch:
jobs:
  upload:
    runs-on: ubuntu-latest
    steps:
      - name: generate a bunch of files
        run: |
          for i in {0..201}; do
            echo $i > file-number-$i.txt
          done
      - uses: actions/upload-artifact@v3
        with:
          name: file-number-0
          path: file-number-0.txt
      - uses: actions/upload-artifact@v3
        with:
          name: file-number-1
          path: file-number-1.txt
#     - uses: actions/upload-artifact@v3
#       with:
#         name: file-number-<n>
#         path: file-number-<n>.txt

Compiled change, and then tested which was then able to get all 201 artifacts

@faubion-hbo
faubion-hbo requested a review from a team as a code owner June 14, 2022 18:49
@faubion-hbo
faubion-hbo requested review from mislav and removed request for a team June 14, 2022 18:49
@cliAutomation cliAutomation added the external pull request originating outside of the CLI core team label Jun 14, 2022
Comment on lines +77 to 85
var linkRE = regexp.MustCompile(`<([^>]+)>;\s*rel="([^"]+)"`)

return response.Artifacts, nil
func findNextPage(resp *http.Response) string {
for _, m := range linkRE.FindAllStringSubmatch(resp.Header.Get("Link"), -1) {
if len(m) > 2 && m[2] == "next" {
return m[1]
}
}
return ""
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, I copied this code from elsewhere in the repository, and I would like to be able to reuse that instead of copying. I wasn't able to get import/referencing to work; I would try:

import (
...
  cmdApi "github.com/cli/cli/v2/pkg/cmd/api"
)
...
cmdApi.findNextPage()

but that would fail with findNextPage() is undefined in api package...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's perfectly fine to copy the function over. Lowercase functions in Go (e.g. findNextPage vs. FindNextPage) are private to a package and cannot be used from other packages, like you have found out when trying to invoke cmdApi.findNextPage().

@faubion-hbo

Copy link
Copy Markdown
Contributor Author

would also appreciate advice or PR for testing the change. any pointers welcomed; new to golang

@mislav mislav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good, but the code has inconsistent indentation. You can tell Go to format your code by saying go fmt ./pkg/cmd/run/shared. You can also configure your text editor to always format your Go code on file save (it's what I do).

As for testing, you can create a mock http.Client like this:

reg := &httpmock.Registry{}
defer reg.Verify(t)
reg.Register(
httpmock.REST("GET", "repos/OWNER/REPO/check-runs/123456/annotations"),
httpmock.StatusStringResponse(404, "not found"))
httpClient := &http.Client{Transport: reg}

You can then pass that client to ListArtifacts and use the registry to stub responses, like so

firstRes = httpmock.WithHeader(firstRes, "Link", `<https://api.github.com/search/repositories?page=2&per_page=100&q=org%3Agithub>; rel="next"`)

Comment on lines +77 to 85
var linkRE = regexp.MustCompile(`<([^>]+)>;\s*rel="([^"]+)"`)

return response.Artifacts, nil
func findNextPage(resp *http.Response) string {
for _, m := range linkRE.FindAllStringSubmatch(resp.Header.Get("Link"), -1) {
if len(m) > 2 && m[2] == "next" {
return m[1]
}
}
return ""
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's perfectly fine to copy the function over. Lowercase functions in Go (e.g. findNextPage vs. FindNextPage) are private to a package and cannot be used from other packages, like you have found out when trying to invoke cmdApi.findNextPage().

Comment thread pkg/cmd/run/shared/artifacts.go Outdated
@faubion-hbo

Copy link
Copy Markdown
Contributor Author

@mislav with regards to cmdApi.findNextPage(): would the maintainers be open to making that code public and it being the single source for GitHub API pagination logic? i could see it being a improvement for maintainability

@faubion-hbo
faubion-hbo requested a review from mislav July 5, 2022 22:24

@mislav mislav left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thank you 🙇

Re: findNextPage duplication: you don't have to worry about it. If it gets too repetitive in our codebase, we will extract it separately. Thank you!

@mislav
mislav enabled auto-merge (squash) July 12, 2022 09:46
@mislav
mislav merged commit 11b1059 into cli:trunk Jul 12, 2022
@faubion-hbo

Copy link
Copy Markdown
Contributor Author

thanks for the merge, @mislav! glad to have my 1st OSS contribution here :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external pull request originating outside of the CLI core team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cant download all artifacts it the run have more than 100

3 participants