From 6b1908564cfd186134dd02db8acefbebdb8912fe Mon Sep 17 00:00:00 2001 From: eq19 Date: Sat, 22 Feb 2025 13:58:09 +0000 Subject: [PATCH 0001/1030] cleaning strategies --- .github/CODEOWNERS | 89 +- .github/FUNDING.yml | 1 + .github/ISSUE_TEMPLATE/bug_report.yml | 54 - .github/ISSUE_TEMPLATE/config.yml | 5 - .github/ISSUE_TEMPLATE/feature_request.yml | 19 - .github/ISSUE_TEMPLATE/other.yml | 19 - .github/actions/1-addition/1-maps/action.yml | 158 + .github/actions/1-addition/2-feed/action.yml | 182 + .github/actions/1-addition/3-lexer/action.yml | 134 + .../2-multiplication/1-maps/action.yml | 139 + .../2-multiplication/2-feed/action.yml | 217 + .../2-multiplication/3-lexer/action.yml | 166 + .../3-exponentiation/4-parser/action.yml | 105 + .../3-exponentiation/5-syntax/action.yml | 120 + .../3-exponentiation/6-grammar/action.yml | 140 + .../actions/4-identition/4-parser/action.yml | 98 + .../actions/4-identition/5-syntax/action.yml | 122 + .../actions/4-identition/6-grammar/action.yml | 139 + .github/actions/reserves/ubuntu/README.md | 4 + .../ubuntu/download-rootfs/action.yaml | 103 + .../download-rootfs/download-rootfs.ps1 | 110 + .../reserves/ubuntu/wsl-bash/action.yaml | 60 + .../reserves/ubuntu/wsl-checkout/action.yaml | 38 + .../reserves/ubuntu/wsl-install/action.yaml | 57 + .github/csc.json | 18 + .github/dco.yml | 4 + .github/dependabot.yml | 14 +- .github/entrypoint/artifact.sh | 124 + .github/entrypoint/artifact/BosonJet.c | 326 + .github/entrypoint/artifact/BosonJet.h | 92 + .github/entrypoint/artifact/BosonJetBorn.c | 10 + .github/entrypoint/artifact/BosonJetBorn.h | 9 + .github/entrypoint/artifact/BosonJetCSS.c | 133 + .github/entrypoint/artifact/BosonJetCSS.h | 33 + .github/entrypoint/artifact/BosonJetLL.c | 42 + .github/entrypoint/artifact/BosonJetLL.h | 23 + .github/entrypoint/artifact/BosonJetLO.c | 198 + .github/entrypoint/artifact/BosonJetLO.h | 31 + .github/entrypoint/artifact/BosonJetNLL.c | 210 + .github/entrypoint/artifact/BosonJetNLL.h | 55 + .github/entrypoint/artifact/BosonJetResum.c | 280 + .github/entrypoint/artifact/BosonJetResum.h | 78 + .github/entrypoint/artifact/ElectroWeak.c | 27 + .github/entrypoint/artifact/ElectroWeak.h | 20 + .github/entrypoint/artifact/Makefile | 99 + .github/entrypoint/artifact/Particle.c | 30 + .github/entrypoint/artifact/Particle.h | 26 + .github/entrypoint/artifact/QCD.c | 122 + .github/entrypoint/artifact/QCD.h | 39 + .github/entrypoint/artifact/README.md | 3 + .github/entrypoint/artifact/StandardModel.h | 19 + .github/entrypoint/artifact/born.c | 52 + .github/entrypoint/artifact/css.c | 72 + .github/entrypoint/artifact/main.c | 114 + .github/entrypoint/artifact/nllR.c | 67 + .github/entrypoint/artifact/python/LICENSE | 21 + .github/entrypoint/artifact/python/README.md | 32 + .../artifact/python/colour_matrices.py | 123 + .../entrypoint/artifact/python/gellmann.ipynb | 815 ++ .../entrypoint/artifact/python/gellmann.py | 92 + .../entrypoint/artifact/python/inventory.py | 218 + .../entrypoint/artifact/python/keywords.lst | 7614 +++++++++++++++++ .../entrypoint/artifact/python/particle.ipynb | 374 + .../entrypoint/artifact/python/pyproject.toml | 29 + .github/entrypoint/artifact/python/setup.py | 29 + .../artifact/python/src/lgt/__init__.py | 0 .../artifact/python/src/lgt/group/__init__.py | 0 .../python/src/lgt/group/generators.py | 55 + .../artifact/python/src/lgt/group/group.py | 403 + .../python/src/lgt/lattice/__init__.py | 0 .../src/lgt/lattice/su3/numpy/__init__.py | 0 .../src/lgt/lattice/su3/numpy/lattice.py | 178 + .../src/lgt/lattice/su3/pytorch/__init__.py | 0 .../lgt/lattice/su3/tensorflow/__init__.py | 0 .../src/lgt/lattice/su3/tensorflow/lattice.py | 169 + .../python/src/lgt/lattice/u1/__init__.py | 0 .../src/lgt/lattice/u1/numpy/__init__.py | 0 .../src/lgt/lattice/u1/numpy/lattice.py | 204 + .../src/lgt/lattice/u1/pytorch/__init__.py | 0 .../src/lgt/lattice/u1/pytorch/lattice.py | 258 + .../src/lgt/lattice/u1/tensorflow/__init__.py | 0 .../src/lgt/lattice/u1/tensorflow/lattice.py | 263 + .../artifact/python/src/params/spaces.json | 54 + .github/entrypoint/artifact/resum.c | 100 + .github/entrypoint/artifact/resumLL.c | 88 + .github/entrypoint/artifact/test_qcd.c | 19 + .github/entrypoint/artifact/tot.c | 54 + .github/entrypoint/cleanup_bundler.rb | 10 + .github/entrypoint/config.sh | 14 + .github/entrypoint/entrypoint_.sh | 202 + .github/entrypoint/front.sh | 41 + .github/entrypoint/gcp_access.py | 102 + .github/entrypoint/gcp_token.py | 101 + .github/entrypoint/gist.sh | 30 + .github/entrypoint/github_pages.sh | 20 + .github/entrypoint/idle.sh | 12 + .github/entrypoint/init.sh | 103 + .github/entrypoint/pinned_repos.rb | 31 + .github/entrypoint/prime_list.py | 47 + .github/entrypoint/remote.sh | 16 + .github/entrypoint/restore.sh | 24 + .github/entrypoint/sort.sh | 34 + .github/entrypoint/startup.sh | 7 + .github/entrypoint/test.py | 322 + .github/entrypoint/wiki.sh | 40 + .github/pull_request_template.md | 21 - .github/stale.yml | 63 - .github/templates/CODE_OF_CONDUCT.md | 49 + .github/templates/CONTRIBUTING.md | 16 + .github/templates/PULL_REQUEST_TEMPLATE.md | 29 + .github/templates/bug_report.md | 32 + .github/templates/bug_report.yml | 58 + .github/templates/config.yml | 14 + .github/templates/feature_request.md | 16 + .github/templates/feature_request.yml | 38 + .github/templates/mapping-errors.md | 14 + .github/templates/other-issues.md | 10 + .../templates/standards-change-proposal.md | 10 + .github/workflows/main.yml | 170 + .github/workflows/reserves/__config.yml | 239 + .../workflows/reserves/account/blog-posts.yml | 27 + .../workflows/reserves/account/star-fork.yml | 23 + .../workflows/reserves/account/starred.yml | 32 + .../reserves/account/topFollowers.yml | 35 + .../workflows/reserves/account/wakatime.yml | 19 + .../{ => reserves/algorithms}/build.yml | 2 +- .../algorithms}/directory_writer.yml | 0 .../algorithms}/project_euler.yml | 4 +- .../{ => reserves/algorithms}/ruff.yml | 2 +- .../{ => reserves/algorithms}/sphinx.yml | 4 +- .../workflows/reserves/basic-validation.yml | 17 + .github/workflows/reserves/build.yml | 111 + .github/workflows/reserves/check-dist.yml | 17 + .github/workflows/reserves/ci.yml | 83 + .../reserves/cloud-init/alpine-unittests.yml | 74 + .../reserves/cloud-init/check_format.yml | 114 + .github/workflows/reserves/cloud-init/cla.yml | 40 + .../reserves/cloud-init/doc-autolabel.yml | 12 + .../reserves/cloud-init/integration.yml | 89 + .../reserves/cloud-init/linkcheck.yml | 39 + .../reserves/cloud-init/scheduled.yml | 59 + .../workflows/reserves/cloud-init/stale.yml | 24 + .../workflows/reserves/cloud-init/unit.yml | 45 + .../workflows/reserves/codeql-analysis.yml | 71 + .github/workflows/reserves/codeql.yml | 70 + .github/workflows/reserves/config.yml | 20 + .github/workflows/reserves/conformance.yml | 53 + .github/workflows/reserves/dependabot.yml | 21 + .../reserves/devcontainer-cli/build-chat.yml | 56 + .../devcontainer-cli/dev-containers.yml | 122 + .../publish-dev-containers.yml | 46 + .../test-plan-item-validator.yml | 25 + .../devcontainer-cli/test-windows.yml | 63 + .../reserves/devcontainer/ci_branch.yml | 33 + .../reserves/devcontainer/ci_common.yml | 1188 +++ .../reserves/devcontainer/ci_main.yml | 22 + .../reserves/devcontainer/clean_tags.sh | 75 + .../reserves/devcontainer/clean_untagged.sh | 34 + .../reserves/devcontainer/codeql-analysis.yml | 71 + .../reserves/devcontainer/pr-bot.yml | 81 + .../reserves/devcontainer/pr-closed.yml | 26 + .../reserves/devcontainer/pr_auto.yml | 26 + .../devcontainer/untagged-image-cleanup.yml | 23 + .github/workflows/reserves/distribution.yml | 128 + .github/workflows/reserves/e2e-tests.yml | 483 ++ .github/workflows/reserves/e2e.yml | 39 + .github/workflows/reserves/elixir-ci.yml | 127 + .github/workflows/reserves/fossa.yml | 22 + .../reserves/github-script/check-dist.yml | 42 + .../workflows/reserves/github-script/ci.yml | 16 + .../github-script/codeql-analysis.yml | 72 + .../reserves/github-script/integration.yml | 325 + .../reserves/github-script/licensed.yml | 24 + .../github-script/pull-request-test.yml | 45 + .../reserves/github-script/stale.yml | 31 + .../iree/ARTIFACT_SUMMARY_TEMPLATE.md | 47 + .github/workflows/reserves/iree/benchmark.yml | 296 + .../reserves/iree/benchmark_compilation.yml | 114 + .../reserves/iree/benchmark_execution.yml | 186 + .../reserves/iree/benchmark_large.yml | 138 + .../reserves/iree/benchmark_trigger.yml | 144 + .github/workflows/reserves/iree/build_all.yml | 124 + .../reserves/iree/build_benchmark_tools.yml | 132 + .../iree/build_e2e_test_artifacts.yml | 145 + .../workflows/reserves/iree/build_package.yml | 320 + .../reserves/iree/bump_torch_mlir.yml | 75 + .github/workflows/reserves/iree/ci.yml | 408 + .../reserves/iree/ci_linux_arm64_clang.yml | 62 + .../reserves/iree/ci_linux_x64_clang.yml | 67 + .../reserves/iree/ci_linux_x64_clang_asan.yml | 78 + .../iree/ci_linux_x64_clang_byollvm.yml | 40 + .../iree/ci_linux_x64_clang_debug.yml | 61 + .../reserves/iree/ci_linux_x64_clang_tsan.yml | 57 + .../reserves/iree/ci_linux_x64_gcc.yml | 48 + .../reserves/iree/ci_macos_x64_clang.yml | 58 + .../reserves/iree/ci_windows_x64_msvc.yml | 52 + .github/workflows/reserves/iree/lint.yml | 23 + .../iree/oneshot_candidate_release.yml | 54 + .github/workflows/reserves/iree/pkgci.yml | 87 + .../reserves/iree/pkgci_build_packages.yml | 144 + .../reserves/iree/pkgci_regression_test.yml | 376 + .../reserves/iree/pkgci_test_amd_mi250.yml | 69 + .../reserves/iree/pkgci_test_amd_mi300.yml | 74 + .../reserves/iree/pkgci_test_amd_w7900.yml | 63 + .../reserves/iree/pkgci_test_android.yml | 91 + .../reserves/iree/pkgci_test_nvidia_t4.yml | 77 + .../iree/pkgci_test_tensorflow_cpu.yml | 49 + .../reserves/iree/pkgci_unit_test.yml | 62 + .../reserves/iree/post_benchmark_comment.yaml | 69 + .../reserves/iree/publish_website.yml | 77 + .github/workflows/reserves/iree/samples.yml | 90 + .../iree/schedule_candidate_release.yml | 78 + .github/workflows/reserves/iree/setup.yml | 116 + .../iree/validate_and_publish_release.yml | 137 + .../reserves/kernel/check-coverage.ps1 | 54 + .../reserves/kernel/close-inactive-issues.yml | 22 + .../reserves/kernel/codeql-analysis.yml | 79 + .../reserves/kernel/dotnet-build-and-test.yml | 176 + .../workflows/reserves/kernel/dotnet-ci.yml | 127 + .../reserves/kernel/dotnet-format.yml | 93 + .../kernel/dotnet-integration-tests.yml | 65 + .../kernel/generate-pr-description.yml | 61 + .../workflows/reserves/kernel/java-build.yml | 92 + .../kernel/java-integration-tests.yml | 92 + .../reserves/kernel/label-issues.yml | 54 + .../workflows/reserves/kernel/label-pr.yml | 21 + .../reserves/kernel/label-title-prefix.yml | 73 + .../kernel/markdown-link-check-config.json | 39 + .../reserves/kernel/markdown-link-check.yml | 23 + .../reserves/kernel/merge-gatekeeper.yml | 32 + .../reserves/kernel/python-build-wheel.yml | 45 + .../kernel/python-integration-tests.yml | 269 + .../workflows/reserves/kernel/python-lint.yml | 28 + .../reserves/kernel/python-test-coverage.yml | 65 + .../reserves/kernel/python-unit-tests.yml | 57 + .github/workflows/reserves/kernel/typos.yaml | 29 + .../reserves/kernel/update-version.sh | 107 + .github/workflows/reserves/licensed.yml | 15 + .../workflows/reserves/llvm-project/README.md | 1 + .../llvm-project/build-ci-container.yml | 112 + .../ci-post-commit-analyzer-run.py | 34 + .../llvm-project/ci-post-commit-analyzer.yml | 95 + .../reserves/llvm-project/clang-tests.yml | 38 + .../llvm-project/commit-access-review.py | 418 + .../llvm-project/commit-access-review.yml | 34 + .../github-action-ci/bootstrap.patch | 13 + .../github-action-ci/stage1.Dockerfile | 42 + .../github-action-ci/stage2.Dockerfile | 29 + .../containers/github-action-ci/storage.conf | 4 + .../workflows/reserves/llvm-project/docs.yml | 169 + .../reserves/llvm-project/email-check.yaml | 46 + .../llvm-project/get-llvm-version/action.yml | 26 + .../llvm-project/issue-release-workflow.yml | 69 + .../llvm-project/issue-subscriber.yml | 37 + .../reserves/llvm-project/issue-write.yml | 157 + .../llvm-project/libclang-abi-tests.yml | 168 + .../llvm-project/libclang-python-tests.yml | 41 + .../reserves/llvm-project/libclc-tests.yml | 39 + .../llvm-project/libcxx-build-and-test.yaml | 278 + .../libcxx-check-generated-files.yml | 24 + .../libcxx-restart-preempted-jobs.yaml | 132 + .../reserves/llvm-project/lld-tests.yml | 38 + .../reserves/llvm-project/lldb-tests.yml | 39 + .../reserves/llvm-project/llvm-bugs.yml | 63 + .../llvm-project/llvm-project-tests.yml | 148 + .../llvm-project-workflow-tests.yml | 32 + .../reserves/llvm-project/llvm-tests.yml | 192 + .../reserves/llvm-project/merged-prs.yml | 41 + .../reserves/llvm-project/new-issues.yml | 23 + .../reserves/llvm-project/new-prs.yml | 75 + .../reserves/llvm-project/pr-code-format.yml | 101 + .../llvm-project/pr-request-release-note.yml | 49 + .../reserves/llvm-project/pr-subscriber.yml | 34 + .../llvm-project/release-asset-audit.py | 66 + .../llvm-project/release-asset-audit.yml | 54 + .../llvm-project/release-binaries-all.yml | 98 + .../release-binaries-save-stage/action.yml | 44 + .../release-binaries-setup-stage/action.yml | 59 + .../llvm-project/release-binaries.yml | 481 ++ .../llvm-project/release-documentation.yml | 88 + .../reserves/llvm-project/release-doxygen.yml | 72 + .../reserves/llvm-project/release-lit.yml | 79 + .../reserves/llvm-project/release-sources.yml | 108 + .../reserves/llvm-project/release-tasks.yml | 123 + .../reserves/llvm-project/scorecard.yml | 62 + .../set-release-binary-outputs.sh | 34 + .../reserves/llvm-project/spirv-tests.yml | 29 + .../unprivileged-download-artifact/action.yml | 81 + .../reserves/llvm-project/version-check.py | 36 + .../reserves/llvm-project/version-check.yml | 31 + .github/workflows/reserves/main.yml | 44 + .../workflows/reserves/major-release-num.yml | 23 + .github/workflows/reserves/marty.yml | 140 + .github/workflows/reserves/matching.yml | 83 + .github/workflows/reserves/math/Benchmark.yml | 32 + .github/workflows/reserves/math/CPPBuild.yml | 94 + .github/workflows/reserves/math/CPPTest.yml | 99 + .../workflows/reserves/math/CSharpBuild.yml | 30 + .../workflows/reserves/math/CSharpTest.yml | 75 + .../reserves/math/EverythingBuild.yml | 39 + .../workflows/reserves/math/FSharpBuild.yml | 38 + .../workflows/reserves/math/FSharpTest.yml | 43 + .../reserves/math/IntegrationTest.yml | 116 + .../reserves/math/InteractiveBuild.yml | 38 + .../reserves/math/InteractiveTest.yml | 45 + .github/workflows/reserves/math/Nuget.yml | 57 + .../reserves/math/TerminalNightly.yml | 61 + .github/workflows/reserves/matrix-go.yml | 128 + .github/workflows/reserves/matrix-ruby.yml | 90 + .github/workflows/reserves/no-response.yml | 13 + .../reserves/project-euler/build.yml | 34 + .../project-euler/directory_writer.yml | 25 + .../reserves/project-euler/project_euler.yml | 39 + .../workflows/reserves/project-euler/ruff.yml | 16 + .../reserves/release-new-action-version.yml | 29 + .github/workflows/reserves/settings.yml | 31 + .../setup-dotnet/basic-validation.yml | 19 + .../reserves/setup-dotnet/check-dist.yml | 19 + .../reserves/setup-dotnet/codeql-analysis.yml | 14 + .../reserves/setup-dotnet/e2e-tests.yml | 483 ++ .../reserves/setup-dotnet/licensed.yml | 15 + .../release-new-action-version.yml | 29 + .../reserves/setup-dotnet/test-dotnet.yml | 35 + .../setup-dotnet/update-config-files.yml | 11 + .../reserves/setup-gcloud/dependabot.yml | 10 + .../reserves/setup-gcloud/draft-release.yml | 24 + .../reserves/setup-gcloud/integration.yml | 125 + .../reserves/setup-gcloud/release.yml | 13 + .../workflows/reserves/setup-gcloud/unit.yml | 49 + .../setup-wsl/check-action-typing.main.kts | 53 + .../setup-wsl/check-action-typing.yaml | 51 + .../check-dependency-versions.main.kts | 73 + .../setup-wsl/check-dependency-versions.yaml | 63 + .../reserves/setup-wsl/test.main.kts | 1084 +++ .../workflows/reserves/setup-wsl/test.yaml | 1738 ++++ .../validate-gradle-wrapper.main.kts | 53 + .../setup-wsl/validate-gradle-wrapper.yaml | 51 + .../workflow-with-copyright.main.kts | 59 + .github/workflows/reserves/stale.yml | 18 + .../workflows/reserves/tensorflow-io/api.yml | 150 + .../reserves/tensorflow-io/benchmarks.yml | 96 + .../reserves/tensorflow-io/build.arm64.yml | 145 + .../reserves/tensorflow-io/build.bazel.sh | 101 + .../reserves/tensorflow-io/build.gpu.sh | 82 + .../tensorflow-io/build.instruction.py | 24 + .../reserves/tensorflow-io/build.space.sh | 9 + .../reserves/tensorflow-io/build.wheel.sh | 54 + .../reserves/tensorflow-io/build.yml | 873 ++ .../reserves/tensorflow-io/release.note.yml | 34 + .../reserves/tensorflow-io/release.yml | 82 + .../reserves/tensorflow-io/reviewdog.yml | 59 + .../reserves/tensorflow-io/scorecard.yml | 64 + .github/workflows/reserves/test-dotnet.yml | 35 + .../reserves/test_of_push_and_pull.yml | 37 + .../reserves/test_of_validate_package.yml | 29 + .../reserves/ubuntu-wsl/build-pr.yaml | 64 + .../reserves/ubuntu-wsl/build-wsl.yaml | 359 + .../ubuntu-wsl/detect-update-releases.yaml | 62 + .../workflows/reserves/ubuntu-wsl/e2e.yaml | 112 + .../reserves/ubuntu-wsl/sync-gh-jira.yaml | 12 + .../reserves/ubuntu-wsl/wsl-example.yaml | 75 + .../reserves/update-config-files.yml | 11 + .github/workflows/reserves/update.yml | 34 + .github/workflows/reserves/validate.yml | 35 + .github/workflows/reserves/validate_links.yml | 28 + .../workflows-call/component-build.yml | 88 + .../reserves/workflows-call/component-ci.yml | 92 + .../workflows-call/component-ci_no-docker.yml | 46 + .../workflows-call/component-deploy.yml | 70 + .../workflows-call/component-package.yml | 30 + .github/workflows/retry.yml | 18 + 371 files changed, 40907 insertions(+), 274 deletions(-) create mode 100644 .github/FUNDING.yml delete mode 100644 .github/ISSUE_TEMPLATE/bug_report.yml delete mode 100644 .github/ISSUE_TEMPLATE/config.yml delete mode 100644 .github/ISSUE_TEMPLATE/feature_request.yml delete mode 100644 .github/ISSUE_TEMPLATE/other.yml create mode 100644 .github/actions/1-addition/1-maps/action.yml create mode 100644 .github/actions/1-addition/2-feed/action.yml create mode 100644 .github/actions/1-addition/3-lexer/action.yml create mode 100644 .github/actions/2-multiplication/1-maps/action.yml create mode 100644 .github/actions/2-multiplication/2-feed/action.yml create mode 100644 .github/actions/2-multiplication/3-lexer/action.yml create mode 100644 .github/actions/3-exponentiation/4-parser/action.yml create mode 100644 .github/actions/3-exponentiation/5-syntax/action.yml create mode 100644 .github/actions/3-exponentiation/6-grammar/action.yml create mode 100644 .github/actions/4-identition/4-parser/action.yml create mode 100644 .github/actions/4-identition/5-syntax/action.yml create mode 100644 .github/actions/4-identition/6-grammar/action.yml create mode 100644 .github/actions/reserves/ubuntu/README.md create mode 100644 .github/actions/reserves/ubuntu/download-rootfs/action.yaml create mode 100644 .github/actions/reserves/ubuntu/download-rootfs/download-rootfs.ps1 create mode 100644 .github/actions/reserves/ubuntu/wsl-bash/action.yaml create mode 100644 .github/actions/reserves/ubuntu/wsl-checkout/action.yaml create mode 100644 .github/actions/reserves/ubuntu/wsl-install/action.yaml create mode 100644 .github/csc.json create mode 100644 .github/dco.yml create mode 100755 .github/entrypoint/artifact.sh create mode 100644 .github/entrypoint/artifact/BosonJet.c create mode 100644 .github/entrypoint/artifact/BosonJet.h create mode 100644 .github/entrypoint/artifact/BosonJetBorn.c create mode 100644 .github/entrypoint/artifact/BosonJetBorn.h create mode 100644 .github/entrypoint/artifact/BosonJetCSS.c create mode 100644 .github/entrypoint/artifact/BosonJetCSS.h create mode 100644 .github/entrypoint/artifact/BosonJetLL.c create mode 100644 .github/entrypoint/artifact/BosonJetLL.h create mode 100644 .github/entrypoint/artifact/BosonJetLO.c create mode 100644 .github/entrypoint/artifact/BosonJetLO.h create mode 100644 .github/entrypoint/artifact/BosonJetNLL.c create mode 100644 .github/entrypoint/artifact/BosonJetNLL.h create mode 100644 .github/entrypoint/artifact/BosonJetResum.c create mode 100644 .github/entrypoint/artifact/BosonJetResum.h create mode 100644 .github/entrypoint/artifact/ElectroWeak.c create mode 100644 .github/entrypoint/artifact/ElectroWeak.h create mode 100644 .github/entrypoint/artifact/Makefile create mode 100644 .github/entrypoint/artifact/Particle.c create mode 100644 .github/entrypoint/artifact/Particle.h create mode 100644 .github/entrypoint/artifact/QCD.c create mode 100644 .github/entrypoint/artifact/QCD.h create mode 100644 .github/entrypoint/artifact/README.md create mode 100644 .github/entrypoint/artifact/StandardModel.h create mode 100644 .github/entrypoint/artifact/born.c create mode 100644 .github/entrypoint/artifact/css.c create mode 100644 .github/entrypoint/artifact/main.c create mode 100644 .github/entrypoint/artifact/nllR.c create mode 100644 .github/entrypoint/artifact/python/LICENSE create mode 100644 .github/entrypoint/artifact/python/README.md create mode 100644 .github/entrypoint/artifact/python/colour_matrices.py create mode 100644 .github/entrypoint/artifact/python/gellmann.ipynb create mode 100644 .github/entrypoint/artifact/python/gellmann.py create mode 100644 .github/entrypoint/artifact/python/inventory.py create mode 100644 .github/entrypoint/artifact/python/keywords.lst create mode 100644 .github/entrypoint/artifact/python/particle.ipynb create mode 100644 .github/entrypoint/artifact/python/pyproject.toml create mode 100644 .github/entrypoint/artifact/python/setup.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/group/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/group/generators.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/group/group.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/lattice.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/su3/pytorch/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/lattice.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/lattice.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/lattice.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/__init__.py create mode 100644 .github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/lattice.py create mode 100644 .github/entrypoint/artifact/python/src/params/spaces.json create mode 100644 .github/entrypoint/artifact/resum.c create mode 100644 .github/entrypoint/artifact/resumLL.c create mode 100644 .github/entrypoint/artifact/test_qcd.c create mode 100644 .github/entrypoint/artifact/tot.c create mode 100644 .github/entrypoint/cleanup_bundler.rb create mode 100755 .github/entrypoint/config.sh create mode 100755 .github/entrypoint/entrypoint_.sh create mode 100755 .github/entrypoint/front.sh create mode 100644 .github/entrypoint/gcp_access.py create mode 100644 .github/entrypoint/gcp_token.py create mode 100755 .github/entrypoint/gist.sh create mode 100755 .github/entrypoint/github_pages.sh create mode 100755 .github/entrypoint/idle.sh create mode 100755 .github/entrypoint/init.sh create mode 100755 .github/entrypoint/pinned_repos.rb create mode 100644 .github/entrypoint/prime_list.py create mode 100755 .github/entrypoint/remote.sh create mode 100755 .github/entrypoint/restore.sh create mode 100755 .github/entrypoint/sort.sh create mode 100644 .github/entrypoint/startup.sh create mode 100644 .github/entrypoint/test.py create mode 100755 .github/entrypoint/wiki.sh delete mode 100644 .github/pull_request_template.md delete mode 100644 .github/stale.yml create mode 100644 .github/templates/CODE_OF_CONDUCT.md create mode 100644 .github/templates/CONTRIBUTING.md create mode 100644 .github/templates/PULL_REQUEST_TEMPLATE.md create mode 100644 .github/templates/bug_report.md create mode 100644 .github/templates/bug_report.yml create mode 100644 .github/templates/config.yml create mode 100644 .github/templates/feature_request.md create mode 100644 .github/templates/feature_request.yml create mode 100644 .github/templates/mapping-errors.md create mode 100644 .github/templates/other-issues.md create mode 100644 .github/templates/standards-change-proposal.md create mode 100644 .github/workflows/main.yml create mode 100644 .github/workflows/reserves/__config.yml create mode 100644 .github/workflows/reserves/account/blog-posts.yml create mode 100644 .github/workflows/reserves/account/star-fork.yml create mode 100644 .github/workflows/reserves/account/starred.yml create mode 100644 .github/workflows/reserves/account/topFollowers.yml create mode 100644 .github/workflows/reserves/account/wakatime.yml rename .github/workflows/{ => reserves/algorithms}/build.yml (96%) rename .github/workflows/{ => reserves/algorithms}/directory_writer.yml (100%) rename .github/workflows/{ => reserves/algorithms}/project_euler.yml (93%) rename .github/workflows/{ => reserves/algorithms}/ruff.yml (87%) rename .github/workflows/{ => reserves/algorithms}/sphinx.yml (95%) create mode 100644 .github/workflows/reserves/basic-validation.yml create mode 100644 .github/workflows/reserves/build.yml create mode 100644 .github/workflows/reserves/check-dist.yml create mode 100644 .github/workflows/reserves/ci.yml create mode 100644 .github/workflows/reserves/cloud-init/alpine-unittests.yml create mode 100644 .github/workflows/reserves/cloud-init/check_format.yml create mode 100644 .github/workflows/reserves/cloud-init/cla.yml create mode 100644 .github/workflows/reserves/cloud-init/doc-autolabel.yml create mode 100644 .github/workflows/reserves/cloud-init/integration.yml create mode 100644 .github/workflows/reserves/cloud-init/linkcheck.yml create mode 100644 .github/workflows/reserves/cloud-init/scheduled.yml create mode 100644 .github/workflows/reserves/cloud-init/stale.yml create mode 100644 .github/workflows/reserves/cloud-init/unit.yml create mode 100644 .github/workflows/reserves/codeql-analysis.yml create mode 100644 .github/workflows/reserves/codeql.yml create mode 100644 .github/workflows/reserves/config.yml create mode 100644 .github/workflows/reserves/conformance.yml create mode 100644 .github/workflows/reserves/dependabot.yml create mode 100644 .github/workflows/reserves/devcontainer-cli/build-chat.yml create mode 100644 .github/workflows/reserves/devcontainer-cli/dev-containers.yml create mode 100644 .github/workflows/reserves/devcontainer-cli/publish-dev-containers.yml create mode 100644 .github/workflows/reserves/devcontainer-cli/test-plan-item-validator.yml create mode 100644 .github/workflows/reserves/devcontainer-cli/test-windows.yml create mode 100644 .github/workflows/reserves/devcontainer/ci_branch.yml create mode 100644 .github/workflows/reserves/devcontainer/ci_common.yml create mode 100644 .github/workflows/reserves/devcontainer/ci_main.yml create mode 100755 .github/workflows/reserves/devcontainer/clean_tags.sh create mode 100755 .github/workflows/reserves/devcontainer/clean_untagged.sh create mode 100644 .github/workflows/reserves/devcontainer/codeql-analysis.yml create mode 100644 .github/workflows/reserves/devcontainer/pr-bot.yml create mode 100644 .github/workflows/reserves/devcontainer/pr-closed.yml create mode 100644 .github/workflows/reserves/devcontainer/pr_auto.yml create mode 100644 .github/workflows/reserves/devcontainer/untagged-image-cleanup.yml create mode 100644 .github/workflows/reserves/distribution.yml create mode 100644 .github/workflows/reserves/e2e-tests.yml create mode 100644 .github/workflows/reserves/e2e.yml create mode 100644 .github/workflows/reserves/elixir-ci.yml create mode 100644 .github/workflows/reserves/fossa.yml create mode 100644 .github/workflows/reserves/github-script/check-dist.yml create mode 100644 .github/workflows/reserves/github-script/ci.yml create mode 100644 .github/workflows/reserves/github-script/codeql-analysis.yml create mode 100644 .github/workflows/reserves/github-script/integration.yml create mode 100644 .github/workflows/reserves/github-script/licensed.yml create mode 100644 .github/workflows/reserves/github-script/pull-request-test.yml create mode 100644 .github/workflows/reserves/github-script/stale.yml create mode 100644 .github/workflows/reserves/iree/ARTIFACT_SUMMARY_TEMPLATE.md create mode 100644 .github/workflows/reserves/iree/benchmark.yml create mode 100644 .github/workflows/reserves/iree/benchmark_compilation.yml create mode 100644 .github/workflows/reserves/iree/benchmark_execution.yml create mode 100644 .github/workflows/reserves/iree/benchmark_large.yml create mode 100644 .github/workflows/reserves/iree/benchmark_trigger.yml create mode 100644 .github/workflows/reserves/iree/build_all.yml create mode 100644 .github/workflows/reserves/iree/build_benchmark_tools.yml create mode 100644 .github/workflows/reserves/iree/build_e2e_test_artifacts.yml create mode 100644 .github/workflows/reserves/iree/build_package.yml create mode 100644 .github/workflows/reserves/iree/bump_torch_mlir.yml create mode 100644 .github/workflows/reserves/iree/ci.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_arm64_clang.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_clang.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_clang_asan.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_clang_byollvm.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_clang_debug.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_clang_tsan.yml create mode 100644 .github/workflows/reserves/iree/ci_linux_x64_gcc.yml create mode 100644 .github/workflows/reserves/iree/ci_macos_x64_clang.yml create mode 100644 .github/workflows/reserves/iree/ci_windows_x64_msvc.yml create mode 100644 .github/workflows/reserves/iree/lint.yml create mode 100644 .github/workflows/reserves/iree/oneshot_candidate_release.yml create mode 100644 .github/workflows/reserves/iree/pkgci.yml create mode 100644 .github/workflows/reserves/iree/pkgci_build_packages.yml create mode 100644 .github/workflows/reserves/iree/pkgci_regression_test.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_amd_mi250.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_amd_mi300.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_amd_w7900.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_android.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_nvidia_t4.yml create mode 100644 .github/workflows/reserves/iree/pkgci_test_tensorflow_cpu.yml create mode 100644 .github/workflows/reserves/iree/pkgci_unit_test.yml create mode 100644 .github/workflows/reserves/iree/post_benchmark_comment.yaml create mode 100644 .github/workflows/reserves/iree/publish_website.yml create mode 100644 .github/workflows/reserves/iree/samples.yml create mode 100644 .github/workflows/reserves/iree/schedule_candidate_release.yml create mode 100644 .github/workflows/reserves/iree/setup.yml create mode 100644 .github/workflows/reserves/iree/validate_and_publish_release.yml create mode 100644 .github/workflows/reserves/kernel/check-coverage.ps1 create mode 100644 .github/workflows/reserves/kernel/close-inactive-issues.yml create mode 100644 .github/workflows/reserves/kernel/codeql-analysis.yml create mode 100644 .github/workflows/reserves/kernel/dotnet-build-and-test.yml create mode 100644 .github/workflows/reserves/kernel/dotnet-ci.yml create mode 100644 .github/workflows/reserves/kernel/dotnet-format.yml create mode 100644 .github/workflows/reserves/kernel/dotnet-integration-tests.yml create mode 100644 .github/workflows/reserves/kernel/generate-pr-description.yml create mode 100644 .github/workflows/reserves/kernel/java-build.yml create mode 100644 .github/workflows/reserves/kernel/java-integration-tests.yml create mode 100644 .github/workflows/reserves/kernel/label-issues.yml create mode 100644 .github/workflows/reserves/kernel/label-pr.yml create mode 100644 .github/workflows/reserves/kernel/label-title-prefix.yml create mode 100644 .github/workflows/reserves/kernel/markdown-link-check-config.json create mode 100644 .github/workflows/reserves/kernel/markdown-link-check.yml create mode 100644 .github/workflows/reserves/kernel/merge-gatekeeper.yml create mode 100644 .github/workflows/reserves/kernel/python-build-wheel.yml create mode 100644 .github/workflows/reserves/kernel/python-integration-tests.yml create mode 100644 .github/workflows/reserves/kernel/python-lint.yml create mode 100644 .github/workflows/reserves/kernel/python-test-coverage.yml create mode 100644 .github/workflows/reserves/kernel/python-unit-tests.yml create mode 100644 .github/workflows/reserves/kernel/typos.yaml create mode 100755 .github/workflows/reserves/kernel/update-version.sh create mode 100644 .github/workflows/reserves/licensed.yml create mode 100644 .github/workflows/reserves/llvm-project/README.md create mode 100644 .github/workflows/reserves/llvm-project/build-ci-container.yml create mode 100644 .github/workflows/reserves/llvm-project/ci-post-commit-analyzer-run.py create mode 100644 .github/workflows/reserves/llvm-project/ci-post-commit-analyzer.yml create mode 100644 .github/workflows/reserves/llvm-project/clang-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/commit-access-review.py create mode 100644 .github/workflows/reserves/llvm-project/commit-access-review.yml create mode 100644 .github/workflows/reserves/llvm-project/containers/github-action-ci/bootstrap.patch create mode 100644 .github/workflows/reserves/llvm-project/containers/github-action-ci/stage1.Dockerfile create mode 100644 .github/workflows/reserves/llvm-project/containers/github-action-ci/stage2.Dockerfile create mode 100644 .github/workflows/reserves/llvm-project/containers/github-action-ci/storage.conf create mode 100644 .github/workflows/reserves/llvm-project/docs.yml create mode 100644 .github/workflows/reserves/llvm-project/email-check.yaml create mode 100644 .github/workflows/reserves/llvm-project/get-llvm-version/action.yml create mode 100644 .github/workflows/reserves/llvm-project/issue-release-workflow.yml create mode 100644 .github/workflows/reserves/llvm-project/issue-subscriber.yml create mode 100644 .github/workflows/reserves/llvm-project/issue-write.yml create mode 100644 .github/workflows/reserves/llvm-project/libclang-abi-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/libclang-python-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/libclc-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/libcxx-build-and-test.yaml create mode 100644 .github/workflows/reserves/llvm-project/libcxx-check-generated-files.yml create mode 100644 .github/workflows/reserves/llvm-project/libcxx-restart-preempted-jobs.yaml create mode 100644 .github/workflows/reserves/llvm-project/lld-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/lldb-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/llvm-bugs.yml create mode 100644 .github/workflows/reserves/llvm-project/llvm-project-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/llvm-project-workflow-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/llvm-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/merged-prs.yml create mode 100644 .github/workflows/reserves/llvm-project/new-issues.yml create mode 100644 .github/workflows/reserves/llvm-project/new-prs.yml create mode 100644 .github/workflows/reserves/llvm-project/pr-code-format.yml create mode 100644 .github/workflows/reserves/llvm-project/pr-request-release-note.yml create mode 100644 .github/workflows/reserves/llvm-project/pr-subscriber.yml create mode 100644 .github/workflows/reserves/llvm-project/release-asset-audit.py create mode 100644 .github/workflows/reserves/llvm-project/release-asset-audit.yml create mode 100644 .github/workflows/reserves/llvm-project/release-binaries-all.yml create mode 100644 .github/workflows/reserves/llvm-project/release-binaries-save-stage/action.yml create mode 100644 .github/workflows/reserves/llvm-project/release-binaries-setup-stage/action.yml create mode 100644 .github/workflows/reserves/llvm-project/release-binaries.yml create mode 100644 .github/workflows/reserves/llvm-project/release-documentation.yml create mode 100644 .github/workflows/reserves/llvm-project/release-doxygen.yml create mode 100644 .github/workflows/reserves/llvm-project/release-lit.yml create mode 100644 .github/workflows/reserves/llvm-project/release-sources.yml create mode 100644 .github/workflows/reserves/llvm-project/release-tasks.yml create mode 100644 .github/workflows/reserves/llvm-project/scorecard.yml create mode 100644 .github/workflows/reserves/llvm-project/set-release-binary-outputs.sh create mode 100644 .github/workflows/reserves/llvm-project/spirv-tests.yml create mode 100644 .github/workflows/reserves/llvm-project/unprivileged-download-artifact/action.yml create mode 100755 .github/workflows/reserves/llvm-project/version-check.py create mode 100644 .github/workflows/reserves/llvm-project/version-check.yml create mode 100644 .github/workflows/reserves/main.yml create mode 100644 .github/workflows/reserves/major-release-num.yml create mode 100644 .github/workflows/reserves/marty.yml create mode 100644 .github/workflows/reserves/matching.yml create mode 100644 .github/workflows/reserves/math/Benchmark.yml create mode 100644 .github/workflows/reserves/math/CPPBuild.yml create mode 100644 .github/workflows/reserves/math/CPPTest.yml create mode 100644 .github/workflows/reserves/math/CSharpBuild.yml create mode 100644 .github/workflows/reserves/math/CSharpTest.yml create mode 100644 .github/workflows/reserves/math/EverythingBuild.yml create mode 100644 .github/workflows/reserves/math/FSharpBuild.yml create mode 100644 .github/workflows/reserves/math/FSharpTest.yml create mode 100644 .github/workflows/reserves/math/IntegrationTest.yml create mode 100644 .github/workflows/reserves/math/InteractiveBuild.yml create mode 100644 .github/workflows/reserves/math/InteractiveTest.yml create mode 100644 .github/workflows/reserves/math/Nuget.yml create mode 100644 .github/workflows/reserves/math/TerminalNightly.yml create mode 100644 .github/workflows/reserves/matrix-go.yml create mode 100644 .github/workflows/reserves/matrix-ruby.yml create mode 100644 .github/workflows/reserves/no-response.yml create mode 100644 .github/workflows/reserves/project-euler/build.yml create mode 100644 .github/workflows/reserves/project-euler/directory_writer.yml create mode 100644 .github/workflows/reserves/project-euler/project_euler.yml create mode 100644 .github/workflows/reserves/project-euler/ruff.yml create mode 100644 .github/workflows/reserves/release-new-action-version.yml create mode 100644 .github/workflows/reserves/settings.yml create mode 100644 .github/workflows/reserves/setup-dotnet/basic-validation.yml create mode 100644 .github/workflows/reserves/setup-dotnet/check-dist.yml create mode 100644 .github/workflows/reserves/setup-dotnet/codeql-analysis.yml create mode 100644 .github/workflows/reserves/setup-dotnet/e2e-tests.yml create mode 100644 .github/workflows/reserves/setup-dotnet/licensed.yml create mode 100644 .github/workflows/reserves/setup-dotnet/release-new-action-version.yml create mode 100644 .github/workflows/reserves/setup-dotnet/test-dotnet.yml create mode 100644 .github/workflows/reserves/setup-dotnet/update-config-files.yml create mode 100644 .github/workflows/reserves/setup-gcloud/dependabot.yml create mode 100644 .github/workflows/reserves/setup-gcloud/draft-release.yml create mode 100644 .github/workflows/reserves/setup-gcloud/integration.yml create mode 100644 .github/workflows/reserves/setup-gcloud/release.yml create mode 100644 .github/workflows/reserves/setup-gcloud/unit.yml create mode 100755 .github/workflows/reserves/setup-wsl/check-action-typing.main.kts create mode 100644 .github/workflows/reserves/setup-wsl/check-action-typing.yaml create mode 100755 .github/workflows/reserves/setup-wsl/check-dependency-versions.main.kts create mode 100644 .github/workflows/reserves/setup-wsl/check-dependency-versions.yaml create mode 100755 .github/workflows/reserves/setup-wsl/test.main.kts create mode 100644 .github/workflows/reserves/setup-wsl/test.yaml create mode 100755 .github/workflows/reserves/setup-wsl/validate-gradle-wrapper.main.kts create mode 100644 .github/workflows/reserves/setup-wsl/validate-gradle-wrapper.yaml create mode 100644 .github/workflows/reserves/setup-wsl/workflow-with-copyright.main.kts create mode 100644 .github/workflows/reserves/stale.yml create mode 100644 .github/workflows/reserves/tensorflow-io/api.yml create mode 100644 .github/workflows/reserves/tensorflow-io/benchmarks.yml create mode 100644 .github/workflows/reserves/tensorflow-io/build.arm64.yml create mode 100755 .github/workflows/reserves/tensorflow-io/build.bazel.sh create mode 100644 .github/workflows/reserves/tensorflow-io/build.gpu.sh create mode 100644 .github/workflows/reserves/tensorflow-io/build.instruction.py create mode 100644 .github/workflows/reserves/tensorflow-io/build.space.sh create mode 100755 .github/workflows/reserves/tensorflow-io/build.wheel.sh create mode 100644 .github/workflows/reserves/tensorflow-io/build.yml create mode 100644 .github/workflows/reserves/tensorflow-io/release.note.yml create mode 100644 .github/workflows/reserves/tensorflow-io/release.yml create mode 100644 .github/workflows/reserves/tensorflow-io/reviewdog.yml create mode 100644 .github/workflows/reserves/tensorflow-io/scorecard.yml create mode 100644 .github/workflows/reserves/test-dotnet.yml create mode 100644 .github/workflows/reserves/test_of_push_and_pull.yml create mode 100644 .github/workflows/reserves/test_of_validate_package.yml create mode 100644 .github/workflows/reserves/ubuntu-wsl/build-pr.yaml create mode 100644 .github/workflows/reserves/ubuntu-wsl/build-wsl.yaml create mode 100644 .github/workflows/reserves/ubuntu-wsl/detect-update-releases.yaml create mode 100644 .github/workflows/reserves/ubuntu-wsl/e2e.yaml create mode 100644 .github/workflows/reserves/ubuntu-wsl/sync-gh-jira.yaml create mode 100644 .github/workflows/reserves/ubuntu-wsl/wsl-example.yaml create mode 100644 .github/workflows/reserves/update-config-files.yml create mode 100644 .github/workflows/reserves/update.yml create mode 100644 .github/workflows/reserves/validate.yml create mode 100644 .github/workflows/reserves/validate_links.yml create mode 100644 .github/workflows/reserves/workflows-call/component-build.yml create mode 100644 .github/workflows/reserves/workflows-call/component-ci.yml create mode 100644 .github/workflows/reserves/workflows-call/component-ci_no-docker.yml create mode 100644 .github/workflows/reserves/workflows-call/component-deploy.yml create mode 100644 .github/workflows/reserves/workflows-call/component-package.yml create mode 100644 .github/workflows/retry.yml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 3cc25d1bae1c..f82824ad56f9 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,86 +1,13 @@ -# This is a comment. +# Lines starting with '#' are comments. # Each line is a file pattern followed by one or more owners. -# More details are here: https://help.github.com/articles/about-codeowners/ - -# The '*' pattern is global owners. +# These owners will be the default owners for everything in the repo. +* @eq19 # Order is important. The last matching pattern has the most precedence. +# So if a pull request only touches javascript files, only these owners +# will be requested to review. +# *.js @octocat @github/js -/.* @cclauss - -# /backtracking/ - -# /bit_manipulation/ - -# /blockchain/ - -# /boolean_algebra/ - -# /cellular_automata/ - -# /ciphers/ - -# /compression/ - -# /computer_vision/ - -# /conversions/ - -# /data_structures/ - -# /digital_image_processing/ - -# /divide_and_conquer/ - -# /dynamic_programming/ - -# /file_transfer/ - -# /fuzzy_logic/ - -# /genetic_algorithm/ - -# /geodesy/ - -# /graphics/ - -# /graphs/ - -# /greedy_method/ - -# /hashes/ - -# /images/ - -# /linear_algebra/ - -# /machine_learning/ - -# /maths/ - -# /matrix/ - -# /networking_flow/ - -# /neural_network/ - -# /other/ - -# /project_euler/ - -# /quantum/ - -# /scheduling/ - -# /scripts/ - -# /searches/ - -# /sorts/ - -# /strings/ - -# /traversals/ - -/web_programming/ @cclauss +# You can also use email addresses if you prefer. +# .github/* docs@example.com diff --git a/.github/FUNDING.yml b/.github/FUNDING.yml new file mode 100644 index 000000000000..79693d22b3ac --- /dev/null +++ b/.github/FUNDING.yml @@ -0,0 +1 @@ +custom: ["https://git.io/JJAW8"] diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml deleted file mode 100644 index 4ccdb52cad24..000000000000 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ /dev/null @@ -1,54 +0,0 @@ -name: Bug report -description: Create a bug report to help us address errors in the repository -labels: [bug] -body: - - type: markdown - attributes: - value: > - Before requesting please search [existing issues](https://github.com/TheAlgorithms/Python/labels/bug). - Usage questions such as "How do I...?" belong on the - [Discord](https://discord.gg/c7MnfGFGa6) and will be closed. - - - type: input - attributes: - label: "Repository commit" - description: > - The commit hash for `TheAlgorithms/Python` repository. You can get this - by running the command `git rev-parse HEAD` locally. - placeholder: "a0b0f414ae134aa1772d33bb930e5a960f9979e8" - validations: - required: true - - - type: input - attributes: - label: "Python version (python --version)" - placeholder: "Python 3.10.7" - validations: - required: true - - - type: textarea - attributes: - label: "Dependencies version (pip freeze)" - description: > - This is the output of the command `pip freeze --all`. Note that the - actual output might be different as compared to the placeholder text. - placeholder: | - appnope==0.1.3 - asttokens==2.0.8 - backcall==0.2.0 - ... - validations: - required: true - - - type: textarea - attributes: - label: "Expected behavior" - description: "Describe the behavior you expect. May include images or videos." - validations: - required: true - - - type: textarea - attributes: - label: "Actual behavior" - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml deleted file mode 100644 index 62019bb08938..000000000000 --- a/.github/ISSUE_TEMPLATE/config.yml +++ /dev/null @@ -1,5 +0,0 @@ -blank_issues_enabled: false -contact_links: - - name: Discord community - url: https://discord.gg/c7MnfGFGa6 - about: Have any questions or need any help? Please contact us via Discord diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml deleted file mode 100644 index 20823bd58ab1..000000000000 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ /dev/null @@ -1,19 +0,0 @@ -name: Feature request -description: Suggest features, propose improvements, discuss new ideas. -labels: [enhancement] -body: - - type: markdown - attributes: - value: > - Before requesting please search [existing issues](https://github.com/TheAlgorithms/Python/labels/enhancement). - Do not create issues to implement new algorithms as these will be closed. - Usage questions such as "How do I...?" belong on the - [Discord](https://discord.gg/c7MnfGFGa6) and will be closed. - - - type: textarea - attributes: - label: "Feature description" - description: > - This could include new topics or improving any existing implementations. - validations: - required: true diff --git a/.github/ISSUE_TEMPLATE/other.yml b/.github/ISSUE_TEMPLATE/other.yml deleted file mode 100644 index 44d6ff541506..000000000000 --- a/.github/ISSUE_TEMPLATE/other.yml +++ /dev/null @@ -1,19 +0,0 @@ -name: Other -description: Use this for any other issues. PLEASE do not create blank issues -labels: ["awaiting triage"] -body: - - type: textarea - id: issuedescription - attributes: - label: What would you like to share? - description: Provide a clear and concise explanation of your issue. - validations: - required: true - - - type: textarea - id: extrainfo - attributes: - label: Additional information - description: Is there anything else we should know about this issue? - validations: - required: false diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml new file mode 100644 index 000000000000..4c1dcbf117be --- /dev/null +++ b/.github/actions/1-addition/1-maps/action.yml @@ -0,0 +1,158 @@ +name: 'Docker Deploy Action' + +description: > + 🪂 A Github Action to deploy pages conveniently + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + initiate_pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + +outputs: + artifact: + description: Collect artifact outputs + value: ${{ toJSON(steps.artifact.outputs) }} + variables: + description: Collect variables outputs + value: ${{ toJSON(steps.variables.outputs) }} + +runs: + using: composite + steps: + - name: Remove Existing Connection + shell: bash + env: + JOB_ID: ${{ inputs.id }} + run: | + HEADER="Accept: application/vnd.github+json" + /home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh ${{ inputs.action_path }} + + #See: https://docs.github.com/en/rest/actions/self-hosted-runners + TOTAL_COUNT=$(gh api -H "${HEADER}" /repos/${{ github.repository }}/actions/runners --jq '.total_count') + RUNNER_ID=$(gh api -H "${HEADER}" /repos/${{ github.repository }}/actions/runners --jq '.runners.[].id') + if (( $TOTAL_COUNT != 0 )); then gh api --method DELETE -H "${HEADER}" /repos/${{ github.repository }}/actions/runners/${RUNNER_ID}; fi + + - name: View context attributes + uses: actions/github-script@v7 + env: + JOB_ID: ${{ inputs.id }} + with: + script: console.log(context) + + - name: Login to DockerHub + uses: docker/login-action@v3 + with: + username: ${{ inputs.hub_username }} + password: ${{ inputs.hub_token }} + + - uses: actions/setup-python@v5 + with: + python-version: 3.12 + allow-prereleases: true + + - uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip + + - name: Install dependencies + shell: bash + run: | + python -m pip -q install --upgrade pip + python -m pip -q install -r ${{ inputs.action_path }}/dataFile/requirements.txt + + ##python -m pip -q install -e ./.github/entrypoint/artifact/python + ##iree-compile --iree-llvmcpu-list-targets && iree-run-module --list_drivers + + ##Ref: https://github.com/iree-org/iree-experimental + ##python ${{ inputs.action_path }}/pythonCode/testIREE.py + + ##Ref: https://github.com/tsoding/JelloVM + cd ${{ inputs.action_path }} && javac javaCode/Main.java + python app.py output.txt && cat dataFile/results/output.txt + + ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode + ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py + ##cd ${{ inputs.action_path }}/pythonCode && python npreader.py -h && python map.py -h + + - name: 🚀 Initiate Lexer + uses: devcontainers/ci@v0.3 + with: + push: always + skipContainerUserIdUpdate: true + imageName: ${{ inputs.image_name }} + imageTag: target-${{ github.run_number }} + configFile: ${{ inputs.action_path }}/.devcontainer.json + runCmd: ls -al / + + - name: 💎 Variables + id: variables + shell: bash + run: | + BASE_NAME=$(basename ${{ inputs.image_name }}) + IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) + for ((i=0; i < ${#array[@]}; i++)); do + if [ "${array[$i]}" != "latest" ]; then + HUB_TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d "{\"username\": \"${{ inputs.hub_username }}\", \"password\": \"${{ inputs.hub_password }}\"}" https://hub.docker.com/v2/users/login/ | jq -r .token) + curl -s -i -X DELETE -H "Accept: application/json" -H "Authorization: JWT $HUB_TOKEN" https://hub.docker.com/v2/namespaces/${{ inputs.hub_username }}/repositories/$BASE_NAME/tags/${array[$i]} + echo "deleted: ${{ inputs.image_name }}:${array[$i]}" + fi + done diff --git a/.github/actions/1-addition/2-feed/action.yml b/.github/actions/1-addition/2-feed/action.yml new file mode 100644 index 000000000000..da588b6b0863 --- /dev/null +++ b/.github/actions/1-addition/2-feed/action.yml @@ -0,0 +1,182 @@ +name: 'Docker Deploy Action' + +description: > + 🪂 A Github Action to deploy pages conveniently + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + default: ${{ github.action_repository }} + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + initiate_pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + rerun_runner: + description: Redeploy hosted runner + default: 'false' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + +outputs: + artifact: + description: Collect artifact outputs + value: ${{ toJSON(steps.artifact.outputs.repo) }} + set_output: + description: Collect set_output outputs + value: ${{ toJSON(steps.set_output.outputs) }} + +runs: + using: composite + steps: + - name: 💎 Set ID Token + id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + token_format: 'id_token' + create_credentials_file: false + credentials_json: ${{ inputs.credentials }} + id_token_audience: 'https://us-central1-feedmapping.cloudfunctions.net/function' + + - name: 💎 Set Artifact + id: artifact + shell: bash + env: + TMP: ${{ inputs.tmp }} + RUN: ${{ github.run_id }} + USER: ${{ github.actor }} + #TOKEN: ${{ inputs.token }} + REPO: ${{ github.repository }} + ACTOR: ${{ github.triggering_actor }} + OWNER: ${{ github.repository_owner }} + BEARER: ${{ steps.auth.outputs.id_token }} + run: | + shopt -s dotglob + rm -rf ${{ inputs.action_path }}/.git + + mv -f ${{ inputs.action_path }}/_config.yml ${RUNNER_TEMP}/ + export PATH=$GITHUB_WORKSPACE/.github/entrypoint:$PATH && artifact.sh + + mv -f ${RUNNER_TEMP}/.env ${{ inputs.action_path }}/ + mv -f ${RUNNER_TEMP}/_config.yml ${{ inputs.action_path }}/ + + mkdir -p ${{ inputs.action_path }}/_data + mv -f ${RUNNER_TEMP}/*.json ${{ inputs.action_path }}/_data/ + + mv -f ${{ inputs.action_path }} ${{ github.workspace }}/_site + mkdir -p ${{ inputs.action_path }} && cp -R ${{ github.workspace }}/_site/. ${{ inputs.action_path }}/ + + - uses: ikalnytskyi/action-setup-postgres@v7 + id: postgres + with: + username: yoda + password: GrandMaster + database: jedi_order + port: 34837 + postgres-version: "17" + ssl: true + + - name: 🪂 Test postgres + env: + CONNECTION_URI: ${{ steps.postgres.outputs.connection-uri }} + SERVICE_NAME: ${{ steps.postgres.outputs.service-name }} + CERTIFICATE_PATH: ${{ steps.postgres.outputs.certificate-path }} + EXPECTED_CONNECTION_URI: postgresql://yoda:GrandMaster@localhost:34837/jedi_order?sslmode=verify-ca&sslrootcert=${{ steps.postgres.outputs.certificate-path }} + EXPECTED_SERVICE_NAME: yoda + EXPECTED_SERVER_VERSION: 17 + EXPECTED_SSL: true + shell: bash + run: | + ##cd /home/runner/work/_actions/eq19/eq19/v2/integrations/tensorflow + ##pip install -e python_projects/* && iree-import-tflite -h && iree-import-tf -h + + ##cd /home/runner/work/_actions/eq19/eq19/v2/integrations/pjrt + ##pip install -e ctstools python_packages/iree_cpu_plugin + + ##pip install -r requirements.txt && cd python_packages/iree_cpu_plugin/build/cmake && ninja + ##JAX_PLATFORMS=iree-cpu pytest -n4 --max-worker-restart=9999 -p openxla_pjrt_artifacts --openxla-pjrt-artifact-dir=/tmp/foobar test/test_simple.py + + #cp ${{ github.workspace }}/.github/entrypoint/test.py . && python3 -m pytest -vv test.py + #python ${{ github.workspace }}/web_programming/cc_exchanges.py + + - name: 🚀 Initiate Feed Container + if: runner.os != 'Windows' + uses: devcontainers/ci@v0.3 + id: set_output + with: + push: always + imageTag: spin-${{ env.ID }} + skipContainerUserIdUpdate: true + imageName: ${{ inputs.image_name }} + configFile: _site/.devcontainer.json + env: | + TOKEN=${{ env.GH_TOKEN }} + REMOTE_REPO=${{ env.REMOTE_REPO }} + RERUN_RUNNER=${{ inputs.rerun_runner }} + BEARER=${{ steps.auth.outputs.id_token }} + TARGET_REPOSITORY=${{ env.TARGET_REPOSITORY }} + runCmd: bash user_data/ft_client/test_client/test_rest_client.sh + + - name: 🪂 Remove existing images + id: variables + shell: bash + run: | + BASE_NAME=$(basename ${{ inputs.image_name }}) + IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) + for ((i=0; i < ${#array[@]}; i++)); do + if [[ "${array[$i]}" != "latest" ]]; then + HUB_TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d "{\"username\": \"${{ inputs.hub_username }}\", \"password\": \"${{ inputs.hub_password }}\"}" https://hub.docker.com/v2/users/login/ | jq -r .token) + curl -s -i -X DELETE -H "Accept: application/json" -H "Authorization: JWT $HUB_TOKEN" https://hub.docker.com/v2/namespaces/${{ inputs.hub_username }}/repositories/$BASE_NAME/tags/${array[$i]} + echo "deleted: ${{ inputs.image_name }}:${array[$i]}" + fi + done diff --git a/.github/actions/1-addition/3-lexer/action.yml b/.github/actions/1-addition/3-lexer/action.yml new file mode 100644 index 000000000000..57357e65c2f9 --- /dev/null +++ b/.github/actions/1-addition/3-lexer/action.yml @@ -0,0 +1,134 @@ +name: 'Docker Deploy Action' + +description: > + 🪂 A Github Action to deploy pages conveniently + +branding: + icon: 'command' + color: 'green' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + default: ${{ github.action_repository }} + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + initiate_pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + +outputs: + artifact: + description: Collect artifact outputs + value: ${{ toJSON(steps.artifact.outputs) }} + variables: + description: Collect variables outputs + value: ${{ toJSON(steps.variables.outputs) }} + +runs: + using: composite + steps: + - name: 🪂 Adapt feed mapping + shell: bash + run: | + TARGET_REPO='https://github.com/${{ env.TARGET_REPOSITORY }}' + cd ${{ github.workspace }}/_site && rm -rf .git .devcontainer.json Dockerfile + git clone --single-branch -b gh-pages ${TARGET_REPO} ${{ inputs.action_path }}/_site &>/dev/null + + cd ${{ inputs.action_path }}/_site + shopt -s dotglob && rm -rf ${RUNNER_TEMP}/* + [[ -d docs ]] && mv -f docs ${RUNNER_TEMP}/ + [[ -d .sass-cache ]] && mv -f .sass-cache ${RUNNER_TEMP}/ + [[ -f .jekyll-metadata ]] && mv -f .jekyll-metadata ${RUNNER_TEMP}/ + + rm -rf * + #git rm -rfq . + #git clean -fxd + mv -f ${RUNNER_TEMP}/* . + mv -f ${{ github.workspace }}/_site/* . + cat ${{ inputs.action_path }}/Dockerfile + + - name: 🚀 Initiate Lexer + uses: devcontainers/ci@v0.3 + id: artifact + with: + push: always + imageTag: id-${{ env.ID }} + skipContainerUserIdUpdate: true + runCmd: ls -alR /home/runner/_site + imageName: ${{ inputs.image_name }} + configFile: ${{ inputs.action_path }}/.devcontainer.json + env: | + CXX=clang++ + CC=clang + FC=gfortran + TARGET_REPOSITORY=${{ env.TARGET_REPOSITORY }} + LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib + + - name: 💎 Variables + id: variables + shell: bash + env: + id: ${{ inputs.id }} + TARGET_REPOSITORY: ${{ env.TARGET_REPOSITORY }} + run: | + BASE_NAME=$(basename ${{ inputs.image_name }}) + echo 'output_${id}='${TARGET_REPOSITORY} >> ${GITHUB_OUTPUT} + IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) + for ((i=0; i < ${#array[@]}; i++)); do + if [ "${array[$i]}" != "latest" ]; then + HUB_TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d "{\"username\": \"${{ inputs.hub_username }}\", \"password\": \"${{ inputs.hub_password }}\"}" https://hub.docker.com/v2/users/login/ | jq -r .token) + curl -s -i -X DELETE -H "Accept: application/json" -H "Authorization: JWT $HUB_TOKEN" https://hub.docker.com/v2/namespaces/${{ inputs.hub_username }}/repositories/$BASE_NAME/tags/${array[$i]} + echo "deleted: ${{ inputs.image_name }}:${array[$i]}" + fi + done diff --git a/.github/actions/2-multiplication/1-maps/action.yml b/.github/actions/2-multiplication/1-maps/action.yml new file mode 100644 index 000000000000..4d2cc0a3bdad --- /dev/null +++ b/.github/actions/2-multiplication/1-maps/action.yml @@ -0,0 +1,139 @@ +name: 'Docker Deploy Action' +description: > + 🪂 A Github Action to deploy pages conveniently +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: "Name for the container to be created. Example: nginx" + required: false + image_tag: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + build_args: + description: Build args + required: false + default: "" + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + rerun_runner: + description: 'Redeploy of runner' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + action_path: + description: 'The action path' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + +branding: + icon: 'command' + color: 'yellow' + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + using: composite + steps: + - name: Login to DockerHub + uses: docker/login-action@v3 + if: runner.os == '-Windows' + with: + username: ${{ inputs.hub_username }} + password: ${{ inputs.hub_token }} + + - name: 💎 Set ID Token + uses: 'google-github-actions/auth@v2' + with: + credentials_json: ${{ inputs.credentials }} + + - name: 💎 Setup Gcloud + uses: 'google-github-actions/setup-gcloud@v2' + #with: + #install_components: 'alpha,cloud-datastore-emulator' + + # Ref: https://cloud.google.com/compute/docs/instances/create-start-instance + - name: 💎 Remove Existing Self-Hosted Runner + shell: bash + env: + JOB_ID: ${{ inputs.id }} + TOKEN: ${{ inputs.token }} + run: | + /Users/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh + + - name: View context attributes + uses: actions/github-script@v7 + env: + JOB_ID: ${{ inputs.id }} + GH_TOKEN: ${{ inputs.token }} + with: + script: console.log(context) + + - name: Set Docker Hub description + uses: peter-evans/dockerhub-description@v4 + with: + enable-url-completion: true + password: ${{ inputs.hub_token }} + username: ${{ inputs.hub_username }} + repository: ${{ inputs.image_name }} + readme-filepath: ${{ inputs.action_path }}/README.md + short-description: ${{ github.event.repository.description }} + + - name: Build and Push my_awesome_microservice + uses: mr-smithers-excellent/docker-build-push@v5 + if: runner.os == '-Windows' + with: + image: my_awesome_microservice + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ inputs.token }} + platform: windows/amd64 + tags: development-latest + + # Ref: https://cloud.google.com/compute/docs/instances/create-start-instance + - name: 💎 Remove Existing Self-Hosted Runner + if: inputs.rerun_runner == 'true' + shell: bash + run: | + for instance_uri in $(gcloud compute instances list --uri 2> /dev/null); do + instance_name=${instance_uri##*/} + instance_zone_uri=${instance_uri/\/instances\/${instance_name}/} + instance_zone=${instance_zone_uri##*/} + gcloud compute instances update "${instance_name}" --zone "${instance_zone}" --no-deletion-protection + if [ -n "${instance_name}" ] && [ -n "${instance_zone}" ] && gcloud compute instances delete ${instance_name} --zone ${instance_zone} --quiet; then + echo "deleted: ${instance_zone}/${instance_name}" + fi + done diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml new file mode 100644 index 000000000000..062adb50a36f --- /dev/null +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -0,0 +1,217 @@ +name: 'Docker Deploy Action' + +description: > + 🪂 A Github Action to deploy pages conveniently + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: "Name for the container to be created. Example: nginx" + required: false + image_tag: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + build_args: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + chat_id: + description: 'The telegram chat id' + required: false + bot_token: + description: 'The monitor token' + required: false + logs_token: + description: 'The warning token' + required: false + action_path: + description: 'The action path' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + initiate_pauli_sum: + description: Build args + required: false + default: "" + +outputs: + variables: + description: Collect variables outputs + value: ${{ toJSON(steps.variables.outputs) }} + metadata: + description: Collect metadata outputs + value: ${{ toJSON(steps.metadata.outputs) }} + +runs: + using: composite + steps: + - name: 🪂 Get tag of dependencies + id: variables + shell: bash + run: | + brew update --quiet + brew install --quiet coreutils doxygen pandoc gsl cmake ninja lcov gcc@12 qt@5 texlive + brew info qt@5 && brew info gsl + IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) + for ((i=0; i < ${#array[@]}; i++)); do + if [[ "${array[$i]}" != "latest" ]]; then + image_tag="${array[$i]}" + echo 'IMAGE_TAG='${image_tag##*-} >> ${GITHUB_ENV} + fi + done + + # See https://github.com/thesofproject/sof/blob/main/zephyr/docker-run.sh + - name: 🪂 Cloud Config + id: metadata + shell: bash + run: | + cat << EOL >> cloud-config + #cloud-config + users: + - default + - uid: 1000 + name: runner + groups: admin,adm,video,docker,google-sudoers + + write_files: + - path: /etc/systemd/system/firewall.service + permissions: "0644" + owner: root + content: | + [Unit] + Description=Configures the host firewall + + [Service] + Type=oneshot + RemainAfterExit=true + ExecStart=/sbin/iptables -A INPUT -p tcp --match multiport --dports 80,5432,6379 -j ACCEPT + + - path: /etc/systemd/system/redis.service + permissions: "0644" + owner: root + content: | + [Unit] + Description=Redis service + After=docker.service firewall.service + Wants=docker.service firewall.service + + [Service] + Restart=always + Environment="HOME=/home/runner" + ExecStop=/usr/bin/docker exec myredis -s quit + ExecStartPre=/usr/bin/docker-credential-gcr configure-docker + ExecStart=/usr/bin/docker run --rm -v /mnt:/mnt --name=myredis -p 6379:6379 --pull=always eq19/maps:target-${{ github.run_number }} + + - path: /etc/systemd/system/postgres.service + permissions: "0644" + owner: root + content: | + [Unit] + Description=Postgres service + After=redis.service + Wants=redis.service + + [Service] + Restart=always + Environment="HOME=/home/runner" + ExecStop=/usr/bin/docker exec mydb -s quit + ExecStartPre=/usr/bin/docker-credential-gcr configure-docker + ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt -v /mnt/disks/deeplearning/home/runner:/home/runner --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} + + - path: /etc/systemd/system/runner.service + permissions: "0644" + owner: root + content: | + [Unit] + Description=gcloud compute + Wants=postgres.service + After=postgres.service + + [Service] + RestartSec=5s + Restart=on-failure + Environment="HOME=/home/runner" + ExecStop=/usr/bin/docker stop myrunner + ExecStopPost=/usr/bin/docker rm myrunner + ExecStartPre=/usr/bin/docker-credential-gcr configure-docker + ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=Google-optimized-instance -e RUNNER_REPOSITORY_URL=https://github.com/${{ inputs.repository }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name myrunner --pull=always eq19/lexer:id-${{ env.IMAGE_TAG }} + + bootcmd: + - fsck.ext4 -tvy /dev/sdb1 + - mkdir -p /mnt/disks/deeplearning + - mount -o nosuid,nodev,commit=30,defaults -t ext4 /dev/sdb1 /mnt/disks/deeplearning + + runcmd: + - systemctl stop docker + - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos /mnt/disks/deeplearning/home + - mv /var/lib/docker/* /mnt/disks/deeplearning/var/lib/docker/cos/ || true + - find /var/lib/docker -mindepth 1 -delete + - rmdir /var/lib/docker || true + - chattr +i /var/lib/docker + - mount --bind /mnt/disks/deeplearning/var/lib/docker/cos /var/lib/docker + - systemctl daemon-reload + - systemctl start docker + - docker run --rm --pull=always --entrypoint "" -v /mnt/disks/deeplearning/home:/mnt/disks/deeplearning/home eq19/feed:spin-${{ env.IMAGE_TAG }} sh -c "cp -r /home/runner /mnt/disks/deeplearning/home/" + - systemctl start runner.service + EOL + + # Ref: https://stackoverflow.com/a/47018417/4058484 + - name: 🪂 Setup Runner by Gcloud + #if: runner.os == 'Windows' + id: instances + shell: bash + run: > + gcloud compute instances create grid-${{ env.IMAGE_TAG }} + --service-account=project-owner@feedmapping.iam.gserviceaccount.com + --scopes=https://www.googleapis.com/auth/cloud-platform --deletion-protection + --zone=us-central1-a --machine-type=e2-micro --enable-display-device --tags=https-server + --boot-disk-device-name=cos --image-project cos-cloud --image-family cos-stable --metadata-from-file=user-data=cloud-config + --create-disk=boot=no,auto-delete=yes,name=tensorflow,device-name=deeplearning,mode=rw,size=50,type=pd-standard,image=projects/ml-images/global/images/c1-deeplearning-tf-2-11-cu113-v20230925-debian-10-py37 diff --git a/.github/actions/2-multiplication/3-lexer/action.yml b/.github/actions/2-multiplication/3-lexer/action.yml new file mode 100644 index 000000000000..78017617e3f1 --- /dev/null +++ b/.github/actions/2-multiplication/3-lexer/action.yml @@ -0,0 +1,166 @@ +name: 'Docker Deploy Action' + +description: > + 🪂 A Github Action to deploy pages conveniently + +branding: + icon: 'command' + color: 'green' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + default: ${{ github.action_repository }} + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + initiate_pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + config: + description: The config of secrets variables + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + repository: + description: 'The deploy repository' + default: ${{ github.repository }} + required: false + +outputs: + describe: + description: Collect describe outputs + value: ${{ steps.describe.outputs.ip }} + compute: + description: Collect compute outputs + value: ${{ steps.describe.outputs.function }} + +runs: + using: composite + steps: + - name: configure + #if: runner.os == 'Windows' + shell: bash + env: + CXX: g++-12 + CC: gcc-12 + FC: gfortran-12 + run: | + mkdir ${{ inputs.action_path }}/build + cd ${{ inputs.action_path }}/build + echo "GSL prefix: $(brew --prefix gsl)" + echo "Content of GSL prefix:" + ls $(brew --prefix gsl) + export PATH=$PATH:/usr/local/bin + export CPATH=$CPATH:/usr/local/include:$(brew --prefix gsl)/include + export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include:$(brew --prefix gsl)/include + export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + cmake .. -DQt5_DIR=$(brew --prefix qt5)/lib/cmake/Qt5 -DCMAKE_PREFIX_PATH=$(brew --prefix qt5)/lib/cmake + make && sudo make install && ctest --output-on-failure + + #Ref: https://gist.github.com/xtrmstep/92aab2d465e348ce1c07ed903ca0dfb8 + - id: describe + if: runner.os == 'Windows' + shell: bash + #env: + #CONFIG_JSON: ${{ inputs.config }} + run: | + #echo -n "$CONFIG_JSON" > config.json + cd ${{ inputs.action_path }} && doxygen > /dev/null 2>&1 + rm -rf ${{ github.workspace }}/xml && mv -f xml ${{ github.workspace }}/ + + COMMIT="${{ env.LATEST_COMMIT }}" + cd ${{ github.workspace }} && rm -rf cloud-config + + # Remove double quotes using parameter extension + git add . && git commit --allow-empty -m "${COMMIT//\"/}" && git push + + #ssh-keygen -t rsa -b 3072 -N '' -f $HOME/.ssh/google_compute_engine -C root + #sed -i.backup "1s|^|runner: |" $HOME/.ssh/google_compute_engine.pub && gh secret set GCP_ID_RSA < $HOME/.ssh/google_compute_engine + + # Ref: https://console.cloud.google.com/security/secret-manager?hl=en&project=feedmapping + #gcloud secrets describe freqtrade-config --project feedmapping || \ + #gcloud secrets create freqtrade-config --project feedmapping --data-file=config.json + #gcloud secrets versions add freqtrade-config --project feedmapping --data-file=config.json + + gcloud compute instances add-metadata grid-${{ env.IMAGE_TAG }} --zone=us-central1-a \ + --metadata=startup-script=/home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/startup.sh + + #gcloud compute instances describe grid-${{ env.IMAGE_TAG }} --zone=us-central1-a --format='value(metadata.items)' + #gcloud compute scp ${{ github.workspace }}/.github/entrypoint/artifact/python/manual_v2.ipynb grid-${{ env.IMAGE_TAG }}:/home/runner/.docker --zone=us-central1-a + + echo $(gcloud compute instances describe grid-${{ env.IMAGE_TAG }} --zone=us-central1-a --format json) > instance + echo 'ip='$(gcloud compute instances describe grid-${{ env.IMAGE_TAG }} --zone=us-central1-a --format='value(networkInterfaces.accessConfigs[0].natIP)') >> ${GITHUB_OUTPUT} + echo 'function='$(curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function -H "Authorization: Bearer $(gcloud auth print-identity-token)" -H "Content-Type: application/json" --data @instance | base64) >> ${GITHUB_OUTPUT} + + #Ref: https://blog.benoitblanchon.fr/github-action-run-ssh-commands/ + - name: Configure SSH + if: runner.os == 'Windows' + id: compute + shell: bash + env: + IP: ${{ steps.describe.outputs.ip }} + run: | + #localedef -i en_US -f UTF-8 en_US.UTF-8 + ssh-keyscan -t rsa $IP >> $HOME/.ssh/known_hosts + chmod 600 $HOME/.ssh/id_rsa + cat << EOL >> $HOME/.ssh/config + Host staging + HostName $IP + User runner + IdentityFile $HOME/.ssh/id_rsa + StrictHostKeyChecking no + EOL + #Ref: https://github.com/cloudkite-io/google-container-optimized-os-tools + ssh staging 'sudo whoami && \ + sudo echo ${{ steps.setup_runner.outputs.function }} > /home/runner/.ssh/function.json && \ + sudo chown root:root /home/runner/.ssh/*.json && sudo mv -f /home/runner/.ssh/*.json /home/runner/.docker/' + if [ $? -eq 0 ]; then + ssh staging 'sudo ls -alR /home/runner' + else + exit 1 + fi diff --git a/.github/actions/3-exponentiation/4-parser/action.yml b/.github/actions/3-exponentiation/4-parser/action.yml new file mode 100644 index 000000000000..4a8cf9e1281f --- /dev/null +++ b/.github/actions/3-exponentiation/4-parser/action.yml @@ -0,0 +1,105 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + failure_status: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + - name: 💎 Reinvented + shell: bash + env: + JOB_ID: ${{ inputs.id }} + TOKEN: ${{ inputs.token }} + BASE: "https://github.com/eq19/eq19.github.io.wiki.git" + WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" + run: | + if [ -d /mnt/disks/deeplearning/usr/local/sbin ]; then + export PAULI_SUM=$(echo -n "${{ inputs.pauli_sum }}" | base64 --decode); + export PATH=/home/runner/_work/_actions/eq19/eq19/v2/.github/entrypoint:$PATH && init.sh + else + exit 1 + fi + + - name: View context attributes + uses: actions/github-script@v7 + env: + JOB_ID: ${{ inputs.id }} + GH_TOKEN: ${{ inputs.token }} + with: + script: console.log(context) + + - name: 🏗️ Collect outputs + uses: mathiasvr/command-output@v2.0.0 + id: command + with: + run: | + date +'%Y-%m-%d' diff --git a/.github/actions/3-exponentiation/5-syntax/action.yml b/.github/actions/3-exponentiation/5-syntax/action.yml new file mode 100644 index 000000000000..cc2aa02b3bba --- /dev/null +++ b/.github/actions/3-exponentiation/5-syntax/action.yml @@ -0,0 +1,120 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + api_key: + description: 'The api key' + required: true + api_secret: + description: 'The api secret' + required: false + token: + description: 'The deploy token' + required: false + workspace: + description: 'The deploy repository' + required: false + failure_status: + description: 'The deploy branch' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + #Fill in metadata with ${{ steps.artifact.outputs.orgs_json }} + - name: Run identity + shell: bash + run: | + TOKEN=$(/mnt/disks/deeplearning/usr/bin/gcloud auth print-identity-token) + curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ + -H "Authorization: Bearer ${TOKEN}" -H "Content-Type: application/json" \ + --data @${{ github.workspace }}/docs/data.json | jq '.' + + # Bring GitHub Env Variables in to the config file on another container usong docker from the attached deeplearning image + /mnt/disks/deeplearning/usr/bin/docker exec mydb sed -i "s|your_exchange_key|${{ inputs.api_key }}|g" /home/runner/user_data/config.json + /mnt/disks/deeplearning/usr/bin/docker exec mydb sed -i "s|your_exchange_secret|${{ inputs.api_secret }}|g" /home/runner/user_data/config.json + + - name: 💎 Connect to Redis + # Runs a script that creates a Redis client, populates + # the client with data, and retrieves data + env: + # The hostname used to communicate with the Redis client service container + # Ref: https://gitlab.com/NamingThingsIsHard/crypto/freqtrade/clients + # Ref: https://stackoverflow.com/a/48547074/4058484 + REDIS_PORT: 6379 + REDIS_HOST: 172.17.0.1 + shell: bash + run: | + npm run test + echo '${{ inputs.pauli_sum }}' + + - name: 💎 Connect to postgres + env: + # use the internal host here because we have specified a container for the job. + # If we were running the job on the VM this would be localhost + POSTGRES_PORT: 5432 + POSTGRES_HOST: 172.17.0.1 + shell: bash + run: | + node /home/runner/scripts/post.js + echo '${{ inputs.pauli_sum }}' diff --git a/.github/actions/3-exponentiation/6-grammar/action.yml b/.github/actions/3-exponentiation/6-grammar/action.yml new file mode 100644 index 000000000000..7bc9107ba4d3 --- /dev/null +++ b/.github/actions/3-exponentiation/6-grammar/action.yml @@ -0,0 +1,140 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + workspace: + description: 'The deploy repository' + required: false + failure_status: + description: 'The deploy branch' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + - name: ⏩ Next wiki's jobs on applying + shell: bash + env: + BASE: "D:/a/_actions/eq19/eq19/v2/.github" + run: | + if [[ '${{ inputs.failure_status }}' == 'false' ]]; then + + TARGET_WIKI="https://${{ github.actor }}:${{ inputs.token }}@github.com/$TARGET_REPOSITORY.wiki.git" + git clone $TARGET_WIKI ${{ inputs.runner_temp }}/wiki + cd ${{ inputs.runner_temp }}/wiki + cat _Sidebar.md + + #ls -alR /home/runner/xml + #ls -alR /home/runner/templates/doxygen/doc + + fi + + - name: ⏪ Rewind all jobs on failure + shell: bash + env: + GH_TOKEN: ${{ inputs.token }} + run: | + if [[ '${{ inputs.failure_status }}' == 'true' ]]; then + + git config --global user.name "$GITHUB_ACTOR" + git config --global --add safe.directory "$GITHUB_WORKSPACE" + git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" + + REMOTE_REPO="https://$GITHUB_ACTOR:$GH_TOKEN@github.com/$GITHUB_REPOSITORY.git" + git clone --single-branch --branch gh-pages $REMOTE_REPO gh-pages && cd gh-pages + git add . && git commit --allow-empty -m "rerun due to job failure" && git push + + else + + cd /home/runner/_site && rm -rf .git .github .env .ssh .cache node-modules + + cd ${{ github.workspace }} + find -not -path "./.git/*" -not -name ".git" | grep git + find -not -path "./.git/*" -not -name ".git" -delete + shopt -s dotglob && mv -f /home/runner/_site/* . + + # Assign the environment variable correctly + COMMIT="${{ env.LATEST_COMMIT }}" + + # Remove double quotes + MESSAGE=$(echo "$COMMIT" | tr -d '"') + + # Debug output to verify + echo "Original: $COMMIT" + echo "Processed: $MESSAGE" + + # Commit and push changes + git add . + git commit --allow-empty -m "$MESSAGE" + git push + + fi diff --git a/.github/actions/4-identition/4-parser/action.yml b/.github/actions/4-identition/4-parser/action.yml new file mode 100644 index 000000000000..3767836dbeab --- /dev/null +++ b/.github/actions/4-identition/4-parser/action.yml @@ -0,0 +1,98 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + runner_temp: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + workspace: + description: 'The deploy repository' + required: false + failure_status: + description: 'The deploy branch' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + - name: Run identity + shell: bash + env: + JOB_ID: ${{ inputs.id }} + TOKEN: ${{ inputs.token }} + run: | + /d/a/_actions/eq19/eq19/v2/.github/entrypoint/init.sh + + - name: View context attributes + uses: actions/github-script@v7 + env: + JOB_ID: ${{ inputs.id }} + GH_TOKEN: ${{ inputs.token }} + with: + script: console.log(context) diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml new file mode 100644 index 000000000000..15c8f27eca46 --- /dev/null +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -0,0 +1,122 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + workspace: + description: 'The deploy repository' + required: false + failure_status: + description: 'The deploy branch' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + - name: 💎 Setup Ruby + uses: ruby/setup-ruby@v1 + if: runner.os == 'Windows' + with: + cache-version: 1 + rubygems: default + bundler-cache: true + ruby-version: '3.3.4' + + - name: Setup .NET SDK + uses: actions/setup-dotnet@v4 + with: + dotnet-version: 8.0.x + + - name: 💎 Set ID Token + id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + token_format: 'id_token' + create_credentials_file: false + credentials_json: ${{ inputs.credentials }} + id_token_audience: 'https://us-central1-feedmapping.cloudfunctions.net/function' + + #Fill in metadata with ${{ steps.artifact.outputs.orgs_json }} + - name: Run identity + shell: bash + env: + JOB_ID: ${{ inputs.id }} + GH_TOKEN: ${{ inputs.token }} + GCP_TOKEN: ${{ steps.auth.outputs.id_token }} + BASE_GIT: "/d/a/_actions/eq19/eq19/v2/.github" + BASE: "https://github.com/eq19/eq19.github.io.wiki.git" + WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" + run: | + curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ + -H "Authorization: Bearer ${GCP_TOKEN}" -H "Content-Type: application/json" \ + --data @${{ inputs.workspace }}/docs/data.json | jq '.' + export PATH=$BASE_GIT/entrypoint:$PATH && github_pages.sh + diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml new file mode 100644 index 000000000000..e566e16d9359 --- /dev/null +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -0,0 +1,139 @@ +name: 'Jekyll Deploy Action' + +description: > + A Github Action to deploy the Jekyll site conveniently for GitHub Pages + +branding: + icon: 'command' + color: 'red' + +inputs: + id: + description: The deploy id + required: false + hub_username: + description: Username for Docker Hub + default: ${{ github.actor }} + required: true + hub_password: + description: Docker Hub authentication token + required: true + hub_token: + description: Docker Hub authentication token + required: true + action_path: + description: Path to the dockerfile from which to build the image + required: true + credentials: + description: 'The gcloud credentials' + required: true + image_name: + description: Tag to use for the new image + required: true + dockerfile_path: + description: Path to the dockerfile from which to build the image + required: true + pauli_sum: + description: Build args + required: false + default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true + cache_src: + description: Path for the docker cache source + required: false + default: /tmp/.buildx-cache + cache_dest: + description: Path for the docker cache destination + required: false + default: /tmp/.buildx-cache-new + provider: + description: 'The deploy provider' + required: true + owner: + description: 'The deploy owner' + required: false + token: + description: 'The deploy token' + required: false + workspace: + description: 'The deploy repository' + required: false + failure_status: + description: 'The deploy branch' + required: false + runner_temp: + description: 'The deploy branch' + required: false + +outputs: + all_outputs: + description: Collect all outputs + value: ${{ toJSON(steps.*.outputs) }} + +runs: + #using: 'node20' + #main: '.github/run/main.js' + #post: '.github/run/post.js' + #post-if: 'success()' + using: composite + + steps: + - name: ⏪ Rewind all jobs on failure + shell: bash + env: + GH_TOKEN: ${{ inputs.token }} + BASE: "D:/a/_actions/eq19/eq19/v2/.github" + run: | + gh variable set RERUN_RUNNER --body "false" + if [[ '${{ inputs.failure_status }}' == 'true' ]]; then + + git config --global user.name "$GITHUB_ACTOR" + git config --global --add safe.directory "$GITHUB_WORKSPACE" + git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" + + REMOTE_REPO="https://$GITHUB_ACTOR:$GH_TOKEN@github.com/$GITHUB_REPOSITORY.git" + git clone --single-branch --branch gh-pages $REMOTE_REPO gh-pages && cd gh-pages + git add . && git commit --allow-empty -m "rerun due to job failure" && git push + + else + + cd ${GITHUB_WORKSPACE//\\//} + + echo -e "\n$hr\nConfig\n$hr" + cat _config.yml + + echo -e "\n$hr\nSource Pages\n$hr" + ls -al ${GITHUB_WORKSPACE//\\//} + + echo -e "\n$hr\nBuild Pages\n$hr" + #Jekyll Quick Reference https://gist.github.com/DrOctogon/bfb6e392aa5654c63d12 + JEKYLL_GITHUB_TOKEN=${{ inputs.token }} DISABLE_WHITELIST=true bundle exec jekyll build --profile -t -p ${GITHUB_WORKSPACE//\\//}/_plugins -d ${GITHUB_WORKSPACE//\\//}/docs + + if [[ "${TARGET_REPOSITORY}" == "eq19/eq19.github.io" ]]; then echo "www.eq19.com" > ${GITHUB_WORKSPACE//\\//}/docs/CNAME; fi + rm -rf ${GITHUB_WORKSPACE//\\//}/docs/.nojekyll && touch ${GITHUB_WORKSPACE//\\//}/docs/.nojekyll + + echo -e "\n$hr\nFinal Pages\n$hr" + ls -al ${GITHUB_WORKSPACE//\\//}/docs + + # Assign the environment variable correctly + COMMIT="${{ env.LATEST_COMMIT }}" + + # Remove double quotes + MESSAGE=$(echo "$COMMIT" | tr -d '"') + + # Debug output to verify + echo "Original: $COMMIT" + echo "Processed: $MESSAGE" + + # Commit and push changes + git add . + git commit --allow-empty -m "$MESSAGE" + git push + + fi diff --git a/.github/actions/reserves/ubuntu/README.md b/.github/actions/reserves/ubuntu/README.md new file mode 100644 index 000000000000..cf195015b3d9 --- /dev/null +++ b/.github/actions/reserves/ubuntu/README.md @@ -0,0 +1,4 @@ +# Github actions for WSL +This page has been moved to our documentation + +https://canonical-ubuntu-wsl.readthedocs-hosted.com/en/latest/ diff --git a/.github/actions/reserves/ubuntu/download-rootfs/action.yaml b/.github/actions/reserves/ubuntu/download-rootfs/action.yaml new file mode 100644 index 000000000000..ac58e3643af7 --- /dev/null +++ b/.github/actions/reserves/ubuntu/download-rootfs/action.yaml @@ -0,0 +1,103 @@ +name: Download Ubuntu WSL Rootfs +description: Download the latest Rootfs for a particular release of Ubuntu WSL + +inputs: + distros: + description: Space-separated list of distros to download + required: true + default: "Ubuntu" + path: + description: Directory to store the rootfs in (${path}\${distro}.tar.gz) + required: true + default: "Ubuntu" + +runs: + using: "composite" + steps: + - name: Check out repo + uses: actions/checkout@v3 + - name: Install python + uses: actions/setup-python@v4 + with: + python-version: "3.10" + - name: Install dependencies + shell: powershell + run: python -m pip install launchpadlib + - name: Set up Go + uses: actions/setup-go@v3 + with: + go-version: "1.21.4" + - name: Generate WSL release info + shell: powershell + # Repository root + working-directory: ${{ github.action_path }}\..\..\..\ + run: | + # Prepare environment + + # Install PowerShell + winget install --id Microsoft.Powershell --silent --accept-package-agreements --accept-source-agreements --source winget + + Write-Output "::group::Build wsl release info generator" + go build .\wsl-builder\release-info\ + if ( "${LastExitCode}" -ne "0" ) { + Write-Output "::endgroup::" + Exit(1) + } + Write-Output "::endgroup::" + + Write-Output "::group::Fetch release info" + python .\wsl-builder\lp-distro-info -o "${{ inputs.path }}\ubuntu-releases.csv" + Get-Content "${{ inputs.path }}\ubuntu-releases.csv" + Write-Output "::endgroup::" + + Write-Output "::group::Filter release info" + if(!(Test-Path "${{ inputs.path }}")) { + New-Item -Type Directory -Path "${{ inputs.path }}" + } + .\release-info.exe "${{ inputs.path }}\ubuntu-releases.csv" ${{ inputs.distros }} --rootfs amd64 --short 1> "${{ inputs.path }}\wsl-releases.csv" + Write-Output "::endgroup::" + + # Check for success + if (Test-Path "${{ inputs.path }}\wsl-releases.csv") { + Exit(0) + } + Write-Error "Did not write WSL releases file" + Exit(1) + - name: Download Ubuntu rootfs + shell: powershell + run: | + # Download rootfs + cd "${{ inputs.path }}" + + $releaseList = $(Get-Content "wsl-releases.csv") -split "\n" + $allSuccess = $true + + foreach ($release in $releaseList) { + $data = $release -split "\s+" + + # $name is the one in inputs.distros + $name = $data[0] + $file = ".\${name}.tar.gz" + $url = $data[1] + + & ${{ github.action_path }}\download-rootfs.ps1 -Path "${file}" -URL "$url" + if ( ! $? ) { + $allSuccess = $false + Write-Output "Error: Could not download ${name}" + } + + Write-Output "::endgroup::" + } + + if ( "${allSuccess}" -eq "$false" ) { + Exit(1) + } + Exit(0) + - name: Cleanup + if: always() + shell: powershell + run: | + # Cleanup + Remove-Item "${{ inputs.path }}\ubuntu-releases.csv" 2>&1 | Out-Null + Remove-Item "${{ inputs.path }}\wsl-releases.csv" 2>&1 | Out-Null + Exit(0) diff --git a/.github/actions/reserves/ubuntu/download-rootfs/download-rootfs.ps1 b/.github/actions/reserves/ubuntu/download-rootfs/download-rootfs.ps1 new file mode 100644 index 000000000000..d37e6e2e8360 --- /dev/null +++ b/.github/actions/reserves/ubuntu/download-rootfs/download-rootfs.ps1 @@ -0,0 +1,110 @@ +param ( + [Parameter(Mandatory = $true, HelpMessage = "The path where to download the tarball")] + [string]$Path, + [Parameter(Mandatory = $true, HelpMessage = "The URL of the tarball")] + [Uri]$URL +) + +# Global variables +$RootFS = "${Path}" +$Directory = Split-Path -Path "${RootFS}" -Parent +$Checksums = "${Directory}\SHA256SUMS" + +# Helper functions + +function Get-URIParent { + param ( + [Parameter(Mandatory=$true)] [Uri]$URI + ) + + $segments = $URI.Segments + $end = $segments.Length - 2 + $trailing = [string]::Join("", $segments[0..$end]) + + return "$($URI.Scheme)://$($URI.Authority)${trailing}" +} + +function Remove-File { + param ( + [Parameter(Mandatory=$true)] [string]$Path + ) + if (!(Test-Path -Path $Path)) { + return + } + Remove-Item -Path "${Path}" -Force 2>&1 | Out-Null +} + +# Test-Checksums returns true if the current rootfs checksum matches the remote one. +# Returns false otherwise, or if any error arises. +function Test-Checksums { + # Get Hash of the current RootFS + if (!(Test-Path -Path ${RootFS})) { + return $false + } + + $oldSHA256 = (Get-FileHash "${Rootfs}" -Algorithm SHA256).Hash + if ($oldSHA256 -eq "") { + Write-Warning "Could not compute hash of ${Rootfs}" + return $false + } + + # Download remote checksum + $checksumURL = "$(Get-URIParent -URI "${URL}")SHA256SUMS" + pwsh.exe -Command pwsh.exe -Command Invoke-WebRequest -Uri "${checksumURL}" -OutFile "${Checksums}" -Resume -MaximumRetryCount 5 -RetryIntervalSec 2 + if (! $?) { + Write-Warning "Could not download checksums" + return $false + } + + if (!(Test-Path "${Checksums}")) { + Write-Warning "Could not download checksums" + return $false + } + + # Parse checksum file + $image = $URL.Segments[$Url.Segments.Length - 1] + $pattern = "(\w+) ${image}" + + $newSHA256 = (Select-String -Pattern "${pattern}" -Path "${Checksums}").Matches[0].Groups[1] + if ($newSHA256 -eq "") { + Write-Warning "Could not find $image in checksums file" + return $false + } + + # Compare checksums + if ($oldSHA256 -ne $newSHA256) { + return $false + } + return $true +} + +function main { + New-Item -Type Directory -Force "${Directory}" 2>&1 | Out-Null + + # Validate checksum + if ( Test-Checksums ) { + Write-Warning "Cache hit: skipping download" + return 0 + } + + # Download Rootfs into temp file + pwsh.exe -Command Invoke-WebRequest -Uri "${URL}" -OutFile "${RootFS}.tmp" -Resume -MaximumRetryCount 5 -RetryIntervalSec 2 + if ( ! $? ) { + Write-Error "Error: Failed download" + return 1 + } + + # Move downloaded tmp file into position + Move-Item -Force "${RootFS}.tmp" "${RootFS}" + return 0 +} + +# Main function +$exitCode = main + +# Clean artefacts +Remove-File -Path "${RootFS}.tmp" +Remove-File -Path "${Checksums}" + +Exit($exitCode) + diff --git a/.github/actions/reserves/ubuntu/wsl-bash/action.yaml b/.github/actions/reserves/ubuntu/wsl-bash/action.yaml new file mode 100644 index 000000000000..efc56f72533e --- /dev/null +++ b/.github/actions/reserves/ubuntu/wsl-bash/action.yaml @@ -0,0 +1,60 @@ +name: Azure WSL script runner +description: 'Runs scripts on WSL in Azure machines.' + +inputs: + exec: + description: Script to run on WSL + required: true + working-dir: + description: The WSL directory to run the script in. Must be absolute, but ~ is allowed. + required: false + default: "~" + distro: + description: Which (installed) distro to run the script on. Use the name as shown in 'wsl -l -v'. + required: false + default: Ubuntu + +runs: + using: 'composite' + steps: + - name: Copy script into WSL + shell: powershell + env: + exec: ${{ inputs.exec }} + scriptDir: C:\Temp\Ubuntu-WSL-actions\${{github.run_number}}_${{github.run_attempt}} + run: | + Write-Output '::group::Storage of script into WSL' + + New-Item -Force -Path "${{ env.scriptDir }}" -ItemType Directory | Out-Null + + # The following function is used because: + # - It does not write the BOM (0xff 0xfe at file start) + # - It does not write \r at line ends + # - Overwrites a pre-existing file if there was one + [IO.File]::WriteAllText( "${{ env.scriptDir }}\script.sh" , "${env:exec}") + $exit=$? + + Write-Output '::endgroup::' + if ( "${exit}" -eq "$False" ) { Exit(1) } + - name: Run linux script + shell: powershell + env: + scriptDir: C:\Temp\Ubuntu-WSL-actions\${{github.run_number}}_${{github.run_attempt}} + WSL_UTF8: "1" + run: | + # Run script + + # We use this --cd trick otherwise bash gets confused with the \ path separators + $linuxPath = wsl.exe -d "${{ inputs.distro }}" --cd "${{ env.scriptDir }}" -- wslpath -ua . + + wsl.exe -d ${{ inputs.distro }} --cd '${{ inputs.working-dir }}' -- bash "$linuxPath/script.sh" + if ( ! $? ) { Exit(1) } + - name: Cleanup + if: always() + shell: powershell + env: + scriptDir: C:\Temp\Ubuntu-WSL-actions\${{github.run_number}}_${{github.run_attempt}} + run: | + if(Test-Path -Path "${{ env.scriptDir }}") { + Remove-Item -Recurse -Force -Path "${{ env.scriptDir }}" + } \ No newline at end of file diff --git a/.github/actions/reserves/ubuntu/wsl-checkout/action.yaml b/.github/actions/reserves/ubuntu/wsl-checkout/action.yaml new file mode 100644 index 000000000000..241fcf267ad0 --- /dev/null +++ b/.github/actions/reserves/ubuntu/wsl-checkout/action.yaml @@ -0,0 +1,38 @@ +name: Azure WSL repo clone +description: 'Clones your repo into WSL.' + +inputs: + distro: + description: Which (installed) distro to use. Use the name as shown in 'wsl -l -v'. + required: true + working-dir: + description: Where the repo should be cloned + required: false + default: "." + submodules: + description: Whether to fetch submodules or not + required: false + default: "false" + +runs: + using: 'composite' + steps: + - name: Clone repo + uses: ubuntu/WSL/.github/actions/wsl-bash@main + with: + distro: ${{ inputs.distro }} + exec: | + set -eu + + export GIT_TERMINAL_PROMPT=0 + + mkdir -p ${{ inputs.working-dir }} + git clone --quiet --depth 1 https://github.com/${{ github.repository }}.git ${{ inputs.working-dir }} + + cd ${{ inputs.working-dir }} + git fetch --quiet --depth 1 --no-tags --prune --update-head-ok origin +${{ github.sha }}:${{ github.ref }} + git checkout --quiet ${{ github.sha }} + + if [ "${{ inputs.submodules }}" = true ] ; then + git submodule --quiet update --init --recursive --depth 1 + fi \ No newline at end of file diff --git a/.github/actions/reserves/ubuntu/wsl-install/action.yaml b/.github/actions/reserves/ubuntu/wsl-install/action.yaml new file mode 100644 index 000000000000..a41594b33c18 --- /dev/null +++ b/.github/actions/reserves/ubuntu/wsl-install/action.yaml @@ -0,0 +1,57 @@ +name: Install Ubuntu WSL +description: 'Install WSL from the store' + +inputs: + distro: + description: Distro name as shown in 'wsl -l -v'. Leave blank to install no distro. + required: false + default: "" + +runs: + using: 'composite' + steps: + - name: Install WSL + shell: powershell + continue-on-error: true + run: | + Write-Output "::group::Install Windows Subsystem for Linux" + winget install --name 'Windows Subsystem for Linux' --accept-source-agreements --accept-package-agreements --silent --verbose + Write-Output "::endgroup::" + + - name: Install the new distro + if: ${{ inputs.distro != '' }} + shell: powershell + run: | + # Install the new distro + switch ( "${{ inputs.distro }}" ) + { + "Ubuntu" { $storeId="9PDXGNCFSCZV" ; $launcher="ubuntu.exe" ; $appxname="Ubuntu" } + "Ubuntu-Preview" { $storeId="9P7BDVKVNXZ6" ; $launcher="ubuntupreview.exe" ; $appxname="UbuntuPreview" } + "Ubuntu-22.04" { $storeId="9PN20MSR04DW" ; $launcher="ubuntu2204.exe" ; $appxname="Ubuntu22.04LTS" } + "Ubuntu-20.04" { $storeId="9MTTCL66CPXJ" ; $launcher="ubuntu2004.exe" ; $appxname="Ubuntu20.04LTS" } + "Ubuntu-18.04" { $storeId="9PNKSF5ZN4SW" ; $launcher="ubuntu1804.exe" ; $appxname="Ubuntu18.04LTS" } + default { Write-Output "Error: Unknown distro ${distro}" ; Exit(1) } + } + + # Uninstall pre-existing appx + $pkgFullName = (Get-AppxPackage | Where-Object Name -like "CanonicalGroupLimited.${appxname}").PackageFullName + if ( "$pkgFullName" -ne "" ) { + Remove-AppxPackage "${pkgFullName}" 3>&1 2>&1 | Out-Null + } + + # Unregister leftover distros + $env:WSL_UTF8=1 + wsl.exe --shutdown + wsl.exe --unregister ${{ inputs.distro }} 3>&1 2>&1 | Out-Null + + Write-Output "::group::Install a new distro" + winget install --id "${storeId}" --accept-source-agreements --accept-package-agreements --silent + Write-Output "::endgroup::" + + # Register the new distro + & ${launcher} install --root --ui=none + + # Testing succesful install + & ${launcher} run echo Created a new distro with user: '$(whoami)' + + if ( ! $? ) { Exit(1) } \ No newline at end of file diff --git a/.github/csc.json b/.github/csc.json new file mode 100644 index 000000000000..d2888fd15929 --- /dev/null +++ b/.github/csc.json @@ -0,0 +1,18 @@ +{ + "problemMatcher": [ + { + "owner": "csc", + "pattern": [ + { + "regexp": "^([^\\s].*)\\((\\d+)(?:,\\d+|,\\d+,\\d+)?\\):\\s+(error|warning)\\s+([a-zA-Z]+(? ${RUNNER_TEMP}/pinned_repos + else + QUERY='{"query":"{\n organization(login: \"'${OWNER}'\") {\n pinnedItems(first: 6, types: REPOSITORY) {\n nodes {\n ... on Repository {\n name\n }\n }\n }\n }\n}"' + curl -s -X POST "${GITHUB_GRAPHQL_URL}" -H "Authorization: bearer ${GH_TOKEN}" --data-raw "${QUERY}" | jq --raw-output '.data.organization.pinnedItems.nodes[].name' | yq eval -P | sed "s/ /, /g" > ${RUNNER_TEMP}/pinned_repos + sed -i "1s|^|maps, feed, lexer, parser, syntax, grammar, |" ${RUNNER_TEMP}/pinned_repos + fi + IFS=', '; array=($(cat ${RUNNER_TEMP}/pinned_repos)) + else + gh api -H "${HEADER}" /user/orgs --jq '.[].login' | sort -uf | yq eval -P | sed "s/ /, /g" > ${RUNNER_TEMP}/user_orgs + IFS=', '; array=($(cat ${RUNNER_TEMP}/user_orgs)) + echo "[" > ${RUNNER_TEMP}/orgs.json + for ((i=0; i < ${#array[@]}; i++)); do + QUERY='{"query":"{\n organization(login: \"'${array[$i]}'\") {\n pinnedItems(first: 6, types: REPOSITORY) {\n nodes {\n ... on Repository {\n name\n }\n }\n }\n }\n}"' + IFS=', '; pr=($(curl -s -X POST "${GITHUB_GRAPHQL_URL}" -H "Authorization: bearer ${GH_TOKEN}" --data-raw "${QUERY}" | jq --raw-output '.data.organization.pinnedItems.nodes[].name' | yq eval -P | sed "s/ /, /g")) + gh api -H "${HEADER}" /orgs/${array[$i]} | jq '. + + {"key1": ["maps","feed","lexer","parser","syntax","grammar"]} + + {"key2": ["'${pr[0]}'","'${pr[1]}'","'${pr[2]}'","'${pr[3]}'","'${pr[4]}'","'${pr[5]}'"]}' >> ${RUNNER_TEMP}/orgs.json + if [[ "$i" -lt "${#array[@]}-1" ]]; then echo "," >> ${RUNNER_TEMP}/orgs.json; fi + done + echo "]" >> ${RUNNER_TEMP}/orgs.json + curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function -H "Authorization: Bearer ${BEARER}" -H "Content-Type: application/json" --data @${RUNNER_TEMP}/orgs.json | jq '.' > ${RUNNER_TEMP}/gist.json + fi + + # Iterate the Structure + printf -v array_str -- ',,%q' "${array[@]}" + if [[ ! "${array_str},," =~ ",,$1,," ]]; then + SPAN=0; echo ${array[0]} + elif [[ "${array[-1]}" == "$1" ]]; then + SPAN=${#array[@]}; echo $2 | sed "s|${OWNER}.github.io|${ENTRY}.github.io|g" + if [[ -n "$CELL" ]]; then + if [[ "${ENTRY}" == "eq19" ]]; then + echo "maps, feed, lexer, parser, syntax, grammar" > ${RUNNER_TEMP}/pinned_repos + else + QUERY='{"query":"{\n organization(login: \"'${ENTRY}'\") {\n pinnedItems(first: 6, types: REPOSITORY) {\n nodes {\n ... on Repository {\n name\n }\n }\n }\n }\n}"' + curl -s -X POST "${GITHUB_GRAPHQL_URL}" -H "Authorization: bearer ${GH_TOKEN}" --data-raw "${QUERY}" | jq --raw-output '.data.organization.pinnedItems.nodes[].name' | yq eval -P | sed "s/ /, /g" > ${RUNNER_TEMP}/pinned_repos + sed -i "1s|^|maps, feed, lexer, parser, syntax, grammar, |" ${RUNNER_TEMP}/pinned_repos + fi + fi + else + for ((i=0; i < ${#array[@]}; i++)); do + if [[ "${array[$i]}" == "$1" && "$i" -lt "${#array[@]}-1" ]]; then + SPAN=$(( $i + 1 )); echo ${array[$SPAN]} + fi + done + fi + + # Generate id from the Structure + [[ -z "$SPIN" ]] && if [[ "$1" != "$2" ]]; then SPIN=0; else SPIN=13; fi + if [[ -n "$CELL" ]]; then + SPANPLUS=$(($SPAN + 1)) + if (( $CELL == 0 )); then MOD=7; else MOD=13; fi + if (( $SPANPLUS == $MOD )); then + SPANPLUS=0 + CELLPLUS=$(($CELL + 1)) + if (( $CELLPLUS == 14 )); then CELLPLUS=0; fi + else + CELLPLUS=$(($CELL + 0)) + fi + + echo "SPIN=[${CELLPLUS}, ${SPANPLUS}]" >> ${GITHUB_ENV} + echo " spin: [${CELLPLUS}, ${SPANPLUS}]" >> ${RUNNER_TEMP}/_config.yml + echo " pinned: [$(cat ${RUNNER_TEMP}/pinned_repos)]" >> ${RUNNER_TEMP}/_config.yml + echo " organization: [$(cat ${RUNNER_TEMP}/user_orgs)]" >> ${RUNNER_TEMP}/_config.yml + fi + return $(( $SPAN + $SPIN )) +} + +jekyll_build() { + + echo -e "\n$hr\nCONFIG\n$hr" + + [[ $1 == *"github.io"* ]] && OWNER=$2 + if [[ $1 != "eq19.github.io" ]]; then SITEID=$(( $3 + 2 )); else SITEID=1; fi + + if [[ "${OWNER}" == "eq19" ]]; then + sed -i "1s|^|description: An attempt to discover the Final Theory\n\n|" ${RUNNER_TEMP}/_config.yml + else + DESCRIPTION=$(gh api -H "${HEADER}" /orgs/${OWNER} --jq '.description') + sed -i "1s|^|description: ${DESCRIPTION}\n\n|" ${RUNNER_TEMP}/_config.yml + fi + + # Note: If you need to use a workflow run's URL from within a job, you can combine + # these variables: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID + sed -i "1s|^|action: ${REPO}/actions/runs/${RUN}\n|" ${RUNNER_TEMP}/_config.yml + sed -i "1s|^|repository: ${OWNER}/$1\n|" ${RUNNER_TEMP}/_config.yml + [[ $1 != *"github.io"* ]] && sed -i "1s|^|baseurl: /$1\n|" ${RUNNER_TEMP}/_config.yml + + FOLDER="span$(( 19 - $SITEID ))" + TARGET_REPOSITORY=${OWNER}/$1 + gh variable set FOLDER --body "$FOLDER" + echo 'FOLDER='${FOLDER} >> ${RUNNER_TEMP}/.env + echo 'repo='${TARGET_REPOSITORY} >> ${GITHUB_OUTPUT} + gh variable set TARGET_REPOSITORY --body "$TARGET_REPOSITORY" + echo 'TARGET_REPOSITORY='${TARGET_REPOSITORY} >> ${GITHUB_ENV} + + sed -i "1s|^|title: eQuantum\n|" ${RUNNER_TEMP}/_config.yml + sed -i "1s|^|span: ${FOLDER}\n|" ${RUNNER_TEMP}/_config.yml + sed -i "1s|^|user: ${USER}\n|" ${RUNNER_TEMP}/_config.yml + sed -i "1s|^|id: ${SITEID}\n|" ${RUNNER_TEMP}/_config.yml + + echo 'ID='${SITEID} >> ${GITHUB_ENV} + cat ${RUNNER_TEMP}/_config.yml + +} + +# Get structure on gist files +PATTERN='sort_by(.created_at)|.[] | select(.public == true).files.[] | select(.filename != "README.md").raw_url' +HEADER="Accept: application/vnd.github+json" #&& echo ${TOKEN} | gh auth login --with-token +gh api -H "${HEADER}" "/users/eq19/gists" --jq "${PATTERN}" > ${RUNNER_TEMP}/gist_files + +# Capture the string and return status +if [[ "${OWNER}" != "${USER}" ]]; then ENTRY=$(set_target ${OWNER} ${USER}); else ENTRY=$(set_target ${OWNER}); fi +CELL=$? && TARGET_REPOSITORY=$(set_target $(basename ${REPO}) ${OWNER}.github.io) +jekyll_build ${TARGET_REPOSITORY} ${ENTRY} $? diff --git a/.github/entrypoint/artifact/BosonJet.c b/.github/entrypoint/artifact/BosonJet.c new file mode 100644 index 000000000000..be74a78f4c23 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJet.c @@ -0,0 +1,326 @@ +#include "BosonJet.h" +#include +#define PI 3.141592653589793 + +BosonJet::BosonJet(){ + invGeV2topb=3.893793656233564944e8; + _nf=5;_calls=50000; +} + +BosonJet::BosonJet(double Ecm, Particle *boson){ + _boson=boson;_Ecm=Ecm;invGeV2topb=3.893793656233564944e8; + _nf=5;_calls=50000; + + //calculate some universal prefactors: + _s=_Ecm*_Ecm; + _preXsection=2.0*PI*_ew.alpha()/(_s*_s); + _mB2=_boson->m()*_boson->m(); + +} + +void BosonJet::setpT(double pT){ + _pT=pT; +} + +double BosonJet::getpT(){ + return _pT; +} + +void BosonJet::getKinematics(){ + cout << "x1=" << _x1 << ", x2=" << _x2 << ", s=" << _shat << ", t=" << _that << ", u=" << _uhat << endl; +} + +double BosonJet::M2toXsection(double pT, double x1, double x2, double Q){ + /* + This function gives the prefactor for + dsigma/dpTdyBdyJ=2*pi*pT*aEM*aS*M2/(x1*x2*s^2)*pdf1(x2)*pdf2(x2) + without pdfs and M2 and aS. + */ + return _preXsection*pT*_qcd.alphas(Q)/(x1*x2); +} + +double BosonJet::M2toXsection(double Q){ + /* + This function gives the prefactor for + dsigma/dpTdyBdyJ=2*pi*pT*aEM*aS*M2/(x1*x2*s^2)*pdf1(x2)*pdf2(x2) + without pdfs and M2 and aS. + */ + return _preXsection*_pT*_qcd.alphas(Q)/(_x1*_x2); +} + +void BosonJet::setEcm(double Ecm){ + _Ecm=Ecm;_s=_Ecm*_Ecm;_preXsection=2.0*PI*_ew.alpha()/(_Ecm*_Ecm*_Ecm*_Ecm); +} + +void BosonJet::setBoson(Particle *boson){ + _boson=boson; _mB2=_boson->m()*_boson->m(); +} + +void BosonJet::setNf(unsigned int nf){ + _nf=nf; +} + +void BosonJet::getxBs(double *xB, double pT, double yB, double yJ){ + /* + p1=x_1 (_Ecm,0,0,_Ecm)/2, p2=x_2 (_Ecm,0,0,-_Ecm)/2, + pB=(sqrt(pT^2+m^2)cosh(yB),pT cos(phiB),pT sin(phiB),sqrt(pT^2+m^2)sinh(yB)), + pJ=(sqrt(pT^2+m^2) cosh(yJ),pT cos(phiJ),pT sin(phiJ),sqrt(pT^2+m^2)sinh(yJ)). + */ + double mT=sqrt(1.0+_boson->m()*_boson->m()/(pT*pT)); + double xT=pT/_Ecm; + xB[0]=xT*(mT*exp(yB)+exp(yJ));xB[1]=xT*(mT*exp(-yB)+exp(-yJ)); +} + +void BosonJet::Mandelstam(double x1, double x2, double pT, double yB, double &s, double &t, double &u){ + /* p1=xB[0] (_Ecm, 0,0,_Ecm)/2, p2=xB[1](_Ecm,0,0,-_Ecm)/2, + pB=(sqrt(pT^2+mB^2)cosh(yB), pT cos(phiB), pT sin(phiB),sqrt(pT^2+mB^2)sinh(yB)), + */ + double mT=sqrt(_mB2+pT*pT);double EcmmT=_Ecm*mT; + s=x1*x2*_s; t=_mB2-x1*EcmmT*exp(-yB); u=_mB2-x2*EcmmT*exp(yB); +} + +void BosonJet::setKinematics(double pT, double yB, double yJ){ + _pT=pT;_yB=yB;_yJ=yJ; + double xB[2];getxBs(xB, pT, yB, yJ);_x1=xB[0];_x2=xB[1]; + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u);_shat=s;_that=t;_uhat=u; + //cout << _shat << ", " << _that << ", " << _uhat << ", " << _x1 << ", " << _x2 << endl; + //cout << _shat*_pT*_pT/(_that*_uhat) << endl; +} + +bool BosonJet::kinematicsQ(){ + return _x1>0.0&&_x1<1.0&&_x2>0.0&&_x2<1.0; +} + +/* + M^2/(g_s^2 e_q^2) aveaged and summed over the spin and color indices repectively in the intial and final states. + See Table 7.2 in page 265 in QCD and Collider Physics. +*/ + +double BosonJet::Mgq2(double s, double t, double u){ + /* + M^2/(e^2g^2) for gq->BJ. For qg->BJ, swap t and u. + */ + return -(s*s+u*u+2.0*t*(s+t+u))/(_qcd._Nc*s*u); +} + +double BosonJet::Mgq2(){ + /* + M^2/(e^2g^2) for gq->BJ. + */ + return -(_shat*_shat+_uhat*_uhat+2.0*_that*(_shat+_that+_uhat))/(_qcd._Nc*_shat*_uhat); +} + +double BosonJet::Mqg2(){ + /* + M^2/(e^2g^2) for qg->BJ. + */ + return -(_shat*_shat+_that*_that+2.0*_uhat*(_shat+_uhat+_that))/(_qcd._Nc*_shat*_that); +} + +double BosonJet::Mqqb2(double s, double t, double u){ + /*M^2/(e^2g^2) for qqb->BJ. The same for qbq->BJ. + */ + return 2.0*_qcd._CF*(t*t+u*u+2.0*s*(s+t+u))/(_qcd._Nc*t*u); +} + +double BosonJet::Mqqb2(){ + /*M^2/(e^2g^2) for qqb->BJ. The same for qbq->BJ. + */ + return 2.0*_qcd._CF*(_that*_that+_uhat*_uhat+2.0*_shat*(_shat+_that+_uhat))/(_qcd._Nc*_that*_uhat); +} + +double BosonJet::e2(double eq){ + double e2eff; + switch(_boson->id()){ + case 22: + e2eff=eq*eq;//photon + break; + case 23: + e2eff=1.0-2.0*fabs(eq)*_ew.sw2();e2eff=(e2eff*e2eff+4.0*eq*eq*_ew.sw2()*_ew.sw2())/(8.0*_ew.sw2()*_ew.cw2());//Z^0 + } + return e2eff; +} + +//differential cross sectoin at LO + +double BosonJet::sigmagqLO(double pT, double yB, double yJ, double Q){ + /* + dsigma/dpTdyZdyJ in unit pb/GeV for gq->ZJ & gqb->ZJ + */ + double res; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>1.0||xB[1]>1.0||xB[0]<0.0||xB[1]<0.0) res=0.0; + else{ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + //cout << s << ", " << t << ", " << u << endl; + double dsgq=0.0; + for(unsigned int i=1;i<=_nf;i++){ + dsgq+=(_qcd.pdf(i,xB[1],Q)+_qcd.pdf(-i,xB[1],Q))*e2(_qcd._partons[i].e());//g(p1) q(p2)->boson(pB) jet(pJ) + } + res=(_qcd.pdf(21,xB[0],Q)*dsgq*Mgq2(s,t,u))*M2toXsection(pT, xB[0], xB[1], Q); + //cout << (_qcd.pdf(21,xB[0],Q)*dsgq*Mgq2(s,t,u)) << ", " << M2toXsection(pT, xB[0], xB[1], Q) << endl; + + } + return res*invGeV2topb;//return with unit pb +} + +double BosonJet::sigmagqLO(double Qh, double QPDF){ + /* + dsigma/dpTdyZdyJ in unit pb/GeV for gq->ZJ & gqb->ZJ + */ + double res=0.0; + if(kinematicsQ()){ + double dsgq=0.0; + for(unsigned int i=1;i<=_nf;i++){ + dsgq+=(_qcd.pdf(i,_x2,QPDF)+_qcd.pdf(-i,_x2,QPDF))*e2(_qcd._partons[i].e());//g(p1) q(p2) or g(p1) qb(p2)->boson(pB) jet(pJ) + } + res=(_qcd.pdf(21,_x1,QPDF)*dsgq*Mgq2())*M2toXsection(Qh); + //cout << (_qcd.pdf(21,xB[0],Q)*dsgq*Mgq2(s,t,u)) << ", " << M2toXsection(pT, xB[0], xB[1], Q) << endl; + + } + return res*invGeV2topb;//return with unit pb +} + + +double BosonJet::sigmaqgLO(double pT, double yB, double yJ, double Q){ + /* + dsigma/dpTdyZdyJ in unit pb/GeV for qg->ZJ and qbg->ZJ + */ + double res; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>1.0||xB[1]>1.0||xB[0]<0.0||xB[1]<0.0) res=0.0; + else{ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + double dsqg=0.0; + for(unsigned int i=1;i<=_nf;i++){ + dsqg+=(_qcd.pdf(i,xB[0],Q)+_qcd.pdf(-i,xB[0],Q))*e2(_qcd._partons[i].e());//q(p1) g(p2) or qb(p1) g(p2)->boson(pB) jet(pJ) + } + res=dsqg*_qcd.pdf(21,xB[1],Q)*Mgq2(s,u,t)*M2toXsection(pT, xB[0], xB[1], Q); + } + return res*invGeV2topb; +} + +double BosonJet::sigmaqgLO(double Q,double QPDF){ + /* + dsigma/dpTdyZdyJ in unit pb/GeV for qg->ZJ and qbg->ZJ + */ + double res=0.0; + if(kinematicsQ()){ + double dsqg=0.0; + for(unsigned int i=1;i<=_nf;i++){ + dsqg+=(_qcd.pdf(i,_x1,QPDF)+_qcd.pdf(-i,_x1,QPDF))*e2(_qcd._partons[i].e());//q(p1) g(p2) or qb(p1) g(p2)->boson(pB) jet(pJ) + } + res=dsqg*_qcd.pdf(21,_x2,QPDF)*Mqg2()*M2toXsection(Q); + } + return res*invGeV2topb; +} + +double BosonJet::sigmaqqbLO(double pT, double yB, double yJ, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qqb->ZJ and qbq->ZJ + */ + double res; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>1.0||xB[1]>1.0||xB[0]<0.0||xB[1]<0.0) res=0.0; + else{ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + double ds=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //qg->Zj + ds+=_qcd.pdf(i,xB[0],Q)*_qcd.pdf(-i,xB[1],Q)*e2(_qcd._partons[i].e());//g(p1) q(p2)->boson(pB) jet(pJ) + //qbg->ZJ + ds+=_qcd.pdf(i,xB[1],Q)*_qcd.pdf(-i,xB[0],Q)*e2(_qcd._partons[i].e());//q(p1) g(p2)->boson(pB) jet(pJ) + } + res=ds*Mqqb2(s,t,u)*M2toXsection(pT, xB[0], xB[1], Q); + } + return res*invGeV2topb;//return with unit pb +} + +double BosonJet::sigmaqqbLO(double Q, double QPDF){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qqb->ZJ and qbq->ZJ + */ + double res=0.0; + if(kinematicsQ()){ + double ds=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //qqb and qb q->Zj + ds+=(_qcd.pdf(i,_x1,QPDF)*_qcd.pdf(-i,_x2,QPDF)+_qcd.pdf(i,_x2,QPDF)*_qcd.pdf(-i,_x1,QPDF))*e2(_qcd._partons[i].e()); + } + res=ds*Mqqb2()*M2toXsection(Q); + } + return res*invGeV2topb;//return with unit pb +} + +double BosonJet::sigmaLO(double pT, double yB, double yJ, double Q){ + return sigmaqgLO(pT, yB, yJ, Q)+sigmagqLO(pT, yB, yJ, Q)+sigmaqqbLO(pT, yB, yJ, Q); +} + +double BosonJet::sigmaLO(double Q, double QPDF){ + double res=0.0; + if(kinematicsQ()){ + res=sigmaqgLO(Q, QPDF)+sigmagqLO(Q, QPDF)+sigmaqqbLO(Q, QPDF); + } + return res; +} + +//MC integration + +void BosonJet::setupMC(double (*pds)(double *, size_t, void *), size_t dim, double *ymin, double *ymax, void *para){ + + gsl_rng_env_setup (); + _T = gsl_rng_default; + _r = gsl_rng_alloc (_T); + + _dim=dim;_mcFun.f=pds;_mcFun.dim=dim;_mcFun.params=para; +#ifdef MC_MISER + cout << "miser is used..." << endl; + _mcStatus = gsl_monte_miser_alloc (dim); +#else + cout << "vegas is used..." << endl; + _mcStatus = gsl_monte_vegas_alloc (dim); +#endif + _ymin=ymin;_ymax=ymax; + +} + +void BosonJet::cleanupMC(){ +#ifdef MC_MISER + gsl_monte_miser_free (_mcStatus); +#else + gsl_monte_vegas_free (_mcStatus); +#endif + gsl_rng_free (_r); + +} + +void BosonJet::setCalls(size_t calls){ + _calls=calls; +} + +void BosonJet::calculate(double qT, ostream &out){ + double res, err; +#ifdef MC_MISER + gsl_monte_miser_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res, &err); +#else + gsl_monte_vegas_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res, &err); +#endif + out << setprecision(15) << qT << " " << qT*res << " " << qT*err << endl;//here, in resum, we actually calculate dsigam/qTdqT but we only save dsigma/dqT. +} + +void BosonJet::calculate(double qT, double *res){ +#ifdef MC_MISER + gsl_monte_miser_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res[0], &res[1]); +#else + gsl_monte_vegas_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res[0], &res[1]); +#endif +} + +void BosonJet::calculate(double *res){//calculate the total cross section in pb +#ifdef MC_MISER + gsl_monte_miser_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res[0], &res[1]); +#else + gsl_monte_vegas_integrate (&_mcFun, _ymin, _ymax, _dim, _calls, _r, _mcStatus, &res[0], &res[1]); +#endif +} + diff --git a/.github/entrypoint/artifact/BosonJet.h b/.github/entrypoint/artifact/BosonJet.h new file mode 100644 index 000000000000..35c1018863b2 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJet.h @@ -0,0 +1,92 @@ +//#define MC_MISER +#include "QCD.h" +#include "ElectroWeak.h" +#include +#include +#include +#include + +using namespace std; + + +class BosonJet{ + + protected: + + unsigned int _nf;//flavors of quarks, 5 by default + + //pdfs and parameters + double _Ecm;//=sqrt{s} characterizes the collision. We always choose the CM frame. + QCD _qcd; + ElectroWeak _ew; + Particle *_boson; + + double _preXsection;//prefactor for calculting cross secton + double _s; + double _mB2;//boson mass squared + + double _x1, _x2, _yB, _yJ, _shat, _that, _uhat,_pT;//Kinematics for a scattering process + + //unit conversion 1 GeV fm=0.1973269788 from PDG 2018 + double invGeV2topb;//1/GeV^2 to pb + + //Variables for Monte Carlo + const gsl_rng_type *_T; + gsl_rng *_r; + +#ifdef MC_MISER + gsl_monte_miser_state * _mcStatus; +#else + gsl_monte_vegas_state * _mcStatus; +#endif + gsl_monte_function _mcFun; + size_t _dim, _calls; + + double *_ymin, *_ymax;//integration boundary + + public: + + BosonJet(); + BosonJet(double Ecm, Particle *boson); + + //calculte the effective charge + double e2(double); + + void setEcm(double Ecm); + void setBoson(Particle *boson); + void getxBs(double *xB, double pT, double yB, double yJ);//calculate Bjorken x's from Ecm, pT and the rapidities of the two produced particles + void Mandelstam(double x1, double x2, double pT, double yB, double &s, double &t, double &u); + void setNf(unsigned int nf); + void setKinematics(double pT, double yB, double yJ); + bool kinematicsQ();//return whether it is allowed kinematics. + void getKinematics(); + void setpT(double pT); + double getpT(); + + //Born amplitudes squared: basically the hard function + double Mgq2(double s, double t, double u);//averged amplitude squared for gq channel + double Mgq2();//averged amplitude squared for gq channel + double Mqg2();//averged amplitude squared for gq channel + double Mqqb2(double s, double t, double u);//averaged amplitude squared for qqbar channel + double Mqqb2();//averaged amplitude squared for qqbar channel + + //differential cross section at LO, used for tests + double M2toXsection(double Q);//The prefactor*M^2 gives Born cross section. + double M2toXsection(double pT, double x1, double x2, double Q);//The prefactor*M^2 gives Born cross section. + double sigmagqLO(double pT, double yB, double yJ, double Q);//differential cross section for gq channel + double sigmagqLO(double Qh, double QPDF);//differential cross section for gq+gqb channel + double sigmaqgLO(double Q, double QPDF);//differential cross section for gq+gqb channel + double sigmaqgLO(double pT, double yB, double yJ, double Q);//differential cross section for qg+qbg channel + double sigmaqqbLO(double pT, double yB, double yJ, double Q);//differential cross section for qqb+qbq channel + double sigmaqqbLO(double Q,double QPDF);//differential cross section for qq channel + double sigmaLO(double pT, double yB, double yJ, double Q); + double sigmaLO(double Q,double QPDF); + + //Monte Carlo + void setupMC(double (*pds)(double *, size_t, void *), size_t dim, double *ymin, double *ymax, void *para); + void cleanupMC();void setCalls(size_t); + void calculate(double, ostream &); + void calculate(double, double *); + void calculate(double *); +}; + diff --git a/.github/entrypoint/artifact/BosonJetBorn.c b/.github/entrypoint/artifact/BosonJetBorn.c new file mode 100644 index 000000000000..e3f6c0e89d75 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetBorn.c @@ -0,0 +1,10 @@ +#include "BosonJetBorn.h" + + +BosonJetBorn::BosonJetBorn(double Ecm, Particle *boson):BosonJet(Ecm, boson){ +} + +void BosonJetBorn::getFilename(stringstream &filename, double yJmax){ + filename << "s" << _Ecm << "etaJ" << yJmax << "Born.dat"; + +} diff --git a/.github/entrypoint/artifact/BosonJetBorn.h b/.github/entrypoint/artifact/BosonJetBorn.h new file mode 100644 index 000000000000..c3f4ce93b801 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetBorn.h @@ -0,0 +1,9 @@ +#include "BosonJet.h" + +class BosonJetBorn: public BosonJet{ + + public: + BosonJetBorn(double Ecm, Particle *boson); + void getFilename(stringstream &filename, double ymax); + +}; diff --git a/.github/entrypoint/artifact/BosonJetCSS.c b/.github/entrypoint/artifact/BosonJetCSS.c new file mode 100644 index 000000000000..90e010a45ce2 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetCSS.c @@ -0,0 +1,133 @@ +#include +#include "BosonJetCSS.h" +#include +#include +#include +#include +#include +#include + +//#define SCREEN_ONLY + +using namespace std; + +BosonJetCSS::BosonJetCSS():BosonJetResum(){ +} + +BosonJetCSS::BosonJetCSS(double Ecm, Particle *boson):BosonJetResum(Ecm, boson){ +} + + +//CSS +double BosonJetCSS::H1qqb(double muh){ + double lmuhomz2=log(muh*muh/_mB2), lRpTomuh2=_R*_pT/muh,mz2t=_mB2/(_mB2-_that),mz2u=_mB2/(_mB2-_uhat); + double lsomz2=log(_shat/_mB2),ltomz2=log(-_that/_mB2),luomz2=log(-_uhat/_mB2),lmz2t=log(mz2t),lmz2u=log(mz2u); + lRpTomuh2=2.0*log(lRpTomuh2); + return 0.5*_qcd.alphas(muh)*( _qcd._Nc*( -_qcd._beta0*lRpTomuh2/6.0 +0.5*lRpTomuh2*lRpTomuh2 + gsl_sf_dilog(mz2t)+ gsl_sf_dilog(mz2u)-lmuhomz2*log(_shat*_mB2/(_that*_uhat)) - 0.5*lmuhomz2*lmuhomz2+0.5*lsomz2*lsomz2-0.5*(ltomz2+luomz2)*(ltomz2+luomz2) + ltomz2*luomz2 + 0.5*lmz2t*lmz2t+0.5*lmz2u*lmz2u-2.0*log(_R)*log(_R)-2.0*M_PI*M_PI/3.0+67.0/9.0-23.0*_nf/54.0 ) + _qcd._CF*( 2.0*lsomz2*lmuhomz2-lmuhomz2*lmuhomz2-3.0*lmuhomz2-lsomz2*lsomz2+M_PI*M_PI-8.0 ) )/M_PI; +} + +double BosonJetCSS::H1qg(double muh){ + double lmuhomz2=log(muh*muh/_mB2), lRpTomuh2=_R*_pT/muh,mz2t=_mB2/(_mB2-_that),mz2u=_mB2/(_mB2-_uhat); + double lsomz2=log(_shat/_mB2),ltomz2=log(-_that/_mB2),luomz2=log(-_uhat/_mB2),lmz2t=log(mz2t),lmz2u=log(mz2u); + lRpTomuh2=2.0*log(lRpTomuh2); + return 0.5*_qcd.alphas(muh)*( _qcd._CF*( -1.5*lRpTomuh2 +0.5*lRpTomuh2*lRpTomuh2 + 2.0*luomz2*lmuhomz2-lmuhomz2*lmuhomz2-3.0*lmuhomz2-luomz2*luomz2-2.0*log(_R)*log(_R)-2.0*M_PI*M_PI/3.0-1.5) + _qcd._Nc*( gsl_sf_dilog(mz2t)-gsl_sf_dilog(_mB2/_shat) - lmuhomz2*log(_uhat*_mB2/(_shat*_that))-0.5*lmuhomz2*lmuhomz2-0.5*(lsomz2+ltomz2)*(lsomz2+ltomz2)+lsomz2*ltomz2-lsomz2*log(_that/(_mB2-_shat)) -0.5*lsomz2*lsomz2+0.5*lmz2t*lmz2t+0.5*luomz2*luomz2+0.5*M_PI*M_PI ) )/M_PI; +} +/* + +double BosonJetCSS::H1qg(double muh){ + double lmuhomz2=log(muh*muh/_mB2), lRpTomuh2=_R*_pT/muh,mz2t=_mB2/(_mB2-_that),mz2u=_mB2/(_mB2-_uhat); + double lsomz2=log(_shat/_mB2),ltomz2=log(-_that/_mB2),luomz2=log(-_uhat/_mB2),lmz2t=log(mz2t),lmz2u=log(mz2u); + lRpTomuh2=2.0*log(lRpTomuh2); + return 0.5*_qcd.alphas(muh)*( _qcd._CF*( -1.5*lRpTomuh2 +0.5*lRpTomuh2*lRpTomuh2 + 2.0*luomz2*lmuhomz2-lmuhomz2*lmuhomz2-3.0*lmuhomz2-luomz2*luomz2-2.0*log(_R)*log(_R)-2.0*M_PI*M_PI/3.0-1.5) + _qcd._Nc*( gsl_sf_dilog(mz2t)-gsl_sf_dilog(_mB2/_shat) - lmuhomz2*log(_uhat*_mB2/(_shat*_that))-0.5*lmuhomz2*lmuhomz2-0.5*(lsomz2+ltomz2)*(lsomz2+ltomz2)+lsomz2*ltomz2+lsomz2*log(_shat-_mB2) -0.5*lsomz2*lsomz2+0.5*lmz2t*lmz2t+0.5*luomz2*luomz2+0.5*M_PI*M_PI ) )/M_PI; +} +*/ + +double BosonJetCSS::aHqgCSS(){ + return -2.0*(_qcd._Nc+_qcd._CF)/M_PI; +} + +double BosonJetCSS::bHqgCSS(double muh){ + return (_qcd._CF*(2.0*log(-_uhat/(_pT*muh*_R))-1.5)+_qcd._Nc*log(_shat*_that/(_uhat*muh*muh))-0.5*_qcd._beta0)/M_PI; +} + +double BosonJetCSS::aHqqbCSS(){ + return _aHqqb+_aJqqb; +} + +double BosonJetCSS::bHqqbCSS(double muh){ + //cout << "s=" << _shat << endl; + return ( _qcd._CF*(2.0*log(_shat/(muh*muh))-3.0)-2.0*_qcd._Nc*log(_R) )/M_PI; +} + +double BosonJetCSS::aSqgCSS(){ + return _asbbqg+_acoftqg; +} + +double BosonJetCSS::bSqgCSS(double muxT){ + return (_qcd._CF*(1.5-2.0*log(-_uhat/(_pT*muxT*_R)))+2.0*_qcd._Nc*log(-_pT*muxT/_that)+0.5*_qcd._beta0)/M_PI; +} + +double BosonJetCSS::CHqgCSS(double mu, double muh){ + return rge(mu,muh,aHqgCSS(),bHqgCSS(muh)); +} + +double BosonJetCSS::CHqqbCSS(double mu, double muh){ + return rge(mu,muh,aHqqbCSS(),bHqqbCSS(muh)); +} + +double BosonJetCSS::CSqgCSS(double mu, double muxT){ + return rge(mu,muxT,aSqgCSS(),bSqgCSS(muxT)); +} + + + +double BosonJetCSS::sigmaqgCSS(double xT, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV for qg->ZJ+qbg->ZJ. + */ + + setKinematics(pT,yB,yJ);double res=0.0; + if(kinematicsQ()){ + double muT=muxTCSS(xT),muh=muhCSS(); + res=0.5*xT*NonPert(xT,muh)*gsl_sf_bessel_J0(_qT*xT)*CHqgCSS(muT,muh)*(1.0+H1qg(muh))*sigmaqgLO(muh,muT)/M_PI; + } + return res; +} + +double BosonJetCSS::sigmaqqbCSS(double xT, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV for qqb and qbq->ZJ+qbg->ZJ. + */ + + setKinematics(pT,yB,yJ);double res=0.0; + if(kinematicsQ()){ + double muT=muxTCSS(xT),muh=muhCSS(); + res=xT*NonPert(xT,muh)*gsl_sf_bessel_J0(_qT*xT)*CHqqbCSS(muT,muh)*(1.0+H1qqb(muh))*sigmaqqbLO(muh,muT); + } + return res; +} + +double BosonJetCSS::sigmaCSS(double xT, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV for qqb and qbq->ZJ+qbg->ZJ. + */ + + setKinematics(pT,yB,yJ);double res=0.0; + if(kinematicsQ()){ + double muT=muxTCSS(xT),muh=muhCSS(); + res=xT*NonPert(xT,muh)*gsl_sf_bessel_J0(_qT*xT)*((1.0+H1qqb(muh))*CHqqbCSS(muT,muh)*sigmaqqbLO(muh,muT)+2.0*(1.0+H1qg(muh))*CHqgCSS(muT,muh)*sigmaqgLO(muh,muT)); + } + return res; +} + +double BosonJetCSS::RGInvariance(double mu, double muxT, double muh){ + return aHqgCSS()*log(mu/muh)+bHqgCSS(muh)+aSqgCSS()*log(mu/muxT)+bSqgCSS(muxT); +} + +//filename + +void BosonJetCSS::getFilename(stringstream &filename, double ymin, double ymax){ + filename << "R" << _R << "pT" << ymin << "." << ymax; + filename << "CSS.dat"; + +} diff --git a/.github/entrypoint/artifact/BosonJetCSS.h b/.github/entrypoint/artifact/BosonJetCSS.h new file mode 100644 index 000000000000..6cf4da9336d5 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetCSS.h @@ -0,0 +1,33 @@ + +#include "BosonJetResum.h" + +class BosonJetCSS: public BosonJetResum{ + public: + BosonJetCSS(); + BosonJetCSS(double Ecm, Particle *boson); + + //CSS formalism: H=Hard+Jet, S=Soft+Coft + double H1qg(double muh); + double H1qqb(double muh); + + double aHqgCSS();//a for anomalous dimension of hard function Hqg + double bHqgCSS(double muh);//b for anomalous dimension of hard function Hqg + + double aHqqbCSS();//a for anomalous dimension of hard function Hqg + double bHqqbCSS(double muh);//b for anomalous dimension of hard function Hqg + + double aSqgCSS();//a for anomalous dimension of soft function Hqg + double bSqgCSS(double);//b for anomalous dimension of soft function Hqg + + double CHqgCSS(double mu, double muH);//nll resummed for hard function: Hqg in CSS + + double CHqqbCSS(double mu, double muH);//nll resummed for hard function: Hqg in CSS + + double CSqgCSS(double mu, double muS);//nll resummed for hard function: Hqg + double sigmaqgCSS(double xT, double pT, double yB, double yJ); + double sigmaqqbCSS(double xT, double pT, double yB, double yJ); + double sigmaCSS(double xT, double pT, double yB, double yJ); + + double RGInvariance(double mu, double muxT, double muh); + void getFilename(stringstream &filename, double ymin, double ymax); +}; diff --git a/.github/entrypoint/artifact/BosonJetLL.c b/.github/entrypoint/artifact/BosonJetLL.c new file mode 100644 index 000000000000..f125060b9b1b --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetLL.c @@ -0,0 +1,42 @@ +#include +#include "BosonJetLL.h" +#include +#include +#include +#include +#include +#include + +//#define SCREEN_ONLY + +BosonJetLL::BosonJetLL():BosonJetNLL(){ + initLL(); +} + +BosonJetLL::BosonJetLL(double Ecm, Particle *boson):BosonJetNLL(Ecm, boson){ + initLL(); +} + +void BosonJetLL::initLL(){ + _ngl=false; +} + +double BosonJetLL::bHqg(){ + cout << "We are here..." << endl; + return -(_cf*(2.0*log(-_muh*_muh/_uhat))+_nc*log(_muh*_muh*_uhat/(_shat*_that))); +} + +//filename + +void BosonJetLL::getFilename(stringstream &filename, double ymin, double ymax){ + filename << "R" << _R << "pT" << ymin << "." << ymax << "LL"; + if(_css){ + filename << "CSS"; + } + if(_nonpert){ + filename << "NP"; + } + filename <<"muh" << _muhV << "mub" << _muxTV << "muJ" << _muJV << "mut" << _mutV; + filename <<".dat"; +} + diff --git a/.github/entrypoint/artifact/BosonJetLL.h b/.github/entrypoint/artifact/BosonJetLL.h new file mode 100644 index 000000000000..f5153a6a6bcd --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetLL.h @@ -0,0 +1,23 @@ + +#include "BosonJetNLL.h" + +class BosonJetLL: public BosonJetNLL{ + +protected: + +public: + + BosonJetLL(); + BosonJetLL(double Ecm, Particle *boson); + + void initLL(); + + virtual double bHqg();//b for anomalous dimension of hard function Hqg + + + //NGL resummation + + void getFilename(stringstream &filename, double ymin, double ymax); + +}; + diff --git a/.github/entrypoint/artifact/BosonJetLO.c b/.github/entrypoint/artifact/BosonJetLO.c new file mode 100644 index 000000000000..73591a108c2a --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetLO.c @@ -0,0 +1,198 @@ +#include "BosonJetLO.h" + +#define PI 3.141592653589793 + +BosonJetLO::BosonJetLO(double Ecm, Particle *boson):BosonJet(Ecm, boson){ +} + + +double BosonJetLO::nllqg(double qT, double pT, double yB, double yJ, double R, double Q){ + /* + Double log + non-PDF single log at O(alpha_s) for qg->BJ channel. It is the same for qbg->BJ. + For gq->BJ and gqb->BJ, it is given by t<->u. + */ + double xB[2],res;getxBs(xB, pT, yB, yJ); + + if(xB[0]>1.0||xB[1]>1.0||xB[0]<0.0||xB[1]<0.0) res=0.0; + else{ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + + res=-_qcd.alphas(Q)*(2.0*(_qcd._Nc*log(-pT*qT/t)+_qcd._CF*log(-pT*qT*R/u))+( 11.0*_qcd._Nc/6.0-_nf/3.0+3.0*_qcd._CF/2.0))/(PI); + } + // + //res=-_qcd.alphas(Q)*2.0*(_qcd._Nc+_qcd._CF)*(log(0.5*qT)+0.5772156649015329)/(PI); + return res; +} + +double BosonJetLO::nllqq(double qT, double pT, double yB, double yJ, double R, double Q){ + /* + Double log + non-PDF single log at O(alpha_s) for qqb->BJ channel. It is the same for qbq->BJ. + */ + double xB[2],res;getxBs(xB, pT, yB, yJ); + if(xB[0]>1.0||xB[1]>1.0||xB[0]<0.0||xB[1]<0.0) res=0.0; + else{ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + res=-_qcd.alphas(Q)*(_qcd._CF*(3.0+2.0*log(-pT*qT/t)+2.0*log(-pT*qT/u))+2.0*_qcd._Nc*log(R))/(PI); + } + return res; +} + +double BosonJetLO::sigmaqg(double qT, double pT, double yB, double yJ, double R, double Q){ + /* + Non-PDF LL and NLL for qg+qbg channel. + */ + return sigmaqgLO(pT,yB,yJ,Q)*nllqg(qT, pT,yB,yJ,R,Q); +} + +double BosonJetLO::sigmaqg(double qT, double pT, double yB, double yJ, double z, double R, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qg->ZJ+qbg->ZJ. + */ + double res=0.0; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]<1.0&&xB[1]<1.0&&xB[0]>0.0&&xB[1]>0.0&&z>0.0&&z<1.0){//z has to be in the whole range (0,1) in order to get the delta function in splitting function + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + + double dg=0.0,dq=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //qg->q'g->BJ+qbg->qb'g + dq+=(_qcd.Pqq(i,z,xB[0],Q)+_qcd.Pqq(-i,z,xB[0],Q))*e2(_qcd._partons[i].e()); + //qg->qg'->BJ+qbg->qb'g->BJ + dg+=(_qcd.pdf(i,xB[0],Q)+_qcd.pdf(-i,xB[0],Q))*e2(_qcd._partons[i].e()); + //That is all. + } + res=_qcd.alphas(Q)*((dg*_qcd.Pgg(z,xB[1],Q)+dq*_qcd.pdf(21,xB[1],Q))*Mgq2(s,u,t)+ dg*_qcd.Pqg(z,xB[1],Q)*Mqqb2(s,t,u))*M2toXsection(pT, xB[0], xB[1], Q)/PI;//prefactor alphas/PI + } + return res*invGeV2topb+sigmaqgLO(pT,yB,yJ,Q)*nllqg(qT, pT,yB,yJ,R,Q); + //return sigmaqgLO(pT,yB,yJ,Q)*nllqg(qT, pT,yB,yJ,R,Q); +} + +double BosonJetLO::sigmaqqb(double qT, double pT, double yB, double yJ, double R, double Q){ + /* + Non-PDF LL and NLL for qqb+qbq channel. + */ + return sigmaqqbLO(pT,yB,yJ,Q)*nllqq(qT, pT,yB,yJ,R,Q); +} + +double BosonJetLO::sigmaqqb(double qT, double pT, double yB, double yJ, double z, double R, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qqb->ZJ. + */ + double res=0.0; + double xB[2];getxBs(xB, pT, yB, yJ); + //z has to be in the whole range (0,1) in order to get the delta function in splitting function + if(xB[0]<1.0&&xB[1]<1.0&&xB[0]>0.0&&xB[1]>0.0&&z>0.0&&z<1.0){ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + + double ds=0.0,dsqg=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //qqb->qqb'->BJ with ' indicating undergoing splitting + ds+=_qcd.pdf(i,xB[0],Q)*_qcd.Pqq(-i,z,xB[1],Q)*e2(_qcd._partons[i].e()); + //qqb->qg->BJ + dsqg+=_qcd.pdf(i,xB[0],Q)*_qcd.Pgqb(z,xB[1],Q)*e2(_qcd._partons[i].e()); + //qbq->qbq'->BJ + ds+=_qcd.pdf(-i,xB[0],Q)*_qcd.Pqq(i,z,xB[1],Q)*e2(_qcd._partons[i].e()); + //qbq->qbg->BJ + dsqg+=_qcd.pdf(-i,xB[0],Q)*_qcd.Pgq(z,xB[1],Q)*e2(_qcd._partons[i].e()); + //qqb->q'qb->BJ, qqb->gqb->BJ,qbq->qb'q->BJ & qbq->gq->BJ is taken into account by a factor of 2. + } + res=2.0*_qcd.alphas(Q)*( ds*Mqqb2(s,t,u)+dsqg*Mgq2(s,u,t) )*M2toXsection(pT, xB[0], xB[1], Q)/PI;//prefactor: 2 alphas/(PI) + } + + return res*invGeV2topb+sigmaqqbLO(pT,yB,yJ,Q)*nllqq(qT, pT,yB,yJ,R,Q); +} + +double BosonJetLO::sigmagg(double qT, double pT, double yB, double yJ, double z, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for gg->ZJ. + */ + double res=0.0; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>0.0&&xB[1]>0.0&&xB[0]<1.0&&xB[1]<1.0&&z<1.0&&z>xB[0]){ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + + for(unsigned int i=1;i<=_nf;i++){ + //Conly gg->qg->BJ is calculated. + //gg->qbg->BJ, gg->gq->BJ and gg->gqb is taken into account by a factor of 4. + res+=e2(_qcd._partons[i].e());//g(p1) q(p2)->boson(pB) jet(pJ) + } + res*=4.0*_qcd.alphas(Q)*_qcd.Pqg(z,xB[0],Q)*_qcd.pdf(21,xB[1],Q)*Mgq2(s,u,t)*M2toXsection(pT, xB[0], xB[1], Q)/PI;//prefactor: 4*alphas/PI + } + return res*invGeV2topb;//return with unit pb +} + + +double BosonJetLO::sigmaqq(double qT, double pT, double yB, double yJ, double z, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qq->ZJ. + */ + double res=0.0; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>0.0&&xB[1]>0.0&&xB[0]<1.0&&xB[1]<1.0&&z<1.0&&z>xB[0]){ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + + double dsqg=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //Only qq->qg->ZJ is calculated. + //qq->gq->ZJ is taken into account by a factor of 2. + dsqg+=_qcd.pdf(i,xB[0],Q)*e2(_qcd._partons[i].e()); + } + + res=2.0*_qcd.alphas(Q)*dsqg*_qcd.Pgq(z,xB[1],Q)*Mgq2(s,u,t)*M2toXsection(pT, xB[0], xB[1], Q)/PI;//prefactor: 2*alphas/(PI) + + } + return res*invGeV2topb;//return with unit pb +} + +double BosonJetLO::sigmaqbqb(double qT, double pT, double yB, double yJ, double z, double Q){ + /* + dsigma/dpTdyBdyJ in pb/GeV for qbqb->ZJ + */ + double res=0.0; + double xB[2];getxBs(xB, pT, yB, yJ); + if(xB[0]>0.0&&xB[1]>0.0&&xB[0]<1.0&&xB[1]<1.0&&z<1.0&&z>xB[0]){ + double s,t,u;Mandelstam(xB[0], xB[1], pT, yB, s, t, u); + double dsqg=0.0; + for(unsigned int i=1;i<=_nf;i++){ + //Only qb qb->qb g->ZJ is calculated. + //qbqb->gqb->ZJ is taken into account by a factor 2 + dsqg+=_qcd.pdf(-i,xB[0],Q)*e2(_qcd._partons[i].e()); + } + + res=2.0*_qcd.alphas(Q)*(dsqg*_qcd.Pgqb(z,xB[1],Q)*Mgq2(s,u,t))*M2toXsection(pT, xB[0], xB[1], Q)/PI;//prefactor: 2*alphas/PI + + } + return res*invGeV2topb;//return with unit pb +} + +double BosonJetLO::sigma(double qT, double pT, double yB, double yJ,double z, double R, double Q){ + return 2.0*sigmaqg(qT,pT,yB,yJ,z,R,Q)+sigmaqqb(qT,pT,yB,yJ,z,R,Q)+sigmaqq(qT,pT,yB,yJ,z,Q)+sigmaqbqb(qT,pT,yB,yJ,z,Q)+sigmagg(qT,pT,yB,yJ,z,Q); +} + +//output + +void BosonJetLO::getFilename(stringstream &filename, double R, int chan, double ymin, double ymax){ + filename << "R" << R << "pT" << ymin << "." << ymax; + switch(chan){ + case 0: + filename<< "tot"; + break; + case 1: + filename<< "qg"; + break; + case 2: + filename<< "qqb"; + break; + case 3: + filename<< "gg"; + break; + case 4: + filename<< "qq"; + break; + case 5: + filename<< "qbqb"; + break; + } + filename << "LO.dat"; + +} diff --git a/.github/entrypoint/artifact/BosonJetLO.h b/.github/entrypoint/artifact/BosonJetLO.h new file mode 100644 index 000000000000..ecc4333078c5 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetLO.h @@ -0,0 +1,31 @@ +#include "BosonJet.h" + +class BosonJetLO: public BosonJet{ + + public: + BosonJetLO(double Ecm, Particle *boson); + + //differential cross section at LO + double sigmaqg(double qT, double pT, double yB, double yJ, double R, double Q);//differential cross section for gq channel + double sigmaqg(double qT, double pT, double yB, double yJ,double z, double R, double Q);//differential cross section for gq channel + //differential cross section after resummation (RG evolution) + double sigmagq(double qT,double dphi, double pT, double yB, double yJ);//differential cross section for gq channel + double sigmagg(double qT,double pT, double yB, double yJ, double z,double Q);//differential cross section for gq channel + double sigmaqq(double qT,double pT, double yB, double yJ, double z,double Q);//differential cross section for gq channel + double sigmaqbqb(double qT,double pT, double yB, double yJ, double z,double Q);//differential cross section for gq channel + double sigmaqqb(double qT,double pT, double yB, double yJ, double R, double Q);//differential cross section for gq channel + double sigmaqqb(double qT,double pT, double yB, double yJ,double z, double R, double Q);//differential cross section for gq channel + double sigma(double qT,double pT, double yB, double yJ, double z, double R, double Q); + + //Fixed-order calculation + double nllqg(double pT, double qT, double yB, double yJ, double R, double Q); + double nllqq(double pT, double qT, double yB, double yJ, double R, double Q); + + //Output + void getFilename(stringstream &filename, double R, int chan, double ymin, double ymax); + +}; + +struct ParaQT{BosonJetLO *_bj;double _R;double _qT; int _chan;}; + + diff --git a/.github/entrypoint/artifact/BosonJetNLL.c b/.github/entrypoint/artifact/BosonJetNLL.c new file mode 100644 index 000000000000..9b878640a70c --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetNLL.c @@ -0,0 +1,210 @@ +#include +#include "BosonJetNLL.h" +#include +#include +#include +#include +#include +#include + +//#define SCREEN_ONLY + +BosonJetNLL::BosonJetNLL():BosonJetResum(){ + initNLL(); +} + +BosonJetNLL::BosonJetNLL(double Ecm, Particle *boson):BosonJetResum(Ecm, boson){ + initNLL(); +} + +void BosonJetNLL::initNLL(){ + _css=false;_imgcphix=true;;_ngl=true; + _angl=0.85*_qcd._Nc;_bngl=0.86*_qcd._Nc;_cngl=1.33; + _cNGL=_qcd._Nc*_qcd._CF*M_PI*M_PI/(3.0); + _muhV=1.0;_muJV=1.0;_mutV=1.0;_muxTV=1.0; + + _mutMin=10.0; + _ll=false; +} + +void BosonJetNLL::setScaleVar(double vh, double vb, double vJ, double vt){ + _muhV=vh;_muxTV=vb;_muJV=vJ;_mutV=vt; +} + +void BosonJetNLL::setCSS(bool cssQ){ + _css=cssQ; +} + +void BosonJetNLL::setNGL(bool nglQ){ + _ngl=nglQ; +} + +void BosonJetNLL::setImgcphix(bool cssQ){ + _imgcphix=cssQ; +} + +double BosonJetNLL::CHqg(){ + //cout << _mu <<", " << _muh << ", " << _pT << ", " << aHqg() << ", " << bHqg() << ", " << aHqg()*log(_mu/_muh)+bHqg() << endl; + return rge(_mu,_muh,aHqg(),bHqg()); +} + +double BosonJetNLL::CJqg(){ + return rge(_mu,_muJ,aJqg(),bJqg()); +} + +double BosonJetNLL::Ccoftqg(){ + return rge(_mu,_mut,acoftqg(),bcoftqg()); +} + +double BosonJetNLL::Csbbqg(){ + return rge(_mu,_muxT,asbbqg(),bsbbqg()); +} + +double BosonJetNLL::CHqqb(){ + return rge(_mu,_muh,aHqqb(),bHqqb()); +} + +double BosonJetNLL::CJqqb(){ + return rge(_mu,_muJ,aJqqb(),bJqqb()); +} + +double BosonJetNLL::Ccoftqqb(){ + return rge(_mu,_mut,acoftqqb(),bcoftqqb()); +} + +double BosonJetNLL::Csbbqqb(){ + return rge(_mu,_muxT,asbbqqb(),bsbbqqb()); +} + +//R resum +double BosonJetNLL::Ccphixqg(){ + return cos(2.0*M_PI*_qcd._CF*log(_qcd.alphas(_muxT)/_qcd.alphas(_mut))/_qcd._beta0); +} + +double BosonJetNLL::Ccphixqqb(){ + return cos(2.0*M_PI*_qcd._Nc*log(_qcd.alphas(_muxT)/_qcd.alphas(_mut))/_qcd._beta0); +} + +double BosonJetNLL::NGL(){ + double res=1.0; + + if(_mut<_muJ){ + double u=log(_qcd.alphas(_mut)/_qcd.alphas(_muJ))/_qcd._beta0; + double u2=u*u; + + //cout << _mut << ", " << _muJ << endl; + + //cout << u2 << ", " << exp(-_cNGL*u2*(1.0+_angl*_angl*u2)/(1.0+pow(_bngl*u,_cngl))) << endl; + + res=exp(-_cNGL*u2*(1.0+_angl*_angl*u2)/(1.0+pow(_bngl*u,_cngl))); + } + return res; +} + + +void BosonJetNLL::setScales(double xT){ + _muxT=_muxTV*_b0/xT;_muh=_muhV*muh();_muJ=_muJV*_R*_pT;_mut=_mutV*_R*_b0/xT; + _mu=_muJ; + if(_muxT>_muh) _muxT=_muh; + if(_mut>_muJ) _mut=_muJ; + if(_muxT>_pT) _muxT=_pT; + + // if(_mut<_muJ&&_muxT<_muh){ + + //CSS + if(_css){ + _mut=_muxT;_muJ=_muh;_mu=_muxT; + } +} + +double BosonJetNLL::sigmaqgR(double xT, double cphix, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV for qg->ZJ+qbg->ZJ. + */ + + setKinematics(pT,yB,yJ);double res=0.0; + if(kinematicsQ()){ + + _xT=xT; _cphix=cphix; setScales(xT); + if((_qcd.alphas(_muxT)/_qcd.alphas(_mut)>exp(-0.25*_qcd._beta0/_qcd._CF))){ + //cout << "RGI: " << RGInvariance(_mu, _muh, _muxT, _muJ, _mut, xT, cphix) << endl; + if(_mutMin>_mut) _mutMin=_mut; + //cout << _mutMin << ", " << _mut << endl; + res=xT*gsl_sf_bessel_J0(_qT*xT)*Ccphixq()*CHqg()*CJqg()*Csbbqg()*Ccoftqg()*sigmaqgLO(_muh,_muxT)/M_PI; + if(_nonpert){ + res*=NonPert(xT,_muh); + } + if(_imgcphix){ + res*=Ccphixqg(); + } + if(_ngl){ + res*=NGL(); + } + } + //} + } + return res; +} + + +double BosonJetNLL::sigmaqqbR(double xT, double cphix, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV for qqb->ZJ+qbg->ZJ. + */ + + setKinematics(pT,yB,yJ);double res=0.0; + if(kinematicsQ()){ + + _xT=xT; _cphix=cphix; setScales(xT); + //cout << "qqb: " << Ccphixqqb()*Csbbqqb() << ", " << CHqqb()*CJqqb()*Csbbqqb()*Ccoftqqb() << endl; + //cout << "mu=" << _mu << ", muxT=" << _muxT << ", muh=" << _muh << ", muJ" << _muJ << ", mut=" << _mut << endl; + // cout << exp(-0.25*_qcd._beta0/_qcd._Nc) << ", " << exp(-0.25*_qcd._beta0/_qcd._CF) << endl; + if((_qcd.alphas(_muxT)/_qcd.alphas(_mut)>exp(-0.25*_qcd._beta0/_qcd._Nc))){ + if(_mutMin>_mut) _mutMin=_mut; + res=xT*gsl_sf_bessel_J0(_qT*xT)*Ccphixg()*CHqqb()*CJqqb()*Csbbqqb()*Ccoftqqb()*sigmaqqbLO(_muh,_muxT)/M_PI; + if(_nonpert){ + res*=NonPert(xT,_muh); + } + if(_imgcphix){ + res*=Ccphixqqb(); + } + if(_ngl){ + double ngl=NGL(); + res*=pow(ngl,_qcd._Nc/_qcd._CF); + } + //} + } + } + return res; +} + +double BosonJetNLL::sigmaR(double xT, double cphix, double pT, double yB, double yJ){ + /* + dsigma/qTdqTdpTdyBdyJ in pb/GeV. + */ + + return 2.0*sigmaqgR(xT, cphix, pT, yB, yJ)+sigmaqqbR(xT, cphix, pT, yB,yJ); +} + + + +//filename + +void BosonJetNLL::getFilename(stringstream &filename, double ymin, double ymax){ + filename << "R" << _R << "pT" << ymin << "." << ymax; + if(_css){ + filename << "CSS"; + } + if(_nonpert){ + filename << "NP"; + } + if(_ngl){ + filename << "NGL"; + } + if(_ll){ + filename << "LL"; + } + filename <<"muh" << _muhV << "mub" << _muxTV << "muJ" << _muJV << "mut" << _mutV; + filename <<"s"<< _Ecm << ".dat"; +} + diff --git a/.github/entrypoint/artifact/BosonJetNLL.h b/.github/entrypoint/artifact/BosonJetNLL.h new file mode 100644 index 000000000000..7d854fadfab0 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetNLL.h @@ -0,0 +1,55 @@ + +#include "BosonJetResum.h" + +class BosonJetNLL: public BosonJetResum{ + +protected: + bool _css,_imgcphix,_ngl; + double _cNGL,_angl,_bngl,_cngl; + double _muhV,_muxTV,_muJV,_mutV; + + +public: + + BosonJetNLL(); + BosonJetNLL(double Ecm, Particle *boson); + + double _mutMin; + + void initNLL(); + void setScales(double xT);//var is a prefactor for the scale variation + void setScaleVar(double vh, double vb, double vJ, double vt); + + double CHqg();//nll resummed for hard function in qg + double CJqg();//nll resummed for Jet function in qg + double Ccoftqg();//nll resummed for Jet function in qg + double Csbbqg();//nll resummed for SBB in qg + double CHqqb();//nll resummed for hard function: Hqqb + double CJqqb();//nll resummed for Jet function in qqb + double Ccoftqqb();//nll resummed for Jet function in qqb + double Csbbqqb();//nll resummed for SBB: Hqqb + + //R resummation + double Ccphixqg(); + double Ccphixqqb(); + double sigmaqgR(double xT, double cphix, double pT, double yB, double yJ); + double sigmaqqbR(double xT,double cphix, double pT, double yB, double yJ); + double sigmaR(double xT,double cphix, double pT, double yB, double yJ); + + //NGL resummation + double NGL(); + + + void getFilename(stringstream &filename, double ymin, double ymax); + + //check css limit + void setCSS(bool cssQ); + void setImgcphix(bool icphix); + + //switch on and off NGL + void setNGL(bool nglQ); + + //print _mut min + double mutMin(); +}; + diff --git a/.github/entrypoint/artifact/BosonJetResum.c b/.github/entrypoint/artifact/BosonJetResum.c new file mode 100644 index 000000000000..3420f67c7827 --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetResum.c @@ -0,0 +1,280 @@ +#include +#include "BosonJetResum.h" +#include +#include +#include +#include +#include +#include + +//#define SCREEN_ONLY + +using namespace std; + +void BosonJetResum::init(){ + _R=0.0;_qT=0.0;_eps=1.0e-4;_b0=1.1229189671337703; + _aHqg=-2.0*(_qcd._Nc+2.0*_qcd._CF)/M_PI;_asbbqg=2.0*(_qcd._Nc+2.0*_qcd._CF)/M_PI; + _aJqg=2.0*_qcd._CF/M_PI;_acoftqg=-2.0*_qcd._CF/M_PI; + _aHqqb=-2.0*(_qcd._Nc+2.0*_qcd._CF)/M_PI;_aJqqb=2.0*_qcd._Nc/M_PI; + _acoftqqb=-2.0*_qcd._Nc/M_PI;_asbbqqb=2.0*(_qcd._Nc+2.0*_qcd._CF)/M_PI; + + //constant for RG evolution + _b2i=2.0*M_PI/_qcd._beta0;double b2ia=0.5/_qcd._beta0; + _aHqg*=b2ia;_aJqg*=b2ia;_asbbqg*=b2ia;_acoftqg*=b2ia; + _aHqqb*=b2ia;_aJqqb*=b2ia;_asbbqqb*=b2ia;_acoftqqb*=b2ia; + + _beta0=_qcd._beta0/M_PI;_nc=_qcd._Nc/M_PI;_cf=_qcd._CF/M_PI; + + _nonpert=false; +} + +BosonJetResum::BosonJetResum(){ + init(); +} + +BosonJetResum::BosonJetResum(double Ecm, Particle *boson):BosonJet(Ecm, boson){ + init(); +} + +void BosonJetResum::setNonPert(bool npQ){ + _nonpert=npQ; +} + +void BosonJetResum::setLL(bool ll){ + _ll=ll; +} + + +void BosonJetResum::setRegulator(double eps){ + _eps=eps; +} + +void BosonJetResum::setScale(double mu, double muh, double muJ, double muxT, double mut,double xT, double cphix){ + _mu=mu;_muJ=muJ;_muh=muh;_mut=mut;_muxT=muxT, _xT=xT, _cphix=cphix; +} + +double BosonJetResum::rge(double mu, double mu0, double a, double b){ + //double b2i=2.0*M_PI/_qcd._beta0;double b2ia=0.5*a/_qcd._beta0; + //return pow(_qcd.alphas(mu)/_qcd.alphas(mu0),-b2i*b)*exp( -b2ia*log(mu/mu0)*log(_qcd.alphas(mu))-b2ia*b2i*( (log(_qcd.alphas(mu0))+1.0)/_qcd.alphas(mu0)- (log(_qcd.alphas(mu))+1.0)/_qcd.alphas(mu)) ); + double r=_qcd.alphas(mu)/_qcd.alphas(mu0); + double lr=log(r); + return exp(-_b2i*b*log(r)-_b2i*a*(4.0*M_PI*(1.0- 1.0/r-lr)/_qcd.alphas(mu0) + ( _qcd._gcusp1/_qcd._gcusp0-_qcd._beta1/_qcd._beta0 )*(1.0-r+lr) + 0.5*_qcd._beta1*lr*lr/_qcd._beta0 ) ); +} + +void BosonJetResum::setR(double R){ + if(R>0.0) + _R=R; + else{ + cout << "The cone size can not be less than zero!" << endl; + throw exception(); + } +} + +void BosonJetResum::setqT(double qT){ + if(qT>0.0) + _qT=qT; + else{ + cout << "qT can not be less than zero!" << endl; + throw exception(); + } +} + +void BosonJetResum::setmu(double mu){ + if(mu>0.0) + _mu=mu; + else{ + cout << "mu can not be less than zero!" << endl; + throw exception(); + } +} + +void BosonJetResum::setPara(double R, double qT, double mu){ + setR(R);setqT(qT);setmu(mu); +} + +double BosonJetResum::regulator(double xT){ + return exp(-_eps*xT*xT); + } + +double BosonJetResum::aHqg(){ + return _aHqg; +} + +double BosonJetResum::bHqg(){ + double b; + if(_ll){ + b=-(_cf*(2.0*log(-_muh*_muh/_uhat))+_nc*log(_muh*_muh*_uhat/(_shat*_that))); + } + else{ + b=-(0.5*_beta0+_cf*(3.0+2.0*log(-_muh*_muh/_uhat))+_nc*log(_muh*_muh*_uhat/(_shat*_that))); + } + return b; +} + + +double BosonJetResum::aJqg(){ + return _aJqg; +} + +double BosonJetResum::bJqg(){ + double b; + if(_ll){ + b=_cf*(-2.0*log(_pT*_R/_muJ) ); + } + else{ + b=_cf*(1.5-2.0*log(_pT*_R/_muJ) ); + } + return b; +} + +double BosonJetResum::acoftqg(){ + return _acoftqg; +} + +double BosonJetResum::bcoftqg(){ + //return -2.0*_cf*log( 2.0*fabs(_cphix)*_xT*_mut/(_b0*_R) ); + double b; + if(_ll){ + b=-2.0*_cf*log(_xT*_mut/(_b0*_R) ); + } + else{ + b=-2.0*_cf*log( 2.0*_xT*_mut/(_b0*_R) ); + } + return b; +} + +double BosonJetResum::asbbqg(){ + return _asbbqg; +} + +double BosonJetResum::bsbbqg(){ + //return (0.5*_beta0+_cf*(1.5+2.0*log(-2.0*fabs(_cphix)*_pT*_xT*_muxT*_muxT/(_b0*_uhat)) )-2.0*_nc*log(-_that/(_pT*_muxT)) ); + double b; + if(_ll){ + b=(_cf*(2.0*log(-_pT*_xT*_muxT*_muxT/(_b0*_uhat)) )-2.0*_nc*log(-_that/(_pT*_muxT)) ); + } + else{ + b=(0.5*_beta0+_cf*(1.5+2.0*log(-2.0*_pT*_xT*_muxT*_muxT/(_b0*_uhat)) )-2.0*_nc*log(-_that/(_pT*_muxT)) ); + } + return b; +} + +double BosonJetResum::aHqqb(){ + return _aHqqb; +} + +double BosonJetResum::bHqqb(){ + double b; + if(_ll){ + b=(2.0*_nc*log(_pT/_muh)-_cf*(2.0*log(_muh*_muh/_shat))); + } + else{ + b=(2.0*_nc*log(_pT/_muh)-0.5*_beta0-_cf*(3.0+2.0*log(_muh*_muh/_shat))); + } + return b; +} + +double BosonJetResum::aJqqb(){ + return _aJqqb; +} + +double BosonJetResum::bJqqb(){ + double b; + if(_ll){ + b=(-2.0*_nc*log(_pT*_R/_muJ) ); + } + else{ + b=( 0.5*_beta0-2.0*_nc*log(_pT*_R/_muJ) ); + } + return b; + +} + + +double BosonJetResum::acoftqqb(){ + return _acoftqqb; +} + +double BosonJetResum::bcoftqqb(){ + //return -2.0*_nc*log( 2.0*fabs(_cphix)*_xT*_mut/(_b0*_R) ); + double b; + if(_ll){ + b=-2.0*_nc*log(_xT*_mut/(_b0*_R) ); + } + else{ + b=-2.0*_nc*log( 2.0*_xT*_mut/(_b0*_R) ); + } + return b; +} + + +double BosonJetResum::asbbqqb(){ + return _asbbqqb; +} + +double BosonJetResum::bsbbqqb(){ + //return (2.0*_nc*log(2.0*fabs(_cphix)*_xT*_muxT/(_b0))+_cf*(3.0+2.0*log(_muxT*_muxT/_shat)) ); + double b; + if(_ll){ + b=(2.0*_nc*log(_xT*_muxT/(_b0))+_cf*(2.0*log(_muxT*_muxT/_shat)) ); + } + else{ + b=(2.0*_nc*log(2.0*_xT*_muxT/(_b0))+_cf*(3.0+2.0*log(_muxT*_muxT/_shat)) ); + } + return b; +} + +double BosonJetResum::Ccphixq(){ + double pre=4.0*_qcd._CF/_qcd._beta0; + return exp(pre*log(fabs(_cphix))*log(_qcd.alphas(_muxT)/_qcd.alphas(_mut)) ); +} + +double BosonJetResum::Ccphixg(){ + double pre=4.0*_qcd._Nc/_qcd._beta0; + return exp(pre*log(fabs(_cphix))*log(_qcd.alphas(_muxT)/_qcd.alphas(_mut)) ); +} + +double BosonJetResum::muxT(double xT){ + return 1.1229189671337703/xT;//muxT=2.0*exp(-gamma_E)/xT +} + +double BosonJetResum::RGInvariance(double mu, double muh, double muxT, double muJ, double mut, double xT, double cphix){ + double b2ia=0.5/_qcd._beta0; + _mu=mu;_muh=muh; _muxT=muxT; _muJ=muJ; _mut=mut;_xT=xT;_cphix=cphix; + //return aHqg()+asbbqg()+aJqg()+acoftqg(); + //return aHqqb()+asbbqqb()+aJqqb()+acoftqqb(); + cout << aHqg()*log(_mu/_muh)/b2ia+bHqg() << " vs " << (2.0*_cf+_nc)*log(_shat/(_mu*_mu))+_cf*log(_uhat*_uhat/(_pT*_pT*_shat))+_nc*log(_that*_that/(_pT*_pT*_shat))+_cf*log(_pT*_pT/(_shat)) << endl; + cout << asbbqg()*log(_mu/_muxT)/b2ia+bsbbqg() << " vs " << -2.0*_cf*log(-_uhat/(_mu*_pT))-2.0*_nc*log(-_that/(_mu*_pT))+ 2.0*_cf*log(_xT*_mu/_b0)<< endl; + cout << aJqg()*log(_mu/_muJ)/b2ia+bJqg() << " vs " <<-2.0*_cf*log(_pT*_R/_mu) << endl; + cout << acoftqg()*log(_mu/_mut)/b2ia+bcoftqg() << " vs " << 2.0*_cf*log(_R*_b0/(_mu*_xT))<< endl; + cout << aHqqb()*log(_mu/_muh)/b2ia+bHqqb() << " vs " << (2.0*_cf+_nc)*log(_shat/(_mu*_mu))+_cf*log(_uhat*_uhat/(_pT*_pT*_shat))+_cf*log(_that*_that/(_pT*_pT*_shat))+_nc*log(_pT*_pT/(_shat)) << endl; + cout << asbbqqb()*log(_mu/_muxT)/b2ia+bsbbqqb() << " vs " << -2.0*_cf*log(-_uhat/(_mu*_pT))-2.0*_cf*log(-_that/(_mu*_pT))+ 2.0*_nc*log(_xT*_mu/_b0) << endl; + cout << aJqqb()*log(_mu/_muJ)/b2ia+bJqqb() << " vs " <<-2.0*_nc*log(_pT*_R/_mu) << endl; + cout << acoftqqb()*log(_mu/_mut)/b2ia+bcoftqqb() << " vs " << 2.0*_nc*log(_R*_b0/(_mu*_xT)) << endl; + return aHqg()*log(_mu/_muh)/b2ia+bHqg()+asbbqg()*log(_mu/_muxT)/b2ia+bsbbqg()+aJqg()*log(_mu/_muJ)/b2ia+bJqg()+acoftqg()*log(_mu/_mut)/b2ia+bcoftqg()+aHqqb()*log(_mu/_muh)/b2ia+bHqqb()+asbbqqb()*log(_mu/_muxT)/b2ia+bsbbqqb()+aJqqb()*log(_mu/_muJ)/b2ia+bJqqb()+acoftqqb()*log(_mu/_mut)/b2ia+bcoftqqb(); + //return aHqqb()*log(_mu/_muh)/b2ia+bHqqb()+asbbqqb()*log(_mu/_muxT)/b2ia+bsbbqqb()+aJqqb()*log(_mu/_muJ)/b2ia+bJqqb()+acoftqqb()*log(_mu/_mut)/b2ia+bcoftqqb(); +} + +double BosonJetResum::muxTCSS(double xT){ + return muxT(xT)*sqrt(1.0+xT*xT/2.25); +} + +double BosonJetResum::muhCSS(){ + //cout << _mB2 << ", " << _pT << endl; + return sqrt(_mB2+_pT*_pT)+_pT; +} + +double BosonJetResum::muh(){ + //cout << _mB2 << ", " << _pT << endl; + //return (sqrt(_mB2)+7.0*_pT)/6.0; + return sqrt(_mB2+_pT*_pT); +} + +double BosonJetResum::muJ(){ + return _R*_pT; +} + +double BosonJetResum::NonPert(double xT, double Q){ + double bmax=1.5; + return exp(-0.21*xT*xT-0.42*log(Q/1.5491933384829668)*log(1.0+xT*xT/(bmax*bmax))); +} diff --git a/.github/entrypoint/artifact/BosonJetResum.h b/.github/entrypoint/artifact/BosonJetResum.h new file mode 100644 index 000000000000..0cd11554343b --- /dev/null +++ b/.github/entrypoint/artifact/BosonJetResum.h @@ -0,0 +1,78 @@ + +#include "BosonJet.h" + +class BosonJetResum: public BosonJet{ + +protected: + + double _R; + double _qT; + double _mu, _muJ, _muh, _muxT, _cphix, _mut, _xT, _b0; + double _aHqg, _aJqg, _asbbqg, _acoftqg; + double _aHqqb, _aJqqb, _asbbqqb, _acoftqqb; + double _eps;//regulator for xT integration + + double _b2i, _beta0, _nc, _cf;//constant for RG evolution + + bool _nonpert; + bool _ll; + + void init(); + +public: + + BosonJetResum(); + BosonJetResum(double Ecm, Particle *boson); + + void setR(double R); + void setqT(double qT); + void setmu(double mu); + void setPara(double R, double qT, double mu); + void setRegulator(double eps); + void setScale(double mu, double muh, double muJ, double muxT, double mut,double xT, double cphix); + void setNonPert(bool); + void setLL(bool); + + double rge(double mu, double mu0, double a, double b);//gives the solution of RG equation + + double aHqg();//a for anomalous dimension of hard function Hqg + double bHqg();//b for anomalous dimension of hard function Hqg + + double aJqg();//a for anomalous dimension of Jet function in qg + double bJqg();//b for anomalous dimension of Jet function in qg + + double acoftqg();//a for coft function in qg + double bcoftqg(); + + double asbbqg();//a for anomalous dimension of soft+beams in qg->BJ + double bsbbqg();//b for anomalous dimension 0of soft+beams in qg->BJ + + double aHqqb();//a for anomalous dimension of hard function Hqqb + double bHqqb();//b for anomalous dimension of hard function Hqqb + + double aJqqb();//a for anomalous dimension of Jet function in qqb + double bJqqb();//b for anomalous dimension of Jet function in qqb + + double acoftqqb();//a for coft function in qqb + double bcoftqqb();//a for coft function in qqb + + double asbbqqb();//a for anomalous dimension of soft+beams in qqb->BJ + double bsbbqqb();//b for anomalous dimension of soft+beams in qqb->BJ + + double Ccphixq(); + double Ccphixg(); + + double RGInvariance(double mu, double muh, double muxT, double muJ, double mut, double xT, double cphix); + + double muxT(double xT); + double muh(); + double muxTCSS(double xT); + double muhCSS(); + double muJ(); + + //Mathematical tricks + + double NonPert(double xT, double Q); + double regulator(double xT); +}; + diff --git a/.github/entrypoint/artifact/ElectroWeak.c b/.github/entrypoint/artifact/ElectroWeak.c new file mode 100644 index 000000000000..540fdecf48b5 --- /dev/null +++ b/.github/entrypoint/artifact/ElectroWeak.c @@ -0,0 +1,27 @@ +#include "ElectroWeak.h" +#include +#define PI 3.141592653589793 + + +/*ElectroWeak::ElectroWeak(){ + //Here, all the values are taken from PDG 2018 + _GF=1.1663787e-5;//in GeV^{-2} + _mZ=91.1876;//in GeV + _mW=80.379;//in GeV + + //gmu scheme shall be used by taking the above three parameters as an input + _cw2=_mW/_mZ; _cw2*=_cw2;_sw2=1.0-_cw2;_alpha=_mW*_mW*sqrt(2.0)*_GF*_sw2/PI; +} +*/ + + +ElectroWeak::ElectroWeak(){ + //Here, all the values are taken from PDG 2018 + //_GF=1.1663787e-5;//in GeV^{-2} + _mZ=91.1876;//in GeV + _mW=80.398;//in GeV + + //gmu scheme shall be used by taking the above three parameters as an input + _sw2=0.22264585;_cw2=1-_sw2;_alpha=1.0/132.33843228; +} + diff --git a/.github/entrypoint/artifact/ElectroWeak.h b/.github/entrypoint/artifact/ElectroWeak.h new file mode 100644 index 000000000000..e2854e7c131f --- /dev/null +++ b/.github/entrypoint/artifact/ElectroWeak.h @@ -0,0 +1,20 @@ + +class ElectroWeak{ + private: + + double _mZ;//Z boson mass in GeV + double _mW;//W boson mass in GeV + double _GF;//G_F in GeV^{-2} + + double _cw2;//cos(theta_W)^2 + double _sw2;//sin(theta_W)^2 + double _alpha;//elecromagnetic alpha + + public: + ElectroWeak(); + //ElectroWeak~(); + double cw2(){return _cw2;} + double sw2(){return _sw2;} + double alpha(){return _alpha;} +}; + diff --git a/.github/entrypoint/artifact/Makefile b/.github/entrypoint/artifact/Makefile new file mode 100644 index 000000000000..248ccc31d34e --- /dev/null +++ b/.github/entrypoint/artifact/Makefile @@ -0,0 +1,99 @@ +CC=g++ +IPDF=-I/Users/biwu/Software/LHAPDF/include +LPDF=-L/Users/biwu/Software/LHAPDF/lib -lLHAPDF +#IGSL=-I/usr/local/include/ +#LGSL=-L/usr/local/lib -lgsl -lgslcblas -lm + +#IPDF=-I/home/bwu/Softwares/LHAPDF/include +#LPDF=-L/home/bwu/Softwares/LHAPDF/lib -lLHAPDF +IGSL=-I/usr/local/include/ +LGSL=-L/usr/local/lib -lgsl -lgslcblas -lm + + +All: resum + +Particle.o: Particle.c Particle.h + $(CC) -Wall -c Particle.c + +QCD.o: QCD.c QCD.h + $(CC) -Wall $(IPDF) -c QCD.c + +ElectroWeak.o: ElectroWeak.c ElectroWeak.h + $(CC) -Wall -c ElectroWeak.c + +BosonJet.o: BosonJet.c BosonJet.h + $(CC) -Wall $(IPDF) -c BosonJet.c + +#Born cross section + +BosonJetBorn.o: BosonJetBorn.c BosonJetBorn.h + $(CC) -Wall $(IPDF) -c BosonJetBorn.c + +born.o: born.c + $(CC) -Wall $(IPDF) $(IGSL) -c born.c + +born: born.o BosonJetBorn.o BosonJet.o QCD.o ElectroWeak.o Particle.o + $(CC) -Wall born.o BosonJetBorn.o BosonJet.o ElectroWeak.o QCD.o Particle.o $(LPDF) $(LGSL) -o born + +#properties of qcd + +test_qcd.o: test_qcd.c + $(CC) -Wall $(IPDF) -c test_qcd.c + +test_qcd: test_qcd.o QCD.o Particle.o + $(CC) -Wall test_qcd.o QCD.o Particle.o $(LPDF) -o test_qcd + +#Fixed order result at LO (O(alphas)) + +BosonJetLO.o: BosonJetLO.c BosonJetLO.h + $(CC) -Wall $(IPDF) -c BosonJetLO.c + +main.o: main.c + $(CC) -Wall $(IPDF) $(IGSL) -c main.c + +LO: main.o BosonJetLO.o BosonJet.o QCD.o ElectroWeak.o Particle.o + $(CC) -Wall main.o BosonJetLO.o BosonJet.o ElectroWeak.o QCD.o Particle.o $(LPDF) $(LGSL) -o LO + +#Resummed result + +BosonJetResum.o: BosonJetResum.c BosonJetResum.h + $(CC) -Wall $(IPDF) -c BosonJetResum.c + +#css + +BosonJetCSS.o: BosonJetCSS.c BosonJetCSS.h + $(CC) -Wall $(IPDF) -c BosonJetCSS.c + +css.o: css.c + $(CC) -Wall $(IPDF) $(IGSL) -c css.c + +css: css.o BosonJetCSS.o BosonJetResum.o BosonJet.o QCD.o ElectroWeak.o Particle.o + $(CC) -Wall css.o BosonJetCSS.o BosonJetResum.o BosonJet.o ElectroWeak.o QCD.o Particle.o $(LPDF) $(LGSL) -o css + +#R resummed + +BosonJetNLL.o: BosonJetNLL.c BosonJetNLL.h + $(CC) -Wall $(IPDF) -c BosonJetNLL.c + +resum.o: resum.c + $(CC) -Wall $(IPDF) $(IGSL) -c resum.c + +resum: resum.o BosonJetNLL.o BosonJetResum.o BosonJet.o QCD.o ElectroWeak.o Particle.o + $(CC) -Wall resum.o BosonJetNLL.o BosonJetResum.o BosonJet.o ElectroWeak.o QCD.o Particle.o $(LPDF) $(LGSL) -o resum + +#tot + +BosonJetNLL.o: BosonJetNLL.c BosonJetNLL.h + $(CC) -Wall $(IPDF) -c BosonJetNLL.c + +tot.o: tot.c + $(CC) -Wall $(IPDF) $(IGSL) -c tot.c + +tot: tot.o BosonJetNLL.o BosonJetResum.o BosonJet.o QCD.o ElectroWeak.o Particle.o + $(CC) -Wall tot.o BosonJetNLL.o BosonJetResum.o BosonJet.o ElectroWeak.o QCD.o Particle.o $(LPDF) $(LGSL) -o tot + + +clean: + rm *.o + rm resum + rm tot diff --git a/.github/entrypoint/artifact/Particle.c b/.github/entrypoint/artifact/Particle.c new file mode 100644 index 000000000000..66b0439df297 --- /dev/null +++ b/.github/entrypoint/artifact/Particle.c @@ -0,0 +1,30 @@ +#include +#include "Particle.h" + +Particle::Particle(){ + _m=0.0;_e=0.0;_name="UFO";_id=-273; +} + +Particle::Particle(string name, int id, double m, double e){ + _name=name;_id=id;_m=m;_e=e; +} + +void Particle::setParticle(string name, int id, double m, double e){ + _name=name;_id=id;_m=m;_e=e; +} + + +double Particle::m(){ + return _m; +} + +int Particle::id(){return _id;} +double Particle::e(){return _e;} +string Particle::name(){return _name;} + +/* +int main(){ + Particle pl; + return 0; +} +*/ diff --git a/.github/entrypoint/artifact/Particle.h b/.github/entrypoint/artifact/Particle.h new file mode 100644 index 000000000000..753979e97cf5 --- /dev/null +++ b/.github/entrypoint/artifact/Particle.h @@ -0,0 +1,26 @@ +#include + +using namespace std; + +class Particle{ + private: + + string _name;//particle name + double _m;//particle mass + double _e;//electric charge in the unit of the elementary charge e + int _id;//particle id according to PDF particle codes + + + public: + + enum _type {d=1,u=2,s=3,c=4,b=5,t=6,g=21,photon=22,Z0=23,Wp=24}; + + Particle(); + Particle(string name, int id, double m,double e); + //~Particle(){}; + void setParticle(string name, int id, double m,double e); + double m(); + int id(); + double e(); + string name(); +}; diff --git a/.github/entrypoint/artifact/QCD.c b/.github/entrypoint/artifact/QCD.c new file mode 100644 index 000000000000..2dab63d6263c --- /dev/null +++ b/.github/entrypoint/artifact/QCD.c @@ -0,0 +1,122 @@ +#include +#include "QCD.h" +#include + +using namespace std; + +QCD::QCD(unsigned int nf){ + /*Here, one needs to download the pdfset from + https://lhapdf.hepforge.org/pdfsets.html + and put it into the folder which can be found out by lhapdf-config --datadir. + */ + _nf=nf; + init(); +} + +QCD::QCD(){ + /*Here, one needs to download the pdfset from + https://lhapdf.hepforge.org/pdfsets.html + and put it into the folder which can be found out by lhapdf-config --datadir. + */ + _nf=5; + init(); +} + +void QCD::init(){ + const int imem = 0; + _Nc=3.0;_CF=4.0/3.0; + _beta0=11.0*_Nc/3.0 - 2.0*_nf/3.0; + _beta1=34.0*_Nc*_Nc/3.0-10.0*_Nc*_nf/3.0-2.0*_CF*_nf; + _gcusp0=4.0;_gcusp1=(268.0/9.0-4.0*M_PI*M_PI/3.0)*_Nc-40.0*_nf/9.0; + _pdf = mkPDF("CT14nlo", imem); + _partons=new Particle[7]; + _partons[0].setParticle("g",21, 0.0, 0.0); + _partons[1].setParticle("d",1, 0.0, -1.0/3.0); + _partons[2].setParticle("u",2, 0.0, 2.0/3.0); + _partons[3].setParticle("s",3, 0.0, -1.0/3.0); + _partons[4].setParticle("c",4, 0.0, 2.0/3.0); + _partons[5].setParticle("b",5, 0.0, -1.0/3.0); + _partons[6].setParticle("t",6,173.0, 2.0/3.0); + +} + +QCD::~QCD(){ + delete _pdf; + delete [] _partons; +} + +double QCD::pdf(int f,double x,double Q){ + return _pdf->xfxQ(f,x,Q)/x; +} + +double QCD::alphas(double Q){ + return _pdf->alphasQ(Q); +} + +double QCD::Pqg(double z){ + double res=0.0; + if(z>0&&z<=1.0) + res=z*z-z+0.5; + return res; +} + +double QCD::Pqg(double z,double x, double Q){ + double res=0.0; + if(z<1.0&&z>x) + res=Pqg(z)*pdf(21,x/z,Q)/z; + return res; +} + +double QCD::Pgq(double z){ + double res=0.0; + if(z>0&&z<1.0) + res=_CF*(z-2.0+2.0/z); + return res; +} + +double QCD::Pgq(double z,double x, double Q){ + double res=0.0; + if(z<1.0&&z>x){ + for(unsigned int i=1;i<=_nf;i++){ + res+=pdf(i,x/z,Q); + } + res*=Pgq(z)/z; + } + return res; +} + +double QCD::Pgqb(double z,double x, double Q){ + double res=0.0; + if(z<1.0&&z>x){ + for(unsigned int i=1;i<=_nf;i++){ + res+=pdf(-i,x/z,Q); + } + res*=Pgq(z)/z; + } + return res; +} + +double QCD::Pqq(int flavor, double z,double x, double Q){ + double res=0.0; + if(z<1.0&&z>0.0){ + double qsumx=pdf(flavor,x,Q); + res=1.5*qsumx;//Here, we have to make sure the range for z integration is from 0 to 1 + double qsum=0.0; + if(z>x) qsum=pdf(flavor,x/z,Q); + res+=((1.0+z*z)*qsum/z-2.0*qsumx)/((1.0-z)); + res*=_CF; + } + return res; +} + +double QCD::Pgg(double z, double x, double Q){ + double res=0.0; + if(z<1.0&&z>0.0){ + res=(11.0*_Nc/6.0-_nf/3.0)*pdf(21,x,Q);//Here, we have to make sure the range for z integration is from 0 to 1 + if(z>x) + res+=2.0*_Nc*( ( z*z-z+1.0 )*(z*z-z+1.0)*pdf(21,x/z,Q)/(z*z) - pdf(21,x,Q) )/(1.0-z); + else + res+=-2.0*_Nc*pdf(21,x,Q)/(1.0-z); + } + return res; +} diff --git a/.github/entrypoint/artifact/QCD.h b/.github/entrypoint/artifact/QCD.h new file mode 100644 index 000000000000..728065f03573 --- /dev/null +++ b/.github/entrypoint/artifact/QCD.h @@ -0,0 +1,39 @@ +#include "Particle.h" +#include "LHAPDF/LHAPDF.h" + +using namespace LHAPDF; + +/*namespace colorfactor{ + double _Nc=3.0; + double _CF=4.0/3.0; +} +*/ + +class QCD{ + private: + const PDF* _pdf; + unsigned int _nf;//flavors of quarks, 5 by default + + public: + + double _Nc, _CF;//number of colors + //const static double _Nc=3.0, _CF=4.0/3.0;//number of colors + double _beta0, _beta1, _gcusp0, _gcusp1;//_gcusp=gamma^cusp + Particle *_partons; + QCD(); + QCD(unsigned int); + ~QCD(); + void init(); + + double pdf(int,double x,double Q);//PDF as a function of flavor, x and Q + double alphas(double Q);//the strong coupling alpha_s as a function of Q + + double Pqg(double z); + double Pqg(double z, double x, double Q); + double Pgq(double z); + double Pgq(double z, double x, double Q); + double Pgqb(double z, double x, double Q); + double Pgg(double z, double x, double Q); + double Pqq(int flavor, double z, double x, double Q); + +}; diff --git a/.github/entrypoint/artifact/README.md b/.github/entrypoint/artifact/README.md new file mode 100644 index 000000000000..b189dc632d01 --- /dev/null +++ b/.github/entrypoint/artifact/README.md @@ -0,0 +1,3 @@ +# SM +C++ code for hadron colliders. +Published papers using this code: JHEP 1911 (2019) 025. diff --git a/.github/entrypoint/artifact/StandardModel.h b/.github/entrypoint/artifact/StandardModel.h new file mode 100644 index 000000000000..5874957cc56e --- /dev/null +++ b/.github/entrypoint/artifact/StandardModel.h @@ -0,0 +1,19 @@ +/* + This class is used for parameters in ElectroWeak theory + Created on January 5 2019 by Bin Wu. +*/ + +class ElectroWeak{ + private: + + double _mZ;//Z boson mass + double _mW;//W boson mass + double _GF;//Fermi coupling constant G_F + double _sw2;//sin(theta_W)^2 + double _cw2;//cos(theta_W)^2 + double _alpha;//electromagnetic coupling + + public: + ElectroWeak(); + //~ElectroWeak(); +}; diff --git a/.github/entrypoint/artifact/born.c b/.github/entrypoint/artifact/born.c new file mode 100644 index 000000000000..3a4f2c4b0baa --- /dev/null +++ b/.github/entrypoint/artifact/born.c @@ -0,0 +1,52 @@ +#include "BosonJetBorn.h" +#include +#include +#include +#include + +using namespace std; + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={yB,yJ}; + BosonJetBorn *bj=(BosonJetBorn*)p; + bj->setKinematics(bj->getpT(),y[0],y[1]); + return bj->sigmaLO(bj->getpT(),bj->getpT()); +} + + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetBorn ZJ(1.3e4,&Z0); + + //range of the integration: from ymin to ymax + double ymin[]={-2.4,-2.4}; + double ymax[]={2.4,2.4}; + + int dim=2; + size_t calls = 50000; + ZJ.setupMC( &dsigmadqT, dim, ymin,ymax, &ZJ); + ZJ.setCalls(calls); + //output file: + + stringstream filename; + ZJ.getFilename(filename, ymax[1]); + cout << filename.str() << endl; + ofstream f(filename.str(), ofstream::out | ofstream::app); + + double pT[]={35.5, 50., 71., 100.5, 143., 194., 260., 350.}; + for(int i=0;i<21;i++){ + ZJ.setpT(pT[i]); + ZJ.calculate(pT[i],f); + //cout << (0.5772156649015329+log(0.5*qT[i]))/(M_PI*qT[i]*qT[i]) << endl; + } + + f << "\n\n"; + f.close(); + + ZJ.cleanupMC(); + + return 0; +} diff --git a/.github/entrypoint/artifact/css.c b/.github/entrypoint/artifact/css.c new file mode 100644 index 000000000000..2fcb491d04b9 --- /dev/null +++ b/.github/entrypoint/artifact/css.c @@ -0,0 +1,72 @@ +#include "BosonJetCSS.h" +#include +#include +#include +#include + +using namespace std; + + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={xT, pT, yB,yJ}; + BosonJetCSS *bj=(BosonJetCSS *)p; + return bj->sigmaCSS(y[0],y[1],y[2],y[3]); +} + + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetCSS ZJ(1.3e4,&Z0); + + double R=0.2;ZJ.setR(R); + + /* + double pT=30.0,yB=0.1,yJ=0.2,muh=100.0,mu=10.0,muJ=10.0,xT=0.2,qT=1.0;ZJ.setqT(qT); + ZJ.setKinematics(pT,yB,yJ);ZJ.getKinematics(); + ZJ.setScale(mu,muJ,muh); + cout << setprecision(15) << ZJ. RGInvariance(0.1,0.2,0.3,0.4,0.5,0.6,0.7) << endl; + cout << setprecision(15) << ZJ. RGInvariance(1.1,3.2,5.3,9.4,1.5,0.6,0.7) << endl; + + */ + //range of the integration: from ymin to ymax + double ymin[]={0.0,200.0,-10.0,-2.4}; + double ymax[]={3.0,6500.0,10.0,2.4}; + + int dim=4; + size_t calls = 50000; + ZJ.setupMC( &dsigmadqT, dim, ymin,ymax, &ZJ); + ZJ.setCalls(calls); + //output file: + + stringstream filename; + ZJ.getFilename(filename, ymin[1],ymax[1]); + cout << filename.str() << endl; + ofstream f(filename.str(), ofstream::out | ofstream::app); + + + double qT[]={1., 1.1273, 1.27082, 1.4326, 1.61497, 1.82056, 2.05233, 2.3136, + 2.60813, 2.94016, 3.31445, 3.7364, 4.21206, 4.74827, 5.35275, + 6.03418, 6.80235, 7.66832, 8.64453, 9.74502, 10.9856, 12.3841, + 13.9607, 15.7379, 17.7414, 20., 22.5461, 25.4163, 28.6519, 32.2994, + 36.4113, 41.0466, 46.272, 52.1626, 58.8032, 66.2891, 74.728, 84.2412, + 94.9654, 107.055, 120.684, 136.047, 153.366, 172.891, 194.9, 219.712, + 247.682, 279.213, 314.759, 354.829, 400.}; + /*{1., 1.25893, 1.58489, 1.99526, 2.51189, 3.16228, 3.98107, 5.01187, + 6.30957, 7.94328, 10., 12.5893, 15.8489, 19.9526, 25.1189, 31.6228, + 39.8107, 50.1187, 63.0957, 79.4328, 100.};*/ + for(int i=0;i<51;i++){ + ZJ.setqT(qT[i]); + ZJ.calculate(qT[i],f); + //cout << (0.5772156649015329+log(0.5*qT[i]))/(M_PI*qT[i]*qT[i]) << endl; + } + + f << "\n\n"; + f.close(); + + ZJ.cleanupMC(); + + return 0; +} diff --git a/.github/entrypoint/artifact/main.c b/.github/entrypoint/artifact/main.c new file mode 100644 index 000000000000..d33c17e1dafe --- /dev/null +++ b/.github/entrypoint/artifact/main.c @@ -0,0 +1,114 @@ +#include "BosonJetLO.h" +#include +#include +#include +#include +#include +#include +#include + +//#define SCREEN_ONLY + +using namespace std; + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={pT, yB,yJ,z}; + //parameter list p: BosonJet, qT, R, chan + struct ParaQT *para=(struct ParaQT *)p; + double res; + switch(para->_chan){ + case 0: + res=para->_bj->sigma(para->_qT, y[0],y[1],y[2],y[3],para->_R,y[0]); + break; + case 1: + res=para->_bj->sigmaqg(para->_qT, y[0],y[1],y[2],y[3],para->_R,y[0]); + break; + case 2: + res=para->_bj->sigmaqqb(para->_qT, y[0],y[1],y[2],y[3],para->_R,y[0]); + break; + case 3: + res=para->_bj->sigmagg(para->_qT, y[0],y[1],y[2],y[3],y[0]); + break; + case 4: + res=para->_bj->sigmaqq(para->_qT, y[0],y[1],y[2],y[3],y[0]); + break; + case 5: + res=para->_bj->sigmaqbqb(para->_qT, y[0],y[1],y[2],y[3],y[0]); + break; + } + return res; +} + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetLO ZJ(1.3e4,&Z0); + //double qT=0.1, pT=500.0, yB=0.5, yJ=0.1,z=0.9999; + //cout << ZJ.sigmaqqb(qT, pT, yB, yJ, z, 0.01, pT) << endl; + + //range of the integration: from ymin to ymax + double ymin[]={30.0,-10.0,-2.4,0.0}; + double ymax[]={6500.0,10.0,2.4,1.0}; + + //integration + struct ParaQT para; + para._bj=&ZJ; para._R=0.2;para._chan=0; + + int dim=4; + size_t calls = 50000; + ZJ.setupMC( &dsigmadqT, dim, ymin,ymax, ¶); + ZJ.setCalls(calls); + //output file: + + stringstream filename; + ZJ.getFilename(filename, para._R, para._chan, ymin[0], ymax[0]); + cout << filename.str() << endl; + ofstream f(filename.str(), ofstream::out | ofstream::app); + + double qT[]={1., 1.1273, 1.27082, 1.4326, 1.61497, 1.82056, 2.05233, 2.3136, + 2.60813, 2.94016, 3.31445, 3.7364, 4.21206, 4.74827, 5.35275, + 6.03418, 6.80235, 7.66832, 8.64453, 9.74502, 10.9856, 12.3841, + 13.9607, 15.7379, 17.7414, 20., 22.5461, 25.4163, 28.6519, 32.2994, + 36.4113, 41.0466, 46.272, 52.1626, 58.8032, 66.2891, 74.728, 84.2412, + 94.9654, 107.055, 120.684, 136.047, 153.366, 172.891, 194.9, 219.712, + 247.682, 279.213, 314.759, 354.829, 400.};/*{0.106101, 0.119047, 0.133573, 0.149872, 0.168159, 0.188677, + 0.211699, 0.23753, 0.266513, 0.299033, 0.335521, 0.37646, 0.422395, + 0.473935, 0.531764, 0.596649, 0.669452, 0.751137, 0.84279, 0.945625, + 1.06101, 1.19047, 1.33573, 1.49872, 1.68159, 1.88677, 2.11699, + 2.3753, 2.66513, 2.99033, 3.35521, 3.7646, 4.22395, 4.73935, 5.31764, + 5.96649, 6.69452, 7.51137, 8.4279, 9.45625}; + {1., 1.25893, 1.58489, 1.99526, 2.51189, 3.16228, 3.98107, 5.01187, + 6.30957, 7.94328, 10., 12.5893, 15.8489, 19.9526, 25.1189, 31.6228, + 39.8107, 50.1187, 63.0957, 79.4328, 100.};*/ + for(int i=0;i<51;i++){ + para._qT=qT[i]; + if(para._chan>5){ + int chan=para._chan; + double res[2]; + f << qT[i]; + for(para._chan=0;para._chan<=5;para._chan++){ + //cout << para._chan << endl; + ZJ.calculate(qT[i],res); + f << " " << res[0] << " " << res[1]; + } + f << endl; + para._chan=chan; + } + else{ + double res[2]; + f << qT[i]; + //cout << para._chan << endl; + ZJ.calculate(qT[i],res); + f << " " << res[0]/qT[i] << " " << res[1]/qT[i] << endl;//save as dsigma/dqT + } + } + + f << "\n\n"; + f.close(); + + ZJ.cleanupMC(); + + return 0; +} diff --git a/.github/entrypoint/artifact/nllR.c b/.github/entrypoint/artifact/nllR.c new file mode 100644 index 000000000000..6f9628c2b349 --- /dev/null +++ b/.github/entrypoint/artifact/nllR.c @@ -0,0 +1,67 @@ +#include "BosonJetResum.h" +#include +#include +#include +#include + +using namespace std; + +struct ParaQT{BosonJetResum *_bj; int _chan;}; + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={xT, phi, pT, yB,yJ}; + BosonJetResum *bj=(BosonJetResum *)p; + double cphix=cos(y[1]); + return bj->sigmaqqbR(y[0],cphix,y[2],y[3],y[4]); +} + + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetResum ZJ(1.3e4,&Z0); + + double R=0.4;ZJ.setR(R); + + /* + double pT=30.0,yB=0.1,yJ=0.2,muh=100.0,mu=10.0,muJ=10.0,xT=0.2,qT=1.0;ZJ.setqT(qT); + ZJ.setKinematics(pT,yB,yJ);ZJ.getKinematics(); + ZJ.setScale(mu,muJ,muh); + cout << setprecision(15) << ZJ. RGInvariance(0.1,0.2,0.3,0.4,0.5,0.6,0.7) << endl; + cout << setprecision(15) << ZJ. RGInvariance(1.1,3.2,5.3,9.4,1.5,0.6,0.7) << endl; + + */ + //range of the integration: from ymin to ymax + double ymin[]={0.0,-0.5*M_PI,30.0,-10.0,-2.4}; + double ymax[]={3.0,0.5*M_PI,6500.0,10.0,2.4}; + + int dim=5; + size_t calls = 500000; + ZJ.setupMC( &dsigmadqT, dim, ymin,ymax, &ZJ); + ZJ.setCalls(calls); + //output file: + + stringstream filename; + ZJ.getFilename(filename, ymin[2],ymax[2]); + cout << filename.str() << endl; + ofstream f(filename.str(), ofstream::out | ofstream::app); + + + double qT[]={1., 1.25893, 1.58489, 1.99526, 2.51189, 3.16228, 3.98107, 5.01187, + 6.30957, 7.94328, 10., 12.5893, 15.8489, 19.9526, 25.1189, 31.6228, + 39.8107, 50.1187, 63.0957, 79.4328, 100.}; + for(int i=0;i<21;i++){ + ZJ.setqT(qT[i]); + ZJ.calculate(qT[i],f); + //cout << (0.5772156649015329+log(0.5*qT[i]))/(M_PI*qT[i]*qT[i]) << endl; + } + + f << "\n\n"; + f.close(); + + ZJ.cleanupMC(); + + return 0; +} diff --git a/.github/entrypoint/artifact/python/LICENSE b/.github/entrypoint/artifact/python/LICENSE new file mode 100644 index 000000000000..c4f387f15487 --- /dev/null +++ b/.github/entrypoint/artifact/python/LICENSE @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2022 Neural Field Transformations For Lattice QCD + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/.github/entrypoint/artifact/python/README.md b/.github/entrypoint/artifact/python/README.md new file mode 100644 index 000000000000..37ef8dc43b50 --- /dev/null +++ b/.github/entrypoint/artifact/python/README.md @@ -0,0 +1,32 @@ +# lgt +Collection of objects / models for Lattice Gauge Theory + +## Install +`python3 -m pip install lgt`
+`python3 -m pip install -e /path/to/this/repo` + +## Organization +``` +📁 lgt/ +├──📁 group/ +│   ├── generators.py +│   └── group.py +└── 📁 lattice/ + ├──📁 su3/ + │   ├──📁 numpy/ + │   │   └── lattice.py + │   ├──📁 pytorch/ + │   │   └── __init__.py + │   └──📁 tensorflow/ + │   └── lattice.py + ├──📁 sun/ + │   ├── logm.py + │   └── sun.py + └──📁 u1/ + ├──📁 numpy/ + │   └── lattice.py + ├──📁 pytorch/ + │   └── lattice.py + └──📁 tensorflow/ + └── lattice.py +``` diff --git a/.github/entrypoint/artifact/python/colour_matrices.py b/.github/entrypoint/artifact/python/colour_matrices.py new file mode 100644 index 000000000000..160ddba47f1b --- /dev/null +++ b/.github/entrypoint/artifact/python/colour_matrices.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Small program to explore colour matrices and structure constants in QCD. + +Written by Gavin Salam as an illustration for lecture 1 of the QCD +course at the 2024 European School of High-Energy Physics. +""" +import numpy as np + +def main(): + + print("Warning: the A,B,C indices here run from 0 to 7, not 1 to 8 as in the lecture notes.") + print("Warning: the a,b,c indices here run from 0 to 2, not 1 to 3 as in the lecture notes.") + + nF = 3 # number of degrees of freedom in Fundamental representation (quarks) + nA = 8 # number of degrees of freedom in Adjoint representation (gluos) + lambdas = get_lambdas() + ts = [0.5*l for l in lambdas] + + #--------------------------------------------- + header("check Tr(t^A t^B) = T_R delta^{AB}") + for A in range(nA): + for B in range(nA): + trace = np.trace(np.matmul(ts[A],ts[B])) + if abs(trace)>1e-10: print(f"{A=} {B=}: Trace(t^A t^B)={trace}") + + #--------------------------------------------- + header("check Sum_A(t_{ab}^A t_{bc}^A) = C_F \\delta_{ac}") + sum = np.zeros((3,3),dtype=complex) + for A in range(nA): + sum += np.matmul(ts[A],ts[A]) + for a in range(nF): + for c in range(nF): + print(f"{a=} {c=}: Sum_A(t_{{ab}}^A t_{{bc}}^A)={sum[a][c]}") + + #--------------------------------------------- + header("Examine the non-zero fABCs (A 1e-10: print(f"{A=} {B=} {result=}") + +def header(title): + print() + print("-"*len(title)) + print(title) + print("-"*len(title)) + +def non_zero_entries(arr,imin, tag="i"): + res = [] + for i in range(imin, arr.shape[0]): + if abs(arr[i]) > 1e-10: res.append(f"{tag}={i}:{arr[i]}") + return ", ".join(res) + +def get_fABCs(ts): + "returns the fABC array" + + # get the ts supplemented by a unit array, which is needed to + # complete the basis below that we will use in solving for the fABCs + ts_plus_unit = np.array([t.flatten() + for t in ts+[np.diag(np.array([1,1,1],dtype=complex))] ]) + + nA = len(ts) + fABC = np.zeros((nA,nA,nA)) + for A in range(nA): + for B in range(nA): + tt = (np.matmul(ts[A],ts[B]) - np.matmul(ts[B],ts[A]))/complex(0,1) + ff = np.linalg.solve(ts_plus_unit.transpose(), tt.flatten()) + if abs(ff[nA]) > 1e-10: raise("illegal 9th component in nA") + fABC[A,B,:] = np.real(ff[:nA]) + #if B <= A: continue + #print(f"{A=} {B=}, fABC={fABC[A,B,:]}") + + return fABC + + +def get_lambdas(): + """Return the Gell-Mann matrices as a list of 3x3 numpy arrays.""" + i = complex(0,1) + lambdas = [0]*8 + lambdas[0] = [[ 0, 1, 0], + [ 1, 0, 0], + [ 0, 0, 0]] + + lambdas[1] = [[ 0,-i, 0], + [ i, 0, 0], + [ 0, 0, 0]] + + lambdas[2] = [[ 1, 0, 0], + [ 0,-1, 0], + [ 0, 0, 0]] + + lambdas[3] = [[ 0, 0, 1], + [ 0, 0, 0], + [ 1, 0, 0]] + + lambdas[4] = [[ 0, 0,-i], + [ 0, 0, 0], + [ i, 0, 0]] + + lambdas[5] = [[ 0, 0, 0], + [ 0, 0, 1], + [ 0, 1, 0]] + + lambdas[6] = [[ 0, 0, 0], + [ 0, 0,-i], + [ 0, i, 0]] + + lambdas[7] = [[ 1.0/np.sqrt(3), 0, 0], + [ 0, 1.0/np.sqrt(3), 0], + [ 0, 0,-2.0/np.sqrt(3)]] + + return [np.array(l,dtype=complex) for l in lambdas] + +if __name__ == "__main__": + main() diff --git a/.github/entrypoint/artifact/python/gellmann.ipynb b/.github/entrypoint/artifact/python/gellmann.ipynb new file mode 100644 index 000000000000..c772c82a5256 --- /dev/null +++ b/.github/entrypoint/artifact/python/gellmann.ipynb @@ -0,0 +1,815 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "7L2WbBt5uNag" + }, + "source": [ + "# 1. Preliminaries" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 85 + }, + "colab_type": "code", + "executionInfo": { + "elapsed": 5024, + "status": "ok", + "timestamp": 1584469489951, + "user": { + "displayName": "mahkame salimi", + "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhXCkBWY2tBzsPtIAaeDyn3ng2gbWea3ReVrZhe=s64", + "userId": "11757140415455670736" + }, + "user_tz": -210 + }, + "id": "lHgp6b1NuNat", + "outputId": "472e7f3d-53e9-4c23-99a5-bb712c0be181" + }, + "outputs": [], + "source": [ + "##!pip3 install qutip\n", + "import qutip as qp\n", + "import numpy as np\n", + "from scipy.spatial import ConvexHull, Delaunay\n", + "import pandas as pd\n", + "from random import random, choice, randint\n", + "##from multiprocessing import Pool\n", + "##from concurrent.futures import ThreadPoolExecutor" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "CkRTqRe-uNa_" + }, + "source": [ + "# 2. Gell_Mann matrix generator function" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "Xaqtqo1quNbD" + }, + "source": [ + "this function will generate the gellmann matrix of dimension d.\n", + "\n", + "the function \"gellmann(j,k,d)\" will generate the jth_kth gellmann matrix of dimension d. so for every d dimension we have d^2 gellmann matrix. one of them is I, so the number of gellmann matrix that is not trivial is (d^2 - 1)." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "wjlZ7FERuNbH" + }, + "outputs": [], + "source": [ + "\"\"\"Generate generalized Gell-Mann matrices.\n", + " .. module:: gellmann.py\n", + " :synopsis: Generate generalized Gell-Mann matrices\n", + " .. moduleauthor:: Jonathan Gross \n", + "\"\"\"\n", + "from itertools import product\n", + "\n", + "def gellmann(j, k, d):\n", + " r\"\"\"Returns a generalized Gell-Mann matrix of dimension d.\n", + " \n", + " According to the convention in *Bloch Vectors for Qubits* by Bertlmann and\n", + " Krammer (2008), returns :math:`\\Lambda^j` for :math:`1\\leq j=k\\leq d-1`,\n", + " :math:`\\Lambda^{kj}_s` for :math:`1\\leq k k:\n", + " gjkd = np.zeros((d, d), dtype=np.complex128)\n", + " gjkd[j - 1][k - 1] = 1\n", + " gjkd[k - 1][j - 1] = 1\n", + " elif k > j:\n", + " gjkd = np.zeros((d, d), dtype=np.complex128)\n", + " gjkd[j - 1][k - 1] = -1.j\n", + " gjkd[k - 1][j - 1] = 1.j\n", + " elif j == k and j < d:\n", + " gjkd = np.sqrt(2/(j*(j + 1)))*np.diag([1 + 0.j if n <= j\n", + " else (-j + 0.j if n == (j + 1)\n", + " else 0 + 0.j)\n", + " for n in range(1, d + 1)])\n", + " else:\n", + " gjkd = np.diag([1 + 0.j for n in range(1, d + 1)])\n", + "\n", + " return gjkd\n", + "\n", + "def get_basis(d):\n", + " r\"\"\"Return a basis of operators.\n", + " \n", + " The basis is made up of orthogonal Hermitian operators on a Hilbert space\n", + " of dimension d, with the identity element in the last place.\n", + " Parameters\n", + " ----------\n", + " d : int\n", + " The dimension of the Hilbert space.\n", + " Returns\n", + " -------\n", + " list of numpy.array\n", + " The basis of operators.\n", + " \"\"\"\n", + " return [gellmann(j, k, d) for j, k in product(range(1, d + 1), repeat=2)]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "1yB7X5nruNbU" + }, + "source": [ + "in this cell the gellmann matrix of 3 dimension will generate and they will save in a dictionary and labeled them by numbers 0,1,...,8 ." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "trodoU6zuNbY" + }, + "outputs": [], + "source": [ + "def gellmann_basis(d):\n", + " \"\"\" generalization gellmann matrice \n", + " input : dimesion of system\n", + " output: dictionary of gellmann matrices of dimension 'd'\n", + " \"\"\"\n", + " \n", + " gellmann_basis = {} ## a dictionary that saved the gellmann matrices\n", + "\n", + " k=0\n", + " for i in range(1,d+1):\n", + " for j in range(1,d+1):\n", + " gellmann_basis.update({\"G{}\".format(k):gellmann(i,j,d)})\n", + " k+=1\n", + " \n", + " return gellmann_basis\n", + "\n", + "#gellmann_basis(3)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "K4Ck7uDYuNbl" + }, + "source": [ + "# 3. Coefficients of gellmann's expansion" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "_sL95KUXuNbp" + }, + "source": [ + "at this cell a random density matrix will generate by qutip library.\n", + "\n", + "for calculating the coefficients of gellmann expansion we do this iteration:\n", + "1. calculate the tensor product of gellmann matrices for every coefficient\n", + "2. calculate the dot product of \"ro\" and the \"gellmann's tensor product\"\n", + "3. calculate the trace of this matrix and divide it into 4\n", + "4. append it to our index list" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "2Upf2jQDuNbt" + }, + "outputs": [], + "source": [ + "def coef_of_gellmann(ro, d_sub1, d_sub2):\n", + " \"\"\" coefficient of gellmann expansion \n", + " input : density matrix and dimensions of two subsystem\n", + " output: dictionary of coefficient\n", + " \"\"\"\n", + " \n", + " coef = {} ## the coefficient's dictionary (suitable format for saving as a data)\n", + " gellmann_basis_d_sub1 = gellmann_basis(d_sub1) ## the gellmann matrix for a d_sub1 dimension system\n", + " gellmann_basis_d_sub2 = gellmann_basis(d_sub2) ## the gellmann matrix for a d_sub2 dimension system\n", + " \n", + " n1 = d_sub1**2\n", + " n2 = d_sub2**2\n", + " for i in range(n1):\n", + " for j in range(n2):\n", + " tensorP = np.kron(gellmann_basis_d_sub1[\"G{}\".format(i)],gellmann_basis_d_sub2[\"G{}\".format(j)]) ## Tensor product of gelllmann matrices\n", + " c = np.trace(np.dot(ro,tensorP))/4 ## dot product and trace\n", + " c = c.real ## all of gellmann coefs is a real\n", + " c = round(c,8) \n", + " coef.update({\"a{}{}\".format(i,j):c}) ## update the coef dict by a[i][j] and its gellmann coef for g[i][j]\n", + "\n", + " index = np.array(coef) ## change list to np.array() object\n", + " return coef" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "vfa0qkq6uNb6" + }, + "source": [ + "this cell is just a test of this algotithm in 3 dimension bipatite systems." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "uA-NxF1vuNcA" + }, + "outputs": [], + "source": [ + "N = 3\n", + "\n", + "ro = qp.rand_dm(N*N,0.5,dims=[[N,N], [N,N]])\n", + "#print(coef_of_gellmann(ro))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "jSoxsPBiuNcJ" + }, + "source": [ + "# 4. Generate entangled states (PPT)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this cell we define a PPT funcion that check positive partial transpose." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "VWGoxF5EuNcL" + }, + "outputs": [], + "source": [ + "def PPT(ro, d_sub1, d_sub2):\n", + " \"\"\" positive partial transpose \n", + " input : density matrix and the dimension of subsystem 1 and subsystem 2\n", + " output: dictionary of coefficient of gellmann matrices and label 1 or 2 for entangeled states and unknown respectivly\n", + " \"\"\"\n", + " \n", + " ro_pt = qp.partial_transpose(ro,[0,1]) ## partial transpose in subsystem 2\n", + " eig = ro_pt.eigenstates() ## calculate the eigenvalues and eigenstates\n", + " eigv = [round(i,8) for i in eig[0]] \n", + " \n", + " result = coef_of_gellmann(ro, d_sub1, d_sub2)\n", + " \n", + " if (eigv[0]<0 ):\n", + " result.update({\"label\":1}) ## the entangled states that labeled by 1\n", + " return result\n", + " else :\n", + " result.update({\"label\":2}) ## the unknown states that labeled by 2\n", + " return result\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 5. Generate separable states (Convex combination)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "in this cell we define two function that:\n", + "1. Convex_Combination: this function is give a matrix list and return a convex combination of this matrices\n", + "2. generate_ran_den_list: this function is generate a list of random density matrices" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "def Convex_Combination(matrix_list1, matrix_list2, index=10):\n", + " \"\"\" Convex Combination \n", + " input : two list of matrices that you want to build a convex combination of them. index is the number of iteration\n", + " output: a state that is a convex combination of input\n", + " \"\"\"\n", + " \n", + " coef = np.random.dirichlet(np.ones(index), size=1) ## generating list of numbers that is positive and its sum is 1 with length equal to 'index'\n", + " \n", + " seprable_state = qp.Qobj() ## build an empty Qobj for cycle\n", + " for i in range(index):\n", + " a = choice(matrix_list1) ## choice a density matrix randomly\n", + " b = choice(matrix_list2) ## choice a density matrix randomly\n", + " \n", + " tensorP = np.kron(a,b) ## Tensor product of density matrices\n", + " tensorP *= round(coef[0][i],8) ## product of tensorP and a coef that come from the coef list\n", + " \n", + " seprable_state += tensorP ## at the last its a convex combination of tensor product of density matrices\n", + " \n", + " return seprable_state\n", + "\n", + "\n", + "def generate_ran_den_list(d, length= 100):\n", + " \"\"\" generate random density matrix list \n", + " input : dimension of system and the lenght of list\n", + " output: list of random density matrix with dimension 'd'\n", + " \"\"\"\n", + " \n", + " result = []\n", + " for i in range(length):\n", + " density = round(random(),4) ## for build a random number between 0 and 1\n", + "\n", + " rand = qp.rand_dm(d, density) ## generate a random density matrix with dimension N\n", + " result.append(rand) ## appending this random density matrix to ran_den_mat list \n", + " \n", + " return result" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 6. Convex hull approximation method" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Making a Convex Hull for a 2_qutrit system and checking for new points in the convex Hull" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [], + "source": [ + "d_sub1 = 3\n", + "d_sub2 = 3\n", + "num_extreme_points = 83\n", + "index = np.zeros((1,81))\n", + "\n", + "for i in range(num_extreme_points):\n", + " ro = qp.rand_dm(d_sub1 * d_sub2, pure=True)\n", + "\n", + " coef = list(coef_of_gellmann(ro, d_sub1, d_sub2).values())\n", + " coef = np.reshape(np.array(coef),(1,81))\n", + " index = np.vstack((index,coef))\n", + "#print(np.shape(index[1:,]))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Using different functions to build convex hull" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "#generating convex hull from extreme points(Pure separable states)\n", + "hull = Delaunay(index)\n", + "#hull = ConvexHull(index)\n", + "#print(index)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The function that checks if a point is in a convex hull" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [], + "source": [ + "#This function checks if a point is in the convex hull or not\n", + "def in_hull(p, hull):\n", + " \"\"\"\n", + " Test if points in `p` are in `hull`\n", + "\n", + " `p` should be a `NxK` coordinates of `N` points in `K` dimensions\n", + " `hull` is either a scipy.spatial.Delaunay object or the `MxK` array of the \n", + " coordinates of `M` points in `K`dimensions for which Delaunay triangulation\n", + " will be computed\n", + " \"\"\"\n", + " from scipy.spatial import Delaunay\n", + " if not isinstance(hull,Delaunay):\n", + " hull = Delaunay(hull)\n", + "\n", + " return hull.find_simplex(p)>=0" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Test" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "metadata": {}, + "outputs": [], + "source": [ + "ro = qp.rand_dm(9) ## generate the random density matrix\n", + "\n", + "coef = list(coef_of_gellmann(ro,3,3).values())\n", + "coef = np.reshape(np.array(coef),(1,81)) \n", + "\n", + "#print(coef)\n", + "print(in_hull(index[20:,],index))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Making a Convex Hull for a 2_qubit system and checking for new points in the convex Hull." + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [], + "source": [ + "d_sub1 = 2\n", + "d_sub2 = 2\n", + "num_extreme_points = 20\n", + "index = np.zeros((1,16))\n", + "\n", + "for i in range(num_extreme_points):\n", + " ro = qp.rand_dm(d_sub1 * d_sub2, pure=True)\n", + " \n", + " coef = list(coef_of_gellmann(ro, d_sub1, d_sub2).values())\n", + " coef = np.reshape(np.array(coef),(1,16))\n", + " index = np.vstack((index,coef))\n", + "#print(np.real(index[1:,]))" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "hull = ConvexHull(index[:,])\n", + "in_hull(index[2,],index)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 7. Generate data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "in this cell we build 100 000 random states and labeled that. after that we build a data frame for saving and using in future." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "colab": {}, + "colab_type": "code", + "id": "5kfhME1RuNcT" + }, + "outputs": [], + "source": [ + "%%time\n", + "\n", + "states={} #states\n", + "d_sub1 = 3 #dimension of subsystem 1\n", + "d_sub2 = 3 #dimension of subsystem 2\n", + "R1 = 50000 # the number of entangled states\n", + "\n", + "##generate the entangled states\n", + "for i in range (1, R1+1):\n", + " density = round(random(),4) ## for build a random number between 0 and 1\n", + " \n", + " ro = qp.rand_dm(d_sub1*d_sub2, density, dims=[[d_sub1,d_sub2], [d_sub1,d_sub2]]) #generating random density matrix with a random density\n", + " states.update({\"{}\".format(i):PPT(ro, d_sub1, d_sub2)}) # update the states by its number and label and gellmann coefs\n", + "\n", + "\n", + "##generate the separable states\n", + "R2 = 50000 # the number of seperable states\n", + "ran_den_mat1 = generate_ran_den_list(d_sub1, 100) ## a list of random density matrix for a system\n", + "ran_den_mat2 = generate_ran_den_list(d_sub2, 100) ## a list of random density matrix for a system\n", + "\n", + "for i in range(R1+1, R1+R2+1):\n", + " index = randint(10,100) ## build a integer number between 10 and 100 for using in function\n", + " sep = Convex_Combination(ran_den_mat1, ran_den_mat2, index) ## generate a random seprable density matrix\n", + " \n", + " sep_coef = coef_of_gellmann(sep, d_sub1, d_sub2) ## calculate the gellmann coefs\n", + " sep_coef.update({\"label\":0}) ## labeled seprable states by 0\n", + " \n", + " states.update({\"{}\".format(i):sep_coef}) ## update states\n", + " \n", + " \n", + "df = pd.DataFrame(data=states).T # build a data frame by pandas\n", + "\n", + "df.to_csv('data_of_states.csv.zip') #for saving the data in zip format\n", + "#df.to_excel('data_of_states.xlsx') #for saving tha data in an excel file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "in this cell we save our data frame as a zip file" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 163 + }, + "colab_type": "code", + "executionInfo": { + "elapsed": 1231, + "status": "error", + "timestamp": 1584469470718, + "user": { + "displayName": "mahkame salimi", + "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GhXCkBWY2tBzsPtIAaeDyn3ng2gbWea3ReVrZhe=s64", + "userId": "11757140415455670736" + }, + "user_tz": -210 + }, + "id": "rt63ussFuNcc", + "outputId": "dae4bdb2-205b-4bbc-b702-de4e82d70511" + }, + "outputs": [], + "source": [ + "df.to_csv('data_of_states.csv.zip')\n", + "##df.to_csv('data_of_states.csv')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "in the blow cell we define a function that merge all of our data to a one data." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "def merge(list_data):\n", + " N = len(list_data) #the number of datas\n", + " n = [len(i.index) for i in list_data] # length of index of every data\n", + " \n", + " new_datas = [list_data[0]]\n", + " index = {}\n", + " k = n[0]\n", + " for i in range(1,N):\n", + " last_index, new_index = [i for i in range(1,n[i]+1)], []\n", + " for j in range(1+k, n[i]+1+k):\n", + " new_index.append(j)\n", + " \n", + " index = {last_index[p]:new_index[p] for p in range(n[i])}\n", + " df = list_data[i]\n", + " df = df.rename(index = index)\n", + " new_datas.append(df)\n", + " k += n[i]\n", + " \n", + " result_data = pd.concat(new_datas)\n", + " return result_data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "this function dont true work because of 'memory error'.\n", + "\n", + "so we dont merge datas and just fix their index for future." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "at this cell we read the unfixed datas. after that we drop the 'a88' column, because it's same for all." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "#df1 = pd.read_csv('data/data_of_states1.csv.zip', index_col=0, compression = 'zip')\n", + "#df2 = pd.read_csv('data/data_of_states2.csv.zip', index_col=0, compression = 'zip')\n", + "#df3 = pd.read_csv('data/data_of_states3.csv.zip', index_col=0, compression = 'zip')\n", + "#df4 = pd.read_csv('data/data_of_states4.csv.zip', index_col=0, compression = 'zip')\n", + "#df5 = pd.read_csv('data/data_of_states5.csv.zip', index_col=0, compression = 'zip')\n", + "#df6 = pd.read_csv('data/data_of_states6.csv.zip', index_col=0, compression = 'zip')\n", + "#df7 = pd.read_csv('data/data_of_states7.csv.zip', index_col=0, compression = 'zip')\n", + "#df8 = pd.read_csv('data/data_of_states8.csv.zip', index_col=0, compression = 'zip')\n", + "#df9 = pd.read_csv('data/data_of_states_100.csv.zip', index_col=0, compression = 'zip')\n", + "#df10 = pd.read_csv('data/data_of_states_200.csv.zip', index_col=0, compression = 'zip')\n", + "\n", + "df = df.drop(columns=['a88'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "in this cell we fixed (rename) the index." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "k = 1000000\n", + "\n", + "a = [i for i in range(1,100000+1)]\n", + "b = [i for i in range(k+1,k+100000+1)]\n", + "c = {a[i]:b[i] for i in range(len(a))}\n", + "\n", + "df = df.rename(index = c)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [], + "source": [ + "#df10.to_csv('fixed_data/data_of_states_10(100 k).csv.zip')\n", + "#df1.to_csv('fixed_data/data_of_states_2(100 k).csv.zip')\n", + "#df2.to_csv('fixed_data/data_of_states_3(100 k).csv.zip')\n", + "#df3.to_csv('fixed_data/data_of_states_4(100 k).csv.zip')\n", + "#df4.to_csv('fixed_data/data_of_states_5(100 k).csv.zip')\n", + "#df5.to_csv('fixed_data/data_of_states_6(100 k).csv.zip')\n", + "#df6.to_csv('fixed_data/data_of_states_7(100 k).csv.zip')\n", + "#df7.to_csv('fixed_data/data_of_states_8(100 k).csv.zip')\n", + "#df8.to_csv('fixed_data/data_of_states_9(100 k).csv.zip')\n", + "#df9.to_csv('fixed_data/data_of_states_10(100 k).csv.zip')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# 8. Correlation of data" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "the plot shows the correlation of 100000 random states" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAHfCAYAAAD6N2lbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xu8HVV99/HP9+RCggQJIEoJNyGoaBFM4Gkfq4CgptYGqaKhxYYnaAoCsVAvUCgiaBvAFmnlURG52KpcNQaFIAqp1QqGe+QSEymXCAoSFHySQC6/549ZR/bZZ5/b7NmzL/N9v17zOnvPzJq19jknWWdm1vqOIgIzMzNrrb52N8DMzKwK3OGamZmVwB2umZlZCdzhmpmZlcAdrpmZWQnc4ZqZmZXAHa6ZmVWSpFmSVkhaJemUBtuPlbRc0t2Sfihp75ptp6ZyKyS9fVT1eR6umZlVjaRxwM+AtwKrgWXAkRFxf80+W0fEs+n1bOBDETErdbxfBw4A/gD4HrBXRGwark6f4ZqZWRUdAKyKiIci4gXgCuCw2h36O9vkJUD/GephwBUR8XxE/A+wKh1vWIV2uJJmpNPvVZL+VZLS+m0l3SRpZfo6tch6zczMxmgn4LGa96vTugEkHS/p58C5wIKxlK03PndTG/s8MB+4FbgemAXcAJwCfD8iFqbr5KcAHx/uQCv/5O0DrnW/4oZrCm6qWWMbxk0Y8H7Cpg1taolVSf3vHXTe796UKVPUiuPW/39flL1+9N2/IeuT+l0UERel140+y6B2RMSFwIWS/hI4HZg72rL1cne4khYBOwOTgAuA64CtI+LHaftXgHeRdbiHAQelopcDSxmhwzUzM2tG6lwvGmLzarI+rN804PFhDncF2UllnrJAc5eU50XEDGAm2Wn2LqkR/WpPsV8eEU8ApK87NFGvmZlZs5YB0yXtLmkiMAdYXLuDpOk1b/8MWJleLwbmSNpC0u7AdOAnI1XYTIe7QNI9ZJePdwYmNthnTJcJJM2XdLuk26/45eqRC5iZmeUQERuBE4AbgQeAqyLiPklnpRHJACdIuk/S3cDJZJeTiYj7gKuA+4ElwPEjjVCGnJeUJR0EHAr8cUSslbSU7JR6Ws1utafYv5K0Y0Q8IWlH4MlGx609/W/VNX0zM+swas+EmYi4nmy8Ue26M2pef3iYsp8GPj2W+vLew30p8EzqbF8N/BFwJvCcpD8CbgP+Gvi3tP9isr8MFqav3xqpgvpBUr/80/eMuI9ZETptoIpVg3/vel/ePyuWAOMl3QucTXZZGeA44GKyOUk/JxswBVlH+1ZJK8kmGS/M3WIzM7MulOsMNyKeB/50iM2va7D/08AheeoyM7Mep5bMNuo4TpoyMzMrgTtcMzOzEhSaNCVpBnAZMJls5NeHIyIknQf8OfAC2b3d/xMRvxnLsRsNkKofSJVnEFU3pLuYmVn3K/oMtz/acXpaZqX1NwGvi4h9yJ7OcGrB9ZqZWZdSn1qydJrcHa6kRZLuSJOC56f5tVtHxI8je+Zff7QjEfHdNMkYshHN04Y4rJmZWU8qK9pxQDlenC40QG3S1KWXXtpE08zMzDpLM/dwF0g6PL0eVbSjpNOAjcBXGx2wNmnqueeec9KUmVkVtClpqmxlRTsiaS7wTuCQdMnZzMysMkqJdpQ0i+xxfAdGxNqmW50UEf/Y6SOSX+gbOIp64ubObq+ZmTVWVrTj54ApwE2S7pb0hfxNNjMz6z5lRTvumaceMzOrAEc7mpmZWVEK7XAlzZC0XNIqSf8qDfyzRdJHJIWk7Yus18zMrNMVGu3Ii0lTt5JFO84i3ceVtDPZo/keLbjO32tV/GM7eZBU+Savf37A+3WTthjzMdYxbvBx2ZS7TWajVR9X2+kDQwHowFSoViglaSo5H/gYdXNzzczMqqCZM9x5EbFG0mRgGXAPQyRNSZoN/CIi7lFFbo6bmZnVauYe7gJJ95BdPh4yaUrSlsBpwBkjHdDRjmZm1SOpJUunKSNpag9gd6D/7HYacKekAyLil7XHdbSjmZn1qpYnTUXEcmCH/oKSHgZmRsSvm2n4aI2URtVtg6is9fIMkqrnAVLWLl0xSKpeXzVmqJaVNGVmZlZppSRN1ZXdLU+dZmZm3azoebhmZmZj04EDnFqhGhfOzczM2qy0aEdJJ0pakYIyzi2yXjMzs05XSrSjpIOBw4B9IuJ5STsMc4xRyxNhVsQzdM3MzMaqrGjH44CFabAVEfFk0y03M7PeILVm6TDNXFKeFxEzgJnAAmAXhoh2BPYC3iTpNkn/KWn/Juo1MzPrOi2PdkxfxwNTyQIyPgpcVf/oPnC0o5mZ9a4yoh0hO9v9RrrU/BNJm4Htgadqj+toRzOz6lFFkqZaHu2Y9l8EvAVYKmkvsrPhpqMdi4gw68Zn6Hbl8y6tktZr8H8xk2JjG1pi1n5lRTteArxS0k+BK4C56WzXxqi+szUzs+5QSrRjRLwAHJWnLjMzs17gaEczM2uvitzDrcanNDMza7NCz3AlzQAuAyaTJU19OCJC0r7AF4BJwEbgQxHxkyLrLlonP0fXg6Ssm3iQlFmm6DPc/mjH6WmZldafC3wyIvYFzkjvO9Zo4h/bxZ2tdRN3tjYqTpoa3hijHQPYOr1+KS/OzzUzM6uEZi4pz4uINZImA8uAexg62vFvgRslfYask//fTdRrZmbWdcqKdjwOOCkidgZOAr7c6ICOdjQzqx5JLVk6TVnRjnOBD6fXV5OFYwziaEczM+tVZUU7Pg4cCCwli3hc2USbS9eN8Y9mNtjktesHvF+35aQ2tcQG6Ou8s9FWyNvhLgGOTdGOKxgY7XgZ2bSgG3gx2vGDwAWSxgPryUYym5mZVUZZ0Y4/BGbkqcvMzKwXOGnKzMysBM5SNjOz9lI1zv2Kjnb8NNlgqakRsVXN+jcDnwX2AeZERNePMOrk6Ecza8yDpKydiv6z4jrggAbrHwWOBr5WcH1mZmZdIfcZrqRFZIEXk4ALIuKiiLg1bRuwb0Q8nNZvzt1SMzPrTRWZFtTMGe68iJgBzCRLndqu2cY4acrMzHpVM/dwF0g6PL3emezpQE830xgnTZmZWa8qMtrRoxHMzGzMOjH3uBWKjHastNE8Q9cjl82KMXn984PWrZu0RRtaYjZ6ee/hLgHGp2jHs0nRjpLOlbQa2FLSaklnpvX7p/VHAF+UdF/zTTczM+seRUc7LgU+1mD/ZQx8kpCZmVmlOGnKzMzaqyJJU4V+SkmflvSYpN/VrT9W0nJJd0v6oaS9i6zXzMys0xV9hnsd8DkGP+/2axHxBQBJs4F/AWYVXHdHadUzdD1YxMy/81YMSbOAC4BxwMURsbBu+8nAB4CNwFNk+ROPpG2bgOVp10cjYvZI9ZWVNPVszduXAJ5ja2ZmmTYkTUkaB1wIvBVYDSyTtDgi7q/Z7S5gZpqRcxxwLvC+tG1dROw7ljqbOcOdFxFrJE1ODb02IoYMvpB0PHAyMBF4SxP1mpmZNesAYFVEPAQg6QrgMOD3HW5E3FKz/63AUc1U2Mw93AWS7kmN6E+aGlJEXBgRewAfB05vtI+jHc3MrCQ7AY/VvF+d1g3lGOCGmveTUn91q6R3jabCdiRNXQF8vtEGRzuamVWP+lozSlnSfGB+zaqLUj8D0Og6dsN+R9JRZM8NOLBm9S4R8bikVwI3S1oeET8frj2lJE1Jmh4R/QOp/ozBg6oqoYg0Kg8WMTMbndqTuAZWk12d7TcNeLx+J0mHAqcBB6YMiv5jP56+PpROOvcDhu1wS0maAk6QdJ+ku8nu487NWa+ZmVkRlgHTJe0uaSIwB1hcu4Ok/YAvArMj4sma9VMlbZFebw+8kZp7v0MpK2nqw3nqMTMza4WI2CjpBOBGsmlBl0TEfZLOAm6PiMXAecBWwNVp9k3/9J/XkMUUbyY7cV1YN7q5ISdNmZlZe7XpaUERcT1wfd26M2peHzpEuf8G/nCs9VUjT8vMzKzNyop2PFrSUyna8W5JHyiyXjMz62JSa5YOU1a0I8CVEXFCwfV1tVbFP7aToyfNzBrLfYYraZGkO9Lo4/kAEXFrRDxRXPPMzMx6Q2nRjsC7Jb0Z+BlwUkQ8Nsy+ZmZWFS0Kvug0ZUU7XgfsFhH7AN8DLm+0k6MdzcysV5US7Vh35vsl4Jwh9nO0o5mZ9aSyoh13rLm3Oxt4IGe9w3qhb8KgdRM3b2hFVS1TRPxjO3mAlJmNVf0jXXtVWdGOC9LgqnuABcDRzTW7ujq5szUzs6GVFe14KnBqnrrMzMx6QTWGhpmZmbWZs5TNzKy9+qpxD7fQDlfSp4G/BqZGxFY1688HDk5vtwR2iIhtiqwbihsgtWHcwMFXEzYVc9z6FKbRDDDqxTQqM7MqKiXaMSJO6n8t6USyB/VaDo1GLZuZWefL3eFKWkQWeDEJuCAiLoqI/tHKwxU9EvhE3nrNzKzHqBrDicqMdkTSrsDuwM1N1GtmZtZ1yop27DcHuCYiNjXa6GhHMzPrVaVEO9aYAxw/1EZHO5qZWa8qJdoRQNKrgKnAj3PWWZqiRiXXKyL20KOWq60X4kutO6xj3KB1U1pVmaMdhzXWaEfIBktdERE+czUzs8opJdoxlTkzT11mZma9wElTZmbWVqpI0lQ1Jj+ZmZm1WWFnuJK2BK4G9gA2AddFxClp2xbAV4AZwNPA+yLi4aLqrrpuf4aujZ4HSFlZJtNw9mZreNBULp+JiFeTRTe+UVL/fd5jyEY17wmcD5xTcL1mZmYdLXeHK2mRpDvSg+XnR8TaiLgFICJeAO4EpqXdDwMuT6+vAQ7RCPmPZmZmvaSZM9x5ETEDmEmWOrVd/wZJ2wB/Dnw/rdoJeAwgIjYCvwW2o46TpszMKqivrzVLh2nmHu4CSYen1/3Rjk9LGg98HfjXiHgobW90NjtoPq6TpszMrFfl+hOgLtrx9cBdvBjteBGwMiI+W1NkNVmnTOqQXwqsydlmMzOzrlNotKOkT6VtH6jbfzEwlyzW8T3AzU6cap0y4x8nr39+wPsi4ivNrFrUgZd/W6HIaMdpwGnA3sCdku6W1N/xfhnYTtIq4GTglOaabWZm1l2Kjnb8jyH2Xw8ckacuMzOzXlCN83gzM7M2K6zDlbSlpO9IejDNzV1Ys+1kSfdLulfS9yXtWlS9ZmbW5aTWLB2m6IcXfCYibpE0Efi+pD+NiBvIRjHPTIOsjgPOBd5XcN02jFbFP3qQlJnZ6JSSNBURt0TE2lS0f4CVmZlZZTRzhjsvItZImgwsk3RtRDwNA5KmLmhQ7hjghibqNTOzXtKBl39boZl7uAsk3UN2xtqfNMUQSVOkbUeRRUGe1+iAjnY0M7NelesMty5paq2kpQyfNIWkQ8nm6R6YphUN4mhHMzPrVaUkTUnaD/giMCsinmyivVaQMtOoyuLUKzPrZHk73CXAsSlpagUDk6YeJEuaAvhcRFxMdgl5K+DqtP7RiJjdZNvNzKwXVCTasaykqUPz1GNmZtYrqvFnhZmZWZsVHXxhZmY2JvK0oLEZLtqxZp/3SApJM4uq18zMrBuUFe2IpCnAAuC2guu0gowU/9jpo5Y9KtnMOlkp0Y7J2WQZyuubarGZmfWWijy8oJlLyvMiYgZZctQCSdv1b6iJdvx+er8fsHNEfLuZxpqZmXWrlkc7SuoDzgf+bqQDOtrRzKyC+tSapcOUEe04BXgdsDSNRHsFsFjS7Ii4vfa4jnY0M7Ne1fJox4j4LbB9//vUOX+kvrO1ztOqZ+ja0F7omzDg/cTNG9rUEjMrWt5LykuA8Sna8WwGRjvuTRbteLekDwxzDDMzs8ooJdqxruxBeeo0M7MepWqEHlbjU5qZmbWZO1wzM7MSFJY0JWlL4GpgD2ATcF1EnFKz/b3AmUAA90TEXxZVt42siGfF9uIzdDuNB0lZFakDp/C0QinRjpKmA6cCb4yIZyTtUHC9ZmZmHa2saMcPAhdGxDNp+5PNNtzMzKybNHOGOy8i1kiaDCyTdG1EPA0Doh0vSPvuldb/CBgHnBkRS5qo28zMekVfNYYTtTzaMe07Pm0/CDgSuDh1ygM42tHMzMoiaZakFZJWSTqlwfaTJd0v6V5J35e0a822uZJWpmXuaOorI9oRYDVwa0RsAP5H0gqyDnhZ7XEd7WhmZmWQNA64EHgrWR+1TNLiiLi/Zre7gJmpnzuO7Il375O0LfAJsof3BHBHKvvMcHW2PNoxWUR2ZnuZpO3JLjE/hA1SxGjiRlr1rFjHP5pZlzoAWNV/JVbSFcBhwO873P5xScmtwFHp9duBmyJiTSp7EzCL7OrukMqKdrwReFrS/cAtwEf77/eamVnFted5uDsBj9W8X53WDeUY4IacZYGSoh0jIoCT02JmZtZykuYD82tWXZRuXQI06pEb3sqUdBTZ5eMDx1q2VtHzcM3MzDpC7bigBlaTDfjtNw14vH4nSYeSXb09MJ1s9pc9qK7s0pHaU9hYbElbSvqOpAfT3NyFNdt2kXSLpLvSaK93FFWvmZl1N0ktWUawDJguafcU1jQHWFzXrv2ALwKz6/IjbgTeJmmqpKnA29K6YZWSNAWcDlwVEZ+XtDdwPbBbwXX3hFYNbiqL4x/NrBtExEZJJ5B1lOOASyLiPklnAbdHxGLgPGAr4OrUgT8aEbNTBsXZvDjT5qz+AVTDyd3hSlpEdjo+Cbggnbr/PmlKUm3SVABbp9cvpcFpu5mZWZki4nqyE8DadWfUvD50mLKXAJeMpb6ykqbOBL4r6UTgJWRzeM3MzCqjrKSpI4HLImIa8A7g36XBTxx20pSZWQX19bVm6TBlJU0dQzYpmIj4saRJwPbAgIcYOGnKzKyCRh7g1BPy/gkwUtLU39bt/yhwSNrnNWSd81M56zYzM+s6ee/hLgGOTUlTKxiYNPUgWdIUwOci4mLg74AvSTqJbADV0SkMwypgpPhHj1o2syooK2nqfuCNeeoyMzPrBU6aMjOz9vI9XDMzMytKoR2upCWS7knRjl9IzxtE0hFp3WZJM4us08zMrBsUfUn5vRHxrLIRU9cARwBXAD8F/oIsk9Iqzs/QNbNa6sA5s62Q+1NKWiTpjnTmOh8gIp5Nm8cDE0mPK4qIByJiRdOtNTMz61LN/FkxLyJmkD0jcIGk7QAk3UgWaPEc2VmumZlZ5RUe7RgRbwd2BLYA3jKWAzra0cysgqTWLB2mFdGORMR6SYuBw4CbRntcRzuamVmvyjtoqlG041aSdoyIJ9IDDN4B/FdRDbXe5Wfolm/y+ucHvO/25zCbdYO8l5SXAONTtOPZZJeVXwIsTuvuIbuP+wUASYdLWg38MfCddJ/XzMysMoqOdrxyiP2/CXwzT11mZtbj+jrvfmsrVGPyk5mZWZu5wzUzMytBoUlTkpaQTQkaTzZg6viI2CTpbLIRy5vJ7u0eHRGPF1m3mZl1qQ6cwtMKZUU7nhcR/wAgaQFwBnBswXVbD3H8Y2Mv9E0YtG7i5g1jPo5HJZuVr6xox2drir6kf72ZmVlVNHOGOy8i1kiaDCyTdG1EPJ2m/BwA3EBNtKOkTwN/DfwWOLiZRpuZmXWb0qIdI+K0iNgZ+CpwQqMDOtrRzKx61NfXkqXTtCPa8WvAd4BP1B/X0Y5mZhWkzuscW6GUaEdJ0yNiZSo7G3iw2YZbtTj+MZNngJSZdYa8He4S4NgU47iCgdGOWwDjgJtJ0Y7AQkmvIpsW9AgeoWxmZhVTVrTju/PUY2Zm1iuKnodrZmY2Ns5SNjMzs6KUFe14JvBB4Km0699HxPVF1m3N67ZnpI6URlWFQVRm1j3KinYEOD8iPlNwfWZm1uVUkSzlUqIdzczMqq6Ze7jzImIGMJMsdWo7gBTt+CTwHDXRjsAJku6VdImkqU3Ua2Zm1nXKinb8PLAHsC/wBPDPjQ7oaEczM+tVpUQ7RsSvasp+Cfh2o+M62tHMrIIc7TissUY77hgRT6SyhwM/bbbhVrxOH5U8Ej9D18w6WVnRjudK2pdsENXDwN8002gzM7NuU1a04/vz1GNmZhXgpCkzMzMrSllJU1cCr0q7bQP8JiL2LbJuMzOzTlZK0lREvK9/B0n/DPy24HrNBvEzdM26hJOmhpcnaSp1xO8Fvp63XjMzs25UZtIUwJuAX0XEyibqNTMz6zplJU31O5Jhzm6dNGVmZr2qlKSpVGY88BfAjKGO66QpM7PqUUWmBZWSNJUcCjwYEauba7JZfk6jMhudyWvXD145ZUr5DekhZSVNAczBg6XMzKyiSkmaSmWOzlOXmZn1uIo8vKAan9LMzKzNig6+MDMzGxsHX+QnabGkn9a831bSTZJWpq9TW1GvmZlZpyr8DFfSXwC/q1t9CvD9iFgo6ZT0/uPDHWfDuAkD3k/YtCFXeyavf37A+25/5qsVq1Xxj+sYN2jdZDaN+Thm7bJuy0mD1nmMcnMKjXaUtBVwMvCput0PAy5Pry8H3pW3XjMz6zF9as3SYZo5w50XEWskTQaWSboWOB34Z2Bt3b4vj4gnANI83R2aqNfMzKzrFB3tuGdEfDPvAWujHS+/5MtNNM3MzKyzFBntuB8wQ9LD6bg7SFoaEQcBv6pJodqR7OEGg9RGO65Zu97RjmZm1jOKjHY8MyL+AEDSbsC3U2cLsBiYCyxMX781UgV5B0nV8yApG6si4h89QMps9NRXjUiIvJ9yCTA+RTueTXZZeTgLgbdKWgm8Nb0360rOWjazPIqOduzf/jDwupr3TwOH5KnLzMysFSTNAi4gy/+/OCIW1m1/M/BZYB9gTkRcU7NtE7A8vX00ImaPVJ+TpszMrL3akKUsaRxwIdlV19Vks20WR8T9Nbs9ChwNfKTBIdZFxL5jqdMdrpmZVdEBwKqIeAhA0hVkmRG/73DT1VokbS6iwrKiHc+T9KCkeyV9U9I2rajXzMxslHYCHqt5vzqtG61JaRrrrZJGFeZUVrTjTcCpEbFR0jnAqYwQ7WjWKfzQ+mK90DcwtnXi5mJmJFgXa1EqVEpBnF+z6qI0/RSgUaVjmY66S0Q8LumVwM2SlkfEz4crUEq0Y0R8NyI2pre3AtPy1mtmZjYaEXFRRMysWS6q2byaLLSp3zTg8TEc+/H09SFgKVkWxbCauaQ8LyJmADPJUqe2I5si1CjacUA54IYm6jUzM2vWMmC6pN0lTQTmkGVGjEjSVElbpNfbA2+k5t7vUEqNdpR0GrAR+OoQ238f7XjppZc20TQzM7OhpauuJwA3Ag8AV0XEfZLOkjQbQNL+klYDRwBflHRfKv4a4PbUB94CLKwb3dxQWdGOSJoLvBM4JCIaXievjXZ87rnnHO1oZlYBatMD6CPieuD6unVn1LxeRoNboBHx38AfjrW+UqId0+TijwMHRsRwl5vNOl6rnqFbFfWDpOqfWQ2OZLXeVFa04+fInl18k6S7JX0hZ71mZmZdqaxoxz3z1GNmZhXQpkvKZavGIxrMzMzazNGOZmbWXhV5PF9LOlxJi4FXRsTr6tZ/BDgPeFlE/LoVdddbx7gB7/2cUhtJnt8Zp1Hl5wFSVhWF/1kxRLQjknYmeyrDo0XXaWZm1ulKiXZMzgc+xtiyKs3MrNdJrVk6TCnRjim14xcRcc9wB3TSlJmZ9apm7uEukHR4el0b7XhSCr4AQNKWwGnA20Y6oJOmzMysV+U6w62Ldnw9cBcDox1/COyVIh/3AHYH7knbpgF3SnpFs403MzPrFqVEOwI79BdMne7MskYpe1SyjVURvzO9EP9YH7no0cTWKu3KUi5bWdGOZmZmlVZKtGPdtt3y1GlmZtbNnDRlZmbtVZGkqZZ8SkmLJf20bt2JklakebvntqJeMzOzTlX4GW6jpClJBwOHAftExPOSdmhY2KyHdVv8owdJmRWrrKSp44CF6d4vEfFk3nrNzKzHOGlqRKNOmgL2At4k6TZJ/ylp/ybqNTMz6zrNdLgLJN1DNiWoNmnqmw32HQ9MJZuv+1HgKjWYeOVoRzMz61W57uHWJU2tTYlStUlT44EdJC1N4RergW9ERAA/kbQZ2B54qva4jnY0M7NeVVbS1CLgLcBSSXsBE4FSkqasOC/0TRi0buLmDW1oSW/ohTQqs0JUZFpQ3g53CXBsSppawchJU5cAl6SpQi8Ac9PZrpmZWSWUkjQVES8AR+Wpy8zMrBc4acrMzNpKfZ03hacVqnHh3MzMrM1KiXaUtK+kWyXdnab9HNCKes3MzDpVKdGOwLnAJyPiBknvSO8PKrpuay2PSG69keIfPWrZyjJ57frBK6dMKb8hPaSsaMcAtk6vXwo8nrdeMzPrMRWJdmzmDHdeRKyRNBlYJula4HQaRzv+LXCjpM+QdfL/u4l6zcysl6gaw4nKinY8DjgpInYGTgK+3OiAjnY0M7NeVVa041zgw6n41cDFjY7raEczM+tVZUU7Pg4cCCwli3hcmb/JVq8+crEXBzdVNVay256ha71j3ZaTBq3zkKnmlBXt+EHgAknjgfXA/Jz1mplZj6lK8EVZ0Y4/BGbkqcvMzKwXVGNomJmZWZs5S9nMzNqrA+fMtkKhZ7iSlkpakSIc75a0Q1r/Zkl3StooafCoDzMzsx7XijPcv4qI2+vWPQocDXykBfVVXhVG61bhM46GH1rfvaowm8CGV2i041Ai4uGIuBfYnLc+MzOzblZ0tCPApZI2AdcCn4oIB1iYmdnQHO04okbRjn8VEX8IvCkt7x/LAR3taGZmvarIaMdJEfELgIh4TtLXgAOAr4z2uI52NDOzXlVktON4SdtHxK8lTQDeCXyvqIaaWWOOf+wOHiQ1jIokTeW9pLyErIO9Fzib7LLyFmSP4LsXuBv4BfAlAEn7S1oNHAF8UdJ9TbfczMysixQd7fidIfZfBkzLU5eZmVkvcNKUmZm1lZw0ZWZmZkUp9Aw3jVbeEViXVr0tIp6UdCxwPLAJ+B0wPyLuL7JuM8s4jcqsM5UV7fi1iPgCgKTZwL8As1pQt5mZWUfK3eFKWkQWeDEJuCDNoW0oIp6tefsSwHNszcws42lBI5oXETOAmWSpU9ul9ZemJwX9g2ruhEs6XtLPgXOBBY0O6KQpM7MK6utrzdJhmrmkvEDS4el1bbTjLyRq6mDSAAAcgklEQVRNIctSfj8paSoiLgQulPSXwOnA3PoDOmnKzMx6Va4/AeqiHV8P3EVdtCPQH+1Y7wrgXblaa2Zm1qVKiXaUND0iVqayfwasbHhUs2Ty+ucHvF83aYs2taQ3jBT/6FHLZq2Xt8NdAhybYhxXMDDacQIwjqyz/VLa/wRJhwIbgGdocDnZzMwqqiKP5ysr2vHDeeoxMzPrFdX4s8LMzKzNCu1wJS2VtCJNC7pb0g5p/dGSnqpZ/4Ei6zUzs+4lqSVLpykraQrgyog4oQX1WQ/yIKnW8jN0zUDSLOACsnFHF0fEwrrtbwY+C+wDzImIa2q2zSWb4grwqYi4fKT6cp/hSlok6Q5J90man/c4ZmZmZZM0DriQbDzS3sCRkvau2+1R4Giyaa61ZbcFPgH8L7Lpr5+QNHWkOktLmgLeLeleSddI2rmJes3MzJp1ALAqIh6KiBfIMiIOq90hIh6OiHuBzXVl3w7cFBFrIuIZ4CZG8XyAZjrcBZLuIZsSVJs09YfAm9Ly/rTvdcBuEbEP2XShhqfejnY0M6ugPrVkqe1T0lJ7NXYn4LGa96vTutHIVTbXPdy6pKm16bF8A5KmJPUnTX0lIp6uKf4l4JxGx3W0o5mZFaW2T2mg0aiq0fY7ucqWlTS1Y0Q8kcrOBh7IWa9Z1+rk9Cw/Q9cqaDXZ1dl+04DHx1D2oLqyS0cqVFbS1IL0HNyNwBqym9BmZmbQnik8y4DpknYHfgHMAf5ylGVvBP6xZqDU24BTRypUVtLUqaNpjJmZWRkiYqOkE8g6z3HAJRFxn6SzgNsjYrGk/YFvAlOBP5f0yYh4bUSskXQ2WacNcFZErBmpzlbMwzUzM+t4EXE9cH3dujNqXi8ju1zcqOwlwCVjqc8drpmZtZcfXjB2abTyjsC6tOptEfGkpPOBg9O6LYEdImKbIus2MzPrZKVEO0bESf2vJZ0I7NeCes06WieNSh4Nxz+aFasd0Y5HAl/PW6+ZmVk3auYMd14aqTUZWCbp2rT+UkmbgGvJAp1/PxlY0q7A7sDNTdRrZmY9RH2d92SfVigr2rHfHOCaiNjU6ICOdjQzs15VSrRjTdE5wPFDHdfRjmZmFdSBz65thVKiHQEkvYps8vCPm220mZXP8Y9mzSkr2hGywVJX1N7TNTMzq4pSoh1TmTPz1GVmZj2urxrBF9X4lGZmZm3mDtfMzKwERUc7TgQ+R/acwM3AaRFxraQtyEYrzwCeBt4XEQ8XWbeZlc9pVFYEeZRyLqcBT0bEXpL6gG3T+mPIRjXvKWkOcA7wvoLrNrMO487W7EVFRzvOA/4JICI2R8Sv0/rDgMvT62uAQ1SVP2nMzMxo7h7uvIiYAcwkS53aLq0/W9Kdkq6W9PK0bifgMcge+gv8Fthu0BHNzMx6VJHRjq8he1DvjyLiDWQBF59J+zY6mx00H9fRjmZmFdTX15qlwxQZ7TgBWAt8M+12Ndm9W4DVZJ3yaknjyZKq1tQf19GOZmbWq4qMdgzgOrIRyjcDhwD3p/0XA3PJznrfA9zcyYlT6zXw2zIpNrapJWbdxaOWzYZWZLQjwMeBf5f0WeAp4P+k9V9O61eRndnOyd9kMzPrKRUZQ1t0tCPAmxvsvx44Ik9dZmZmvaDz7iqbmZn1IHe4ZmZmJSgr2vFk4APARrJ7u/Mi4pEi6y6SB0nZSF7omzDg/cTNG9rUks7mZ+jaqPT5Hm4eQ0U73gXMTKOajwPOxdGO1qXqO1szs9EoJdoxIm6JiLVpn1vJAjLMzMwqo6xox1rHADc0OqCTpszMqkfqa8nSaZq5pLxA0uHpdX2048npvu1ngPf3F5B0FFkHfWCjAzppyszMelWuPwHqoh1fT3aPtlG04xtqyhxKdo93dprHa2ZmVhmlRDtK2g/4IjArIp5sttFm7dRoRPLk9QP/hlw3aYuymtN1HP9ogzhpalhjjXY8D9gKuDo9BvfRiJidu9VmZmZdpqxox0Pz1GNmZhVQkXm4nTeMy8zMrAcV2uFKmijpIkk/k/SgpHfXbX+PpJA0s8h6zczMOl1ZSVNImgIsAG4ruE6ztvMgqfwc/2h04JzZViglaSo5myzScX3u1pqZmXWpUpKm0rSgnSPi202218zMrCs10+EukHQP2ZSg+qSpNwA/Bj6TLi2fD/zdSAd0tKOZmfWqXPdw65Km1kpaSuOkqWOAKcDrgKVpDu4rgMWSZkfE7bXHdbSjmVn1qCLTglqeNBURvwW27y+YOueP1He2Zmb9nEZlvaispCkzs8K4s7VuVErSVF3Zg/LUaWZmPaoiWcrVmPxkZmbWZu5wzczMSlBatKOk90q6PwVlfK3Ies3MzDpdKdGOkqYDpwJvjIhnJO1QcL0dp/75qOD4P7O8PGq5x1XkHm7uDlfSIrLAi0nABWkO7Tzg1ZBFOwL90Y4fBC6MiGfSNj+E3szMKqWUaEdgL2AvST+SdKukWc002szMrNu0PNox7TsemE4WinEkcLGkbeoP6GhHM7PqUV9fS5ZOU0a0I8Bq4NaI2AD8j6QVZB3wstrjOtrRzMx6VcujHdP+i8jObC+TtD3ZJeaHmmh3x/MAKbPW8TN0rRuVFe14I/A2SfcDm4CPRsTT+ZttZmY9owMv/7ZCKdGOERHAyWkxMzOrnKLn4ZqZmY1NRebhVuM83szMrM0KPcOVNBH4HNnAqc3AaRFxraRdgMuBbYBxwCkRcX2RdZtZtTmNyjpdKdGOwOnAVRHxeUl7A9cDuxVct5mZdaO+alxSLivaMYCt0+uXAo/nrdfMzKwblRXteCZwlKTVZGe3JzZRr5mZWdMkzZK0QtIqSac02L6FpCvT9tsk7ZbW7yZpnaS70/KF0dRXVrTjkcBlETENeAfZXN1BdTva0czMyiBpHHAh2RTXvYEj0y3PWseQhTztCZwPnFOz7ecRsW9ajh1NnWVFOx4DzAKIiB9LmgRsDwx4apCjHc3MqqfB+VcZDgBWRcRDWRt0BXAYLyYkkt6fmV5fA3xOyj+Hqaxox0fT+8skvYbsvu9TeRvdCSavXT/g/botJ7WpJWbWiOMfbQQ7AY/VvF8N/K+h9omIjZJ+C/TfPt1d0l3As8DpEfFfI1VYVrTj3wFfknQSWcd8dEqfMjMzawlJ84H5NasuSldSARqdqdb3S0Pt8wSwS0Q8LWkGsEjSayPi2eHaU1a04/3AG/PUZWZmPa5FSVO1tykbWE02/qjfNAbPoOnfZ7Wk8WRXd9ekE8bnUx13SPo52UN5bh+uPU6aMjOzKloGTJe0ewptmgMsrttnMTA3vX4PcHNEhKSXpUFXSHol2eNmR3wCXmEdrqQpNUOk75b063RpGUlvTlOFNkoaHP9iZmZWoojYCJxA9jS7B8jCme6TdJak2Wm3LwPbSVpF9vCd/qlDbwbuTTN1rgGOjYg1I9WpVt1KlXQHcFJE/CDNXdoa+AiwOCJGHKngUcpm1mqOfxybKVOmtOTa7/r7HmzJ//eTXvvqjoqwKjppqn/bdGAH4L8AIuLhtH5zM401M7Me5GjHEc2LiDWSJgPLJF1b81D5I4ErPRLZzMwsU2TS1PSabXOAr4/1gE6aMjOzXlVk0tSktO31wPiIuGOsx3XSlJlZBfkB9MNqlDTV70hynN2amZn1slyjlCVtASwii71aAbwMODMilkp6CHhHRDxYs//+ZBnLU4H1wC8j4rXD1eEzXDNrB8c/Dq1lo5Qf/FlrRim/eq+OOnUuPGkqIl7ZYN0yshQPMzOzAdr08ILSVeNTmpmZtZk7XDMzsxKUFe14rKTlaf0PGzzk18zMrKeVFe24df9ji1JG5YciYtZw5T1oysw6gQdRvahVg6aeX/nzlvx/v8X0PTpq0FTuM1xJiyTdIem+9MzB2m310Y61zwh8CYOfOWhmZtbTSot2lHQ82dMWJgJvaaJeMzPrJX3VGE5UWrRjRFwYEXsAHwdOb3RARzuamVmvake04xXA5xttcLSjmZn1qryXlMcU7ShpekSsTG//DFiJmVkXqB8k5WfoFk8VyVLO2+EuAY6VdC9ZtOOtNdveC7yjbv8TJB0KbACeAebmrNfMzKwrlRXt+OE89ZiZmfWKZkYpm5mZNc+jlM3MzKwoZUU7Hi3pqZptHyiqXjMzs25Q2CXliHgO2Lf/fYp2/EbNLldGxAlF1Wdm1g6NRiQ7/tFGo5RoRzMzsyFJrVk6TDOXlOdFxAxgJlnq1HY12wZFOwLvlnSvpGsk7dxEvWZmZl2nrGjH64DdImIf4HvA5Y0O6GhHMzPrVaVEO9Y81ADgS8A5jY7raEczswrqwMu/rVBWtOOOEfFEejsbeCBnvWZmHcfxjzYaZUU7LkgPnt8IrAGOzlmvmZlZVyor2vFU4NQ8dZmZmfUCRzuamVlbqa8a93Ad7WhmZlaCws5wJU1hYNDFNOA/IuJvJZ0PHJzWbwnsEBHbFFW3WZW90DdhwPuJmze0qSXWz2lU1kgp0Y4RcVLN+hOB/Yqq18zMupyqcbG1HdGOg6YNmZmZ9bpmznDnRcQaSZOBZZKurQm4aBTtiKRdgd2Bm5uo18zMeklFgi/KinasXX9NRGxqdEBHO5qZWa8qJdqxxhzg+KGO62hHMzPrVaVEOwJIehUwFfhxzjrNrAGPSu4OI8U/VnrUckXm4ZYV7QhZR3xF/X1dMzOzKigl2jGtPzNPXWZmZr3A0Y5mZtZW8jzcsZN0pKTlku6VtETS9mn9tpJukrQyfZ1aZL1mZmadTkXdUpU0Hngc2Dsifi3pXGBtRJyZXq+JiIWSTgGmRsTHhzvemrXrBzRswiYPDDGz3tUNz9CdMmVKS0Y3bfzVUy0Z2zP+5S/rqNFYRSZNKS0vkSRga7IOGOAw4PL0+nLgXU202czMrOsUljQFXAscBywH/h+wkhfn3L48Ip4AiIgnJO3QRL1mZtZLKjItqMikqb3JOtz9gD8A7mWMD52vTZq6/JIvN9E0MzOzzlJk0tQfAUTEz9M+VwGnpCK/krRjOrvdEXiy0XFrk6bq7+GamZl1syKTpi4C9pb0soh4Cngr8EDafzEwF1iYvn5rpAo8SMrMqqTTnqG7XoO7hyktqmvdpC1actxWtTevIpOmHgc+CfxA0gbgEeDotP9C4CpJxwCPAkc002gzM7NuU3TS1FLgCw32fxo4JE9dZmZmvaAa8R5mZmZt5g7XzMysBGVFO54n6cG0/puStimyXjMzs05XVrTj24CbI2KjpHMARop29APozcwGanf8Y6uiHVv1/32r2ptXKdGOEfHdiNiYit4KTGuy3WZmZl2lrGjHAeWAK5uo18zMrOuUGu0o6TRgI/DVRgesjXa89NJLm2iamZlZZykr2hFJc4F3AofEEDeOa6MdfQ/XzMx6SSnRjpJmAR8HDoyItQW028yscjot/tHGJu8l5SXA+BTteDaDox3vBfYF/jHt/zmyWMubJN0taVAalZmZWZkkzZK0QtIqSac02L6FpCvT9tsk7Vaz7dS0foWkt4+mvrKiHffMU4+ZmVkrSBoHXEh2NXY1sEzS4oi4v2a3Y8iu5u4paQ5wDvA+SXsDc4DXko1Z+p6kvSJi03B1OmnKzMyq6ABgVUQ8FBEvAFcAh9XtcxhweXp9DXBImvZ6GHBFRDwfEf8DrErHG5Y7XDMzq6KdgMdq3q9O6xruk7IkfgtsN8qyg5QS7Viz/SOSon69mZlZ0WqnmqZlfu3mBkXqZ8cMtc9oyg7STPDFACna8QIGRjueAJyZtu9Mdq380aLqHI0N4yYMeO8H25tZL6kflexRyy+qnWrawGqyDIl+00jpiA32WZ36uJcCa0ZZdpBSoh2T84GPMYq/AszMzFpsGTBd0u6SJpINglpct89iYG56/R6yZwJEWj8njWLeHZgO/GSkCkuJdpQ0G/hFRNyT9cVmZmbtkx6mcwJwIzAOuCQi7pN0FnB7RCwGvgz8u6RVZGe2c1LZ+1K40/1k6YnHjzRCGUqIdpS0JXAacMZIB3S0o5mZlSUiro+IvSJij4j4dFp3RupsiYj1EXFEROwZEQdExEM1ZT+dyr0qIm4YTX1lRDt+C9gd6D+7nQbcKemAiPhl3Yd3tKOZmfWklkc7RsRyYIf+gpIeBmZGxK+ba/roeJCUmVXJSIOoGu1j5cjb4S4Bjk0RjisYHO24AXgEOLqIRpqZmXW7UqId68rulqdOMzOzbuakKTMzsxK4wzUzMyuBhngWfL6DSUcCf08WbvE4cFT/4ChJJ5IlT20EvhMRHxvuWB6lbGbWGnnTqKZMmdKSIIVW/X/fqvbmVdgZbk2048ERsQ/ZPNwT0raDyZ6usE9EvBb4TFH1mpmZdYOyoh2PAxamwVZExJNNttvMzKyrNHOGOy8iZgAzgQVkHWx/tOPjZMlTX0777gW8SdJtkv5T0v5N1GtmZtZ1Wh7tmPYdD0wlC8j4KHCVGoQqO9rRzMx6VRnRjpA9yugb6SkLP5G0GdgeeKr2uI52NDOzXtXyaMe0/yLgLcBSSXsBE4FSoh3NzGygTot/rH9uea8qK9rxEuASST8FXgDmRpHzkczMrGtVpTcoJdoxIl4AjspTl5mZWS9w0pSZmVkJRuxwJf1uhO27pUvFoybpMkmDbxqYmZn1qLz3cK2DrGPcgPeT2dSmlnSfyWvXD1q3bstJbWhJY/Xt66S2Weut18D/oifFxpbU02iAVKOBVFN+eGNL6t9ckZu4o76kLGkrSd+XdKek5ZIOq9k8XtLlku6VdI2kLVOZGSno4g5JN0rasfBPYGZm1gXGcg93PXB4RLwBOBj455rwilcBF6UM5WeBD0maAPwb8J6USHUJ8Onimm5mZtY9xtLhCvjHNBXoe8BOwMvTtsci4kfp9X8Af0LWCb8OuEnS3cDpwLRhK3DSlJlZ5URES5ZOM5Z7uH8FvAyYEREbJD0M9N9Qqv9kQdZB3xcRfzzaCpw0ZWZmvWosZ7gvBZ5Mne3BwK4123aR1N+xHgn8kCwQ42X96yVNkPTaIhptZmbWdUZxSv679HV74MfA7cDFZLGNu6XlfrLAi3uBa4EtU5l9gR8A9wD3AR9M6y8ju7c72ssC83NcSujYMp3ePn8f/Jm6qX3+PpT/mbzkW9regFE1Em7vpTKd3j5/H/yZuql9/j6U/5m85FucNGVmZlYCd7hmZmYl6JYO96IeK1NmXZ1cpsy6/Jnylymzrk4uU2Zdnf6ZLAela/hmZmbWQt1yhmtmZtbV3OGamZmVwB2umZlZCdzh1pC03Rj23VbS1Fa2p6aurdOTl0ZdX6e3r2xj+dmWbaxt6+SfbZltS/V1bPv878IGafdE4LoJ2C8FFgIPAk+n5YG0bpthyr0C+DxwIbAdcCawHLgK2HGIMguB7dPrmcBDwCrgEeDAIcrsAlwBPAWsTPs/mdbtNkSZVwM3AN8B9iBL2foN8BPgNUOU+Y+atr0deIzsgRGPAEcM833o2PZ1wc92zO0rq21d8LMdc9s6vX1lta0bfve8FLe0vQF1vxA3Ah8HXlGz7hVp3U3DlFsCnAicQhYv+fH0j+xE4FtDlFle8/oWYP/0ei+GSF4hi7Z8HzCuZt04YA5w6xBlfgD8OVnG9CNpX6V13x9F2/67/z8FsnjNe4b5PnRs+7rgZzvm9pXVti742Y65bZ3evrLa1g2/e16KW9regLpfiBU5t91V8/rRum13D1HmQWB8en1r3bblQ5RZOUwbGm6ra9uqum13DlHmPmDr9PqHQF/ttrG2oRPa1wU/2zG3r6y2dcHPdsxt6/T2lfzvtqN/97wUt4zl8XxleETSx4DLI+JXAJJeDhxNdnlmKLX3or8yzLZaFwLXS1oILJH0WeAbwCHA3UOUuUPS/wUur2nPzsBc4K4hyoyref0vddsmDlHmk8Atki4EfgRcLelbwFvI/rIdSie3r9N/tnnaV1bboLN/tnna1untK/Pfbaf/7llR2t3j1/2VNRU4h+wvsWfS8kBat+0w5c4Ctmqwfk/gmmHKHQxcSfaPbjnZPZu/ASYMsf9E4DiyfzzLgZ+mMh8CthiizN8M07bPDtO2PdPn/iZwHdn9mreP8P3r2PbV/WzXpKWTfrZjbl8TbTuoQdvmD9W2YX62SzrkZzvmtnVA+zri38UQv3sj/r9X1r8LL8UuTpoyMzMrQcdNC5L0dkmfl7RY0rfS61mtKJe3riGOdUYnlEmf6RhJu9atnzeKMrvlKDOqepR5r6Qj0utDJP2rpA9JGvL3ME+5vHU1OM7No9231WUkbV/3/qj0meZLUpvLHC5p2/T6ZZIul7Rc0pWSpg3zmerLfWWkcnnqkvQvkt44VDvaWSaV21bSGenfkyT9vaRvSzpPQ0wpqinzgVTmtDGUGXU9VqyOOsNVdk9hL7J7EqvT6mnAX5MNbvhwUeXy1jVM2x+NiF3aWUbSPwJ/AtxJNprysxHxb2nbnRHxhgZl/gl44xjL5Knn/wI7kF3eexbYguyS2zuAXw3zsx1zuZxl7q1fRfb7sQIgIvZpV5lU7vffV0mnA28Cvga8E1gdESe1scz9EbF3en0lcCtwNXAo8FcR8dYhPtOYy+Us8xTZSOOXkV1O/XpEDHdvubQyqdz1ZJd3twZew4vTe94KvD4iDmtXGStYu69p1y7Az4ZYL4YfaTjmcjnLPDvE8hywsZ1lUrnlvDgKcRvgeuD89P6udpdJXyeQzTOcmN6PZ/jRuWMul7PMYrJ5lK8GdgV2IxuwsiuwazvL1H9fyf7QeUnNZxzqM5VVZkXN6zvqtjUcMZu3XM4yd6Wv04F/IBtN/CDwCWCvdpapbTfZ/z2/GOVnKqWMl2KXTrukvF7SAQ3W7w+sL7hcnjK/AaZHxNZ1yxTgiTaXgawT3AgQEb8hO/vcWtLVDD2ysqwy/ftvAJZFxAvp/UZg0zCfKU+5MZeJiNnAtWSPKnt9RDwMbIiIRyLikXaWSSZL2k/SDLL5pP+v5jMO9X0oq8xSSWdJmpxevwtA0sHAb4f5THnK5SkT6TOsjIizI+K1wHuBSWR/LLazDEBfuqS7M7CV0q0dZSlQQ/17KquMFandPX7tArwBuA24H/gucFN6fRvwhiLL5SzzKeCAIbad084yadu3aZAYk463qc1lbqDxqMpXAD8Z5jONuVzeutI+W5FNA1lMdgl1NL+3LS9DFlRQu+yY1m/H0GEeZZWZQJZ09GhaNpNdjfkasMswn2nM5XKWaXjVZYTvdyllUrkjgV+l5d1k6VQ3Ab8A5rezjJdil466h9tP0ivIUlMmkt1XfRwgIn5QdLkm6ppKduloUv+6dpdJf/UTEesalNs5Ir7arjLDfKatyC5bXj1cmbzlmiizF7Af2R9ho/05tbxMg880nmxqyw0dUmZbsvuDE8g6w9F+pjGXG20ZSVtFxO8afCbaXaam7DiyS71TgNcCf0Q2tuTxYf69l1LGCtTuHr/RAnyA7D7hM2R/Za8Dbm5FuV4r0+nt82fy96Gb2ufvg5cil7Y3YIhfiOVkfxn23+R/NXBlK8r1WplOb58/k78P3dQ+fx+8FLl02qCpfusjYj2ApC0i4kHgVS0q12tlOr19/kz5y3R6+/yZ8pfphvZZkzotS7nfaknbAIuAmyQ9Q7q32oJyvVam09vnz5S/TKe3z58pf5luaJ81qSMHTdWSdCDZ8yKXRJre0apyvVam09vnz5S/TKe3z58pf5luaJ/l0/EdrpmZWS/o1Hu4ZmZmPcUdrpmZWQnc4ZqZmZXAHa6ZmVkJ3OGamZmV4P8DZkG8oqtTGr8AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "import seaborn\n", + "\n", + "#import the data\n", + "df = pd.read_csv('fixed_data/data_of_states_4(100 k).csv.zip', index_col=0, compression = 'zip')\n", + "\n", + "# Compute the correlation matrix\n", + "corr = df.corr()\n", + "\n", + "# Set up the matplotlib figure\n", + "f, ax = plt.subplots(figsize=(8, 8))\n", + "\n", + "# Generate a custom diverging colormap\n", + "cmap = seaborn.diverging_palette(220, 10, as_cmap=True)\n", + "\n", + "# Draw the heatmap with the mask and correct aspect ratio\n", + "#data_mini = d.sample(10000)\n", + "seaborn.heatmap(corr, cmap=cmap, vmax=.3, center=0)\n", + "plt.show()" + ] + } + ], + "metadata": { + "colab": { + "collapsed_sections": [], + "name": "PPt.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.1" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/.github/entrypoint/artifact/python/gellmann.py b/.github/entrypoint/artifact/python/gellmann.py new file mode 100644 index 000000000000..f2126f7c6d10 --- /dev/null +++ b/.github/entrypoint/artifact/python/gellmann.py @@ -0,0 +1,92 @@ +"""Generate generalized Gell-Mann matrices. + + .. module:: gellmann.py + :synopsis: Generate generalized Gell-Mann matrices + .. moduleauthor:: Jonathan Gross + + .. references:: + - https://arxiv.org/pdf/0806.1174.pdf + - https://github.com/iree-org/iree/tree/main/integrations/pjrt + - https://mathworld.wolfram.com/GeneralizedGell-MannMatrix.html + - https://github.com/husisy/klearning/blob/master/misc/draft_gellman.py + +""" +import numpy as np +import itertools as it + +from pylab import * +from sparse import COO +from pprint import pprint + +def gellmann(j, k, d, sparse=False): + r"""Returns a generalized Gell-Mann matrix of dimension d. + + According to the convention in *Bloch Vectors for Qubits* by Bertlmann and + Krammer (2008), returns :math:`\Lambda^j` for :math:`1\leq j=k\leq d-1`, + :math:`\Lambda^{kj}_s` for :math:`1\leq k k: + coords = [[j - 1, k - 1], + [k - 1, j - 1]] + data = [1, 1] + elif k > j: + coords = [[j - 1, k - 1], + [k - 1, j - 1]] + data = [-1j, 1j] + elif j == k and j < d: + coords = [list(range(j + 1)), + list(range(j + 1))] + data = np.sqrt(2/(j*(j + 1)))*np.array(list(it.repeat(1 + 0j, j)) + + [-j + 0j]) + else: + coords = [list(range(d)), + list(range(d))] + data = list(it.repeat(1 + 0j, d)) + + if sparse: + gjkd = COO(coords, data, shape=(d, d)) + else: + gjkd = np.zeros((d, d), dtype=np.complex128) + for val, m, n in zip(data, *coords): + gjkd[m][n] = val + + return gjkd + +def get_basis(d, sparse=False): + r"""Return a basis of operators. + + The basis is made up of orthogonal Hermitian operators on a Hilbert space + of dimension d, with the identity element in the last place. + + Parameters + ---------- + d : int + The dimension of the Hilbert space. + + Returns + ------- + list of numpy.array + The basis of operators. + + """ + return [gellmann(j, k, d, sparse) + for j, k in it.product(range(1, d + 1), repeat=2)] + +pprint(get_basis(13)) diff --git a/.github/entrypoint/artifact/python/inventory.py b/.github/entrypoint/artifact/python/inventory.py new file mode 100644 index 000000000000..2ae2333fc0eb --- /dev/null +++ b/.github/entrypoint/artifact/python/inventory.py @@ -0,0 +1,218 @@ +# This Python file uses the following encoding: utf-8 + +from gametext import * + +states = {} + +states["currentrm"] = 'entryrm' + +states["entryrm"] = { + roomd: 0, + e_door: 1, + w_door: 1, + chair: 1, + sagan: 1 + } + +states["mainrm"] = { + roomd: 0, + n_door: 1, + e_door: 1, + w_door: 1, + s_door: 1, + # fireplace: 1 + } + +#QUARKS + +states["strangerm"] = { + roomd: 0, + n_door: 1, + e_door: 1, + w_door: 1, + s_door: 1, + dirac: 1 + } + +states["charmrm"] = { + roomd: 0, + s_door: 1 + } + +states["downrm"] = { + roomd: 0, + n_door: 1, + portal: 1, + e_door: 1, + gellmann: 1, + wardrobe: 1, + rope: 0, + teaspoon: 0, + tripod: 0, + hadrons: 0 + } + +states["uprm"] = { + roomd: 0, + s_door: 1, + portal: 1, + hadrons: 1, + rope: 0 + } + +states["bottomrm"] = { + roomd: 0, + w_door: 1, + stairs: 1, + pauli: 1, + trapdoor: 1, + upstairs: 1, + downstairs: 1 + } + +states["toprm"] = { + roomd: 0, + stairs: 1, + trapdoor: 1, + downstairs: 1, + bed: 1, + chest: 0, + powder: 0, + chair: 1, + desk: 1, + drawer: 0, + paperclip: 0, + seashell: 0, + bricabrac: 0, + floorlamp: 1, + drapes: 1, + window: 1, + pauli: 0 + } + + +#LEPTONS + +states["muneurm"] = { + roomd: 0, + n_door: 1, + s_door: 1, + e_door: 1, + w_door: 1, + feynman: 1, + tripod: 1 + } + +states["eneurm"] = { + roomd: 0, + s_door: 1, + e_door: 1 + } + +states["tneurm"] = { + roomd: 0, + s_door: 1, + w_door: 1 + } + +states["elecrm"] = { + roomd: 0, + n_door: 1, + e_door: 1 + } + +states["muonrm"] = { + roomd: 0, + n_door: 1, + e_door: 1, + w_door: 1 + } + +states["taurm"] = { + roomd: 0, + n_door: 1, + w_door: 1 + } + +# BOSONS + +states["photonrm"] = { + roomd: 0 + } + +states["gluonrm"] = { + roomd: 0 + } + +states["Zrm"] = { + roomd: 0 + } + +states["Wrm"] = { + roomd: 0 + } + +states["higgsrm"] = { + roomd: 0 + } + +#INVENTORY + +states["inv"] = { + beads: 0, + key: 9, + key_bt: 0, + bottle: 1, + powder: 0, + refresher: 0, + bongos: 1, + tripod: 0, + up_bongos: 0, + rope: 0, + teaspoon: 0, + paperclip: 0, + seashell: 0 + } + +states[beads] = { + 'u': 0, + 'd': 0, + 'c': 0, + 's': 0, + 't': 0, + 'b': 0, + 've': 0, + 'e': 0, + 'vmu': 0, + 'mu': 0, + 'vtau': 0, + 'tau': 0, + 'y': 0, + 'g': 0, + 'Z': 0, + 'W': 0 + } + +#MAP + +states['map'] = { + "entryrm": 1, + "mainrm": 0, + "strangerm": 0, + "charmrm": 0, + "downrm": 0, + "uprm": 0, + "toprm": 0, + "bottomrm": 0, + "muneurm": 0, + "eneurm": 0, + "tneurm": 0, + "elecrm": 0, + "muonrm": 0, + "taurm": 0, + "photonrm": 0, + "gluonrm": 0, + "Zrm": 0, + "Wrm": 0, + "higgsrm": 0 + } diff --git a/.github/entrypoint/artifact/python/keywords.lst b/.github/entrypoint/artifact/python/keywords.lst new file mode 100644 index 000000000000..01926db38e06 --- /dev/null +++ b/.github/entrypoint/artifact/python/keywords.lst @@ -0,0 +1,7614 @@ +(2+1) dimensional topological gravity +(n p) reaction +1s slater type function +2d lattice +3 j symbol +3d scanner +4 current +4 divergence +4 laplacian +4 tensor +4 vector +4 vector potential +4 velocities +4 velocity +4pi microscope +6 j symbol +9 j symbol +ab initio multiple spawning +abelian group +abelian sandpile model +aberration equation +above threshold ionization +abraham lorentz equation of motion +abraham lorentz model of electron +abrikosov vortex +absolute zero +absorbance +absorption band +absorption coefficient +absorption cross section +absorption edge +absorption spectra +absorption spectroscopy +absorption spectrum +absorptivity +abstract differential geometry +ac stark shift +acceleration field +accelerator mass spectrometry +accelerator operator +acceleron +accidental symmetry +acholeplasma +acoustic droplet vaporization +acoustic metric +acoustic paramagnetic resonance +acoustic phonon +acoustic resonance spectroscopy +acousto optic +acousto optical spectrometer +actinide chemistry +actinide concept +action principle +activation product +active laser medium +adaptive measurement +adaptive optic +addition of velocities +addition theorem for spherical harmonic +adhm construction +adiabatic +adiabatic approximation +adiabatic dynamic +adiabatic elimination +adiabatic evolution +adiabatic flame temperature +adiabatic invariance +adiabatic invariant +adiabatic passage +adiabatic quantum algorithm +adiabatic quantum computation +adiabatic quantum computing +adiabatic quantum optimization +adiabatic theorem +adiabatic transport +adiabatically eliminated +adinkra symbol +adjacency matrix +admittance +ads/cft correspondence +advanced green function for wave equation +advanced potential +advanced tactical laser +affine gauge theory +affine lie algebra +afshar experiment +aharonov bohm effect +aharonov bohm phase +aharonov bohm potential +aharonov casher effect +airy beam +airy disk +airy integral +aklt model +akns system +alfven velocities +alfven velocity +alfven wave +algebraic holography +algebraic quantum field theory +algorithmic cooling +alikhanian alikhanov spectrometer +alkali atom +alkali metal atom +all gas phase iodine laser +all optical switching +alpha particle spectroscopy +alpha particle x ray spectrometer +alpha process +alpine planetary boundary layer +american superconductor +ampac +amperes law +amplified spontaneous emission +amplitude amplification +amplitude damping channel +amplitude modulation +amplitude of low frequency fluctuation +amplituhedron +amsterdam density functional +an exceptionally simple theory of everything +anaem +analytic continuation +analytic properties of the index of refraction +analytical regularization +anatoly dyatlov +ancilla bit +ancillary qubit +ancillary system +anderson impurity model +anderson localisation +anderson localization +anderson model +anderson transition +andersons rule +andreev reflection +angle resolved photoemission spectroscopy +angles of incidence +angular distribution of radiation +angular momentum +angular momentum coupling +angular momentum diagram +angular momentum operator +angular velocities +angular velocity +anharmonic oscillator +anisotropy parameter +annihilation +annihilation radiation +annni model +annular dark field imaging +anomalon +anomalous diffusion +anomalous dispersion +anomalous magnetic dipole moment +anomalous magnetic moment +anomaly +anomaly matching condition +antenna in wave guide +anti stokes raman spectroscopy +anti symmetric operator +anti zeno effect +antiferromagnetism +antifreeze protein +antihydrogen +antimatter +antimatter comet +antimatter gravity measurement +antimatter rocket +antimatter tests of lorentz violation +antimatter weapon +antineutron +antiparticle +antiproton +antiproton decelerator +antiprotonic helium +antisymmetric exchange +antisymmetrizer +anyon +anyonic excitation +apparent paradox +apparent violation +applied radiochemistry +applied spectroscopy +approximate quantum error correction +approximation scheme +arago spot +arc fusion reactor +argon fluoride laser +argonaut class reactor +argus laser +arnolds cat map +aromatic ring current +arrival time +arrow of time +arthrobacter +arthrobacter antarcticus +arthrobacter chlorophenolicus +arthrobacter cryotolerans +arthrobacter cumminsii +arthrobacter flavus +arthrobacter livingstonensis +arthrobacter roseus +arthrobacter russicus +arthrobacter soli +arthrobacter subterraneus +arthrobacter woluwensis +arthur komar +arthur maitland +artificial atom +artificial disintegration +artificial insemination +artificial intelligence +ashtekar variable +associated legendre function +associated legendre polynomial +asterix iv laser +astro comb +astronomical spectroscopy +asymmetric entanglement +asymmetrically entangled quantum state +asymmetry measure +asymptotic entanglement +asymptotic freedom +asymptotic iteration method +asymptotic limit +asymptotic regime +asymptotic safety in quantum gravity +atiyah jones conjecture +atmospheric diffraction +atmospheric lidar +atmospheric optics ray tracing code +atmospheric turbulence +atom +atom atom interaction +atom beam +atom bose einstein condensate +atom cavity system +atom center of mass motion +atom chip +atom clock +atom coherence +atom decay +atom dynamic +atom entanglement +atom excitation +atom field coupling +atom field entanglement +atom field interaction +atom force microscope +atom gas +atom hydrogen +atom interacting +atom interaction +atom interferometry +atom inversion +atom laser +atom level +atom memory +atom motion +atom nuclei +atom optic +atom photon entanglement +atom position +atom quantum memory +atom qubit +atom resonance +atom spontaneous emission +atom transition +atom transition frequency +atom vibration +atomic absorption spectroscopy +atomic and molecular astrophysic +atomic decay rate +atomic electron transition +atomic emission spectroscopy +atomic form factor +atomic heraldry +atomic internal degree +atomic mass +atomic mass unit +atomic nucleus +atomic number +atomic operator +atomic optic +atomic orbital +atomic physic +atomic recoil +atomic spacing +atomic spectroscopy +atomic theory +atomic unit +atomistix toolkit +atomistix virtual nanolab +atoms in molecule +attenuated total reflectance +attenuation coefficient +attenuation in optical fiber +attenuation length +attosecond +attosecond chronoscopy +audience scanning +aufbau principle +auger architectomic +auger effect +auger electron spectroscopy +austin model +autler townes effect +autler townes splitting +autoignition temperature +autoionization +automated mineralogy +autonomous landing hazard avoidance technology +auxiliary field +auxiliary field monte carlo +auxiliary qubit +auxiliary system +avalanche diode +averaging procedure +avoided crossing +avoided level crossing +awpp +axial vector +axino +axiomatic quantum field theory +axion +azeotrope +azimuthal quantum number +azimuthal symmetry +b integral +b meson +b tagging +b0 +babinets principle +back action +back action evasion +background field +background field method +background noise +bad cavity limit +baecklund transform +bakers map +balanced homodyne detection +ballistic conduction +ballistic conduction in single walled carbon nanotube +banach space +bancroft point +band bending +band diagram +band emission +band gap +band head +band offset +band structure +bandwidth limited pulse +banks zaks fixed point +bare mass +bargmann state +bargmann wigner equation +bargmanns limit +bariloche atomic centre +barrett crane model +baryon +baryon acoustic oscillation +baryon number +baryonic dark matter +basis set +basis set superposition error +basis state +basis vector +batalin vilkovisky formalism +bathochromic shift +bayesian approach +bayesian estimate +bb84 +bb84 protocol +bcfw recursion +bcs theory +beam dump +beam expander +beam homogenizer +beam parameter product +beam powered propulsion +beam splitter +beans critical state model +bechgaard salt +beer lambert law +bekenstein bound +belavkin equation +bell chsh inequalities +bell chsh inequality +bell diagonal state +bell experiment +bell inequalities +bell inequality +bell inequality violation +bell kochen specker theorem +bell measurement +bell nonlocality +bell operator +bell pair +bell state +bell state measurement +bell test +bell test experiment +bell theorem +bell type inequalities +bell type inequality +bell violation +belle ii detector +bells theorem +benchtop nuclear magnetic resonance spectrometer +beneath the city street +benesi hildebrand method +bennetts law +berezin integral +berezinian +beriev a 60 +berry connection and curvature +berry curvature +berry phase +bessel beam +bessel equation +bessel function +bessels differential equation +bessels inequalities +bessels inequality +beta decay +beta decay stable isobar +beta function +beta m +betatron +bethe ansatz +bethe feynman formula +bethe formula +bethe heitler formula +bethe lattice +bethe salpeter equation +bf model +bhabha scattering +bht algorithm +biexciton +big bang +big bang nucleosynthesis +bigdft +biham middleton levine traffic model +bilepton +binary collision approximation +binding energy +biocentric universe +biological half life +biological small angle scattering +biological system +biomedical spectroscopy +biot and savart law +biot savart law +bipartite case +bipartite correlation +bipartite entanglement +bipartite gaussian state +bipartite mixed state +bipartite pure state +bipartite quantum state +bipartite quantum system +bipartite state +bipartite system +biphoton +bipolaron +birefringence of the ionosphere +birge sponer method +bismuth strontium calcium copper oxide +bispinor +bistability +bistatic cross section +bistochastic quantum channel +bit commitment +bit error rate +bit string physic +bks theory +black body radiation +black hole +black hole complementarity +black hole electron +black hole entropy +black hole evaporation +black hole information paradox +black hole physic +black hole thermodynamic +black star +blackbody radiation +blasius boundary layer +blind quantum computation +bloch ball +bloch equation +bloch grueneisen temperature +bloch oscillation +bloch spectrum +bloch sphere +bloch vector +bloch wave +blood irradiation therapy +blown flap +blue laser +blueshift +bmt equation for spin +boeing laser avenger +boeing yal 1 +bogdanov map +bogoliubov causality condition +bogoliubov inner product +bogoliubov parasyuk theorem +bogoliubov transformation +bogomolnyi prasad sommerfield bound +bogomolnyi prasad sommerfield state +bohmian interpretation +bohmian mechanic +bohmian quantum mechanics +bohmian trajectory +bohr einstein debate +bohr magneton +bohr model +bohr radius +bohrs classical formula +boiling +boiling chip +boiling point +bolometer +boltzmann distribution +boltzmanns distribution law +bond fluctuation model +bond hardening +bond softening +boolean network +boost momentum +boost parameter +borchers algebra +borel summation +born approximation +born huang approximation +born infeld nonlinear electrodynamic +born markov approximation +born oppenheimer approximation +born probability rule +born rule +born serie +borromean nucleus +bose condensate +bose einstein condensate +bose einstein condensation +bose einstein statistic +bose gas +bose hubbard hamiltonian +bose hubbard model +bosenova +boson +boson sampling +bosonic atom +bosonic environment +bosonic field +bosonic gaussian channel +bosonic particle +bosonic system +bosonization +bottom eta meson +bottom quark +bottomness +bound electron dynamic +bound entangled +bound entanglement +bound state +bound state energy +bound state problem +bound state solution +bound states in wave guide +boundary condition +boundary conformal field theory +boundary layer +boundary layer control +boundary layer suction +boundary layer thickness +boundary value problem +bounded error +bounded operator +boussos holographic bound +bpst instanton +bqp +bra ket notation +bragg plane +bragg scattering +braggs law +braid group +braid statistic +braided vector space +branch point +branching fraction +brauer wall group +braunstein ghosh severini entropy +breast mri +breathalyzer +breit equation +breit interaction +bremsstrahlung +bretherton equation +brewsters angle +brian josephson +brillouin precursor +brillouin scattering +brillouin spectroscopy +brillouin zone +brillouins theorem +broadband +broadband acoustic resonance dissolution spectroscopy +broken symmetry +brownian motion +brownian particle +brst quantization +brus equation +btz black hole +bubble fusion +bubble point +buffer gas +bulk richardson number +bullough dodd model +bunch davies vacuum +bures distance +bures metric +butcher group +byers yang theorem +c parity +c rot gate +c symmetry +c theorem +c70 molecule +cabibbo kobayashi maskawa matrix +cadpac +caesium cadmium bromide +cal pt symmetry +caldeira leggett model +calkin correspondence +callan symanzik equation +calligraphic projection +calogero conjecture +caloron +camassa holm equation +canadian penning trap mass spectrometer +canonical commutation relation +canonical ensemble +canonical momentum +canonical quantization +canonical quantum gravity +canonical state +canonical stress tensor +canonical transformation +canonical variable +capacitance +capacitive coupling +capacitive resistance +capacitively coupled +capacity region +capillary rise method +capping inversion +car parrinello method +car parrinello molecular dynamic +carbon 13 nmr satellite +carbon 13 nuclear magnetic resonance +carbon burning process +carbon dioxide +carbon dioxide laser +carbon nanotube +cardiac magnetic resonance imaging +cardiac magnetic resonance imaging perfusion +cardiac ventriculography +cardy formula +carnobacterium pleistocenium +carnot efficiency +caroline herzenberg +carrier scattering +cartan decomposition +cartesian coordinate +cascading gauge theory +casimir effect +casimir energy +casimir force +casimir force measurement +casimir free energy +casimir interaction +casimir interaction energy +casimir polder force +casimir polder interaction +casimir polder potential +casimir pressure +castep +cat state +catastrophic optical damage +categorical quantum mechanic +category of finite dimensional hilbert space +category theory +cathode ray +cathodoluminescence +cauchy boundary condition +cauchy hadamard formula +cauchy riemann differential equation +cauchy schwarz inequalities +cauchy schwarz inequality +cauchys integral formula +cauchys theorem +causal dynamical triangulation +causal fermion system +causal green function +causal interpretation +causal patch +causal perturbation theory +causal set +causal structure +causality +cavendishs apparatu +cavities +cavity +cavity cooling +cavity decay +cavity field +cavity frequency +cavity loss +cavity mirror +cavity mode +cavity optomechanic +cavity optomechanical system +cavity photon +cavity qed +cavity qed system +cavity quantum electrodynamic +cavity resonance +cavity ring down spectroscopy +cayley graph +ccr and car algebras +cecoin5 +cells alive system +cellular potts model +center fed antenna +center of inversion +center of mass +center of mass motion +center vortex +central charge +central field approximation +central limit theorem +central potential +central role +central spin +centre for magnetic resonance investigation +cesium atom +cghs model +ch inequalities +ch inequality +chain of circle procedure +chameleon field +chameleon particle +channel capacity +channel loss +channel state duality +chaotic behaviour +chaotic dynamic +chaotic motion +chaotic regime +chaotic scattering +chaotic state +chaotic system +chaplygin gas model +characteristic time +charge +charge carrier +charge conjugation +charge conservation +charge contrast imaging +charge density +charge density wave +charge noise +charge ordering +charge qubit +charge radius +charge state +charge transfer insulator +charge transport +charged current +charged particle +charged particle equilibrium +chargino +charm +charm quark +charmed baryon +charmed eta meson +chebyshev polynomial +chelpg +chemical bond +chemical computer +chemical imaging +chemical laser +chemical oxygen iodine laser +chemical potential +chemical shift +chemical shift index +chemiluminescence +chep conference +cherenkov angle +cherenkov radiation +chern number +chern simons theory +chip integration +chiral algebra +chiral anomaly +chiral gauge theory +chiral model +chiral molecule +chiral perturbation theory +chiral potts curve +chiral potts model +chiral superfield +chiral symmetry +chiral symmetry breaking +chirality +chirped pulse amplification +chladnis law +chromism +chromo weibel instability +chronon +chsh bell inequalities +chsh bell inequality +chsh inequalities +chsh inequality +cidnp +circuit model +circuit qed +circuit quantum electrodynamic +circular current loop +circular polarization +circularly polarized wave +cladding +claes goeran granqvist +clas detector +classical action +classical algorithm +classical analogue +classical and quantum conductivity +classical approach +classical behaviour +classical bit +classical bound +classical capacity +classical case +classical channel +classical chaos +classical communication +classical complexity +classical computation +classical computing +classical concept +classical correlation +classical counterpart +classical cryptography +classical diffusion +classical domain +classical dynamic +classical electrodynamic +classical electron radius +classical entanglement +classical environment +classical equation +classical evolution +classical field +classical field theory +classical hamiltonian +classical heisenberg model +classical information channel +classical information theory +classical interpretation +classical level +classical light +classical limit +classical logic +classical measurement +classical mechanic +classical memory +classical method +classical mixture +classical model +classical motion +classical noise +classical notion +classical object +classical observable +classical optic +classical orbit +classical particle +classical phase space +classical physic +classical post processing +classical probability distribution +classical probability theory +classical problem +classical property +classical protocol +classical quantum channel +classical quantum correspondence +classical random walk +classical regime +classical resource +classical result +classical setting +classical simulation +classical solution +classical state +classical statistic +classical statistical mechanic +classical strategy +classical system +classical theory +classical trajectory +classical transition +classical treatment +classical variable +classical version +classical xy model +classically chaotic system +classically correlated state +classically forbidden region +clauser horne inequalities +clauser horne inequality +clauser horne shimony holt inequalities +clauser horne shimony holt inequality +clausius mossotti formula +clausius mossotti relation +clear physical interpretation +clebsch gordan coefficient +clifford algebra +clifford gate +clifford group +clock +cloning +cloning machine +closed expression +closed form expression +closed form solution +closed formula +closed loop +closed quantum system +closed system +closed timelike curve +cloud aerosol transport system +cloud quantum computing +cluster decomposition theorem +cluster expansion approach +cluster state +cmos +cndo/2 +cno cycle +cnot +coanda effect +coarse graining +coaxial cable +coaxial line +cobalt 59 +coblentz society +cobordism hypothesis +coefficient matrix +coffee ring effect +cognitive system +coherence +coherence function +coherence length +coherence property +coherence representation +coherence time +coherent addition +coherent anti stokes raman spectroscopy +coherent attack +coherent backscattering +coherent control +coherent coupling +coherent destruction +coherent diffraction imaging +coherent dynamic +coherent evolution +coherent excitation +coherent feedback +coherent field +coherent information +coherent interaction +coherent light +coherent manipulation +coherent oscillation +coherent perfect absorber +coherent population trapping +coherent spectroscopy +coherent state +coherent superposition +coherent transfer +coherent transport +coherent tunneling +coherently control +coherently manipulated +coil +coin operator +coincidence counting +coincidence detection +coincidence measurement +coincidence probability +cold atom ensemble +cold atom experiment +cold atom gas +cold atom laboratory +cold atom system +cold dark matter +cold fission +cold fusion +cold polar molecule +cold trapped ion +cold vapour +coleman mandula theorem +coleman weinberg potential +collaborative computing project for nmr +collapse model +collection efficiency +collective atomic excitation +collective attack +collective behaviour +collective coupling +collective decoherence +collective dephasing +collective effect +collective excitation +collective measurement +collective mode +collective motion +collective noise +collective spin +collective state +collinear quasi phase matching +collision energy +collision induced absorption and emission +collision problem +collisional broadening limit +collisional decoherence +collisional excitation +collisional process +colocalization benchmark source +color center +color charge +color code +color confinement +color flavor locking +color glass condensate +color superconductivity +colored noise +colored particle in cell +colorimeter +colossal magnetoresistance +combinatorial hierarchy +combining dimensions +common bath +common reservoir +communicating party +communication capacity +communication channel +communication complexity +communication cost +communication protocol +commutation relation +commutator subspace +commuting observable +commuting operator +compact fluorescent lamp +compact operator on hilbert space +compact quantum group +compact reconnaissance imaging spectrometer +compacton +comparative analysis +comparative study +complementarity +complementarity principle +complementarity relation +complementary observable +complementary screen +complete active space +complete graph +complete information +complete positivity +complete set +complete set of commuting observable +complete set of function +complete solution +complete system +complete theory +completely mixed state +completely positive map +completely positive trace preserving map +completely suppressed +completeness +completeness relation +complex conjugation +complex differentiable +complex function +complex hadamard matrix +complex hilbert space +complex network +complex number +complex oxide +complex plane +complex potential +complex quantum system +complexity theory +composite fermion +composite field +composite gravity +composite pulse +composite quantum system +composite reaction texturing +composite system +compound system +compressed sensing +compton edge +compton effect +compton scattering +compton wavelength +computation time +computational bases +computational basis +computational complexity +computational cost +computational fluid dynamic +computational power +computational problem +computational resource +compute analytically +computed efficiently +computer algorithm +computer science +computer simulation +concurrence +condensation +condensed matter +condensed matter physic +condensed matter system +conditional dynamic +conditional entropy +conditional measurement +conditional preparation +conditional probability +conditional quantum entropy +conditional state +conductance quantum +conduction band +conduction electron +conduction in a moving medium +conductivity +conductivity near the percolation threshold +conductor +configuration interaction +configuration space +configuration state function +confining potential +conformal bootstrap +conformal family +conformal field theory +conformal loop ensemble +conformal supergravity +conformal symmetry +congruent melting +conical hole or point +conical intersection +conjugate bases +conjugate basis +conjugate momenta +conjugate variable +conquest +conscious observer +conservation law +conservation law of energy +conservation law of momentum +conservation of angular momentum +conserved quantity +consistency condition +consistent description +consistent histories +consistent history approach +consistent theory +constant diffusion process +constant magnetic field +constant time +constituent quark +constituive equation +constitutive relation +constrained system +constraint algebra +constraints imposed +constructive interference +constructive quantum field theory +contact interaction +contact process +contextuality +continuity equation +continuous cooling transformation +continuous degree +continuous family +continuous limit +continuous observation +continuous quantum computation +continuous quantum measurement +continuous quantum variable +continuous scan laser doppler vibrometry +continuous set +continuous slowing down approximation range +continuous spectra +continuous spectrum +continuous spontaneous localisation +continuous symmetry +continuous time quantum walk +continuous variable +continuous variable entanglement +continuous variable quantum cryptography +continuous variable quantum key distribution +continuous variable quantum system +continuous variable quantum teleportation +continuous variable regime +continuous variable state +continuous variable system +continuous variable teleportation +continuous wave +continuum limit +continuum percolation theory +contraction +contractive state +contravariant vector +control error +control field +control parameter +control pulse +control qubit +control scheme +control theory +controlled disorder +controlled generation +controlled not gate +controlled phase gate +controlled z +convective boundary layer +convective derivative +conventional superconductor +conventional transmission electron microscope +convergence in square mean +convergence property +conversion efficiency +conversion electron moessbauer spectroscopy +conversion process +convex combination +convex hull +convex optimization +convex set +convolution theorem +cooling process +cooper pair +cooper pair box +cooperative effect +coorbit theory +copenhagen +copenhagen interpretation +copper vapor laser +coprecipitation +core charge +core electron +core excited shape resonance +corner transfer matrix +coronal radiative loss +correlated noise +correlated photon +correlated photon pair +correlated state +correlation +correlation effect +correlation energy +correlation function +correlation inequalities +correlation inequality +correlation length +correlation matrix +correlation property +correlation time +correspondence principle +correspondence rule +coset construction +cosmic acceleration +cosmic microwave background +cosmic ray spallation +cosmic string spacetime +cosmological constant problem +cosmological interpretation +cost function +coster kronig transition +cotlar stein lemma +coulomb barrier +coulomb blockade +coulomb collision +coulomb excitation +coulomb explosion +coulomb field +coulomb force +coulomb gauge +coulomb interaction +coulomb law +coulomb operator +coulomb potential +coulomb repulsion +coulomb scattering state +coulombs law +coumarin +counter rotating term +counterfactual definiteness +counterfactual quantum computation +counting statistic +coupled bose einstein condensate +coupled cavity array +coupled cluster +coupled dynamic +coupled equation +coupled harmonic oscillator +coupled oscillator +coupled quantum dot +coupled qubit +coupled spin +coupled system +coupling constant +coupling efficiency +coupling field +coupling parameter +coupling strength +covafiant lagrangian formalism +covalent superconductor +covariance +covariance mapping +covariance matrix +cp violation +cp2k +cpt symmetry +craic technologies +cramer rao bound +creation and annihilation operator +critical behaviour +critical coupling +critical exponent +critical field +critical fluctuation +critical line +critical opalescence +critical phenomena +critical point +critical property +critical radius +critical system +critical temperature +critical value +critical variable +cross kerr nonlinearity +cross phase modulation +cross section +crossed crystal configuration +crowther criterion +cryochemistry +cryoelectronic +cryogenic particle detector +cryogenic temperature +cryoneurolysis +cryoneuromodulation +cryophorus +cryoprotectant +cryoscopic constant +cryostasis +cryosurgery +cryozoa +cryptographic protocol +crypton +crystal base +crystal momentum +crystal oven +crystallization +crystallization of polymer +css code +cubic harmonic +cubical atom +cumulant expansion +cuprate +curie tempenature +curie temperature +curies law +current algebra +current density +current density imaging +current four vector +current quark +curtright field +curvaton +curvature of space +curved space +curved space time +custodial symmetry +cutoff +cyclic evolution +cyclic group +cyclic operation +cyclops laser +cylindrical capacitor +c�as lemma +d dimensional hilbert space +d level system +d meson +d term +d wave system +d wave two +dacapo +dalitz plot +damped harmonic oscillator +damped lyman alpha system +damped oscillator +damping rate +damping time +dangerously irrelevant operator +darboux transformation +dark count +dark count rate +dark energy +dark energy star +dark matter +dark photon +dark state +dark state polariton +dasar +data compression +data processing inequalities +data processing inequality +david deutsch +davisson germer experiment +davydov soliton +dbar problem +de broglie bohm interpretation +de broglie bohm theory +de broglie relation +de broglie wavelength +de haas van alphen effect +de sitter invariant special relativity +de sitter spacetime +de wolf +dead time +debye waller factor +decay channel +decay energy +decay exponentially +decay process +decay product +decay rate +decay time +decision making +decision problem +decision theory +decoherence +decoherence channel +decoherence control +decoherence dynamic +decoherence effect +decoherence factor +decoherence free subspace +decoherence functional +decoherence induced +decoherence mechanism +decoherence process +decoherence property +decoherence rate +decoherence suppression +decoherence theory +decoherence time +decoherence time scale +decoherent history approach +deconfinement +decoy state +decoy state method +decoy state quantum key distribution +deelectrification +deelectrification factor +deep inelastic scattering +deep level transient spectroscopy +deferred measurement principle +deflection +deformation parameter +deformation polarization +deformation quantization +degasperis procesi equation +degenerate energy level +degenerate ground state +degenerate operation +degenerate parametric oscillator +degenerate state +degree of coherence +delay time +delayed choice +delayed choice experiment +delayed choice quantum eraser +delayed neutron +delbrueck scattering +delocalized electron +delta baryon +delta potential +dennis m. bushnell +dense coding +density +density fluctuation +density functional theory +density matrix +density matrix renormalization group +density of state +density operator +density profile +dephasing +dephasing channel +dephasing due +dephasing effect +dephasing noise +dephasing rate +dephasing rate sp formula +dephasing time +depletion effect +deployable tactical engagement system +depolarization ratio +depolarizing channel +depolarizing noise +deposition +derivations of the lorentz transformation +derricks theorem +deslandres table +desy +detection efficiency +detection inefficiency +detection loophole +detection scheme +detector efficiency +detector inefficiency +determination of nuclear radii +deterministic entanglement +deterministic generation +deterministic model +deterministic quantum computation +deuterium burning +deuterium nmr +deutsch jozsa algorithm +deutsch jozsa problem +device independent quantum cryptography +dew point +dexter electron transfer +di positronium +diabatic +diamagnetic +diamagnetism +diamond +diatom molecule +dicke effect +dicke model +dicke state +dielectric +dielectric displacement +dielectric function +dielectric media +dielectric medium +dielectric permittivity +dielectric polarization +dielectric property +dielectric spectroscopy +dielectric susceptibility +diffeomorphism constraint +differential cross section +differential dynamic microscopy +differential geometry +differential operator +diffraction +diffraction efficiency +diffraction formalism +diffraction grating +diffraction in time +diffraction limit +diffraction limited system +diffraction pattern +diffraction tomography +diffractive beam splitter +diffuse interstellar band +diffuse reflectance infrared fourier transform +diffuse serie +diffusing wave spectroscopy +diffusion coefficient +diffusion monte carlo +diffusion mri +diffusion process +diffusionless transformation +diffuson +digital fish library +dihydrogen cation +diis +dilaton +dimensional entangled quantum system +dimensional regularization +dimensional transmutation +dinger cat state +dinger equation +diode +diode pumped solid state laser +dipolar bose einstein condensate +dipolar interaction +dipole approximation +dipole blockade +dipole density +dipole dipole interaction +dipole layer +dipole moment +dipole vector +diquark +dirac adjoint +dirac bracket +dirac delta function +dirac electron +dirac equation +dirac equation in curved spacetime +dirac equation in the algebra of physical space +dirac fermion +dirac field +dirac hamiltonian +dirac hole theory +dirac operator +dirac oscillator +dirac particle +dirac point +dirac sea +dirac spectrum +dirac spinor +dirac string +dirac von neumann axiom +diracs relativistic generalization of +direct and indirect band gap +direct quantum chemistry +directed percolation +dirichlet boundary condition +discovery of the neutron +discrete dipole approximation +discrete dipole approximation code +discrete fourier transform +discrete phase space +discrete spectrum +discrete symmetry +discrete time quantum walk +discrete wigner function +disjoint interval +disk laser +disordered system +dispersion force +dispersion relation +dispersion relation for electromagnetic in vacuum +dispersion relation for electromagnetic wave +dispersion relation in a conducting medium +dispersive bistability +dispersive interaction +dispersive media +dispersive medium +dispersive regime +displaced vacuum +displacement current +displacement operator +displacement polarization +dissipation model for extended environment +dissipation rate +dissipative dynamic +dissipative effect +dissipative environment +dissipative preparation +dissipative process +dissipative quantum system +dissipative soliton +dissipative system +distant atom +distant node +distant particle +distillable entanglement +distillation +distillation protocol +distinguishability +distinguishable particle +distribute entanglement +distributed bragg reflector laser +distributed entanglement +distributed feedback laser +distributed multipole analysis +distributed quantum computation +distributed quantum computing +distribution +divergence +dixmier trace +dm interaction +dmol3 +domain wall +domino tiling +doppler cooling +doppler cooling limit +doppler effect +doppler shift +doppler spectroscopy +dose fractionation theorem +double exchange mechanism +double ionization +double quantum dot +double slit +double slit experiment +doublet state +doubly degenerate +dove prism +down converted photon +down quark +drell yan process +dressed state +driven harmonic oscillator +driven system +driving field +driving force +driving frequency +driving laser +dropleton +drude model +drudes formula +dual superconductor model +duanes hypothesis +ductile brittle transition +dunham expansion +durcupan +duru kleinert transformation +dwell time +dyall hamiltonian +dye laser +dym equation +dynamic contrast enhanced mri +dynamic light scattering +dynamic structure factor +dynamical decoupling +dynamical mean field theory +dynamical picture +dynamical reduction +dyon +dyson serie +dzyaloshinskii moriya interaction +early universe +earths field nmr +eavesdropping +eberhard inequalities +eberhard inequality +ebullioscopic constant +eckart condition +eckhaus equation +ecton +edans +eddy covariance +edge mode +edge of the wedge theorem +edge state +efda +effective action +effective atomic number +effective coupling +effective current +effective dynamic +effective field theory +effective hamiltonian +effective interaction +effective magnetic field +effective mass +effective photon mass +effective potential +effective temperature +effective theory +effective voltage +effects of nuclear explosion +efficiency +efficient algorithm +efficient classical simulation +efficient computation +efficient coupling +efficient generation +efimov effect +efimov physic +efimov state +ehrenfest theorem +ehrenfest time +ehrenfest tolman effect +eigenspinor +eigenstate thermalization hypothesis +eigenvalue equation +eigenvalue perturbation +eigenvalue problem +eigenvalue spectrum +eight vertex model +eikonal approximation +einselection +einstein brillouin keller method +einstein coefficient +einstein de haas effect +einstein maxwell dirac equation +einstein podolsky rosen +einstein podolsky rosen correlation +einstein podolsky rosen paradox +einstein podolsky rosen steering +einsteins summation convention +ekman layer +elastic scattering +electret +electric charge +electric dipole spin resonance +electric dipole transition +electric field gradient +electric field intensity +electric field nmr +electric motor +electric potential +electric susceptibility +electrical aerosol spectrometer +electrically detected magnetic resonance +electricoil +electrochromism +electrolaser +electromagnetic absorption by water +electromagnetic cavities +electromagnetic cavity +electromagnetic field strength tensor +electromagnetic interaction +electromagnetic momentum density +electromagnetic radiation +electromagnetic vacuum fluctuation +electromagnetic wave +electromagnetic waves in matter +electromagnetic waves in vacuum +electromagnetically induced grating +electromagnetically induced transparency +electron +electron affinity +electron backscatter diffraction +electron beam additive manufacturing +electron beam freeform fabrication +electron beam furnace +electron beam induced current +electron beam induced deposition +electron beam ion trap +electron beam machining +electron beam physical vapor deposition +electron beam prober +electron beam processing +electron beam technology +electron beam texturing +electron beam tomography +electron bubble +electron configuration +electron correlation +electron degeneracy pressure +electron density +electron diffraction +electron energy loss spectroscopy +electron excitation +electron gas +electron gun +electron hole +electron liquid +electron localization function +electron magnetic circular dichroism +electron magnetic moment +electron multiplier +electron neutrino +electron nuclear double resonance +electron pair +electron paramagnetic resonance +electron phenomenological spectroscopy +electron phonon coupling +electron positron pair production +electron precipitation +electron rest mass +electron scattering +electron shell +electron spectrometer +electron spectroscopy +electron spin +electron spin qubit +electron spin resonance +electron therapy +electron tomography +electron transfer +electron transport +electron tunneling +electron wake +electron wave packet interference +electronic anticoincidence +electronic band structure +electronic correlation +electronic density +electronic energy transfer +electronic excitation +electronic spin +electronic state +electronic structure +electronic transport +electrostatic +electrostatic energy +electrostatic lens +electrostatic nuclear accelerator +electrostatic potential +electrostatic unit +electrostriction +electrothermal feedback +electroweak interaction +electroweak scale +elementary excitation +elementary gate +elementary particle +elitzur vaidman bomb tester +elitzurs theorem +elko field +ellen gleditsch +elliott formula +ellipsometry +elliptic flow +elliptically polarized wave +emc effect +emission spectra +emission spectrum +empty lattice approximation +encoded qubit +endovenous laser treatment +energy band +energy catalyzer +energy conservation +energy density of the electric field +energy dispersive x ray diffraction +energy dispersive x ray spectroscopy +energy flow +energy level +energy level splitting +energy momentum 4 vector +energy momentum tensor +energy operator +energy transfer +energy transfer upconversion +engineering diffraction +englert greenberger yasin duality relation +ensemble interpretation +entangled atom +entangled coherence +entangled light +entangled pair +entangled particle +entangled photon +entangled photon pair +entangled quantum state +entangled quantum system +entangled qubit +entangled qutrit +entangled state +entangled system +entanglement +entanglement assisted capacity +entanglement assisted classical capacity +entanglement breaking channel +entanglement class +entanglement classification +entanglement concentration +entanglement cost +entanglement creation +entanglement criteria +entanglement criterion +entanglement decay +entanglement detection +entanglement dimensionality +entanglement distillation +entanglement distribution +entanglement entropy +entanglement evolution +entanglement fidelity +entanglement generation +entanglement manipulation +entanglement measure +entanglement monotone +entanglement of formation +entanglement purification +entanglement renormalization +entanglement representation +entanglement resource +entanglement sharing +entanglement spectra +entanglement spectrum +entanglement sudden death +entanglement swapping +entanglement teleportation +entanglement transfer +entanglement transformation +entanglement witness +entangling capability +entangling gate +entangling power +entropic inequalities +entropic inequality +entropic quantity +entropic uncertainty +entropic uncertainty relation +entropy exchange +entropy increase +entropy inequality +entropy of black hole +entropy of entanglement +entropy production +environment +environment induced decoherence +environmental decoherence +environmental degree +environmental effect +environmental interaction +environmental noise +environmental scanning electron microscope +enyi entropy +ep quantum mechanic +epcard +epistemological letter +epr argument +epr correlation +epr entanglement +epr experiment +epr pair +epr paradox +epr state +epsilon radiation +eqp +equilibrium state +equivalence class +equivalence principle +equivalent width +er:yag laser +er=epr +ergodic hypothesis +ermakov lewis invariant +ernst angle +error ellipse +essential singularity +eta meson +eternally collapsing object +ether drift experiment +euclidean quantum gravity +euclidean space +eugene mallove +eugene podkletnov +euler heisenberg lagrangian +euler lagrange equation +euler maclaurin formula +european spallation source +eutectic +eutectic system +eutectoid +evanescent field +evanescent mode +evanescent wave +evaporation +evaporative cooling +even and odd atomic nuclei +event horizon +event symmetry +everett interpretation +evershed effect +evolution equation +evolution operator +evolution time +ex situ conservation +exablate +exchange coupling +exchange energy +exchange force +exchange interaction +exchange operator +exchange symmetry +excimer laser +excitation +excitation energy transfer +excitation function +excitation population +excitation probability +excitation spectra +excitation spectrum +excited atom +excited state +exciton +exciton polariton +excitonic state +exclusion principle +exclusive correlation spectroscopy +exoelectron emission +exotic atom +exotic baryon +exotic hadron +exotic meson +exotic particle +expanding universe +expectation value +exponential decay law +exponential scaling +exponential separation +exponential speed +exponential speedup +extended hueckel method +extended interaction oscillator +external beam radiotherapy +extinction coefficient +extinction ratio +extraordinary magnetoresistance +eye surgery +fabry perot cavities +fabry perot cavity +factoring algorithm +factorization method +faddeev equation +faddeev popov ghost +fair sampling +fair sampling assumption +fajans paneth hahn law +fano factor +fano resonance +far infrared laser +faraday effect +faradays law of induction +fast fission +fast low angle shot magnetic resonance imaging +fast neutron therapy +faster than light +fault current limiter +fault tolerance +fault tolerant quantum computation +fault tolerant quantum computing +fayet iliopoulos d term +fazia +feedback control +feedback loop +fellgetts advantage +femtochemistry +femtosecond pulse shaping +fenna matthews olson complex +fermi arc +fermi dirac statistic +fermi energy +fermi gas +fermi heap and fermi hole +fermi level +fermi liquid theory +fermi motion +fermi point +fermi resonance +fermi sea +fermi statistic +fermi surface +fermi surface of superconducting cuprate +fermion +fermion doubling +fermion system +fermionic atom +fermionic condensate +fermionic field +fermionic system +fermis golden rule +fermis interaction +ferrimagnetism +ferroelectric crystal +ferroelectrics +ferromagnetic phase +ferromagnetic superconductor +ferromagnetics +ferromagnetism +feshbach fano partitioning +feshbach resonance +few body system +feynman checkerboard +feynman diagram +feynman heaviside expressions for field +feynman parametrization +feynman slash notation +fiber based wavelength division multiplexer +fiber bragg grating +fiber disk laser +fiber laser +fiber source +fibonacci anyon +fibre bundle +fidelity decay +fidelity of quantum state +fidelity susceptibility +field approximation +field effect +field electron emission +field equation +field strength tensor +field tensor +fierz identity +filling factor +fine structure +fine structure constant +fine structure splitting +finite amount +finite automata +finite bandwidth +finite conductivity +finite difference method +finite dimension +finite dimensional case +finite dimensional hilbert space +finite dimensional quantum system +finite dimensional representation +finite dimensional system +finite element analysis +finite potential well +firestrike +firewall +first order coherence +first order correlation function +first quantization +fisher information +fission product +fission product pilot plant +fitzgerald lorentz contraction +fixed orbit +fizeaus experiment +flack parameter +flash +flash freezing +flash point +flat spacetime +flavour +flip angle +floer homology +floquet state +floquet theory +flow separation +fluctuation dissipation theorem +fluctuation theorem +fluctuation x ray scattering +fluid analogs in quantum mechanic +fluid attenuated inversion recovery +fluorescein +fluorescence correlation spectroscopy +fluorescence cross correlation spectroscopy +fluorescence intensity decay shape microscopy +fluorescence intermittency +fluorescence loss in photobleaching +fluorescence microscope +fluorescence recovery after photobleaching +fluorescence spectroscopy +fluorescence spectrum +fluorescent tag +fluorine 19 nuclear magnetic resonance +fluorometer +flux footprint +flux pinning +flux pumping +flux quantization +flux quantum +flux qubit +fluxon +fmo complex +fmri adaptation +fock matrix +fock space +fock state +focused ion beam +foerster coupling +fokker planck equation +foldy wouthuysen transformation +forbidden mechanism +force density +force equation +force spectroscopy +form factor +formation of the universe +forward scattering amplitude +four dimensional laplacian operator +four divergence +four fermion interaction +four gradient +four level system +four potential +four tensor +four vector +four wave mixing +fourier analysis +fourier bessel serie +fourier domain mode locking +fourier integral +fourier serie +fourier spherical bessel series +fourier transform +fourier transform infrared spectroscopy +fourier transform spectroscopy +fourth order +fractional anisotropy +fractional crystallization +fractional freezing +fractional quantum hall effect +fractional quantum hall state +fractional quantum mechanic +fractional revival +fractional schroedinger equation +fractional statistic +fractional supersymmetry +fractional vortices +fracton +franck condon principle +franck hertz experiment +franco british nuclear forum +fraunhofer diffraction +fraunhofer distance +fraunhofer line +fraxel +fredholm module +fredkin gate +free carrier absorption +free electron +free electron laser +free electron model +free energy +free evolution +free fall +free fall atomic model +free fermion +free field +free induction decay +free motion +free orbit experiment +free particle +free space +free space optical communication +freeze thaw resistance +freezing +freezing point +freezing point depression +frequency addition source of optical radiation +frequency comb +frequency conversion +frequency dependence +frequency dependence of the conductivity +frequency dependence of the index of refraction +frequency detuning +frequency difference +frequency distribution +frequency distribution of radiation +frequency domain +frequency entanglement +frequency modulation +frequency range +frequency resolved optical gating +frequency shift +frequency spectrum +frequent measurement +fresnel +fresnel diffraction +fresnel number +fresnel zone +fresnels formula +fresnels rhombus +freund rubin compactification +friedmann robertson walker metric +fringe visibility +frobenius manifold +froissart stora equation +frontier molecular orbital theory +frozen ark +frozen mirror image method +frozen zoo +frw/cft duality +fr�edericksz transition +fubini study metric +fujikawa method +fulde ferrell larkin ovchinnikov phase +full configuration interaction +full spectral imaging +functional integral +functional integration +functional magnetic resonance imaging +functional magnetic resonance spectroscopy of the brain +functional renormalization group +fundamental serie +fusenet +fusion ignition +fusion rule +futbolin +fuzzy cold dark matter +fuzzy sphere +g factor +g strain +gadolinium yttrium garnet +gain medium +gain switching +galerkins method +galilean invariance +galilean relativity +galilean transformation +galilei group +game theory +gamma counter +gamma matrices +gamma ray spectrometer +gammasphere +gamow factor +gamow teller transition +gans theory +gary taubes +gas dynamic laser +gas electron diffraction +gas laser +gasochromism +gate fidelity +gate operation +gate time +gate voltage +gauge anomaly +gauge boson +gauge covariant derivative +gauge field +gauge fixing +gauge gravitation theory +gauge group +gauge invariance +gauge invariant +gauge potential +gauge principle +gauge symmetry +gauge theory +gauge theory gravity +gauge transformation +gauged supergravity +gaugino +gaugino condensation +gauss sum +gauss theorem +gaussian approximation +gaussian beam +gaussian broadening +gaussian channel +gaussian entanglement +gaussian fixed point +gaussian measurement +gaussian method +gaussian noise +gaussian orbital +gaussian pillbox +gaussian position +gaussian quantum discord +gaussian quantum monte carlo +gaussian state +gaussian system of unit +gausson +gausss law +geiger marsden experiment +geiger nuttall law +geisa +gekko xii +gel dosimetry +gelfand naimark segal construction +gelfand zeitlin integrable system +gell mann and low theorem +gell mann matrices +gell mann okubo mass formula +general antiparticle spectrometer +general covariant transformation +general criterion +general integral theorem +general relativity +generalized abel plana formula +generalized bell inequalities +generalized bell inequality +generalized bell state +generalized coherence +generalized entanglement +generalized entropy +generalized fourier coefficient +generalized gibbs ensemble +generalized index of refraction +generalized measurement +generalized momenta +generalized ohms law +generalized permittivity +generalized probabilistic theory +generalized quantum measurement +generalized relative entropy +generalized uncertainty principle +generalized uncertainty relation +generalized valence bond +generate entanglement +generated entanglement +generating entanglement +generating function +generators of the lorentz group +genetic algorithm +genuine multipartite entanglement +genuine tripartite entanglement +geometric algebra +geometric approach +geometric description +geometric discord +geometric entanglement +geometric formulation +geometric interpretation +geometric measure +geometric measure of entanglement +geometric phase +geometric phase analysis +geometric quantization +geometric quantum discord +geometric structure +geometrical interpretation +geometrical property +geometrical structure +geonium atom +gerchberg saxton algorithm +gf method +ghirardi rimini weber theory +ghost imaging +ghz experiment +ghz state +ghz theorem +giant magnetoresistance +giant resonance +gibbs state +gim mechanism +gingerbreadman map +ginzburg landau theory +glashow resonance +glass transition +glauber sudarshan p representation +gleasons theorem +global anomaly +global entanglement +global symmetry +global warming potential +globar +glossary of quantum philosophy +glueball +gluex +gluino +gluon +gluon condensate +gluon field +gluon field strength tensor +goertler vortices +golay cell +golden rule +goldstino +goldstone boson +good quantum number +goos hanchen effect +gopakumar vafa invariant +gottesman knill theorem +graded manifold +gradient enhanced nmr spectroscopy +graph isomorphism problem +graph state +graph theory +graphane +graphene +graphene quantum dot +graphene sheet +graphical unitary group approach +graphite furnace atomic absorption +graphite like zno nanostructure +grassmann integral +grassmann number +gravastar +graviphoton +graviscalar +gravitational anomaly +gravitational effect +gravitational field +gravitational instanton +gravitational interaction +gravitational interaction of antimatter +gravitational radiation +gravitational redshift +gravitational self interaction +gravitational wave +gravitational wave detection +gravitational wave detector +gravitational wave interferometer +gravitino +graviton +gravity +gravity wave +gravity wave interferometer +gray tree frog +grazing incidence small angle scattering +green function +green function in electrostatic +greenberger horne zeilinger +greenberger horne zeilinger state +greenberger horne zeilinger theorem +greens first identity +greens function +greens reciprocation theorem +greens theorem +greg fahy +greninger chart +grenouille +gribov ambiguity +grimm sommerfeld rule +grism +gross neveu model +gross pitaevskii equation +grotrian diagram +ground state +ground state atom +ground state cooling +ground state degeneracy +ground state energy +ground state entanglement +ground state fidelity +ground state phase diagram +ground state property +ground state wave function +group field theory +group hopf algebra +group theory +group velocities +group velocity +grover algorithm +grover search +grover walk +grovers algorithm +grylloblatta campodeiformis +grylloblattidae +gsi anomaly +guided mode +gupta bleuler formalism +gutmann beckett method +gw approximation +gyration frequency +gyration radius +gyromagnetic ratio +gyrotropic media +gyrovector space +haag lopuszanski sohnius theorem +haags theorem +haar measure +hadamard gate +hadamard matrix +hadamard space +hadamard transform +hadamard walk +hadron +hadron spectroscopy +hadronization +hafnium controversy +hagedorn temperature +hagen rubens relation +hair removal +half metal +hall effect +halo nucleus +halochromism +halomonas +hamilton jacobi einstein equation +hamilton jacobi equation +hamiltonian +hamiltonian constraint +hamiltonian constraint of lqg +hamiltonian describing +hamiltonian description +hamiltonian dynamic +hamiltonian evolution +hamiltonian field theory +hamiltonian formalism +hamiltonian formulation +hamiltonian function +hamiltonian lattice gauge theory +hamiltonian matrix +hamiltonian operator +hamiltonian parameter +hamiltonian system +hamiltons optico mechanical analogy +hamiltons principle +hanbury brown +hanbury brown and twiss effect +hanbury brown twiss experiment +hankel function +hankel transform +hanle effect +hard core boson +hard hexagon model +hardening +hardys paradox +harmonic approximation +harmonic chain +harmonic confinement +harmonic generation +harmonic oscillator +harmonic oscillator potential +harmonic potential +harmonic superspace +harmonic trap +hartman effect +hartree fock method +hartree product +hawking effect +hawking radiation +hawking temperature +heat bath +heat deflection temperature +heat engine +heat of combustion +heat transfer +heat transport +heaviside feynman expression +heavy baryon chiral perturbation theory +heavy fermion +heavy fermion superconductor +heavy ion collision +heisenberg algebra +heisenberg chain +heisenberg cut +heisenberg equation +heisenberg group +heisenberg langevin equation +heisenberg limit +heisenberg model +heisenberg picture +heisenberg scaling +heisenberg uncertainty relation +heisenbergs microscope +helical orbit spectrometer +helicity +helicity bases +helicity basis +helion +helion energy +helium +helium atom +helium line ratio +helium neon laser +hellmann feynman theorem +helmholtz coil +helmholtz equation +helram +hemisphere +heralded single photon +heralded single photon source +hermite gauss +hermite polynomial +hermitian hamiltonian +hermitian matrix +hermitian operator +herta regina leng +hertz vector +hertzian dipole +hessian normal form +heteroazeotrope +heterodyne detection +heusler alloy +hexaarylbiimidazole +hexadecupole deformation +hexadecupole moment +hidden measurements interpretation +hidden momentum +hidden subgroup problem +hidden symmetry +hidden variable +hidden variable model +hidden variable theory +higgs boson +higgs field +higgs mechanism +higgs phase +higgsino +high aperture lens +high definition fiber tracking +high dimension +high dimensional entangled +high dimensional entanglement +high dimensional quantum state +high dimensional system +high dimensional transformation +high dimensionally entangled +high efficiency superconducting detector +high energy density physic +high energy liquid laser area defense system +high energy nuclear physic +high energy physic +high fidelity quantum gate +high finesse cavities +high finesse cavity +high finesse optical cavities +high finesse optical cavity +high finesse optical resonator +high flux advanced neutron application reactor +high frequency approximation +high frequency behavior of dielectric constant +high harmonic generation +high intensity focused ultrasound +high order harmonic generation +high resolution electron energy loss spectroscopy +high resolution transmission electron microscopy +high sensitivity +high spatial resolution +high temperature limit +high temperature superconductivity +high visibility +higher dimension +higher dimensional gamma matrices +higher dimensional quantum system +higher dimensional system +higher moment +higher order correlation function +higher temperature +highly charged ion +highly efficient +highly efficient superconducting detector +highly entangled +highly entanglement +highly sensitive +hilbert schmidt distance +hilbert schmidt norm +hilbert space +hilbert space dimension +hilbert space formalism +hilbert space structure +hilbert spectroscopy +hiper +histidine tryptophan ketoglutarate +holbrook superconductor project +holevo bound +holevo capacity +holevo information +holevos theorem +hollow atom +hollow cathode lamp +holographic principle +holography +holomorphic +holomovement +holon +holonomic quantum computation +holstein herring method +holstein primakoff transformation +homess law +homodyne detection +homodyne detector +homodyne measurement +homodyne tomography +homogeneous broadening +homogeneous magnetic field +honeycomb lattice +hong ou mandel +hong ou mandel effect +hong ou mandel interference +hong ou mandel interferometry +hookes atom +hopfield dielectric +horava lifshitz gravity +hot band +hot particle +hpo formalism +hubbard model +hueckel method +hunds case +hunds rule +hunter saxton equation +husimi distribution +husimi function +husimi q representation +huygens fresnel principle +huygens principle +huygenss principle +hybrid entanglement +hybrid functional +hybrid quantum system +hybrid silicon laser +hybrid system +hydrocarbon dew point +hydrodynamic equations for conducting fluid +hydrodynamic quantum analog +hydrogen atom +hydrogen fluoride laser +hydrogen like atom +hydrogen maser +hydrogen molecule +hydroxyl tagging velocimetry +hypercharge +hyperdeformation +hyperfine coupling +hyperfine interaction +hyperfine interaction energy +hyperfine level +hyperfine splitting +hyperfine state +hyperfine structure +hypermultiplet +hypernucleus +hyperon +hyperpolarization +hyperspectral imaging +hypertriton +hyphomonas +hypothermia +hypsochromic shift +hysteresis +hze ion +ice type model +idealizations in electromagnetism +ideally hard superconductor +identical atom +identical boson +identical copy +identical fermion +identical particle +identical photon +identity operator +identity theorem for holomorphic function +identity theorem for power series +idler mode +idler photon +image charge +image mapping spectrometer +imaginary time +imaging spectrometer +imaging spectroscopy +immirzi parameter +immobilized enzyme electron spin resonance +immunogold labelling +impact ionization +impact parameter +impedance +implicate and explicate order +in situ electron microscopy +in vivo magnetic resonance spectroscopy +ince gauss +incoherent broad band cavity enhanced absorption spectroscopy +incoherent scattering +incompatible measurement +incompatible observable +incongruent melting +increasing temperature +incredible utility +independent electron approximation +independent parameter +independent reservoir +independent source +index of reflection +index of refraction +indistinguishability +indistinguishable particle +indistinguishable photon +indistinguishable single photon +individual addressing +individual atom +individual attack +induce charge +induced gamma emission +inductance +induction coefficient +inductive resistance +inductively coupled plasma +inductively coupled plasma atomic emission spectroscopy +inelastic collision +inelastic neutron scattering +inelastic process +inelastic scattering +inequality violation +inert pair effect +inertial redshift +infinite dimension +infinite dimensional case +infinite dimensional hilbert space +infinite dimensional quantum system +infinite dimensional system +infinite energy +infinite family +infinite temperature +information capacity +information causality +information encoded +information entropy +information exchange +information flow +information gained +information geometry +information loss +information processing task +information theoretic approach +information theoretic security +information theory +information transfer +information transmission +infraparticle +infrared divergence +infrared fixed point +infrared safety +infrared spectroscopy +infrared spectroscopy correlation table +inglis teller equation +inhomogeneous broadening +inhomogeneous plane wave +inhour +injection seeder +inner bremsstrahlung +input output formalism +insensitive nuclei enhanced by polarization transfer +instanton +instanton fluid +integer quantum hall effect +integrable system +integral equations of the first kind +integrated optic +integrated optical circuit +integrating sphere +intelligent laser speckle classification +intense laser field +intensity correlation +intensity dependent coupling +intensity fluctuation +intensity interferometer +interacting atom +interacting boson +interacting boson model +interacting electron +interacting fermion +interacting field +interacting particle +interacting particle system +interacting quantum system +interacting qubit +interacting spin +interacting system +interaction energy +interaction free measurement +interaction hamiltonian +interaction picture +interaction potential +interatomic coulombic decay +interband cascade laser +interface between two media +interference +interference effect +interference experiment +interference fringe +interference pattern +interference phenomena +interference term +interference visibility +interferometric measurement +interferometric scheme +interferometry +internal conversion +internal conversion coefficient +internal degree +internal dynamic +internal energy +internal field +internal space time symmetry +internal symmetry +interpretations of quantum mechanics +intersubband polariton +intersystem crossing +intervalence charge transfer +interventional magnetic resonance imaging +intracavity field +intralase +intralase sbk +intramolecular vibrational energy redistribution +intraoperative electron radiation therapy +intraoperative mri +intravoxel incoherent motion +intrinsic decoherence +intrinsic parity +intrinsic property +intruder state +invariance +invariance mechanics +invariant set postulate +inverse participation ratio +inverse photoemission spectroscopy +inverse scattering problem +inverse scattering transform +inverse square law +inverse square root potential +inverse temperature +inversion symmetry +inversion with respect to a sphere +inxitu +ion coulomb crystal +ion crystal +ion implantation +ion laser +ion milling machine +ion notation +ion reactor +ion trap +ionization +ionization energy +ionized impurity scattering +ionosphere +ir uv mixing +iron based superconductor +iron beam +iron group +iron tetraboride +irrational conformal field theory +irreducible representation +irrotational vector +isadore perlman +ising anyon +ising chain +ising interaction +ising model +iskra laser +island of inversion +isobar +isodiapher +isogenous serie +isolated quantum system +isolated singularity +isolated system +isomeric shift +isosbestic point +isospin +isothermal transformation diagram +isotone +isotopic shift +isotropic state +isovalent hybridization +isttok +iterative algorithm +ito equation +ivan dryer +ivar giaever +j coupling +j/psi meson +j1 j2 model +jablonski diagram +jacobi identity +jacobi matrix +jacobi operator +jacobi polynomial +jacobian +janus laser +jarzynski equality +jaynes cummings hamiltonian +jaynes cummings hubbard model +jaynes cummings model +jefimenkos expression +jellium +jemris +joanna fowler +johari goldstein relaxation +john bell +john marburger +joint quantum entropy +jones polynomial +jordan map +jordan wigner transformation +josephson charge qubit +josephson effect +josephson energy +josephson junction +josephson junction count +josephson penetration depth +josephson phase +josephson voltage standard +josephson vortex +jost function +joulean heat +judd ofelt theory +julian schwinger +k alpha +k band multi object spectrograph +k beta +k line +k space +kadomtsev petviashvili equation +kadowaki woods ratio +kagome lattice +kane quantum computer +kaon +kaonic hydrogen +kaonium +kapitsa dirac effect +kapitza dirac regime +karnovsky fixative +karplus equation +kashas rule +kato theorem +kautsky effect +kcbs pentagram +kdv hierarchy +kelvin helmholtz instability +kerr effect +kerr medium +kerr nonlinearity +kibble zurek mechanism +kicked rotator +kicked rotor +kikuchi line +killing spinor +kinetic equation +kinetic inductance +kinetic inductance detector +kinetic theory +kinoshita lee nauenberg theorem +kirchhoff diffraction +kirchhoff integral theorem +kirchhoffs diffraction formula +kirchhoffs integral +kirchhoffs integral representation +kirszbraun theorem +kitaev model +klein gordon equation +klein nishina formula +klein paradox +klein transformation +klein tunneling +klystron +kmb09 protocol +kms state +knight shift +knizhnik zamolodchikov equation +kochen specker theorem +kodama state +kogut susskind fermion +kohn sham equation +kolmogorov complexity +kondo model +konishi anomaly +kontsevich quantization formula +koopmans theorem +korringa kohn rostoker approximation +korteweg de vries equation +kosterlitz thouless transition +kozai mechanism +kramers kronig dispersion relation +kramers kronig relation +kramers law +kramers theorem +kramers wannier duality +kraus representation +krefts dichromaticity index +krypton +krypton fluoride laser +ktp crystal +kubo formula +kubo gap +kuipers theorem +kuramoto model +kurchatov medal +kurt wuethrich +kuzyk quantum gap +ladder operator +lageos +lagrangian +lagrangian density +lagrangian foliation +lagrangian grassmannian +laguerre gauss +laguerre polynomial +lamb dicke limit +lamb dicke regime +lamb moessbauer factor +lamb shift +lambda baryon +lambda configuration +lambda point +lambda system +lambda transition +lambert w step potential +laminar turbulent transition +lanczos tensor +landau level +landau pole +landau quantization +landau zener formula +landau zener problem +landau zener transition +landauer formula +land� g factor +land� interval rule +langevin equation +langmuir state +lanthanum barium copper oxide +laplace equation +laplace expansion +laplace transform +laplaces method +laplacian operator +laporte rule +large angular momentum +large detuning +large gauge transformation +large helical device +large molecule +large quantum system +large scale quantum computation +large scale quantum computer +large separation +larmor formula +larmor formula for power +larmor precession +laser +laser ablation +laser absorption spectrometry +laser accelerometer +laser antibunching +laser assisted new attachment procedure +laser based angle resolved photoemission spectroscopy +laser beam machining +laser beam profiler +laser beam quality +laser beam welding +laser blended vision +laser bonding +laser broom +laser capture microdissection +laser cavities +laser cavity +laser coagulation +laser communication +laser communication in space +laser communications relay demonstration +laser construction +laser converting +laser cooled atom +laser cooling +laser cutting +laser cutting bridge +laser designator +laser detuning +laser diffraction analysis +laser diode +laser diode rate equation +laser doppler velocimetry +laser doppler vibrometer +laser dye +laser engineered net shaping +laser engraving +laser fence +laser field +laser fluctuation +laser frequency +laser guide star +laser gun +laser hair removal +laser harp +laser hybrid welding +laser induced breakdown spectroscopy +laser induced fluorescence +laser induced incandescence +laser induced thermotherapy +laser inertial fusion energy +laser integration line +laser intensity +laser jammer +laser lamp +laser level +laser light +laser lighting display +laser line level +laser linewidth +laser lithotripsy +laser machine control +laser marking +laser medicine +laser m�gajoule +laser parameter +laser peening +laser physic +laser pointer +laser power +laser power scaling +laser printing +laser projector +laser propulsion +laser pulse +laser pumping +laser radiation +laser rangefinder +laser resurfacing +laser scalpel +laser scanning +laser scanning vibrometry +laser science +laser shaft alignment +laser speckle +laser surgery +laser survey +laser tag +laser tracker +laser ultrasonic +laser warning receiver +laser weapon system +lasik +lasik md +lasing threshold +lasing without inversion +lasnex +latent internal energy +lateral casimir force +lattice boltzmann method +lattice density functional theory +lattice depth +lattice field theory +lattice gauge theory +lattice light sheet microscopy +lattice model +lattice potential +lattice qcd +lattice scattering +lattice site +lattice spacing +lattice system +laughlin wavefunction +laurent serie +laurents expansion +laws of reflection +laws of refraction +lax pair +lboz +lcp theory +lead star +legendre differential equation +legendre function +legendre polynomial +leggett garg +leggett garg inequalities +leggett garg inequality +leggett inequalities +leggett inequality +lehmann spectral representation +length measurement +lenzs law +lepton +leroy radius +leslie lawrance foldy +level crossing +level repulsion +level shift +level spacing +lever rule +liblas +libquantum +libration +lidar +lidar detector +lie algebra +lie conformal algebra +lie group +lie superalgebra +lieb conjecture +lieb oxford inequalities +lieb oxford inequality +lieb robinson bound +lienard wiechert potential +lifshitz formula +lifshitz theory +ligand k edge +light cone +light dressed state +light front computational method +light front quantization +light front quantization application +light harvesting complexes +light matter interaction +light polarization +light propagation +light pulse +light scattered +light scattering +light section +light sheet fluorescence microscopy +light source +light storage +light transport +light wave +lightcraft +lightest supersymmetric particle +limit theorem +limiting speed +lindblad equation +lindblad form +lindblad master equation +lindblad operator +lindblad resonance +lindblad superoperator +lindblad theory +line integral +line laser +linear antenna +linear combination of atomic orbital +linear dichroism +linear energy transfer +linear entropy +linear momentum +linear motor +linear operator +linear optical device +linear optical element +linear optical quantum computation +linear optical quantum computing +linear optics quantum computation +linear optics quantum computing +linear paul trap +linear programming +linear quantum system +linear response theory +linear stability +linear superposition +linear transform model +linearly polarized +linewidth +liouville equation +liouville von neumann equation +liouvillian operator +lipkin meshkov glick model +lippmann schwinger equation +liquid crystal +liquid crystal laser +liquid crystal thermometer +liquid helium +liquid liquid critical point +liquid metal cooled reactor +liquid sky +liquid state nuclear magnetic resonance +liquidus +lithium atom +lithium burning +little hierarchy problem +little parks effect +lloyd cross +local action +local causality +local control +local decoherence +local density approximation +local distinguishability +local hamiltonian +local hamiltonian problem +local hidden variable +local hidden variable model +local hidden variable theory +local oscillator +local perturbation +local quantum field theory +local quantum operation +local quench +local realism +local realistic model +local realistic theory +local state +local symmetry +local system +local theory +local transformation +local unitary equivalence +local unitary invariant +local unitary operation +local unitary transformation +local variable +locality +locality condition +localizable entanglement +localized mode +localized molecular orbital +localized state +locally compact quantum group +locc +lockin effect +log wind profile +logarithmic conformal field theory +logarithmic schroedinger equation +logic gate +logical gate +logical operation +logical qubit +logical state +london equation +london moment +london penetration depth +long coherence time +long distance communication +long distance entanglement +long distance experiment +long distance quantum communication +long distance quantum experiment +long josephson junction +long lived quantum memory +long path laser +long period fiber grating +long range correlation +long range entanglement +long range interaction +long slit spectroscopy +long time behaviour +long time dynamic +long time evolution +long time limit +long time nuclear waste +loop group +loop integral +loop quantum cosmology +loop quantum gravity +loop representation +loophole +loophole free bell test +loophole free test +loopholes in bell test experiment +lorentz boost +lorentz force +lorentz gauge +lorentz group +lorentz invariance +lorentz invariance in loop quantum gravity +lorentz invariant +lorentz lorenz relation +lorentz symmetry +lorentz transformation +lorenz condition +lorenz gauge +loschmidt echo +loss divincenzo quantum computer +lossy beam splitter +lossy cavities +lossy cavity +lossy channel +low angle laser light scattering +low dimension +low energy electron diffraction +low energy electron microscopy +low energy excitation +low energy limit +low energy spectrum +low field nuclear magnetic resonance +low level laser therapy +low lying excitation +low lying state +low temperature +low temperature limit +low temperature regime +low voltage electron microscope +lower critical solution temperature +lowest landau level +lsz reduction formula +luca turin +luli2000 +luminosity +lunar atmosphere and dust environment explorer +lunar reconnaissance orbiter +lund string model +luttinger liquid +luttingers theorem +lyapunov control +lyapunov exponent +lyman alpha blob +lyman alpha forest +lyman alpha line +lyman limit +mach zehnder interferometer +mach zehnder interferometry +machine learning +macroscopic average +macroscopic body +macroscopic entanglement +macroscopic field +macroscopic limit +macroscopic maxwell equation +macroscopic object +macroscopic quantum coherence +macroscopic quantum electrodynamic +macroscopic quantum phenomena +macroscopic quantum self trapping +macroscopic quantum state +macroscopic quantum system +macroscopic quantum tunneling +macroscopic realism +macroscopic scale +macroscopic superposition +macroscopic system +macroscopic world +macroscopically distinct state +madelung equation +madison symmetric torus +magic angle +magic angle spinning +magic state +magic state distillation +magnesium diboride +magnetic bottle +magnetic catalysis +magnetic charge +magnetic circular dichroism +magnetic diffusion +magnetic dipole +magnetic dipole dipole interaction +magnetic energy +magnetic field +magnetic field gradient +magnetic field in matter +magnetic field intensity +magnetic field strength +magnetic flux +magnetic flux density +magnetic flux quantum +magnetic impurity +magnetic induction +magnetic induction tomography +magnetic inequivalence +magnetic microtrap +magnetic mirror +magnetic moment +magnetic monopole +magnetic multipole field +magnetic permeability +magnetic photon +magnetic polarization +magnetic pressure +magnetic property +magnetic proton recoil neutron spectrometer +magnetic quantum number +magnetic resonance +magnetic resonance angiography +magnetic resonance cholangiopancreatography +magnetic resonance elastography +magnetic resonance force microscopy +magnetic resonance imaging +magnetic resonance imaging of the brain +magnetic resonance microscopy +magnetic resonance neurography +magnetic resonance spectroscopic imaging +magnetic resonance velocimetry +magnetic scalar potential +magnetic semiconductor +magnetic shielding +magnetic skyrmion +magnetic susceptibility +magnetic system +magnetic tension +magnetic translation +magnetic trap +magnetic wave +magnetization +magnetization transfer +magnetized liner inertial fusion +magnetized sphere +magnetizing current +magneto inertial fusion +magneto optical trap +magnetocapacitance +magnetohydrodynamics +magnetometer +magnetophosphene +magnetoreception +magnetoresistance +magnetoresistive random access memory +magnetostatics +magnon +major actinide +majorana bound state +majorana equation +majorana fermion +majorana mode +majorana representation +majoron +majumdar ghosh model +man in the middle +mandel parameter +mandel q parameter +mandelstam variable +mangler transformation +manick sorcar +manipulation of atoms by optical field +many body dynamic +many body effect +many body entanglement +many body hamiltonian +many body interaction +many body localisation +many body physic +many body problem +many body quantum system +many body state +many body system +many body theory +many minds interpretation +many particle system +many worlds interpretation +marchenko equation +margolus levitin theorem +markov approximation +markov chain +markov chain in the ising model +markov property +markovian approximation +markovian dynamic +markovian limit +markovian master equation +mars climate sounder +mars telecommunications orbiter +martin fleischmann +maryland lidar availability +maser +mass excess +mass gap +mass motion +mass number +mass of photon +mass spectrometer +mass spectrum +massive particle +massive scalar field +massless particle +massless scalar field +master equation +material property +materials oscilloscope +mathematical formulation of quantum mechanic +mathematical universe hypothesis +matrix algebra +matrix element +matrix isolation +matrix mechanic +matrix product operator +matrix product state +matrix representation +matsubara frequency +mattauch isobar rule +matter field +matter interferometry +matter wave +matter wave clock +matter wave interferometer +matter wave interferometry +mattis bardeen theory +maxim chernodub +maximal entanglement +maximally entangled mixed state +maximally entanglement +maximally mixed state +maximon +maximum entanglement +maximum entropy principle +maximum entropy spectral estimation +maximum likelihood estimation +maximum violation +maxwell +maxwell bloch equation +maxwell equation +maxwell stress tensor +maxwell wagner sillars polarization +maxwellian stress tensor +maxwells equation +mccumber relation +mcstas +mean field approach +mean field approximation +mean field dynamic +mean field theory +mean free path +measurable quantity +measurement apparatus +measurement back action +measurement based quantum computation +measurement based quantum computing +measurement device independent quantum cryptography +measurement device independent quantum key distribution +measurement in quantum mechanic +measurement limit +measurement operator +measurement precision +measurement problem +measurement process +measurement statistic +measurement strategy +measurement strength +measurement theory +measurement time +measurement uncertainty +mechanical frequency +mechanical mode +mechanical motion +mechanical oscillator +mechanical resonator +mechanochromism +meir wingreen formula +meissner effect +melting +melting point +melting point depression +membrane paradigm +memory effect +mercury laser +mermin inequalities +mermin inequality +mermin wagner theorem +meron +meson +mesonic molecule +mesophase +mesoscopic physic +mesoscopic system +messential singularity +metal induced gap state +metal k edge +metal l edge +metallic plate +metallomesogen +metastable helium +metastable inner shell molecular state +metastable state +method of continued fraction +method of electric image +method of image +method of steepest descent +metric operator +metric tensor +metrology +mhv amplitude +michael coey +michael mckubre +michael nobel +michel parameter +michelson interferometry +michelson morley experiment +michigan spin physics center +micro cavities +micro cavity +micro spectrophotometry +micro spider +microcanonical ensemble +microcoil +microprobe +microring resonator +microscopic degree +microscopic derivation +microscopic description +microscopic dynamic +microscopic system +microscopic theory +microscopy +microwave +microwave field +microwave frequency +microwave photon +microwave radiation +microwave regime +microwave resonator +mictomagnetism +millikelvin temperature +mimos +min entropy +mindo +mini tes +minimal coupling +minimal model +minimal monotone +minimal output entropy +minimal subtraction scheme +minimal supersymmetric standard model +minimum design metal temperature +minimum distance +minimum error discrimination +minimum output entropy +minimum uncertainty state +minisuperspace +minkowski space +minkowski space time +minkowski vacuum +minnesota functional +minority interpretations of quantum mechanic +miracl +mirror nuclei +miscibility gap +mixed anomaly +mixed entanglement +mixed layer +mixed phase space +mixed quantum state +mixed state +mixed state entanglement +mixed two qubit state +mndo +moata +mode entanglement +mode locking +mode structure +mode volume +modern valence bond theory +modulation frequency +moduli +moessbauer effect +moessbauer spectroscopy +molecular aggregate +molecular beacon +molecular beam +molecular dynamic +molecular electronic transition +molecular gases +molecular hamiltonian +molecular ion +molecular magnet +molecular multipole moment +molecular orbital +molecular orbital theory +molecular physic +molecular polarizability +molecular rotation +molecular system +molecular tagging velocimetry +molecular term symbol +molecular vibration +molecule +mollow triplet +moment problem +momentum distribution +momentum entanglement +momentum observable +momentum operator +momentum representation +momentum space +momentum transfer +momentum transfer cross section +monin obukhov length +monochromatic wavelength +monochromatic wavelength dispersive x ray fluorescence +monochromator +monogamy inequalities +monogamy inequality +monogamy relation +monopole moment +monopole radiation field +monotone metric +monte carlo method +monte carlo methods for electron transport +monte carlo simulation +montonen olive duality +mood ring +morse palais lemma +morse potential +moseleys law +mosquito laser +mossbauer effect +motional degree +motional ground state +motional narrowing +motional state +mott bethe formula +mott criterion +mott insulator +mott insulator state +mott phase +mott problem +mott scattering +movable mirror +moving charge +moving mirror +moving object +moyal bracket +moyal product +mpc&a +mpemba effect +mtz black hole +mu problem +muffin tin approximation +mulliken population analysis +multi configuration time dependent hartree +multi configurational self consistent field +multi particle entangled state +multi particle entanglement +multi particle system +multi partite entangled state +multi partite entanglement +multi photon entangled state +multi photon experiment +multi photon state +multi photon state generation +multi qubit state +multi qubit system +multi scale entanglement renormalization ansatz +multi unit spectroscopic explorer +multiangle light scattering +multicritical point +multifocal diffractive lens for laser application +multifocal plane microscopy +multiparticle entanglement +multipartite case +multipartite correlation +multipartite entanglement +multipartite pure state +multipartite quantum correlation +multipartite quantum state +multipartite quantum system +multipartite setting +multipartite state +multipartite system +multipass spectroscopic absorption cell +multiphoton interference +multiple exciton generation +multiple histories +multiple integrated laser engagement system +multiple prism grating laser oscillator +multiple qubit +multiple scale analysis +multiple scattering +multiplet +multiplication factor +multiplicative quantum number +multiplicity +multipolarity of gamma radiation +multipole expansion +multipole field +multipole moment +multipole radiation +multipoles constructed of point charge +multiqubit state +multireference configuration interaction +multiscale tomography +multispectral image +multispectral segmentation +multivariate optical computing +multivariate optical element +multiverse +munnerlyn formula +muon capture +muon catalyzed fusion +muon neutrino +muon spin spectroscopy +muonium +mutual information +mutual interaction +mutually unbiased bases +mutually unbiased basis +mutually unbiased measurement +mycoplasma mycoides +myron mathisson +m�ller plesset perturbation theory +m�ller scattering +n = 4 supersymmetric yang mills theory +n electron valence state perturbation theory +n qubit state +n qubit system +n slit interferometric equation +n vector model +n00n state +n=8 supergravity +nahm equation +nakajima zwanzig equation +nambu jona lasinio model +nanocrystal solar cell +nanolaser +nanomechanical oscillator +nanomechanical resonator +nanophotonic coherent imager +nanophotonic resonator +nanowire +narrow resonance +nash equilibria +nash equilibrium +national ignition facility +national lidar dataset +national radiological protection board +natural bond orbital +natural orbital +natural width +nd:gdvo4 +nd:yag laser +nd:ycob +nddo +near infrared camera and multi object spectrometer +near infrared spectroscopy +nearest neighbor +nearest neighbor interaction +nearest neighbour interaction +nearly free electron model +negative eigenvalue +negative energy density +negative energy state +negative probability +negative refraction +negative temperature +negatively charged nitrogen vacancy +negativity +neighboring site +neighboring spin +nematicon +neodymium doped yttrium lithium fluoride +neodymium doped yttrium orthovanadate +neon +neon burning process +nephelauxetic effect +nernst heat theorem +nested neutron spectrometer +neumann boundary condition +neutral atom +neutral current +neutral kaon +neutral kaon system +neutral particle +neutral particle oscillation +neutralino +neutrino +neutrino oscillation +neutron +neutron acceptance diagram shading +neutron activation +neutron backscattering +neutron bomb +neutron capture +neutron capture nucleosynthesis +neutron cross section +neutron degenerate matter +neutron detection +neutron diffraction +neutron economy +neutron emission +neutron flux +neutron imaging +neutron interferometry +neutron moisture gauge +neutron number +neutron probe +neutron proton ratio +neutron radiation +neutron reflectometry +neutron reflector +neutron research facility +neutron scattering +neutron source +neutron spectroscopy +neutron spin echo +neutron star +neutron stimulated emission computed tomography +neutron temperature +neutron time of flight scattering +neutron transport +neutronium +new worlds mission +newton wigner localization +newtonian gravity +next to minimal supersymmetric standard model +nichols algebra +nielsen ninomiya theorem +nielsen olesen vortex +nielsen olsen string +nike laser +nikiforov uvarov method +niobium germanium +niobium nitride +niobium tin +niobium titanium +nitrogen +nitrogen 15 nuclear magnetic resonance spectroscopy +nitrogen laser +nitrogen rule +nitrogen vacancy +nitrogen vacancy center +nmr quantum computation +nmr quantum computer +nmr quantum information processor +nmr spectroscopy +nmr tube +nmrpipe +no broadcast theorem +no cloning +no communication theorem +no go theorem +no signaling +no teleportation theorem +noble gas configuration +noether theorem +noethers second theorem +noethers theorem +noise correlation +noise equivalent power +noise immune cavity enhanced optical heterodyne molecular spectroscopy +noise level +noise model +noise property +noise ratio +noise reduction +noise source +noise spectrum +noise strength +noise threshold +noiseless quantum channel +noiseless subsystem +noisy channel +noisy environment +noisy quantum channel +noisy storage model +non abelian anyon +non abelian gauge transformation +non abelian geometric phase +non abelian statistic +non adiabatic transition +non classical correlation +non classical feature +non classical light +non classical property +non classical state +non commuting observable +non commuting operator +non demolition +non equilibrium dynamic +non equilibrium steady state +non exponential decay +non extensive self consistent thermodynamical theory +non gaussian state +non hermitian hamiltonian +non hermitian operator +non hermitian system +non inertial frame +non interacting fermion +non interacting particle +non linear coherent state +non linear sigma model +non local correlation +non local property +non markovian behaviour +non markovian character +non markovian dynamic +non markovian effect +non markovian environment +non markovian master equation +non markovian noise +non markovian open quantum system +non markovian quantum dynamic +non markovian quantum state diffusion +non markovian regime +non maximally entanglement +non orthogonal quantum state +non orthogonal state +non perturbative +non relativistic case +non relativistic limit +non relativistic quantum mechanic +non topological soliton +non unitary evolution +nonclassical correlation +nonclassical effect +nonclassical feature +nonclassical light +nonclassical photon counting +nonclassical property +nonclassical state +noncommutative geometry +noncommutative quantum field theory +noncommutative quantum mechanic +noncommutative space +noncommuting observable +noncontextuality inequalities +noncontextuality inequality +nonequilibrium dynamic +nonequilibrium steady state +noninertial frame +nonlinear cavities +nonlinear cavity +nonlinear coherences +nonlinear coupling +nonlinear crystal +nonlinear dirac equation +nonlinear dynamic +nonlinear electrodynamic effect +nonlinear equation +nonlinear interaction +nonlinear medium +nonlinear optic +nonlinear optical process +nonlinear oscillator +nonlinear processe +nonlinear quantum mechanic +nonlinear refractive index +nonlinear regime +nonlinear response +nonlinear schroedinger equation +nonlinear susceptibility +nonlinear system +nonlocal correlation +nonlocal lagrangian +nonlocal nature +nonlocal property +nonlocality +nonlocality in time +nonorthogonal state +nonrelativistic limit +nonrelativistic quantum mechanic +nontrivial topology +nonzero probability +nonzero temperature +noon state +normal dispersion +normal mode +normal mode splitting +normal order +northwest nuclear consortium +nova +novette laser +novikov veselov equation +np complete +npdgamma experiment +nuclear astrophysic +nuclear binding energy +nuclear chain reaction +nuclear chemistry +nuclear collision length +nuclear cross section +nuclear data +nuclear density +nuclear drip line +nuclear emulsion +nuclear energy in kenya +nuclear energy in saudi arabia +nuclear force +nuclear forensic +nuclear fusion +nuclear instruments and methods in physics research +nuclear interaction length +nuclear isomer +nuclear magnetic moment +nuclear magnetic resonance +nuclear magnetic resonance chemical shift re referencing +nuclear magnetic resonance crystallography +nuclear magnetic resonance database method +nuclear magnetic resonance decoupling +nuclear magnetic resonance in porous media +nuclear magnetic resonance quantum computer +nuclear magnetic resonance spectra database +nuclear magnetic resonance spectroscopy +nuclear magnetic resonance spectroscopy of carbohydrate +nuclear magnetic resonance spectroscopy of nucleic acid +nuclear magnetic resonance spectroscopy of protein +nuclear magnetic resonance spectroscopy of stereoisomer +nuclear matter +nuclear microscopy +nuclear orientation +nuclear overhauser effect +nuclear physic +nuclear pumped laser +nuclear quadrupole moment +nuclear quadrupole resonance +nuclear resonance fluorescence +nuclear resonance vibrational spectroscopy +nuclear shell model +nuclear spin +nuclear structure +nuclear transparency +nucleic acid quantitation +nucleocosmogenesis +nucleogenic +nucleon +nucleon spin structure +nucleosynthesis +nuclep +nukleonik +number operator +number state +number state expansion +number theory +numerical algorithm +numerical analysis +numerical approach +numerical calculation +numerical computation +numerical evidence +numerical example +numerical experiment +numerical method +numerical optimization +numerical renormalization group +nv center +n�el temperature +objective collapse theory +oblate +obliquity factor in diffraction +oblivious transfer +observable +observable consequence +observer +obstacles in wave guide +occupation number +octacube +octupole moment +od600 +oddo harkins rule +oerstedts law +of the poisson equation +ohmic power +ohmic resistance +ohms law +oil drop experiment +old quantum theory +oldstone conference +omega baryon +omega meson +on demand source +on shell and off shell +on shell renormalization scheme +on thermonuclear war +one dimensional chain +one dimensional harmonic oscillator +one dimensional harmonic trap +one dimensional hubbard model +one dimensional lattice +one dimensional optical lattice +one dimensional quantum mechanic +one dimensional quantum system +one dimensional system +one dimensional waveguide +one electron universe +one loop feynman diagram +one parameter family +one photon interference +one qubit gate +one way quantum computation +one way quantum computer +onetep +oniom +onium +onset of deconfinement +oocyte selection +oops leon +opacity +opals +open boundary condition +open quantum dynamic +open quantum system +open shell +open system dynamic +operando spectroscopy +operating temperature +operational interpretation +operator algebra +operator norm +operator quantum error correction +oppenheimer phillips process +optical amplifier +optical autocorrelation +optical beam +optical bistability +optical bloch equation +optical cavities +optical cavity +optical coherence +optical coherence tomography +optical control +optical decay +optical depth +optical detection +optical device +optical dipole trap +optical equivalence theorem +optical excitation +optical fiber +optical fibre +optical field +optical frequency comb +optical frequency multiplier +optical interferometry +optical isolator +optical lattice +optical loss +optical microcavities +optical microcavity +optical mode +optical nanofiber +optical network +optical parametric amplification +optical parametric amplifier +optical parametric oscillator +optical phase +optical phase space +optical photon +optical potential +optical pulse +optical pumping +optical quantum computing +optical quantum information processing +optical qubit +optical realization +optical resonator +optical response +optical scheme +optical setup +optical spectrometer +optical state +optical theorem +optical transition +optical trap +optical tweezer +optical wavelength +optimal choice +optimal cloning +optimal control problem +optimal control theory +optimal discrimination +optimal entanglement witnesses +optimal fidelity +optimal measurement +optimal protocol +optimal quantum cloning +optode +optomechanic +optomechanical cavities +optomechanical cavity +optomechanical coupling +optomechanical interaction +optomechanical system +optomechanically induced transparency +oracle query +orbit modeling +orbital angular momentum +orbital free density functional theory +orbital hybridisation +orbital magnetization +orbital motion +orbital overlap +orbital stability +orbiton +orchestrated objective reduction +order operator +orders of approximation +organic laser +organic molecule +organic superconductor +orientation entanglement +orientation polarization +ornstein uhlenbeck process +orthodox quantum mechanic +orthogonal function +orthogonal state +orthogonality relation +orthomodular lattice +orthonormal bases +orthonormal basis +orthonormal system +orthonormalization +oscillating circuit +oscillation threshold +oscillator representation +oscillator strength +oscillon +osculating orbit +out of equilibrium dynamic +output correlation matrix +output coupler +ova bank +overtone band +oxygen burning process +oxypnictide +ozi rule +p adic quantum mechanic +p form electrodynamic +p nuclei +p process +page time +painted turtle +pair creation +pair production +pairwise entanglement +pake doublet +pandemonium effect +pandya theorem +parallel plate capacitor +parallel transport +paramagnetic +paramagnetic nuclear magnetic resonance spectroscopy +paramagnetism +parameter estimation +parametric amplification +parametric amplifier +parametric down conversion +parametric downconversion +parametric oscillator +parametric process +parametric resonance +parastatistic +paraxial approximation +paraxial limit +paraxial wave equation +pariser parr pople method +parity +parity anomaly +parity check matrices +parsec +parsevals equation +parsevals theorem +partial cloning +partial information +partial polarization +partial separability +partial trace +partial transposition criterion +partial wave analysis +particle creation +particle entanglement +particle in a box +particle in a one dimensional lattice +particle in a ring +particle in a spherically symmetric potential +particle induced x ray emission +particle loss +particle motion +particle number operator +particle physics and representation theory +particle production +particle trajectory +partite entanglement +partition function +parton +pascal photocoagulator +paser +path distinguishability +path entanglement +path integral +path integral formulation +path integral molecular dynamic +path integral monte carlo +path integration +path length +path ordering +patterson power cell +paul kuroda +paul trap +pauli channel +pauli equation +pauli exclusion principle +pauli group +pauli lubanski pseudovector +pauli matrix +pauli operator +pauli principle +pauli spin operator +pauli villars regularization +paulings principle of electroneutrality +pbr theorem +peakon +pearl vortex +peccei quinn theory +pegasus toroidal experiment +peierls substitution +peierls transition +penetration depth +penguin diagram +penning trap +penrose interpretation +pentaquark +per cent mille +percolation critical exponent +percolation theory +percolation threshold +perdaix +peregrine soliton +peres horodecki criterion +perfect conductor +perfect correlation +perfect quantum state transfer +perfect state transfer +perfect teleportation +periodic boundary condition +periodically poled ktp crystal +periodically poled lithium niobate +permanent magnet +permanent magnetization +permeability +permittivity +permutation group +permutation symmetry +perpendicular magnetic field +persistent current +personnel halting and stimulation response rifle +perturbation +perturbation problem beyond all order +perturbation theory +perturbative qcd +petahertz +peter l. hagelstein +peter van riper +pglo +phase boundary +phase coherence +phase control +phase damping channel +phase decoherence +phase dependent correlation function +phase diagram +phase difference +phase diffusion +phase distribution +phase error +phase estimation +phase factor +phase fluctuation +phase gate +phase information +phase instability +phase matching +phase measurement +phase modulation +phase noise +phase of plane wave +phase operator +phase quadrature +phase qubit +phase retrieval +phase sensitivity +phase separation +phase shift +phase shift for scattering by sphere +phase shifter +phase space approach +phase space density +phase space description +phase space distribution +phase space formulation +phase space representation +phase state +phase structure +phase super resolution +phase time +phase transition +phase transitions and critical phenomena +phase velocities +phase velocity +phason +phi meson +philosophical interpretation of classical physic +phonon +phonon bath +phonon drag +phonon interaction +phonon mode +phonon noise +phonon number +phonon scattering +phosphorescence +phosphorus 31 nuclear magnetic resonance +phosphorus donor +photino +photo detection +photo meson +photoacoustic effect +photoacoustic spectroscopy +photobleaching +photochromic lens +photochromism +photodarkening +photodetection +photodisintegration +photodynamic therapy +photoelectric effect +photoelectrochemical process +photoelectron photoion coincidence spectroscopy +photoemission spectroscopy +photofission +photoinduced phase transition +photoionization +photoionization mode +photoluminescence +photoluminescence excitation +photomagneton +photomedicine +photon +photon absorption +photon antibunching +photon blockade +photon bunching +photon correlation +photon counter +photon counting +photon counting distribution +photon counting statistic +photon detection +photon detector +photon distribution +photon emission +photon energy +photon entanglement +photon field +photon flux +photon generation +photon loss +photon mass +photon number +photon number distribution +photon number resolving detector +photon number splitting attack +photon number state +photon number statistic +photon pair +photon polarization +photon scattering +photon source +photon state +photon statistic +photon structure function +photon subtraction +photon wave function +photonic band gap +photonic crystal +photonic crystal cavities +photonic crystal cavity +photonic entanglement +photonic laser thruster +photonic molecule +photonic orbital angular momentum +photonic qubit +photonic state +photonic system +photorefractive keratectomy +photorejuvenation +photosynthesis +photosynthetic complexes +photothermal microspectroscopy +photothermal spectroscopy +photovoltaic +physical limit +physics applications of asymptotically safe gravity +physics of magnetic resonance imaging +pi josephson junction +pi pulse +pickering series +piezochromism +piezoelectric effect +piezoelectricity +pilot wave +pilot wave theory +pinning force +pion +pion decay constant +pionium +planck constant +planck distribution +planck einstein relation +planck length +planck mass +planck particle +planck postulate +planck scale +plancks law +plane of incidence +plane wave +plane wave expansion +planetary boundary layer +planetary fourier spectrometer +plasma +plasma acoustic shield system +plasma frequency +plasma model +plasma science society of india +plasmaron +plasmon +plasmonic laser +plasmonic nanoparticle +plate trick +plekton +plum pudding model +pockels cell +pockels effect +pocono conference +poeschl teller potential +poincare group +poincare stresses +poincar� group +poincar� lindstedt method +point dipole +point interaction +point particle +pointer bases +pointer basis +pointer state +poisson bracket +poisson equation +poisson superalgebra +poissonian pumping +poissonian statistic +poissons equation +polar coordinate +polar decomposition +polar molecule +polar substance +polar vector +polariton +polariton laser +polarizability +polarization +polarization correlation +polarization degree +polarization entangled photon +polarization entangled photon pair +polarization entanglement +polarization interferometer +polarization optic +polarization potential +polarization property +polarization qubit +polarization rotator +polarization sensitive imaging +polarization spectroscopy +polarization squeezing +polarization state +polarization vector +polarized light +polarized photon +polaron +polyamorphism +polymer field theory +polynomial time algorithm +polynomial wigner ville distribution +polythiazyl +pomeron +ponderomotive energy +ponderomotive force +pontecorvo maki nakagawa sakata matrix +poole frenkel effect +pople diagram +pople notation +poppers experiment +population dynamic +population imbalance +population inversion +population transfer +population trapping +porods law +port wine stain +position and momentum space +position dependent effective mass +position dependent mass +position measurement +position operator +position space +positive definite kernel +positive operator valued measure +positive partial transpose +positive partial transposition +positron +positron annihilation spectroscopy +positron emission tomography +positronium +positronium hydride +post hartree fock +post selection +postbqp +potential energy +potential energy of protein +potential energy surface +potential well +potts model +pour point +pourbaix diagram +povm +powder diffraction +power flow +power law decay +power loss +power series +poynting robertson effect +poynting theorem +poynting vector +poyntings theorem +ppt state +practical quantum key distribution +precession electron diffraction +precession frequency +precision measurement +precision tests of qed +precursor +pregeometry +preon +present day technology +presolar grain +pressuron +primary field +prime dimension +prime number +prime power dimension +primon gas +principal quantum number +principal value +principle of locality +principle of superposition +prior entanglement +prior information +prior knowledge +prior work +prism compressor +prism spectrometer +privacy amplification +probabilistic description +probabilistic interpretation +probabilistic teleportation +probability amplitude +probability current +probability density function +probability distribution +probability interpretation +probability measure +probability representation +probability theory +probe beam +probe field +probe laser +probe light +probe pulse +probe qubit +problem of time +proca action +proca lagrangian +process tomography +product bases +product basis +product numerical range +product operator formalism +product state +project 1640 +project sherwood +project valkyrie +projected entangled pair state +projection measurement +projection operator +projection postulate +projective hilbert space +projective measurement +projective superspace +projector augmented wave method +prolate +prompt neutron +propagator +proper time +protective measurement +protein chemical shift prediction +protein chemical shift re referencing +protein protein interaction +protocol on blinding laser weapon +proton +proton boron fusion +proton coupled electron transfer +proton decay +proton emission +proton enhanced nuclear induction spectroscopy +proton exchange membrane +proton exchange membrane fuel cell +proton magnetic moment +proton nuclear magnetic resonance +proton pack +proton proton chain reaction +proton pump +proton spin crisis +proton therapy +proton to electron mass ratio +proton transfer reaction mass spectrometry +proton tunneling +protonium +protonophore +proximity effect +pseudogap +pseudomonas +pseudomonas genome database +pseudopotential +pseudoscalar meson +pseudospin symmetry +pseudovector meson +psi meson +psychrophile +pt symmetric hamiltonian +pt symmetric quantum mechanic +pt symmetric system +pt symmetry +pt symmetry breaking +ptc rubber +public channel +pulay stress +pulse duration +pulse height analyzer +pulse laser +pulse programming +pulse sequence +pulse shape +pulse width +pulsed electron paramagnetic resonance +pulsed excitation +pulsed field gradient +pulsed field magnet +pulsed laser deposition +pump beam +pump depletion +pump field +pump laser +pump power +pure dephasing +pure entanglement +pure gaussian state +pure quantum state +pure spinor +pure state +pure state entanglement +pure three qubit state +purely classical +purely quantum +purification of quantum state +purity +push broom scanner +pyquante +pyroelectric effect +pyroelectricity +pyrromethene +q ball +q switching +q value +qbism +qcd matter +qcd string +qcd sum rule +qcd vacuum +qcma +qed vacuum +qkd protocol +qkd system +qma +qmc@home +qnd measurement +quadratic configuration interaction +quadrature phase measurement +quadrature phase operator +quadrature squeezing +quadrature variance +quadrupole deformation +quadrupole moment +quadrupole splitting +quadrupole transition +quantisation +quantitative susceptibility mapping +quantization +quantization commutes with reduction +quantization of charge +quantization of the electromagnetic field +quantized electromagnetic field +quantized field +quantum 1/f noise +quantum action +quantum adiabatic algorithm +quantum adiabatic evolution +quantum adiabatic theorem +quantum advantage +quantum adversary +quantum affine algebra +quantum algebra +quantum algorithm +quantum algorithm for linear systems of equation +quantum amplifier +quantum annealer +quantum annealing +quantum anomaly +quantum aspects of life +quantum attack +quantum automata +quantum baker +quantum bayesianism +quantum beat +quantum billiard +quantum biology +quantum bit +quantum bit commitment +quantum bit error rate +quantum black hole +quantum brownian motion +quantum bus +quantum byzantine agreement +quantum calculus +quantum capacitance +quantum capacity +quantum carpet +quantum cascade laser +quantum catalyst +quantum cellular automata +quantum cellular automaton +quantum channel +quantum channel capacity +quantum chaos +quantum chaotic system +quantum chemistry +quantum chemistry composite method +quantum chromodynamic +quantum circuit +quantum classical boundary +quantum classical correspondence +quantum classical transition +quantum clock +quantum cloning +quantum code +quantum cognition +quantum coherence +quantum communication +quantum communication channel +quantum communication complexity +quantum compass +quantum complex network +quantum complexity +quantum complexity theory +quantum computation +quantum computation based +quantum computer +quantum computing +quantum computing application +quantum concentration +quantum configuration space +quantum confined stark effect +quantum contextuality +quantum convolutional code +quantum correction +quantum correlation +quantum cosmology +quantum counterpart +quantum coupling +quantum cramer rao bound +quantum critical point +quantum criticality +quantum cryptographic protocol +quantum cryptography +quantum cylindrical quadrupole +quantum darwinism +quantum data bus +quantum decay +quantum decoherence +quantum defect +quantum degeneracy +quantum degree +quantum dense coding +quantum depolarizing channel +quantum description +quantum device +quantum diffusion +quantum digital signature +quantum dimer model +quantum discord +quantum dissipation +quantum dissipative system +quantum domain +quantum dot +quantum dot cellular automaton +quantum dot coupled +quantum dot display +quantum dot laser +quantum dot solar cell +quantum dynamic +quantum dynamical semigroup +quantum effect +quantum efficiency +quantum electrochemistry +quantum electrodynamic +quantum electromagnetic field +quantum emitter +quantum energy teleportation +quantum ensemble +quantum entanglement +quantum entropy +quantum equation +quantum equilibrium +quantum eraser +quantum eraser experiment +quantum erasure +quantum ergodicity +quantum error correcting code +quantum error correction +quantum error correction code +quantum espresso +quantum estimation theory +quantum evolution +quantum factoring algorithm +quantum feedback +quantum feedback control +quantum fiction +quantum fidelity +quantum field +quantum field theory +quantum field theory in curved spacetime +quantum filtering +quantum fingerprinting +quantum finite automata +quantum fisher information +quantum fluctuation +quantum fluid +quantum flux parametron +quantum foam +quantum formalism +quantum foundation +quantum fourier transform +quantum game +quantum game theory +quantum gas +quantum gate +quantum gate operation +quantum gauge theory +quantum geometry +quantum graph +quantum gravity +quantum ground state +quantum group +quantum gyroscope +quantum hall effect +quantum hamilton jacobi equation +quantum hardware +quantum harmonic oscillator +quantum heat engine +quantum heat engines and refrigerators +quantum heterostructure +quantum hilbert space +quantum history +quantum hydrodynamic +quantum hypothesis testing +quantum illumination +quantum imaging +quantum immortality +quantum indeterminacy +quantum inequalities +quantum inequality +quantum information +quantum information processor +quantum information protocol +quantum instrument +quantum interference +quantum interference effect +quantum interferometry +quantum internet +quantum inverse scattering method +quantum ising chain +quantum ising model +quantum jump +quantum jump approach +quantum jump method +quantum key +quantum key distribution +quantum kicked rotor +quantum kz equation +quantum langevin equation +quantum lattice system +quantum lc circuit +quantum light +quantum limit +quantum lithography +quantum localisation +quantum logic +quantum logic gate +quantum logic operation +quantum lower bound +quantum machine +quantum machine learning +quantum magnetism +quantum markov chain +quantum master equation +quantum matter +quantum measurement +quantum measurement problem +quantum mechanic +quantum mechanical bell test prediction +quantum mechanical modification +quantum mechanical scattering of photon and nucleus +quantum mechanics of time travel +quantum memory +quantum metamaterial +quantum metrology +quantum mirage +quantum money +quantum monte carlo +quantum mutual information +quantum network +quantum neural network +quantum no deleting theorem +quantum noise +quantum noise reduction +quantum non demolition measurement +quantum non equilibrium +quantum non locality +quantum nondemolition measurement +quantum nonlocality +quantum number +quantum operation +quantum optic +quantum optical spectroscopy +quantum optomechanic +quantum oscillator +quantum paradox +quantum parallelism +quantum parameter estimation +quantum pendulum +quantum phase +quantum phase estimation +quantum phase estimation algorithm +quantum phase gate +quantum phase space +quantum phase transition +quantum phenomena +quantum phenomenon +quantum point contact +quantum potential +quantum prediction +quantum principle +quantum probability +quantum probability theory +quantum process +quantum process tomography +quantum processor +quantum programming +quantum protocol +quantum pseudo telepathy +quantum pure state +quantum quench +quantum query complexity +quantum rabi model +quantum radar +quantum random number generator +quantum random walk +quantum randomness +quantum readout of puf +quantum reality +quantum realm +quantum recurrance phenomena +quantum refereed game +quantum reference frame +quantum reflection +quantum refrigerator +quantum regime +quantum register +quantum regression theorem +quantum relative entropy +quantum repeater +quantum resonance +quantum resource +quantum revival +quantum rotor model +quantum scattering +quantum search algorithm +quantum searching +quantum secret sharing scheme +quantum secure direct communication +quantum sensing +quantum sensor +quantum shannon theory +quantum side information +quantum signal +quantum simulator +quantum sort +quantum space time +quantum speed limit +quantum speedup +quantum spin +quantum spin chain +quantum spin hall effect +quantum spin model +quantum spin tunneling +quantum stabilizer code +quantum state diffusion +quantum state discrimination +quantum state engineering +quantum state estimation +quantum state preparation +quantum state reconstruction +quantum state reduction +quantum state space +quantum state tomography +quantum state transfer +quantum statistical mechanic +quantum statistical property +quantum steering +quantum stein +quantum stochastic calculus +quantum stochastic differential equation +quantum storage +quantum strategy +quantum structure +quantum subsystem +quantum suicide +quantum superposition +quantum superposition principle +quantum switch +quantum t design +quantum technology +quantum teleportation +quantum thermodynamic +quantum threshold theorem +quantum tic tac toe +quantum to classical transition +quantum tomography +quantum topology +quantum trajectory approach +quantum transition +quantum transport +quantum triviality +quantum tunneling +quantum tunnelling +quantum tunnelling composite +quantum turing machine +quantum two level system +quantum uncertainty +quantum universe +quantum vacuum fluctuation +quantum vacuum thruster +quantum vortex +quantum walk +quantum wave function +quantum wave packet +quantum well +quantum well laser +quantum wire +quantum yield +quantum zeno dynamic +quantum zeno effect +quark +quark gluon plasma +quark lepton complementarity +quark model +quark nova +quark star +quarkonium +quartic interaction +quasi harmonic approximation +quasi set theory +quasi static approximation +quasi stationary current +quasiatom +quasielastic neutron scattering +quasielastic scattering +quasiparticle +quasiparticle excitation +quasiprobability distribution +qubit +qubit channel +qubit coherence +qubit entanglement +qubit field theory +qubit frequency +qubit gate +qubit measurement +qudit +quench dynamic +quenched approximation +quenching +query complexity +quiteron +qunit +qutrit +r parity +r process +r symmetry +rabi cycle +rabi frequency +rabi hamiltonian +rabi model +rabi oscillation +rabi problem +rabi resonance method +racah parameter +racetrack memory +radial mode +radial number +radially polarized +radiated electromagnetic energy +radiation chemistry +radiation cross section +radiation damping +radiation field +radiation impedance +radiation length +radiation pressure +radiation pressure force +radiation reaction +radiation resistance +radiation zone +radiative auger effect +radiative energy loss +radiative heat transfer +radiative reaction +radio frequency +radio spectrum scope +radioactive displacement law +radiofrequency coil +radius of convergence +raman amplification +raman cooling +raman laser +raman nath regime +raman optical activity +raman process +raman scattering +raman spectroscopy +raman transition +ramsauer townsend effect +ramsey interferometry +ramsey spectroscopy +random bit +random coil index +random error +random laser +random matrix theory +random number +random phase approximation +random pure state +random state +random variable +random walk +randomized benchmarking +randomness +range criterion +rapid phase transition +rapid single flux quantum +rapidity +rare earth ion +rare earth ion doped crystal +rarita schwinger equation +rashba spin orbit coupling +rate equation +rational conformal field theory +rational number +rayleigh criterion +rayleigh jeans law +rayleigh scattering +rb atom +rb vapor +rds 3 +reaction center +reaction coordinate +real hilbert space +real time mri +realist interpretation +realistic interpretation +realistic theory +reality +realized experimentally +recalescence +reciprocal lattice +reciprocation theorem of green +recoil energy +recoil temperature +recrystallization +rectangular potential barrier +recycling antimatter +redfield equation +redshift +reduced density matrix +reduced density operator +reduced dynamic +reduction criterion +reed muller code +reeh schlieder theorem +reentrant superconductivity +reflection coefficient +reflection high energy electron diffraction +reflection law +reflectometric interference spectroscopy +refraction law +refractive index +regenerative amplification +regge theory +regular +regularization +regularly pumped laser +relational approach to quantum physic +relational quantum mechanic +relative entropy +relative error +relative frequency +relative intensity noise +relative locality +relative phase +relative state +relativistic addition of velocities +relativistic breit wigner distribution +relativistic causality +relativistic correction +relativistic covariance +relativistic dynamic +relativistic effect +relativistic electron beam +relativistic hamilton equation +relativistic heavy ion collider +relativistic lagrange equation +relativistic motion +relativistic particle +relativistic quantum chemistry +relativistic quantum field theory +relativistic quantum information +relativistic quantum mechanic +relativistic quantum theory +relativistic regime +relativistic runaway electron avalanche +relativistic symmetry +relativistic theory +relativistic transformation of acceleration +relativistic wave equation +relativity theory +relaxation +relaxation dynamic +relaxation process +relaxation rate +relaxation time +relaxometry +relic particle +remanence +remission +remote sensing atmospheric boundary layer +remote state preparation +renner teller effect +renninger negative result experiment +renormalization +renormalization group +renormalization procedure +renormalon +renyi entanglement entropy +renyi entropy +representation theory of the galilean group +representation theory of the lorentz group +representation theory of the poincar� group +representations of the atmospheric boundary layer +representer theorem +reproducing kernel hilbert space +reptation monte carlo +repulsive interaction +repulsive state +research reactor +reservoir engineering +residual chemical shift anisotropy +residual dipolar coupling +residual entanglement +residual resistance ratio +residue +residue theorem +resolution +resolved sideband cooling +resolved sideband regime +resonance +resonance condition +resonance energy +resonance enhanced multiphoton ionization +resonance fluorescence +resonance frequency +resonance raman spectroscopy +resonance state +resonances in scattering +resonant absorption +resonant cavities +resonant cavity +resonant coupling +resonant enhancement +resonant excitation +resonant frequency +resonant inelastic x ray scattering +resonant interaction +resonant line shape +resonant state +resonant tunneling +resonating valence bond theory +resource theory +response factor +response function +response time +rest frame +rest mass +resting state fmri +restricted open shell hartree fock +reststrahlen effect +resummation +retarded green function +retarded potential +retarded time +retrocausality +retrograde condensation +reverse reconciliation +reversible circuit +reversibly assembled cellular composite material +revival time +rho meson +rhodamine +rhodamine 123 +rhodamine 6g +rhodamine b +riafp +richard feynman +riemann hilbert problem +riemann surface +riemann zeta function +rietveld refinement +rigged hilbert space +right hand rule +rigid rotor +ring cavities +ring cavity +ring laser +ring laser gyroscope +rishon model +ritz method +robert w. bussard +robin boundary condition +robin bullough +robinson oscillator +rodriguess formula +rolling ball argument +ron rocco +ronens golden rule +room temperature superconductor +roothaan equation +roper resonance +rosenau hyman equation +rossiter mclaughlin effect +rotating frame +rotating magnetic field +rotating wave approximation +rotation group +rotation operator +rotational invariance +rotational spectroscopy +rotational state +rotational symmetry +rotational temperature +rotational vibrational spectroscopy +roton +round trip gain +rovibronic coupling +rp process +rrkm theory +rst model +rubidium +ruby laser +rule 184 +rule of mutual exclusion +runge gross theorem +rushbrooke inequalities +rushbrooke inequality +rusi taleyarkhan +rutherford atom model +rutherford backscattering spectrometry +rutherford medal and prize +rutherford model +rutherford scattering +rydberg atom +rydberg blockade +rydberg constant +rydberg correction +rydberg excitation +rydberg formula +rydberg ionization spectroscopy +rydberg klein rees method +rydberg molecule +rydberg ritz combination principle +rydberg state +s factor +s function +s knot +s matrix +s process +s pulse +s wave scattering length +sacla +saddlepoint approximation method +sagnac interferometer +sagnac interferometry +saha ionization equation +sakurais bell inequalities +sakurais bell inequality +sam1 +sarg04 +satellite laser ranging +saturated absorption spectroscopy +saturated spectroscopy +saturation dome +saturation magnetization +saxion +scalable architecture +scalable quantum computation +scalable quantum computer +scalable quantum computing +scalable quantum information processing +scalar boson +scalar curvature +scalar electrodynamic +scalar field +scalar field theory +scalar meson +scalar potential +scalar tensor theory +scale free ideal gas +scale invariance +scale relativity +scaling behaviour +scaling dimension +scaling law +scaling limit +scaling property +scanning confocal electron microscopy +scanning electron microscope +scanning mobility particle sizer +scanning squid microscope +scanning squid microscopy +scanning transmission electron microscopy +scanning tunneling spectroscopy +scattered light +scattered photon +scattering +scattering amplitude +scattering approach +scattering channel +scattering cross section +scattering from rough surface +scattering length +scattering matrix +scattering media absorption spectroscopy +scattering of particle +scattering of radiation +scattering phase shift +scattering potential +scattering problem +scattering process +scattering state +scattering theory +schaefer bergmann diffraction +scharnhorst effect +scheelite +scherrer equation +schmidt coefficient +schmidt decomposition +schmidt number +schmidt rank +schmidts orthogonalization +schoenflies notation +schottky effect +schreinemakers analysi +schrieffer wolff transformation +schrodinger cat state +schrodinger equation +schroedinger cat +schroedinger equation +schroedinger field +schroedinger functional +schroedinger group +schroedinger newton equation +schroedinger picture +schroedingers cat +schroedingers cat trilogy +schumann resonance +schumann runge band +schwarzschild black hole +schwind eye tech solution +schwinger dyson equation +schwinger function +schwinger limit +schwinger magnetic induction +schwinger model +schwinger parametrization +schwinger variational principle +schwingers quantum action principle +sciamachy +science by press conference +scientists against nuclear arm +scientists for global responsibility +scissors mode +scop formalism +scotophor +screening by atomic electron +screening effect +scully lamb form +scully lamb laser +search algorithm +search for the higgs boson +second harmonic generation +second order correlation function +second order phase transition +second quantization +second solar spectrum +second sound +secondary electron +secondary emission +secret key rate +secret sharing +secular resonance +secure communication +secure direct communication +secure key distribution +secure key rate +seed nucleus +seesaw mechanism +seiberg duality +seiberg witten theory +selected area diffraction +selection rule +selective laser melting +selective laser sintering +self adjoint extension +self adjoint operator +self amplified spontaneous emission +self assembled quantum dot +self consistency principle in high energy physic +self correcting quantum memory +self energy +self energy part +self focusing +self force +self induction coefficient +self organized criticality +self organized criticality control +self pulsation +self pulsing +semi classical approximation +semi classical limit +semi classical theory +semi empirical mass formula +semi empirical quantum chemistry method +semi rigid molecule +semicircular potential well +semiclassical analysis +semiclassical approach +semiclassical approximation +semiclassical behaviour +semiclassical description +semiclassical dynamic +semiclassical gravity +semiclassical limit +semiclassical method +semiclassical model +semiclassical physic +semiclassical propagator +semiclassical regime +semiclassical solution +semiclassical steady state +semiconductor bloch equation +semiconductor laser theory +semiconductor luminescence equation +semiconductor microcavities +semiconductor microcavity +semiconductor nanostructures +semiconductor optical gain +semiconductor quantum dot +semiconductor quantum well +semiconductor ring laser +semidefinite program +semifluxon +semileptonic decay +semilinear response +sensitivity +separability condition +separability criteria +separability criterion +separable operation +separable quantum state +separable state +sequence transformation +sequential walking +sergio focardi +series acceleration +sery expansion +sfermion +sgoldstino +shannon entropy +shape invariance +shape invariant potential +shape resonance +shared entanglement +shared randomness +sharp series +sharp transition +shelter island conference +shielding effect +shiftcor +shim +shin sung chul +shinnar le roux algorithm +shiva laser +shoolerys rule +shors algorithm +short distance +short laser pulse +short pulse +short range interaction +short supermultiplet +short time behaviour +short time limit +short time scale +shot noise +shot noise level +shot noise limit +shpolskii matrix +siberian salamander +sic povm +sideband cooling +sidney kimmel +siegbahn notation +sievert chamber +sigma baryon +sigma model +sign problem +signal beam +signal field +signal photon +signal to noise ratio +signal variance +signal velocities +signal velocity +silicon burning process +silicon vacancy centre +silsbee effect +simon glatzel equation +simons problem +simplified perturbations model +simulated annealing +simulated fluorescence process algorithm +simultaneous measurement +sindo +sine gordon equation +single atom +single bit +single cavity mode +single crystal diamond +single electron +single electron spin +single electron transistor +single ion +single mode approximation +single mode cavity field +single mode fiber +single mode field +single mode optical fiber +single mode radiation field +single molecule +single molecule experiment +single molecule magnet +single nitrogen vacancy center +single nv center +single photon detection +single photon detector +single photon emission +single photon generation +single photon interference +single photon level +single photon pulse +single photon sensitive camera +single photon source +single photon state +single photon wave packet +single quantum dot +single qubit gate +single qubit measurement +single qubit operation +single qubit rotation +single qudit +single semiconductor quantum dot +single spin measurement +single trapped atom +single trapped ion +single two level atom +single two level system +singlet fission +singlet state +singleton field +singular perturbation +singular trace +sisyphus cooling +six state protocol +size consistency and size extensivity +skin depth +skin effect +skyrmion +slater condon rule +slater determinant +slater integral +slater type orbital +slaters rule +slavnov taylor identitie +slip melting point +slitless spectroscopy +slope efficiency +slow light +small angle neutron scattering +small angle scattering +small angle x ray scattering +small arms weapons effects simulator +small tight aspect ratio tokamak +smart onboard data interface module +smith purcell effect +snapshot hyperspectral imaging +snellius law of refraction +snells law +sobolev space +social thermodynamics theory +society for applied spectroscopy +society for cryobiology +soft collinear effective theory +soft photon +soft photon emission +soft retooling +soft susy breaking +soft tissue laser surgery +soft x ray emission spectroscopy +sokol eshelon +solar neutrino +solar pumped laser +solarization +soldering +solenoid +solenoidal vector +soler model +solid harmonic +solid state device +solid state dye laser +solid state laser +solid state nuclear magnetic resonance +solid state nuclear track detector +solid state physic +solid state quantum memory +solid state qubit +solid state system +solidus +soliton +solomon equation +solvated electron +solvatochromism +solvent suppression +sommerfeld expansion +sommerfeld kossel displacement law +sommerfeld parameter +sonochemistry +soret peak +source field +soviet laser pistol +soxs +space dimension +space like separat +space time +space time symmetry +spacetime dimension +spacetime geometry +spallation neutron source +spaser +spatial coherence +spatial correlation +spatial degree +spatial dimension +spatial distribution +spatial entanglement +spatial filter +spatial inversion +spatial light modulator +spatial mode +spatial profile +spatial resolution +spatial separation +spatial structure +spatial wavefunction +spatially offset raman spectroscopy +spatiospectral scanning +spatiotemporal wavefunction +spc file format +spdc +special conformal transformation +special sensor ultraviolet limb imager +special theory of relativity +specific heat +specim +speckle pattern +spectral analysis +spectral atlas +spectral band +spectral characteristic +spectral component +spectral correlation +spectral decomposition +spectral density +spectral diffusion +spectral distribution +spectral energy distribution +spectral feature +spectral filtering +spectral flux density +spectral function +spectral gap +spectral hole burning +spectral imaging +spectral line +spectral line ratio +spectral line shape +spectral phase interferometry +spectral property +spectral purity +spectral resolution +spectral signature +spectral singularity +spectral statistic +spectral theory +spectral wavefunction +spectral width +spectro polarimetric high contrast exoplanet research +spectrochemistry +spectrofluorometer +spectrohelioscope +spectrometer +spectronic 20 +spectrophotometer for nucleic acid measurement +spectrophotometry +spectroradiometer +spectroscopic notation +spectroscopy +spectrum analyzer +speed of light +spekkens toy model +sphaleron +spherical bases +spherical basis +spherical bessel function +spherical capacitor +spherical cavities +spherical cavity +spherical coordinate +spherical harmonic +spherical model +spherical wave +spherium +sphingomona +spin +spin 1 particle +spin 1/2 particle +spin 1/2 system +spin angular momentum +spin bath +spin boson model +spin canting +spin chain +spin charge separation +spin chemistry +spin coherence +spin component +spin connection +spin contamination +spin correlation +spin current +spin decoherence +spin destruction collision +spin diffusion +spin dynamic +spin echo +spin engineering +spin ensemble +spin entanglement +spin exchange interaction +spin foam +spin glasses +spin group +spin hall effect +spin label +spin lattice +spin lattice relaxation +spin lattice relaxation in the rotating frame +spin magnetic moment +spin model +spin network +spin operator +spin orbit coupling +spin orbit interaction +spin pair +spin polarization +spin polarized electron +spin precession +spin projection +spin pumping +spin quantum number +spin qubit +spin relaxation +spin representation +spin spherical harmonic +spin spin interaction +spin spin relaxation +spin squeezed state +spin squeezing +spin state +spin statistics connection +spin statistics theorem +spin stiffness +spin symmetry +spin system +spin tensor +spin transfer torque +spin transistor +spin transition +spin valve +spin wave +spin � +spinhenge@home +spinless fermion +spinless particle +spinlock srl +spinmechatronic +spinodal decomposition +spinon +spinor +spinor bose einstein condensate +spinor field +spinors in three dimension +spinplasmonic +spintronic +spontaneous breaking +spontaneous decay rate +spontaneous emission +spontaneous emission rate +spontaneous four wave mixing +spontaneous parametric down conversion +spontaneous parametric downconversion +spontaneous symmetry breaking +spontaneous wave function collapse +spontaneously broken +spring peeper +spurion +sputter coating +square lattice +square lattice ising model +squashed entanglement +squeeze operator +squeezed coherence +squeezed coherent state +squeezed light +squeezed state +squeezed thermal state +squeezed vacuum +squeezed vacuum state +squeezing generator +squeezing parameter +squeezing property +squeezing spectrum +squeezing vacuum +squid +stability analysis +stability of the solar system +stabilizer code +stabilizer formalism +stabilizer group +stabilizer state +staggered fermion +standard model +standard model extension +standard quantum limit +standing wave +stanene +star product +stark effect +stark shift +stark spectroscopy +state collapse +state discrimination +state disturbance +state estimation +state evolution +state merging +state preparation +state reconstruction +state reduction +state space +state tomography +state transfer +state universal coupled cluster +state vector collapse +static disorder +static forces and virtual particle exchange +static imperfection +static light scattering +static magnetic field +stationary phase +stationary phase approximation +stationary state +stationary wave integrated fourier transform spectrometry +statistical distribution +statistical ensemble +statistical error +statistical field theory +statistical fluctuation +statistical inference +statistical interpretation +statistical mechanic +statistical mixture +statistical operator +statistical physic +statistical prediction +statistical property +statistical theory +steady state +steady state entanglement +steady state free precession imaging +steady state solution +steane code +stebbins whitford effect +steering +steering inequalities +steering inequality +stellar nucleosynthesis +stereoelectronic +stereolithography +stereoscopic spectroscopy +sterile neutrino +sterilization +stern gerlach experiment +stimulated emission +stimulated raman adiabatic passage +stochastic cellular automaton +stochastic differential equation +stochastic dynamic +stochastic electrodynamic +stochastic evolution +stochastic interpretation +stochastic local operation +stochastic master equation +stochastic matrix +stochastic mechanic +stochastic model +stochastic process +stochastic vacuum model +stokes boundary layer +stokes operator +stokes parameter +stokes shift +stokes theorem +stone von neumann theorem +stong basis set +stop squark +stopping and range of ions in matter +stopping power +storage time +storing quantum information +strain encoded magnetic resonance imaging +strain scanning +strained quantum well laser +strange b meson +strange matter +strange quark +strangelet +strangeness +strangeness production +streak +stream function +stress tensor +strictly correlated electrons density functional theory +string net liquid +stripping reaction +strong correlation +strong coupling expansion +strong coupling limit +strong coupling regime +strong interaction +strong subadditivity of quantum entropy +strongly correlated material +strongly correlated quantum spin liquid +strongly correlated system +strongly coupled +strongly interacting boson +strongly interacting regime +strongly monotone +strontium ruthenate +strontium vapor laser +structural property +structure based assignment +struve sahade effect +stueckelberg action +sub doppler cooling +sub poissonian statistic +sub shot noise +subatomic particle +subatomic scale +sublimation +sublimation apparatus +subsequent measurement +subsystem code +subtracted dispersion relation +sudden birth +sudden death +sudden quench +sudden transition +sugawara theory +sum frequency generation +sum frequency generation spectroscopy +sum rule +sum rule in quantum mechanics +super linear algebra +super minkowski space +super poincar� algebra +super poissonian statistic +super qcd +super resolution +super vector space +superalgebra +superatom +supercharge +supercommutative algebra +superconducting cavities +superconducting cavity +superconducting charge qubit +superconducting circuit +superconducting coherence length +superconducting detector +superconducting device +superconducting flux qubit +superconducting logic +superconducting magnet +superconducting magnetic energy storage +superconducting nanowire single photon detector +superconducting phase qubit +superconducting quantum circuit +superconducting quantum computing +superconducting quantum interference device +superconducting qubits coupled +superconducting radio frequency +superconducting resonator +superconducting steel +superconducting system +superconducting transmission line resonator +superconducting tunnel junction +superconducting wire +superconductivity +superconductor classification +superconductor insulator transition +superconformal algebra +supercritical angle fluorescence microscopy +supercritical liquid gas boundaries +supercurrent +superdeformation +superdense coding +superdeterminism +superdiamagnetism +superfield +superfluid helium +superfluid phase +superfluid state +supergeometry +superghost +supergravity +supergrid +supergroup +superinsulator +superlattice +superluminal communication +superluminal propagation +superluminal signaling +superluminal signalling +supermanifold +supermathematic +supermatrix +supermembrane +supermetric +supermodule +supermultiplet +supernova nucleosynthesis +superoperator +superparamagnetic iron platinum particle +superpartner +superposition +superposition principle +superposition state +superpotential +superradiance +superradiant laser +superradiant phase +superradiant phase transition +superresolution +superselection +superselection rule +superspace +superstripe +superstructure +supersymmetric gauge theory +supersymmetric partner +supersymmetric quantum mechanic +supersymmetry +supersymmetry algebra +supersymmetry as a quantum group +supersymmetry breaking +supersymmetry breaking scale +supersymmetry nonrenormalization theorem +supertrace +surface acoustic wave +surface charge +surface charge density +surface code +surface core level shift +surface current +surface energy transfer +surface enhanced raman spectroscopy +surface extended x ray absorption fine structure +surface hopping +surface impedance +surface layer +surface phonon +surface plasmon +surface resistance +surface state +susceptibility +susceptibility weighted imaging +susskind glogower operator +svetlichny inequalities +svetlichny inequality +swampland +swap gate +swedish radiation safety authority +sweeping jet actuator +swihart velocities +swihart velocity +swissfel +switching time +symmetric group +symmetric logarithmic derivative +symmetry group +symmetry in quantum mechanic +symmetry property +symmetry protected topological order +sympathetic cooling +symplectic geometry +symplectic group +symplectic structure +symplectite +synchrotron light source +synchrotron radiation +synthesis of precious metal +synthetic gauge field +system bath coupling +system bath interaction +system density matrix +system environment coupling +system environment interaction +system evolution +systems of orthonormal function +systems of unit +systolic freedom +sznajd model +t hooft operator +t hooft polyakov monopole +t hooft symbol +t j model +t meson +t symmetry +tachyon +tachyon condensation +tachyonic antitelephone +tachyonic field +tachyons in fiction +tactical high energy laser +tadahiko mizuno +tadpole +tailing in graphics or image revelation +talbot cavities +talbot cavity +talbot effect +tangloid +tardigrade +targeted temperature management +tattoo removal +tau +tau neutrino +tauonium +tavis cummings model +taylors theorem +te wave +tea laser +technical noise +technicolor +technological applications of superconductivity +telecom band +telecom wavelength +telecommunication wavelength +telegraph equation +teleportation +teleportation fidelity +teleportation process +teleportation protocol +teleportation scheme +tem wave +temperature coefficient +temperature dependence +temperley lieb algebra +temporal mode +tensor analysis +tensor force +tensor network +tensor operator +tensor product +tensor product of hilbert space +terachem +terahertz spectroscopy and technology +terahertz time domain spectroscopy +term symbol +terra 3 +tesla turbine +tetracene +tetraneutron +tetraquark +tetron model +thallium barium calcium copper oxide +theoretical limit +thermal bath +thermal blooming +thermal casimir effect +thermal casimir force +thermal conductance quantum +thermal correction +thermal desorption spectroscopy +thermal emission spectrometer +thermal energy +thermal entanglement +thermal environment +thermal equilibrium state +thermal excitation +thermal field +thermal fluctuation +thermal hall effect +thermal infrared spectroscopy +thermal light +thermal motion +thermal noise +thermal photon +thermal printing +thermal property +thermal quantum field theory +thermal radiation +thermal reduced state +thermal reservoir +thermal shock +thermal state +thermally stimulated current spectroscopy +thermionic emission +thermochromism +thermodesulforhabdus norvegica +thermodynamic entropy +thermodynamic equilibrium +thermodynamic function +thermodynamic limit +thermodynamic property +thermodynamic quantity +thermodynamical limit +thermodynamical property +thermofield dynamic +thermonuclear fusion +theta meson +theta representation +thirring model +thirring wess model +thomas athol rafter +thomas factor +thomas fermi model +thomas jefferson national accelerator facility +thomas precession +thomson cross section +thomson formula +thomson medal and prize +thomson problem +thomson scattering +thomsons theorem +thought experiment +thouless energy +three body force +three body interaction +three body problem +three body recombination +three dimensional space +three jet event +three level atom +three level system +three qubit entanglement +three qubit pure state +three qubit state +three qubit system +three stage quantum cryptography protocol +ti polaron +ti sapphire laser +tight binding +tight binding approximation +tight binding model +time asymmetry +time bin encoding +time delay +time dependent density functional theory +time dependent electric field +time dependent frequency +time dependent green function +time dependent hamiltonian +time dependent harmonic oscillator +time dependent magnetic field +time dependent parameter +time dependent perturbation theory +time dependent potential +time dependent schrodinger equation +time dependent schroedinger equation +time dependent variational monte carlo +time dependent variational principle +time dilatation +time energy uncertainty relation +time evolution +time evolution operator +time evolving block decimation +time independent hamiltonian +time like separation +time of flight +time operator +time resolution +time resolved fluorescence energy transfer +time resolved spectroscopy +time reversal +time reversal invariance +time reversal symmetry +time reversal symmetry breaking +time scale +time symmetry +time variation +timeline of nuclear fusion +timeline of quantum computing +timir datta +tip enhanced raman spectroscopy +tm wave +tnp atp +toda field theory +toda lattice +toffoli gate +tokamak fusion test reactor +tokamak � configuration variable +tolman electronic parameter +tomographic probability representation +tomographic reconstruction +tomography +tomography imaging spectrometer +tomography laser mammography +tonks girardeau gas +top eta meson +top quark +tophat beam +topicity +topness +topoflight +topological charge +topological code +topological defect +topological degeneracy +topological entanglement entropy +topological entropy +topological insulator +topological invariant +topological order +topological phase +topological phase transition +topological property +topological protection +topological quantum computation +topological quantum computer +topological quantum computing +topological quantum field theory +topological quantum memory +topological quantum number +topological qubit +topological state +topological tensor product +topologically protected +topos theory +tore supra +toric code +toric code model +toroidal ring model +total active reflection coefficient +total angular momentum +total angular momentum quantum number +total energy +total momentum +total noise +total reflection +total scattering cross section +totalitarian principle +townsend discharge +toy model +trace distance +trace norm +tractography +trans planckian problem +transactinide element +transactional interpretation +transcranial photobiomodulation +transfer efficiency +transfer function +transfer matrix method +transformation property +transformation theory +transformer +transistor laser +transition amplitude +transition dipole moment +transition edge sensor +transition frequency +transition matrix +transition of state +transition point +transition probability +transition rate +transition temperature +translation invariance +translation operator +translational invariance +translational motion +translational partition function +translational symmetry +transmission coefficient +transmission electron aberration corrected microscope +transmission electron microscopy +transmission line resonator +transmission probability +transmission raman spectroscopy +transmission spectrum +transmissometer +transmit information +transmittance +transmon +transmon qubit +transport efficiency +transport property +transuranium element +transverse doppler shift +transverse electric wave +transverse electromagnetic wave +transverse field ising model +transverse ising model +transverse magnetic wave +transverse mode +transverse relaxation optimized spectroscopy +transverse spatial mode +transverse vector field +transverse wave +trap electrodes +trap frequency +trapped atom +trapped atom ion +trapped bose einstein condensate +trapped ion +trapped ion quantum computer +trapped ion qubit +trapping frequency +trapping potential +traveling wave solution +treated classically +treated quantum mechanically +tree kernel +triality +triangular lattice +tribimaximal mixing +tribochromism +tricritical point +trident laser +trimaximal mixing +trinity paradox +trion +tripartite entanglement +tripartite state +tripartite system +triple alpha process +triple point +triple resonance nuclear magnetic resonance spectroscopy +triplet state +tristimulus colorimeter +trochoid +trojan wave packet +tropospheric emission spectrometer +true muonium +truncated wigner approximation +tsallis entropy +tsirelson bound +tsirelsons bound +tunable diode laser absorption spectroscopy +tunable laser +tunnel coupling +tunnel injection +tunnel ionization +tunnel junction +tunnel magnetoresistance +tunneling dynamic +tunneling probability +tunneling process +tunneling rate +tunneling time +tunnelling time +turbulent atmosphere +turbulent quantum channel +turing machine +twin beam +twisted geometries +twistor theory +two atom system +two body dirac equation +two body force +two body interaction +two component bose einstein condensate +two dimensional array +two dimensional case +two dimensional correlation analysis +two dimensional electron gas +two dimensional hilbert space +two dimensional infrared spectroscopy +two dimensional lattice +two dimensional nuclear magnetic resonance spectroscopy +two dimensional potential +two dimensional system +two electron atom +two level atom +two level quantum system +two level system +two mode gaussian state +two mode squeezed state +two mode squeezed vacuum state +two mode squeezing +two parameter family +two particle system +two path interferometer +two photon absorption +two photon entanglement +two photon interference +two photon photoelectron spectroscopy +two photon resonance +two photon state +two photon transition +two point correlation function +two qubit case +two qubit entanglement +two qubit gate +two qubit interaction +two qubit operation +two qubit state +two qubit system +two sided cavities +two sided cavity +two slit experiment +two spin system +two state quantum system +two state system +two state vector formalism +two time correlation function +tyndall effect +type 1.5 superconductor +type i superconductor +type ii superconductor +typical subspace +u bit +uk atomic energy authority constabulary +ultra cold atom +ultra high energy cosmic ray +ultra strong coupling regime +ultrabright +ultracold atom +ultracold atom gas +ultracold boson +ultracold fermion +ultracold gas +ultracold molecule +ultracold neutron +ultracold polar molecule +ultracold quantum gas +ultrafast electron diffraction +ultrafast laser spectroscopy +ultrafast x ray +ultramicrotomy +ultrashort laser pulse +ultrashort pulse +ultrashort pulse laser +ultrasonic grating +ultrasound attenuation spectroscopy +ultrastrong coupling regime +ultrastructure +ultraviolet catastrophe +ultraviolet divergence +ultraviolet fixed point +ultraviolet photoelectron spectroscopy +ultraviolet visible spectroscopy +ultraviolet visible spectroscopy of stereoisomer +umbelliferone +umklapp scattering +unambiguous discrimination +unambiguous state discrimination +unbroken pt symmetry +uncertainty principle +uncertainty relation +uncleftish beholding +uncomputation +unconditional security +unconditionally secure quantum bit commitment +unconventional superconductor +undetected photon +unentanglements +unextendible product bases +unextendible product basis +uniform acceleration +uniform magnetic field +unique ground state +uniqueness theorem +unit sphere +unit time +unitarity +unitarity gauge +unitary dynamic +unitary equivalence +unitary evolution +unitary gate +unitary group +unitary limit +unitary matrix +unitary operation +unitary operator +unitary representation +unitary time evolution +unitary transformation +universal behaviour +universal conductance fluctuation +universal control +universal feature +universal gate +universal gate set +universal linear accelerator +universal property +universal quantum computation +universal quantum computer +universal quantum computing +universal quantum gate +universal resource +universal scaling +universal set +universal wavefunction +universality +universality class +unpaired electron +unrestricted hartree fock +unruh effect +unruh temperature +unsorted database +unstable particle +unstable state +unstable system +up conversion +up quark +upper critical solution temperature +upsilon meson +uranium rhodium germanium +uv completion +v particle +vacuum +vacuum angle +vacuum energy +vacuum energy density +vacuum expectation value +vacuum field +vacuum fluctuation +vacuum level +vacuum manifold +vacuum polarization +vacuum rabi oscillation +vacuum rabi splitting +vacuum state +vafa witten theorem +vainshtein radius +vakhitov kolokolov stability criterion +valence and conduction band +valence bond theory +valence electron +valley exit jet +valleytronic +van allen belt +van der waals force +van der waals interaction +van der waerden notation +vanadium gallium +vapochromism +variable pathlength cell +variational approach +variational method +variational monte carlo +variational perturbation theory +variational principle +varphi josephson junction +vector beam +vector boson +vector diagram +vector field +vector greens theorem +vector meson +vector meson dominance +vector model of the atom +vector plane wave +vector potential +vector radiative transfer +vector space +velocity field +velocity interferometer system +verlinde algebra +vertex function +vertex model +vertex operator algebra +vertical cavity surface emitting laser +vertical external cavity surface emitting laser +vertico spatially modulated illumination +viaspan +vibration theory of olfaction +vibrational circular dichroism +vibrational degree +vibrational ground state +vibrational level +vibrational mode +vibrational motion +vibrational state +vibrational temperature +vibronic coupling +vibronic spectroscopy +video spectroscopy +vienna ab initio simulation package +virasoro algebra +virtual black hole +virtual particle +virtual photon +virtual quanta +viscous fingering +visible light photon counter +vitaly ginzburg +vitess +vitrification +vnmr +voigt profile +voigt thomson law +voitenko compressor +volterra lattice +volume operator +von k�rm�n constant +von neumann algebra +von neumann entropy +von neumann equation +von neumann measurement +von neumann wigner interpretation +vortex beam +vortex state +voter model +vsepr theory +vulcan laser +vulvar intraepithelial neoplasia +w algebra +w and z boson +w state +wannier equation +ward takahashi identity +wards conjecture +water retention on mathematical surface +waterloo institute for nanotechnology +wattless current +wave equation +wave function +wave function collapse +wave function renormalization +wave guide +wave mechanic +wave number +wave packet +wave packet dynamic +wave particle duality +wave vector +wavefront shaping +wavefunction +wavelength dispersive x ray spectroscopy +weak coherence +weak coherent pulse +weak convergence +weak coupling limit +weak coupling regime +weak equivalence principle +weak field limit +weak hypercharge +weak interaction +weak isospin +weak localization +weak measurement +weak nonlinearity +weak probe field +weak trace class operator +weak value +weakless universe +wehrl entropy +weinberg angle +weinberg witten theorem +weiss domain +weizsacker williams method +werner state +werthamer helfand hohenberg theory +wess zumino gauge +wess zumino model +wess zumino witten model +western chorus frog +wet electron +wetting layer +weyl brauer matrice +weyl equation +weyl heisenberg group +weyl quantization +weyl semimetal +weyls eigendifferential +wheeler dewitt equation +wheelers delayed choice experiment +which path information +whispering gallery mode +whistler +white noise +wick rotation +wicks theorem +wide angle x ray scattering +widom scaling +wien filter +wien2k +wiens displacement law +wiggler +wightman axiom +wightman function +wigner distribution function +wigner eckart theorem +wigner function +wigner inequalities +wigner inequality +wigner quasiprobability distribution +wigner representation +wigner rotation +wigner transform +wigner weisskopf theory +wigner weyl transform +wigners classification +wigners friend +wigners theorem +wilson bappu effect +wilson loop +wind profile power law +winding number +wiswessers rule +witness operator +witt algebra +witten index +wkb approximation +wolf effect +wood frog +woods saxon potential +woodwards rule +work function +workable approximation to +world line +worlds interpretation +wronskian +wssus model +wu experiment +wu yang monopole +x and y boson +x ray +x ray absorption fine structure +x ray absorption near edge structure +x ray burster +x ray crystallography +x ray fluorescence +x ray fluorescence holography +x ray free electron laser +x ray laser +x ray magnetic circular dichroism +x ray notation +x ray photoelectron spectroscopy +x ray raman scattering +x ray reflectivity +x ray scattering technique +x ray spectroscopy +x ray standing wave +x(3872) +xi baryon +xstream system +xy spin chain +xylomannan +y(4140) +y(4260) +yamartino method +yang baxter equation +yang mills existence and mass gap +yang mills theory +yangian +yarkovsky effect +yarkovsky okeefe radzievskii paddack effect +yb:luvo4 +yetter drinfeld category +yoshiaki arata +yoshimine sort +youngs interference experiment +yrast +yttrium aluminium garnet +yttrium barium copper oxide +yukawa interaction +yukawa potential +yvette mattern +z n model +z(4430) +zc(3900) +zeeman doppler imaging +zeeman effect +zeeman energy +zeeman slower +zeeman splitting +zeeman sublevel +zeno dynamic +zeno effect +zero curtain effect +zero field nmr +zero field splitting +zero phonon line and phonon sideband +zero point energy +zero point fluctuation +zero sound +zerona +zeta function +zinc selenide +zindo +zitterbewegung +zm 87 +zone axis +zz diboson diff --git a/.github/entrypoint/artifact/python/particle.ipynb b/.github/entrypoint/artifact/python/particle.ipynb new file mode 100644 index 000000000000..0e551656a592 --- /dev/null +++ b/.github/entrypoint/artifact/python/particle.ipynb @@ -0,0 +1,374 @@ +{ + "cells": [ + { + "cell_type": "code", + "execution_count": 30, + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import pandas as pd\n", + "import numpy as np\n", + "from scipy import interpolate" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# class Particle\n", + "\n", + "*The Particle object represents a particles with id given by:*\n", + "\n", + "|QUARKS|no|LEPTONS|no|GAUGE AND HIGGS BOSONS|no|\n", + "|------|---|------|----|--|--|\n", + "|d| 1|$e^-$|11|$g$|(9)21|\n", + "|u| 2|$\\nu_e$|12|$\\gamma$|22|\n", + "|s| 3|$\\mu^-$|13|$Z^0$|23|\n", + "|c| 4|$\\nu_\\mu$|14|$W^+$|24|\n", + "|b| 5|$\\tau^-$|15|$H^0$|25|\n", + "|t| 6|$\\nu_\\tau$|16|||\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# class QCD\n", + "\n", + "*This class includes pdf and splitting functions.*\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## $\\alpha_s$" + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "metadata": {}, + "outputs": [], + "source": [ + "import sympy as sp\n", + "from sympy import I, pi, oo\n", + "sp.init_printing()" + ] + }, + { + "cell_type": "code", + "execution_count": 94, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAO8AAAAaCAYAAABfLwsTAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAI4ElEQVR4Ae2ciVEdORCGgXIAzziChQx8RGCcAawjWJyBKSLYYjPwOgIbZwCOYNdkYG8EHBmw/6enVmnuS/Nmxn5dNU8aXf13q1vq0QzsPj4+7oxJu7u7K41/7nkc+PQP8X0Yk2+XsZeAsYs827a/hgae5MWUIR+r7EHOdZ2v63l/obHeWV+N/0H5b7oOrWwG6RIwzkBNWwhTaEA+g32e5XnvxQXecd8kdFyGP9W4RxGfC+UPVPY8Kps6uwSMU+toy386DXySv1zl2QfnVaULb+NdMt+45z277r89+26q2xIwbkoXWz4z04B88kaQLuWj72Nou6pw996zL3X/d9wgdV582HmPxWdOYXNGzCVgzADe3vwSGpBdfpegL+Q77rzIOa8KOUj6psKnbbTg27NbsVvbwRNjXNU5v/oRKl/qCgCUH4XmiFGY0Bfyv9R1N+UCNicseQPw2E5V/kaX2RfNnH3Z3Ep/hedAGm2aNoVXfNh5X0nuEyejMqQcIvFQvNN00U4XB07P821VhmGyCBTGUblzbqWrsvqUZeIxa4zoSNeHlDL3HWtOWJBBhIHe+zRjKyrjMJW5ZQc67Stzyn7CsTG84sXij5KcXpyT+YKjOqF8R4yO1a/gnFamehSfMUzd47iE5MaPHfjA7lOlGhPhZo0RWUX88OhQqcdN1c0Fi3AwdxzK4JgZp411oTo2GvSX3H5iPk15MOraOF7xxL7dwqV0B0cK3lwFWm3o9L2q3srVht1Xt8FRbceFj11MQOUEWd+uqcZcAkZO3hv13VX2Pu2FY05YmDsW/lq78Jjv+8ibso9wTIJXfPEdt4HynpcJ/CHBHpSWkmJtOuB4L0obZAt/cKs+hNWckiGk7YhUOVJdePdrZUPSJWD08vEcV6vvIXro2HcWWDR3hMLY10mdHXrZ7pR+9vlJkonxEpn8juA47ytddY7LzsnhwbV3RmVrCUeF9vlRn1aHYLTtS1Lm7DFGsrFY3ggzz0oQp+6spF/c3WZ/Jsfi5w5dsKC10QGbA5vJJDQDvGyIK+FY4bw4G6tZFdmJXluFcZIKuR14nR39dwkYiUbQNTsM5D4R9WX3Sg9lvBvT2YywWATWyr6kIzYaDLiWkM+3rW3Xo3IUvB1wmK8e4LzskHVGw+oMtVKY2mGcD5s0RPFbAkbBDDhfm2GRytCo4yT1LzK6J4yEbnU9UxtbnFxhoh/TWRMWFhzCtHfC0eaxqSs85IZSfY5ri+R/0iOyNdrtmn3r38F4hQvdM6dsdDwCsCBB6JoyFp6q7yCs7f4T38G8WbcFIiQl/K1zcOvkYnHdjGFsxqMsXQJGcPOMeSNd2gSYoVH3jB/vuLdqY458pDLCavqmpDZYWIjNyTGsMYjNA2pjX05fsf7WXQu/5yoBL3aR2nkH4xX+a80pun2pvO3kul2T6jgnqqLgqzgvhmSAyjq0Vqo6s2NgnKN+pVUCcjBG7zQMPeZux6qa/1SUMuifdeLOF177PIsmE32li1cjreS0vg1pIxbxw/B5Pj9uGGtINcbYKsT1ONBBpUOqDYvNJ12c07hFXWlKSoWXxbMq2qgqRw7z1bu9FlKhqIwSUFDJhFqot/76o8XACZsMwihZwM5ud6aLHQ9nKXwIngAvq21+VTV94aTomR2DBTUm7umbkmqxpGTUMJYzVC97Q1P3dVGl4/rO9pYDJ+cwNjWlwssiE2xM8scLJItPFWEfkLMR3stWfnihOiY5fFSgPCeDMGaQ9zJ2JW6V4x1d7xfn6svqy07T+VLfQRjVH+yZL8Z0j2AFeVQ2BCdjho9hGEsXvO2lu8mR4aH6whdF9O2jK+uj/rVYrB2pCMOqfMevut5Y1JcFCybOlmK+cV71LLC1fOIxlOcArOprv9pxYr75vMYcjFdj2Dw7m/P3lT4YY1BbfA99rfb0Q7hGCFVK6shKxzPsR1sdVcZrIzz/UGUoiRDgN5X1Cus0hjNipfldScM20xCMXib4I09M3KPkQENxaiBehTD5Riycn4W/6TFjX+3A6CgBDsbpi2UNwv8OxeJthujjQmMR6mZIZce6cNw/1TY/R6Gt2mDUcbjJghfr2rVVO/TI6X5fW8PGh+IFK/TWy/ZR+aaIwnXQDzI9oAuOORmI7ftpg3IwZDsIMEYomxfrQWkCY2GLqtqT+qFsqPcfLcBb/Tth9H2YyIz8Hg/fe2ccawhO9cVwMERkLbzfVT0T4+rENyyEKmelRc/hPegQHBrLDsoqsdDGSLzYedFF6QnoUCzw0RimG2ecnvedUj6rDfblywuJ+rOJxM7NOODOzCsdp8Yr/i5cllzuEFL3zpk7yLmvticuRJU8hG4hnFNFq9BVfQARvtFVnpB6SEiCsnv3L8Ot8Woxqh6Hd2FI3F9l6KQ0lFN5cpzG2/NtG8KPhsPwkHp5K8PmqE3SuYsx1OWFrzBPKsN5AZ/RpY2j8o3ozvjFqXhX2lbcriyvvmw07jFrTxmId01uFXB33X7e0lyrB8rgv3DEq1+3kdZ/7jSkfxW/OozGj9A0ppVuwu4XVyjPn2VZv1zV4Ft2egur0Ct5TvDLsIyJo6sgk2Dx+inMRaQvnLiMpsILHmyrMZrIg5as9GOzwV/d55GkhBxfdXV9P4tB8S9k3PKsfO+X+B7YrcZITbUYmWTxZvJRTJ7s8SCUj4jT8RCeM/Hg+Y+QFn0QqoZXR66RfsbGAR/xwNB4D8kCwr8uwk7Ygd07aOUdbQKL8YpTj4dvC3CE8Hjj8aA/6Fz3RJLxIwdzPYatOYZVP8JBZOo2ElLdE/52cWJk/aI+2KviCh8iK8vhSQiBrbwuVXtWAUIArk598+Oq/yh/n6lxGzGqDRMdQi/lMdaqk8pRcOb10XQvfLPAAc45YWnS2xLxmkzSM+ch4Q1I/G9wWI2+qmHv3VP9F0taBW2ltt2Od77rFW6xUm2B/ywakH1yOMyZQoh6gvMipBrYc2vhk62fRQlbObYaWJoG5Jc8vvAPLjLnUnuxIKrkuYCvi8KBSVy/zW81sNXAJBpgM7Uv8QKA/wGK0WrBgZxbEwAAAABJRU5ErkJggg==\n", + "text/latex": [ + "$\\displaystyle \\left( Q^{2}, \\ Q^{2}_{0}, \\ b_{0}, \\ b_{1}, \\ C_{A}, \\ C_{F}\\right)$" + ], + "text/plain": [ + "(Q², Q²₀, b₀, b₁, C_A, C_F)" + ] + }, + "execution_count": 94, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "Q2, x, Q02, b0, b1, CA, CF = sp.symbols('Q^2, x, Q_0^2 b_0, b_1, C_A, C_F')\n", + "Q2, Q02, b0, b1, CA, CF" + ] + }, + { + "cell_type": "code", + "execution_count": 109, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJkAAAAaCAYAAACkeP7MAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAG4klEQVRoBe2agXUcNRCGfX4U4JgKcDpwcAdJBwYqIOnAeVTACx0EOiB0YFMBIR0kHcS+Dsz/ySOdpJW0t3u7x56f9d5Y0sxIM5oZjbQ6r+7v748eS1mtVidayy+2njOrf9Ya149ljXOsY3a7EWQxaBGXgpcx7lDa0vt9rCt9wecY99RO/Y09prKb5nlXsm8pwBJHlQYtFadFYrGwQdQmm4E7X6rOS9DLbLSz3bCz4Dpf07GQrviUKYY3HrePWnL9sTaFOHT/2DfRxDL7xM1Cn3gNk9hNsfNJi/0g3a7iRYcggyjgeNlbMWWI/kmKFvm7IL5/YbwvtvhYxlluiJi49PaS7YYPZL830pH7sSsrIY+EIJv8q/Yzw89eSSZ3vwvJfNsSZroRLCjtAwh9ScssqFg0juBl47wQnx8XeEVntxGAfwXkATQOwW5mW3z7gzOpGtRksOKlDfrUIFkEDEHdnBud4BN07lTCEUDFOYR3Qaj6pCWD8X08rfH7pqErOvfJFc//ajfTk8By9ndOVie5MPctYle65BHUr2vzoJyAAOhcIuMxot8Jkg8V9QmwD55PbTLame/HtfCvBcn4mL60Nrqic00v0RZjN+mC/5yuqp0TQtTVFjAlXjLvWvOJjoK9Tw/iIZtpqoeMqLbPYASWBxxTzWiiEahVup97CTW6tvQQfTF2ky7Y3SUJ1UfcTXod2lrcEJpkvRRUU75oKEfQd47IXI54OBYCr9oEDP0E8nFxX7xhx8X4pbWl50HZTfoSV25THKtxIVgLmkWXOb7I3gvecbGzfvKp2pxgQ+QyeLPpblrMqR5H2I2czOdwX+F4oJzyR2OeCfiYSQBao6DLqwZ9NMlsNIXN0OHQ7Ib/TmQD56NrdfruPuwiskS42zBGQMYYdNSIv5o5RPNZ7FKB0vwogG5z0Qh6bTMu5tFYgrqaWWPeIW3NOZnNkIuOguJ9TPjF2U06cV1B8XMyGVmgmsksErn7vNViv6j2hTZPANWxnjGriezbDOe7OIbSm8VMLxayzvRyEwz4gy5utw0Y02SdwWbIOzS7+bg4JchayrM47j3w/EknKgTEmDemVlBzXJLB4mCORCbNH63XfGdLRpQ7yHJyy+RR2KlthhKHZreQSL6R8kQcC6gVnMkdyUcmj7cEHY7hyJyybBNcXj6O/CS9kgdZ6Qae8lXwreh9QdhynptoxJ992gz1drbbiDX22c3H1C1BVi0WTARUfny5LCIH1i7wOBonE5i34oszHgZhzlJBDr8EhCIdyJjc++I5fCA9vCgbtwXYV/H+BoqxAu6brYs9uoRdZ1ONriSP+YApbYY+s9mNyUeUPrtBp6w5LlvKOy79+cc3rMa5zogy6qUgHDdq+/cRHP1R8JON8RUODfweafWv1Mxp9ZXhb4RzbdVc1Alyfi7Kd7D7MrUxHLtsAgKtJg9Wdlw+j8+W0MeWKW2GDnPabcwai3aLJgqZjCDDGN9HxNCUk9bqEEzBSeZk+gQQhd+oYidB45kDh/NhkGQb4ZjvQtAp4oXG8faHDwzh/FH9XDgCmKz0XSaToEAuuwed40L/PEZk7RfqJ8e+5mKeO9V80Q0q0gt5U9sMHWax26DFpcwdu6VkFzN8lDl/cBzxqVl8igAv4OuSTMIxhcNwqMclj6ZG94+iybOHBIrsPu2bj782B/PjfGQCzBX+58nmCrJFQ6/OOoT7LCh++tsc0DtPIDYOWtEujK0BYwTePjvbzPTET5PbrbaGPjy6CDp28+NEIyG4n/ecodTpONAzD6kRKnBOsTaZ4CqfQzgUDAGS00t98RNw4f1MbYI+BIDatSBjbR0dkKGCvlXHicaxHWSU9NoVZzr02sz0ndxuY/Q3nat2M13xvdvcx2pQeJ5oXY4d0xZ/CAR39EoQRyh3otLHAbubf98ZWtz9TscYzn8lGS4V2yS+fZpNSlaJj/OYzNGMLrXCVcDPW+PZFb+tzZAzh93G6N+0m1032PQPz14WdSCaP77C1weag3sYAUDt2rUxomPcarrNx4nXv2qTgoBOJhSOrJXgjbcjR3iyWPWXDtEIzmIGzHXbpS8ZW9sMOSqT222I/pLftJvpyJo2/wnjBYAUhOPI4+eqJQsnVp2cyxWv2wiqCaSinsKz00NgqM09pviTkfA4q3oUila9x+W67bOPzui+rUzx9tpt27ngQ7agajfjSe5r7j9jGWkp7m8x8dWwlyKZ7Aou88mD6i7CNac//niney7g57DkyBMPO42v1toxuosKs4+dw27bKL2N3YyHIHRvlcwbgsx1NnedMfelbfR84nnEFrDg559Ak/u9v/i7pYvIq/q1mDlmnsqTBYZagOSUv4se/Qfjy0cO5L61PQAAAABJRU5ErkJggg==\n", + "text/latex": [ + "$\\displaystyle \\left( a_{s}({Q_0^2}), \\ a_{s}({Q^2})\\right)$" + ], + "text/plain": [ + "(a_{s}({Q_0__2}), a_{s}({Q__2}))" + ] + }, + "execution_count": 109, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "a0, a = sp.symbols('a_{s}({Q_0^2}), a_{s}({Q^2})')\n", + "a0, a" + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAIUAAAAwCAYAAAA/3bn6AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAFsElEQVR4Ae2cjVUUMRDHOR8FoHZwdoBagdiBaAVAB/gsATtQOxA7ACtA6UA7EOgA/799mZj9uNvkdrm9i5n3wmazk8nMP5PJx94yu7+/3ynUj8BsNtsX17nSc2F2119jezl2t1f1h9dcjrCnVr4o3Si9UJorZU/FKZZ0sYsIh7DIQU51IVpkT4+yt7AYmIxAcYpkyPKvUJwi/z5OtrA4RTJk+VcoTpF/HydbWJwiGbL8KxSnyL+Pky0sTpEMWf4VilPk38fJFhanSIYs/wqz8kJseSfreJuXYLwD4d0H199K10pXwu6jrtlRcYrsunS4QWX6GI5hdhKKU2TXpcMNKk4xHMPsJBSnyK5LhxtUnGI4htlJKE6RXZcON6g4xXAMs5Mwk0UcyHx311gDD3VwwwFOoQwRGO3wSid/5VuBTBxkNKfIBI9ihhAoa4riBi0EilO0ICkFk34M5L7A+uC6Ye6uR+4jnKjekYy5+HlzWSgSgT7Mpo4UZ+rQ9y7xJRaf5/2MtO2/+morFpNIvrkcgy/eOmlqpziWcgeBZmfKo3Dv53nieSPep3Kob1ZfZXtKF0q3Sr+sfFOvU+krzC7BRO2DYYsmnT6kzYnSj5ZWPQWAKZYPMu55yKr7O92/1nOiTbLcUNY68lPqq7Y/gpPSpdPDmzxppJAynxsK4SS/VdZ3MEZE+eStaGeINBft4vFLBOqB0vFAyWvTt6EnGIJlndQBO5uQpBXAEPL3+vQRz+0iHj1jOuIgrVfOIhkp5WqHEHyaUifkVd216hu2TV50q1TDatJIgUaQRtpcFzy29x+CMDLFt2y38ZrnMpipZBtoan3B8m0I1OhrCnUa872F05fKHynR6e+UIH7wGi4OK4dQGeDgIESMO90v6nh2KdVCCf4OwmmuJcdW1890fxG22VFnyqJR9E3FPTAYLMH+sy8TWKNOHxL8yWQqT8fwa+gqvPJM6VfwHIdg7scRLMFTC2fGz1XEIvI4LLM89ZRggqeSEZRxnjGqrU6flaePQLfB+kpWNO4hDqrHAP4Zlj1SwWgkb6WB5sIF0CwyPGk8BwxGCldLdPiy0E/Hc57RRciCXpkMu6rMb7+k55lLp1yrGtP8idWXrTbbdzBqEc9U2LRjGe6hDLAEU0+7PjdO5oc6IQz7hKVrK9O1+ldB1pTuH1s+4YpjLXIaa88/F2Bm8FPa0D3g/VHb1Tcbumf3wPRSTV/wdJF4iGDWiSEL+iCXnVOTsL1mc4MhRl8iqLVrtjTE7CTh3qhMfxGx/5GUHj2kmky1QihfeWVucsKr5LFaPgjLLK9yRpIPo5TDq0Tmjbun/r7VcWU8X2l6Qa7SSjaqXq++pqdrx0+9Vt51FW807uIFn9pubtTpQ8I9MQLdjU0djCbC4CJv93V7Mnj2IhmMqmaItZF6qbYZEdS9UwqJe+qum5bqu4oyK+AOHrXpeDSncB3OEbM5QxVO5cnhdMIpZLNDUm3HgHq4q0vw7aGTHrHdOnHtcg/VQHD3T6on6/+zTN9ebUbAHbu9DjQ4mlNIFs5AunGdAfDeAZyzXKlsKF1LwMsFQohKocOw8/kqh/i33equCDDmMN0cD1O6qr6hNkNx51VB7fR3N5Q+MM9+F/BRknXKiRyByMECjZB+ozI/lcCzImEAMrvoSIXsLNgKcz7B+iJs05wUJ7C8spVD1EYLhWugPn1jVBiKO/1V37kItAdbaD6UbBnBcXhtsRjbluptzEIzVmf4RCxooxaasXIlj6jakjnm9CH5ayM8u2sLGKOAj2Ywu2nNb5tjBDR4iDhh1Gk83ujb99KuHiVUsLU/3GVqkv4sIJPDvuoaEH8kg2mGH/psZMdKV0YzA4Awv6+EUzO6B/1vDCeX6bV1PrPNTrEncM67jFJ5oR4E3KDi+53WYNhap8Bm5+0cZPXtLnog+r8eCzeOxflxTWeU/Qu5SQi/lrfnfwAAAABJRU5ErkJggg==\n", + "text/latex": [ + "$\\displaystyle - \\frac{1}{x^{2} \\left(b_{0} + b_{1} x\\right)}$" + ], + "text/plain": [ + " -1 \n", + "──────────────\n", + " 2 \n", + "x ⋅(b₀ + b₁⋅x)" + ] + }, + "execution_count": 111, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "-1/(x**2*(b0 + b1*x))" + ] + }, + { + "cell_type": "code", + "execution_count": 113, + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxIAAAByCAYAAADH2OcMAAAACXBIWXMAAA7EAAAOxAGVKw4bAAAgAElEQVR4Ae2di9HUthqGgUkBXCo40AGQCkI6gJwKAh0kkwoypAOSCk5IB0AFCXQQOgD+DjjvYyTj9foi7dpr2X414/Va10+PJEufJdnXP3/+fG3t5vr16y+Uh7fKy+9rz4vlN4EpCbhtTEnzcnGp3G4qtTc6vtN97Wos5Vz/Y/Gt0d11fY2lZpkvQcBt4xKUp08j976e638qiW9MFdFS8YQGcttKxFIl4HRLJeC2UWrJjMsVlIcf5fNN6BwGA+X6H4xshY6u6yssNIt8EQJuGxfBPEsiuff1XP9TCb1qRUIN5KlAPBS8J1MBcTwmsAUCbhvrL0Xd194pF8y2vkzJTa7/lDjX4Md1fQ2lZBmXIOC2sQT1adPMva/n+p9C2utKdIp4Lh6HGsh9JcrU/3+Uh9Gp/4sL6ARNYCECbhsLgZ8pWZXnK0X9Sve531KSyPWfEmepflzXSy0Zy7U0AbeNpUtg2vRz7+u5/s+Rds2KxFtl/LU615/PAeCwJrA1ArqBuG1sqFBVnneVnX913NP97v1Y1nL9j8VXsrvresmlY9mWJOC2sST96dPOva/n+j9H4lUqEgL0WJlmuv+WZyPOKX6H3RoBt43lS1RlwGzpHzo+6v70/RQSKU6WON1NjS/X/xQyXjoO1/VLE3d6ayHgtrF8SakMdtMPrFWR4OncX+pUdzkbEW4SV8r/6+WbS54Ekp230fwSQvGkFfOj8pK1PE3xPN9r+X9B1v0rLptpG1PVlW5S89pKdh50vJ+qjio+z0q0imxLdb2VtaRL5Z8Hau4HdjoOGKokW2obysskY4YhXnO5SfZd9AOrUyRUMGh5LN3Y5WyE8k/n8b0GKM/mqvxzxiv5XzRl51rpPZLdvZx0Qz1AmZjkiW9O2qX6DUw20zamqitLlJdk/6R0eXUrG6YnMYqTvRLvFGfSA5Rc/5MIeaFIlDf3A+4HroV64H6g0e621jaUn0nGDA1EF/sr2XfRD6xRkUDDu7nHAaQqJZr5G+X9wcVaghJSuiypGF2bnSKT4mJ3P4pQNZtC3LrmKfoD2dWDrpQ05Ye3dlEXkjahyu+mjXgs3jZSyi21EBTXZHUlNc0p/EnuOMiNA/5vFW/2rFtbFsVLfWfQdKvt1nWd678rjlLtlLfF6/pSbJR39wMN+KGeux8ITEpoG5KhyDFDo9rM/lcMdtMP3Jid5vQJxP0R08dcfox0njzBv5hRY/hJidEgpjLMpPyTENndkHavVw2o+ADhM/mjY7W5dm3RtlFyXblw5Xik9FC8fw9K7t/6/3wCGVC+b4oz5Zxicv2nxFmKn0Xr+sIQ3A80CsD9QAPGl7+Ltg33A3V57KYfmFSRYECn45WOTzp4yjypUZxxQHuRvQGkp+OtDvLDzXsxo/R5cs83My729W6lyQ3pjtL8ayjjyKbjuY4X4cz/lzp4gnpgkF/HVcMSxYK15PVsBG66jjMWY4MmFCs2ti5ilMci6ghyBAAXaRtt2Ep/DXWlLfZc1yy3Yzo+1nPOD0lMnGJbeaz/tBPadZJRfCgnHEnL+XL9JwmR4El5cj+QwOkUL6G+uB84hud+QExUP9wPpD9oOa5F09rspx9QZ8OgbdJDZcE6bTrSqeNlYKpop413LD6liVL0cszfnO7w1MGyhkmZ9sWntHjK/7bPPdojE/503I928Sw7lK/eOAijA7ZMS3fmi/Aj7sjJEpjeOPrintJe6S9aR5T+Im0DhrDX0VvOkbP8LFpXohxzn5VP1sWy76eq0/pP263uHzofcGpfxzB95xDXQRx9frHP9T8UV64beSP93HBj/hXnInVd6S7axuECTx3uB0LbinVFTNwPfKkfi7SNUDfdDzTqperkbvqBSWckBC4aBohsDJzaoOEdPLmeOoGe+CbZH9ATd6o1N4g5mPalz6CPTqvThKeOUYE42N8QA+jmwhfHeQJ7FE94skYahL2KYTrOVcfZYV9ZhbDUiR/6/KTYS55HOmB8qlm6jizVNuC1irqSUrAT1AOSoUNtLt+jbv5PcWPfnoGgfWCfahjMxqeOKWFy/afEmerH/UAqqXR/7gc6WLkfqKG4HwjKdk3khD/uB/KgTa5IUABBhDmWWMQ1Z3m5XLlvMY0Dh+bgZO5c/aCb89AyqjcSgFkAblxDhnpwMEBXfhhMVW/aoAMgf8HuKJ4gww9yHxpswQWl5RxD/ENpnBP3JcIu2TbWVFfGymKKelArlaq31P1/VI9ZHnhbx8eWAFy3lYuWl4PLKm7azIFt/0Wu//6YMlwkn/uBDF4pXhtl7n6gG5j7Ac2ECk19/+nGNJut+4FDtHU5bL0fmFyREEcGlqx5vzpkOskVnXxdOJPEuGAkqlypA4jq5jAT0yMCYRDQy1nuzBIwkEkZvFfxKEw18Al5Jvyv2AV79km0B1iyqg1xDM048NS1WoNeh9jIn8ArJTeLtA3JN9hxyb20upLC8lw/tAv2P/CiAr5GHZVt6jjKRNNwnXOvjO0y9d6R678p2zn/3Q8k0sto4+4H3A+M1Sr3A18IjY0ZxjhO4b6bfuCbKWi14uBm9y50ojjxfYBX4Ykc1+caBo1FGOWRRsuyjigTeWUt9MFsjPzBpNpUrDMDAJYoVQ1ebrAZ+4bCt/I/ONhQPMTL6ybx90EHT0AfK+5TXo1KAzjIg64rE9LhKetrxf0uWA+dyCcmDqCqPQ+65lwbxQWfPoMsDEz6ZkiQgw2ezJAMcupLYC57ZFLcl6gjZCHWw8Hs7LyuDLKZwjG0i6P6TN0U+ziwj0nx0KVtF926ztFvbE9dfpp2uf6bYc/5737A/UCz/rgfcD9Q0pihWTdn+b+nfmBSRSIMmu6HUqnemx7seOsRT+Zip5ZdcAofn8ANPbnOjvfUAJKHfLKZmG8i1PmSPW+tYuNxNYAP/lAc6g/oyY5NOD/Lz1/6nzLw5Sbcm2/FQaeNLOw3qGSRHWmy7r/9liRZjxqe7vMkucvEd+P3ubfDxJmCSi7Jl/QO/FYkDJDJY5+JbKgjKBVFGLG/SB1ROsltQ373XleWrhtPVAYoln/r4AEBSnuyUftBGcF/HJgNhs31PxhZoqPkQzb3A+4HmjXG/cDMYwW1O/cDX2vc2Jjhq89l/m2qH7gxMcM42OOLrtUAOZ6VzuOYFh2Njqc6Dp5KR/eec1LH2RN2DmsG7n8pf7USERJhoM2yhtiR/qLr9lIvnrBXA3KF73zyH+KKp9v6U/GMFvEMS/1HFhSTpiz8b6cbg42diTMOztt+YxmPDtiDbHC4asnWjnPsGlmGyj+ygVNJ5lJ1ZIhNzcN1pUax2B/agY7qIUI4N9tsjlx3cjzLb67/zOgPvMd7hPsB9wO8kpT7k/uBw/6ZBjP1WMH9wNfb0NiY4avPBf5trR/4ZmKGLD95J0hxYBdvIiRTdWS6qXBDiR1NUsUnsEwcJNZxf7E+/g03rjdyyYn/ieQeHRyTWsgD2j9PFQ8Mccgdu//qSIrvIILuC/JBw+gyPN3E/c+WI4xZ3nSKgXUf5+qpBw0hIeK4r6FSmhL893khrfi0pctPH5sjvyobZlJi/Wu6V/VL7kdLUuSJOp315PjCdSS1bbiuhBKfqx4o3uodnc2KlfNf9ay6eSSEybm3EV2u/wQRer24H3A/0Kwc7gcuM1ZwP/C11o2NGSqf7ge+Ajvn39SKBNOX7TdKxCnNatDNQFt+GGzXMxSJGYiDxdEOUWkwCH6QGO8p3uKglnT6zP3gwMCV5U7N9fu4MahLNaQTbxLtMNykWXtYy0JaskPGOV4Xm6JARAWSPDII79vbIOckQ97r/HWEiGxiHenw8sVKsnQpCshLfbwr91P2lHSld8k6EvNNuQ8Z15VAZ656oHhTFYGhclq7m/uBryXIvR7jfqDVD+ieG/tA9vTx4dOxB07uB6qq1Puz2X5ghrpSQXQ/0FuXshymViS4aXLDbJr4JDdlCU8zXMn/42B6aOAW/aBYMZD+Q40hasm8+vTcwXUcrCPDuxas6gmQ0uhkHholN28G5x/lrz1zgZx9eSOtAyVQ8fGEH0WpGU/sJGL5y8vJBlniTbIrkigr+SnFxPKPsnXJFf3MVkdioioj5ODYe12JSFZ5DuWI7Kkb62P9S/I/ERT3A19Bzt7G19i2JTP9wwf1GXEvIfv5ePEIs1l9hrrsfqCPToK960oCpBV4CeWIpEn39Vz/uQhu5AZI8B9vnHGgy6D2mW4QVwlhh7zE8DyVWNQoLwzGkOfopqcCi8tmokLFNW9yYukUa6M55yoRMOUm2mfaS6wYvFcDRsnzWEd8Ok6ZVE/GJAM3cAawLMFqG27WdZiW469cE284/xTcX8uu+q8zb3Wi3OvN38HPqSfKvK5XHZHEOjHUyXQEm89KfC9ZR3Laxt7rynyFHmJW/ee1xm91zDEjGOt6LPOx/OT6H4sv1b1ur+LAvcv9gPuBZt2p3vwXLXS/5KEXykRfv4NX9wMRWPc53hNim+/29cV2Tf3AHHVliM0kbqrLu+kHplYkeCLdvBG81PWfJwycuwoyDhLplEow30kIPpR2vyUMT1p+CwNJnGjc1ebrULFyv2RLHDR6lgscGKVB3AxYa+ZKg0bHNUoC5lv5qzt1XeOGPPhj42fXjAFxfqvjyIR8MQXNDEuVruzi0qp7skNRQcH6Tyvdo7gyLFimNjQoQw42dMcbaUbUs3q9VB0ZbRuBjevKrMX9JXKxhjNtjvPUJt7/YpmPxZ/rfyy+FHf3A1+Wrrof6OgHQr9BvWzfr7lu96fN+uZ+oEnj+H+8J8Q2f+Rjbf3AjHXliM3UFrvqB5TZa1MdKggqMANJnkxz5jsGnfHjpuPfPvcue/nntakvutzmsFNa3NQYwLKBkrT5zzr6Kk/810E+UR44UJweRfd4lh1vpyKO5sGU1E/Rz9BZ/pjVICzLhw54YqeDdGGODMiMXNGOV9HWYYI7eYl5qvMT/cmN9AbLJsRDGjCJ+Sfeg/zjL8Z76llxwOpIzhif3CgDZn3qfOb+V3jqY1J5NONWmFLqyGjbkKy7ryvNsuv6f2o9aMeleCiPs+t+R7zxXnDQztr+4rVkyPIfw51zVpruB1r3QXjKuB/4woF75lF/Jjvu80/76l5wdz8w0M+J0ab6AeVnlrrSV8eivdI9aTwQw8dzKI/N9wMnD7wiqFPPoaAGB6vtuBWGGzHrKBeTOzdt5CWvMZz+08ky0K8UKZ2TBq/ydzRIj3GmnkO6lTIS/sOzM33Zc1PPagDy384rys2R8pMqL/5kYDVYT+ROPno7oJT0FH6SG0dKWm0/SrvNLbuOBAaTtQ3FB/fN1ZU2+/a18nx2PVAcsfOj/nOgcJ/VDqKcioeZxKNBWHRvn3P9t8PPfS35qvtgTjoK437gjD5Q/BZt20o/to+DNiF7+ri+/sj9QA+bZtsRv0nbhuLbZF1pMuv6r3y7H8i4x9wQsDUZlus8XIvAmpajMqL0MNVfGf2/0sGSIux4kn+0z+KLz6NfXu+a6vcocLBgwFrxQwb9Z11q54Zs2SNb5xuOZD9kqj0XIe98rO9qyHOCG8uokKXTKB0G3XRM8DnHIOe5smanP2EdmbptbK6uJBbOFPWAWQDaFx+DZC/S3zp667DccgzLO3K+D5PrP0eWpfxOXddnzceEbRw5t9APxPvs7Rb4m7puLsNtOrsfSOv/p24bS/cDk9eVZqUa+O9+YADOkZM6uos+3ZcAaLh0qmjOPFmrlkKlyCG/WU/jUuKc04/k5cbY+2RfbgdPoodkkV8Gy5+G/Iy5KTz8UG44V/+HwsgP8vVOJbfDyi9lSZnGI2tGoyM+6srgU3a5k4+zljW1073ktWSfpI4EDslPqsfyGOLbVF0Zy/NU7mJHu6mfrAaWb2P8uq7qtc4HT2Sj+9BZYbifJNf3XP9DaU/pJrncD4S+Vyx21w8oz0dL/2TH/euov8FOh/sB9d1jbVCc6A831Q8oP5PWlTGGU7lTZ3Xsoh+4DrS1mPD0mUrFm494ol+8CTL/EgT9EM53dGYAyX6P5A2ZioslEv+7VN6D7AxakmZC5B9l540OzI/nyqn4aIiUNU8HOo38MLBi5qPvSVZnuJIsA+ez6kiIY7G2EdIvuq5cqszFIpZDNdunaxTs26qjT/SfNsKTWOx4q1lv3Zb7gQmMiZu34I2++S3X/0FiBV808uV+4ALlFHhP2rYVJw8TD17/qmtei84M2oGRX/cDiWOFpdvGGurKQeWa8UIsdtMPrEqRoMzDTYWp/VOW3cxYbeaPOjTSN10327lSV5o8DWJj5+jAZUoZlC5PVngbVK+CEPzwVJflI7s34kGHu1jbKLmuXLJyiANPZ26pXlZKgq7pUA4Ua9mhAOcqEswQ8TDh3lC7kHtllEaW/xhuDWflbdG6viQj5Z2HUKvvB5SPuNyPB2z3dPB69APFWn7cD2RWtqXbhtKffMwwVV3JRHmWd8m8m35gjYoENxaeXNw6q5RXGliVk8EBT+B3p0g1iyzcrJjRSZotaYbd6n8x2XXbKKVcVQ61khDKhCfnB/W06SdVboVhFuOh4jp6atsVR67/rjhKtQtc3Q+4H2DQ6n6g0VD33jYaKBb9q3LYTT9wY1HSpyXOZrObKiQ2NO7OaBDBkq5Xe81/o8BRpLq+gdHwsru/u24bBZU29ZJvtfDGJmYPDpSIM+T8QWF/zQif6z8j6sW97rquux+o65/7gRpF/WfXbaOmsPyf3fQDq5uRoG6og2ZKlOU2SU/mlq9PlsAELkPAbeMynM9NReVUP61KiSsoJb/onpc0E5vrP0WG0vy4rpdWIpanFAJuG6WUxLAcKqdN9AOrVCQoGhUA646/U8eavFl5uEjtagLbIOC2UW45qmzYbM1sKpvreXLI22hGXxwRyvRgn4XC9ppc/70RFe7gul54AVm8xQi4bSyGfjRhlc2m+oE1KxKsB+ftJZ6VGK229rAnArpJuW1sqMBVniyR+m/qvS7X/xAqxYXSwys5L/qyhSGZmm6u600a/m8CXwm4bXxlsYV/uff1XP/nMFrjHokqv7FjC7DOYeCwJrApAm4b2ylO3d/YTMrsRdJ+oFz/CaR4QxBHkcZ1vchisVAFEHDbKKAQJhIh976e6/9cMVerSISMf6fzswDtXBYObwJbIuC2sfLS1H2NATyvOWVJU+9rkGM2c/3HcBs4u65voBCdhVkIuG3MgvVykebe13P9T5GTVSsS6lx55zRvRHkZ4E3BxHGYwOoJuG2svgjJAN+M4BWno3soQm5z/Ydg6z65rq+7/Cz9fATcNuZje8GYc+/ruf7Pzspq90g0cy4lgul/PkzmjddNMP6/ewJuG+usAuHBCN+MqL6OPZaLXP9j8UV3xct3a9gj8Vu0K/Xsul5qyViupQm4bSxdAqeln3tfz/V/mlTHoTahSBxnyzYmYAImYALnElDHtBpF4ty8OrwJmIAJmEA+gesKwjrcN+GcGgNfavXT/1Ra9mcCJmACBROQwsBXs7s+8nk7iP2xQ/x36geSNoF3hLWVCZiACZhAQQTCjEa2PjDZjIQE+FwQD4tiAiZgAiYwQEBKAA+SBk3ujIT7gUGcdjQBEzCBogik9ANjAn8z5iHVfQphUtOyPxMwARMwgfIIuB8or0wskQmYgAnMSWDVb22aE4zjNgETMAETMAETMAETMAET6Ccw2YxEfxJ2MYF1EwjrBvkoGIY3hGF4tz+vH7YxARMwARPYOAH3AxsvYGfvZAJWJE5G54A7IsC7/J/F/KpDYWPqWx33op3PJmACJmACmybgfmDTxevMnUrAS5tOJedweyLwVMpD8402z5X5u7K7vycIzqsJmIAJ7JiA+4EdF76z3k/AikQ/G7vsgADT1Tpe6fik49+eLDMb8U+Pm61NYMsEWL7nJXxbLmHn7Zr7AVcCEzidwGSvfz1dBIc0geUJqCNhqdI/zSVMfVLJLzMSj+XXS5v6INneBEzABFZGwP3AygrM4hZBwDMSRRSDhSiAAMuUXo3JoY4Gf3zt98GYX7ubgAmYgAmsioD7gVUVl4UtgYAViRJKwTIsSkDKQdz/8HpIEPm7K3dmIx74jU1DpOxmAiZgAusi4H5gXeVlacsh4Lc2lVMWlmQ5At8r6fdDykFUIuQHv6yp5cnVla7fc21jAiZgAiawagLuB1ZdfBZ+KQLeI7EUeadbDAEpBeyPQCH4OwjF3odXUhL+4jooEbzy9Weug2ED9s9Dykf06LMJmIAJmEDZBNwPlF0+lq5cAlYkyi0bS3YBAuo8biqZTzre6fgOxaBhd48ZB13jjr8DI7frBxa+MAETMAETWB2Bxj3f/cDqSs8CL03AeySWLgGnvzSBuD+iUiIQpjHLwKZqrm+hNLQP3GxMwARMwARWT8D9wOqL0BlYioAViaXIO91SCLAu9l1DeajeKR6Eu1OKkJbDBEzABExgNgLuB2ZD64i3TsCbrbdews7fGIGH8tD+2Bx2mLhnAuWCtzVhPui4I8WjuV+icvCPCZiACZjAKgm4H1hlsVnoEgh4RqKEUrAMSxLg7Utstm6aJ+Gieh1sUCI+oDzo+E1ufAl79JsTzQj93wRMwARMoFgC7geKLRoLVjoBKxKll5DluwSB+hWuYdPdD0r0WWO501Nd19+YkD3/H8nv3UsI5zRMwARMwARmJ+B+YHbETmCLBKxIbLFUnaccArzitakQvNT1n1IWfieSoCzwxqYrrhuGa55i2ZiACZiACaybgPuBdZefpV+QgPdILAjfSRdB4EdJ8VwKw0868/2IF1Iiqu9HBOnia18/hut44vp2vPDZBEzABExgtQTcD6y26Cz40gSsSCxdAk5/UQJh+RIfl8s1KBFRycgNa/8mYAImYAKFEHA/UEhBWIxVEvDSplUWm4W+IIG4pKk9+4ASUa+pvaA8TsoETMAETOCyBNwPXJa3U1sRASsSKyosi3p5AnpShbJAJ4Li0DZ8BdXGBEzABExgwwTcD2y4cJ21swlYkTgboSPYAQE2Xscvn7IBm/98xM4zEjsofGfRBEzABETA/YCrgQl0ELiuwVCHta1MwASaBKQ8ND9Ix6ZsvikRp7ubXv3fBEzABExggwTcD2ywUJ2lswlYkTgboSMwARMwARMwARMwARMwgf0R8NKm/ZW5c2wCJmACJmACJmACJmACZxMoXpHQVOJ9Hf/q6NrsejYARzAfAZfdfGwdswmYgAmYgAmYgAksTaDI70gEpeEPweGjXw913F0alNNPI+CyS+NkXyZgAiZgAiZgAiawdgJFKhJhE+sT4GpgyheH768d9F7kd9ntpaSdTxMwARMwARMwgb0TKH5p094LyPk3ARMwARMwARMwARMwgRIJWJEosVQskwmYgAmYgAmYgAmYgAkUTsCKROEFZPFMwARMwARMwARMwARMoEQCViRKLBXLZAImYAImYAImYAImYAKFE7AiUXgBWTwTMAETMAETMAETMAETKJGAFYkSS8UymYAJmIAJmIAJmIAJmEDhBKxIFF5AFs8ETMAETMAETMAETMAESiRgRaLEUrFMJmACJmACJmACJmACJlA4ASsShReQxTMBEzABEzABEzABEzCBEglYkSixVCyTCZiACZiACZiACZiACRRO4Prnz5+LFPH69esvJdhNHQ/D+b3O73T8LZl/09mmUAIuu0ILxmKZgAmYgAmYgAmYwIQEilUkJsyjozIBEzABEzABEzABEzABE5iYgJc2TQzU0ZmACZiACZiACZiACZjAHghYkVi4lLUM6JGOpwuL4eRNwARMwARMwARMwARMIIuAFYksXLN4Zh8Ih40JmIAJmIAJmIAJmIAJrIaAFYnVFJUFNQETMAETMAETMAETMIFyCFiRKKcsLIkJmIAJmIAJmIAJmIAJrIaAFYnVFJUFNQETMAETMAETMAETMIFyCFiRKKcsLIkJmIAJmIAJmIAJmIAJrIZA9R0JvTWozK/SrQSjPpB3fUxUMX4hP486/N0Odh873N4p7icd9rWVy65GcdKflLI7KWIHMgETMAETMAETMIGNE/AH6RYuYCkCjyXCXX+te+GCcPImYAImYAImYAImYAJZBLy0KQuXPZuACZiACZiACZiACZiACUDgm7Vi0JN8vr3wS5D/bjj/qCf7V2vNk+U2ARMwARMwARMwARMwgbUQuKgiocE/S3jeTwTnueJ6FuNS3OxBeKvjXrTjPHGazah39X9KjoprVAmcMr1dFZQzawImYAImYAImYAIXInCxpU0aGP6kPN2fMF9PFWdz8/JzxX1Xdu00sCNtmxMJzFB2KIE/h4PN5Gw0RwlsGpdbk4b/m4AJmIAJmIAJmEBhBC6iSGggyobiOxo4/jWUf/lj8Phcx4tw5v9LHU87wjEb8U+H/YGV0nyNheJAhiMj+5s6Xun4pOPfIw87txCTOcpuVAl0ue284jn7JmACJmACJmACxROY/a1NGoiyjOWNBoYPhmjIHzMKzDCwz+Fd06/cXuqaZVG9cYTwj+XnYGlTjEfuPPH+Tu6deyiC+z9yr5dLxbBznpUueSZvv8+ZzilxS7ZZyk7xohj+GctC1+xxQYl7ILt22RdZbqfwdBgTMAETMAETMAET2BKBSygS1d6FvoFyHKwK6kf5+b4Prvx9khuDz6OBvtxYzoSywUC0T1Fg8Ir7UXjZM2PBtzSeyH1w1gS/ezFiMnvZwVLpoER2KoFyc7ntpcI5nyZgAiZgAiZgAqsicAlF4pMG57f6qGigyBPnm/LTOZMQw8kfigKDzYOPv8mep9kMeFECOpWIRhwoI/9p+1MczAq80nGr7RbD7vEsLrOWHUyVRooS6HLbYwV0nk3ABEzABEzABIomMOseiTBA731Lk9xRABhIDn69ORCs4gkDz8pK/1Ei2Lj7PQoAbsEuBDk6EccPR7bXrjET8t5KxFcy4ohyNVvZkVIsP/3tnUnCn4zL7QsH/5qACZiACZiACZhAMTPj3SQAAAYlSURBVAS+aUsSBnc/y56n+x90sNSHmYDfdM41KAjVZud2wJAOy1ZeK+6DdfFtv+Ga9fqY2/yE8CgiP+s/ygiGZUvI3meQBaWhvR+BQfM7xRPf7sTsyCvJtaplToFJ8WUntrH8KiUwXFOGV2LepbxsutzIv40JmIAJmIAJmIAJrI3AgSKhgSgD6rjXoBrQyY4lP490/l2DvMGlQx2Zfyg7BvtdJg74+9zbYYgLEwea1ZIoXXOujWTs3AMRPLChlzzWRvlCQYmKSPVBu2DHW5zu9Qxs6/Cl/JGsqyk7yRqXo6UqgZstt1Lqj+UwARMwARMwARMwgVwCtSIRBs8oEbzfPw7WiY//py77YZDONwK6TBzQj85GBNkOnlhLxt59F12JBTtkiTMb0VuUo36jk+JmmRTuvPq0monRNRuCMczS3JGfqAhVlkv+rK3sxCpXCdxkuS1ZZ5y2CZiACZiACZiACZxLoFYkFBEDZQbZf7YiZaB96hKf2wp71YovXvJU+lpLaYlu7XPc13Du4B2lqEq3kQBLnd6hPES7MDDn8g4/QYn4ID9RqWCGhqVPvW+ZItwFzarKTtxylcCtltsFq4iTMgETMAETMAETMIFpCTQVCQbr7FdoD6gZeLO8aWrTnPXojTsM6hkoM9g/2NsQBviETZ0l6FJsWDL1D5E0TFxG9XewYy/Hd9FdcrxGkdDB9x+S8hHDznReXdllcthquWVisHcTMAETMAETMAETKIdApUiEwTqzEe1lRtVMAAPnLpHDQJ5BPMoH34Foz1wwyCbeLkNaLB2qjeJj9oNXwTbjiUuKDt7sFNOW35xZAmRhmUzTsGSqvU8jpoXCgCJFOPLYNFwTdlFFQvIhG8dqyk6y5hryt6lyywVg/yZgAiZgAiZgAiZQGoEbLYHiE/hozYC6GqBqwPo4DKorN/1n8M3yHgbyPNH/b+Vw+MPgr72UKPr4lT/EG84/BQcG79V/nZkJQJnh9aDtAXv1xqcQhiVSKDssOepLD6882W7Hg31tp/AMWknzmeK80plrTHsgyzXxlWLWVHa5zLZcbrks7N8ETMAETMAETMAEiiBQKRJhwIzCUA/CwyCe67js59vWYB6358Efm7HjU/xmxojz26ZF/C//uLHn4Y84+JddXFp1T3YoKuxB4ANy9UBf1/HVoQzwGeg3DdfMEvSZB3JoL9Ni9qPOt/6z4ZwvaB8so5Jd2zC4jUpG2+1i15KTPK+m7E4Es7lyO5GDg5mACZiACZiACZhAMQTqL1tr4M6g+A8dPNlmk/H/dDBIZWkRdgffe5B/BuxvdBAOf0ezBvLDUqUXGuz2frU6xPNLiIcBMYbZBr5UXS+pwp+uK/cQhjf/HHyJWva8JpRvE3QqAcGdj9fVionskJ88Evbo+xFyR8mo3FrhPsseGZvLsGR1eRPysIqyO4XOVsvtFBYOYwImYAImYAImYAKlEKg3W2tAjDLQNatwZBcG18xC3Ar/eYrPEqVqv0LMnNxZpsQMQq0ERLd4lh+Ug4M08C87BviV0TVLnTqVg+AlnnpnCRRHNeug9GolgkC6Jt+9357Av8Lip5aHcMFExSdeL3IOeThgGAQ5sgscSiy7TnZbLrfODNvSBEzABEzABEzABFZC4MaJcrI8qHqzURiYM3NQzx604uRpf+9AveW3eVntudBAEgWFWQQG89HE/ygOTcNg/0BRaDiyjApZTjEoMcyuVEYy8Z+3SPWlFXwWeVq67HKhuNxyidm/CZiACZiACZiACVyAQL20KSctDaRZevRRRxzId72xqY5S/hm8snk5aeAt/+yPII1oWDZ18PRffj7JkY/I1fayY7nR0deoZc9sBEusTv7ug+KISghvqWIJFB/uiwqNLtdhlI/Fyy6VlMstlZT9mYAJmIAJmIAJmMDlCZykSOSKqQEhMwUvUwfy8s/SJvZfYH5UuKN9CPLDwP7gI3G6Zn8EG3MPjPyiyLCfYXUD/4OMLHAxR9mlZsPllkrK/kzABEzABEzABEzg8gQuokiQLQ0KmRV4pMF8yl6HJBKKc3SWQH54As9G8aTZkKSEd+ZpjrIbQ+hyGyNkdxMwARMwARMwARNYlsD/AcEBLd11eKBjAAAAAElFTkSuQmCC\n", + "text/latex": [ + "$\\displaystyle - \\frac{b_{1} \\left(- \\log{\\left(a_{s}({Q^2}) \\right)} + \\log{\\left(a_{s}({Q^2}) + \\frac{b_{0}}{b_{1}} \\right)}\\right)}{b_{0}^{2}} + \\frac{b_{1} \\left(- \\log{\\left(a_{s}({Q_0^2}) \\right)} + \\log{\\left(a_{s}({Q_0^2}) + \\frac{b_{0}}{b_{1}} \\right)}\\right)}{b_{0}^{2}} - \\frac{1}{a_{s}({Q_0^2}) b_{0}} + \\frac{1}{a_{s}({Q^2}) b_{0}}$" + ], + "text/plain": [ + " ⎛ ⎛ b₀⎞⎞ ⎛ \n", + " b₁⋅⎜-log(a_{s}({Q__2})) + log⎜a_{s}({Q__2}) + ──⎟⎟ b₁⋅⎜-log(a_{s}({Q_0__2}\n", + " ⎝ ⎝ b₁⎠⎠ ⎝ \n", + "- ────────────────────────────────────────────────── + ───────────────────────\n", + " 2 \n", + " b₀ \n", + "\n", + " ⎛ b₀⎞⎞ \n", + ")) + log⎜a_{s}({Q_0__2}) + ──⎟⎟ \n", + " ⎝ b₁⎠⎠ 1 1 \n", + "─────────────────────────────── - ────────────────── + ────────────────\n", + " 2 a_{s}({Q_0__2})⋅b₀ a_{s}({Q__2})⋅b₀\n", + " b₀ " + ] + }, + "execution_count": 113, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sp.integrate(-1/(x**2*(b0 + b1*x)), (x, a0, a))" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.6100939485189322\n", + "0.24485952713564962\n" + ] + } + ], + "source": [ + "N_c = 3.0\n", + "C_A = N_c\n", + "n_f = 5\n", + "C_F = (N_c*N_c-1)/(2.0*N_c)\n", + "b0 = (11.0*C_A - 2.0*n_f)/(12.0*np.pi)\n", + "b1 = (17.0*C_A**2 - 5.0*C_A*n_f - 3.0*C_F*n_f)/(24.0*np.pi**2)\n", + "print(b0)\n", + "print(b1)" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAEOCAYAAACHE9xHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de3xcdZ3/8dcnk8k9adImaek1BUpKi5aWUMAqtFKgiFq8LEsBuYg/FgX8uepvxd+6yy7sb1d3vV+xasULFCoooKKAXKwol6ZCKQV6obQ0vaZJr7lNMvn8/phJnaa5TNJJZjLzfj4e88jM93zPzKffR/o+J2fO+R5zd0REJH1lJbsAEREZWgp6EZE0p6AXEUlzCnoRkTSnoBcRSXMKehGRNJfdXwczmwT8FBgHdAJL3f0b3foY8A3gPUAzcK27/zW67BrgC9Gu/+HuP+nvM8vLy72qqmoA/wwRkcy2evXqve5e0dOyfoMe6AA+4+5/NbNiYLWZPe7ur8b0uRiYFn2cBXwPOMvMRgO3ATWAR9d92N339fWBVVVV1NbWxlGaiIgAmNnW3pb1e+jG3Xd27Z27+yHgNWBCt26LgZ96xHNAqZmdAFwEPO7ujdFwfxxYNMh/h4iIDMKAjtGbWRUwG3i+26IJwLaY13XRtt7aRURkmMQd9GZWBDwAfMrdD3Zf3MMq3kd7T+9/g5nVmlltfX19vGWJiEg/4jlGj5kFiYT83e7+yx661AGTYl5PBHZE2+d3a3+6p89w96XAUoCamhpNwCMifWpvb6euro7W1tZklzKs8vLymDhxIsFgMO514jnrxoAfAa+5+1d76fYwcLOZ3Uvky9gD7r7TzB4F/tPMyqL9LgQ+H3d1IiK9qKuro7i4mKqqKiIxlf7cnYaGBurq6pg6dWrc68WzRz8P+Aiw1sxeirb9X2By9IPvBB4hcmrlJiKnV14XXdZoZncAq6Lr3e7ujXFXJyLSi9bW1owKeQAzY8yYMQz08Ha/Qe/uz9DzsfbYPg7c1MuyZcCygRTVqQM3IhKHTAr5LoP5N6fklbGHW9uTXYKISNpIyaA/1NqR7BJERNJGSgb9wdZ2dOcrEUknLS0tnHfeeYTD4SNtDzzwAGeddRazZs2ipqaGRx99FIBQKMS5555LR0didnpTMug7Op3Xdh5KdhkiIgmzbNkyPvjBDxIIBAC45557+PKXv8xDDz3EmjVrWL58Oddccw11dXXk5ORw/vnnc9999yXks1My6AGeWr8n2SWIiPRr3bp1LFy4kFNOOYU77riDW265hVWrVh3T7+6772bx4sUANDU1ceutt7JixQrGjRsHwLRp05g/fz5PPPEEAJdeeil33313QmqM64Kp4ZYfDPD0+j3ctODkZJciIiPAv/96Ha/u6H7B/vGZMb6E2943s88+ra2t/N3f/R2/+MUvOPHEE5k+fTpnnHEGZ5555lH9QqEQmzdvpmtW3nvvvZc5c+YwadKko/rl5ubS3NwMwGmnndbjBmMwUnKPvjgvm9Vb93GgWWffiEjq+sMf/sDs2bOZOXMm+fn5hEIhPvOZzxzTb+/evZSWlh55/corrzBr1qxj+q1Zs4bp06cDEAgEyMnJ4dCh4z+MnZJ79MV5QZodVm6s532zxie7HBFJcf3teQ+VF198kTlz5gCwY8cOioqKmDdvHtu2beP2229n1KhRLFq0iDPOOOOoqRpKSkoIhUJHvdezzz5LU1MT55133pG2trY28vLyjrvOlNyjL8gJUFYQ1HF6EUlpubm51NXVAfD5z3/+SHi//vrr5OTk8MlPfpKFCxdSVlZGOBw+EvaXXHIJK1asOHKF64YNG/jYxz7GsmXLyMqKxHJDQwMVFRUDmtOmNykZ9ADnnVLBH9fX06nLZEUkRV1xxRWsXLmS6upqZs2axTnnnMOnPvUpLrjgAm655RZuvvlmtm/fDsCFF17IM888A8DcuXP5whe+wMKFC5k+fTqXXHIJP/zhD3nXu9515L2feuop3vOe9ySkzpQ8dAOwYHolD760g5e3H+D0SaX9ryAiMswmTpzI6tWrj2n/3Oc+RzgcZvLkyVRWVgJw880389WvfpWFCxcCcN1113Hdddexe/du3v3ud1NYWHjUe9xzzz3813/9V0LqTNmgP3daBWbw1Ot7FPQiMqJ86UtfOqZt9uzZLFiwgHA4fORceoCxY8eybt26o/qGQiEuvfRSqqurE1JPyh66KSvMYfakUp7WcXoRSRMf/ehHjwr53uTk5HD11Vcn7HNTNugBFlRXsqbuAPWH2pJdiojIiJXaQT89cmxr5QbdWlBEZLBSOuhnji+hsjhXp1mKSI8ycfLDwfybUzrozYz51RWs3FBPR7gz2eWISArJy8ujoaEho8K+61aCA72IKmXPuumyoLqSFbV1vLhtP2dWjU52OSKSIiZOnEhdXd2Ab6s30nXdHHwg4rk5+DLgvcAedz+th+X/B7gy5v1OBSqi94vdAhwCwkCHu9cMqDpg3rRysrOMp17fo6AXkSOCweCAbpCdyeI5dHMXsKi3he7+P+5+urufDnwe+GO3G4AviC4fcMgDlOQFqakq46n1mbXVFhFJlH6D3t1XAo399YtaAiw/rop6sKC6ktd2HmTngZZEv7WISNpL2JexZlZAZM//gZhmBx4zs9VmdkM/699gZrVmVtv9mFvXaZZPa69eRGTAEnnWzfuAP3c7bDPP3ecAFwM3mdm5va3s7kvdvcbdayoqKo5aNq2yiAml+Tz1uk6zFBEZqEQG/eV0O2zj7juiP/cAvwLmDuaNzYwF0yv486a9tHWE+19BRESOSEjQm9ko4DzgoZi2QjMr7noOXAi8MtjPWFBdSVMoTO2WfcdbrohIRonn9MrlwHyg3MzqgNuAIIC73xnt9gHgMXdvill1LPArM+v6nHvc/feDLfSck8aQk53FU6/vYd7J5YN9GxGRjNNv0Lv7kjj63EXkNMzYts3AsTdFHKSCnGzOPnEMT63fwxfeOyNRbysikvZSegqE7hZUV/BGfRNvNTQnuxQRkRFjhAV99DTLDTr7RkQkXiMq6KvKC5laXqjTLEVEBmBEBT3A/OoK/vJGAy0hnWYpIhKPERf0C6oraevo5LnNDckuRURkRBhxQT936mjygwHdjEREJE4jLujzggHmnVzOk6/vyagbDoiIDNaIC3qABdMrqNvXwhv1Tf13FhHJcCMy6Od3nWapwzciIv0akUE/oTSf6rHFOk4vIhKHERn0APOnV/DCm40cbutIdikiIiltxAb9gupK2sPOnzftTXYpIiIpbcQG/RlTyijOzdZxehGRfozYoA8GsnjXKeU89Xq9TrMUEenDiA16iJx9s+tgK6/tPJTsUkREUtYID/rIvWV19o2ISO9GdNBXFufxtgmjdJxeRKQP/Qa9mS0zsz1m1uP9Xs1svpkdMLOXoo9/jVm2yMzWm9kmM7s1kYV3WVBdweqt+zjQ3D4Uby8iMuLFs0d/F7Conz5/cvfTo4/bAcwsAHwHuBiYASwxs4TfA3D+9Eo6HVZurE/0W4uIpIV+g97dVwKNg3jvucAmd9/s7iHgXmDxIN6nT7MmllJWENRxehGRXiTqGP05ZrbGzH5nZjOjbROAbTF96qJtCRXIMs47pYI/rq+ns1OnWYqIdJeIoP8rMMXdZwHfAh6MtlsPfXtNYjO7wcxqzay2vn5gh2EWTK+koSnE2u0HBrSeiEgmOO6gd/eD7n44+vwRIGhm5UT24CfFdJ0I7OjjfZa6e42711RUVAyohnOnVWCm0yxFRHpy3EFvZuPMzKLP50bfswFYBUwzs6lmlgNcDjx8vJ/Xk7LCHGZPKtVNw0VEepDdXwczWw7MB8rNrA64DQgCuPudwIeBj5tZB9ACXO6ROQk6zOxm4FEgACxz93VD8q8A3j29ki8/toH6Q21UFOcO1ceIiIw4/Qa9uy/pZ/m3gW/3suwR4JHBlTYw86sjQb9yQz0fOmPicHykiMiIMKKvjI01c3wJlcW5Ok4vItJN2gS9mTG/uoKVG+rpCHcmuxwRkZSRNkEPkZuRHGzt4MVt+5NdiohIykiroJ83rZzsLNPZNyIiMdIq6EvygtRUlfHUes17IyLSJa2CHiKHb17beZBdB1qTXYqISEpIv6CfXgmgOepFRKLSLuinVRYxoTSfJ3WcXkQESMOgNzMWTK/gz5v20tYRTnY5IiJJl3ZBD5Hj9E2hMLVb9iW7FBGRpEvLoD/npDHkZGfpNEsREdI06Atysjn7xDGaDkFEhDQNeojcNPyN+ibeamhOdikiIkmVxkEfPc1yg/bqRSSzpW3QV5UXMrW8UMfpRSTjpW3QA8yvruAvbzTQ2q7TLEUkc6V10C+orqSto5NnNzckuxQRkaRJ66A/68TR5AcDOnwjIhmt36A3s2VmtsfMXull+ZVm9nL08RczmxWzbIuZrTWzl8ysNpGFxyM3O8C8k8t58vU9RG5jKyKSeeLZo78LWNTH8jeB89z97cAdwNJuyxe4++nuXjO4Eo/PgukV1O1r4Y36pmR8vIhI0vUb9O6+EmjsY/lf3L1rroHngJS6M/f8as1mKSKZLdHH6K8Hfhfz2oHHzGy1md3Q14pmdoOZ1ZpZbX194m4cMqE0n+qxxTz26u6EvaeIyEiSsKA3swVEgv5zMc3z3H0OcDFwk5md29v67r7U3WvcvaaioiJRZQGwePZ4XnizkU17DiX0fUVERoKEBL2ZvR34IbDY3Y+cy+juO6I/9wC/AuYm4vMG6rKaSeQEsvj5c28l4+NFRJLquIPezCYDvwQ+4u4bYtoLzay46zlwIdDjmTtDrbwol/e8bRwPrK6jOdSRjBJERJImntMrlwPPAtVmVmdm15vZjWZ2Y7TLvwJjgO92O41yLPCMma0BXgB+6+6/H4J/Q1w+cs4UDrV18OCLO5JVgohIUmT318Hdl/Sz/GPAx3po3wzMOnaN5JgzuYwZJ5Tw02e3sGTuJMws2SWJiAyLtL4yNpaZ8ZFzpvD6rkP89S3deUpEMkfGBD3A4tPHU5ybzc+e3ZrsUkREhk1GBX1BTjYfOmMij6zdxd7DbckuR0RkWGRU0ANcdfYUQuFO7lu1LdmliIgMi4wL+pMri5h38hjuef4twp2a6ExE0l/GBT3AR86ewvb9LZq+WEQyQkYG/cJTxzK2JJefPacvZUUk/WVk0GcHsrhi7hT+uKGerQ2avlhE0ltGBj3AkrmTyM4yfq69ehFJcxkb9JUleVx02jhW1Nbp5uEiktYyNugh8qXsgZZ2fr1G89+ISPrK6KA/a+poplUW6fCNiKS1jA76rvlv1tQdYM22/ckuR0RkSGR00AN8YPYECnMCOtVSRNJWxgd9cV6QD8yZwK/X7GBfUyjZ5YiIJFzGBz1E5r9p6+jk/tV1yS5FRCThFPTA9HElzK0azc+f30qn5r8RkTQTV9Cb2TIz22NmPd7z1SK+aWabzOxlM5sTs+waM9sYfVyTqMIT7apzprC1oZmVG+uTXYqISELFu0d/F7Coj+UXA9OijxuA7wGY2WjgNuAsYC5wm5mVDbbYobRo5jjKi3J1qqWIpJ24gt7dVwKNfXRZDPzUI54DSs3sBOAi4HF3b3T3fcDj9L3BSJqc7CyWzJ3EE6/vYVtjc7LLERFJmEQdo58AxN7Joy7a1lt7SloydzIGLH/hrWSXIiKSMIkKeuuhzftoP/YNzG4ws1ozq62vT85x8vGl+Sw8dSz3rdpGW4fmvxGR9JCooK8DJsW8ngjs6KP9GO6+1N1r3L2moqIiQWUN3EfOmUJDU4jfrd2VtBpERBIpUUH/MHB19Oybs4ED7r4TeBS40MzKol/CXhhtS1nzTipnanmhrpQVkbQR7+mVy4FngWozqzOz683sRjO7MdrlEWAzsAn4AfAJAHdvBO4AVkUft0fbUlZWlnHV2VNYvXUfr2w/kOxyRESOm7mn3gVCNTU1Xltbm7TPP9DSzju/9CRnnziGH1xdk7Q6RETiZWar3b3HwNKVsT0YlR/khnedyOOv7uYlzWopIiOcgr4X171zKqMLc/jKY+uTXYqIyHFR0PeiKDebT8w/iT9t3MtzmxuSXY6IyKAp6Ptw1dlTGFuSy5cfXU8qfpchIhIPBX0f8oIBbnn3NGq37uPpDZrsTERGJgV9Py6rmcSk0fl85THt1YvIyKSg70dOdhafOv8UXtl+kN+/oqtlRWTkUdDH4dLZEzi5soivPL6BsG5MIiIjjII+DoEs49MXnMKmPYd56KXtyS5HRGRAFPRxWjRzHDPHl/D1P2ykPdyZ7HJEROKmoI9TVpbxmQtP4a3GZlbUbut/BRGRFKGgH4AF1ZXMmVzKt57YRGu75qsXkZFBQT8AZsZnL6pm18FW3VtWREYMBf0AveOkcuadPIbvPf0GTW0dyS5HRKRfCvpB+OyF1TQ0hfjxn99MdikiIv1S0A/C7MllLDy1ku+v3Mz+5lCyyxER6ZOCfpA+e1E1TW0d/M+jmsZYRFJbvLcSXGRm681sk5nd2sPyr5nZS9HHBjPbH7MsHLPs4UQWn0zTx5Vw7Tumcs8Lb+nmJCKS0voNejMLAN8BLgZmAEvMbEZsH3f/R3c/3d1PB74F/DJmcUvXMnd/fwJrT7pPX3gKlcW5/POv1tKhi6hEJEXFs0c/F9jk7pvdPQTcCyzuo/8SYHkiikt1RbnZ/Ot7Z7Jux0F+ptMtRSRFxRP0E4DYS0Hrom3HMLMpwFTgyZjmPDOrNbPnzOzSQVeaot7ztnGce0oFX3lsA7sPtia7HBGRY8QT9NZDW29TOF4O3O/usZeNTo7emfwK4OtmdlKPH2J2Q3SDUFtfP3Ju8mFm3P7+mYTCndzxm1eTXY6IyDHiCfo6YFLM64nAjl76Xk63wzbuviP6czPwNDC7pxXdfam717h7TUVFRRxlpY6q8kJumn8yv3l5J3/aOHI2UiKSGeIJ+lXANDObamY5RML8mLNnzKwaKAOejWkrM7Pc6PNyYB6Qlru9N84/kanlhfzLg69oHhwRSSn9Br27dwA3A48CrwEr3H2dmd1uZrFn0SwB7vWj77d3KlBrZmuAp4AvuntaBn1udoA7Fp/GloZm7vzjG8kuR0TkCEvF+6DW1NR4bW1tsssYlFuWv8ij63bx2KfOpaq8MNnliEiGMLPV0e9Dj6ErYxPsXy45ldxAFv/y0Cu6mbiIpAQFfYJVluTx2Yuq+dPGvfxidV2yyxERUdAPhY+cPYVzThzDvz+8jrcampNdjohkOAX9EMjKMr582SyysoxPr3iJcKcO4YhI8ijoh8iE0nzuWHwatVv36SwcEUkqBf0QWnz6eN779hP42uMbWFt3INnliEiGUtAPITPj/136NsqLcvnUfS/SEtKFVCIy/BT0Q2xUQZCvXDaLN+qb+OLvXkt2OSKSgRT0w2DeyeV8dN5UfvLsVp5evyfZ5YhIhlHQD5N/WlTNKWOL+Owv1rDrgKYzFpHho6AfJnnBAN+9cg4toTCfuHs1oQ7dkUpEhoeCfhidXFnMf394Fn99az//+YiO14vI8FDQD7NL3n4C179zKnf9ZQsPvbQ92eWISAZQ0CfBrRdPZ27VaG59YC3rdx1KdjkikuYU9EkQDGTx7StmU5SXzY0/X83B1vZklyQiaUxBnySVJXl898o5bGts5n8vf5GOsL6cFZGhoaBPojOrRvPvi2fy1Pp63VhcRIZMdrILyHRXnjWFLXub+MGf3qSqvJDr5k1Ndkkikmbi2qM3s0Vmtt7MNpnZrT0sv9bM6s3spejjYzHLrjGzjdHHNYksPl3cevGpXDhjLHf85lWeeG13sssRkTTTb9CbWQD4DnAxMANYYmYzeuh6n7ufHn38MLruaOA24CxgLnCbmZUlrPo0Ecgyvn756cwcP4pblr/IK9s106WIJE48e/RzgU3uvtndQ8C9wOI43/8i4HF3b3T3fcDjwKLBlZreCnKy+dE1NZTmB7n2x6vY2tCU7JJEJE3EE/QTgG0xr+uibd19yMxeNrP7zWzSANfFzG4ws1ozq62vr4+jrPRTWZLHT6+fS7izk6t+9Dy7D2pOHBE5fvEEvfXQ1v3eeL8Gqtz97cAfgJ8MYN1Io/tSd69x95qKioo4ykpPJ1cWc9d1c2k8HOLqH73A/uZQsksSkREunqCvAybFvJ4I7Ijt4O4N7t4WffkD4Ix415VjzZpUytKra3hzbxPX3bWKpraOZJckIiNYPEG/CphmZlPNLAe4HHg4toOZnRDz8v1A14xdjwIXmllZ9EvYC6Nt0o95J5fzzSWzWbNtP9fdtYrmkMJeRAan36B39w7gZiIB/Rqwwt3XmdntZvb+aLdPmtk6M1sDfBK4NrpuI3AHkY3FKuD2aJvEYdFp4/j65bOp3dLIdT9W2IvI4Jh7j4fMk6qmpsZra2uTXUbKeOil7fzjfS8xd+poll17JgU5us5NRI5mZqvdvaanZZoCYQRYfPoEvvb3p/PCm40sWfocOw+0JLskERlBFPQjxOLTJ/D9j9Swac9h3vetP1O7RUfARCQ+CvoR5IIZY3nwpnkU5QZY8oPnuOf5t5JdkoiMAAr6EWba2GIeuumdvOOkcv7vr9byz79aq/vPikifFPQj0KiCIMuuPZMbzzuJu59/iyt/+Bz1h9r6X1FEMpKCfoQKZBm3Xjydby2ZzdrtB3jft57h5br9yS5LRFKQgn6Ee9+s8Tzw8XcQyDI+fOez/PKvdckuSURSjII+DcwcP4qHb57HnMmlfHrFGu74zau6NaGIHKGgTxNjinL52fVnce07qvjRM29yzY9fYF+TJkQTEQV9WgkGsvi398/kvz/8dla9uY/3ffsZXtt5MNlliUiSKejT0GU1k7jvH86mPdzJB7/7F1as2qZDOSIZTEGfpmZPLuPXN7+TmeNL+KcHXuaCr63k/tV1CnyRDKSgT2OVJXms+IdzuPOqM8gPBvjsL9bw7q/8kftWvaWLrEQyiGavzBDuzhOv7eGbT27k5boDTCjN5xMLTuLDZ0wkNzuQ7PJE5Dj1NXulgj7DuDtPb6jnG3/YyEvb9nPCqDw+Pv8kLquZRF5QgS8yUino5RjuzjOb9vKNP2ykdus+KotzufG8k7jirMkKfJERSEEvvXJ3nt3cwDef2MhzmxspL8rlH849kSvPnqwbnIiMIMcd9Ga2CPgGEAB+6O5f7Lb808DHgA6gHviou2+NLgsDa6Nd33L399MPBX1yPL+5gW89uYlnNu2lODebC2aO5b1vP4F3nlxBTra+txdJZccV9GYWADYAFwB1RO79usTdX43pswB43t2bzezjwHx3//vossPuXjSQghX0ybV6ayPLX9jGY+t2cbC1g5K8bC6cOY5L3n4C804qV+iLpKC+gj6ev83nApvcfXP0ze4FFgNHgt7dn4rp/xxw1eDLlWQ7Y8pozpgymtAH3safN+3lNy/v5NF1u7h/dR0ledlc1BX6J5cTDCj0RVJdPEE/AdgW87oOOKuP/tcDv4t5nWdmtUQO63zR3R8ccJWSFDnZWSyYXsmC6ZW0dZzGMxv38tu1O/n9K7v4xeo6SguCXDQjEvrnnDRGoS+SouIJeuuhrcfjPWZ2FVADnBfTPNndd5jZicCTZrbW3d/oYd0bgBsAJk+eHEdZMpxyswOcf+pYzj91LG0dYf60YS+PrN3Jb9fu5L7abZQWBLng1LGcc9IYzqwazcSyfMx6+tURkeEWT9DXAZNiXk8EdnTvZGYLgX8GznP3I7c7cvcd0Z+bzexpYDZwTNC7+1JgKUSO0cf/T5DhlpsdYOGMsSycMZbW9jB/2riX3768g0fXRfb0AU4YlceZVaM5s6qMM6eO5pTKYrKyFPwiyRBP0K8CppnZVGA7cDlwRWwHM5sNfB9Y5O57YtrLgGZ3bzOzcmAe8N+JKl6SLy8Y4IIZY7lgxlg6O531uw+xaksjL7zZyPNvNvDwmsg+QUleNjVVozmzajRzp5bxtgml+lJXZJj0G/Tu3mFmNwOPEjm9cpm7rzOz24Fad38Y+B+gCPhF9M/1rtMoTwW+b2adRObV+WLs2TqSXrKyjFNPKOHUE0q4+pwq3J26fS288GZjJPy3NPLk65H9gNzsLGZNKuXMqjJmnDCK6nHFVI0pIFvH+UUSThdMybDae7iN2i2NvPDmPmq3NrJux0HCnZHfwZxAFidVFlE9tohTxhVTPbaYU8YWM6E0X4d9RPqhK2MlZbW2h9m05zDrdx1iw+5DrN99iA27DrHjQOuRPoU5AaaNjQZ/dAMwtaKQE0rytAEQiTre8+hFhkxeMMBpE0Zx2oRRR7UfbG1n4+5DrN91mA27IxuBJ17fzX21fzvTNyc7i0ll+VSNKWTymIKjfk4ozdd3ACJRCnpJSSV5wSMXbsXae7iNDbsPsWVvM1sbm9i6t5mtjc08u7mB5lD4SL8sgwll+UwZXciUMQVMGVPAxLICxpfmM35UHuVFufprQDKGgl5GlPKiXMqLcnnHSUe3uzt7D4fY2tDEloZm3or+3NrYzG/X7mR/c/tR/YMBY9yoPE4YFQn+8aX5nFD6t+fjR+VTkp+tawEkLSjoJS2YGRXFuVQU51JTNfqY5Qea29m+v4Ud+1vYeaCFHQdaI8/3t7Jqyz52H9xJR+fR31cV5AQYNyqPyuJcKoq7fuZSWZxLZXHekeelBUFtECSlKeglI4wqCDKqIMiM8SU9Lg93OnsPt7E9Gv47D7SwfX8Lew62sedQKy/X7WfPwTZa2sPHrBsMGBVFuVSU5EV+FucypjCH0YU5jCmK/BxdmMOYwlzKCoO6o5cMOwW9CBDIMsaW5DG2JA/6mIHjcFsH9Yfa2HOwlfrDbew52Hbk555DrdTta+albftobArR2csJbcW52Ywu6gr/HMoKchhdFPlZmh+ktCDIqPwcSgsiz0vzc8gLZumvBhk0Bb3IABTlZlOUm83U8sI++3V2Ogda2mloCtHYFKKxqS3y/HAopi3E9v2trN1+gH1N7YTCvd+wPSc7i7Jo6I8qCFKaH2RU9FGcF6QkP5uSvCAl+UGK87qeZ1OSH6QoJ1tfPGc4Bb3IEMjKMsoKcygrzImrv7vT0h5mf3N75NES4rxyTnUAAAlwSURBVEBzO/tbur1ubmdfc4i3GpvZ39zOwdb2o8426olZZANVkve3jUBRXmSDdeRn7t9eF8e0F+dlU5QbpDA3QEFONgFtMEYkBb1ICjAzCnKyKcjJZnxp/oDWbQ93cri1g4Ot7Rxs6eBQa/uR55Gf7RyMWX64rZ09h1rZXN/B4bbIo7W9978mYuUFsyjMyaYwN5uCnACFuZHnhTmRDUFRboCCmNcFOQHyj3keoCCYfeR5fjCgvziGmIJeZIQLBrIG9NdDT9rDnTS1dXCoNRL8TW0dHGrr4HBrpK051EFTW5jmUGR5cyhMU1sHTaEODra0s+tAC01tYZpCHTS3hfs8DNWTvGAWBTnZ5AcjG4P8YOSRG8w6qi2v2/K8YBZ50fa86Ovc7Jj26PPc6LKcQGZ+16GgFxGCgSxKC3IoLRj8xiJWqKOTllCY5vbIRqElFKY5FNlQHHneHqYl1H15mNb2yKOlPXzky++u1y2hMK3tnQPekHQxi0yo17URyA1mkZv9t41DbnYg8jr2eXZkQ9H1PCfaPyc7suHIjW5AYttj+3b1i30+3BsbBb2IJFxXqI0iOCTvH+70o8K/rSOyAYhsJKI/j2oL09bxt+et7Z20dUTa2mKet7SH2d8SirZ1Hlmva3mipgYLBuzo8I9uAIKBv20ggoGYn4EsggEjGMgiGLPhONLWz6yvCnoRGXECWXbk+4Hh4u60h51QuJO29sjhqVBHZIPQ9bOtI0yo4+j2rn5dz9s6OmmPbYvt0629qa2DUNgJdYRpDzvt4ci6Xe/RHvYjs7/2RUEvIhIHMyMn28jJzqJoGDcw/Ql3RjYA+V/qvU/qVCsiIgMWyDICWX1fba15XEVE0pyCXkQkzcUV9Ga2yMzWm9kmM7u1h+W5ZnZfdPnzZlYVs+zz0fb1ZnZR4koXEZF49Bv0ZhYAvgNcDMwAlpjZjG7drgf2ufvJwNeAL0XXnQFcDswEFgHfjb6fiIgMk3j26OcCm9x9s7uHgHuBxd36LAZ+En1+P3C+Ra4IWAzc6+5t7v4msCn6fiIiMkziCfoJwLaY13XRth77uHsHcAAYE+e6AJjZDWZWa2a19fX18VUvIiL9iifoe7pWt/sZ+r31iWfdSKP7UnevcfeaioqKOMoSEZF4xHMefR0wKeb1RGBHL33qzCwbGAU0xrnuMVavXn3YzNbHUdtAjCLyl0Yi1+lreW/Lemrv3tbX63Jgbx81DcZQjE1/feIZh57ahntseqvrePsn4ndHY9N3e1/j0X3ZSPl/1dfyab2u5e59PohsDDYDU4EcYA0ws1ufm4A7o88vB1ZEn8+M9s+Nrr8ZCMTxmbX99RnoA1ia6HX6Wt7bsp7au7f19XqkjM1gxicVx2Yw4zMUYzPQscj0sYljPLovGxH/rwYzNu7e/x69u3eY2c3Ao0AAWObu68zs9ujgPAz8CPiZmW0isid/eXTddWa2AngV6ABucve+75IwdH49BOv0tby3ZT21d2/r73WiDcXY9NcnnnHoqW24x2YwnzEUY9NTu8am7/a+xiMVxyaedQYzNlh0S5BSzKzW3WuSXUcq0tj0TmPTO41N39J9fFL1ytilyS4ghWlseqex6Z3Gpm9pPT4puUcvIiKJk6p79CIikiAKehGRNKegFxFJcyMq6M3sRDP7kZndn+xaUoGZFZrZT8zsB2Z2ZbLrSTX6femdmV0a/b15yMwuTHY9qcTMTjWzO83sfjP7eLLrSYRhC3ozW2Zme8zslW7tfU6BHMsjE6tdP7SVJtcAx+mDwP3u/r+A9w97sUkwkPHJhN+XWAMcmwejvzfXAn+fhHKH1QDH5jV3vxG4DEiLUy6Hc4/+LiJTFR/R2xTIZvY2M/tNt0flMNaaTHcR5zgRmVKia9K4ZF2INtzuIv7xyTR3MfCx+UJ0ebq7iwGMjZm9H3gGeGJ4yxwawxb07r6SyFWzsXqcAtnd17r7e7s99gxXrck0kHEiMpfQxGifEXUYbrAGOD4ZZSBjYxFfAn7n7n8d7lqH20B/b9z9YXd/B5AWh0STHQ5xT2MMYGZjzOxOYLaZfX6oi0shvY3TL4EPmdn3GJ5LulNVj+OTwb8vsXr73bkFWAh82MxuTEZhKaC335v5ZvZNM/s+8EhySkuseGavHEpxT2MM4O4NQCb+UvY4Tu7eBFw33MWkoN7GJ1N/X2L1NjbfBL453MWkmN7G5mng6eEtZWgle49+UNMYZyCNU980Pr3T2PQuY8Ym2UG/CphmZlPNLIfIrJcPJ7mmVKRx6pvGp3cam95lzNgM5+mVy4FngWozqzOz6z1y28GuKZBfIzKP/brhqikVaZz6pvHpncamd5k+NprUTEQkzSX70I2IiAwxBb2ISJpT0IuIpDkFvYhImlPQi4ikOQW9iEiaU9CLiKQ5Bb2ISJpT0EtGMbMPmdnzZrbGzGrN7CIzqzKzFjN7KabfWDO7x8w2m9lqM3vWzD7Qz3s/bWYXdWv7lJl918zyzewlMwuZWflQ/ftEeqKgl4xhZlcAnyVyz4NZwBLgJ0Qms3rD3U+P9jPgQWClu5/o7mcQmQdlYs/vfMTyaL9YlwPL3b0l+v5pOWmWpDYFvWQEMysEvghc5u67ANx9I5HpaM/v1v3dQMjd7+xqcPet7v6tmPe7ysxeiO6lfz96t6L7gfeaWW60TxUwnsidikSSRkEvmeJy4K/uvq1bexvQ/e5lM4Fe77pkZqcSuc/qvOheehi4Mjr//Qv87ZZ1lwP3uSaUkiRT0EumOA1Y00P7LKC1rxXN7DvRY/qrok3nA2cAq6LH9c8HTowuiz18c3n0tUhSKeglUxwEcmIbzOwcoBD4Y7e+64A5XS/c/SYiYV7RtSrwE3c/Pfqodvd/iy57EDjfzOYA+ZlwP1ZJfQp6yRS/BS4zswoAMzsF+CHwUaCzW98ngTwz+3hMW0HM8yeI3Gu1Mvpeo81sCoC7HyZy3H8Z2puXFJHse8aKDAt3f8HM/gP4Q/TL0gBwtbs/G/3SNLavm9mlwNfM7J+AeqAJ+Fx0+atm9gXgMTPLAtqBm4Ct0bdYTuTG7d3PwBFJCt14RDKOmY0lste+xN1fjgb9b9z9tGH47C1AjbvvHerPEumiQzeScdx9t7vPdPeXo01hYFTsBVOJ1nXBFBDk2ENFIkNKe/QiImlOe/QiImlOQS8ikuYU9CIiaU5BLyKS5hT0IiJpTkEvIpLmFPQiImlOQS8ikub+P6EBDyFNFhA6AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], + "source": [ + "alphas_df = pd.read_csv('alphas.csv')\n", + "alphas_df = alphas_df.set_index('$Q$[GeV]')\n", + "alphas = interpolate.interp1d(alphas_df.index, alphas_df[r'$\\alpha_s(Q)$'])\n", + "\n", + "alphas_df.plot(logx=True)" + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array(0.95798)" + ] + }, + "execution_count": 62, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "alphas(0.3)" + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array(0.4405)" + ] + }, + "execution_count": 66, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "alphas(1.0)" + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array(0.30426603)" + ] + }, + "execution_count": 67, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "alphas(2.0)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## PDFs" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "def plot(url, width='100%', height=500):\n", + " \"\"\"Return an IFrame plot\"\"\"\n", + " from IPython.display import IFrame\n", + " return IFrame(url, width=width, height=height) " + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + " \n", + " " + ], + "text/plain": [ + "" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "link_pdf = \"http://hepdata.cedar.ac.uk/pdf/pdf3.html\"\n", + "plot(link_pdf, height=900)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.3" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/.github/entrypoint/artifact/python/pyproject.toml b/.github/entrypoint/artifact/python/pyproject.toml new file mode 100644 index 000000000000..e4a8cb2ba94b --- /dev/null +++ b/.github/entrypoint/artifact/python/pyproject.toml @@ -0,0 +1,29 @@ +[tool.poetry] +name = "lgt" +version = "0.1.3" +description = "Python library for Lattice Gauge Theory" +authors = ["Sam Foreman "] + +[tool.poetry.dependencies] +python = ">=3.8,<3.11" +tensorflow = "^2.8.0" +numpy = "^1.22.3" +torch = "^1.11.0" +scipy = "^1.8.0" +itermplot = "^0.331" +ipython = "^8.2.0" + +[tool.poetry.dev-dependencies] +pyright = "^1.1.232" +pynvim = "^0.4.3" +ptipython = "^1.0.1" +# ipdb = {version = "*", optional = true} +pudb = {version = "*", optional = true} +# ptipython = {optional = true} + +[tool.poetry.extras] +dev_tools = ["pudb"] + +[build-system] +requires = ["poetry-core>=1.0.0"] +build-backend = "poetry.core.masonry.api" diff --git a/.github/entrypoint/artifact/python/setup.py b/.github/entrypoint/artifact/python/setup.py new file mode 100644 index 000000000000..4e5b539cd7eb --- /dev/null +++ b/.github/entrypoint/artifact/python/setup.py @@ -0,0 +1,29 @@ +import setuptools + +with open("README.md", "r", encoding="utf-8") as fh: + long_description = fh.read() + +setuptools.setup( + name="nftqcd/lgt", + version="0.1.2", + author="Sam Foreman", + author_email="saforem2@gmail.com", + description=( + "A package containing a collection of" + " utilities for Lattice Gauge Theory" + ), + long_description=long_description, + long_description_content_type="text/markdown", + url="https://github.com/nftqcd/lgt", + project_urls={ + "Bug Tracker": "https://github.com/nftqcd/lgt/issues", + }, + classifiers=[ + "Programming Language :: Python :: 3", + "License :: OSI Approved :: MIT License", + "Operating System :: OS Independent", + ], + package_dir={"": "src"}, + packages=setuptools.find_packages(where="src"), + python_requires=">=3.8", +) diff --git a/.github/entrypoint/artifact/python/src/lgt/__init__.py b/.github/entrypoint/artifact/python/src/lgt/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/group/__init__.py b/.github/entrypoint/artifact/python/src/lgt/group/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/group/generators.py b/.github/entrypoint/artifact/python/src/lgt/group/generators.py new file mode 100644 index 000000000000..0d520bc4e4c1 --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/group/generators.py @@ -0,0 +1,55 @@ +""" +generators.py + +Contains methods for generating elements of SU(2), SU(3) gauge groups. + +Note: + - The `eps` (type: float) argument to the below functions controls the + 'distance' from the identity matrix. +""" +from __future__ import absolute_import, print_function, division, annotations + +import numpy as np + +# from re import split +# from multiprocessing import pool + + +def generate_SU2(eps: float) -> np.ndarray: + """Returns a single randomly initialized SU(2) matrix.""" + r_rand_nums = np.random.uniform(0, 0.5, (4)) + r = np.empty((4)) + + r[1:] = eps * r_rand_nums[1:] / np.linalg.norm(r_rand_nums[1:]) + r[0] = np.sign(r_rand_nums[0]) * np.sqrt(1 - eps ** 2) + + r11 = +r[0] + 1j * r[3] + r12 = +r[2] + 1j * r[1] + r21 = -r[2] + 1j * r[1] + r22 = +r[0] - 1j * r[3] + + return np.array([[r11, r12], [r21, r22]]) + + +def generate_SU3(eps: float) -> np.ndarray: + """Returns a single randomly initialized SU(3) mtx.""" + r = np.identity(3, dtype=np.complexfloating) + s = np.identity(3, dtype=np.complexfloating) + t = np.identity(3, dtype=np.complexfloating) + + r[:2, :2] = generate_SU2(eps) + s[0:3:2, 0:3:2] = generate_SU2(eps) + t[1:, 1:] = generate_SU2(eps) + + return np.dot(np.dot(r, s), t) + + +def generate_SU3_array(n: int, eps: float) -> np.ndarray: + """Generates a 2*n array of SU(3) mtxs; eps controls dist from Identity""" + arr = np.zeros((2 * n, 3, 3), dtype=np.complexfloating) + for i in range(n): + mtx = generate_SU3(eps) + arr[2 * i] = mtx + arr[2 * i + 1] = mtx.conj().T + + return arr diff --git a/.github/entrypoint/artifact/python/src/lgt/group/group.py b/.github/entrypoint/artifact/python/src/lgt/group/group.py new file mode 100644 index 000000000000..640499b072c2 --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/group/group.py @@ -0,0 +1,403 @@ +""" +group.py + +Contains implementations of various (special) unitary groups. + +Date: 2022/03/25 + +Original version from: + + https://github.com/nftqcd/nthmc/blob/master/lib/group.py + +written by Xiao-Yong Jin +""" +from __future__ import absolute_import, division, print_function, annotations + + +import numpy as np +import tensorflow as tf +import math +from typing import Callable + +Array = np.array +Tensor = tf.Tensor +PI = tf.convert_to_tensor(math.pi) +SQRT1by3 = tf.math.sqrt(1. / 3.) + +TF_FLOAT = tf.keras.backend.floatx() + +# def adj(x: Array): +# return x.conj().T + + +class Group: + """Gauge group represented as matrices in the last two dims in tensors.""" + def mul( + self, + a: Tensor, + b: Tensor, + adjoint_a: bool = False, + adjoint_b: bool = False + ) -> Tensor: + return tf.linalg.matmul(a, b, adjoint_a=adjoint_a, adjoint_b=adjoint_b) + + +class U1Phase(Group): + def mul( + self, + a: Tensor, + b: Tensor, + adjoint_a: bool = False, + adjoint_b: bool = False + ) -> Tensor: + if adjoint_a and adjoint_b: + return tf.subtract(tf.math.negative(a), b) + elif adjoint_a: + return tf.add(tf.math.negative(a), b) + elif adjoint_b: + return tf.subtract(a, b) + else: + return tf.add(a, b) + + def adjoint(self, x: Tensor) -> Tensor: + return tf.math.negative(x) + + def trace(self, x: Tensor) -> Tensor: + return tf.math.cos(x) + + def diff_trace(self, x: Tensor) -> Tensor: + return tf.math.negative(tf.math.sin(x)) + + def diff2trace(self, x: Tensor) -> Tensor: + return tf.math.negative(tf.math.cos(x)) + + def compat_proj(self, x: Tensor) -> Tensor: + return tf.math.floormod(x + PI, 2 * PI) - PI + + def random(self, shape: list[int]): + return self.compat_proj(tf.random.uniform(shape, *(-4, 4))) + + def random_momentum(self, shape: list[int]) -> Tensor: + return tf.random.normal(shape) + + def kinetic_energy(self, p: Tensor) -> Tensor: + return tf.reduce_sum( + tf.square(tf.reshape(p, [p.shape[0], -1])), + axis=1 + ) + + +class SU3(Group): + dtype = tf.complex128 + size = [3, 3] + shape = (3, 3) + + def mul( + self, + a: Tensor, + b: Tensor, + adjoint_a: bool = False, + adjoint_b: bool = False, + ) -> Tensor: + return tf.linalg.matmul(a, b, adjoint_a=adjoint_a, adjoint_b=adjoint_b) + + def adjoint(self, x: Tensor) -> Tensor: + return tf.linalg.adjoint(x) + + def trace(self, x: Tensor) -> Tensor: + return tf.linalg.trace(x) + + def diff_trace(self, x: Tensor): # type: ignore + print('TODO') + + def diff2Trace(self, x: Tensor): # -> Tensor: + print('TODO') + + def exp(self, x: Tensor) -> Tensor: + return exp(x) + + def projectTAH(self, x: Tensor) -> Tensor: + return projectTAH(x) + + def random(self, shape: tuple) -> Tensor: + r = tf.random.normal(shape, dtype=TF_FLOAT) + i = tf.random.normal(shape, dtype=TF_FLOAT) + return projectSU(tf.dtypes.complex(r, i)) + + def random_momentum(self, shape: tuple) -> Tensor: + return randTAH3(shape[:-2]) + + def kinetic_energy(self, p: Tensor) -> Tensor: + p2 = norm2(p) - tf.constant(8.0) # - 8.0 ?? + return ( + 0.5 * tf.math.reduce_sum(tf.reshape(p2, [p.shape[0], -1]), axis=1) + ) + + +def norm2(x: Tensor, axis=[-2, -1]) -> Tensor: + """No reduction if axis is empty""" + n = tf.math.real(tf.math.multiply(tf.math.conj(x), x)) + if len(axis) == 0: + return n + + return tf.math.reduce_sum(n, axis=axis) + + +# Converted from qex/src/maths/matrixFunctions.nim +# Last two dims in a tensor contain matrices. +# WARNING: below only works for SU3 for now +def randTAH3(shape, s): + s2 = 0.70710678118654752440 # sqrt(1/2) + s3 = 0.577350269189625750 # sqrt(1/3) + r3 = s2 * s.normal(shape, dtype=TF_FLOAT) + r8 = s2 * s3 * s.normal(shape, dtype=TF_FLOAT) + # m00 = tf.dtypes.complex(tf.cast(0.0,TF_FLOAT), r8+r3) + # m11 = tf.dtypes.complex(tf.cast(0.0,TF_FLOAT), r8-r3) + # m22 = tf.dtypes.complex(tf.cast(0.0,TF_FLOAT), -2*r8) + m00 = tf.dtypes.complex(0.0, r8+r3) + m11 = tf.dtypes.complex(0.0, r8-r3) + m22 = tf.dtypes.complex(0.0, -2*r8) + r01 = s2 * s.normal(shape, dtype=TF_FLOAT) + r02 = s2 * s.normal(shape, dtype=TF_FLOAT) + r12 = s2 * s.normal(shape, dtype=TF_FLOAT) + i01 = s2 * s.normal(shape, dtype=TF_FLOAT) + i02 = s2 * s.normal(shape, dtype=TF_FLOAT) + i12 = s2 * s.normal(shape, dtype=TF_FLOAT) + m01 = tf.dtypes.complex(r01, i01) + m10 = tf.dtypes.complex(-r01, i01) + m02 = tf.dtypes.complex(r02, i02) + m20 = tf.dtypes.complex(-r02, i02) + m12 = tf.dtypes.complex(r12, i12) + m21 = tf.dtypes.complex(-r12, i12) + return tf.stack([ + tf.stack([m00, m10, m20], axis=-1), + tf.stack([m01, m11, m21], axis=-1), + tf.stack([m02, m12, m22], axis=-1), + ], axis=-1) + + +def eigs3(tr, p2, det): + tr3 = (1.0/3.0)*tr + p23 = (1.0/3.0)*p2 + tr32 = tr3*tr3 + q = tf.math.abs(0.5*(p23-tr32)) + r = 0.25*tr3*(5*tr32-p2) - 0.5*det + sq = tf.math.sqrt(q) + sq3 = q*sq + isq3 = 1.0/sq3 + maxv = tf.constant(3e38, shape=isq3.shape, dtype=isq3.dtype) + minv = tf.constant(-3e38, shape=isq3.shape, dtype=isq3.dtype) + isq3c = tf.math.minimum(maxv, tf.math.maximum(minv, isq3)) + rsq3c = r * isq3c + maxv = tf.constant(1, shape=isq3.shape, dtype=isq3.dtype) + minv = tf.constant(-1, shape=isq3.shape, dtype=isq3.dtype) + rsq3 = tf.math.minimum(maxv, tf.math.maximum(minv, rsq3c)) + t = (1.0/3.0)*tf.math.acos(rsq3) + st = tf.math.sin(t) + ct = tf.math.cos(t) + sqc = sq*ct + sqs = 1.73205080756887729352*sq*st # sqrt(3) + ll = tr3 + sqc + e0 = tr3 - 2*sqc + e1 = ll + sqs + e2 = ll - sqs + return e0, e1, e2 + + +def rsqrtPHM3f(tr, p2, det): + l0, l1, l2 = eigs3(tr, p2, det) + sl0 = tf.math.sqrt(tf.math.abs(l0)) + sl1 = tf.math.sqrt(tf.math.abs(l1)) + sl2 = tf.math.sqrt(tf.math.abs(l2)) + u = sl0 + sl1 + sl2 + w = sl0 * sl1 * sl2 + d = w*(sl0+sl1)*(sl0+sl2)*(sl1+sl2) + di = 1.0/d + c0 = (w*u*u+l0*sl0*(l1+l2)+l1*sl1*(l0+l2)+l2*sl2*(l0+l1))*di + c1 = -(tr*u+w)*di + c2 = u*di + return c0, c1, c2 + + +def rsqrtPHM3(x: Tensor) -> Tensor: + tr = tf.math.real(tf.linalg.trace(x)) + x2 = tf.linalg.matmul(x, x) + p2 = tf.math.real(tf.linalg.trace(x2)) + det = tf.math.real(tf.linalg.det(x)) + c0, c1, c2 = rsqrtPHM3f(tr, p2, det) + c0_ = tf.reshape(c0, c0.shape + [1, 1]) + c1_ = tf.reshape(c1, c1.shape + [1, 1]) + c2_ = tf.reshape(c2, c2.shape + [1, 1]) + term0 = tf.cast(c0_ * tf.eye(3, batch_shape=[1] * len(c0.shape)), x.dtype) + term1 = tf.multiply(x, tf.cast(c1_, x.dtype)) + term2 = tf.multiply(x, tf.cast(c2_, x.dtype)) + return term0 + term1 + term2 + + +def projectU(x: Tensor) -> Tensor: + """x (x'x)^{-1/2}""" + # nc = x.shape[-1] + t = tf.linalg.matmul(x, x, adjoint_a=True) + t2 = rsqrtPHM3(t) + return tf.linalg.matmul(x, t2) + + +def projectSU(x: Tensor) -> Tensor: + nc = tf.constant(x.shape[-1], TF_FLOAT) + m = projectU(x) + d = tf.linalg.det(m) + tmp = tf.math.atan2(tf.math.imag(d), tf.math.real(d)) + p = tmp * tf.constant(-1.0) / nc + # p = -(1.0 / nc) * tf.math.atan2(tf.math.imag(d), tf.math.real(d)) + # p = tf.math.multiply(tf.math.negative(tf.constant(1.0) / nc), + # tf.math.atan2(tf.math.imag(d), tf.math.real(d))) + p_ = tf.reshape(tf.dtypes.complex(tf.math.cos(p), tf.math.sin(p)), + p.shape + [1, 1]) + return p_ * m + + +def projectTAH(x: Tensor) -> Tensor: + """Returns R = 1/2 (X - X†) - 1/(2 N) tr(X - X†) + R = - T^a tr[T^a (X - X†)] + = T^a ∂_a (- tr[X + X†]) + """ + nc = tf.constant(x.shape[-1]) + r = 0.5 * (x - tf.linalg.adjoint(x)) + d = tf.linalg.trace(r) / nc + r -= tf.reshape(d, d.shape + [1, 1]) * eyeOf(x) + + return r + +def checkU(x: Tensor) -> tuple[Tensor, Tensor]: + """Returns the average and maximum of the sum of the deviations of X†X""" + nc = tf.constant(x.shape[-1]) + d = norm2(tf.linalg.matmul(x, x, adjoint_a=True) - eyeOf(x)) + a = tf.math.reduce_mean(d, axis=range(1, len(d.shape))) + b = tf.math.reduce_max(d, axis=range(1, len(d.shape))) + c = 2 * (nc * nc + 1) + + return tf.math.sqrt(a / c), tf.math.sqrt(b / c) + + +def checkSU(x: Tensor) -> tuple[Tensor, Tensor]: + """Returns the average and maximum of the sumf of deviations of: + - X† X + - det(x) + from unitarity + """ + nc = tf.constant(x.shape[-1]) + d = norm2(tf.linalg.matmul(x, x, adjoint_a=True) - eyeOf(x)) + det = tf.linalg.det(x) + d = tf.math.add(d, norm2(tf.constant(1., dtype=det.dtype) + det, axis=[])) + # d = tf.math.add(d, norm2(tf.constant(1.) + tf.linalg.det(x), axis=[])) + # d += norm2(-1 + tf.linalg.det(x), axis=[]) + a = tf.math.reduce_mean(d, axis=range(1, len(d.shape))) + b = tf.math.reduce_max(d, axis=range(1, len(d.shape))) + c = tf.cast(2 * (nc * nc + 1), TF_FLOAT) + return tf.math.sqrt(a / c), tf.math.sqrt(b / c) + + +def su3vec(x: Tensor) -> Tensor: + """Only for x in 3x3 anti-Hermitian. + + Return 8 real numbers, X^a T^a = X - 1/3 tr(X) + + Convention: tr{T^a T^a} = -1/2 + X^a = - 2 tr[T^a X] + """ + c = -2 + x00 = x[..., 0, 0] # type:ignore + x01 = x[..., 0, 1] # type:ignore + x11 = x[..., 1, 1] # type:ignore + x02 = x[..., 0, 2] # type:ignore + x12 = x[..., 1, 2] # type:ignore + x22 = x[..., 2, 2] # type:ignore + return tf.stack([ + c * tf.math.imag(x01), + c * tf.math.real(x01), + tf.math.imag(x11) - tf.math.imag(x00), + c * tf.math.imag(x02), + c * tf.math.real(x02), + c * tf.math.imag(x12), + c * tf.math.real(x12), + SQRT1by3 * ( + 2 * tf.math.imag(x22) - tf.math.imag(x11) - tf.math.imag(x00) + ), + ], axis=-1) + + +def su3fromvec(v: Tensor) -> Tensor: + """ + X = X^a T^a + tr{X T^b} = X^a tr{T^a T^b} = X^a (-1/2) 𝛅^ab = -1/2 X^b + X^a = -2 X_{ij} T^a_{ji} + """ + s3 = 0.577350269189625751 # sqrt(1/3) + c = -0.5 + zero = tf.zeros(v[..., 0].shape, dtype=v[..., 0].dtype) # type:ignore + x01 = c * tf.dtypes.complex(v[..., 1], v[..., 0]) # type:ignore + x02 = c * tf.dtypes.complex(v[..., 4], v[..., 3]) # type:ignore + x12 = c * tf.dtypes.complex(v[..., 6], v[..., 5]) # type:ignore + x2i = s3 * v[..., 7] # type:ignore + x0i = c * (x2i + v[..., 2]) # type:ignore + x1i = c * (x2i - v[..., 2]) # type:ignore + + def neg_conj(x: Tensor) -> Tensor: + return tf.math.negative(tf.math.conj(x)) + + # ---------------------------------------------------- + # NOTE: Returns matrix of the form: + # + # M = [[ cmplx(x0i), -x01*, -x02*], + # [ x01, cmplx(x1i), -x12*], + # [ x02, x12, cmplx(x2i)]] + # ---------------------------------------------------- + v1 = tf.stack([ + tf.dtypes.complex(zero, x0i), neg_conj(x01), neg_conj(x02) + ], axis=-1) + v2 = tf.stack([ + x01, tf.dtypes.complex(zero, x1i), neg_conj(x12) + ], axis=-1) + v3 = tf.stack([ + x02, x12, tf.dtypes.complex(zero, x2i) + ], axis=-1) + + return tf.stack([v1, v2, v3]) + + +def eyeOf(m): + batch_shape = [1] * (len(m.shape) - 2) + return tf.eye(*m.shape[-2:], batch_shape=batch_shape, dtype=m.dtype) + + +def exp(m: Tensor, order: int = 12): + eye = eyeOf(m) + x = eye + m / tf.constant(order) + for i in tf.range(order-1, 0, -1): + x = eye + tf.linalg.matmul(m, x) / tf.constant(tf.cast(i, m.dtype)) + + return x + + +def SU3GradientTF( + f: Callable[[Tensor], Tensor], + x: Tensor, +) -> tuple[Tensor, Tensor]: + """Compute gradient using TensorFlow GradientTape. + + y = f(x) must be a real scalar value. + + Returns: + - (f(x), D), where D = T^a D^a = T^a ∂_a f(x) + + NOTE: Use real vector derivatives, e.g. + D^a = ∂_a f(x) + = ∂_t f(exp(T^a) x) |_{t=0} + """ + zeros = tf.zeros(8) + with tf.GradientTape(watch_accessed_variables=False) as tape: + tape.watch(zeros) + y = f(tf.linalg.matmul(exp(su3fromvec(zeros)), x)) + d = tape.gradient(y, zeros) + + return y, d diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/lattice.py b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/lattice.py new file mode 100644 index 000000000000..07964bf6e1fa --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/numpy/lattice.py @@ -0,0 +1,178 @@ +""" +su3/numpy/lattice.py + +Implements SU3BaseLattice Class +""" +from __future__ import absolute_import, print_function, division, annotations + +# import jax.numpy as jnp +import numpy as np + +from lgt.group import group as g +# from typing import Generator +# from l2hmc.lattice.generators import generate_SU3 + +Array = np.ndarray +PI = np.pi +TWO_PI = 2. * np.pi + + +def pbc(tup: tuple[int], shape: tuple[int]) -> list: + return np.mod(tup, shape).tolist() + + +def mat_adj(mat: Array) -> Array: + return mat.conj().T + + +Site = tuple[int, int, int, int] # t, x, y, z +Link = tuple[int, int, int, int, int] # t, x, y, z, dim +Buffer = tuple[int, int, int, int, int, int] # b, t, x, y, z, dim + + +class BaseLatticeSU3: + """4D Lattice with SU(3) links.""" + dim = 4 + + def __init__( + self, + nb: int, + shape: tuple[int, int, int, int], + c1: float = 0.0, + ) -> None: + """4D SU(3) Lattice object for dealing with lattice quantities. + + NOTE: + `x.shape = [nb, 4, nt, nx, nx, nx, 3, 3]`. + + Args: + - nb (int): Batch dimension, i.e. number of chains to run in parallel + - nt (int): Temporal extent of lattice + - nx (int): Spatial extent of lattice + - dim (int): Number of dimensions, 4 by default (t, x, y, z) + """ + self.dim = 4 + self.g = g.SU3() + assert len(shape) == 4 # (nb, nt, nx, dim) + self.c1 = c1 + self.link_shape = self.g.shape + self.nt, self.nx, self.ny, self.nz = shape + self._shape = (nb, 4, *shape, *self.g.shape) + self.volume = self.nt * self.nx * self.ny * self.nz + # ---------------------------------------------------------------- + # NOTE: + # - self.link_shape: [*e] = [3, 3] + # - self.site_idxs: [nb, *n] = [nb, t, x, y, z] + # - self.link_idxs: [nb, *n, d] = [nb, t, x, y, z, d] + # - self.shape: [nb, *n, d, *e] = [nb, t, x, y, z, d, 3, 3] + # + # - where: t in [0, 1, ..., (nt - 1)] + # (x, y, z) in [0, 1, ..., (nx - 1)] + # d in [0, 1, ..., (dim - 1)] + # ---------------------------------------------------------------- + self.site_idxs = tuple( + [self.nt] + [self.nx for _ in range(self.dim - 1)] + ) + self.nplaqs = self.nt * self.nx + self._lattice_shape = shape + self.nsites = np.cumprod(shape)[-1] + self.nlinks = self.nsites * self.dim + self.link_idxs = tuple(list(self.site_idxs) + [self.dim]) + + def coeffs(self, beta: Array) -> dict[str, Array]: + """Coefficients for the plaquette and rectangle terms.""" + rect_coeff = beta * self.c1 + plaq_coeff = beta * (1.0 - 8.0 * self.c1) + + return {'plaq': plaq_coeff, 'rect': rect_coeff} + + def _link_staple_op(self, link: Array, staple: Array) -> Array: + return self.g.mul(link, staple) + + def _plaquette(self, x: Array, u: int, v: int): + xuv = self.g.mul(x[:, u], np.roll(x[:, v], shift=-1, axis=u + 1)) + xvu = self.g.mul(x[:, v], np.roll(x[:, u], shift=-1, axis=v + 1)) + return self.g.trace(self.g.mul(xuv, xvu, adjoint_b=True)) + + def _wilson_loops( + self, + x: Array, + needs_rect: bool = False + ) -> tuple[list[Array], list[Array]]: + # x.shape = [nb, nt, nx, nx, nx, d, 3, 3] + # y.shape = [nb, d, nt, nx, nx, nx, 3, 3] + assert len(x.shape) == 8 + plaqs = [] + rects = [] + for u in range(1, 4): + for v in range(0, u): + yuv = self.g.mul(x[:, u], np.roll(x[:, v], shift=-1, axis=u+1)) + yvu = self.g.mul(x[:, v], np.roll(x[:, u], shift=-1, axis=v+1)) + plaq = self.g.trace(self.g.mul(yuv, yvu, adjoint_b=True)) + plaqs.append(plaq) + if needs_rect: + yu = np.roll(x[:, u], shift=-1, axis=v+1) + yv = np.roll(x[:, v], shift=-1, axis=u+1) + uu = self.g.mul(x[:, v], yuv, adjoint_a=True) + ur = self.g.mul(x[:, u], yvu, adjoint_a=True) + ul = self.g.mul(yuv, yu, adjoint_b=True) + ud = self.g.mul(yvu, yv, adjoint_b=True) + ul_ = np.roll(ul, shift=-1, axis=u+1) + ud_ = np.roll(ud, shift=-1, axis=v+1) + rects.append( + self.g.trace(self.g.mul(ur, ul_, adjoint_b=True)) + ) + rects.append( + self.g.trace(self.g.mul(uu, ud_, adjoint_b=True)) + ) + + return plaqs, rects + + def _plaquettes(self, x: Array) -> Array: + ps, _ = self._wilson_loops(x) + psum = 0.0 + for p in ps: # NOTE: len(ps) == 6 + psum += np.sum(p.real, axis=range(1, len(p.shape))) + + # NOTE: return psum / (len(ps) * dim(link) * volume) + return psum / (6 * 3 * self.volume) + + def action(self, x: Array, beta: Array): + """Returns the action""" + coeffs = self.coeffs(beta) + ps, rs = self._wilson_loops(x, needs_rect=self.c1 != 0) + psum = 0.0 + for p in ps: + psum += np.sum( + p.real, + axis=range(1, len(p.shape)) + ) + + action = coeffs['plaq'] * psum + + if self.c1 != 0: + rsum = 0.0 + for r in rs: + rsum += np.sum( + r.real, + axis=range(1, len(r.shape)) + ) + action += coeffs['rect'] * rsum + # action += tf.math.multiply(coeffs['rect'], rsum) + + return action * (-1.0 / 3.0) + + def random(self): + return self.g.random(self._shape) + + # def grad_action(self, x: Array, beta: Array) -> Array: + # """Returns the derivative of the action""" + # if tf.executing_eagerly(): + # with tf.GradientTape(watch_accessed_variables=False) as tape: + # tape.watch(x) + # action = self.action(x, beta) + # g = tape.gradient(action, x) + # else: + # g = tf.gradients(self.action(x, beta), [x])[0] + + # return self.g.projectTAH(self.g.mul(g, x, adjoint_b=True)) diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/su3/pytorch/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/pytorch/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/lattice.py b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/lattice.py new file mode 100644 index 000000000000..324479f6bbaa --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/lattice/su3/tensorflow/lattice.py @@ -0,0 +1,169 @@ +""" +lattice.py + +Contains implementation of generic GaugeLattice object. +""" +from __future__ import absolute_import, print_function, division, annotations + +import numpy as np +import tensorflow as tf + +from lgt.group import group as g +from lgt.lattice.su3.numpy.lattice import BaseLatticeSU3 +# from typing import Generator +# from l2hmc.lattice.generators import generate_SU3 + +Array = np.ndarray +PI = np.pi +TWO_PI = 2. * np.pi + +Tensor = tf.Tensor + + +def pbc(tup: tuple[int], shape: tuple[int]) -> list: + return np.mod(tup, shape).tolist() + + +def mat_adj(mat: Array) -> Array: + return mat.conj().T + + +Site = tuple[int, int, int, int] # t, x, y, z +Link = tuple[int, int, int, int, int] # t, x, y, z, dim +Buffer = tuple[int, int, int, int, int, int] # b, t, x, y, z, dim + + +class LatticeSU3(BaseLatticeSU3): + """4D Lattice with SU(3) links.""" + dim = 4 + + def __init__( + self, + nb: int, + shape: tuple[int, int, int, int], + c1: float = 0.0, + ) -> None: + """4D SU(3) Lattice object for dealing with lattice quantities. + + NOTE: + `x.shape = [nb, 4, nt, nx, ny, nz, 3, 3]`. + + Args: + - nb (int): Batch dimension, i.e. number of chains to run in parallel + - shape: Lattice shape, list or tuple of 4 ints + - c1: Constant indicating whether or not to use rectangle terms ? + """ + self.dim = 4 + self.g = g.SU3() + assert len(shape) == 4 # (nb, nt, nx, dim) + self.c1 = tf.constant(c1) + self.link_shape = self.g.shape + self.nt, self.nx, self.ny, self.nz = shape + self._shape = (nb, 4, *shape, *self.g.shape) + self.volume = self.nt * self.nx * self.ny * self.nz + self.site_idxs = tuple( + [self.nt] + [self.nx for _ in range(self.dim - 1)] + ) + self.nplaqs = self.nt * self.nx + self._lattice_shape = shape + self.nsites = np.cumprod(shape)[-1] + self.nlinks = self.nsites * self.dim + self.link_idxs = tuple(list(self.site_idxs) + [self.dim]) + + def coeffs(self, beta: Tensor) -> dict[str, Tensor]: + """Coefficients for the plaquette and rectangle terms.""" + rect_coeff = beta * self.c1 + plaq_coeff = beta * (tf.constant(1.0) - tf.constant(8.0) * self.c1) + + return {'plaq': plaq_coeff, 'rect': rect_coeff} + + def _link_staple_op(self, link: Tensor, staple: Tensor) -> Tensor: + return self.g.mul(link, staple) + + def _plaquette(self, x: Tensor, u: int, v: int): + xuv = self.g.mul(x[:, u], tf.roll(x[:, v], shift=-1, axis=u + 1)) + xvu = self.g.mul(x[:, v], tf.roll(x[:, u], shift=-1, axis=v + 1)) + return self.g.trace(self.g.mul(xuv, xvu, adjoint_b=True)) + + def _wilson_loops( + self, + x: Tensor, + needs_rect: bool = False + ) -> tuple[list[Tensor], list[Tensor]]: + # x.shape = [nb, nt, nx, nx, nx, d, 3, 3] + # y.shape = [nb, d, nt, nx, nx, nx, 3, 3] + x = tf.reshape(x, self._shape) + assert len(x.shape) == 8 + plaqs = [] + rects = [] + for u in range(1, 4): + for v in range(0, u): + yuv = self.g.mul(x[:, u], tf.roll(x[:, v], shift=-1, axis=u+1)) + yvu = self.g.mul(x[:, v], tf.roll(x[:, u], shift=-1, axis=v+1)) + plaq = self.g.trace(self.g.mul(yuv, yvu, adjoint_b=True)) + plaqs.append(plaq) + if needs_rect: + yu = tf.roll(x[:, u], shift=-1, axis=v+1) + yv = tf.roll(x[:, v], shift=-1, axis=u+1) + uu = self.g.mul(x[:, v], yuv, adjoint_a=True) + ur = self.g.mul(x[:, u], yvu, adjoint_a=True) + ul = self.g.mul(yuv, yu, adjoint_b=True) + ud = self.g.mul(yvu, yv, adjoint_b=True) + ul_ = tf.roll(ul, shift=-1, axis=u+1) + ud_ = tf.roll(ud, shift=-1, axis=v+1) + rects.append( + self.g.trace(self.g.mul(ur, ul_, adjoint_b=True)) + ) + rects.append( + self.g.trace(self.g.mul(uu, ud_, adjoint_b=True)) + ) + + return plaqs, rects + + def _plaquettes(self, x: Tensor) -> Tensor: + ps, _ = self._wilson_loops(x) + psum = tf.constant(0.0) + for p in ps: # NOTE: len(ps) == 6 + psum += tf.reduce_sum(tf.math.real(p), axis=range(1, len(p.shape))) + + # NOTE: return psum / (len(ps) * dim(link) * volume) + return psum / (6 * 3 * self.volume) + + def action(self, x: Tensor, beta: Tensor): + """Returns the action""" + coeffs = self.coeffs(beta) + ps, rs = self._wilson_loops(x, needs_rect=self.c1 != 0) + psum = tf.constant(0.0) + for p in ps: + psum += tf.reduce_sum( + tf.math.real(p), + axis=range(1, len(p.shape)) + ) + + action = tf.math.multiply(coeffs['plaq'], psum) + + if self.c1 != 0: + rsum = tf.constant(0.0) + for r in rs: + rsum += tf.reduce_sum( + tf.math.real(r), + axis=range(1, len(r.shape)) + ) + action += tf.math.multiply(coeffs['rect'], rsum) + + return action * tf.constant(-1.0 / 3.0) + + def random(self): + return self.g.random(self._shape) + + def grad_action(self, x: Tensor, beta: Tensor) -> Tensor: + """Returns the derivative of the action""" + if tf.executing_eagerly(): + with tf.GradientTape(watch_accessed_variables=False) as tape: + tape.watch(x) + action = self.action(x, beta) + g = tape.gradient(action, x) + else: + g = tf.gradients(self.action(x, beta), [x])[0] + + return self.g.projectTAH(self.g.mul(g, x, adjoint_b=True)) diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/lattice.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/lattice.py new file mode 100644 index 000000000000..dfe88e043dbd --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/numpy/lattice.py @@ -0,0 +1,204 @@ +""" +lattice.py + +Implements BaseLatticeU1 class in numpy +""" +from __future__ import absolute_import, division, print_function, annotations +from dataclasses import dataclass +from math import pi as PI +from typing import Optional +from scipy.special import i1, i0 + +import numpy as np + +TWO_PI = 2. * PI +Array = np.ndarray + + +@dataclass +class Charges: + intQ: Array + sinQ: Array + + def asdict(self): + return {'intQ': self.intQ, 'sinQ': self.sinQ} + + +@dataclass +class LatticeMetrics: + plaqs: Array + charges: Charges + p4x4: Array + + def asdict(self): + return { + 'plaqs': self.plaqs, + 'p4x4': self.p4x4, + 'sinQ': self.charges.sinQ, + 'intQ': self.charges.intQ, + } + + +def area_law(beta: float, nplaqs: int): + return (i1(beta) / i0(beta)) ** nplaqs + + +def plaq_exact(beta: float): + return area_law(beta, nplaqs=1) + + +def project_angle(x: Array) -> Array: + return x - TWO_PI * np.floor((x + PI) / TWO_PI) + + +class BaseLatticeU1: + def __init__(self, nb: int, shape: tuple[int, int]): + self.nb = nb + self._dim = 2 + assert len(shape) == 2 + self.nt, self.nx = shape + self.xshape = (self._dim, *shape) + self._shape = (nb, *self.xshape) + + self.nplaqs = self.nt * self.nx + self.nlinks = self.nplaqs * self._dim + + def draw_uniform_batch(self): + unif = np.random.uniform(self.xshape) + return TWO_PI * unif - PI + + def unnormalized_log_prob(self, x: Array) -> Array: + return self.action(x=x) + + def action( + self, + x: Optional[Array] = None, + wloops: Optional[Array] = None + ) -> Array: + """Calculate the Wilson gauge action for a batch of lattices.""" + wloops = self._get_wloops(x) if wloops is None else wloops + return (1. - np.cos(wloops)).sum((1, 2)) + + def calc_metrics(self, x: Array) -> dict[str, Array]: + wloops = self.wilson_loops(x) + plaqs = self.plaqs(wloops=wloops) + charges = self.charges(wloops=wloops) + return {'plaqs': plaqs, 'intQ': charges.intQ, 'sinQ': charges.sinQ} + + def observables(self, x: Array) -> LatticeMetrics: + wloops = self.wilson_loops(x) + return LatticeMetrics(p4x4=self.plaqs4x4(x=x), + plaqs=self.plaqs(wloops=wloops), + charges=self.charges(wloops=wloops)) + + def wilson_loops(self, x: Array) -> Array: + """Calculate the Wilson loops by summing links in CCW direction.""" + # -------------------------- + # NOTE: Watch your shapes! + # -------------------------- + # * First, x.shape = [-1, 2, Lt, Lx], so + # (x_reshaped).T.shape = [2, Lx, Lt, -1] + # and, + # x0.shape = x1.shape = [Lx, Lt, -1] + # where x0 and x1 are the links along the 2 (t, x) dimensions. + # + # * The Wilson loop is then: + # wloop = U0(x, y) + U1(x+1, y) - U0(x, y+1) - U(1)(x, y) + # and so output = wloop.T, with output.shape = [-1, Lt, Lx] + # -------------------------- + x0, x1 = x.reshape(-1, *self.xshape).transpose(1, 2, 3, 0) + return (x0 + np.roll(x1, -1, axis=0) - np.roll(x0, -1, axis=1) - x1).T + + def wilson_loops4x4(self, x: Array) -> Array: + """Calculate the 4x4 Wilson loops""" + x0, x1 = x.reshape(-1, *self.xshape).transpose(1, 0, 2, 3) + return ( + x0 # Ux [x, y] + + x0.roll(-1, dims=2) # Ux [x+1, y] + + x0.roll(-2, dims=2) # Ux [x+2, y] + + x0.roll(-3, dims=2) # Ux [x+3, y] + + x0.roll(-4, dims=2) # Ux [x+4, y] + + x1.roll((-4, -1), dims=(2, 1)) # Uy [x+4, y+1] + + x1.roll((-4, -2), dims=(2, 1)) # Uy [x+4, y+2] + + x1.roll((-4, -3), dims=(2, 1)) # Uy [x+4, y+3] + - x0.roll((-3, -4), dims=(2, 1)) # -Ux [x+3, y+4] + - x0.roll((-2, -4), dims=(2, 1)) # -Ux [x+2, y+4] + - x0.roll((-1, -4), dims=(2, 1)) # -Ux [x+1, y+4] + - x1.roll(-4, dims=1) # -Uy [x, y+4] + - x1.roll(-3, dims=1) # -Uy [x, y+3] + - x1.roll(-2, dims=1) # -Uy [x, y+2] + - x1.roll(-1, dims=1) # -Uy [x, y+1] + - x1 # -Uy [x, y] + ).T + + def plaqs( + self, + x: Optional[Array] = None, + wloops: Optional[Array] = None, + # beta: float = None + ) -> Array: + """Calculate the average plaquettes for a batch of lattices.""" + if wloops is None: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + wloops = self.wilson_loops(x) + + return np.cos(wloops).mean((1, 2)) + + def _plaqs4x4(self, wloops4x4: Array) -> Array: + return np.cos(wloops4x4).mean((1, 2)) + + def plaqs4x4( + self, + x: Optional[Array] = None, + wloops4x4: Optional[Array] = None + ) -> Array: + """Calculate the 4x4 Wilson loops for a batch of lattices.""" + if wloops4x4 is None: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + wloops4x4 = self.wilson_loops4x4(x) + + return self._plaqs4x4(wloops4x4) + + def _sin_charges(self, wloops: Array) -> Array: + """Calculate sinQ from Wilson loops.""" + return np.sin(wloops).sum((1, 2)) / TWO_PI + + def _int_charges(self, wloops: Array) -> Array: + """Calculate intQ from Wilson loops.""" + return project_angle(wloops).sum((1, 2)) / TWO_PI + + def _get_wloops(self, x: Optional[Array] = None) -> Array: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + return self.wilson_loops(x) + + def sin_charges( + self, + x: Optional[Array] = None, + wloops: Optional[Array] = None + ) -> Array: + """Calculate the real-valued charge approximation, sin(Q).""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._sin_charges(wloops) + + def int_charges( + self, + x: Optional[Array] = None, + wloops: Optional[Array] = None + ) -> Array: + """Calculate the integer valued topological charge, int(Q).""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._int_charges(wloops) + + def charges( + self, + x: Optional[Array] = None, + wloops: Optional[Array] = None + ) -> Charges: + """Calculate both charge representations and return as single object""" + wloops = self._get_wloops(x) if wloops is None else wloops + sinQ = self._sin_charges(wloops) + intQ = self._int_charges(wloops) + return Charges(intQ=intQ, sinQ=sinQ) diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/lattice.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/lattice.py new file mode 100644 index 000000000000..bb16532b68e5 --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/pytorch/lattice.py @@ -0,0 +1,258 @@ +""" +lattice.py + +Contains pytorch implementation of Lattice object. + +Author: Sam Foreman +Date: 06/02/2021 +""" +from __future__ import absolute_import, annotations, division, print_function +from dataclasses import dataclass +from typing import Optional +from math import pi as PI + +from scipy.special import i0, i1 +import torch + +from lgt.lattice.u1.numpy.lattice import BaseLatticeU1 + +TWO_PI = 2. * PI +Tensor = torch.Tensor + + +@dataclass +class Charges: + intQ: Tensor + sinQ: Tensor + + +@dataclass +class LatticeMetrics: + plaqs: Tensor + charges: Charges + p4x4: Tensor + + +def area_law(beta: float, nplaqs: int): + return (i1(beta) / i0(beta)) ** nplaqs + + +def plaq_exact(beta: float | Tensor): + return i1(beta) / i0(beta) + + +def project_angle(x: Tensor) -> Tensor: + """For x in [-4pi, 4pi], returns x in [-pi, pi].""" + return x - TWO_PI * torch.floor((x + PI) / TWO_PI) + + +class LatticeU1(BaseLatticeU1): + def __init__(self, nb: int, shape: tuple[int, int]): + super().__init__(nb, shape=shape) + + def draw_uniform_batch(self, requires_grad=True) -> Tensor: + """Draw batch of samples, uniformly from [-pi, pi).""" + unif = torch.rand(self._shape, requires_grad=requires_grad) + return TWO_PI * unif - PI + + def unnormalized_log_prob(self, x: Tensor) -> Tensor: + return self.action(x=x) + + def action( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the Wilson gauge action for a batch of lattices.""" + wloops = self._get_wloops(x) if wloops is None else wloops + return (1. - torch.cos(wloops)).sum((1, 2)) + + def plaqs_diff( + self, + beta: float, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None, + ) -> Tensor: + """Calculate the difference between plaquettes and expected value""" + wloops = self._get_wloops(x) if wloops is None else wloops + plaqs = self.plaqs(wloops=wloops) + pexact = plaq_exact(beta) * torch.ones_like(plaqs) + return pexact - self.plaqs(wloops=wloops) + + def calc_metrics( + self, + x: Tensor, + beta: Optional[float] = None + ) -> dict[str, Tensor]: + """Calculate various metrics and return as dict""" + wloops = self.wilson_loops(x) + plaqs = self.plaqs(wloops=wloops) + charges = self.charges(wloops=wloops) + metrics = {'plaqs': plaqs} + if beta is not None: + metrics.update({ + 'plaqs_err': plaq_exact(torch.from_numpy(beta)) - plaqs + }) + metrics.update({ + 'intQ': charges.intQ, 'sinQ': charges.sinQ + }) + + return metrics + + def observables(self, x: Tensor) -> LatticeMetrics: + """Calculate observables and return as LatticeMetrics object""" + wloops = self.wilson_loops(x) + return LatticeMetrics(p4x4=self.plaqs4x4(x=x), + plaqs=self.plaqs(wloops=wloops), + charges=self.charges(wloops=wloops)) + + def wilson_loops(self, x: Tensor) -> Tensor: + """Calculate the Wilson loops by summing links in CCW direction.""" + # -------------------------- + # NOTE: Watch your shapes! + # -------------------------- + # * First, x.shape = [-1, 2, Lt, Lx], so + # (x_reshaped).T.shape = [2, Lx, Lt, -1] + # and, + # x0.shape = x1.shape = [Lx, Lt, -1] + # where x0 and x1 are the links along the 2 (t, x) dimensions. + # + # * The Wilson loop is then: + # wloop = U0(x, y) + U1(x+1, y) - U0(x, y+1) - U(1)(x, y) + # and so output = wloop.T, with output.shape = [-1, Lt, Lx] + # -------------------------- + x0, x1 = x.reshape(-1, *self.xshape).transpose(0, 1) + x0 = x0.T + x1 = x1.T + + return (x0 + x1.roll(-1, dims=0) - x0.roll(-1, dims=1) - x1).T + + def wilson_loops4x4(self, x: Tensor) -> Tensor: + """Calculate the 4x4 Wilson loops""" + x0, x1 = x.reshape(-1, *self.xshape).transpose(0, 1) + x0 = x0.T + x1 = x1.T + return ( + x0 # Ux [x, y] + + x0.roll(-1, dims=2) # Ux [x+1, y] + + x0.roll(-2, dims=2) # Ux [x+2, y] + + x0.roll(-3, dims=2) # Ux [x+3, y] + + x0.roll(-4, dims=2) # Ux [x+4, y] + + x1.roll((-4, -1), dims=(2, 1)) # Uy [x+4, y+1] + + x1.roll((-4, -2), dims=(2, 1)) # Uy [x+4, y+2] + + x1.roll((-4, -3), dims=(2, 1)) # Uy [x+4, y+3] + - x0.roll((-3, -4), dims=(2, 1)) # -Ux [x+3, y+4] + - x0.roll((-2, -4), dims=(2, 1)) # -Ux [x+2, y+4] + - x0.roll((-1, -4), dims=(2, 1)) # -Ux [x+1, y+4] + - x1.roll(-4, dims=1) # -Uy [x, y+4] + - x1.roll(-3, dims=1) # -Uy [x, y+3] + - x1.roll(-2, dims=1) # -Uy [x, y+2] + - x1.roll(-1, dims=1) # -Uy [x, y+1] + - x1 # -Uy [x, y] + ).T + + def plaqs( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None, + # beta: float = None + ) -> Tensor: + """Calculate the average plaquettes for a batch of lattices.""" + if wloops is None: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + wloops = self.wilson_loops(x) + + return torch.cos(wloops).mean((1, 2)) + + def _plaqs4x4(self, wloops4x4: Tensor) -> Tensor: + return torch.cos(wloops4x4).mean((1, 2)) + + def plaqs4x4( + self, + x: Optional[Tensor] = None, + wloops4x4: Optional[Tensor] = None + ) -> Tensor: + """Calculate the 4x4 Wilson loops for a batch of lattices.""" + if wloops4x4 is None: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + wloops4x4 = self.wilson_loops4x4(x) + + return self._plaqs4x4(wloops4x4) + + def _sin_charges(self, wloops: Tensor) -> Tensor: + """Calculate sinQ from Wilson loops.""" + return torch.sin(wloops).sum((1, 2)) / TWO_PI + + def _int_charges(self, wloops: Tensor) -> Tensor: + """Calculate intQ from Wilson loops.""" + return project_angle(wloops).sum((1, 2)) / TWO_PI + + def _get_wloops( + self, + x: Optional[Tensor] = None + ) -> Tensor: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified.') + return self.wilson_loops(x) + + def sin_charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the real-valued charge approximation, sin(Q).""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._sin_charges(wloops) + + def int_charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the integer valued topological charge, int(Q).""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._int_charges(wloops) + + def charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Charges: + """Calculate both charge representations and return as single object""" + wloops = self._get_wloops(x) if wloops is None else wloops + sinQ = self._sin_charges(wloops) + intQ = self._int_charges(wloops) + return Charges(intQ=intQ, sinQ=sinQ) + + def plaq_loss( + self, + acc: Tensor, + x1: Optional[Tensor] = None, + x2: Optional[Tensor] = None, + wl1: Optional[Tensor] = None, + wl2: Optional[Tensor] = None, + ) -> float: + wloops1 = self._get_wloops(x1) if wl1 is None else wl1 + wloops2 = self._get_wloops(x2) if wl2 is None else wl2 + dwloops = 2. * (1. - torch.cos(wloops2 - wloops1)) + ploss = acc * dwloops.sum((1, 2)) + 1e-4 + + return -ploss.mean(0) + + def charge_loss( + self, + acc: Tensor, + x1: Optional[Tensor] = None, + x2: Optional[Tensor] = None, + wl1: Optional[Tensor] = None, + wl2: Optional[Tensor] = None, + ): + wloops1 = self._get_wloops(x1) if wl1 is None else wl1 + wloops2 = self._get_wloops(x2) if wl2 is None else wl2 + q1 = self._sin_charges(wloops=wloops1) + q2 = self._sin_charges(wloops=wloops2) + dq = (q2 - q1) ** 2 + qloss = acc * dq + 1e-4 + return -qloss.mean(0) diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/__init__.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/.github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/lattice.py b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/lattice.py new file mode 100644 index 000000000000..dbb75a64d9fb --- /dev/null +++ b/.github/entrypoint/artifact/python/src/lgt/lattice/u1/tensorflow/lattice.py @@ -0,0 +1,263 @@ +""" +lattice.py + +TensorFlow implementation of the Lattice object. +""" +from __future__ import absolute_import, annotations, division, print_function +from dataclasses import asdict, dataclass + +import numpy as np +import tensorflow as tf +from typing import Optional + +from lgt.lattice.u1.numpy.lattice import BaseLatticeU1 + +TF_FLOAT = tf.keras.backend.floatx() +PI = tf.constant(np.pi) +TWO_PI = 2. * PI + +Tensor = tf.Tensor + + +@dataclass +class Charges: + intQ: Tensor + sinQ: Tensor + + def asdict(self): + return asdict(self) + + +@dataclass +class LatticeMetrics: + plaqs: Tensor + charges: Charges + p4x4: Tensor + + def asdict(self): + return { + 'plaqs': self.plaqs, + 'sinQ': self.charges.sinQ, + 'intQ': self.charges.intQ, + 'p4x4': self.p4x4, + } + + +def area_law(beta: float, num_plaqs: int) -> float: + """Returns the expected value of the Wilson loop containing `num_plaqs`.""" + return (tf.math.bessel_i1(beta) / tf.math.bessel_i0(beta)) ** num_plaqs + + +def plaq_exact(beta: float | Tensor) -> Tensor: + """Computes the expected value of the avg. plaquette for 2D U(1).""" + beta = tf.constant(beta, dtype=TF_FLOAT) + pexact = tf.constant(tf.math.bessel_i1(beta) / tf.math.bessel_i0(beta)) + return pexact + + +def project_angle(x): + """Returns the projection of an angle `x` from [-4 π, 4 π] to [-π, π].""" + return x - TWO_PI * tf.math.floor((x + PI) / TWO_PI) + + +class LatticeU1(BaseLatticeU1): + def __init__(self, nb: int, shape: tuple[int, int]): + super().__init__(nb, shape=shape) + + def draw_uniform_batch(self) -> Tensor: + """Draw batch of samples, uniformly from [-pi, pi).""" + return tf.random.uniform(self._shape, *(-PI, PI), dtype=TF_FLOAT) + + def unnormalized_log_prob(self, x: Tensor) -> Tensor: + return self.action(x) + + def action( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the Wilson gauge action for a batch of lattices.""" + wloops = self._get_wloops(x) if wloops is None else wloops + local_action = tf.ones_like(wloops) - tf.math.cos(wloops) + return tf.reduce_sum(local_action, (1, 2)) + + def calc_metrics( + self, + x: Tensor, + beta: Optional[Tensor] = None, + ) -> dict[str, Tensor]: + wloops = self.wilson_loops(x) + plaqs = self.plaqs(wloops=wloops) + charges = self.charges(wloops=wloops) + metrics = {'plaqs': plaqs} + if beta is not None: + pexact = plaq_exact(beta) * tf.ones_like(plaqs) + metrics.update({ + 'plaqs_err': pexact - plaqs + }) + + metrics.update({ + 'intQ': charges.intQ, 'sinQ': charges.sinQ + }) + return metrics + + def observables(self, x: Tensor) -> LatticeMetrics: + """Calculate Lattice observables.""" + wloops = self.wilson_loops(x) + return LatticeMetrics(p4x4=self.plaqs4x4(x=x), + plaqs=self.plaqs(wloops=wloops), + charges=self.charges(wloops=wloops)) + + def wilson_loops(self, x: Tensor) -> Tensor: + """Calculate the Wilson loops by summing links in CCW direction.""" + # -------------------------- + # NOTE: Watch your shapes! + # -------------------------- + # * First, x.shape = [-1, 2, Lt, Lx], so + # (x_reshaped).T.shape = [2, Lx, Lt, -1] + # and, + # x0.shape = x1.shape = [Lx, Lt, -1] + # where x0 and x1 are the links along the 2 (t, x) dimensions. + # + # * The Wilson loop is then: + # wloop = U0(x, y) + U1(x+1, y) - U0(x, y+1) - U1(x, y) + # and so output = wloop.T, with output.shape = [-1, Lt, Lx] + # -------------------------- + x = tf.transpose(tf.reshape(x, self._shape), (1, 2, 3, 0)) + x0 = x[0] # type:ignore NOTE: x0 = t links + x1 = x[1] # type:ignore NOTE: x1 = x links + return tf.transpose( + x0 + tf.roll(x1, -1, axis=0) - tf.roll(x0, -1, axis=1) - x1 + ) + + def wilson_loops4x4(self, x: Tensor) -> Tensor: + """Calculate the 4x4 Wilson loops""" + x = tf.transpose(tf.reshape(x, (-1, *self.xshape)), (1, 2, 3, 0)) + x0 = x[0] # type:ignore + x1 = x[1] # type:ignore + return tf.transpose( + x0 # Ux [x, y] + + tf.roll(x0, -1, 2) # Ux [x+1, y] + + tf.roll(x0, -2, 2) # Ux [x+2, y] + + tf.roll(x0, -3, 2) # Ux [x+3, y] + + tf.roll(x0, -4, 2) # Ux [x+4, y] + + tf.roll(x1, (-4, -1), (2, 1)) # Uy [x+4, y] + + tf.roll(x1, (-4, -2), (2, 1)) # Uy [x+4, y+1] + + tf.roll(x1, (-4, -3), (2, 1)) # Uy [x+4, y+3] + - tf.roll(x0, (-3, -4), (2, 1)) # -Ux [x+3, y+4] + - tf.roll(x0, (-2, -4), (2, 1)) # -Ux [x+2, y+4] + - tf.roll(x0, (-1, -4), (2, 1)) # -Ux [x+1, y+4] + - tf.roll(x1, -4, 1) # -Uy [x, y+4] + - tf.roll(x1, -3, 1) # -Uy [x, y+3] + - tf.roll(x1, -2, 1) # -Uy [x, y+2] + - tf.roll(x1, -1, 1) # -Uy [x, y+1] + - x1 # -Uy [x, y] + ) + + def plaqs_diff( + self, + beta: float, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None, + ) -> Tensor: + wloops = self._get_wloops(x) if wloops is None else wloops + plaqs = self.plaqs(wloops=wloops) + pexact = plaq_exact(beta) * tf.ones_like(plaqs) + return pexact - self.plaqs(wloops=wloops) + + def plaqs( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the avg plaq for each of the lattices in x.""" + wloops = self._get_wloops(x) if wloops is None else wloops + return tf.reduce_mean(tf.math.cos(wloops), (1, 2)) + + def _plaqs4x4(self, wloops4x4: Tensor) -> Tensor: + return tf.reduce_mean(tf.math.cos(wloops4x4), (1, 2)) + + def plaqs4x4( + self, + x: Optional[Tensor] = None, + wloops4x4: Optional[Tensor] = None + ) -> Tensor: + """Calculate 4x4 wilson loops.""" + if wloops4x4 is None: + if x is None: + raise ValueError('One of `x` or `wloops` must be specified') + wloops4x4 = self.wilson_loops4x4(x) + + return self._plaqs4x4(wloops4x4) + + def _sin_charges(self, wloops: Tensor) -> Tensor: + """Calculate sinQ from Wilson loops.""" + return tf.reduce_sum(tf.math.sin(wloops), (1, 2)) / TWO_PI + + def _int_charges(self, wloops: Tensor) -> Tensor: + return tf.reduce_sum(project_angle(wloops), (1, 2)) / TWO_PI + + def _get_wloops(self, x: Optional[Tensor] = None) -> Tensor: + if x is None: + raise ValueError('Expected input `x`') + return self.wilson_loops(x) + + def sin_charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the real-valued charge approximation, sin(Q)""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._sin_charges(wloops) + + def int_charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Tensor: + """Calculate the integer valued charges.""" + wloops = self._get_wloops(x) if wloops is None else wloops + return self._int_charges(wloops) + + def charges( + self, + x: Optional[Tensor] = None, + wloops: Optional[Tensor] = None + ) -> Charges: + """Calculate both charge representations and return as single object""" + wloops = self._get_wloops(x) if wloops is None else wloops + sinQ = self._sin_charges(wloops) + intQ = self._int_charges(wloops) + return Charges(intQ=intQ, sinQ=sinQ) + + def plaq_loss( + self, + acc: Tensor, + x1: Optional[Tensor] = None, + x2: Optional[Tensor] = None, + wl1: Optional[Tensor] = None, + wl2: Optional[Tensor] = None, + ) -> float: + wloops1 = self._get_wloops(x1) if wl1 is None else wl1 + wloops2 = self._get_wloops(x2) if wl2 is None else wl2 + dwl = tf.subtract(wloops2, wloops1) + dwloops = 2. * (tf.ones_like(wl1) - tf.math.cos(dwl)) + ploss = acc * tf.reduce_sum(dwloops, axis=(1, 2)) + 1e-4 + + return tf.reduce_mean(-ploss, axis=0) + + def charge_loss( + self, + acc: Tensor, + x1: Optional[Tensor] = None, + x2: Optional[Tensor] = None, + wl1: Optional[Tensor] = None, + wl2: Optional[Tensor] = None, + ) -> float: + wloops1 = self._get_wloops(x1) if wl1 is None else wl1 + wloops2 = self._get_wloops(x2) if wl2 is None else wl2 + q1 = self._sin_charges(wloops=wloops1) + q2 = self._sin_charges(wloops=wloops2) + qloss = (acc * tf.math.subtract(q2, q1) ** 2) + 1e-4 + return tf.reduce_mean(-qloss, axis=0) diff --git a/.github/entrypoint/artifact/python/src/params/spaces.json b/.github/entrypoint/artifact/python/src/params/spaces.json new file mode 100644 index 000000000000..858ee348e189 --- /dev/null +++ b/.github/entrypoint/artifact/python/src/params/spaces.json @@ -0,0 +1,54 @@ +{ + "strategy_name": "fibbo", + "params": { + "roi": { + "0": 0.0786, + "4": 0.0238, + "11": 0.006, + "19": 0 + }, + "stoploss": { + "stoploss": -0.137 + }, + "trailing": { + "trailing_stop": true, + "trailing_stop_positive": 0.099, + "trailing_stop_positive_offset": 0.178, + "trailing_only_offset_is_reached": false + }, + "max_open_trades": { + "max_open_trades": Infinity + }, + "buy": { + "buy_additional_indicator": "NONE", + "buy_fast_dema": "13", + "buy_fib_level": "0.618", + "buy_rsi": 45, + "buy_slow_ema": "34", + "buy_swing_period": 50, + "period": 14, + "SmoothK": 2, + "buy_stoch_osc": 29, + "smoothD": 3 + }, + "sell": { + "sell_additional_indicator": "NONE", + "sell_fib_level": "0.382", + "sell_rsi": 70, + "sell_rsi_threshold": 64, + "sell_stoch_osc": 85 + }, + "protection": { + "cooldown_lookback": 7, + "low_profit_trade_limit": 6, + "max_drawdown_trade_limit": 4, + "stop_duration": 156, + "trade_limit": 9, + "use_low_profit": true, + "use_max_drawdown_protection": false, + "use_stop_protection": true + } + }, + "ft_stratparam_v": 1, + "export_time": "2025-02-20 13:28:29.884316+00:00" +} diff --git a/.github/entrypoint/artifact/resum.c b/.github/entrypoint/artifact/resum.c new file mode 100644 index 000000000000..62d225e04604 --- /dev/null +++ b/.github/entrypoint/artifact/resum.c @@ -0,0 +1,100 @@ +#include "BosonJetNLL.h" +#include +#include +#include +#include + +//#define NLL_DEBUG + +using namespace std; + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={xT, phi, pT, yB,yJ}; + BosonJetNLL *bj=(BosonJetNLL *)p; + double cphix=cos(y[1]); + return bj->sigmaR(y[0],cphix,y[2],y[3],y[4]); +} + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetNLL ZJ(1.3e4,&Z0); + + double R=0.4;ZJ.setR(R); + + #ifdef NLL_DEBUG + double pT=100.0,yB=0.1,yJ=0.2,muh=100.0,mu=30.0,muJ=10.0, mut=5.0, muxT=70.0, xT=1.2,qT=1.0,cphix=0.4;ZJ.setqT(qT); + ZJ.setKinematics(pT,yB,yJ);ZJ.getKinematics(); + ZJ.setScale(mu,muh,muJ,muxT,mut,xT,cphix); + cout << ZJ.aHqqb()<< ", " << ZJ.bHqqb() << endl; + ZJ.setLL(true); + cout << ZJ.RGInvariance(mu,muh,muxT,muJ,mut,xT,cphix) << endl; + /* + cout << M_PI*ZJ.sigmaqgR(xT, cphix, pT, yB, yJ) < +#include +#include +#include + +//#define LL_DEBUG + +using namespace std; + +double dsigmadqT(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={xT, phi, pT, yB,yJ}; + BosonJetLL *bj=(BosonJetLL *)p; + double cphix=cos(y[1]); + return bj->sigmaR(y[0],cphix,y[2],y[3],y[4]); +} + + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetLL ZJ(1.3e4,&Z0); + + double R=0.6;ZJ.setR(R); + + #ifdef LL_DEBUG + double pT=100.0,yB=0.1,yJ=0.2,muh=100.0,mu=30.0,muJ=10.0, mut=5.0, muxT=70.0, xT=1.2,qT=1.0,cphix=0.4;ZJ.setqT(qT); + ZJ.setKinematics(pT,yB,yJ);ZJ.getKinematics(); + ZJ.setScale(mu,muh,muJ,muxT,mut,xT,cphix); + cout << ZJ.aHqqb()<< ", " << ZJ.bHqqb() << endl; + cout << ZJ.RGInvariance(mu,muh,muxT,muJ,mut,xT,cphix) << endl; + /* + cout << M_PI*ZJ.sigmaqgR(xT, cphix, pT, yB, yJ) < + +using namespace std; + +int main(){ + QCD qcd; + /* + for(double Q=0.1;Q<100;Q+=0.1) + cout << Q << " " << qcd.alphas(Q) << " " << qcd.pdf(21, 0.1,Q) << endl; + */ + double Q = + + for(double x=0.001;x<1.0;x+=0.001){ + cout << x << " " << qcd.pdf(1,x,3.3) << endl; + } + + return 0; +} diff --git a/.github/entrypoint/artifact/tot.c b/.github/entrypoint/artifact/tot.c new file mode 100644 index 000000000000..330ee971825f --- /dev/null +++ b/.github/entrypoint/artifact/tot.c @@ -0,0 +1,54 @@ +#include "BosonJetNLL.h" +#include +#include +#include +#include + +using namespace std; + +double sigma(double y[], size_t dim, void *p){ + //dim=5 dimensional integration variables: y[]={xT, phi, pT, yB,yJ, qT}; + BosonJetNLL *bj=(BosonJetNLL *)p; + double cphix=cos(y[1]); + bj->setqT(y[5]); + return y[5]*(bj->sigmaR(y[0],cphix,y[2],y[3],y[4])); +} + +int main(){ + + //Particle photon("photon",Particle::photon,0.0,0.0); + Particle Z0("Z0",Particle::Z0,91.1876,0.0); + //BosonJet gammaJ(1.3e4,&photon); + BosonJetNLL ZJ(1.3e4,&Z0); + + double R=0.1;ZJ.setR(R); + + //range of the integration: from ymin to ymax + double pT=200.0; + double ymin[]={0.0,-0.5*M_PI,pT,-10.0,-2.4,0.0}; + double ymax[]={1.5,0.5*M_PI,6500.0,10.0,2.4,pT}; + + int dim=6; + size_t calls = 500000; + ZJ.setupMC( &sigma, dim, ymin,ymax, &ZJ); + ZJ.setCalls(calls); + //output file: + + ZJ.setImgcphix(true); + ZJ.setCSS(false); + ZJ.setNGL(true); + ZJ.setLL(false); + ZJ.setNonPert(false); + ZJ.setScaleVar(1.0,1.0,1.0,1.0); + + double res[2]; + double Rls[]={0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0}; + + for(int i=0;i<10;i++){ + ZJ.setR(Rls[i]);ZJ.calculate(res); + cout << Rls[i] << " " << res[0] << " " << res[1] << endl; + } + ZJ.cleanupMC(); + + return 0; +} diff --git a/.github/entrypoint/cleanup_bundler.rb b/.github/entrypoint/cleanup_bundler.rb new file mode 100644 index 000000000000..a579baf8bc15 --- /dev/null +++ b/.github/entrypoint/cleanup_bundler.rb @@ -0,0 +1,10 @@ +#!/usr/bin/env ruby + +gempaths = %x(gem env gempath).split(":") +gempaths.each do |gempath| + # lookup bundler-*.gemspec files and delete them + # this is the only way to completely cleanup default bundler + # Note: the bundler gemspecs' paths are different for CRuby and JRuby + Dir.glob(gempath.strip + "/specifications/**/bundler-*.gemspec").each { |p| File.delete(p) } +end + diff --git a/.github/entrypoint/config.sh b/.github/entrypoint/config.sh new file mode 100755 index 000000000000..5657f5c0c848 --- /dev/null +++ b/.github/entrypoint/config.sh @@ -0,0 +1,14 @@ +#!/usr/bin/env bash + +git config --global user.name "${GITHUB_ACTOR}" +git config --global --add safe.directory "${GITHUB_WORKSPACE}" +git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com" + +#git config --global credential.helper store +#echo "https://${GITHUB_ACTOR}:${GITHUB_ACCESS_TOKEN}@github.com" > ~/.git-credentials + +echo 'REMOTE_REPO="https://${GITHUB_ACTOR}:${GITHUB_ACCESS_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"' >> ${GITHUB_ENV} +COMMIT=$(curl -s "https://api.github.com/users/eq19/events/public" | jq ".[0].payload.commits[0].message") + +# Remove double quotes using parameter expansion +echo 'LATEST_COMMIT="${COMMIT//\"/}"' >> ${GITHUB_ENV} diff --git a/.github/entrypoint/entrypoint_.sh b/.github/entrypoint/entrypoint_.sh new file mode 100755 index 000000000000..4e062fd313ed --- /dev/null +++ b/.github/entrypoint/entrypoint_.sh @@ -0,0 +1,202 @@ +#!/usr/bin/env bash + +set -e +export WORKING_DIR=${PWD} +chown -R $(whoami) ${WORKING_DIR} +export hr=$(printf '=%.0s' {1..80}) + +# Initial default value +export TOKEN=${INPUT_TOKEN} +export ACTOR=${GITHUB_ACTOR} +export REPOSITORY=${INPUT_REPOSITORY} +export OWNER=${GITHUB_REPOSITORY_OWNER} +export BRANCH=${INPUT_BRANCH:=gh-pages} +export PROVIDER=${INPUT_PROVIDER:=github} +export BUNDLER_VER=${INPUT_BUNDLER_VER:=>=0} +export JEKYLL_BASEURL=${INPUT_JEKYLL_BASEURL:=} +export PRE_BUILD_COMMANDS=${INPUT_PRE_BUILD_COMMANDS:=} +export SCRIPT_DIR=/maps/Scripts && chmod -R +x ${SCRIPT_DIR} +export PREFIX=${WORKING_DIR}/docs/_data/ta-lib/linux/ta-lib + +# https://stackoverflow.com/a/42137273/4058484 +export JEKYLL_SRC=${WORKING_DIR} +export JEKYLL_GITHUB_TOKEN=${TOKEN} + +if [[ "${OWNER}" == "chetabahana" ]]; then + export JEKYLL_SRC=${WORKING_DIR}/docs +fi +export JEKYLL_CFG=${JEKYLL_SRC}/_config.yml +sed -i -e "s/eq19/${OWNER}/g" ${JEKYLL_CFG} + +if [[ -z "${TOKEN}" ]]; then + echo -e "Please set the TOKEN environment variable." + exit 1 +fi + +# Pre-handle Jekyll baseurl +if [[ -n "${JEKYLL_BASEURL-}" ]]; then + JEKYLL_BASEURL="--baseurl ${JEKYLL_BASEURL}" +fi + +# Initialize environment +export RUBYOPT=W0 +export RUBY_VERSION=2.7.0 +export RAILS_VERSION=5.0.1 +export BUNDLER_VER=${BUNDLER_VER} +export BUNDLE_GEMFILE=/maps/Gemfile +export BUNDLE_SILENCE_ROOT_WARNING=1 +#export TA_LIBRARY_PATH=${PREFIX}/lib +export NOKOGIRI_USE_SYSTEM_LIBRARIES=1 +#export TA_INCLUDE_PATH=${PREFIX}/include +export PAGES_REPO_NWO=$GITHUB_REPOSITORY +export VENDOR_BUNDLE=${WORKING_DIR}/vendor/bundle +export SSL_CERT_FILE=$(realpath .github/hook-scripts/cacert.pem) + +# identity +echo -e "\n$hr\nWHOAMI\n$hr" +whoami +pwd +id + +# os version +echo -e "\n$hr\nOS VERSION\n$hr" +cat /etc/os-release +uname -r + +# file system +echo -e "\n$hr\nFILE SYSTEM\n$hr" +df -h + +# rootdir +echo -e "$hr\nROOT DIR\n$hr" +cd / && pwd && ls -al + +# packages +echo -e "$hr\nVENV DIR\n$hr" +cd /maps && pwd && ls -al + +# current +echo -e "$hr\nWORK DIR\n$hr" +cd ${WORKING_DIR} && pwd && ls -al + +#echo -e "$hr\nPRIOR INSTALLATION\n$hr" +#chown -R root:root ${HOME} && dpkg -l + +# cloning default repository +# https://stackoverflow.com/a/74439875/4058484 +git config --global user.name "${ACTOR}" +git config --global user.email "${ACTOR}@users.noreply.github.com" +git config --global credential.helper store &>/dev/null +echo "https://{ACTOR}:${TOKEN}@github.com" > /root/.git-credentials +git clone --quiet --recurse-submodules -j8 ${REPOSITORY} /maps/feed/default + +export NPM_CACHE_DIR=${VENDOR_BUNDLE}/npm +apt-get install -qq npm &>/dev/null && apt-get install -qq yarn &>/dev/null +npm install --prefix /maps --cache ${NPM_CACHE_DIR} &>/dev/null + +export PATH=${HOME}/.local/bin:$PATH +export PIP_CACHE_DIR=${VENDOR_BUNDLE}/pip + +#wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz &>/dev/null +cd ${PREFIX} #tar -xzf ta-lib-0.4.0-src.tar.gz &>/dev/null && cd ta-lib/ +./configure --prefix=/usr &>/dev/null && make &>/dev/null && make install &>/dev/null + +cd ${WORKING_DIR} +# https://pypi.org/project/pipx/ +python -m pip install --upgrade pip setuptools six wheel &>/dev/null +python -m pip install pytest-cov -r /maps/requirements.txt &>/dev/null +python -m pip install jax[cuda11_cudnn82] \ + -f https://storage.googleapis.com/jax-releases/jax_releases.html &>/dev/null + +export GEM_PATH=${VENDOR_BUNDLE}/gem +export GEM_HOME=${GEM_PATH}/ruby/${RUBY_VERSION} +export PATH=$PATH:${GEM_HOME}/bin + +# https://stackoverflow.com/a/30369485/4058484 +apt-get install -qq ruby ruby-dev ruby-bundler build-essential &>/dev/null +gem install rails --version "$RAILS_VERSION" --quiet --silent &>/dev/null + +# installed packages +#echo -e "\n$hr\nUPON INSTALLATION\n$hr" +#dpkg -l + +# Setting default ruby version +echo -e "$hr\nTENSORFLOW VERSION\n$hr" +pip show tensorflow-gpu && pip -V + +# https://stackoverflow.com/a/60945404/4058484 +ruby -v && bundler version && python -V +node -v && npm -v + +# Restore modification time (mtime) of git files +echo -e "$hr\nEPOCH TEST\n$hr" +${SCRIPT_DIR}/restore.sh +${SCRIPT_DIR}/prime_list.sh +${SCRIPT_DIR}/init_environment.sh + +# Clean up bundler cache +CLEANUP_BUNDLER_CACHE_DONE=false +bundle config path ${GEM_PATH} +bundle config cache_all true + +cleanup_bundler_cache() { + ${SCRIPT_DIR}/cleanup_bundler.sh + rm -rf ${GEM_HOME} && mkdir -p ${GEM_HOME} + gem install bundler -v "${BUNDLER_VER}" &>/dev/null + echo -e "\nCLEANUP BUNDLE\n$hr" && bundle install + CLEANUP_BUNDLER_CACHE_DONE=true +} + +# If the vendor/bundle folder is cached in a differnt OS (e.g. Ubuntu), +# it would cause `jekyll build` failed, we should clean up the cache firstly. +OS_NAME_FILE=${GEM_PATH}/os-name +os_name=$(cat /etc/os-release | grep '^NAME=') +os_name=${os_name:6:-1} + +if [[ "$os_name" != "$(cat $OS_NAME_FILE 2>/dev/null)" ]]; then + cleanup_bundler_cache + echo -e $os_name > $OS_NAME_FILE +fi + +# Check and execute pre_build_commands commands +cd ${JEKYLL_SRC} + +if [[ ${PRE_BUILD_COMMANDS} ]]; then + eval "${PRE_BUILD_COMMANDS}" +fi + +build_jekyll() { + echo -e "\n$hr\nJEKYLL BUILD\n$hr" && pwd + # https://gist.github.com/DrOctogon/bfb6e392aa5654c63d12 + JEKYLL_GITHUB_TOKEN=${TOKEN} bundle exec jekyll build --trace --profile \ + ${JEKYLL_BASEURL} -c ${JEKYLL_CFG} -d ${WORKING_DIR}/build + + # vendor/bundle + echo -e "\n$hr\nVENDOR BUNDLE\n$hr" && chown -R root:root ${HOME} + cd ${HOME} && mv -f .gem gem && mv -f gem/* ${VENDOR_BUNDLE}/gem/ + echo ${VENDOR_BUNDLE} && ls -al ${VENDOR_BUNDLE} && echo -e "\n" + echo ${NPM_CACHE_DIR} && ls -al ${NPM_CACHE_DIR} && echo -e "\n" + echo ${PIP_CACHE_DIR} && ls -al ${PIP_CACHE_DIR} && echo -e "\n" + echo ${GEM_PATH} && ls -al ${GEM_PATH} && echo -e "\n" + echo ${GEM_HOME} && ls -al ${GEM_HOME} && echo -e "\n" + chmod -R a+rwx,o-w ${PIP_CACHE_DIR} + echo -e "Source cleaning..\n" + rm -rf gem && ls -al +} + +build_jekyll || { + $CLEANUP_BUNDLER_CACHE_DONE && exit -1 + echo -e "\nRebuild all gems and try to build again\n" + cleanup_bundler_cache + build_jekyll +} + + +cd ${WORKING_DIR} +echo -e "$hr\nDEPLOYMENT\n$hr" +# https://unix.stackexchange.com/a/83895/158462 +git submodule foreach -q ${SCRIPT_DIR}/github_pages.sh + + +apt-get clean && rm -rf /var/lib/apt/lists/* ${RUNNER_TEMP}/* /var${RUNNER_TEMP}/* +exit $? diff --git a/.github/entrypoint/front.sh b/.github/entrypoint/front.sh new file mode 100755 index 000000000000..b6230c651082 --- /dev/null +++ b/.github/entrypoint/front.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash + +edit_file () { + + NUM=$(($2 + 0)) + while IFS=' ' read -ra SPIN; do + T+=("${SPIN[0]}") + R+=("${SPIN[1]}") + A+=("${SPIN[2]}") + C+=("${SPIN[3]}") + K+=("${SPIN[4]}") + I+=("${SPIN[5]}") + N+=("${SPIN[6]}") + G+=("${SPIN[7]}") + done < ${RUNNER_TEMP//\\//}/spin.txt + + IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP//\\//}/Sidebar.md; unset IFS; + for ((i=0; i < ${#LINE[@]}; i++)); do + TEXT="${LINE[$i]}"; + IFS='|'; array=($TEXT); unset IFS; + if [[ "${array[1]}" == "${NUM}" ]]; then + FRONT="---\n" + FRONT+="sort: $i\n" + FRONT+="spin: $NUM\n" + FRONT+="span: ${N[$NUM]}\n" + FRONT+="suit: ${I[$NUM]}\n" + FRONT+="description: ${TEXT##*|}\n" + FRONT+="---\n# ${TEXT%%|*}\n\n" + sed -i "1s|^|$FRONT|" $1 + fi + done + + #if [[ $NUM -lt 2 || $NUM == 18 || $NUM -gt 29 ]]; then + mv -f $1 ${1%/*}/README.md + #sed '1,8!d' ${1%/*}/README.md + #fi + +} + +FILE=${1##*/} && SORT=${FILE%.*} +[[ $SORT =~ ^-?[0-9]+$ ]] && edit_file $1 $SORT diff --git a/.github/entrypoint/gcp_access.py b/.github/entrypoint/gcp_access.py new file mode 100644 index 000000000000..bc97a5824313 --- /dev/null +++ b/.github/entrypoint/gcp_access.py @@ -0,0 +1,102 @@ +import urllib +import argparse +import requests + +import google.auth.transport.requests +import google.oauth2.id_token + +METADATA_URL = "http://metadata.google.internal/computeMetadata/v1/" +METADATA_HEADERS = {"Metadata-Flavor": "Google"} +SERVICE_ACCOUNT = "default" + + +def make_authorized_get_request(): + """ + make_authorized_get_request makes a GET request to the specified HTTP endpoint + by authenticating with the ID token obtained from the google-auth client library + using the specified audience value. + """ + + # Cloud Functions uses your function's URL as the `audience` value + # audience = https://project-region-projectid.cloudfunctions.net/myFunction + audience = "https://us-central1-feedmapping.cloudfunctions.net/function" + + # For Cloud Functions, `endpoint` and `audience` should be equal + # Ref: https://cloud.google.com/functions/docs/securing/authenticating + endpoint = audience + req = urllib.request.Request(endpoint) + + auth_req = google.auth.transport.requests.Request() + id_token = google.oauth2.id_token.fetch_id_token(auth_req, audience) + + req.add_header("Authorization", f"Bearer {id_token}") + response = urllib.request.urlopen(req) + + return response.read() + + +def get_access_token() -> str: + """ + Retrieves access token from the metadata server. + + Returns: + The access token. + """ + url = f"{METADATA_URL}instance/service-accounts/{SERVICE_ACCOUNT}/token" + + # Request an access token from the metadata server. + # Ref: https://cloud.google.com/compute/docs/access/authenticate-workloads#applications + r = requests.get(url, headers=METADATA_HEADERS) + r.raise_for_status() + + # Extract the access token from the response. + access_token = r.json()["access_token"] + + return access_token + + +def list_buckets(project_id: str, access_token: str) -> dict: + """ + Calls Storage API to retrieve a list of buckets. + + Args: + project_id: name of the project to list buckets from. + access_token: access token to authenticate with. + + Returns: + Response from the API. + """ + url = "https://www.googleapis.com/storage/v1/b" + params = {"project": project_id} + headers = {"Authorization": f"Bearer {access_token}"} + + r = requests.get(url, params=params, headers=headers) + r.raise_for_status() + + return r.json() + + +def main(project_id: str) -> None: + """ + Retrieves access token from metadata server and uses it to list + buckets in a project. + + Args: + project_id: name of the project to list buckets from. + """ + #access_token = get_access_token() + access_token = make_authorized_get_request() + #buckets = list_buckets(project_id, access_token) + #print(buckets) + return access_token + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter + ) + parser.add_argument("project_id", help="Your Google Cloud project ID.") + + args = parser.parse_args() + + main(args.project_id) diff --git a/.github/entrypoint/gcp_token.py b/.github/entrypoint/gcp_token.py new file mode 100644 index 000000000000..471e8425f386 --- /dev/null +++ b/.github/entrypoint/gcp_token.py @@ -0,0 +1,101 @@ +# +# Copyright 2019 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import google.oauth2.credentials +from google.oauth2 import id_token +from google.oauth2 import service_account +from google.auth import impersonated_credentials +import google.auth +import google.auth.transport.requests +from google.auth.transport.requests import AuthorizedSession +from google.auth import compute_engine + +# pip install google-auth requests + +target_audience = 'https://us-central1-feedmapping.cloudfunctions.net/function' + +url = 'https://httpbin.org/get' +certs_url='https://www.googleapis.com/oauth2/v1/certs' +metadata_identity_doc_url = "http://metadata/computeMetadata/v1/instance/service-accounts/default/identity" + +svcAccountFile = '/path/to/svcaccount.json' + +def GetIDTokenFromServiceAccount(svcAccountFile, target_audience): + creds = service_account.IDTokenCredentials.from_service_account_file( + svcAccountFile, + target_audience= target_audience) + request = google.auth.transport.requests.Request() + creds.refresh(request) + return creds.token + +def GetIDTokenFromComputeEngine(target_audience): + request = google.auth.transport.requests.Request() + #creds = compute_engine.IDTokenCredentials(request=request, target_audience=target_audience, use_metadata_identity_endpoint=True) + #creds.refresh(request) + #return creds.token + + token = id_token.fetch_id_token(request, target_audience) + return token + +# FOR ADC local +# ADC account must have roles/iam.serviceAccountTokenCreator permission on target_principal +def GetIDTokenFromADC(target_prinicpal, target_audience): + credentials, project_id = google.auth.default() + target_scopes = ['https://www.googleapis.com/auth/cloud-platform'] + target_credentials = impersonated_credentials.Credentials( + source_credentials = credentials, + target_principal = target_principal, + target_scopes = target_scopes, + delegates=[], + lifetime=5) + + id_creds = impersonated_credentials.IDTokenCredentials(target_credentials, target_audience=target_audience, include_email=True) + request = google.auth.transport.requests.Request() + id_creds.refresh(request) + token = id_creds.token + return token + +def VerifyIDToken(token, certs_url, audience=None): + request = google.auth.transport.requests.Request() + result = id_token.verify_token(token,request,certs_url=certs_url) + if audience in result['aud']: + return True + return False + +def MakeAuthenticatedRequest(id_token, url): + creds = google.oauth2.credentials.Credentials(id_token) + authed_session = AuthorizedSession(creds) + r = authed_session.get(url) + print(r.status_code) + print(r.text) + +# For ServiceAccount +#token = GetIDTokenFromServiceAccount(svcAccountFile,target_audience) + +# For ADC on Compute Engine, Cloud Run, GCF +#token = GetIDTokenFromComputeEngine(target_audience) + +# For ADC local +target_principal = 'target-serviceaccount@YOUR_PROJECT.iam.gserviceaccount.com' +token = GetIDTokenFromADC(target_principal,target_audience) + +print('Token: ' + token) +if VerifyIDToken(token=token,certs_url=certs_url, audience=target_audience): + print('token Verified with aud: ' + target_audience) +print('Making Authenticated API call:') +MakeAuthenticatedRequest(token,url) + +# to verify ECDSA use certificate certs_url, not JWK +# print(VerifyIDToken(token=token,certs_url='https://www.gstatic.com/iap/verify/public_key', audience="/projects/248066739582/apps/fabled-ray-104117")) diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh new file mode 100755 index 000000000000..d2ed098c4768 --- /dev/null +++ b/.github/entrypoint/gist.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash + +rm -rf ${RUNNER_TEMP}/*dir + +gh gist clone 0ce5848f7ad62dc46dedfaa430069857 ${RUNNER_TEMP}/gistdir/identition/span1 +gh gist clone b32915925d9d365e2e9351f0c4ed786e ${RUNNER_TEMP}/gistdir/identition/span2 +gh gist clone 88d09204b2e5986237bd66d062406fde ${RUNNER_TEMP}/gistdir/identition/span3 +gh gist clone 8cab5e72d52ecb338a2f2187082a1699 ${RUNNER_TEMP}/gistdir/identition/span4 +gh gist clone 54600a56d20163c2da8910dd804ec406 ${RUNNER_TEMP}/gistdir/identition/span5 +gh gist clone f1af4317b619154719546e615aaa2155 ${RUNNER_TEMP}/gistdir/identition/span6 +gh gist clone 6c89c3b0f109e0ead561a452720d1ebf ${RUNNER_TEMP}/gistdir/identition/span7 +gh gist clone f21abd90f8d471390aad23d6ecc90d6d ${RUNNER_TEMP}/gistdir/identition/span8 +gh gist clone 6e2fcc2138be6fb68839a3ede32f0525 ${RUNNER_TEMP}/gistdir/identition/span9 +gh gist clone b541275ab7deda356feef32d600e44d8 ${RUNNER_TEMP}/gistdir/identition/span10 +gh gist clone 80c8098f16f3e6ca06893b17a02d910e ${RUNNER_TEMP}/gistdir/identition/span11 +gh gist clone 4ffc4d02579d5cfd336a553c6da2f267 ${RUNNER_TEMP}/gistdir/identition/span12 + +gh gist clone f78d4470250720fb18111165564d555f ${RUNNER_TEMP}/gistdir/exponentiation/span13 +gh gist clone 765ddc69e339079a5a64b56c1d46e00f ${RUNNER_TEMP}/gistdir/exponentiation/span14 +gh gist clone b9f901cda16e8a11dd24ee6b677ca288 ${RUNNER_TEMP}/gistdir/exponentiation/span15 +gh gist clone dc30497160f3389546d177da901537d9 ${RUNNER_TEMP}/gistdir/exponentiation/span16 +gh gist clone e84a0961dc7636c01d5953d19d65e30a ${RUNNER_TEMP}/gistdir/exponentiation/span17 +gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef ${RUNNER_TEMP}/gistdir/exponentiation/span18 + +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +find ${RUNNER_TEMP}/gistdir -type f -name "README.md" -exec rm -rf {} \; + +git clone ${BASE} ${RUNNER_TEMP}/workdir +mv -f ${RUNNER_TEMP}/workdir/Home.md ${RUNNER_TEMP}/workdir/README.md +find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh new file mode 100755 index 000000000000..1dee0d06e8d9 --- /dev/null +++ b/.github/entrypoint/github_pages.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Structure: Cell Types – Modulo 6 + +hr='------------------------------------------------------------------------------------' + +if [[ "${WIKI}" != "${BASE}" ]]; then + rm -rf ${RUNNER_TEMP//\\//}/wikidir + + git clone $WIKI ${RUNNER_TEMP//\\//}/wikidir + cd ${RUNNER_TEMP//\\//}/wikidir && mv -f Home.md README.md + + find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; + +fi + +#echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 +find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +cd ${GITHUB_WORKSPACE//\\//} && pwd && ls -alR . diff --git a/.github/entrypoint/idle.sh b/.github/entrypoint/idle.sh new file mode 100755 index 000000000000..2e620d0ed020 --- /dev/null +++ b/.github/entrypoint/idle.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash + +# Run script on system idle https://askubuntu.com/a/506975/486942 +while sleep 30s ; do + state=$(gnome-screensaver-command --query | grep -o "\w*active\w*" >> /dev/null) + if [[ $state == active ]] + then + ./start.sh + else + ./stop.sh + fi +done \ No newline at end of file diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh new file mode 100755 index 000000000000..3437b845ba47 --- /dev/null +++ b/.github/entrypoint/init.sh @@ -0,0 +1,103 @@ +#!/usr/bin/env bash +# Structure: Cell Types – Modulo 6 +# https://www.hexspin.com/proof-of-confinement/ + +hr='------------------------------------------------------------------------------------' + +git config --global user.name "${GITHUB_ACTOR}" +git config --global user.email "${GITHUB_ACTOR}@users.noreply.github.com" + +git config --global --add safe.directory "${GITHUB_WORKSPACE}" +[[ "$RUNNER_OS" == "Windows" ]] && YQ=$(choco install yq) +[[ "$RUNNER_OS" == "Windows" ]] && git config --global core.safecrlf false + +git config --global credential.helper store +echo "https://${GITHUB_ACTOR}:${GH_TOKEN}@github.com" > ~/.git-credentials + +TARGET_REPO="https://${GITHUB_ACTOR}:${GH_TOKEN}@github.com/${TARGET_REPOSITORY}.git" +REMOTE_REPO="https://${GITHUB_ACTOR}:${GH_TOKEN}@github.com/${GITHUB_REPOSITORY}.git" + +API_URL="https://api.github.com/users/eq19/events/public" +COMMIT=$(curl -s $API_URL | jq -r 'map(select(.type == "PushEvent")) | .[0].payload.commits[0].message') + +# Remove double quotes using parameter expansion +LATEST_COMMIT="${COMMIT//\"/}" + +if [[ -z "$LATEST_COMMIT" ]] || [[ "$LATEST_COMMIT" == "null" ]]; then + echo 'LATEST_COMMIT="update by workspace"' >> ${GITHUB_ENV} +else + echo 'LATEST_COMMIT='$LATEST_COMMIT >> ${GITHUB_ENV} +fi + +if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then + + echo -e "\n$hr\nENVIRONTMENT\n$hr" + echo 'TARGET_REPO='${TARGET_REPO} >> ${GITHUB_ENV} + echo 'REMOTE_REPO='${REMOTE_REPO} >> ${GITHUB_ENV} + + if [[ -f _config.yml ]]; then + FOLDER=$(yq '.span' _config.yml) + export FOLDER=$(eval echo $FOLDER) + elif [[ -f /home/runner/_site/.env ]]; then + set -a && . /home/runner/_site/.env && set +a + fi + printenv | sort + +fi + +echo -e "\n$hr\nWORKSPACE\n$hr" +if [[ "${JOB_ID}" == "1" ]]; then + + #[[ "${LATEST_COMMIT}" == *"RERUN_RUNNER"* ]] && gh variable set RERUN_RUNNER --body "true" + + cd ${GITHUB_WORKSPACE} && rm -rf .github + cp -r /home/runner/work/_actions/eq19/eq19/v2/.github . + chown -R "$(whoami)" .github + + git remote set-url origin ${REMOTE_REPO} + git add . && git commit -m "update workflows" --quiet && git push --quiet + if [ $? -eq 0 ]; then + git clone --single-branch --branch gh-pages $REMOTE_REPO gh-pages && cd gh-pages + git add . && git commit --allow-empty -m "rerun due to job update" && git push + exit 1 + else + PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json + mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json + + mv -f $1/dataFile/user_data ${GITHUB_WORKSPACE}/ + cd ${GITHUB_WORKSPACE} && ls -al ${GITHUB_WORKSPACE} + fi + +elif [[ "${JOB_ID}" == "2" ]]; then + + ls -alR ${GITHUB_WORKSPACE} + +elif [[ "${JOB_ID}" == "3" ]]; then + + cat /home/runner/_site/_config.yml + gist.sh ${TARGET_REPOSITORY} ${FOLDER} + + find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; + mv -f ${RUNNER_TEMP}/workdir/* /home/runner/_site/ + + rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md + sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md + cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && ls -lR . + +else + + cd ${RUNNER_TEMP//\\//} && rm -rf gh-source + git clone --single-branch --branch gh-source $TARGET_REPO gh-source + + cd ${GITHUB_WORKSPACE//\\//} + #find -not -path "./.git/*" -not -name ".git" | grep git + find -not -path "./.git/*" -not -name ".git" -delete + + rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git + shopt -s dotglob && mv -f ${RUNNER_TEMP//\\//}/gh-source/* . && ls -lR . + +fi + +if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then + echo -e "\n$hr\nGITHUB CONTEXT\n$hr" +fi diff --git a/.github/entrypoint/pinned_repos.rb b/.github/entrypoint/pinned_repos.rb new file mode 100755 index 000000000000..29ef7861a6cb --- /dev/null +++ b/.github/entrypoint/pinned_repos.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby + +# get a user's pinned repos +# https://stackoverflow.com/q/43352056/4058484 +# https://stackoverflow.com/q/7785793/4058484 + +require 'uri' +require 'rubygems' +require 'nokogiri' +require 'open-uri' + +# if ARGV.length != 1 +# STDERR.puts "usage: /maps/pinned_repos.rb " +# exit 1 +# end + +token = ENV['INPUT_TOKEN'] +username = ARGV[0] +mode = ARGV[1] + +# open("http://...", :http_basic_authentication=>[user, password]) +# profile_url = "https://github.com/#{username}" + +profile_url = "https://github.com/#{username}?view_as=#{mode}" + +uri = URI.parse( profile_url ) +params = { :view_as => "#{mode}" } +uri.query = URI.encode_www_form( params ) + +page = Nokogiri::HTML(URI.open(profile_url, "Authorization" => "Bearer #{token}")) +page.css("span.repo").each { |repo| puts repo.text } diff --git a/.github/entrypoint/prime_list.py b/.github/entrypoint/prime_list.py new file mode 100644 index 000000000000..a99d0da8e586 --- /dev/null +++ b/.github/entrypoint/prime_list.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python + +import os +import glob +import talib +import numpy as np +import pandas as pd +import pandas_ta as ta + +# Suppress warnings +os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' +import tensorflow as tf +from tensorflow.keras import layers + +tf.get_logger().setLevel('ERROR') +tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR) + +# Make numpy values easier to read. +np.set_printoptions(precision=3, suppress=True) +os.chdir("/github/workspace/docs/_data/primes") + +# use glob to get all the csv files in the folder +# https://www.geeksforgeeks.org/how-to-read-all-csv-files-in-a-folder-in-pandas/ + +path = os.getcwd() +pd.DataFrame().ta.indicators() +print(talib.get_function_groups()) +csv_files = glob.glob(os.path.join(path, "*.csv")) + +for f in csv_files: + # read the csv file + df_train = pd.read_csv(f, names=["Length", "Diameter", "Height", "Whole weight", "Shucked weight","Viscera weight", "Shell weight", "Age"]) + +df_train.head() +df_features = df_train.copy() +df_labels = df_features.pop('Age') +df_features = np.array(df_features) +df_features + +df_model = tf.keras.Sequential([ + layers.Dense(64), + layers.Dense(1) +]) + +df_model.compile(loss = tf.keras.losses.MeanSquaredError(), + optimizer = tf.keras.optimizers.Adam()) +df_model.fit(df_features, df_labels, epochs=10) diff --git a/.github/entrypoint/remote.sh b/.github/entrypoint/remote.sh new file mode 100755 index 000000000000..5ae0d60a0721 --- /dev/null +++ b/.github/entrypoint/remote.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +deploy_remote() { + echo -e "Deploying to $1 on branch gh-pages" + REMOTE_REPO="https://${ACTOR}:${TOKEN}@github.com/$1.git" + + git config --global user.name "${ACTOR}" + git config --global user.email "${ACTOR}@users.noreply.github.com" + + git clone -b gh-pages --single-branch ${REMOTE_REPO} &>/dev/null + cd "$(basename "${REMOTE_REPO}" .git)" && rm -rf * + + mv -v ${GITHUB_WORKSPACE}/_site/* . && touch .nojekyll && git add . + git commit -m "jekyll build from cction ${GITHUB_SHA}" + git push -u origin gh-pages +} diff --git a/.github/entrypoint/restore.sh b/.github/entrypoint/restore.sh new file mode 100755 index 000000000000..9d4ab0511c9a --- /dev/null +++ b/.github/entrypoint/restore.sh @@ -0,0 +1,24 @@ +#!/usr/bin/env bash + +# Restore the modification time (mtime) of all files in work tree, based on the +# date of the most recent commit that modified the file. +for f in $(git ls-files); do + mtime=$(git log -1 --format='%at' -- "$f") + [[ "$mtime" == "" ]] && continue + mtime=$(date -d @"$mtime" "+%Y-%m-%dT%H:%M:%S") + touch -d "$mtime" "$f" +done + +cd ${WORKING_DIR} +git submodule update --init --recursive +git submodule foreach --recursive git fetch +git submodule foreach 'git fetch origin; git checkout $(git rev-parse --abbrev-ref HEAD); git reset --hard origin/$(git rev-parse --abbrev-ref HEAD); chown -R $(whoami) .git; chmod -R +x .git; git config --unset http.https://github.com/.extraheader; git config --unset-all http.https://github.com/.extraheader; git config --global --unset http.https://github.com/.extraheader; git config --system --unset http.https://github.com/.extraheader;git submodule update --recursive; git clean -dfx' + +cd ${WORKING_DIR}/docs/_feed/parser +mv _sass ${JEKYLL_SRC}/ && ln -s ${JEKYLL_SRC}/_sass _sass +mv assets ${JEKYLL_SRC}/ && ln -s ${JEKYLL_SRC}/assets assets +mv _plugins ${JEKYLL_SRC}/ && ln -s ${JEKYLL_SRC}/_plugins _plugins +mv _layouts ${JEKYLL_SRC}/ && ln -s ${JEKYLL_SRC}/_layouts _layouts +mv _includes ${JEKYLL_SRC}/ && ln -s ${JEKYLL_SRC}/_includes _includes + +cd ${WORKING_DIR} diff --git a/.github/entrypoint/sort.sh b/.github/entrypoint/sort.sh new file mode 100755 index 000000000000..632ac1c967a7 --- /dev/null +++ b/.github/entrypoint/sort.sh @@ -0,0 +1,34 @@ +#!/usr/bin/env bash + +edit_file () { + + while ((i++)); IFS=' ' read -ra SORT; do + + T+=("${SORT[0]}") + R+=("${SORT[1]}") + A+=("${SORT[2]}") + C+=("${SORT[3]}") + K+=("${SORT[4]}") + I+=("${SORT[5]}") + N+=("${SORT[6]}") + G+=("${SORT[7]}") + + done < ${RUNNER_TEMP}/spin.txt + + while ((j++)); IFS=' ' read -ra SORT; do + + SPIN="$((i+j-1))" + SPIN+=" ${SORT[0]}" + SPIN+=" ${SORT[1]}" + SPIN+=" ${SORT[2]}" + SPIN+=" ${SORT[3]}" + + echo "${SPIN} ${SORT[0]}" >> ${RUNNER_TEMP}/spin.txt + + done < $1 + +} + +FILE=${1##*_} && NAME=${FILE%.*} +[[ $(($NAME + 0)) < 3 ]] && edit_file $1 + diff --git a/.github/entrypoint/startup.sh b/.github/entrypoint/startup.sh new file mode 100644 index 000000000000..516b7b6b12ce --- /dev/null +++ b/.github/entrypoint/startup.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +# Fetch the config.json from Google Secret Manager +/mnt/disks/deeplearning/usr/bin/gcloud secrets versions access latest --secret=freqtrade-config > /home/runner/config.json + +# Start Freqtrade +freqtrade trade --config /home/runner/config.json diff --git a/.github/entrypoint/test.py b/.github/entrypoint/test.py new file mode 100644 index 000000000000..fef0ba98341b --- /dev/null +++ b/.github/entrypoint/test.py @@ -0,0 +1,322 @@ +import locale +import os +import pathlib +import subprocess +import typing as t + +import cryptography.x509 as x509 +import psycopg +import furl +import pytest + + +ConnectionFactory = t.Callable[[str], psycopg.Connection] + + +@pytest.fixture(scope="function") +def is_windows() -> bool: + """Returns True if running on Windows.""" + + return os.name == "nt" + + +@pytest.fixture(scope="function") +def is_windows_server_2019(is_windows: bool) -> bool: + """Returns True if running on Windows Server 2019.""" + + if not is_windows: + return False + + windows_caption = subprocess.check_output(["wmic", "os", "get", "Caption"], text=True) + return "Windows Server 2019" in windows_caption + + +@pytest.fixture(scope="function") +def connection_uri() -> str: + """Read and return connection URI from environment.""" + + connection_uri = os.getenv("CONNECTION_URI") + if connection_uri is None: + pytest.fail("CONNECTION_URI: environment variable is not set") + return connection_uri + + +@pytest.fixture(scope="function") +def service_name() -> str: + """Read and return connection URI from environment.""" + + service_name = os.getenv("SERVICE_NAME") + if service_name is None: + pytest.fail("SERVICE_NAME: environment variable is not set") + return service_name + + +@pytest.fixture(scope="function") +def connection_factory() -> ConnectionFactory: + """Return 'psycopg.Connection' factory.""" + + def factory(connection_uri: str) -> psycopg.Connection: + return psycopg.connect(connection_uri) + return factory + + +@pytest.fixture(scope="function", params=["uri", "kv-string"]) +def connection( + request: pytest.FixtureRequest, + connection_factory: ConnectionFactory, + connection_uri: str, + service_name: str, +) -> psycopg.Connection: + """Return 'psycopg.Connection' for connection URI set in environment.""" + + if request.param == "uri": + return connection_factory(connection_uri) + elif request.param == "kv-string": + return connection_factory(f"service={service_name}") + raise RuntimeError("f{request.param}: unknown value") + + +def test_connection_uri(connection_uri: str): + """Test that CONNECTION_URI matches EXPECTED_CONNECTION_URI.""" + + assert connection_uri == os.getenv("EXPECTED_CONNECTION_URI") + + +def test_service_name(service_name: str): + """Test that SERVICE_NAME matches EXPECTED_SERVICE_NAME.""" + + assert service_name == os.getenv("EXPECTED_SERVICE_NAME") + + +def test_certificate_path(): + """Test that CERTIFICATE_PATH points to the certificate.""" + + certificate_path = os.getenv("CERTIFICATE_PATH") + + if os.getenv("EXPECTED_SSL") == "true": + assert certificate_path + certificate_bytes = pathlib.Path(certificate_path).read_bytes() + certificate = x509.load_pem_x509_certificate(certificate_bytes) + assert certificate.subject.rfc4514_string() == "CN=localhost" + else: + assert not certificate_path + + +def test_server_encoding(connection: psycopg.Connection): + """Test that PostgreSQL's encoding matches the one we passed to initdb.""" + + assert connection.execute("SHOW SERVER_ENCODING").fetchone()[0] == "UTF8" + + +def test_locale(connection: psycopg.Connection, is_windows_server_2019: bool): + """Test that PostgreSQL's locale matches the one we paased to initdb.""" + + locale_exp = "en_US.UTF-8" + + if is_windows_server_2019: + locale_exp = "en-US" + + record = connection \ + .execute("SELECT datcollate, datctype FROM pg_database WHERE datname = 'template0'") \ + .fetchone() + assert record + assert locale.normalize(record[0]) == locale_exp + assert locale.normalize(record[1]) == locale_exp + + record = connection \ + .execute("SELECT datcollate, datctype FROM pg_database WHERE datname = 'template1'") \ + .fetchone() + assert record + assert locale.normalize(record[0]) == locale_exp + assert locale.normalize(record[1]) == locale_exp + + +def test_environment_variables(is_windows: bool): + """Test that only expected 'PG*' variables are set.""" + + pg_environ = {k: v for k, v in os.environ.items() if k.startswith("PG")} + + # In case of Windows, there might be a mix of forward and backward slashes + # as separators. So let's compare paths semantically instead. + pg_servicefile = pathlib.Path(pg_environ.pop("PGSERVICEFILE", "")) + pg_servicefile_exp = pathlib.Path(os.environ["RUNNER_TEMP"], "pgdata", "pg_service.conf") + assert pg_servicefile.resolve() == pg_servicefile_exp.resolve() + + if is_windows: + pg_environ_exp = { + "PGBIN": "", + "PGDATA": "", + "PGPASSWORD": "", + "PGROOT": "", + "PGUSER": "", + } + else: + pg_environ_exp = {} + assert pg_environ == pg_environ_exp + + +def test_server_version(connection: psycopg.Connection): + """Test that PostgreSQL's version is expected.""" + + server_version = connection.execute("SHOW SERVER_VERSION").fetchone()[0] + assert server_version.split(".")[0] == os.getenv("EXPECTED_SERVER_VERSION") + + +def test_server_ssl(connection: psycopg.Connection): + """Test that connection is SSL encrypted.""" + + expected = os.getenv("EXPECTED_SSL") == "true" + assert connection.info.pgconn.ssl_in_use is expected + + +def test_user_permissions(connection: psycopg.Connection): + """Test that a user has super/createdb permissions.""" + + with connection: + record = connection \ + .execute("SELECT usecreatedb, usesuper FROM pg_user WHERE usename = CURRENT_USER") \ + .fetchone() + assert record + + usecreatedb, usesuper = record + assert usecreatedb + assert usesuper + + +def test_user_create_insert_select(connection: psycopg.Connection): + """Test that a user has CRUD permissions in a database.""" + + table_name = "test_setup_postgres" + + with connection, connection.transaction(force_rollback=True): + records = connection \ + .execute(f"CREATE TABLE {table_name}(eggs INTEGER, rice VARCHAR)") \ + .execute(f"INSERT INTO {table_name}(eggs, rice) VALUES (1, '42')") \ + .execute(f"SELECT * FROM {table_name}") \ + .fetchall() + assert records == [(1, "42")] + + +def test_user_create_insert_non_ascii(connection: psycopg.Connection): + """Test that non-ASCII characters can be stored and fetched.""" + + table_name = "test_setup_postgres" + + with connection, connection.transaction(force_rollback=True): + records = connection \ + .execute(f"CREATE TABLE {table_name}(eggs INTEGER, rice VARCHAR)") \ + .execute(f"INSERT INTO {table_name}(eggs, rice) VALUES (1, 'Україна')") \ + .execute(f"INSERT INTO {table_name}(eggs, rice) VALUES (2, 'ウクライナ')") \ + .execute(f"SELECT * FROM {table_name}") \ + .fetchall() + assert records == [(1, "Україна"), (2, "ウクライナ")] + + +def test_user_create_drop_database(connection: psycopg.Connection): + """Test that a user has permissions to create databases.""" + + # CREATE/DROP DATABASE statements don't work within transactions, and with + # autocommit disabled transactions are created by psycopg automatically. + connection.autocommit = True + + database = "databas3" + connection.execute(f"CREATE DATABASE {database}") + connection.execute(f"DROP DATABASE {database}") + + +def test_user_create_drop_user( + connection: psycopg.Connection, + connection_factory: ConnectionFactory, + connection_uri: str +): + """Test that a user has permissions to create users.""" + + # CREATE/DROP USER statements don't work within transactions, and with + # autocommit disabled transactions are created by psycopg automatically. + connection.autocommit = True + + username = "us3rname" + password = "passw0rd" + database = "databas3" + + connection.execute(f"CREATE USER {username} WITH PASSWORD '{password}'") + connection.execute(f"CREATE DATABASE {database} WITH OWNER '{username}'") + + try: + # Smoke test that created user can successfully log-in and execute + # queries for its own database. + connection_uri = furl.furl( + connection_uri, username=username, password=password, path=database).url + test_user_create_insert_select(connection_factory(connection_uri)) + + finally: + connection.execute(f"DROP DATABASE {database}") + connection.execute(f"DROP USER {username}") + + +def test_client_applications( + connection_factory: ConnectionFactory, + service_name: str, + connection_uri: str, + monkeypatch: pytest.MonkeyPatch, +): + """Test that PostgreSQL client applications can be used.""" + + # Request connection parameters from the connection service file prepared + # by our action. + monkeypatch.setenv("PGSERVICE", service_name) + + username = "us3rname" + password = "passw0rd" + database = "databas3" + + subprocess.check_call(["createuser", username]) + subprocess.check_call(["createdb", "--owner", username, database]) + subprocess.check_call(["psql", "-c", f"ALTER USER {username} WITH PASSWORD '{password}'"]) + + try: + # Smoke test that created user can successfully log-in and execute + # queries for its own database. + connection_uri = furl.furl( + connection_uri, username=username, password=password, path=database).url + test_user_create_insert_select(connection_factory(connection_uri)) + + finally: + subprocess.check_call(["dropdb", database]) + subprocess.check_call(["dropuser", username]) + + +def test_libpq_applications(service_name: str, monkeypatch: pytest.MonkeyPatch): + """Test that libpq-using applications can be used.""" + + # Request connection parameters from the connection service file prepared + # by our action. + monkeypatch.setenv("PGSERVICE", service_name) + + with psycopg.connect() as connection: + assert connection \ + .execute("SELECT usename FROM pg_user WHERE usename = CURRENT_USER") \ + .fetchone() + + +def test_auth_wrong_username(connection_factory: ConnectionFactory, connection_uri: str): + """Test that wrong username is rejected!""" + + connection_furl = furl.furl(connection_uri, username="wrong") + + with pytest.raises(psycopg.OperationalError) as excinfo: + connection_factory(connection_furl.url) + + assert 'password authentication failed for user "wrong"' in str(excinfo.value) + + +def test_auth_wrong_password(connection_factory: ConnectionFactory, connection_uri: str): + """Test that wrong password is rejected!""" + + connection_furl = furl.furl(connection_uri, password="wrong") + username = connection_furl.username + + with pytest.raises(psycopg.OperationalError) as excinfo: + connection_factory(connection_furl.url) + + assert f'password authentication failed for user "{username}"' in str(excinfo.value) diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh new file mode 100755 index 000000000000..ef6077e4eb96 --- /dev/null +++ b/.github/entrypoint/wiki.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +hr='------------------------------------------------------------------------------------' + +gh gist clone 0ce5848f7ad62dc46dedfaa430069857 $1/identition/span1 +gh gist clone b32915925d9d365e2e9351f0c4ed786e $1/identition/span2 +gh gist clone 88d09204b2e5986237bd66d062406fde $1/identition/span3 +gh gist clone 8cab5e72d52ecb338a2f2187082a1699 $1/identition/span4 +gh gist clone 54600a56d20163c2da8910dd804ec406 $1/identition/span5 +gh gist clone f1af4317b619154719546e615aaa2155 $1/identition/span6 +gh gist clone 6c89c3b0f109e0ead561a452720d1ebf $1/identition/span7 +gh gist clone f21abd90f8d471390aad23d6ecc90d6d $1/identition/span8 +gh gist clone 6e2fcc2138be6fb68839a3ede32f0525 $1/identition/span9 +gh gist clone b541275ab7deda356feef32d600e44d8 $1/identition/span10 +gh gist clone 80c8098f16f3e6ca06893b17a02d910e $1/identition/span11 +gh gist clone 4ffc4d02579d5cfd336a553c6da2f267 $1/identition/span12 + +gh gist clone f78d4470250720fb18111165564d555f $1/exponentiation/span13 +gh gist clone 765ddc69e339079a5a64b56c1d46e00f $1/exponentiation/span14 +gh gist clone b9f901cda16e8a11dd24ee6b677ca288 $1/exponentiation/span15 +gh gist clone dc30497160f3389546d177da901537d9 $1/exponentiation/span16 +gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 +gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 + +find $1 -type f -name "README.md" -exec rm -rf {} \; +rm -rf ${RUNNER_TEMP//\\//}/README.md + +mv -f ${RUNNER_TEMP//\\//}/wikidir/README.md ${RUNNER_TEMP//\\//}/README.md +cp -R ${RUNNER_TEMP//\\//}/wikidir/* $1 + +rm -rf ${RUNNER_TEMP//\\//}/spin.txt && touch ${RUNNER_TEMP//\\//}/spin.txt +find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done + +rm -rf ${RUNNER_TEMP//\\//}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP//\\//}/Sidebar.md +sed -i 's/0. \[\[//g' ${RUNNER_TEMP//\\//}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP//\\//}/Sidebar.md + +find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' + +echo -e "\n$hr\nSPIN\n$hr" +cat ${RUNNER_TEMP//\\//}/spin.txt diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md deleted file mode 100644 index e2ae0966cda5..000000000000 --- a/.github/pull_request_template.md +++ /dev/null @@ -1,21 +0,0 @@ -### Describe your change: - - - -* [ ] Add an algorithm? -* [ ] Fix a bug or typo in an existing algorithm? -* [ ] Add or change doctests? -- Note: Please avoid changing both code and tests in a single pull request. -* [ ] Documentation change? - -### Checklist: -* [ ] I have read [CONTRIBUTING.md](https://github.com/TheAlgorithms/Python/blob/master/CONTRIBUTING.md). -* [ ] This pull request is all my own work -- I have not plagiarized. -* [ ] I know that pull requests will not be merged if they fail the automated tests. -* [ ] This PR only changes one algorithm file. To ease review, please open separate PRs for separate algorithms. -* [ ] All new Python files are placed inside an existing directory. -* [ ] All filenames are in all lowercase characters with no spaces or dashes. -* [ ] All functions and variable names follow Python naming conventions. -* [ ] All function parameters and return values are annotated with Python [type hints](https://docs.python.org/3/library/typing.html). -* [ ] All functions have [doctests](https://docs.python.org/3/library/doctest.html) that pass the automated testing. -* [ ] All new algorithms include at least one URL that points to Wikipedia or another similar explanation. -* [ ] If this pull request resolves one or more open issues then the description above includes the issue number(s) with a [closing keyword](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue): "Fixes #ISSUE-NUMBER". diff --git a/.github/stale.yml b/.github/stale.yml deleted file mode 100644 index 0939e1f223ff..000000000000 --- a/.github/stale.yml +++ /dev/null @@ -1,63 +0,0 @@ -# Configuration for probot-stale - https://github.com/probot/stale - -# Number of days of inactivity before an Issue or Pull Request becomes stale -daysUntilStale: 30 - -# Number of days of inactivity before an Issue or Pull Request with the stale label is closed. -# Set to false to disable. If disabled, issues still need to be closed manually, but will remain marked as stale. -daysUntilClose: 7 - -# Only issues or pull requests with all of these labels are check if stale. Defaults to `[]` (disabled) -onlyLabels: [] - -# Issues or Pull Requests with these labels will never be considered stale. Set to `[]` to disable -exemptLabels: - - "Status: on hold" - -# Set to true to ignore issues in a project (defaults to false) -exemptProjects: false - -# Set to true to ignore issues in a milestone (defaults to false) -exemptMilestones: false - -# Set to true to ignore issues with an assignee (defaults to false) -exemptAssignees: false - -# Label to use when marking as stale -staleLabel: stale - -# Limit the number of actions per hour, from 1-30. Default is 30 -limitPerRun: 5 - -# Comment to post when removing the stale label. -# unmarkComment: > -# Your comment here. - -# Optionally, specify configuration settings that are specific to just 'issues' or 'pulls': -pulls: - # Comment to post when marking as stale. Set to `false` to disable - markComment: > - This pull request has been automatically marked as stale because it has not had - recent activity. It will be closed if no further activity occurs. Thank you - for your contributions. - - # Comment to post when closing a stale Pull Request. - closeComment: > - Please reopen this pull request once you commit the changes requested - or make improvements on the code. If this is not the case and you need - some help, feel free to seek help from our [Gitter](https://gitter.im/TheAlgorithms/community) - or ping one of the reviewers. Thank you for your contributions! - -issues: - # Comment to post when marking as stale. Set to `false` to disable - markComment: > - This issue has been automatically marked as stale because it has not had - recent activity. It will be closed if no further activity occurs. Thank you - for your contributions. - - # Comment to post when closing a stale Issue. - closeComment: > - Please reopen this issue once you add more information and updates here. - If this is not the case and you need some help, feel free to seek help - from our [Gitter](https://gitter.im/TheAlgorithms/community) or ping one of the - reviewers. Thank you for your contributions! diff --git a/.github/templates/CODE_OF_CONDUCT.md b/.github/templates/CODE_OF_CONDUCT.md new file mode 100644 index 000000000000..fa1504469b99 --- /dev/null +++ b/.github/templates/CODE_OF_CONDUCT.md @@ -0,0 +1,49 @@ +# Contributor Covenant Code of Conduct + +## Our Pledge + +In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation. + +## Our Standards + +Examples of behavior that contributes to creating a positive environment include: + +* Using welcoming and inclusive language +* Being respectful of differing viewpoints and experiences +* Gracefully accepting constructive criticism +* Focusing on what is best for the community +* Showing empathy towards other community members + +Examples of unacceptable behavior by participants include: + +* The use of sexualized language or imagery and unwelcome sexual attention or advances +* Trolling, insulting/derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or electronic address, without explicit permission +* Other conduct which could reasonably be considered inappropriate in a professional setting + +## Our Responsibilities + +Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior. + +Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. + +## Scope + +This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at hello+community@mirumee.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately. + +Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version] + +[homepage]: http://contributor-covenant.org +[version]: http://contributor-covenant.org/version/1/4/ + +For answers to common questions about this code of conduct, see +https://www.contributor-covenant.org/faq diff --git a/.github/templates/CONTRIBUTING.md b/.github/templates/CONTRIBUTING.md new file mode 100644 index 000000000000..004fb79d116a --- /dev/null +++ b/.github/templates/CONTRIBUTING.md @@ -0,0 +1,16 @@ +# Installation + +1. Install a full [Ruby development environment](https://jekyllrb.com/docs/installation/) +2. Change into your new directory + +```sh +make +``` + +4. Build the site and make it available on a local server + +```sh +make server +``` + +5. Browse to [http://localhost:4000](http://localhost:4000) diff --git a/.github/templates/PULL_REQUEST_TEMPLATE.md b/.github/templates/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 000000000000..9a76d57fac34 --- /dev/null +++ b/.github/templates/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,29 @@ +### Summary + +Main Code: []() +``` + +``` + +### Checklist + +1. [ ] Privileged views and APIs are guarded by proper permission checks. +1. [ ] All visible strings are translated with proper context. +1. [ ] All data-formatting is locale-aware (dates, numbers, and so on). +1. [ ] Database queries are optimized and the number of queries is constant. +1. [ ] Database migration files are up to date. +1. [ ] The changes are tested. +1. [ ] The code is documented (docstrings, project documentation). +1. [ ] GraphQL schema and type definitions are up to date. +1. [ ] Changes are mentioned in the changelog. + +### Reference + + +- []() +- []() +- []() + +### Screenshots + diff --git a/.github/templates/bug_report.md b/.github/templates/bug_report.md new file mode 100644 index 000000000000..2746c69591a1 --- /dev/null +++ b/.github/templates/bug_report.md @@ -0,0 +1,32 @@ +--- +name: Bug report +about: Create a bug report +title: '' +labels: bug, needs triage +assignees: '' + +--- + +**Description:** +A clear and concise description of what the bug is. + +**Task version:** +Specify the task version + +**Platform:** +- [ ] Ubuntu +- [ ] macOS +- [ ] Windows + +**Runner type:** +- [ ] Hosted +- [ ] Self-hosted + +**Repro steps:** +A description with steps to reproduce the issue. If your have a public example or repo to share, please provide the link. + +**Expected behavior:** +A description of what you expected to happen. + +**Actual behavior:** +A description of what is actually happening. \ No newline at end of file diff --git a/.github/templates/bug_report.yml b/.github/templates/bug_report.yml new file mode 100644 index 000000000000..b412302c8877 --- /dev/null +++ b/.github/templates/bug_report.yml @@ -0,0 +1,58 @@ +name: 🐞 Bug Report +description: Report an issue +labels: ["bug 🐞"] +body: + - type: markdown + attributes: + value: | + :star2: Thanks for taking the time to report this issue! :star2: + + Please search through [other recent issues](https://github.com/iree-org/iree/issues) to see if your report overlaps with an existing issue. + - type: textarea + id: what-happened + attributes: + label: What happened? + description: | + Also tell us, what did you expect to happen? + + For compiler errors, we like to see the MLIR output leading up to whatever error occurred. + placeholder: Tell us what you see! Please also copy/paste or link to any relevant log output. + validations: + required: true + - type: textarea + id: reproducing + attributes: + label: Steps to reproduce your issue + description: | + For compiler errors, we like to see the input IR and the commands that you ran. + value: | + 1. Go to '...' + 2. Click on '....' + 3. Scroll down to '....' + 4. See error + - type: dropdown + id: component + attributes: + label: What component(s) does this issue relate to? + description: | + If you're not sure, that's okay! This helps us route issues to the right developers. + multiple: true + options: + - Frontends + - MLIR + - Python + - Compiler + - Runtime + - Other + - type: textarea + id: version + attributes: + label: Version information + description: What version of our software are you running? + placeholder: This could be a git commit hash or release version number. + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context about the problem here. + placeholder: For example, did what you were trying work before? diff --git a/.github/templates/config.yml b/.github/templates/config.yml new file mode 100644 index 000000000000..4616203c27f1 --- /dev/null +++ b/.github/templates/config.yml @@ -0,0 +1,14 @@ +blank_issues_enabled: true +contact_links: + - name: 📄 Blank Issue + url: https://github.com/iree-org/iree/issues/new + about: If you know what you're doing (especially collaborators) + - name: 📧 Send an email to our mailing list + url: https://lists.lfaidata.foundation/g/iree-technical-discussion + about: For asynchronous discussion + - name: 🗣 Start a discussion on GitHub + url: https://github.com/iree-org/iree/discussions + about: An alternative platform for asynchronous discussion + - name: 💬 Join us on Discord + url: https://discord.gg/wEWh6Z9nMU + about: For realtime communication with the team and collaborators diff --git a/.github/templates/feature_request.md b/.github/templates/feature_request.md new file mode 100644 index 000000000000..2c9d47d9c179 --- /dev/null +++ b/.github/templates/feature_request.md @@ -0,0 +1,16 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: feature request, needs triage +assignees: '' +--- + +**Description:** +Describe your proposal. + +**Justification:** +Justification or a use case for your proposal. + +**Are you willing to submit a PR?** + \ No newline at end of file diff --git a/.github/templates/feature_request.yml b/.github/templates/feature_request.yml new file mode 100644 index 000000000000..cf060ebc8a33 --- /dev/null +++ b/.github/templates/feature_request.yml @@ -0,0 +1,38 @@ +name: ➕ Feature Request +description: Ask for a new feature to be added +labels: ["enhancement ➕"] +body: + - type: markdown + attributes: + value: | + :star2: Thanks for taking the time to file this request! :star2: + + We also work closely with upstream communities like the MLIR project, which uses the [MLIR Discourse forum](https://discourse.llvm.org/c/mlir/31) for coordination. + - type: textarea + id: overview + attributes: + label: Request description + description: | + What would you like added or changed? + validations: + required: true + - type: dropdown + id: component + attributes: + label: What component(s) does this issue relate to? + description: | + If you're not sure, that's okay! This helps us route issues to the right developers. + multiple: true + options: + - Frontends + - MLIR + - Python + - Compiler + - Runtime + - Other + - type: textarea + id: additional-context + attributes: + label: Additional context + description: Add any other context about your request here. + placeholder: For example, is a similar feature already implemented? diff --git a/.github/templates/mapping-errors.md b/.github/templates/mapping-errors.md new file mode 100644 index 000000000000..782ec3038f51 --- /dev/null +++ b/.github/templates/mapping-errors.md @@ -0,0 +1,14 @@ +--- +name: Mapping errors +about: For errors in the mapping data (typos, wrong info, etc.) +title: '' +labels: bug, triage +assignees: '' + +--- + +**Minecraft version:** *[What version of Minecraft does the mappings belong to, ex. 1.16.5]* +**Mappings version:** *[What version of the mappings does the error appear in, ex. 2020.01.01]* + +*[Describe the error from the mapping data. Use Official/Mojmap names preferentially.]* +*[It is recommended to also suggest a solution for the error, or even contribute the solution yourself. If you plan on contributing a fix, please note it.]* diff --git a/.github/templates/other-issues.md b/.github/templates/other-issues.md new file mode 100644 index 000000000000..ecc31fc1f807 --- /dev/null +++ b/.github/templates/other-issues.md @@ -0,0 +1,10 @@ +--- +name: Other issues +about: For any other issues not covered by other templates. +title: '' +labels: triage +assignees: '' + +--- + +*[If your topic fits an already existing template, please use those for the proper labels to be applied.]* diff --git a/.github/templates/standards-change-proposal.md b/.github/templates/standards-change-proposal.md new file mode 100644 index 000000000000..10a81b4c929f --- /dev/null +++ b/.github/templates/standards-change-proposal.md @@ -0,0 +1,10 @@ +--- +name: Standards change proposal +about: Discussion for changes to the mapping standards or processes. +title: '' +labels: meta, triage +assignees: '' + +--- + +*[Elaborate on what change should be made to the mapping standards or processes, and why.]* diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml new file mode 100644 index 000000000000..532bad526be0 --- /dev/null +++ b/.github/workflows/main.yml @@ -0,0 +1,170 @@ +# 💎 Main Workflow: eQuantum MEC30 +name: COmpendium of RElations – Modulo 6 + +on: + #push: + workflow_run: + types: [completed] + workflows: ["pages-build-deployment"] + +# 🕸️ Allow concurrent +concurrency: + group: "pages" + cancel-in-progress: true + +# ⚠️ Set token permissions +permissions: write-all + +# 🕸️ Global Environment +env: + GH_TOKEN: ${{ secrets.ACCESS_TOKEN }} + PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} + TARGET_REPOSITORY: ${{ vars.TARGET_REPOSITORY }} + +jobs: + lexering: + strategy: + max-parallel: 1 + fail-fast: true + matrix: + config: + - { os: ubuntu-latest, id: 1 } + - { os: macos-latest, id: 2 } + env: + ID: ${{ matrix.config.id }} + DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' + + # 🔧 Commutation relations + runs-on: ${{ matrix.config.os }} + continue-on-error: ${{ matrix.config.id != 1 && false || true }} + + steps: + - name: 📂 Checkout + uses: actions/checkout@v4 + with: + submodules: recursive + token: ${{ secrets.ACCESS_TOKEN }} + repository: ${{ env.ID != 1 && env.DOC || '' }} + + - name: ⚙️ Build Maps + uses: eq19/maps@v8 + id: build-parser + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + rerun_runner: ${{ vars.RERUN_RUNNER }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + docker_hub_token: ${{ secrets.DOCKER_HUB_TOKEN }} + docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }} + initiate_pauli_sum: ${{ toJSON(steps.set-lexering.outputs) }} + + - name: 🪂 Feed Mapping + uses: eq19/feed@v6 + id: feed-mapping + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + chat_id: ${{ secrets.MESSAGE_API }} + rerun_runner: ${{ vars.RERUN_RUNNER }} + bot_token: ${{ secrets.MONITOR_TOKEN }} + logs_token: ${{ secrets.WARNING_TOKEN }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + docker_hub_token: ${{ secrets.DOCKER_HUB_TOKEN }} + docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }} + initiate_pauli_sum: ${{ toJSON(steps.build-parser.outputs) }} + + - name: 🏃‍♂️ Lexering Runner + uses: eq19/lexer@v4 + id: set-lexering + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + config: ${{ secrets.CONFIG_JSON }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + docker_hub_token: ${{ secrets.DOCKER_HUB_TOKEN }} + docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }} + initiate_pauli_sum: ${{ toJSON(steps.feed-mapping.outputs) }} + + - name: 📋 Generate Outputs + id: gen_outputs + shell: bash + run: | + id=${{ matrix.config.id }} + output=${{ matrix.config.id == 1 && + steps.feed-mapping.outputs.addition || + steps.set-lexering.outputs.multiplication }} + echo "output_${id}=${output}" >> "${GITHUB_OUTPUT}" + + outputs: + # ♻️ Simulate Gell-Mann Matrices + target: ${{ steps.gen_outputs.outputs.output_1 }} + address: ${{ steps.gen_outputs.outputs.output_2 }} + function: ${{ steps.set-lexering.outputs.function }} + + parsering: + strategy: + matrix: + config: + - { os: self-hosted, id: 3 } + - { os: windows-latest, id: 4 } + + # 💢 Completeness relations + max-parallel: 1 + fail-fast: true + + env: + IP_ADDRESS: '${{ needs.lexering.outputs.address }}' + PASS: ${{ contains(needs.lexering.result, 'success') }} + + ## ✔️ Completeness status + needs: lexering + runs-on: ${{ matrix.config.os }} + #if: ${{ (needs.lexering.result == 'success') }} + continue-on-error: ${{ matrix.config.id == 3 && true || false }} + + steps: + - name: 📂 Checkout + uses: actions/checkout@v4 + with: + submodules: recursive + token: ${{ secrets.ACCESS_TOKEN }} + repository: ${{ env.TARGET_REPOSITORY }} + ref: ${{ matrix.config.id == 3 && 'gh-source' || 'gh-pages' }} + + - name: 🚀 Setup Parser + if: ${{ env.PASS == 'true' }} + uses: eq19/parser@v2 + id: set-parser + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + pauli_sum: ${{ needs.lexering.outputs.function }} + failure_status: ${{ contains(steps.*.conclusion, 'failure') || + (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} + + - name: 🌀 Evaluate Syntax + if: ${{ env.PASS == 'true' }} + uses: eq19/syntax@v2 + id: eval-syntax + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + api_key: ${{ secrets.ACCESS_API }} + api_secret: ${{ secrets.ACCESS_KEY }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + pauli_sum: ${{ toJSON(steps.set-parser.outputs) }} + failure_status: ${{ contains(steps.*.conclusion, 'failure') || + (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} + + - name: ⚧️ Set Grammar Rules + uses: eq19/grammar@v1 + id: grammar-rules + if: always() + with: + id: ${{ matrix.config.id }} + token: ${{ secrets.ACCESS_TOKEN }} + credentials: ${{ secrets.GCP_CREDENTIALS }} + pauli_sum: ${{ toJSON(steps.eval-syntax.outputs) }} + failure_status: ${{ contains(steps.*.conclusion, 'failure') || + (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} diff --git a/.github/workflows/reserves/__config.yml b/.github/workflows/reserves/__config.yml new file mode 100644 index 000000000000..8920eb42c10e --- /dev/null +++ b/.github/workflows/reserves/__config.yml @@ -0,0 +1,239 @@ +# General Settings +# https://jekyllrb.com/docs/configuration/ +# https://jekyllrb.com/docs/configuration/options/ +# https://learn.cloudcannon.com/jekyll-cheat-sheet/ + +input_types: +- text +- date +- email +- radio +- submit +- number +- checkbox +- textarea + +form_engines: +- formspree +- formbucket +- cloudcannon + +debug: + dist: false + compress: true + shortcodes: true + +# Repository Settings +# https://blog.webjeda.com/jekyll-redirect/ +# https://github.com/ethanresnick/json-api-1/blob/gh-pages/_config.yml +# https://github.com/daattali/beautiful-jekyll/blob/master/_config.yml + +include: + - app.js + - feed.xml + - feed.json + - .htaccess + +exclude: + - src/ + - docs/ + - dist/ + - assets/ + - .babelrc + - Gemfile + - Makefile + - yarn.lock + - Dockerfile + - update.sh + - Gemfile.lock + - Gemfile.lock + - vendor/ruby/ + - vendor/gems/ + - vendor/cache/ + - vendor/bundle/ + - package-lock.json + - webpack.config.js + - docker_compose.yml + - requirements.txt + - node_modules/ + - package.json + - package-lock.json + - webpack.config.js + - jekyll-rtd-theme.gemspec + +encoding: utf-8 +livereload: true +limit_posts: 1000 +collections_dir: maps +timezone: Asia/Jakarta +remote_theme: eQ19/parser +excerpt_separator: + +# Plugins Setting +# https://pages.github.com/versions/ +# https://github.com/jekyll/jekyll-feed +# https://en.wikipedia.org/wiki/CoffeeScript +# https://github.com/jekyll/jekyll-redirect-from + +plugins: + - jemoji + - jekyll-gist + - jekyll-feed + - jekyll-avatar + - jekyll-sitemap + - jekyll-paginate + - jekyll-mentions + - jekyll-remote-theme + - jekyll-coffeescript + - jekyll-readme-index + - jekyll-redirect-from + - jekyll-relative-links + - jekyll-default-layout + - jekyll-github-metadata + - jekyll-titles-from-headings + - jekyll-optional-front-matter + +# Jekyll Tag Pages Generator +# https://jekyllrb.com/docs/liquid/tags/ +# https://github.com/pattex/jekyll-tagging +# https://jekyllrb.com/docs/plugins/generators/ + +# tag_page_dir: tagz +# tag_feed_layout: tag_feed +# tag_page_layout: tag_page + +# Default Setting +# https://stackoverflow.com/a/49428493 +# https://jekyllrb.com/docs/collections/ +# https://jekyllrb.com/docs/configuration/front-matter-defaults + +defaults: + - + scope: + path: "" + type: feeds + values: + layout: feeds + - + scope: + path: "" + type: maps + values: + layout: redirect + - + scope: + path: "" + type: categories + values: + layout: base + permalink: /sitemap/:category.xml + +# Permalinks +# https://jekyllrb.com/docs/permalinks/ +# https://stackoverflow.com/a/28754526/4058484 +# https://github.com/NiklasEi/jekyll_custom_permalink +# https://github.com/avillafiorita/jekyll-datapage_gen + +collections: + maps: + output: true + permalink: /sitemap/index_:categories:output_ext + feeds: + output: true + permalink: /sitemap/index_:categories:output_ext + posts: + output: true + permalink: /sitemap/:year/:y_day/:month/:title:output_ext + +# Feed and Submodul Setting +# https://stackoverflow.com/a/55885186/4058484 +# https://github.com/jekyll/jekyll-feed/issues/2 +# https://www.w3schools.com/xml/xsl_elementref.asp +# https://github.com/jekyll/jekyll-feed/issues/207#issuecomment-787490785 + +feed: + tags: true + posts_limit: 1000 + excerpt_only: true + collections: + maps: + path: /leads.xml # https://eq19.github.io/grammar/leads.xml + feeds: + categories: + - main # https://eq19.github.io/grammar/feed/feeds/main.xml + - user # https://eq19.github.io/grammar/feed/feeds/user.xml + - diagram # https://eq19.github.io/grammar/feed/feeds/diagram.xml + - section # https://eq19.github.io/grammar/feed/feeds/section.xml + path: /feeds.xml # https://eq19.github.io/grammar/feeds.xml + posts: + path: /posts.xml # https://eq19.github.io/grammar/posts.xml + +# Kramdown Options +# https://kramdown.gettalong.org/options.html +# https://jekyllrb.com/docs/configuration/markdown/ + +highlighter: rouge +markdown: kramdown + +kramdown: + html_to_native: true + syntax_highlighter: pygments + math_engine_opts configuration: mathjax + +excerpt_separator: + +# Custom Headers +# https://shopify.github.io/liquid/ +# https://jekyllrb.com/docs/configuration/webrick/ +# https://en.wikipedia.org/wiki/Content_Security_Policy + +google: + gtag: GTM-PGWT4RS + #adsense: ca-pub-3677197599836940 +webrick: + headers: + Content-Security-Policy: default-src 'none'; script-src 'self'; connect-src 'self'; img-src 'self'; style-src 'self'; + #My-Other-Header: My-Other-Value +#sumo: 2b0c75104f236f75954a2d200c80651a1ebf311e7f2c868ac12d7284ecd4fbd4 + +# Tags Setting +# http://codinfox.github.io/blog/tags/ +# https://longqian.me/2017/02/09/github-jekyll-tag/ +# https://github.com/chetabahana/chetabahana.github.io/tags + +case_labels: +- Buka Toko +- Stok Barang +- Merchant Center +- Peluang Terbaik +- Portfolio +- Network + +flow_labels: +- Optimasi +- Cloud-Task-API +- Cloud-Site-API +- Google-Ads-API +- Google-Trend-API +- eShop SEO + +# Site Setting +# https://jekyllrb.com/docs/variables/ +# https://stackoverflow.com/a/55094996/4058484 +# https://stackoverflow.com/a/17260660/4058484 +# https://stackoverflow.com/a/30828197/4058484 + +basedir: _feeds +organization: marketleader +repository: eq19/eq19.github.io + +# Site Metadata +# https://docs.github.com/en/pages +# https://help.github.com/articles/repository-metadata-on-github-pages/ +# https://www.google.com/search?q=tortoisegit+keep+asking+password+site%3Astackoverflow.com +# https://stackoverflow.com/questions/3490505/is-there-a-notepad-with-github-plugin/41884020 +# https://stackoverflow.com/questions/14000173/tortoisegit-save-user-authentication-credentials + +slogan: eQuantum Project | A Unique Mapping System +disclaimer: The definite key to identify whether you use our concept is when there a kind of development item lies a unified assignment in hexagonal form by six (6) corresponding sets while each sets pick a combination of six (6) routes with a pairing of six (6) to six (6) of all channels. +license: Apache License, Version 2.0 diff --git a/.github/workflows/reserves/account/blog-posts.yml b/.github/workflows/reserves/account/blog-posts.yml new file mode 100644 index 000000000000..9b1490db80b1 --- /dev/null +++ b/.github/workflows/reserves/account/blog-posts.yml @@ -0,0 +1,27 @@ +name: Blog Posts + +on: workflow_dispatch + +jobs: + update-posts: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Update post list + run: | + sleep 1m + curl -LO https://github.com/ouuan/ouuan.moe/raw/dist/feed.json + node src/listPosts.js + rm feed.json + - name: Commit changes + run: | + git config --local user.email "github-actions[bot]@users.noreply.github.com" + git config --local user.name "github-actions[bot]" + git add -A + git diff-index --quiet HEAD || git commit -m "Update blog posts" + - name: Pull changes + run: git pull -r + - name: Push changes + uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df + with: + github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/account/star-fork.yml b/.github/workflows/reserves/account/star-fork.yml new file mode 100644 index 000000000000..1587d68660d5 --- /dev/null +++ b/.github/workflows/reserves/account/star-fork.yml @@ -0,0 +1,23 @@ +name: This repo has x stars y forks + +on: + push: + branches: + - master + fork: + watch: + types: + - started + schedule: + - cron: '0 * * * *' + workflow_dispatch: + +jobs: + update: + runs-on: ubuntu-latest + steps: + - name: This repo has x stars y forks + uses: ouuan/This-repo-has-x-stars-y-forks-action@v3 + with: + token: ${{ secrets.STAR_FORK_TOKEN }} + template: "A profile README with stars and forks 🌟" diff --git a/.github/workflows/reserves/account/starred.yml b/.github/workflows/reserves/account/starred.yml new file mode 100644 index 000000000000..16dc3bcc318e --- /dev/null +++ b/.github/workflows/reserves/account/starred.yml @@ -0,0 +1,32 @@ +name: Update awesome-stars + +on: + schedule: + - cron: 0 18 * * * + workflow_dispatch: + +jobs: + awesome-stars: + name: update awesome-stars + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: 3.8 + - run: pip install starred + - name: Update star list + run: starred --username ${{ github.repository_owner }} --sort --token ${{ secrets.GITHUB_TOKEN }} > AWESOME-STARS.md + - name: Commit changes + run: | + git config --local user.email "github-actions[bot]@users.noreply.github.com" + git config --local user.name "github-actions[bot]" + git add -A + git diff-index --quiet HEAD || git commit -m "Update awesome stars" + - name: Pull changes + run: git pull -r + - name: Push changes + uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df + with: + github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/account/topFollowers.yml b/.github/workflows/reserves/account/topFollowers.yml new file mode 100644 index 000000000000..6e0ebdb59ff3 --- /dev/null +++ b/.github/workflows/reserves/account/topFollowers.yml @@ -0,0 +1,35 @@ +name: Get Top Followers + +on: + push: + branches: + - master + schedule: + - cron: '0 22 * * *' + workflow_dispatch: + +jobs: + top-followers: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Setup python + uses: actions/setup-python@v5 + with: + python-version: 3.8 + - name: Install requests + run: pip install requests + - name: Update README + run: python src/getTopFollowers.py ${{ github.repository_owner }} ${{ secrets.GITHUB_TOKEN }} README.md + - name: Commit changes + run: | + git config --local user.email "github-actions[bot]@users.noreply.github.com" + git config --local user.name "github-actions[bot]" + git add -A + git diff-index --quiet HEAD || git commit -m "Update top followers" + - name: Pull changes + run: git pull -r + - name: Push changes + uses: ad-m/github-push-action@d91a481090679876dfc4178fef17f286781251df + with: + github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/account/wakatime.yml b/.github/workflows/reserves/account/wakatime.yml new file mode 100644 index 000000000000..491d3af25eb9 --- /dev/null +++ b/.github/workflows/reserves/account/wakatime.yml @@ -0,0 +1,19 @@ +name: WakaTime Readme + +on: + schedule: + - cron: '0 20 * * *' + workflow_dispatch: + +jobs: + update-readme: + runs-on: ubuntu-latest + steps: + - uses: athul/waka-readme@8539bedb8277066fc9fe1b47d1f9b108eb866494 + with: + API_BASE_URL: ${{ secrets.WAKAPI_BASE_URL }} + WAKATIME_API_KEY: ${{ secrets.WAKAPI_API_KEY }} + COMMIT_MESSAGE: Update Wakapi stats + BLOCKS: "⬜🟨🟩" + TIME_RANGE: last_30_days + SHOW_TIME: true diff --git a/.github/workflows/build.yml b/.github/workflows/reserves/algorithms/build.yml similarity index 96% rename from .github/workflows/build.yml rename to .github/workflows/reserves/algorithms/build.yml index 62829b2b45a5..a6f308715cc2 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/reserves/algorithms/build.yml @@ -10,7 +10,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v5 + - uses: astral-sh/setup-uv@v4 with: enable-cache: true cache-dependency-glob: uv.lock diff --git a/.github/workflows/directory_writer.yml b/.github/workflows/reserves/algorithms/directory_writer.yml similarity index 100% rename from .github/workflows/directory_writer.yml rename to .github/workflows/reserves/algorithms/directory_writer.yml diff --git a/.github/workflows/project_euler.yml b/.github/workflows/reserves/algorithms/project_euler.yml similarity index 93% rename from .github/workflows/project_euler.yml rename to .github/workflows/reserves/algorithms/project_euler.yml index 8d51ad8850cf..84c55335451e 100644 --- a/.github/workflows/project_euler.yml +++ b/.github/workflows/reserves/algorithms/project_euler.yml @@ -15,7 +15,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v5 + - uses: astral-sh/setup-uv@v4 - uses: actions/setup-python@v5 with: python-version: 3.x @@ -25,7 +25,7 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v5 + - uses: astral-sh/setup-uv@v4 - uses: actions/setup-python@v5 with: python-version: 3.x diff --git a/.github/workflows/ruff.yml b/.github/workflows/reserves/algorithms/ruff.yml similarity index 87% rename from .github/workflows/ruff.yml rename to .github/workflows/reserves/algorithms/ruff.yml index cfe127b3521f..2c6f92fcf7bf 100644 --- a/.github/workflows/ruff.yml +++ b/.github/workflows/reserves/algorithms/ruff.yml @@ -12,5 +12,5 @@ jobs: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v5 + - uses: astral-sh/setup-uv@v4 - run: uvx ruff check --output-format=github . diff --git a/.github/workflows/sphinx.yml b/.github/workflows/reserves/algorithms/sphinx.yml similarity index 95% rename from .github/workflows/sphinx.yml rename to .github/workflows/reserves/algorithms/sphinx.yml index 16ff284a74f2..e3e2ce81a95d 100644 --- a/.github/workflows/sphinx.yml +++ b/.github/workflows/reserves/algorithms/sphinx.yml @@ -23,10 +23,10 @@ concurrency: jobs: build_docs: - runs-on: ubuntu-24.04-arm + runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - - uses: astral-sh/setup-uv@v5 + - uses: astral-sh/setup-uv@v4 - uses: actions/setup-python@v5 with: python-version: 3.13 diff --git a/.github/workflows/reserves/basic-validation.yml b/.github/workflows/reserves/basic-validation.yml new file mode 100644 index 000000000000..a4b1c0f26c5f --- /dev/null +++ b/.github/workflows/reserves/basic-validation.yml @@ -0,0 +1,17 @@ +name: Basic validation + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + call-basic-validation: + name: Basic validation + uses: actions/reusable-workflows/.github/workflows/basic-validation.yml@main diff --git a/.github/workflows/reserves/build.yml b/.github/workflows/reserves/build.yml new file mode 100644 index 000000000000..b02df6d5dfe0 --- /dev/null +++ b/.github/workflows/reserves/build.yml @@ -0,0 +1,111 @@ +name: build + +on: + push: + branches: [ master ] + pull_request: + branches: [ master ] + +permissions: + contents: read + +jobs: + + build: + + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Run Python unit tests + run: python3 -u -m unittest tests/tests.py + + - name: Verify that the Docker image for the action builds + run: docker build . --file Dockerfile + + - name: Create new uncommitted html file for testing + run: touch tests/uncommitted.html + + - name: Integration test 1 + id: integration + uses: ./ + with: + path-to-root: tests + base-url-path: https://TESTING.FAKE.WEB.ADDRESS.TESTING/ + + - name: Output stats test 1 + run: | + echo "sitemap-path = ${{ steps.integration.outputs.sitemap-path }}" + echo "url-count = ${{ steps.integration.outputs.url-count }}" + echo "excluded-count = ${{ steps.integration.outputs.excluded-count }}" + + - name: Integration test 2 + id: integration2 + uses: ./ + with: + path-to-root: tests + base-url-path: https://TESTING.FAKE.WEB.ADDRESS.TESTING/ + sitemap-format: txt + additional-extensions: docx pptx + + - name: Output stats test 2 + run: | + echo "sitemap-path = ${{ steps.integration2.outputs.sitemap-path }}" + echo "url-count = ${{ steps.integration2.outputs.url-count }}" + echo "excluded-count = ${{ steps.integration2.outputs.excluded-count }}" + + - name: Integration test 3 + id: integration3 + uses: ./ + with: + path-to-root: tests/subdir + base-url-path: https://TESTING.FAKE.WEB.ADDRESS.TESTING/ + drop-html-extension: true + + - name: Output stats test 3 + run: | + echo "sitemap-path = ${{ steps.integration3.outputs.sitemap-path }}" + echo "url-count = ${{ steps.integration3.outputs.url-count }}" + echo "excluded-count = ${{ steps.integration3.outputs.excluded-count }}" + + - name: Integration test 4 + id: integration4 + uses: ./ + with: + path-to-root: tests/subdir + base-url-path: https://TESTING.FAKE.WEB.ADDRESS.TESTING/ + sitemap-format: txt + additional-extensions: docx pptx + drop-html-extension: true + + - name: Output stats test 4 + run: | + echo "sitemap-path = ${{ steps.integration4.outputs.sitemap-path }}" + echo "url-count = ${{ steps.integration4.outputs.url-count }}" + echo "excluded-count = ${{ steps.integration4.outputs.excluded-count }}" + + - name: Integration test 5 + id: integration5 + uses: ./ + with: + path-to-root: tests/exclude + base-url-path: https://TESTING.FAKE.WEB.ADDRESS.TESTING/ + exclude-paths: /excludeSubDir /exc1.html /subdir/exc4.html + + - name: Output stats test 5 + run: | + echo "sitemap-path = ${{ steps.integration5.outputs.sitemap-path }}" + echo "url-count = ${{ steps.integration5.outputs.url-count }}" + echo "excluded-count = ${{ steps.integration5.outputs.excluded-count }}" + + - name: Verify integration test results + run: python3 -u -B -m unittest tests/integration.py + diff --git a/.github/workflows/reserves/check-dist.yml b/.github/workflows/reserves/check-dist.yml new file mode 100644 index 000000000000..c9522913185d --- /dev/null +++ b/.github/workflows/reserves/check-dist.yml @@ -0,0 +1,17 @@ +name: Check dist/ + +on: + push: + branches: + - main + paths-ignore: + - '**.md' + pull_request: + paths-ignore: + - '**.md' + workflow_dispatch: + +jobs: + call-check-dist: + name: Check dist/ + uses: actions/reusable-workflows/.github/workflows/check-dist.yml@main diff --git a/.github/workflows/reserves/ci.yml b/.github/workflows/reserves/ci.yml new file mode 100644 index 000000000000..940f7a924514 --- /dev/null +++ b/.github/workflows/reserves/ci.yml @@ -0,0 +1,83 @@ +name: CI + +on: + push: + branches: [develop] + pull_request: + branches: [develop] + +jobs: + checkout: + runs-on: ubuntu-latest + outputs: + version: ${{ steps.matching.outputs.version }} + steps: + - uses: actions/checkout@main + - uses: rundocs/checkout@master + - name: Match the commit message whether to Release! + run: | + MESSAGE=$(git log -1 --format="%s") + if [[ ${MESSAGE} =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then + echo "::set-output name=version::${MESSAGE:1}" + fi + id: matching + if: ${{ github.event_name == 'push' }} + release: + runs-on: ubuntu-latest + needs: checkout + if: ${{ needs.checkout.outputs.version }} + env: + VERSION: ${{ needs.checkout.outputs.version }} + steps: + - uses: actions/checkout@main + with: + fetch-depth: 0 + - uses: actions/setup-node@main + - uses: actions/setup-ruby@main + + - name: Generate changelog and Auto update version! + run: | + LAST_TAG=$(git describe --tags --abbrev=0) + git log ${LAST_TAG}..HEAD --no-merges --format="- %h %s @%an" > action.changelog.md + sed -i "s/${LAST_TAG:1}/${VERSION}/g" jekyll-rtd-theme.gemspec + sed -i "s/${LAST_TAG:1}/${VERSION}/g" _includes/rest/defaults.liquid + + - name: Prettier check and Build theme assets! + run: | + npm install + npm run format && npm run build + + - name: Save changes! + run: | + git remote set-url origin "https://${GITHUB_ACTOR}:${{ secrets.AUTH_TOKEN }}@github.com/${GITHUB_REPOSITORY}.git" + git config user.name "${GITHUB_ACTOR}" + git config user.email "${GITHUB_ACTOR}@users.noreply.github.com" + git add . + git commit -m "💎 v${VERSION}" && git push + + - name: Create release! + uses: actions/create-release@main + with: + release_name: ${{ needs.checkout.outputs.version }} + tag_name: v${{ needs.checkout.outputs.version }} + commitish: ${{ github.ref }} + body_path: action.changelog.md + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Configure gem credentials and Build gemspec! + run: | + CONFIG=${HOME}/.gem/credentials + mkdir -p ${HOME}/.gem && touch ${CONFIG} && chmod 0600 ${CONFIG} + echo "---" > ${CONFIG} + echo ":github: Bearer ${GITHUB_TOKEN}" >> ${CONFIG} + echo ":rubygems_api_key: ${RUBYGEMS_AUTH_TOKEN}" >> ${CONFIG} + gem build *.gemspec + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RUBYGEMS_AUTH_TOKEN: ${{ secrets.RUBYGEMS_AUTH_TOKEN }} + + - name: Publish rubygems! + run: | + gem push *.gem + gem push --KEY github --host https://rubygems.pkg.github.com/${{ github.repository_owner }} *.gem diff --git a/.github/workflows/reserves/cloud-init/alpine-unittests.yml b/.github/workflows/reserves/cloud-init/alpine-unittests.yml new file mode 100644 index 000000000000..eac2c7a81e47 --- /dev/null +++ b/.github/workflows/reserves/cloud-init/alpine-unittests.yml @@ -0,0 +1,74 @@ + +name: Alpine Unittests + +on: + pull_request: + branches-ignore: + - 'ubuntu/**' + push: + branches: + - main + +concurrency: + group: 'ci-${{ github.workflow }}-${{ github.ref }}' + cancel-in-progress: true + +defaults: + run: + shell: sh -ex {0} + +jobs: + build: + runs-on: ubuntu-latest + steps: + + - name: "Checkout" + uses: actions/checkout@v4 + with: + # Fetch all tags for tools/read-version + fetch-depth: 0 + + - name: Setup LXD + uses: canonical/setup-lxd@v0.1.1 + with: + channel: latest/candidate + + - name: Create alpine container + # the current shell doesn't have lxd as one of the groups + # so switch groups to run lxd commands + run: lxc launch images:alpine/edge alpine + + - name: Check networking (for debugging) + run: | + lxc exec alpine -- ping -c 1 google.com || true + lxc exec alpine -- ping -c 1 dl-cdn.alpinelinux.org || true + lxc exec alpine -- nslookup www.google.com || true + lxc exec alpine -- ping -c 1 dl-cdn.alpinelinux.org || true + + - name: Install dependencies + run: lxc exec alpine -- apk add py3-tox git tzdata + + - name: Mount source into container directory + run: lxc config device add alpine gitdir disk source=$(pwd) path=/root/cloud-init-ro + + - name: Create a r/w directory to run tests in + # without this, tox fails during package install + run: | + lxc exec alpine -- git config --global --add safe.directory /root/cloud-init-ro/.git + lxc exec alpine -- git clone cloud-init-ro cloud-init-rw + + - name: Set a timezone + # test for regression of GH-5158 + # https://github.com/canonical/cloud-init/issues/5158 + run: lxc exec alpine -- ln -s /usr/share/zoneinfo/Europe/Brussels /etc/localtime + + - name: Set up tox environment + # Setup the environment and then tell pytest to do essentially nothing + run: lxc exec alpine --cwd /root/cloud-init-rw -- tox -e py3 -- --cache-show= + + - name: Stop network + # Take down network interfaces to ensure tests don't use network + run: lxc exec alpine -- ifdown -a + + - name: Run unittests + run: lxc exec alpine --cwd /root/cloud-init-rw -- tox -e py3 diff --git a/.github/workflows/reserves/cloud-init/check_format.yml b/.github/workflows/reserves/cloud-init/check_format.yml new file mode 100644 index 000000000000..ba44e665811d --- /dev/null +++ b/.github/workflows/reserves/cloud-init/check_format.yml @@ -0,0 +1,114 @@ +name: Lint Tests +on: + pull_request: + push: + branches: + - main + +concurrency: + group: 'ci-${{ github.workflow }}-${{ github.ref }}' + cancel-in-progress: true +defaults: + run: + shell: sh -ex {0} + +jobs: + check_format: + strategy: + fail-fast: false + matrix: + env: [ruff, mypy, pylint, black, isort] + name: Check ${{ matrix.env }} + runs-on: ubuntu-20.04 + steps: + - name: "Checkout #1" + uses: actions/checkout@v3.0.0 + + - name: "Checkout #2 (for tools/read-version)" + run: | + git fetch --unshallow + git remote add upstream https://git.launchpad.net/cloud-init + + - name: Dependencies + run: | + sudo DEBIAN_FRONTEND=noninteractive apt-get -qy update + sudo DEBIAN_FRONTEND=noninteractive apt-get -qy install tox + + - name: Print version + run: python3 --version + + - name: Test + env: + # matrix env: not to be confused w/environment variables or testenv + TOXENV: ${{ matrix.env }} + run: tox + schema-format: + strategy: + fail-fast: false + name: Check json format + runs-on: ubuntu-22.04 + steps: + - name: "Checkout #1" + uses: actions/checkout@v3.0.0 + + - name: "Test format" + run: | + tools/check_json_format.sh cloudinit/config/schemas/schema-cloud-config-v1.json + tools/check_json_format.sh cloudinit/config/schemas/schema-network-config-v1.json + tools/check_json_format.sh cloudinit/config/schemas/versions.schema.cloud-config.json + + doc: + strategy: + fail-fast: false + name: Check docs + runs-on: ubuntu-22.04 + steps: + - name: "Checkout #1" + uses: actions/checkout@v3.0.0 + + - name: "Checkout #2 (for tools/read-version)" + run: | + git fetch --unshallow + git remote add upstream https://git.launchpad.net/cloud-init + - name: "Install Python 3.10" + uses: actions/setup-python@v4 + with: + python-version: '3.10.8' + - name: "Install dependencies" + run: | + sudo DEBIAN_FRONTEND=noninteractive apt-get -qy update + sudo DEBIAN_FRONTEND=noninteractive apt-get -qy install tox + - name: "Spellcheck" + run: | + tox + env: + TOXENV: doc-spelling + - name: "Build docs" + env: + TOXENV: doc + run: | + tox + + shell-lint: + name: Shell Lint + runs-on: ubuntu-22.04 + steps: + - name: Checkout + uses: actions/checkout@v3.0.0 + + - name: Install ShellCheck + run: | + sudo apt-get update + sudo apt install shellcheck + + - name: Run ShellCheck + run: | + shellcheck ./tools/ds-identify + + check-cla-signers: + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@v3.0.0 + + - name: Check CLA signers file + run: tools/check-cla-signers diff --git a/.github/workflows/reserves/cloud-init/cla.yml b/.github/workflows/reserves/cloud-init/cla.yml new file mode 100644 index 000000000000..fd438b2106e5 --- /dev/null +++ b/.github/workflows/reserves/cloud-init/cla.yml @@ -0,0 +1,40 @@ +name: Verify Contributor License Agreement + +on: [pull_request] + +jobs: + cla-validate: + + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v1 + - name: Check CLA signing status for ${{ github.event.pull_request.user.login }} + run: | + cat > unsigned-cla.txt < /home/$USER/.config/pycloudlib.toml + sudo adduser $USER lxd + # Jammy GH Action runners have docker installed, which edits iptables + # in a way that is incompatible with lxd. + # https://documentation.ubuntu.com/lxd/en/latest/howto/network_bridge_firewalld/#prevent-connectivity-issues-with-lxd-and-docker + sudo iptables -I DOCKER-USER -j ACCEPT + sudo lxd init --auto + - name: Run integration Tests + run: | + sg lxd -c 'CLOUD_INIT_CLOUD_INIT_SOURCE="$(ls cloud-init*.deb)" tox -e integration-tests-ci' diff --git a/.github/workflows/reserves/cloud-init/linkcheck.yml b/.github/workflows/reserves/cloud-init/linkcheck.yml new file mode 100644 index 000000000000..aeae6840a46d --- /dev/null +++ b/.github/workflows/reserves/cloud-init/linkcheck.yml @@ -0,0 +1,39 @@ +name: scheduled-linkcheck +on: + schedule: + - cron: '3 14 * * *' +concurrency: + group: 'ci-${{ github.workflow }}-${{ github.ref }}' + cancel-in-progress: true + +jobs: + linkcheck: + runs-on: ubuntu-22.04 + continue-on-error: ${{ !(github.event_name == 'workflow_dispatch' && github.event.inputs.failOnError == 'true') }} + + steps: + - name: Checkout Repository + uses: actions/checkout@v3.0.0 + + - name: "Install Python 3.10" + uses: actions/setup-python@v4 + with: + python-version: '3.10.8' + + - name: Install tox + run: pip install tox + + - name: Run Sphinx linkcheck + run: tox -e linkcheck | tee output.txt + + - name: Check for broken links below threshold + run: | + broken_count=$(grep -c "broken" output.txt) + if [[ $broken_count -ge 5 ]]; then + echo "Too many broken links detected: $broken_count" + broken_matches=$(grep "broken" output.txt) + echo "Broken links \n$broken_matches" + exit 1 + else + echo "Number of broken links is below threshold: $broken_count" + fi diff --git a/.github/workflows/reserves/cloud-init/scheduled.yml b/.github/workflows/reserves/cloud-init/scheduled.yml new file mode 100644 index 000000000000..aca4f9b7e871 --- /dev/null +++ b/.github/workflows/reserves/cloud-init/scheduled.yml @@ -0,0 +1,59 @@ +name: scheduled-daily +on: + schedule: + - cron: '3 14 * * *' +concurrency: + group: 'ci-${{ github.workflow }}-${{ github.ref }}' + cancel-in-progress: false + +jobs: + hypothesis: + name: unittest / hypothesis-slow + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Install dependencies + env: + DEBIAN_FRONTEND: noninteractive + run: | + sudo apt-get -qy update + sudo apt-get -qy install tox + - name: Run fuzztest + env: + PYTEST_ADDOPTS: -v + HYPOTHESIS_PROFILE: ci + run: tox -e hypothesis-slow + devel_tests: + name: unittest / 3.13-dev + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Install Python 3.13-dev + uses: actions/setup-python@v4 + with: + python-version: 3.13-dev + check-latest: true + - name: Install tox + run: pip install tox + - name: Run unittest + run: tox -e py3 + format_tip: + strategy: + fail-fast: false + matrix: + env: [tip-ruff, tip-mypy, tip-pylint, tip-black, tip-isort] + name: format-tip + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Install dependencies + run: | + sudo apt-get -qy update + sudo apt-get -qy install tox + - name: Run Linters tip + env: + TOXENV: ${{ matrix.env }} + run: tox diff --git a/.github/workflows/reserves/cloud-init/stale.yml b/.github/workflows/reserves/cloud-init/stale.yml new file mode 100644 index 000000000000..c763dafded74 --- /dev/null +++ b/.github/workflows/reserves/cloud-init/stale.yml @@ -0,0 +1,24 @@ +name: Mark and close stale pull requests + +on: + schedule: + - cron: "0 0 * * *" # Daily @ 00:00 + +jobs: + stale: + + runs-on: ubuntu-latest + + steps: + - uses: actions/stale@v1 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + days-before-stale: 14 + days-before-close: 7 + stale-pr-message: | + Hello! Thank you for this proposed change to cloud-init. This pull request is now marked as stale as it has not seen any activity in 14 days. If no activity occurs within the next 7 days, this pull request will automatically close. + + If you are waiting for code review and you are seeing this message, apologies! Please reply, tagging TheRealFalcon, and he will ensure that someone takes a look soon. + + (If the pull request is closed and you would like to continue working on it, please do tag TheRealFalcon to reopen it.) + stale-pr-label: 'stale-pr' diff --git a/.github/workflows/reserves/cloud-init/unit.yml b/.github/workflows/reserves/cloud-init/unit.yml new file mode 100644 index 000000000000..5eda03c8e63e --- /dev/null +++ b/.github/workflows/reserves/cloud-init/unit.yml @@ -0,0 +1,45 @@ +name: Unit Tests +on: + pull_request: + push: + branches: + - main +concurrency: + group: 'ci-${{ github.workflow }}-${{ github.ref }}' + cancel-in-progress: true +defaults: + run: + shell: sh -ex {0} +jobs: + unittests: + strategy: + matrix: + python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12"] + toxenv: [py3] + experimental: [false] + check-latest: [false] + continue-on-error: [false] + include: + - python-version: "3.6" + toxenv: lowest-supported + continue-on-error: false + check-latest: false + experimental: false + name: unittest / ${{ matrix.toxenv }} / python ${{matrix.python-version}} + runs-on: ubuntu-20.04 + continue-on-error: ${{ matrix.experimental }} + steps: + - name: "Checkout" + uses: actions/checkout@v4 + with: + # Fetch all tags for tools/read-version + fetch-depth: 0 + - name: Install Python ${{matrix.python-version}} + uses: actions/setup-python@v4 + with: + python-version: ${{matrix.python-version}} + check-latest: ${{matrix.check-latest}} + - name: Install tox + run: pip install tox + - name: Run unittest + run: tox -e ${{ matrix.toxenv }} diff --git a/.github/workflows/reserves/codeql-analysis.yml b/.github/workflows/reserves/codeql-analysis.yml new file mode 100644 index 000000000000..9889ea23606b --- /dev/null +++ b/.github/workflows/reserves/codeql-analysis.yml @@ -0,0 +1,71 @@ +# For most projects, this workflow file will not need changing; you simply need +# to commit it to your repository. +# +# You may wish to alter this file to override the set of languages analyzed, +# or to provide custom queries or build logic. +# +# ******** NOTE ******** +# We have attempted to detect the languages in your repository. Please check +# the `language` matrix defined below to confirm you have the correct set of +# supported CodeQL languages. +# +name: "CodeQL" + +on: + push: + branches: [ master ] + pull_request: + # The branches below must be a subset of the branches above + branches: [ master ] + schedule: + - cron: '36 20 * * 1' + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ 'python' ] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ] + # Learn more: + # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + # queries: ./path/to/local/query, your-org/your-repo/queries@main + + # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + uses: github/codeql-action/autobuild@v3 + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 https://git.io/JvXDl + + # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines + # and modify them (or add more) to build your code if your project + # uses a compiled language + + #- run: | + # make bootstrap + # make release + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 diff --git a/.github/workflows/reserves/codeql.yml b/.github/workflows/reserves/codeql.yml new file mode 100644 index 000000000000..0071d200394f --- /dev/null +++ b/.github/workflows/reserves/codeql.yml @@ -0,0 +1,70 @@ +# For most projects, this workflow file will not need changing; you simply need +# to commit it to your repository. +# +# You may wish to alter this file to override the set of languages analyzed, +# or to provide custom queries or build logic. +# +# ******** NOTE ******** +# We have attempted to detect the languages in your repository. Please check +# the `language` matrix defined below to confirm you have the correct set of +# supported CodeQL languages. +# +name: "CodeQL" + +on: + push: + branches: [ main ] + pull_request: + # The branches below must be a subset of the branches above + branches: [ main ] + schedule: + - cron: '38 3 * * 1' + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ 'javascript', 'ruby' ] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] + # Learn more about CodeQL language support at https://git.io/codeql-language-support + + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v1 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + # queries: ./path/to/local/query, your-org/your-repo/queries@main + + # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + uses: github/codeql-action/autobuild@v1 + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 https://git.io/JvXDl + + # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines + # and modify them (or add more) to build your code if your project + # uses a compiled language + + #- run: | + # make bootstrap + # make release + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v1 diff --git a/.github/workflows/reserves/config.yml b/.github/workflows/reserves/config.yml new file mode 100644 index 000000000000..ebe993a70a18 --- /dev/null +++ b/.github/workflows/reserves/config.yml @@ -0,0 +1,20 @@ +# Behaviorbot config. See https://github.com/behaviorbot/ for more information. +# Note: Please Don't edit this file directly. +# Edit https://github.com/pages-themes/maintenance-scripts instead. + +# Configuration for update-docs - https://github.com/behaviorbot/update-docs +updateDocsComment: "Thanks for the pull request! If you are making any changes to the user-facing functionality, please be sure to update the documentation in the `README` or `docs/` folder alongside your change. :heart:" + +# Configuration for request-info - https://github.com/behaviorbot/request-info +requestInfoReplyComment: Thanks for this. Do you mind providing a bit more information about what problem you're trying to solve? +requestInfoLabelToAdd: more-information-needed + +# Configuration for new-issue-welcome - https://github.com/behaviorbot/new-issue-welcome +#newIssueWelcomeComment: > +# Welcome! + +# Configuration for new-pr-welcome - https://github.com/behaviorbot/new-pr-welcome +newPRWelcomeComment: Welcome! Congrats on your first pull request to the Tactile theme. If you haven't already, please be sure to check out [the contributing guidelines](https://github.com/pages-themes/tactile/blob/master/docs/CONTRIBUTING.md). + +# Configuration for first-pr-merge - https://github.com/behaviorbot/first-pr-merge +firstPRMergeComment: "Congrats on getting your first pull request to the Tactile theme merged! Without amazing humans like you submitting pull requests, we couldn’t run this project. You rock! :tada:

If you're interested in tackling another bug or feature, take a look at [the open issues](https://github.com/pages-themes/tactile/issues), especially those [labeled `help wanted`](https://github.com/pages-themes/tactile/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)." diff --git a/.github/workflows/reserves/conformance.yml b/.github/workflows/reserves/conformance.yml new file mode 100644 index 000000000000..5eadc897dc4d --- /dev/null +++ b/.github/workflows/reserves/conformance.yml @@ -0,0 +1,53 @@ +name: conformance + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + pull_request: + push: + +jobs: + run-conformance-test: + runs-on: ubuntu-latest + steps: + - + name: Checkout + uses: actions/checkout@v3 + with: + fetch-depth: 0 + - + name: Build image + uses: docker/bake-action@v2 + with: + targets: image-local + - + name: Start distribution server + run: | + IP=`hostname -I | awk '{print $1}'` + echo "IP=$IP" >> $GITHUB_ENV + echo "OCI_ROOT_URL=http://$IP:5000" >> $GITHUB_ENV + DISTRIBUTION_REF="registry:local" + docker run --rm -p 5000:5000 -e REGISTRY_STORAGE_DELETE_ENABLED=true -idt "registry:local" + - + name: Run OCI Distribution Spec conformance tests + uses: opencontainers/distribution-spec@main + env: + OCI_ROOT_URL: ${{ env.OCI_ROOT_URL }} + OCI_NAMESPACE: oci-conformance/distribution-test + OCI_TEST_PULL: 1 + OCI_TEST_PUSH: 1 + OCI_TEST_CONTENT_DISCOVERY: 1 + OCI_TEST_CONTENT_MANAGEMENT: 1 + OCI_HIDE_SKIPPED_WORKFLOWS: 1 + - + name: Move test results + run: mkdir -p .out/ && mv {report.html,junit.xml} .out/ + - + name: Upload test results + uses: actions/upload-artifact@v3 + with: + name: oci-test-results-${{ github.sha }} + path: .out/ + if-no-files-found: error diff --git a/.github/workflows/reserves/dependabot.yml b/.github/workflows/reserves/dependabot.yml new file mode 100644 index 000000000000..a2ed19f8acec --- /dev/null +++ b/.github/workflows/reserves/dependabot.yml @@ -0,0 +1,21 @@ +# To get started with Dependabot version updates, you'll need to specify which +# package ecosystems to update and where the package manifests are located. +# Please see the documentation for all configuration options: +# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates + +version: 2 +updates: + + # Configure docker dependabot updates + - package-ecosystem: "docker" + directory: "/" + target-branch: "master" + schedule: + interval: "daily" + + # Configure GitHub Actions dependabot updates + - package-ecosystem: "github-actions" + directory: "/" + target-branch: "master" + schedule: + interval: "daily" diff --git a/.github/workflows/reserves/devcontainer-cli/build-chat.yml b/.github/workflows/reserves/devcontainer-cli/build-chat.yml new file mode 100644 index 000000000000..c6ac67e9f24a --- /dev/null +++ b/.github/workflows/reserves/devcontainer-cli/build-chat.yml @@ -0,0 +1,56 @@ +name: Build Chat + +on: + workflow_run: + workflows: + - '**' + types: + - completed + branches: + - '**' + +permissions: + id-token: write + actions: read + contents: read + +jobs: + main: + runs-on: ubuntu-latest + environment: ci + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + repository: "microsoft/vscode-github-triage-actions" + path: ./actions + - name: Install Actions + run: npm install --production --prefix ./actions + - name: Install Additional Dependencies + # Pulls in a bunch of other packages that arent needed for the rest of the actions + run: npm install @azure/storage-blob@12.1.1 + - name: 'Az CLI login' + uses: azure/login@v1 + with: + client-id: ${{ secrets.AZURE_CLIENT_ID }} + tenant-id: ${{ secrets.AZURE_TENANT_ID }} + subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }} + - name: Read Secrets + id: read-secrets + run: | + SLACK_TOKEN=$(az keyvault secret show --name slack-api-token --vault-name vscode-probot --query value -o tsv) + echo "::add-mask::$SLACK_TOKEN" + echo "SLACK_TOKEN=$SLACK_TOKEN" >> "$GITHUB_OUTPUT" + + STORAGE_CONNECTION_STRING=$(az keyvault secret show --name azure-storage-connection --vault-name vscode-probot --query value -o tsv) + echo "::add-mask::$STORAGE_CONNECTION_STRING" + echo "STORAGE_CONNECTION_STRING=$STORAGE_CONNECTION_STRING" >> "$GITHUB_OUTPUT" + - name: Build Chat + uses: ./actions/build-chat + with: + token: ${{ secrets.GITHUB_TOKEN }} + slack_token: ${{ steps.read-secrets.outputs.SLACK_TOKEN }} + storage_connection_string: ${{ steps.read-secrets.outputs.STORAGE_CONNECTION_STRING }} + workflow_run_url: ${{ github.event.workflow_run.url }} + notify_authors: true + log_channel: bot-log diff --git a/.github/workflows/reserves/devcontainer-cli/dev-containers.yml b/.github/workflows/reserves/devcontainer-cli/dev-containers.yml new file mode 100644 index 000000000000..26b2933e447f --- /dev/null +++ b/.github/workflows/reserves/devcontainer-cli/dev-containers.yml @@ -0,0 +1,122 @@ +name: Dev Containers CI + +on: + push: + branches: + - '**' + pull_request: + branches: + - '**' + +jobs: + cli: + name: CLI + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: '18.x' + registry-url: 'https://npm.pkg.github.com' + scope: '@microsoft' + - name: Install Dependencies + run: yarn install --frozen-lockfile + - name: Type-Check + run: yarn type-check + - name: Lint + run: yarn lint + - name: Package + run: yarn package + - name: TGZ name + run: | + VERSION=$(jq -r '.version' < package.json) + echo "TGZ=devcontainers-cli-${VERSION}.tgz" | tee -a $GITHUB_ENV + echo "TGZ_UPLOAD=devcontainers-cli-${VERSION}-${GITHUB_SHA:0:8}.tgz" | tee -a $GITHUB_ENV + - name: Store TGZ + uses: actions/upload-artifact@v4 + with: + name: ${{ env.TGZ_UPLOAD }} + path: ${{ env.TGZ }} + tests-matrix: + name: Tests Matrix + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + mocha-args: [ + "src/test/container-features/containerFeaturesOrder.test.ts", + "src/test/container-features/e2e.test.ts", + "src/test/container-features/featuresCLICommands.test.ts", + "src/test/cli.build.test.ts", + "src/test/cli.exec.buildKit.1.test.ts", + "src/test/cli.exec.buildKit.2.test.ts", + "src/test/cli.exec.nonBuildKit.1.test.ts", + "src/test/cli.exec.nonBuildKit.2.test.ts", + "src/test/cli.test.ts", + "src/test/cli.up.test.ts", + "src/test/imageMetadata.test.ts", + "src/test/container-features/containerFeaturesOCIPush.test.ts", + # Run all except the above: + "--exclude src/test/container-features/containerFeaturesOrder.test.ts --exclude src/test/container-features/registryCompatibilityOCI.test.ts --exclude src/test/container-features/containerFeaturesOCIPush.test.ts --exclude src/test/container-features/e2e.test.ts --exclude src/test/container-features/featuresCLICommands.test.ts --exclude src/test/cli.build.test.ts --exclude src/test/cli.exec.buildKit.1.test.ts --exclude src/test/cli.exec.buildKit.2.test.ts --exclude src/test/cli.exec.nonBuildKit.1.test.ts --exclude src/test/cli.exec.nonBuildKit.2.test.ts --exclude src/test/cli.test.ts --exclude src/test/cli.up.test.ts --exclude src/test/imageMetadata.test.ts 'src/test/**/*.test.ts'", + ] + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '18.x' + registry-url: 'https://npm.pkg.github.com' + scope: '@microsoft' + - name: Install Dependencies + run: | + yarn install --frozen-lockfile + docker run --privileged --rm tonistiigi/binfmt --install all + - name: Type-Check + run: yarn type-check + - name: Package + run: yarn package + - name: Run Tests + run: yarn test-matrix --forbid-only ${{ matrix.mocha-args }} + env: + CI: true + + features-registry-compatibility: + name: OCI Implementation Registry Compatibility + # TODO: This should be expanded to run on different platforms + # Most notably to test platform-specific credential helper behavior + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '18.x' + registry-url: 'https://npm.pkg.github.com' + scope: '@microsoft' + - name: Install Dependencies + run: yarn install --frozen-lockfile + - name: Type-Check + run: yarn type-check + - name: Package + run: yarn package + - name: Run Tests + run: yarn test-matrix --forbid-only src/test/container-features/registryCompatibilityOCI.test.ts + env: + CI: true + # This variable should only be set in the parent `devcontainers/cli` repo. + RUNNING_IN_DEVCONTAINERS_CLI_REPO_CI: ${{ vars.RUNNING_IN_DEVCONTAINERS_CLI_REPO_CI }} + # Scoped to read private packages in the `devcontainers` org (for testing purposes) + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + # Pull-only credential for a test Azure CR instance + FEATURES_TEST__AZURE_REGISTRY_SCOPED_CREDENTIAL: ${{ secrets.FEATURES_TEST__AZURE_REGISTRY_SCOPED_CREDENTIAL }} + + + tests: + name: Tests + needs: [tests-matrix, features-registry-compatibility] + runs-on: ubuntu-latest + steps: + - name: Done + run: echo Test Matrix done. diff --git a/.github/workflows/reserves/devcontainer-cli/publish-dev-containers.yml b/.github/workflows/reserves/devcontainer-cli/publish-dev-containers.yml new file mode 100644 index 000000000000..8e059287e7eb --- /dev/null +++ b/.github/workflows/reserves/devcontainer-cli/publish-dev-containers.yml @@ -0,0 +1,46 @@ +name: Publish @devcontainers/cli + +on: + push: + tags: + - 'v*' + +jobs: + main: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '18.x' + registry-url: 'https://registry.npmjs.org' + scope: '@devcontainers' + - name: Verify Versions + run: | + node -e " + const packageRef = 'refs/tags/v' + require('./package.json').version; + const githubRef = '${{ github.ref }}'; + if (packageRef !== githubRef && packageRef + '-pre-release' != githubRef) { + console.log('::error::' + 'Version Mismatch.', packageRef, githubRef); + throw Error('Version Mismatch'); + } + " + - name: TGZ name + run: | + VERSION=$(jq -r '.version' < package.json) + echo "TGZ=devcontainers-cli-${VERSION}.tgz" | tee -a $GITHUB_ENV + echo "TGZ_UPLOAD=devcontainers-cli-${VERSION}-${GITHUB_SHA:0:8}.tgz" | tee -a $GITHUB_ENV + - name: Download TGZ + uses: dawidd6/action-download-artifact@09f2f74827fd3a8607589e5ad7f9398816f540fe + with: + workflow: dev-containers.yml + workflow_conclusion: success + commit: ${{ github.sha }} + name: ${{ env.TGZ_UPLOAD }} + path: . + - name: Publish TGZ + run: npm publish ${TGZ} --access public + env: + NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} diff --git a/.github/workflows/reserves/devcontainer-cli/test-plan-item-validator.yml b/.github/workflows/reserves/devcontainer-cli/test-plan-item-validator.yml new file mode 100644 index 000000000000..22f75ba44a25 --- /dev/null +++ b/.github/workflows/reserves/devcontainer-cli/test-plan-item-validator.yml @@ -0,0 +1,25 @@ +name: Test Plan Item Validator +on: + issues: + types: [edited] + +jobs: + main: + runs-on: ubuntu-latest + steps: + - name: Checkout Actions + if: contains(github.event.issue.labels.*.name, 'testplan-item') || contains(github.event.issue.labels.*.name, 'invalid-testplan-item') + uses: actions/checkout@v4 + with: + repository: 'microsoft/vscode-github-triage-actions' + ref: stable + - name: Install Actions + if: contains(github.event.issue.labels.*.name, 'testplan-item') || contains(github.event.issue.labels.*.name, 'invalid-testplan-item') + run: npm install --production + - name: Run Test Plan Item Validator + if: contains(github.event.issue.labels.*.name, 'testplan-item') || contains(github.event.issue.labels.*.name, 'invalid-testplan-item') + uses: ./test-plan-item-validator + with: + label: testplan-item + invalidLabel: invalid-testplan-item + comment: Invalid test plan item. See errors below and the [test plan item spec](https://github.com/microsoft/vscode/wiki/Writing-Test-Plan-Items) for more information. This comment will go away when the issues are resolved. diff --git a/.github/workflows/reserves/devcontainer-cli/test-windows.yml b/.github/workflows/reserves/devcontainer-cli/test-windows.yml new file mode 100644 index 000000000000..ce1cd560bf6e --- /dev/null +++ b/.github/workflows/reserves/devcontainer-cli/test-windows.yml @@ -0,0 +1,63 @@ +name: Test (Windows) + +defaults: + run: + shell: bash + +on: + push: + branches: + - '**' + pull_request: + branches: + - '**' +jobs: + tests-matrix: + name: Tests Matrix (Windows) + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + mocha-args-windows: [ + # "src/test/cli.build.test.ts", + # "src/test/cli.exec.buildKit.1.test.ts", + # "src/test/cli.exec.buildKit.2.test.ts", + # "src/test/cli.exec.nonBuildKit.1.test.ts", + # "src/test/cli.exec.nonBuildKit.2.test.ts", + # "src/test/cli.test.ts", + # "src/test/cli.up.test.ts", + "src/test/container-features/containerFeaturesOCI.test.ts", + # "src/test/container-features/containerFeaturesOCIPush.test.ts", + # "src/test/container-features/containerFeaturesOrder.test.ts", + # "src/test/container-features/e2e.test.ts", + # "src/test/container-features/featureHelpers.test.ts", + # "src/test/container-features/featuresCLICommands.test.ts", + "src/test/container-features/generateFeaturesConfig.test.ts", + # "src/test/container-templates/containerTemplatesOCI.test.ts", + # "src/test/container-templates/templatesCLICommands.test.ts", + # "src/test/dockerComposeUtils.test.ts", + # "src/test/dockerfileUtils.test.ts", + # "src/test/dockerUtils.test.ts", + # "src/test/imageMetadata.test.ts", + # "src/test/variableSubstitution.test.ts", + # "src/test/container-features/registryCompatibilityOCI.test.ts", + # # Run all except the above: + # "--exclude .....", + ] + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Setup Node.js + uses: actions/setup-node@v4 + with: + node-version: '18.x' + registry-url: 'https://npm.pkg.github.com' + scope: '@microsoft' + - name: Install Dependencies + run: yarn install --frozen-lockfile + - name: Type-Check + run: yarn type-check + - name: Package + run: yarn package + - name: Run Tests + run: yarn test-matrix --forbid-only ${{ matrix.mocha-args-windows }} \ No newline at end of file diff --git a/.github/workflows/reserves/devcontainer/ci_branch.yml b/.github/workflows/reserves/devcontainer/ci_branch.yml new file mode 100644 index 000000000000..99fff2972d00 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/ci_branch.yml @@ -0,0 +1,33 @@ +--- +name: CI (branch) +# This workflow is triggered manually and can be used to test workflow changes on a branch (which aren't picked up by the pr-bot builds) + +on: + workflow_dispatch: + inputs: + prNumber: + description: set to the PR number if testing a branch for a PR (used to derive the image tag) + default: '' + type: string + required: false + runFullTests: + description: tick to run the full tests, untick to run tests without credentials + default: true + type: boolean + required: false + +permissions: + contents: write + packages: write + +jobs: + build-test-publish: + name: "Build, test, publish" + uses: ./.github/workflows/ci_common.yml + with: + release: false + prNumber: ${{ github.event.inputs.prNumber }} + # despite the runFullTests input being a boolean, we seem to get a string below! + runFullTests: ${{ github.event.inputs.runFullTests == 'true' }} + secrets: + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} diff --git a/.github/workflows/reserves/devcontainer/ci_common.yml b/.github/workflows/reserves/devcontainer/ci_common.yml new file mode 100644 index 000000000000..20b945bf7ae5 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/ci_common.yml @@ -0,0 +1,1188 @@ +# +# This workflow contains the core logic to build, test and release the GitHub Action and Azure DevOps Task +# It is called via workflow_call from the pr-bot +# +name: "build-test" +on: + workflow_call: + inputs: + prNumber: + description: The PR number (if building for a PR) + type: string + required: false + prRef: + description: The git ref to checkout (e.g. the merge commit for a PR) + type: string + required: false + prHeadSha: + description: >- + For PR builds (where GITHUB_REF would usually be set to the PR but isn't due to the pr-bot + running on a comment trigger), pass the PR's head SHA commit here + type: string + required: false + runFullTests: + description: true to run the full tests, false to run tests without credentials + default: true + type: boolean + required: false + release: + type: boolean + description: indicates whether to make a release + default: false + required: false + AZDO_ORG: + description: The AzDO org for AzDO task verification + type: string + required: false + default: https://dev.azure.com/monacotools + AZDO_PROJECT: + description: The AzDO project for AzDO task verification + type: string + required: false + default: Monaco + AZDO_BUILD: + description: The AzDO build to trigger for AzDO task verification + type: string + required: false + default: devcontainers.ci + secrets: + AZDO_TOKEN: + description: "" + required: true + +permissions: + contents: write + packages: write + +jobs: + build: # run build using the github action! + runs-on: ubuntu-latest + name: build, test, package + outputs: + version: ${{ steps.build.outputs.version }} + version_short: ${{ steps.build.outputs.version_short }} + version_major: ${{ steps.build.outputs.version_major }} + image_tag: ${{ steps.set_image_tag.outputs.image_tag }} + image_push_option: ${{ steps.set_image_push_option.outputs.image_push_option }} + build_number: ${{ steps.build_number.outputs.build_number }} + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - id: set_image_push_option + name: Set image push option + uses: actions/github-script@v6 + with: + script: | + // if running full tests then use the filter option, for the dev container action + // else, set to never to avoid pushing (as we're likely running without secrets) + + const pushOption = ${{ inputs.runFullTests }} ? 'filter' : 'never'; + console.log(`Setting image_push_option=${pushOption}`); + core.setOutput("image_push_option", pushOption); + + - name: Login to GitHub Container Registry + if: ${{ steps.set_image_push_option.outputs.image_push_option == 'filter' }} + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Previously, run_id was used as the patch number (BUILD_NUMBER) + # but this got outside the limit for AzDO extensions (0 to 2147483647) + - name: Generate build number + id: build_number + # only run this if we are going to run the AzDO publishing + if: ${{ steps.set_image_push_option.outputs.image_push_option == 'filter' }} + uses: onyxmueller/build-tag-number@v1 + with: + token: ${{ secrets.GITHUB_TOKEN }} + + - id: set_image_tag + name: Set image_tag... + uses: actions/github-script@v6 + env: + PR_NUMBER: ${{ inputs.prNumber }} + with: + script: | + const {PR_NUMBER} = process.env + if (PR_NUMBER) { + const prTag = `pr-${PR_NUMBER}`; + console.log(`Setting image_tag=${prTag} for pull_request`); + core.setOutput("image_tag", prTag); + return; + } + if ((context.eventName === "push" || context.eventName === "workflow_dispatch") && context.ref === "refs/heads/main") { + console.log("Setting image_tag=latest for push to main"); + core.setOutput("image_tag", "latest"); + return; + } + core.error('Failed to set image_tag'); + console.log(context); + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + node --version + npm --version + echo "** Installing and building common..." + (cd common && npm install && npm run build) + (cd common && npm run tsc_version) + echo "** Installing and building github-action..." + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Build in dev container + uses: ./ + id: build + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + BUILD_NUMBER: ${{ steps.build_number.outputs.build_number }} + IS_PR: ${{ github.head_ref }} + BRANCH: ${{ github.ref }} + with: + imageName: ghcr.io/devcontainers/ci-devcontainer + imageTag: ${{ steps.set_image_tag.outputs.image_tag }} + runCmd: | + echo "Starting" + ./scripts/build-test-package.sh + push: ${{ steps.set_image_push_option.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + env: | + BUILD_NUMBER + IS_CI=1 + IS_PR + BRANCH + + - name: Show version info + env: + VERSION: ${{ steps.build.outputs.version }} + VERSION_SHORT: ${{ steps.build.outputs.version_short }} + VERSION_MAJOR: ${{ steps.build.outputs.version_major }} + run: | + echo "VERSION=${VERSION}" + echo "VERSION_SHORT=${VERSION_SHORT}" + echo "VERSION_MAJOR=${VERSION_MAJOR}" + + - uses: actions/upload-artifact@v3 + name: Create dev AzDO VSIX artifact + with: + name: azdo-task-dev + path: ./output/devcontainers-dev.ci-${{ steps.build.outputs.version }}.vsix + if-no-files-found: error + + - uses: actions/upload-artifact@v3 + name: Create release AzDO VSIX artifact + with: + name: azdo-task + path: ./output/devcontainers.ci-${{ steps.build.outputs.version }}.vsix + if-no-files-found: error + + checks-succeeded: + # simple no-op job to use as step for check status + name: All succeeded + needs: # TODO - check what jobs need adding here + - build + - test-azdo + - test-gh-run-args + - test-gh-build-args + - test-gh-dockerfile-context + - test-gh-feature-docker-from-docker + - test-gh-docker-from-docker-non-root + - test-gh-docker-from-docker-root + - test-gh-skip-user-update + - test-compose-features + - test-config-file + - test-simple + - test-no-run + - test-platform-with-runcmd + - test-env-vars-on-post-create + - test-multiple-tags-job2 + - test-github-output + runs-on: ubuntu-latest + steps: + - name: Simple step + run: | + echo Done! + + pr-checks: + name: Mark PR checks as complete + needs: # TODO - check what jobs need adding here + - checks-succeeded + runs-on: ubuntu-latest + if: ${{ inputs.prHeadSha }} + steps: + # For PR builds triggered from comment builds, the GITHUB_REF is set to main + # so the checks aren't automatically associated with the PR + # If prHeadSha is specified then explicity mark the checks for that SHA + - name: Report check status + if: ${{ inputs.prHeadSha }} + uses: LouisBrunner/checks-action@v1.6.0 + with: + token: ${{ secrets.GITHUB_TOKEN }} + # the name must be identical to the one received by the real job + sha: ${{ inputs.prHeadSha }} + name: "Build, test, publish / All succeeded" + status: "completed" + conclusion: "success" + + release: + name: Create release + runs-on: ubuntu-latest + needs: + - build + - checks-succeeded + if: ${{ inputs.release == true }} + env: + VERSION: ${{ needs.build.outputs.version }} + VERSION_SHORT: ${{ needs.build.outputs.version_short }} + VERSION_MAJOR: ${{ needs.build.outputs.version_major }} + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Show version info + run: | + echo "VERSION=${VERSION}" + echo "VERSION_SHORT=${VERSION_SHORT}" + echo "VERSION_MAJOR=${VERSION_MAJOR}" + + - name: Download release VSIX + uses: actions/download-artifact@v3 + with: + name: azdo-task + path: output + + - name: Show version + run: | + echo "VERSION: $VERSION" + echo "VERSION_SHORT: $VERSION_SHORT" + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: github.ref == 'refs/heads/main' # only need this for push (on `main`) + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Publish AzDO Task + uses: ./ + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + AZDO_PROJECT: ${{ inputs.AZDO_PROJECT }} + AZDO_BUILD: ${{ inputs.AZDO_BUILD }} + AZDO_ORG: ${{ inputs.AZDO_ORG }} + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} + AZURE_DEVOPS_EXT_PAT: ${{ secrets.AZDO_TOKEN }} + BUILD_NUMBER: ${{ github.run_id }} + IS_PR: ${{ github.head_ref }} + BRANCH: ${{ github.ref }} + with: + imageName: ghcr.io/devcontainers/ci-devcontainer + runCmd: | + echo "Starting" + ./scripts/publish-azdo-task.sh + env: | + BUILD_NUMBER + IS_CI=1 + IS_PR + BRANCH + AZDO_TOKEN + AZURE_DEVOPS_EXT_PAT + AZDO_ORG + AZDO_PROJECT + AZDO_BUILD + + - name: Create GitHub Action Release + env: + TAG_NAME: v${{ needs.build.outputs.version_short }} + RELEASE_NAME: v${{ needs.build.outputs.version }} + GH_TOKEN: ${{ github.token }} + run: | + git config user.name "CI build" + git config user.email automated-build@example.com + + ./scripts/gh-release.sh + + # + # Jobs to test + # + # When adding a new test: + # - add to the 'needs' section for the 'checks-succeeded' job + # - add any new images to the image_names in clean_tags.sh + # - review the Azure DevOps pipeline and consider adding there for verification + # + + test-azdo: + name: Run AzDO test + runs-on: ubuntu-latest + needs: build + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Show version + env: + VERSION: ${{ needs.build.outputs.version }} + VERSION_SHORT: ${{ needs.build.outputs.version_short }} + run: | + echo "VERSION: $VERSION" + echo "VERSION_SHORT: $VERSION_SHORT" + + - name: Download workflow artifacts + uses: actions/download-artifact@v3 + with: + path: output + + - name: Download release VSIX + uses: actions/download-artifact@v3 + with: + name: azdo-task-dev + path: output + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: github.ref == 'refs/heads/main' # only need this for push (on `main`) + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run AzDO test pipeline + uses: ./ + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + AZDO_PROJECT: ${{ inputs.AZDO_PROJECT }} + AZDO_BUILD: ${{ inputs.AZDO_BUILD }} + AZDO_ORG: ${{ inputs.AZDO_ORG }} + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} + AZURE_DEVOPS_EXT_PAT: ${{ secrets.AZDO_TOKEN }} + BUILD_NUMBER: ${{ github.run_id }} + IS_PR: ${{ github.head_ref }} + BRANCH: ${{ github.ref }} + IMAGE_TAG: ${{ needs.build.outputs.image_tag }} + with: + imageName: ghcr.io/devcontainers/ci-devcontainer + runCmd: | + echo "Starting" + ./scripts/test-azdo.sh + env: | + BUILD_NUMBER + IMAGE_TAG + IS_CI=1 + IS_PR + BRANCH + AZDO_TOKEN + AZURE_DEVOPS_EXT_PAT + AZDO_ORG + AZDO_PROJECT + AZDO_BUILD + + test-simple: + name: Run simple test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + id: simpletest + with: + subFolder: github-tests/Dockerfile/run-args + runCmd: echo $HOSTNAME && [[ $HOSTNAME == "my-host" ]] + - name: Validate runCmdOutput output + run: | + echo "'runCmdOutput' value: $runCmdOutput" + if [["$runCmdOutput" = *my-host*]]; then + echo "'runCmdOutput' output of simpletest step doesn't contain expected value 'my-host'" + exit 1 + fi + env: + runCmdOutput: ${{ steps.simpletest.outputs.runCmdOutput }} + + test-config-file: + name: Run test with config file + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + id: configfiletest + with: + subFolder: github-tests/Dockerfile/config-file + configFile: github-tests/Dockerfile/config-file/.devcontainer/subfolder/devcontainer.json + runCmd: echo $HOSTNAME && [[ $HOSTNAME == "my-host" ]] + - name: Validate runCmdOutput output + run: | + echo "'runCmdOutput' value: $runCmdOutput" + if [["$runCmdOutput" = *my-host*]]; then + echo "'runCmdOutput' output of configfiletest step doesn't contain expected value 'my-host'" + exit 1 + fi + env: + runCmdOutput: ${{ steps.configfiletest.outputs.runCmdOutput }} + + test-gh-run-args: + name: Run GitHub run-args test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/run-args + imageName: ghcr.io/devcontainers/ci/tests/run-args + runCmd: echo $HOSTNAME && [[ $HOSTNAME == "my-host" ]] + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-env-vars-on-post-create: + name: Run GitHub env-vars-on-post-create test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + env: # TEST_ENV_VALUE1 is set via devcontainer.json using a localEnv reference + TEST_ENV_VALUE1: SetViaDevcontainerJsonLocalEnv + with: + subFolder: github-tests/Dockerfile/env-vars-on-post-create + imageName: ghcr.io/devcontainers/ci/tests/env-vars-on-post-create + env: + | # TEST_ENV_VALUE2 is an additional env var to pass to the container + TEST_ENV_VALUE2=AdditionalEnvVar + runCmd: | + cat marker.txt + cat marker.txt | grep 'post-create: TEST_ENV_VALUE1=SetViaDevcontainerJsonLocalEnv' + cat marker.txt | grep 'post-create: TEST_ENV_VALUE2=AdditionalEnvVar' + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-build-args: + name: Run GitHub build-args test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/build-args + imageName: ghcr.io/devcontainers/ci/tests/build-args + runCmd: echo $BUILD_ARG_TEST && [[ $BUILD_ARG_TEST == "Hello build-args!" ]] + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-dockerfile-context: + name: Run GitHub dockerfile-context test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/dockerfile-context + imageName: ghcr.io/devcontainers/ci/tests/dockerfile-context + runCmd: /tmp/dummy.sh + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-feature-docker-from-docker: + name: Run GitHub feature-docker-from-docker test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/feature-docker-from-docker + imageName: ghcr.io/devcontainers/ci/tests/feature-docker-from-docker + runCmd: make docker-build + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-docker-from-docker-non-root: + name: Run GitHub docker-from-docker-non-root test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/docker-from-docker-non-root + imageName: ghcr.io/devcontainers/ci/tests/docker-from-docker-non-root + runCmd: make docker-build + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-docker-from-docker-root: + name: Run GitHub docker-from-docker-root test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/docker-from-docker-root + imageName: ghcr.io/devcontainers/ci/tests/docker-from-docker-root + runCmd: make docker-build + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-gh-skip-user-update: + name: Run GitHub skipContainerUserIdUpdate test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/skip-user-update + imageName: ghcr.io/devcontainers/ci/tests/skip-user-update + skipContainerUserIdUpdate: true + runCmd: | + echo "Get user/group info ..." + user_id=$(id -u $(whoami)) + echo "User ID: $user_id" + group_id=$(id -g $(whoami)) + echo "Group ID: $group_id" + [[ $user_id == 1000 ]] && [[ $group_id == 1000 ]] + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-compose-features: + name: Run GitHub docker-compose with features test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/docker-compose/features + imageName: ghcr.io/devcontainers/ci/tests/compose-features + runCmd: go version + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + + test-no-run: + name: Run test without runCmd + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + with: + subFolder: github-tests/Dockerfile/build-only + + test-platform-with-runcmd: + name: Run GitHub platform and runCmd test + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up QEMU for multi-architecture builds + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + # This can be omitted once runner images have a version of Skopeo > 1.4.1 + # See https://github.com/containers/skopeo/issues/1874 (and https://github.com/devcontainers/ci/issues/191#issuecomment-1416384710) + - name: Update skopeo + run: | + sudo apt purge buildah golang-github-containers-common podman skopeo + sudo apt autoremove --purge + REPO_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable" + source /etc/os-release + sudo sh -c "echo 'deb ${REPO_URL}/x${NAME}_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list" + sudo wget -qnv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/x${NAME}_${VERSION_ID}/Release.key -O Release.key + sudo apt-key add Release.key + sudo apt-get update + sudo apt-get install skopeo + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Run test + uses: ./ + id: platform-with-runcmd + with: + subFolder: github-tests/Dockerfile/platform-with-runcmd + imageName: ghcr.io/devcontainers/ci/tests/platform-with-runcmd + platform: linux/amd64,linux/arm64 + runCmd: echo $HOSTNAME && [[ $HOSTNAME == "my-host" ]] + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + - name: Validate runCmdOutput output + run: | + echo "'runCmdOutput' value: $runCmdOutput" + if [["$runCmdOutput" = *my-host*]]; then + echo "'runCmdOutput' output of platform-with-runcmd step doesn't contain expected value 'my-host'" + exit 1 + fi + env: + runCmdOutput: ${{ steps.platform-with-runcmd.outputs.runCmdOutput }} + + test-multiple-tags-job1: + # This is split into multiple parts because pushing the image happens during the post-job phase + # and thus testing if the tags exist can't happen as part of job 1 + name: Test multiple tags job 1 + runs-on: ubuntu-latest + needs: build + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Build and push with multiple tags and single platform + uses: ./ + id: multiple-tags-single-platform + with: + subFolder: github-tests/Dockerfile/build-only + imageName: ghcr.io/devcontainers/ci/tests/multiple-tags-single-platform + imageTag: tag1-${{ github.run_id }},tag2-${{ github.run_id }} + push: always + + - name: Set up QEMU for multi-architecture builds + uses: docker/setup-qemu-action@v3 + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v3 + + - name: Install updated Skopeo + # This can be omitted once runner images have a version of Skopeo > 1.4.1 + # See https://github.com/containers/skopeo/issues/1874 + run: | + sudo apt purge buildah golang-github-containers-common podman skopeo + sudo apt autoremove --purge + REPO_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable" + source /etc/os-release + sudo sh -c "echo 'deb ${REPO_URL}/x${NAME}_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list" + sudo wget -qnv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/x${NAME}_${VERSION_ID}/Release.key -O Release.key + sudo apt-key add Release.key + sudo apt-get update + sudo apt-get install skopeo + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + - name: Build and push with multiple tags and multiple platforms + uses: ./ + id: multiple-tags-multiple-platforms + with: + subFolder: github-tests/Dockerfile/platform-with-runcmd + imageName: ghcr.io/devcontainers/ci/tests/multiple-tags-multiple-platforms + imageTag: tag1-${{ github.run_id }},tag2-${{ github.run_id }} + platform: linux/amd64,linux/arm64 + push: always + + test-multiple-tags-job2: + name: Test multiple tags job 2 + runs-on: ubuntu-latest + needs: [build, test-multiple-tags-job1] + steps: + - name: Validate multiple tags single platform + run: | + set -e + echo "Querying GitHub API for versions of package devcontainers-ci/tests/multiple-tags-single-platform..." + versions=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/users/devcontainers/packages/container/ci%2Ftests%2Fmultiple-tags-single-platform/versions) + echo "Checking tag1-${{ github.run_id }} exists... " + echo "$versions" | grep -q "tag1-${{ github.run_id }}" + echo "Tag exists." + echo "Checking tag2-${{ github.run_id }} exists... " + echo "$versions" | grep -q "tag1-${{ github.run_id }}" + echo "Tag exists." + - name: Validate multiple tags multiple platforms + run: | + set -e + echo "Querying GitHub API for versions of package devcontainers-ci/tests/multiple-tags-multiple-platforms..." + versions=$(curl -s -H "Accept: application/vnd.github+json" -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" -H "X-GitHub-Api-Version: 2022-11-28" https://api.github.com/users/devcontainers/packages/container/ci%2Ftests%2Fmultiple-tags-multiple-platforms/versions) + echo "Checking tag1-${{ github.run_id }} exists... " + echo "$versions" | grep -q "tag1-${{ github.run_id }}" + echo "Tag exists." + echo "Checking tag2-${{ github.run_id }} exists... " + echo "$versions" | grep -q "tag1-${{ github.run_id }}" + echo "Tag exists." + + test-github-output: + runs-on: ubuntu-latest + needs: build + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + # if the following value is missing (i.e. not triggered via comment workflow) + # then the default checkout will apply + ref: ${{ inputs.prRef }} + + - name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + + - name: Login to GitHub Container Registry + uses: docker/login-action@v2 + if: ${{ needs.build.outputs.image_push_option == 'filter' }} + with: + registry: ghcr.io + username: ${{ github.repository_owner }} + password: ${{ secrets.GITHUB_TOKEN }} + + + # Published action contains compiled JS, but we need to compile it here + - uses: actions/setup-node@v3 + with: + node-version: 20 + - name: Compile GH action + run: | + (cd common && npm install && npm run build) + (cd github-action/ && npm install && npm run build && npm run package) + + + - id: github-output + name: Build and run test with GITHUB_OUTPUT + uses: ./ + with: + subFolder: github-tests/Dockerfile/outputs + imageName: ghcr.io/devcontainers/ci/tests/outputs + imageTag: ${{ needs.build.outputs.image_tag }} + push: ${{ needs.build.outputs.image_push_option }} + eventFilterForPush: | + push + pull_request + runCmd: | + echo "test=hello" >> "$GITHUB_OUTPUT" + echo "test2=world" >> "$GITHUB_OUTPUT" + echo "TEST_ENV=env_value" >> "$GITHUB_ENV" + echo "/mnt/test/path" >> "$GITHUB_PATH" + + - name: Validate output + run: | + set -e + echo "Test outputs..." + test "${{ steps.github-output.outputs.test }}" == "hello" + test "${{ steps.github-output.outputs.test2 }}" == "world" + + echo "Test Env" + test "${TEST_ENV}" == "env_value" + + echo "Test Path" + echo "${PATH}" | grep -q "/mnt/test/path" diff --git a/.github/workflows/reserves/devcontainer/ci_main.yml b/.github/workflows/reserves/devcontainer/ci_main.yml new file mode 100644 index 000000000000..657f85d68882 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/ci_main.yml @@ -0,0 +1,22 @@ +--- +name: CI (main) +# This workflow is triggered on pushes to main and runs tests and performs the release steps + +on: + push: + branches: [main] + workflow_dispatch: + +permissions: + contents: write + packages: write + +jobs: + build-test-publish: + name: "Build, test, publish" + if: github.ref == 'refs/heads/main' + uses: ./.github/workflows/ci_common.yml + with: + release: true + secrets: + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} diff --git a/.github/workflows/reserves/devcontainer/clean_tags.sh b/.github/workflows/reserves/devcontainer/clean_tags.sh new file mode 100755 index 000000000000..7736bc7dd553 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/clean_tags.sh @@ -0,0 +1,75 @@ +#!/bin/bash +set -e + + +function show_usage() { + echo + echo "clean_tags.sh" + echo + echo "Clean tags across images" + echo + echo -e "\t--tag\t(Required)The tag to clean" + echo +} + + +# Set default values here +tag="" + +# Process switches: +while [[ $# -gt 0 ]] +do + case "$1" in + --tag) + tag=$2 + shift 2 + ;; + *) + echo "Unexpected '$1'" + show_usage + exit 1 + ;; + esac +done + + +if [[ -z $tag ]]; then + echo "--tag" + show_usage + exit 1 +fi + +image_names=( + "ci-devcontainer" + "ci/tests/run-args" + "ci/tests/build-args" + "ci/tests/dockerfile-context" + "ci/tests/feature-docker-from-docker" + "ci/tests/docker-from-docker-non-root" + "ci/tests/docker-from-docker-root" + "ci/tests/skip-user-update" +) + +echo "Checking for tag $tag:" + +for image_name in ${image_names[@]}; +do + echo "Checking image $image_name..." + escaped_image_name=$(echo ${image_name} | sed "s/\//%2f/g") + response=$(curl -s -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/orgs/devcontainers/packages/container/$escaped_image_name/versions") + message=$(echo "$response" | jq -r ".message?") + if [[ -z "$message" || "$message" == "null" ]]; then + version_id=$(echo "$response" | jq -r ".[] | select(.metadata.container.tags | index(\"${tag}\")) | .id") + if [[ -n $version_id ]]; then + echo "Found version '$version_id' for '$image_name:$tag' - deleting..." + curl -s -X DELETE -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/orgs/devcontainers/packages/container/$escaped_image_name/versions/$version_id" + else + echo "Tag '$tag' not found for '$image_name:$tag' - skipping" + fi + else + echo "Error: $message" + fi +done + + +echo "Done." \ No newline at end of file diff --git a/.github/workflows/reserves/devcontainer/clean_untagged.sh b/.github/workflows/reserves/devcontainer/clean_untagged.sh new file mode 100755 index 000000000000..87e77dfb6189 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/clean_untagged.sh @@ -0,0 +1,34 @@ +#!/bin/bash +set -e + + +image_names=( + "ci-devcontainer" + "ci/tests/run-args" + "ci/tests/build-args" + "ci/tests/dockerfile-context" + "ci/tests/feature-docker-from-docker" + "ci/tests/docker-from-docker-non-root" + "ci/tests/docker-from-docker-root" + "ci/tests/skip-user-update" +) + +for image_name in ${image_names[@]}; +do + echo "Checking for untagged versions for $image_name" + escaped_image_name=$(echo ${image_name} | sed "s/\//%2f/g") + response=$(curl -s -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/orgs/devcontainers/packages/container/$escaped_image_name/versions?per_page=100") + message=$(echo "$response" | jq -r ".message?") + if [[ -z "$message" || "$message" == "null" ]]; then + version_ids=$(echo "$response" | jq -r ".[] | select(.metadata.container.tags | length ==0) | .id") + for version_id in ${version_ids[@]}; + do + echo -e "\tDeleting version '$version_id' for '$image_name:$tag' ..." + curl -s -X DELETE -H "Authorization: Bearer $GITHUB_TOKEN" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/orgs/devcontainers/packages/container/$escaped_image_name/versions/$version_id" + done + else + echo "Error: $message" + fi +done + +echo "Done." diff --git a/.github/workflows/reserves/devcontainer/codeql-analysis.yml b/.github/workflows/reserves/devcontainer/codeql-analysis.yml new file mode 100644 index 000000000000..fbc24e27c4bd --- /dev/null +++ b/.github/workflows/reserves/devcontainer/codeql-analysis.yml @@ -0,0 +1,71 @@ +# For most projects, this workflow file will not need changing; you simply need +# to commit it to your repository. +# +# You may wish to alter this file to override the set of languages analyzed, +# or to provide custom queries or build logic. +# +# ******** NOTE ******** +# We have attempted to detect the languages in your repository. Please check +# the `language` matrix defined below to confirm you have the correct set of +# supported CodeQL languages. +# +name: "CodeQL" + +on: + push: + branches: [ main ] + pull_request: + # The branches below must be a subset of the branches above + branches: [ main ] + schedule: + - cron: '41 9 * * 0' + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ 'javascript' ] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ] + # Learn more: + # https://docs.github.com/en/free-pro-team@latest/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#changing-the-languages-that-are-analyzed + + steps: + - name: Checkout repository + uses: actions/checkout@v3 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v1 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + # queries: ./path/to/local/query, your-org/your-repo/queries@main + + # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + uses: github/codeql-action/autobuild@v1 + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 https://git.io/JvXDl + + # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines + # and modify them (or add more) to build your code if your project + # uses a compiled language + + #- run: | + # make bootstrap + # make release + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v1 diff --git a/.github/workflows/reserves/devcontainer/pr-bot.yml b/.github/workflows/reserves/devcontainer/pr-bot.yml new file mode 100644 index 000000000000..7813843d7809 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/pr-bot.yml @@ -0,0 +1,81 @@ +--- +name: pr-bot +# This workflow is triggered on PR comments + + +on: + issue_comment: + types: [created] # only run on new comments + # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#issue_comment + # https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#issue_comment + +permissions: + contents: write + packages: write + +jobs: + pr_comment: + name: PR comment + # https://docs.github.com/en/graphql/reference/enums#commentauthorassociation + # (and https://docs.github.com/en/rest/reference/issues#comments) + + # only allow commands where: + # - the comment is on a PR + # - the commenting user has write permissions (i.e. is OWNER or COLLABORATOR) + if: ${{ github.event.issue.pull_request }} + runs-on: ubuntu-latest + outputs: + command: ${{ steps.check_command.outputs.command }} + prRef: ${{ steps.check_command.outputs.prRef }} + prNumber: ${{ steps.check_command.outputs.prNumber }} + prHeadSha: ${{ steps.check_command.outputs.prHeadSha }} + steps: + # Ensure we have the script file for the github-script action to use + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + + # Determine whether the comment is a command + - id: check_command + name: Check for a command using GitHub script + uses: actions/github-script@v6 + with: + script: | + const script = require('./.github/scripts/build.js') + await script.getCommandFromComment({core, context, github}); + + - name: Output PR details + run: | + echo "PR Details" + echo "==========" + echo "command : ${{ steps.check_command.outputs.command }}" + echo "prNumber : ${{ steps.check_command.outputs.prNumber }}" + echo "prRef : ${{ steps.check_command.outputs.prRef }}" + echo "prHeadSha : ${{ steps.check_command.outputs.prHeadSha }}" + + # If we are skipping the tests and don't run the actual deploy (see the run_test job below) + # we won't receive a check-run status, and will have to send it "manually" + - name: Bypass E2E check-runs status + if: ${{ steps.check_command.outputs.command == 'test-force-approve' }} + uses: LouisBrunner/checks-action@v1.1.1 + with: + token: ${{ secrets.GITHUB_TOKEN }} + # the name must be identical to the one received by the real job + sha: ${{ steps.check_command.outputs.prHeadSha }} + name: "Build, test, publish / All succeeded" + status: "completed" + conclusion: "success" + + run_test: + needs: [pr_comment] + if: ${{ needs.pr_comment.outputs.command == 'run-tests' }} + name: build-test + uses: ./.github/workflows/ci_common.yml + with: + prNumber: ${{ needs.pr_comment.outputs.prNumber }} + prRef: ${{ needs.pr_comment.outputs.prRef }} + prHeadSha: ${{ needs.pr_comment.outputs.prHeadSha }} + release: false + secrets: + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} diff --git a/.github/workflows/reserves/devcontainer/pr-closed.yml b/.github/workflows/reserves/devcontainer/pr-closed.yml new file mode 100644 index 000000000000..709611a5a9ed --- /dev/null +++ b/.github/workflows/reserves/devcontainer/pr-closed.yml @@ -0,0 +1,26 @@ +name: pr_closed + +on: + pull_request: + types: [closed] + +# https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#issue_comment +# https://docs.github.com/en/developers/webhooks-and-events/webhooks/webhook-events-and-payloads#pull_request + +jobs: + + pr_closed: + name: PR Closed + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + + - name: Clean GitHub container images for PR + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + PR_IMAGE_TAG: pr-${{ github.event.pull_request.number }} + run: + ./.github/workflows/clean_tags.sh --tag $PR_IMAGE_TAG diff --git a/.github/workflows/reserves/devcontainer/pr_auto.yml b/.github/workflows/reserves/devcontainer/pr_auto.yml new file mode 100644 index 000000000000..6412806c32e0 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/pr_auto.yml @@ -0,0 +1,26 @@ +--- +name: pr_auto +# This workflow is triggered automatically for pull requests and runs tests without credentials +# I.e. doesn't validate image caching, doesn't perform tests in Azure DevOps. + +on: + pull_request: + branches: [ main ] + paths-ignore: + - README.md + - 'docs/**' + +permissions: + contents: write + packages: write + +jobs: + build-test: + name: "Build, test, publish" + uses: ./.github/workflows/ci_common.yml + with: + prNumber: ${{ github.event.pull_request.number }} + release: false + runFullTests: false + secrets: + AZDO_TOKEN: ${{ secrets.AZDO_TOKEN }} diff --git a/.github/workflows/reserves/devcontainer/untagged-image-cleanup.yml b/.github/workflows/reserves/devcontainer/untagged-image-cleanup.yml new file mode 100644 index 000000000000..4eadfd156034 --- /dev/null +++ b/.github/workflows/reserves/devcontainer/untagged-image-cleanup.yml @@ -0,0 +1,23 @@ +name: untagged_image_cleanup + +on: # yamllint disable-line rule:truthy + schedule: + - cron: "4 18 * * *" + workflow_dispatch: + +jobs: + + clean_images: + name: + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + persist-credentials: false + + - name: Clean untagged GitHub container images + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: + ./.github/workflows/clean_untagged.sh diff --git a/.github/workflows/reserves/distribution.yml b/.github/workflows/reserves/distribution.yml new file mode 100644 index 000000000000..152c7918ccc0 --- /dev/null +++ b/.github/workflows/reserves/distribution.yml @@ -0,0 +1,128 @@ +name: distribution + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + push: + branches: + - 'main' + - 'release/*' + tags: + - 'v*' + pull_request: + +env: + DOCKERHUB_SLUG: distribution/distribution + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + go: + - 1.18 + - 1.19 + steps: + - + name: Checkout + uses: actions/checkout@v3 + - + name: Set up Go + uses: actions/setup-go@v3 + with: + go-version: ${{ matrix.go }} + - + name: Test + run: | + make coverage + - + name: Codecov + uses: codecov/codecov-action@v3 + with: + directory: ./ + + build: + runs-on: ubuntu-latest + needs: + - test + steps: + - + name: Checkout + uses: actions/checkout@v3 + with: + fetch-depth: 0 + - + name: Docker meta + id: meta + uses: docker/metadata-action@v4 + with: + images: | + ${{ env.DOCKERHUB_SLUG }} + ### versioning strategy + ### push semver tag v3.2.1 on main (default branch) + # distribution/distribution:3.2.1 + # distribution/distribution:3.2 + # distribution/distribution:3 + # distribution/distribution:latest + ### push semver prelease tag v3.0.0-beta.1 on main (default branch) + # distribution/distribution:3.0.0-beta.1 + ### push on main + # distribution/distribution:edge + tags: | + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=semver,pattern={{major}} + type=ref,event=pr + type=edge + labels: | + org.opencontainers.image.title=Distribution + org.opencontainers.image.description=The toolkit to pack, ship, store, and deliver container content + - + name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + - + name: Login to DockerHub + if: github.event_name != 'pull_request' + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - + name: Build artifacts + uses: docker/bake-action@v2 + with: + targets: artifact-all + - + name: Move artifacts + run: | + mv ./bin/**/* ./bin/ + - + name: Upload artifacts + uses: actions/upload-artifact@v3 + with: + name: registry + path: ./bin/* + if-no-files-found: error + - + name: Build image + uses: docker/bake-action@v2 + with: + files: | + ./docker-bake.hcl + ${{ steps.meta.outputs.bake-file }} + targets: image-all + push: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }} + - + name: GitHub Release + uses: softprops/action-gh-release@v1 + if: startsWith(github.ref, 'refs/tags/') + with: + draft: true + files: | + bin/*.tar.gz + bin/*.sha256 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/e2e-tests.yml b/.github/workflows/reserves/e2e-tests.yml new file mode 100644 index 000000000000..332eebccb62f --- /dev/null +++ b/.github/workflows/reserves/e2e-tests.yml @@ -0,0 +1,483 @@ +name: e2e tests + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + test-setup-multiple-versions: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 2.2.402, 3.1.404 and 3.0.x + uses: ./ + with: + dotnet-version: | + 2.2.402 + 3.1.404 + 3.0.x + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2.402$", "^3.1.404$", "^3.0" + + test-setup-full-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # Side-by-side install of 2.2 and 3.1 used for the test project + - name: Setup dotnet 2.2.402 + uses: ./ + with: + dotnet-version: 2.2.402 + - name: Setup dotnet 3.1.201 + uses: ./ + with: + dotnet-version: 3.1.201 + # We are including this variable to force the generation of the nuget config file to verify that it is created in the correct place + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1.201$", "^2.2.402$" -CheckNugetConfig + + test-setup-without-patch-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # 2.0, 3.0, 5.0 needs to be in single quotes to interpret as a string instead of as an integer + - name: Setup dotnet '3.1' + uses: ./ + with: + dotnet-version: '3.1' + - name: Setup dotnet '2.2' + uses: ./ + with: + dotnet-version: '2.2' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1", "^2.2" + + test-setup-prerelease-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet '3.1.100-preview1-014459' + uses: ./ + with: + dotnet-version: '3.1.100-preview1-014459' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "3.1.100-preview1-014459" + + test-setup-latest-patch-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.x + uses: ./ + with: + dotnet-version: 3.1.x + - name: Setup dotnet 2.2.X + uses: ./ + with: + dotnet-version: 2.2.X + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2", "^3.1" + + test-ABCxx-syntax: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.4xx + uses: ./ + with: + dotnet-version: '6.0.4xx' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^6\.0\.4\d{2}" + + test-setup-with-wildcard: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.* + uses: ./ + with: + dotnet-version: 3.1.* + - name: Setup dotnet 2.2.* + uses: ./ + with: + dotnet-version: 2.2.* + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1", "^2.2" + + test-setup-global-json-specified-and-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '{"sdk":{"version": "2.2.207","rollForward": "latestFeature"}}' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + dotnet-version: 3.1 + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2", "^3.1" + + test-setup-global-json-only: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '{"sdk":{"version": "2.2.207","rollForward": "latestFeature"}}' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2" + + test-global-json-with-comments: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '/* should support comments */ {"sdk":{"version": "2.2.207","rollForward": "latestFeature"}} // should support comments' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2" + + test-setup-with-dotnet-quality: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 7.0 with preview quality + uses: ./ + with: + dotnet-version: '7.0' + dotnet-quality: 'preview' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^7\.0\.\d+-" + + test-setup-with-cache: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macos-latest] + env: + NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Copy NuGet lock file to root + shell: bash + run: cp ./__tests__/e2e-test-csproj/packages.lock.json ./packages.lock.json + - name: Setup .NET Core 3.1 + id: setup-dotnet + uses: ./ + with: + dotnet-version: 3.1 + cache: true + - name: Verify Cache + if: steps.setup-dotnet.outputs.cache-hit == 'true' + shell: bash + run: if [[ -e ${NUGET_PACKAGES} ]]; then exit 0; else exit 1; fi + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1" + + test-setup-with-cache-dependency-path: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macos-latest] + env: + NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup .NET Core 3.1 + id: setup-dotnet + uses: ./ + with: + dotnet-version: 3.1 + cache: true + cache-dependency-path: './__tests__/e2e-test-csproj/packages.lock.json' + - name: Verify Cache + if: steps.setup-dotnet.outputs.cache-hit == 'true' + shell: bash + run: if [[ -e ${NUGET_PACKAGES} ]]; then exit 0; else exit 1; fi + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1" + + test-dotnet-version-output-during-single-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.401 + uses: ./ + id: step1 + with: + dotnet-version: '6.0.401' + + - name: Verify value of the dotnet-version output + shell: pwsh + run: | + $version = & dotnet --version + Write-Host "Installed version: $version" + if (-not ($version -eq '${{steps.step1.outputs.dotnet-version}}')) { throw "Unexpected output value" } + + test-dotnet-version-output-during-multiple-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.401, 5.0.408, 7.0.100-rc.1.22431.12 + uses: ./ + id: step2 + with: + dotnet-version: | + 7.0.100-rc.1.22431.12 + 6.0.401 + 5.0.408 + + - name: Verify value of the dotnet-version output + shell: pwsh + run: | + $version = "7.0.100-rc.1.22431.12" + if (-not ($version -eq '${{steps.step2.outputs.dotnet-version}}')) { throw "Unexpected output value" } + + test-proxy: + runs-on: ubuntu-latest + container: + image: ubuntu:latest + options: --dns 127.0.0.1 + services: + squid-proxy: + image: ubuntu/squid:latest + ports: + - 3128:3128 + env: + https_proxy: http://squid-proxy:3128 + http_proxy: http://squid-proxy:3128 + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Install Powershell + run: | + apt-get update + apt-get install -y wget apt-transport-https software-properties-common + wget -q "https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb" + dpkg -i packages-microsoft-prod.deb + rm packages-microsoft-prod.deb + apt-get update + apt-get install -y powershell + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 6.0 + uses: ./ + with: + dotnet-version: 6.0 + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: | + __tests__/verify-dotnet.ps1 -Patterns "^6.0" -CheckNugetConfig + + test-bypass-proxy: + runs-on: ubuntu-latest + env: + https_proxy: http://no-such-proxy:3128 + no_proxy: github.com,dotnetcli.blob.core.windows.net,download.visualstudio.microsoft.com,api.nuget.org,dotnetcli.azureedge.net + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.201 + uses: ./ + with: + dotnet-version: 3.1.201 + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1.201$" -CheckNugetConfig + + test-sequential-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + lower-version: ['3.1.426'] + higher-version: ['7.0.203'] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # Install one version, use it for something, then switch to next version + - name: Setup dotnet (lower version) + uses: ./ + with: + dotnet-version: ${{ matrix.lower-version }} + - name: Verify dotnet (lower version) + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.lower-version }}$" + - name: Setup dotnet (higher version) + uses: ./ + with: + dotnet-version: ${{ matrix.higher-version }} + - name: Verify dotnet (higher version) + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.lower-version }}$", "^${{ matrix.higher-version }}$" diff --git a/.github/workflows/reserves/e2e.yml b/.github/workflows/reserves/e2e.yml new file mode 100644 index 000000000000..dc449508351c --- /dev/null +++ b/.github/workflows/reserves/e2e.yml @@ -0,0 +1,39 @@ +name: e2e + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + push: + branches: + - 'main' + - 'release/*' + pull_request: + +jobs: + run-e2e-test: + runs-on: ubuntu-latest + steps: + - + name: Checkout + uses: actions/checkout@v3 + with: + fetch-depth: 0 + - + name: Build image + uses: docker/bake-action@v2 + with: + targets: image-local + - + name: Start distribution server + run: | + IP=`hostname -I | awk '{print $1}'` + echo "IP=$IP" >> $GITHUB_ENV + echo '{"insecure-registries" : ["'$IP':5000"]}' | sudo tee /etc/docker/daemon.json + sudo service docker restart + docker run --rm -p 5000:5000 -p 5001:5001 -idt "registry:local" + - + name: Tests + run: | + bash ./tests/push.sh $IP diff --git a/.github/workflows/reserves/elixir-ci.yml b/.github/workflows/reserves/elixir-ci.yml new file mode 100644 index 000000000000..e504a5962e70 --- /dev/null +++ b/.github/workflows/reserves/elixir-ci.yml @@ -0,0 +1,127 @@ +name: Elixir CI + +on: + push: + branches: ['master'] + pull_request: + branches: ['master'] + +env: + MIX_ENV: test + # NOTE: make sure these versions match in Containerfile and .tool-versions + ELIXIR_VERSION_SPEC: "1.15.1" + OTP_VERSION_SPEC: "26.0.2" + +jobs: + compile: + name: Compile + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Set up Elixir + uses: erlef/setup-beam@v1.16 + with: + elixir-version: ${{ env.ELIXIR_VERSION_SPEC }} + otp-version: ${{ env.OTP_VERSION_SPEC }} + - name: Install dependencies + run: mix deps.get + - name: Compile dependencies + run: mix deps.compile + - name: Compile + run: mix compile --warnings-as-errors + + test: + name: Test + runs-on: ubuntu-22.04 + + services: + invoices_control_db_test: + image: postgres:latest + ports: ['5432:5432'] + env: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + POSTGRES_HOST_AUTH_METHOD: 'trust' + POSTGRES_DB: invoices_control_test + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + env: + MIX_ENV: test + + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Set up Elixir + uses: erlef/setup-beam@v1.16 + with: + elixir-version: ${{ env.ELIXIR_VERSION_SPEC }} + otp-version: ${{ env.OTP_VERSION_SPEC }} + - name: Install dependencies + run: mix deps.get + - name: Compile dependencies + run: mix deps.compile + - name: Setup database + env: + MIX_ENV: test + PGHOST: localhost + POSTGRES_PORT: 5432 + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + run: + mix ecto.drop + mix ecto.create + mix ecto.migrate + - name: Run tests + env: + MIX_ENV: test + PGHOST: localhost + POSTGRES_PORT: 5432 + POSTGRES_USER: postgres + POSTGRES_PASSWORD: postgres + run: + mix test + + check-formatted: + name: Check Formatted + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Set up Elixir + uses: erlef/setup-beam@v1.16 + with: + elixir-version: ${{ env.ELIXIR_VERSION_SPEC }} + otp-version: ${{ env.OTP_VERSION_SPEC }} + - name: Install dependencies + run: mix deps.get + - name: Compile dependencies + run: mix deps.compile + - name: Check formatted + run: mix format --check-formatted + + credo: + name: Credo + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Set up Elixir + uses: erlef/setup-beam@v1.16 + with: + elixir-version: ${{ env.ELIXIR_VERSION_SPEC }} + otp-version: ${{ env.OTP_VERSION_SPEC }} + - name: Install dependencies + + run: mix deps.get + - name: Compile dependencies + run: mix deps.compile + - name: Run credo + run: mix credo --strict diff --git a/.github/workflows/reserves/fossa.yml b/.github/workflows/reserves/fossa.yml new file mode 100644 index 000000000000..025260896f09 --- /dev/null +++ b/.github/workflows/reserves/fossa.yml @@ -0,0 +1,22 @@ +name: FOSSA License Scanning + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + - pull_request + - push + +jobs: + scan-license: + runs-on: ubuntu-latest + + steps: + - name: Checkout code + uses: actions/checkout@v3 + + - name: Run FOSSA scan and upload build data + uses: fossa-contrib/fossa-action@v1 + with: + fossa-api-key: cac3dc8d4f2ba86142f6c0f2199a160f diff --git a/.github/workflows/reserves/github-script/check-dist.yml b/.github/workflows/reserves/github-script/check-dist.yml new file mode 100644 index 000000000000..817ad8b17da1 --- /dev/null +++ b/.github/workflows/reserves/github-script/check-dist.yml @@ -0,0 +1,42 @@ +# `dist/index.js` is a special file in Actions. +# When you reference an action with `uses:` in a workflow, +# `index.js` is the code that will run. +# For our project, we generate this file through a build process +# from other source files. +# We need to make sure the checked-in `index.js` actually matches what we expect it to be. +name: Check dist/ + +on: + push: + branches: + - main + pull_request: + workflow_dispatch: + +jobs: + check-dist: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v3 + + - uses: ./.github/actions/install-dependencies + + - name: Rebuild the dist/ directory + run: npm run build + + - name: Compare the expected and actual dist/ directories + run: | + if [ "$(git diff --ignore-space-at-eol dist/ | wc -l)" -gt "0" ]; then + echo "Detected uncommitted changes after build. See status below:" + git diff + exit 1 + fi + id: diff + + # If index.js was different than expected, upload the expected version as an artifact + - uses: actions/upload-artifact@v3 + if: ${{ failure() && steps.diff.conclusion == 'failure' }} + with: + name: dist + path: dist/ diff --git a/.github/workflows/reserves/github-script/ci.yml b/.github/workflows/reserves/github-script/ci.yml new file mode 100644 index 000000000000..dd2216d96e88 --- /dev/null +++ b/.github/workflows/reserves/github-script/ci.yml @@ -0,0 +1,16 @@ +name: CI + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + ci: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + - run: npm run style:check + - run: npm test diff --git a/.github/workflows/reserves/github-script/codeql-analysis.yml b/.github/workflows/reserves/github-script/codeql-analysis.yml new file mode 100644 index 000000000000..f08e127014b0 --- /dev/null +++ b/.github/workflows/reserves/github-script/codeql-analysis.yml @@ -0,0 +1,72 @@ +# For most projects, this workflow file will not need changing; you simply need +# to commit it to your repository. +# +# You may wish to alter this file to override the set of languages analyzed, +# or to provide custom queries or build logic. +# +# ******** NOTE ******** +# We have attempted to detect the languages in your repository. Please check +# the `language` matrix defined below to confirm you have the correct set of +# supported CodeQL languages. +# +name: "CodeQL" + +on: + push: + branches: [ "main" ] + pull_request: + # The branches below must be a subset of the branches above + branches: [ "main" ] + schedule: + - cron: '32 12 * * 3' + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: [ 'javascript' ] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] + # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support + + steps: + - name: Checkout repository + uses: actions/checkout@v3 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v2 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + + # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs + # queries: security-extended,security-and-quality + + + # Autobuild attempts to build any compiled languages (C/C++, C#, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + uses: github/codeql-action/autobuild@v2 + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun + + # If the Autobuild fails above, remove it and uncomment the following three lines. + # modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance. + + # - run: | + # echo "Run, Build Application using script" + # ./location_of_script_within_repo/buildscript.sh + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v2 diff --git a/.github/workflows/reserves/github-script/integration.yml b/.github/workflows/reserves/github-script/integration.yml new file mode 100644 index 000000000000..13494d98814d --- /dev/null +++ b/.github/workflows/reserves/github-script/integration.yml @@ -0,0 +1,325 @@ +name: Integration + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + test-return: + name: 'Integration test: return' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - id: output-set + uses: ./ + with: + script: return core.getInput('input-value') + result-encoding: string + input-value: output + - run: | + echo "- Validating output is produced" + expected="output" + if [[ "${{steps.output-set.outputs.result}}" != "$expected" ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.output-set.outputs.result}}" + exit 1 + fi + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-relative-require: + name: 'Integration test: relative-path require' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - id: relative-require + uses: ./ + with: + script: return require('./package.json').name + result-encoding: string + - run: | + echo "- Validating relative require output" + if [[ "${{steps.relative-require.outputs.result}}" != "github-script" ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.relative-require.outputs.result}}" + exit 1 + fi + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-npm-require: + name: 'Integration test: npm package require' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + - id: npm-require + uses: ./ + with: + script: return require('@actions/core/package.json').name + result-encoding: string + - run: | + echo "- Validating npm require output" + expected="@actions/core" + if [[ "${{steps.npm-require.outputs.result}}" != "$expected" ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.npm-require.outputs.result}}" + exit 1 + fi + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-previews: + name: 'Integration test: GraphQL previews option' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + - id: previews-default + name: Default previews not set + uses: ./ + with: + script: | + const endpoint = github.request.endpoint + return endpoint({url: "/graphql"}).headers.accept + result-encoding: string + - id: previews-set-single + name: Previews set to a single value + uses: ./ + with: + previews: foo + script: | + const endpoint = github.request.endpoint + return endpoint({url: "/graphql"}).headers.accept + result-encoding: string + - id: previews-set-multiple + name: Previews set to comma-separated list + uses: ./ + with: + previews: foo,bar,baz + script: | + const endpoint = github.request.endpoint + return endpoint({url: "/graphql"}).headers.accept + result-encoding: string + - run: | + echo "- Validating previews default" + expected="application/vnd.github.v3+json" + if [[ "${{steps.previews-default.outputs.result}}" != $expected ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.previews-default.outputs.result}}" + exit 1 + fi + echo "- Validating previews set to a single value" + expected="application/vnd.github.foo-preview+json" + if [[ "${{steps.previews-set-single.outputs.result}}" != $expected ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.previews-set-single.outputs.result}}" + exit 1 + fi + echo "- Validating previews set to multiple values" + expected="application/vnd.github.foo-preview+json,application/vnd.github.bar-preview+json,application/vnd.github.baz-preview+json" + if [[ "${{steps.previews-set-multiple.outputs.result}}" != $expected ]]; then + echo $'::error::\u274C' "Expected '$expected', got ${{steps.previews-set-multiple.outputs.result}}" + exit 1 + fi + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-user-agent: + name: 'Integration test: user-agent option' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + - id: user-agent-default + name: Default user-agent not set + uses: ./ + with: + script: | + const endpoint = github.request.endpoint + return endpoint({}).headers['user-agent'] + result-encoding: string + - id: user-agent-set + name: User-agent set + uses: ./ + with: + user-agent: foobar + script: | + const endpoint = github.request.endpoint + return endpoint({}).headers['user-agent'] + result-encoding: string + - id: user-agent-empty + name: User-agent set to an empty string + uses: ./ + with: + user-agent: '' + script: | + const endpoint = github.request.endpoint + return endpoint({}).headers['user-agent'] + result-encoding: string + - run: | + echo "- Validating user-agent default" + expected="actions/github-script octokit-core.js/" + if [[ "${{steps.user-agent-default.outputs.result}}" != "$expected"* ]]; then + echo $'::error::\u274C' "Expected user-agent to start with '$expected', got ${{steps.user-agent-default.outputs.result}}" + exit 1 + fi + echo "- Validating user-agent set to a value" + expected="foobar octokit-core.js/" + if [[ "${{steps.user-agent-set.outputs.result}}" != "$expected"* ]]; then + echo $'::error::\u274C' "Expected user-agent to start with '$expected', got ${{steps.user-agent-set.outputs.result}}" + exit 1 + fi + echo "- Validating user-agent set to an empty string" + expected="octokit-core.js/" + if [[ "${{steps.user-agent-empty.outputs.result}}" != "$expected"* ]]; then + echo $'::error::\u274C' "Expected user-agent to start with '$expected', got ${{steps.user-agent-empty.outputs.result}}" + exit 1 + fi + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-debug: + strategy: + matrix: + environment: ['', 'debug-integration-test'] + environment: ${{ matrix.environment }} + name: "Integration test: debug option (runner.debug mode ${{ matrix.environment && 'enabled' || 'disabled' }})" + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + - id: debug-default + name: Default debug not set + uses: ./ + with: + script: | + const log = github.log + return { + runnerDebugMode: core.isDebug(), + debug: log.debug === console.debug, + info: log.info === console.info + } + - id: debug-true + name: Debug set to true + uses: ./ + with: + debug: true + script: | + const log = github.log + return { + runnerDebugMode: core.isDebug(), + debug: log.debug === console.debug, + info: log.info === console.info + } + - id: debug-false + name: Debug set to false + uses: ./ + with: + debug: false + script: | + const log = github.log + return { + runnerDebugMode: core.isDebug(), + debug: log.debug === console.debug, + info: log.info === console.info + } + - id: evaluate-tests + name: Evaluate test outputs + env: + RDMODE: ${{ runner.debug == '1' }} + run: | + # tests table, pipe separated: label | output | expected + # leading and trailing spaces on any field are trimmed + tests=$(cat << 'TESTS' + Validating debug default |\ + ${{ steps.debug-default.outputs.result }} |\ + {"runnerDebugMode":${{ env.RDMODE }},"debug":${{ env.RDMODE }},"info":${{ env.RDMODE }}} + Validating debug set to true |\ + ${{ steps.debug-true.outputs.result }} |\ + {"runnerDebugMode":${{ env.RDMODE }},"debug":true,"info":true} + Validating debug set to false |\ + ${{ steps.debug-false.outputs.result }} |\ + {"runnerDebugMode":${{ env.RDMODE }},"debug":false,"info":false} + TESTS + ) + + strim() { shopt -s extglob; lt="${1##+( )}"; echo "${lt%%+( )}"; } + while IFS='|' read label output expected; do + label="$(strim "$label")"; output="$(strim "$output")"; expected="$(strim "$expected")" + echo -n "- $label:" + if [[ "$output" != "$expected" ]]; then + echo $'\n::error::\u274C' "Expected '$expected', got '$output'" + exit 1 + fi + echo $' \u2705' + done <<< "$tests" + + echo $'\u2705 Test passed' | tee -a $GITHUB_STEP_SUMMARY + + test-base-url: + name: 'Integration test: base-url option' + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./.github/actions/install-dependencies + + - id: base-url-default + name: API URL with base-url not set + uses: ./ + with: + script: | + const endpoint = github.request.endpoint + return endpoint({}).url + result-encoding: string + + - id: base-url-default-graphql + name: GraphQL URL with base-url not set + uses: ./ + with: + script: | + const endpoint = github.request.endpoint + return endpoint({url: "/graphql"}).url + result-encoding: string + + - id: base-url-set + name: API URL with base-url set + uses: ./ + with: + base-url: https://my.github-enterprise-server.com/api/v3 + script: | + const endpoint = github.request.endpoint + return endpoint({}).url + result-encoding: string + + - id: base-url-set-graphql + name: GraphQL URL with base-url set + uses: ./ + with: + base-url: https://my.github-enterprise-server.com/api/v3 + script: | + const endpoint = github.request.endpoint + return endpoint({url: "/graphql"}).url + result-encoding: string + + - run: | + echo "- Validating API URL default" + expected="https://api.github.com/" + actual="${{steps.base-url-default.outputs.result}}" + if [[ "$expected" != "$actual" ]]; then + echo $'::error::\u274C' "Expected base-url to equal '$expected', got $actual" + exit 1 + fi + echo "- Validating GraphQL URL default" + expected="https://api.github.com/graphql" + actual="${{steps.base-url-default-graphql.outputs.result}}" + if [[ "$expected" != "$actual" ]]; then + echo $'::error::\u274C' "Expected base-url to equal '$expected', got $actual" + exit 1 + fi + echo "- Validating base-url set to a value" + expected="https://my.github-enterprise-server.com/api/v3/" + actual="${{steps.base-url-set.outputs.result}}" + if [[ "$expected" != "$actual" ]]; then + echo $'::error::\u274C' "Expected base-url to equal '$expected', got $actual" + exit 1 + fi + echo "- Validating GraphQL URL with base-url set to a value" + expected="https://my.github-enterprise-server.com/api/v3/graphql" + actual="${{steps.base-url-set-graphql.outputs.result}}" + if [[ "$expected" != "$actual" ]]; then + echo $'::error::\u274C' "Expected base-url to equal '$expected', got $actual" + exit 1 + fi diff --git a/.github/workflows/reserves/github-script/licensed.yml b/.github/workflows/reserves/github-script/licensed.yml new file mode 100644 index 000000000000..7afc7a306f72 --- /dev/null +++ b/.github/workflows/reserves/github-script/licensed.yml @@ -0,0 +1,24 @@ +name: Licensed + +on: + push: + branches: + - main + pull_request: + branches: + - main + +jobs: + test: + runs-on: ubuntu-latest + name: Check licenses + steps: + - uses: actions/checkout@v3 + with: + fetch-depth: 0 # prefer to use a full fetch for licensed workflows + # https://github.com/jonabc/setup-licensed/releases/tag/v1.1.1 + - uses: jonabc/setup-licensed@82c5f4d19e8968efa74a25b132922382c2671fe2 + with: + version: '3.x' + - uses: ./.github/actions/install-dependencies + - run: licensed status diff --git a/.github/workflows/reserves/github-script/pull-request-test.yml b/.github/workflows/reserves/github-script/pull-request-test.yml new file mode 100644 index 000000000000..fb87ec3a9e5d --- /dev/null +++ b/.github/workflows/reserves/github-script/pull-request-test.yml @@ -0,0 +1,45 @@ +name: Pull Request Test + +on: + pull_request: + branches: [main] + types: [opened, synchronize] + +jobs: + pull-request-test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - uses: ./ + with: + script: | + // Get the existing comments. + const {data: comments} = await github.rest.issues.listComments({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.payload.number, + }) + + // Find any comment already made by the bot. + const botComment = comments.find(comment => comment.user.id === 41898282) + const commentBody = "Hello from actions/github-script! (${{ github.sha }})" + + if (context.payload.pull_request.head.repo.full_name !== 'actions/github-script') { + console.log('Not attempting to write comment on PR from fork'); + } else { + if (botComment) { + await github.rest.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: botComment.id, + body: commentBody + }) + } else { + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: context.payload.number, + body: commentBody + }) + } + } diff --git a/.github/workflows/reserves/github-script/stale.yml b/.github/workflows/reserves/github-script/stale.yml new file mode 100644 index 000000000000..7a3b654accff --- /dev/null +++ b/.github/workflows/reserves/github-script/stale.yml @@ -0,0 +1,31 @@ +name: Stale Issues & PRs + +on: + schedule: + - cron: '0 0 * * *' + workflow_dispatch: + +jobs: + mark_stale: + name: Mark issues and PRs as stale + runs-on: ubuntu-latest + steps: + - uses: actions/stale@v3 + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + exempt-issue-labels: Not Stale + exempt-pr-labels: Not Stale + stale-issue-message: > + This issue is stale because it has been open for 60 days with no + activity. Remove the "Stale" label or comment on the issue, or it + will be closed in 7 days. + stale-pr-message: > + This pull request is stale because it has been open for 60 days + with no activity. Remove the "Stale" label or comment on the pull + request, or it will be closed in 7 days. + close-issue-message: > + This issue has been marked as stale and closed due to no activity + on it. + close-pr-message: > + This pull request has been marked as stale and closed due to no + activity on it. diff --git a/.github/workflows/reserves/iree/ARTIFACT_SUMMARY_TEMPLATE.md b/.github/workflows/reserves/iree/ARTIFACT_SUMMARY_TEMPLATE.md new file mode 100644 index 000000000000..69d366dc3d59 --- /dev/null +++ b/.github/workflows/reserves/iree/ARTIFACT_SUMMARY_TEMPLATE.md @@ -0,0 +1,47 @@ +### :link: Artifact Links + +```sh +# Compiler and host tools archive: +export INSTALL_DIR_URL="${INSTALL_DIR_GCS_ARTIFACT}" +# Directory of e2e test artifacts: +export E2E_TEST_ARTIFACTS_DIR_URL="${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}" +# Directory of benchmark tool binaries: +export BENCHMARK_TOOLS_DIR_URL="${BENCHMARK_TOOLS_GCS_ARTIFACT_DIR}" +# Directory of execution benchmark results: +export EXECUTION_BENCHMARK_RESULTS_DIR_URL="${EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR}" +# Compilation benchmark results: +export COMPILATION_BENCHMARK_RESULTS_URL="${COMPILATION_BENCHMARK_RESULTS_GCS_ARTIFACT}" +``` + +You can list `gs://` directories and download files with [gcloud cli]( +https://cloud.google.com/sdk/gcloud/reference/storage). + +
+Click to show common usages + +```sh +# Get compile and run flags of benchmarks +gcloud storage cp -r "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}/benchmark-flag-dump.txt" /tmp/ +``` + +```sh +# Download MLIR input files and command lines +mkdir /tmp/iree_e2e_test_inputs +gcloud storage cp "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}/*" /tmp/iree_e2e_test_inputs +``` + +```sh +# Download all artifacts (MLIR input files and compiled VMFBs) +mkdir /tmp/iree_e2e_test_artifacts +gcloud storage cp -r "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}/*" /tmp/iree_e2e_test_artifacts +``` + +```sh +# Download execution benchmark results +mkdir /tmp/iree_benchmark_results +gcloud storage cp -r "${EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR}/*" /tmp/iree_benchmark_results +``` +
+ +To run benchmarks locally with the CI-built e2e test artifacts, see [IREE Benchmark Suites]( +https://iree.dev/developers/performance/benchmark-suites/#3-fetch-the-benchmark-artifacts). diff --git a/.github/workflows/reserves/iree/benchmark.yml b/.github/workflows/reserves/iree/benchmark.yml new file mode 100644 index 000000000000..9b4b6a6db3bc --- /dev/null +++ b/.github/workflows/reserves/iree/benchmark.yml @@ -0,0 +1,296 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: Benchmark + +on: + workflow_dispatch: + pull_request: + push: + branches: + - main + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +env: + # This needs to be in env instead of the outputs of setup because it contains + # the run attempt and we want that to be the current attempt, not whatever + # attempt the setup step last ran in. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + build_for_benchmarks: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_for_benchmarks') && needs.setup.outputs.benchmark-presets != '' + uses: ./.github/workflows/build_all.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + write-caches: ${{ needs.setup.outputs.write-caches }} + run-tests: false + + build_benchmark_tools: + needs: [setup, build_for_benchmarks] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_benchmark_tools') && needs.setup.outputs.benchmark-presets != '' + uses: ./.github/workflows/build_benchmark_tools.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + install-dir: ${{ needs.build_for_benchmarks.outputs.install-dir }} + install-dir-archive: ${{ needs.build_for_benchmarks.outputs.install-dir-archive }} + install-dir-gcs-artifact: ${{ needs.build_for_benchmarks.outputs.install-dir-gcs-artifact }} + + build_e2e_test_artifacts: + needs: [setup, build_for_benchmarks] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_e2e_test_artifacts') && needs.setup.outputs.benchmark-presets != '' + uses: ./.github/workflows/build_e2e_test_artifacts.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + install-dir: ${{ needs.build_for_benchmarks.outputs.install-dir }} + install-dir-archive: ${{ needs.build_for_benchmarks.outputs.install-dir-archive }} + install-dir-gcs-artifact: ${{ needs.build_for_benchmarks.outputs.install-dir-gcs-artifact }} + benchmark-presets: ${{ needs.setup.outputs.benchmark-presets }} + shard-count: "c2-standard-60=2,default=1" + + test_benchmark_suites: + needs: [setup, build_for_benchmarks, build_e2e_test_artifacts] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_benchmark_suites') && needs.setup.outputs.benchmark-presets != '' + strategy: + matrix: + target: + - platform: linux + arch: riscv_64 + docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + run_scripts: "./build_tools/cmake/build_riscv.sh && ./build_tools/cmake/test_riscv.sh" + - platform: linux + arch: riscv_32 + docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + run_scripts: "./build_tools/cmake/build_riscv.sh && ./build_tools/cmake/test_riscv.sh" + - platform: linux + arch: x86_64 + docker_image: "gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" + run_scripts: "./build_tools/cmake/test_benchmark_suites_on_linux.sh" + # Requires Intel CascadeLake CPU. + host_machine: c2s601t + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - ${{ matrix.target.host_machine || 'cpu' }} # Default to generic x86_64 VM. + env: + PLATFORM: ${{ matrix.target.platform }} + ARCH: ${{ matrix.target.arch }} + DOCKER_IMAGE: ${{ matrix.target.docker_image }} + RUN_SCRIPTS: ${{ matrix.target.run_scripts }} + INSTALL_DIR: ${{ needs.build_for_benchmarks.outputs.install-dir }} + INSTALL_DIR_ARCHIVE: ${{ needs.build_for_benchmarks.outputs.install-dir-archive }} + INSTALL_DIR_GCS_ARTIFACT: ${{ needs.build_for_benchmarks.outputs.install-dir-gcs-artifact }} + TARGET_BUILD_DIR: build-${{ matrix.target.platform }}-${{ matrix.target.arch }} + E2E_TEST_ARTIFACTS_DIR: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Downloading install dir archive" + run: gcloud storage cp "${INSTALL_DIR_GCS_ARTIFACT}" "${INSTALL_DIR_ARCHIVE}" + - name: "Extracting install directory" + run: tar -xf "${INSTALL_DIR_ARCHIVE}" + # TODO(#11136): Only download the needed artifacts instead of everything. + - name: "Downloading e2e test artifacts" + run: | + mkdir -p ${E2E_TEST_ARTIFACTS_DIR} + gcloud storage cp -r "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}/*" "${E2E_TEST_ARTIFACTS_DIR}" + - name: "Build iree-run-module and test benchmark suite modules" + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_TARGET_PLATFORM=${PLATFORM}" \ + --env "IREE_TARGET_ARCH=${ARCH}" \ + --env "IREE_TARGET_BUILD_DIR=${TARGET_BUILD_DIR}" \ + --env "BUILD_PRESET=benchmark-suite-test" \ + --env "IREE_HOST_BIN_DIR=${INSTALL_DIR}/bin" \ + --env "E2E_TEST_ARTIFACTS_DIR=${E2E_TEST_ARTIFACTS_DIR}" \ + "${DOCKER_IMAGE}" \ + bash -euo pipefail -c \ + "${RUN_SCRIPTS}" + + compilation_benchmarks: + needs: [setup, build_e2e_test_artifacts] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'compilation_benchmarks') && needs.setup.outputs.benchmark-presets != '' + uses: ./.github/workflows/benchmark_compilation.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + e2e-test-artifacts-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + e2e-test-artifacts-build-log: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log }} + e2e-test-artifacts-build-log-gcs-artifact: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log-gcs-artifact }} + + execution_benchmarks: + needs: [setup, build_benchmark_tools, build_e2e_test_artifacts] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'execution_benchmarks') && needs.setup.outputs.benchmark-presets != '' + uses: ./.github/workflows/benchmark_execution.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + e2e-test-artifacts-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + benchmark-tools-gcs-artifact-dir: ${{ needs.build_benchmark_tools.outputs.benchmark-tools-gcs-artifact-dir }} + + process_benchmark_results: + needs: [setup, compilation_benchmarks, execution_benchmarks] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'process_benchmark_results') && needs.setup.outputs.benchmark-presets != '' + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + env: + COMPILE_STATS_RESULTS: ${{ needs.compilation_benchmarks.outputs.compile-stats-results }} + COMPILE_STATS_RESULTS_GCS_ARTIFACT: ${{ needs.compilation_benchmarks.outputs.compile-stats-results-gcs-artifact }} + # Empty if no execution benchmark runs. + EXECUTION_BENCHMARK_RESULTS_DIR: ${{ needs.execution_benchmarks.outputs.benchmark-results-dir }} + # Empty if no execution benchmark runs. + EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR: ${{ needs.execution_benchmarks.outputs.benchmark-results-gcs-artifact-dir }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + # We need the full history (and main branch) to generate the report. + fetch-depth: 0 + - name: Downloading compilation benchmark results + run: | + gcloud storage cp \ + "${COMPILE_STATS_RESULTS_GCS_ARTIFACT}" \ + "${COMPILE_STATS_RESULTS}" + - name: Downloading execution benchmark results + id: download-execution-results + # Skip the download if there is no execution benchmark results (e.g. no + # benchmark matches the preset/filter). In such case, no benchmark job + # is run in benchmark_execution.yml and the output variables are empty. + if: env.EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR != '' + run: | + gcloud storage cp -r \ + "${EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR}/benchmark-results-*.json" \ + "${EXECUTION_BENCHMARK_RESULTS_DIR}" + echo "execution-benchmark-results-pattern=${EXECUTION_BENCHMARK_RESULTS_DIR}/benchmark-results-*.json" >> "${GITHUB_OUTPUT}" + - name: Generating comment + if: fromJson(needs.setup.outputs.is-pr) + id: generate-comment + env: + # Wildcard pattern to match all execution benchmark results. Empty if + # execution_benchmarks is skipped, which results in no match. + EXECUTION_BENCHMARK_RESULTS_PATTERN: ${{ steps.download-execution-results.outputs.execution-benchmark-results-pattern }} + IREE_BUILD_URL: https://github.com/iree-org/iree/actions/runs/${{ github.run_id }}/attempts/${{ github.run_attempt }} + PR_NUMBER: ${{ github.event.pull_request.number }} + BENCHMARK_COMMENT_ARTIFACT: benchmark-comment.json + run: | + build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/benchmark-report@sha256:7498c6f32f63f13faf085463cc38656d4297519c824e63e1c99c8c258147f6ff \ + ./build_tools/benchmarks/generate_benchmark_comment.py \ + --verbose \ + --pr_number="${PR_NUMBER}" \ + --pr_committish="${GITHUB_SHA}" \ + --pr_base_branch="origin/${GITHUB_BASE_REF}" \ + --comment_type="benchmark-summary" \ + --build_url="${IREE_BUILD_URL}" \ + --benchmark_files="${EXECUTION_BENCHMARK_RESULTS_PATTERN}" \ + --compile_stats_files="${COMPILE_STATS_RESULTS}" \ + --output="${BENCHMARK_COMMENT_ARTIFACT}" + echo "benchmark-comment-artifact=${BENCHMARK_COMMENT_ARTIFACT}" >> "${GITHUB_OUTPUT}" + - name: Uploading comment artifact + # Due to security reasons, instead of posting the comment to PR, we only + # upload the comment data in presubmit workflow and trigger the posting + # workflow on the main branch. See post_benchmark_comment.yaml + if: fromJson(needs.setup.outputs.is-pr) + env: + BENCHMARK_COMMENT_ARTIFACT: ${{ steps.generate-comment.outputs.benchmark-comment-artifact }} + BENCHMARK_COMMENT_GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.generate-comment.outputs.benchmark-comment-artifact }} + run: | + gcloud storage cp \ + "${BENCHMARK_COMMENT_ARTIFACT}" \ + "${BENCHMARK_COMMENT_GCS_ARTIFACT}" + - name: Uploading results to dashboard + if: github.ref_name == 'main' + env: + EXECUTION_BENCHMARK_RESULTS_PATTERN: ${{ steps.download-execution-results.outputs.execution-benchmark-results-pattern }} + IREE_DASHBOARD_API_TOKEN: ${{ secrets.IREE_DASHBOARD_API_TOKEN }} + run: | + build_tools/github_actions/docker_run.sh \ + --env "IREE_DASHBOARD_API_TOKEN=${IREE_DASHBOARD_API_TOKEN}" \ + gcr.io/iree-oss/benchmark-report@sha256:7498c6f32f63f13faf085463cc38656d4297519c824e63e1c99c8c258147f6ff \ + ./build_tools/benchmarks/upload_benchmarks_to_dashboard.py \ + --verbose \ + --benchmark_files="${EXECUTION_BENCHMARK_RESULTS_PATTERN}" \ + --compile_stats_files="${COMPILE_STATS_RESULTS}" + + ############################################################################## + + # Depends on all the other jobs to provide a single anchor that indicates the + # final status. Status reporting will become more sophisticated in the future + # and we can hopefully avoid the need to explicitly list every single job... + benchmark_summary: + # Even if you have an explicit if condition, you still need to override + # GitHub's default behavior of not running if any dependencies failed. + if: always() + runs-on: ubuntu-20.04 + needs: + - setup + - build_for_benchmarks + + # Benchmark pipeline + - build_benchmark_tools + - build_e2e_test_artifacts + - test_benchmark_suites + - compilation_benchmarks + - execution_benchmarks + - process_benchmark_results + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: Getting failed jobs + id: failed_jobs + run: | + echo '${{ toJson(needs) }}' + FAILED_JOBS="$(echo '${{ toJson(needs) }}' \ + | jq --raw-output \ + 'map_values(select(.result!="success" and .result!="skipped")) | keys | join(",")' \ + )" + echo "failed-jobs=${FAILED_JOBS}" >> $GITHUB_OUTPUT + if [[ "${FAILED_JOBS}" != "" ]]; then + echo "The following jobs failed: ${FAILED_JOBS}" + exit 1 + fi + - name: Show useful artifact links + if: always() + env: + # If the job of an artifact is skipped or failed, we show "NOT_PRESENT". + INSTALL_DIR_GCS_ARTIFACT: ${{ needs.build_for_benchmarks.outputs.install-dir-gcs-artifact || 'NOT_PRESENT' }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir || 'NOT_PRESENT' }} + BENCHMARK_TOOLS_GCS_ARTIFACT_DIR: ${{ needs.build_benchmark_tools.outputs.benchmark-tools-gcs-artifact-dir || 'NOT_PRESENT' }} + EXECUTION_BENCHMARK_RESULTS_GCS_ARTIFACT_DIR: ${{ needs.execution_benchmarks.outputs.benchmark-results-gcs-artifact-dir || 'NOT_PRESENT' }} + COMPILATION_BENCHMARK_RESULTS_GCS_ARTIFACT: ${{ needs.compilation_benchmarks.outputs.compile-stats-results-gcs-artifact || 'NOT_PRESENT' }} + run: | + envsubst < ./.github/workflows/ARTIFACT_SUMMARY_TEMPLATE.md >> "${GITHUB_STEP_SUMMARY}" + - name: Posting to Discord + uses: sarisia/actions-status-discord@v1.14.3 + if: failure() && github.ref_name == 'main' + with: + webhook: ${{ secrets.DISCORD_WEBHOOK }} + description: "The following jobs failed: ${{ steps.failed_jobs.outputs.failed-jobs }}" + url: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/attempts/${{ github.run_attempt }}" diff --git a/.github/workflows/reserves/iree/benchmark_compilation.yml b/.github/workflows/reserves/iree/benchmark_compilation.yml new file mode 100644 index 000000000000..df6e9c333e62 --- /dev/null +++ b/.github/workflows/reserves/iree/benchmark_compilation.yml @@ -0,0 +1,114 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Workflow for running compilation benchmarks. +# It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Benchmark Compilation + +on: + workflow_call: + inputs: + runner-group: + required: true + type: string + runner-env: + required: true + type: string + e2e-test-artifacts-dir: + required: true + type: string + e2e-test-artifacts-gcs-artifact-dir: + required: true + type: string + e2e-test-artifacts-build-log: + required: true + type: string + e2e-test-artifacts-build-log-gcs-artifact: + required: true + type: string + outputs: + compile-stats-results: + description: | + Local path to the compilation benchmark results. + value: ${{ jobs.compilation_benchmarks.outputs.compile-stats-results }} + compile-stats-results-gcs-artifact: + description: | + GCS path to the uploaded compilation benchmark results. + value: ${{ jobs.compilation_benchmarks.outputs.compile-stats-results-gcs-artifact }} + +permissions: + contents: read + +env: + # This duplicates the variable from benchmark.yml. The variable needs to be in env + # instead of the outputs of setup because it contains the run attempt and we + # want that to be the current attempt, not whatever attempt the setup step + # last ran in. It therefore can't be passed in via inputs because the env + # context isn't available there. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +jobs: + compilation_benchmarks: + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - cpu + - os-family=Linux + env: + E2E_TEST_ARTIFACTS_DIR: ${{ inputs.e2e-test-artifacts-dir }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ inputs.e2e-test-artifacts-gcs-artifact-dir }} + E2E_TEST_ARTIFACTS_BUILD_LOG: ${{ inputs.e2e-test-artifacts-build-log }} + E2E_TEST_ARTIFACTS_BUILD_LOG_GCS_ARTIFACT: ${{ inputs.e2e-test-artifacts-build-log-gcs-artifact }} + outputs: + compile-stats-results: ${{ steps.collect.outputs.compile-stats-results }} + compile-stats-results-gcs-artifact: ${{ steps.upload.outputs.compile-stats-results-gcs-artifact }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Downloading assets" + id: "download-assets" + env: + BENCHMARK_CONFIG: ${{ env.E2E_TEST_ARTIFACTS_DIR }}/compilation-benchmark-config.json + BENCHMARK_CONFIG_GCS_ARTIFACT: ${{ env.E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR }}/compilation-benchmark-config.json + run: | + mkdir -p "${E2E_TEST_ARTIFACTS_DIR}" + gcloud storage cp "${BENCHMARK_CONFIG_GCS_ARTIFACT}" "${BENCHMARK_CONFIG}" + gcloud storage cp \ + "${E2E_TEST_ARTIFACTS_BUILD_LOG_GCS_ARTIFACT}" \ + "${E2E_TEST_ARTIFACTS_BUILD_LOG}" + jq -r \ + --arg GCS_ARTIFACT_DIR "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}" \ + '.module_dir_paths | map("\($GCS_ARTIFACT_DIR)/\(.)") | join("\n")' \ + "${BENCHMARK_CONFIG}" | \ + gcloud storage cp -r --read-paths-from-stdin \ + "${E2E_TEST_ARTIFACTS_DIR}" + echo "benchmark-config=${BENCHMARK_CONFIG}" >> "${GITHUB_OUTPUT}" + - name: "Collecting compilation statistics" + id: collect + env: + BENCHMARK_CONFIG: ${{ steps.download-assets.outputs.benchmark-config }} + COMPILE_STATS_RESULTS: benchmark-results/compile-stats-results.json + run: | + mkdir -p benchmark-results + ./build_tools/benchmarks/collect_compilation_statistics.py \ + --e2e_test_artifacts_dir="${E2E_TEST_ARTIFACTS_DIR}" \ + --build_log="${E2E_TEST_ARTIFACTS_BUILD_LOG}" \ + --compilation_benchmark_config="${BENCHMARK_CONFIG}" \ + --output="${COMPILE_STATS_RESULTS}" + echo "compile-stats-results=${COMPILE_STATS_RESULTS}" >> "${GITHUB_OUTPUT}" + - name: "Uploading benchmark results" + id: upload + env: + COMPILE_STATS_RESULTS: ${{ steps.collect.outputs.compile-stats-results }} + COMPILE_STATS_RESULTS_GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.collect.outputs.compile-stats-results }} + run: | + gcloud storage cp \ + "${COMPILE_STATS_RESULTS}" \ + "${COMPILE_STATS_RESULTS_GCS_ARTIFACT}" + echo "compile-stats-results-gcs-artifact=${COMPILE_STATS_RESULTS_GCS_ARTIFACT}" >> "${GITHUB_OUTPUT}" diff --git a/.github/workflows/reserves/iree/benchmark_execution.yml b/.github/workflows/reserves/iree/benchmark_execution.yml new file mode 100644 index 000000000000..baa1c57abbce --- /dev/null +++ b/.github/workflows/reserves/iree/benchmark_execution.yml @@ -0,0 +1,186 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Workflow for running execution benchmarks. +# It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Benchmark execution + +on: + workflow_call: + inputs: + runner-group: + required: true + type: string + runner-env: + required: true + type: string + e2e-test-artifacts-dir: + required: true + type: string + e2e-test-artifacts-gcs-artifact-dir: + required: true + type: string + benchmark-tools-gcs-artifact-dir: + required: true + type: string + outputs: + benchmark-results-dir: + description: | + Local path that stores all benchmark results. + Empty if no benchmark runs. + value: ${{ jobs.run_benchmarks.outputs.benchmark-results-dir }} + benchmark-results-gcs-artifact-dir: + description: | + GCS path that stores all benchmark results. + Empty if no benchmark runs. + value: ${{ jobs.run_benchmarks.outputs.benchmark-results-gcs-artifact-dir }} + +permissions: + contents: read + +env: + # This duplicates the variable from benchmark.yml. The variable needs to be in env + # instead of the outputs of setup because it contains the run attempt and we + # want that to be the current attempt, not whatever attempt the setup step + # last ran in. It therefore can't be passed in via inputs because the env + # context isn't available there. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + BENCHMARK_CONFIG: ${{ inputs.e2e-test-artifacts-dir }}/execution-benchmark-config.json + BENCHMARK_CONFIG_GCS_ARTIFACT: ${{ inputs.e2e-test-artifacts-gcs-artifact-dir }}/execution-benchmark-config.json + +jobs: + generate_matrix: + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - cpu + - os-family=Linux + outputs: + benchmark-matrix: ${{ steps.generate.outputs.benchmark-matrix }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Generating benchmark matrix" + id: generate + run: | + gcloud storage cp "${BENCHMARK_CONFIG_GCS_ARTIFACT}" "${BENCHMARK_CONFIG}" + # This jq command takes a benchmark config with the following structure: + # + # { "target_device_name" : { + # "host_environment" : { ... }, + # "shards" : [ { ... }, { ... } ] + # }, + # "other_target_device_name" : { ... } + # + # and turns it into a flat list of benchmark jobs: + # + # [ + # { "device_name" : "target_device_name", + # "host_environment" : { ... }, + # "shard" : { index: 0, count: 2 } + # }, + # { "device_name" : "target_device_name", + # "host_environment" : { ... }, + # "shard" : { index: 1, count: 2 } + # }, + # { "device_name" : "other_target_device_name", + # "host_environment" : { ... }, + # "shard" : { index: 0, count: N } + # }, + # ... + # ] + echo benchmark-matrix="$(jq -c '[ . | to_entries[] + | .key as $device_name + | .value.host_environment as $host_environment + | (.value.shards | length) as $count + | .value.shards[] + | {$device_name, $host_environment, shard: {index, $count}} + ]' "${BENCHMARK_CONFIG}")" >> "${GITHUB_OUTPUT}" + + run_benchmarks: + needs: [generate_matrix] + if: needs.generate_matrix.outputs.benchmark-matrix != '[]' + strategy: + # Don't kill other benchmark jobs if one of them failed. + fail-fast: false + # Matrix is dynamically generated by the job generate_matrix. So we only + # runs the benchmarks specified in ${BENCHMARK_CONFIG}. + # All tasks in matrix are seen as a single job in Github CI and the job + # can output a single set of values. + matrix: + benchmark: ${{ fromJSON(needs.generate_matrix.outputs.benchmark-matrix) }} + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - machine-type=${{ matrix.benchmark.device_name }} + env: + DEVICE_NAME: ${{ matrix.benchmark.device_name }} + SHARD_INDEX: ${{ matrix.benchmark.shard.index }} + SHARD_COUNT: ${{ matrix.benchmark.shard.count }} + PLATFORM_ARCH: ${{ matrix.benchmark.host_environment.platform }}-${{ matrix.benchmark.host_environment.architecture }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ inputs.e2e-test-artifacts-gcs-artifact-dir }} + BENCHMARK_RESULTS_DIR: benchmark-results + outputs: + benchmark-results-dir: ${{ env.BENCHMARK_RESULTS_DIR }} + # Ideally this should be defined in env, so it can be used in the upload + # step. But Github CI doesn't allow us to access env.GCS in env. + benchmark-results-gcs-artifact-dir: ${{ env.GCS_DIR }}/${{ env.BENCHMARK_RESULTS_DIR }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Downloading benchmark tools" + id: download-tools + env: + # See `build_benchmark_tools` step for the name format of benchmark tools artifacts. + BENCHMARK_TOOLS_ARCHIVE: ${{ env.PLATFORM_ARCH }}-benchmark-tools.tar + BENCHMARK_TOOLS_GCS_ARTIFACT: ${{ inputs.benchmark-tools-gcs-artifact-dir }}/${{ env.PLATFORM_ARCH }}-benchmark-tools.tar + run: | + gcloud storage cp "${BENCHMARK_TOOLS_GCS_ARTIFACT}" "${BENCHMARK_TOOLS_ARCHIVE}" + echo "benchmark-tools-archive=${BENCHMARK_TOOLS_ARCHIVE}" >> "${GITHUB_OUTPUT}" + - name: "Downloading benchmark assets" + id: download-assets + run: | + gcloud storage cp "${BENCHMARK_CONFIG_GCS_ARTIFACT}" "${BENCHMARK_CONFIG}" + echo "benchmark-config=${BENCHMARK_CONFIG}" >> "${GITHUB_OUTPUT}" + - name: "Unpacking benchmark tools" + id: unpack-tools + env: + BENCHMARK_TOOLS_ARCHIVE: ${{ steps.download-tools.outputs.benchmark-tools-archive }} + # See `build_benchmark_tools` step for the name format of benchmark tools directory. + BENCHMARK_TOOLS_DIR: ${{ env.PLATFORM_ARCH }}-benchmark-tools-dir + run: | + tar -xf ${BENCHMARK_TOOLS_ARCHIVE} + echo "benchmark-tools-dir=${BENCHMARK_TOOLS_DIR}/build/tools" >> "${GITHUB_OUTPUT}" + - name: "Determine Shard Suffix" + id: sharding + run: | + if (( SHARD_COUNT > 1 )); then + echo "suffix=$(printf -- "-%02d-of-%02d" "${SHARD_INDEX}" "${SHARD_COUNT}")" >> "${GITHUB_OUTPUT}" + else + echo "suffix=" >> "${GITHUB_OUTPUT}" + fi + - name: "Running benchmarks" + env: + IREE_EXECUTION_BENCHMARK_CONFIG: ${{ steps.download-assets.outputs.benchmark-config }} + IREE_DOCKER_WRAPPER: ./build_tools/github_actions/docker_run.sh + IREE_BENCHMARK_TOOLS_DIR: ${{ steps.unpack-tools.outputs.benchmark-tools-dir }} + IREE_TARGET_DEVICE_NAME: ${{ env.DEVICE_NAME }} + IREE_SHARD_INDEX: ${{ matrix.benchmark.shard.index }} + IREE_BENCHMARK_RESULTS: ${{ env.BENCHMARK_RESULTS_DIR }}/benchmark-results-${{ matrix.benchmark.device_name }}${{ steps.sharding.outputs.suffix }}.json + run: | + mkdir -p ${BENCHMARK_RESULTS_DIR} + export IREE_E2E_TEST_ARTIFACTS_DIR="${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR/gs:\/\//https://storage.googleapis.com/}" + ./build_tools/benchmarks/run_benchmarks.sh + - name: "Uploading benchmark results" + run: gcloud storage cp -r "${BENCHMARK_RESULTS_DIR}" "${GCS_DIR}/" diff --git a/.github/workflows/reserves/iree/benchmark_large.yml b/.github/workflows/reserves/iree/benchmark_large.yml new file mode 100644 index 000000000000..f512c9f6d4ea --- /dev/null +++ b/.github/workflows/reserves/iree/benchmark_large.yml @@ -0,0 +1,138 @@ +# Copyright 2023 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Executes large benchmarks that are expected to take multple hours. + +name: Benchmark Large + +on: + schedule: + # Scheduled to run at 09:00 UTC and 21:00 UTC. + - cron: "0 09,21 * * *" + workflow_dispatch: + inputs: + shard-count: + description: | + A device-name to integer mapping as a comma separated list. + Allows to assign a distinct shard count for each target device. + The reserved keyword `default` assigns a shard count to all target devices + that are not explicitly listed. + # Please keep this default value in sync with the jobs.build_e2e_test_artifacts.with.shard-count field below + default: a2-highgpu-1g=1,c2-standard-60=2,default=1 + type: string + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +env: + # This needs to be in env instead of the outputs of setup because it contains + # the run attempt and we want that to be the current attempt, not whatever + # attempt the setup step last ran in. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +# Jobs are organized into groups and topologically sorted by dependencies +jobs: + setup: + uses: ./.github/workflows/setup.yml + + build_all: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_all') + uses: ./.github/workflows/build_all.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + write-caches: ${{ needs.setup.outputs.write-caches }} + run-tests: false + + build_benchmark_tools: + needs: [setup, build_all] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_benchmark_tools') + uses: ./.github/workflows/build_benchmark_tools.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + install-dir: ${{ needs.build_all.outputs.install-dir }} + install-dir-archive: ${{ needs.build_all.outputs.install-dir-archive }} + install-dir-gcs-artifact: ${{ needs.build_all.outputs.install-dir-gcs-artifact }} + + build_e2e_test_artifacts: + needs: [setup, build_all] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_e2e_test_artifacts') + uses: ./.github/workflows/build_e2e_test_artifacts.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + install-dir: ${{ needs.build_all.outputs.install-dir }} + install-dir-archive: ${{ needs.build_all.outputs.install-dir-archive }} + install-dir-gcs-artifact: ${{ needs.build_all.outputs.install-dir-gcs-artifact }} + benchmark-presets: cuda-large,comp-stats-large,x86_64-large + build-default-benchmark-suites: 0 + # Please keep the shard count default value in sync with on.workflow_dispatch.shard-count.default + shard-count: ${{ inputs && inputs.shard_count || 'a2-highgpu-1g=1,c2-standard-60=2,default=1' }} + + compilation_benchmarks: + needs: [setup, build_e2e_test_artifacts] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'compilation_benchmarks') + uses: ./.github/workflows/benchmark_compilation.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + e2e-test-artifacts-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + e2e-test-artifacts-build-log: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log }} + e2e-test-artifacts-build-log-gcs-artifact: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log-gcs-artifact }} + + execution_benchmarks: + needs: [setup, build_benchmark_tools, build_e2e_test_artifacts] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'execution_benchmarks') + uses: ./.github/workflows/benchmark_execution.yml + with: + # env.GCS_DIR is also duplicated in this workflow. See the note there on + # why this is. + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + e2e-test-artifacts-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: ${{ needs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + benchmark-tools-gcs-artifact-dir: ${{ needs.build_benchmark_tools.outputs.benchmark-tools-gcs-artifact-dir }} + + benchmark_summary: + # Even if you have an explicit if condition, you still need to override + # GitHub's default behavior of not running if any dependencies failed. + if: always() + runs-on: ubuntu-20.04 + needs: + - setup + - build_all + - build_benchmark_tools + - build_e2e_test_artifacts + - compilation_benchmarks + - execution_benchmarks + steps: + - name: Getting failed jobs + id: failed_jobs + run: | + echo '${{ toJson(needs) }}' + FAILED_JOBS="$(echo '${{ toJson(needs) }}' \ + | jq --raw-output \ + 'map_values(select(.result!="success" and .result!="skipped")) | keys | join(",")' \ + )" + echo "failed-jobs=${FAILED_JOBS}" >> $GITHUB_OUTPUT + if [[ "${FAILED_JOBS}" != "" ]]; then + echo "The following jobs failed: ${FAILED_JOBS}" + exit 1 + fi + - name: Posting to Discord + uses: sarisia/actions-status-discord@v1.14.3 + if: failure() && github.ref_name == 'main' + with: + webhook: ${{ secrets.DISCORD_WEBHOOK }} + description: "The following jobs failed: ${{ steps.failed_jobs.outputs.failed-jobs }}" diff --git a/.github/workflows/reserves/iree/benchmark_trigger.yml b/.github/workflows/reserves/iree/benchmark_trigger.yml new file mode 100644 index 000000000000..9fc3de193f5a --- /dev/null +++ b/.github/workflows/reserves/iree/benchmark_trigger.yml @@ -0,0 +1,144 @@ +# Copyright 2023 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: Benchmark Trigger + +# Trigger benchmark CI workflows when benchmark PR labels are added. +# When a benchmark label is added, this cancels and then re-runs the Benchmark +# workflow. When the workflow is re-run, it will pick up the latest PR +# description and labels. See +# https://github.com/iree-org/iree/issues/10042#issuecomment-1449250094 for +# more background. +# +# This workflow is running on the base branch the PR targets. + +on: + pull_request_target: + types: + - labeled + +env: + PR_NUMBER: ${{ github.event.pull_request.number }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + # Target workflow that runs the benchmarks. + WORKFLOW_NAME: Benchmark + BENCHMARK_LABEL_PREFIX: "benchmarks:" + +jobs: + # Precondition check is a separate step because we can't put the concurrency + # constraint on it; otherwise the irrelevant labeling events will cancel the + # events that meet the preconditions. Even with cancel-in-progress = false, + # the queued events will still be cancelled. + precondition: + runs-on: ubuntu-20.04 + outputs: + found-label: ${{ steps.precondition.outputs.found-label }} + triggered-at: ${{ steps.precondition.outputs.triggered-at }} + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + steps: + - name: "Checking labels" + id: precondition + run: | + echo "found-label=$(jq \ + --arg label_prefix ${BENCHMARK_LABEL_PREFIX} \ + '.label.name | startswith($label_prefix)' \ + ${GITHUB_EVENT_PATH})" >> "${GITHUB_OUTPUT}" + # pull_request_target event doesn't have the workflow start time. Get + # the approximate start time at the beginning. + echo "triggered-at=$(date +%s)" >> "${GITHUB_OUTPUT}" + + trigger: + needs: precondition + if: fromJSON(needs.precondition.outputs.found-label) + runs-on: ubuntu-20.04 + # Required for cancel and rerun APIs. + permissions: + actions: write + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + concurrency: + # Only allows a single trigger to run for a PR concurrently. + # Timestamp-based check below makes sure we don't rerun benchmark twice + # when multiple label events happen the same time. We don't use + # `cancel-in-progress` to avoid that because cancelled jobs are + # considered as failures and show red on Github UI. + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + steps: + - name: "Checking out repository" + # This checkouts from the base branch instead of the pull request. See + # https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request_target + uses: actions/checkout@v4.1.7 + + - name: "Finding the previous workflow run" + id: find-workflow + env: + RUN_JSON: workflow-run.json + run: | + gh api "/repos/${GITHUB_REPOSITORY}/actions/runs" --method GET -F head_sha="${HEAD_SHA}" \ + | jq --arg name "${WORKFLOW_NAME}" '.workflow_runs | map(select(.name == $name))[0]' \ + > "${RUN_JSON}" + echo "workflow-url=$(jq '.url' ${RUN_JSON})" >> "${GITHUB_OUTPUT}" + echo "run-status=$(jq --raw-output '.status' ${RUN_JSON})" >> "${GITHUB_OUTPUT}" + echo "run-started-at=$(jq --raw-output '.run_started_at' ${RUN_JSON})" >> "${GITHUB_OUTPUT}" + + - name: "Checking if the workflow has been rerun" + id: check + if: fromJSON(steps.find-workflow.outputs.workflow-url) != null + env: + RUN_STARTED_AT: ${{ steps.find-workflow.outputs.run-started-at }} + TRIGGERED_AT: ${{ needs.precondition.outputs.triggered-at }} + run: | + # If the latest workflow run started after the trigger event, it means + # the workflow has been rerun and picked up the new labels. Skip rerun + # in this case. + RUN_STARTED_AT_EPOCH="$(date --date="${RUN_STARTED_AT}" +%s)" + if (( RUN_STARTED_AT_EPOCH < TRIGGERED_AT )); then + SHOULD_RERUN="true" + else + SHOULD_RERUN="false" + fi + echo "should-rerun=${SHOULD_RERUN}" >> "${GITHUB_OUTPUT}" + + cat <> "${GITHUB_OUTPUT}" + + - name: "Rerequesting the workflow" + if: | + fromJSON(steps.check.outputs.should-rerun) && + steps.get-sha.outputs.latest-sha == env.HEAD_SHA + env: + WORKFLOW_RUN_URL: ${{ fromJSON(steps.find-workflow.outputs.workflow-url) }} + run: | + gh api "${WORKFLOW_RUN_URL}/rerun" --method POST diff --git a/.github/workflows/reserves/iree/build_all.yml b/.github/workflows/reserves/iree/build_all.yml new file mode 100644 index 000000000000..7e45b5d309a7 --- /dev/null +++ b/.github/workflows/reserves/iree/build_all.yml @@ -0,0 +1,124 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Workflow for building IREE. It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Build All + +on: + workflow_call: + inputs: + runner-group: + required: true + type: string + runner-env: + required: true + type: string + write-caches: + required: true + type: string + run-tests: + required: true + type: boolean + outputs: + install-dir: + description: Local install directory path. + value: ${{ jobs.build_all.outputs.install-dir }} + install-dir-archive: + description: Name of the zipped install directory. + value: ${{ jobs.build_all.outputs.install-dir-archive }} + install-dir-gcs-artifact: + description: GCS path to the uploaded install archive. + value: ${{ jobs.build_all.outputs.install-dir-gcs-artifact }} + install-dir-gcs-url: + description: GCS URL of the uploaded install archive. + value: ${{ jobs.build_all.outputs.install-dir-gcs-url }} + +permissions: + contents: read + +env: + # This duplicates the variables from other workflows. The variables need to be + # in env instead of the outputs of setup because it contains the run attempt + # and we want that to be the current attempt, not whatever attempt the setup + # step last ran in. They therefore can't be passed in via inputs because the + # env context isn't available there. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + GCS_URL: https://storage.googleapis.com/iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +jobs: + build_all: + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - cpu + - os-family=Linux + env: + BUILD_DIR: full-build-dir + INSTALL_DIR: full-build-dir/install + outputs: + install-dir: ${{ env.INSTALL_DIR }} + # Pass the install directory as output for other jobs to use. + # TODO(#16203): replace this with iree-dist packages + # The install directory provides a similar set of files and existing + # jobs are already configured to use the build directory, so this is + # an intermediate step towards a fully package-based CI setup. + install-dir-archive: ${{ steps.install-archive.outputs.install-dir-archive }} + install-dir-gcs-artifact: ${{ steps.install-upload.outputs.install-dir-gcs-artifact }} + install-dir-gcs-url: ${{ steps.install-upload.outputs.install-dir-gcs-url }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building IREE" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ inputs.write-caches }} + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" \ + --env "IREE_BUILD_SETUP_PYTHON_VENV=${BUILD_DIR}/.venv" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_all.sh "${BUILD_DIR}" + # TODO(scotttodd): trim build/tests to exclude integration tests (tests/ folder, check tests) + # that coverage should move to a test_cpu job (matrix with test_gpu?) + - name: "Testing IREE" + if: inputs.run-tests + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}" + - name: "Testing iree-dialects" + if: inputs.run-tests + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/test_iree_dialects.sh "${BUILD_DIR}" + + # TODO(#16203): If this (or iree-dist) is small enough, just use GitHub + # artifacts instead of `tar` commands and GCS + - name: "Creating install dir archive" + id: install-archive + env: + INSTALL_DIR_ARCHIVE: install_dir.tar.zst + run: | + tar -I 'zstd -T0' \ + -cf ${INSTALL_DIR_ARCHIVE} ${INSTALL_DIR} + echo "install-dir-archive=${INSTALL_DIR_ARCHIVE}" >> "${GITHUB_OUTPUT}" + - name: "Uploading install dir archive" + id: install-upload + env: + INSTALL_DIR_ARCHIVE: ${{ steps.install-archive.outputs.install-dir-archive }} + INSTALL_DIR_GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.install-archive.outputs.install-dir-archive }} + INSTALL_DIR_GCS_URL: ${{ env.GCS_URL }}/${{ steps.install-archive.outputs.install-dir-archive }} + run: | + gcloud storage cp "${INSTALL_DIR_ARCHIVE}" "${INSTALL_DIR_GCS_ARTIFACT}" + echo "install-dir-gcs-artifact=${INSTALL_DIR_GCS_ARTIFACT}" >> "${GITHUB_OUTPUT}" + echo "install-dir-gcs-url=${INSTALL_DIR_GCS_URL}" >> "${GITHUB_OUTPUT}" diff --git a/.github/workflows/reserves/iree/build_benchmark_tools.yml b/.github/workflows/reserves/iree/build_benchmark_tools.yml new file mode 100644 index 000000000000..820c2831af51 --- /dev/null +++ b/.github/workflows/reserves/iree/build_benchmark_tools.yml @@ -0,0 +1,132 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Workflow for building benchmark tools. +# It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Build Benchmark Tools + +on: + workflow_call: + inputs: + runner-group: + required: true + type: string + runner-env: + required: true + type: string + install-dir: + required: true + type: string + install-dir-archive: + required: true + type: string + install-dir-gcs-artifact: + required: true + type: string + outputs: + benchmark-tools-gcs-artifact-dir: + description: | + GCS path to benchmark tools archive. + value: ${{ jobs.build_benchmark_tools.outputs.benchmark-tools-gcs-artifact-dir }} + +permissions: + contents: read + +env: + # This duplicates the variable from benchmark.yml. The variable needs to be in env + # instead of the outputs of setup because it contains the run attempt and we + # want that to be the current attempt, not whatever attempt the setup step + # last ran in. It therefore can't be passed in via inputs because the env + # context isn't available there. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +jobs: + build_benchmark_tools: + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - cpu + - os-family=Linux + outputs: + # We can't collect all outputs from the matrix jobs due to Github's + # limitation (https://github.com/orgs/community/discussions/17245). + # Therefore, the output is the GCS directory that stores all benchmark + # tools archives. The following jobs need to construct the archive names + # by themselves and combine with path of GCS directory here to fetch the + # archives. + benchmark-tools-gcs-artifact-dir: ${{ steps.upload.outputs.benchmark-tools-gcs-artifact-dir }} + strategy: + matrix: + target: + - platform: "linux" + arch: "x86_64" + docker_image: "gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a" + # Builds tools on the host and assumes the builder is Linux x86_64. + build_script: "./build_tools/cmake/build_runtime.sh" + - platform: "linux" + arch: "riscv_64" + docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + build_script: "./build_tools/cmake/build_riscv.sh" + - platform: "android" + arch: "armv8.2-a" + docker_image: "gcr.io/iree-oss/android@sha256:66b92a1c920588a73d03316f26025407ea754bab93e8f9bfe40dbf6ed5fe6c7e" + build_script: "./build_tools/cmake/build_android.sh" + env: + PLATFORM: ${{ matrix.target.platform }} + ARCH: ${{ matrix.target.arch }} + DOCKER_IMAGE: ${{ matrix.target.docker_image }} + BUILD_SCRIPT: ${{ matrix.target.build_script }} + BUILD_TOOLS_DIR: ${{ matrix.target.platform }}-${{ matrix.target.arch }}-benchmark-tools-dir + INSTALL_DIR: ${{ inputs.install-dir }} + INSTALL_DIR_ARCHIVE: ${{ inputs.install-dir-archive }} + INSTALL_DIR_GCS_ARTIFACT: ${{ inputs.install-dir-gcs-artifact }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Downloading install dir archive" + run: gcloud storage cp "${INSTALL_DIR_GCS_ARTIFACT}" "${INSTALL_DIR_ARCHIVE}" + - name: "Extracting install directory" + run: tar -xf "${INSTALL_DIR_ARCHIVE}" + - name: "Compiling the benchmark tools" + id: build + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_TARGET_PLATFORM=${PLATFORM}" \ + --env "IREE_TARGET_ARCH=${ARCH}" \ + --env "BUILD_PRESET=benchmark" \ + --env "IREE_HOST_BIN_DIR=${INSTALL_DIR}/bin" \ + "${DOCKER_IMAGE}" "${BUILD_SCRIPT}" "${BUILD_TOOLS_DIR}/build" + - name: "Compiling the benchmark tools with tracing" + id: build-with-tracing + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_TARGET_PLATFORM=${PLATFORM}" \ + --env "IREE_TARGET_ARCH=${ARCH}" \ + --env "BUILD_PRESET=benchmark-with-tracing" \ + --env "IREE_HOST_BIN_DIR=${INSTALL_DIR}/bin" \ + "${DOCKER_IMAGE}" "${BUILD_SCRIPT}" "${BUILD_TOOLS_DIR}/build-traced" + - name: "Creating the benchmark tools archive" + id: archive + env: + BENCHMARK_TOOLS_ARCHIVE: ${{ env.PLATFORM }}-${{ env.ARCH }}-benchmark-tools.tar + run: | + tar -cf "${BENCHMARK_TOOLS_ARCHIVE}" \ + "${BUILD_TOOLS_DIR}"/*/tools/iree-benchmark-module \ + "${BUILD_TOOLS_DIR}"/*/tools/build_config.txt + echo "benchmark-tools-archive=${BENCHMARK_TOOLS_ARCHIVE}" >> "${GITHUB_OUTPUT}" + - name: "Uploading the benchmark tools archive" + id: upload + env: + BENCHMARK_TOOLS_ARCHIVE: ${{ steps.archive.outputs.benchmark-tools-archive }} + BENCHMARK_TOOLS_GCS_ARTIFACT_DIR: ${{ env.GCS_DIR }}/benchmark-tools + run: | + gcloud storage cp "${BENCHMARK_TOOLS_ARCHIVE}" "${BENCHMARK_TOOLS_GCS_ARTIFACT_DIR}/" + echo "benchmark-tools-gcs-artifact-dir=${BENCHMARK_TOOLS_GCS_ARTIFACT_DIR}" >> "${GITHUB_OUTPUT}" diff --git a/.github/workflows/reserves/iree/build_e2e_test_artifacts.yml b/.github/workflows/reserves/iree/build_e2e_test_artifacts.yml new file mode 100644 index 000000000000..99378b517335 --- /dev/null +++ b/.github/workflows/reserves/iree/build_e2e_test_artifacts.yml @@ -0,0 +1,145 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Workflow for building E2E test artifacts. +# It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Build E2E Test Artifacts + +on: + workflow_call: + inputs: + runner-group: + required: true + type: string + runner-env: + required: true + type: string + install-dir: + required: true + type: string + install-dir-archive: + required: true + type: string + install-dir-gcs-artifact: + required: true + type: string + benchmark-presets: + description: | + A comma-separated string of benchmark presets to build. + required: true + type: string + build-default-benchmark-suites: + description: | + Build default benchmark suites for sanity check and e2e model testing + if set to 1. + default: 1 + type: number + shard-count: + description: | + A device-name to integer mapping as a comma separated list. + Allows to assign a distinct shard count for each target device. + The reserved keyword `default` assigns a shard count to all target devices + that are not explicitly listed. + default: default=1 + type: string + outputs: + e2e-test-artifacts-dir: + description: | + Local path that stores compiled test artifacts. + value: ${{ jobs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: + description: | + GCS path to the uploaded test artifacts. + value: ${{ jobs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-gcs-artifact-dir }} + e2e-test-artifacts-build-log: + description: | + Local path that contains the build log. + value: ${{ jobs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log }} + e2e-test-artifacts-build-log-gcs-artifact: + description: | + GCS path to the uploaded build log. + value: ${{ jobs.build_e2e_test_artifacts.outputs.e2e-test-artifacts-build-log-gcs-artifact }} + +env: + # This duplicates the variable from benchmark.yml. The variable needs to be in env + # instead of the outputs of setup because it contains the run attempt and we + # want that to be the current attempt, not whatever attempt the setup step + # last ran in. It therefore can't be passed in via inputs because the env + # context isn't available there. + GCS_DIR: gs://iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +jobs: + build_e2e_test_artifacts: + runs-on: + - self-hosted # must come first + - runner-group=${{ inputs.runner-group }} + - environment=${{ inputs.runner-env }} + - cpu + - os-family=Linux + env: + INSTALL_DIR: ${{ inputs.install-dir }} + INSTALL_DIR_ARCHIVE: ${{ inputs.install-dir-archive }} + INSTALL_DIR_GCS_ARTIFACT: ${{ inputs.install-dir-gcs-artifact }} + outputs: + e2e-test-artifacts-dir: ${{ steps.build.outputs.e2e-test-artifacts-dir }} + e2e-test-artifacts-gcs-artifact-dir: ${{ steps.upload.outputs.e2e-test-artifacts-gcs-artifact-dir }} + e2e-test-artifacts-build-log: ${{ steps.build.outputs.e2e-test-artifacts-build-log }} + e2e-test-artifacts-build-log-gcs-artifact: ${{ steps.upload.outputs.e2e-test-artifacts-build-log-gcs-artifact }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Downloading install dir archive" + run: gcloud storage cp "${INSTALL_DIR_GCS_ARTIFACT}" "${INSTALL_DIR_ARCHIVE}" + - name: "Extracting install directory" + run: tar -xf "${INSTALL_DIR_ARCHIVE}" + - name: "Building e2e test artifacts" + id: build + env: + IREE_BENCHMARK_PRESETS: ${{ inputs.benchmark-presets }} + IREE_BUILD_DEFAULT_BENCHMARK_SUITES: ${{ inputs.build-default-benchmark-suites }} + BUILD_E2E_TEST_ARTIFACTS_DIR: build-e2e-test-artifacts + IREE_SHARD_COUNT: ${{ inputs.shard-count }} + run: | + echo "e2e-test-artifacts-dir=${BUILD_E2E_TEST_ARTIFACTS_DIR}/e2e_test_artifacts" >> "${GITHUB_OUTPUT}" + echo "e2e-test-artifacts-build-log=${BUILD_E2E_TEST_ARTIFACTS_DIR}/.ninja_log" >> "${GITHUB_OUTPUT}" + build_tools/github_actions/docker_run.sh \ + --env "IREE_HOST_BIN_DIR=${INSTALL_DIR}/bin" \ + --env "IREE_BENCHMARK_PRESETS=${IREE_BENCHMARK_PRESETS}" \ + --env "IREE_BUILD_DEFAULT_BENCHMARK_SUITES=${IREE_BUILD_DEFAULT_BENCHMARK_SUITES}" \ + --env "IREE_SHARD_COUNT=${IREE_SHARD_COUNT}" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + build_tools/cmake/build_e2e_test_artifacts.sh \ + "${BUILD_E2E_TEST_ARTIFACTS_DIR}" + - name: "Uploading e2e test artifacts" + id: upload + if: ${{ !cancelled() }} + env: + E2E_TEST_ARTIFACTS_DIR: ${{ steps.build.outputs.e2e-test-artifacts-dir }} + E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR: ${{ env.GCS_DIR }}/e2e-test-artifacts + E2E_TEST_ARTIFACTS_BUILD_LOG: ${{ steps.build.outputs.e2e-test-artifacts-build-log }} + E2E_TEST_ARTIFACTS_BUILD_LOG_GCS_ARTIFACT: ${{ env.GCS_DIR }}/e2e-test-artifacts/ninja_log + run: | + # TODO(#13677): MLIRs of the versioned MLIR source models are fetched + # from GCS and they won't change. We can upload links instead of files + # to save space. + # Uploads all artifacts except for the `*.tflite`, which are the + # source models we imported into MLIR, and empty directories (GCS + # doesn't accept them). + # Not archiving the directory to allow fetching each file as needed + # separately. `find` depth is limited at 1 to exclude the directory + # itself and list all files and subdirectories at the first level. Let + # gcloud cli handle the recursive file traversal. + find "${E2E_TEST_ARTIFACTS_DIR}" -mindepth 1 -maxdepth 1 ! -name "*.tflite" ! -empty \ + | gcloud storage cp --read-paths-from-stdin -r \ + "${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}" + gcloud storage cp "${E2E_TEST_ARTIFACTS_BUILD_LOG}" \ + "${E2E_TEST_ARTIFACTS_BUILD_LOG_GCS_ARTIFACT}" + echo "e2e-test-artifacts-gcs-artifact-dir=${E2E_TEST_ARTIFACTS_GCS_ARTIFACT_DIR}" >> "${GITHUB_OUTPUT}" + echo "e2e-test-artifacts-build-log-gcs-artifact=${E2E_TEST_ARTIFACTS_BUILD_LOG_GCS_ARTIFACT}" >> "${GITHUB_OUTPUT}" diff --git a/.github/workflows/reserves/iree/build_package.yml b/.github/workflows/reserves/iree/build_package.yml new file mode 100644 index 000000000000..25db3faa53c4 --- /dev/null +++ b/.github/workflows/reserves/iree/build_package.yml @@ -0,0 +1,320 @@ +# Builds release packages on supported platforms: +# * main-dist-linux (CMake 'install') +# * py-compiler-pkg (`iree-compiler` Python package) +# * Linux, macOS, Windows +# * All supported Python versions (e.g. 3.9, 3.10, 3.11) +# * py-runtime-pkg (`iree-runtime` Python package) +# * Linux, macOS, Windows +# * All supported Python versions (e.g. 3.9, 3.10, 3.11) +# * py-tf-compiler-tools-pkg (`iree-tools-[tf, tflite]`, pure Python packages) + +name: Build Release Packages + +# Note: this workflow is typically invoked by schedule_candidate_release.yml. +on: + workflow_dispatch: + inputs: + package_suffix: + description: "Suffix to append to package names" + required: false + default: "" + package_version: + description: "Version of the package" + required: true + default: "0.1a1" + release_id: + description: "Release id to upload artifacts to" + default: "" + commit: + description: "Commit to check out" + default: "" + +jobs: + build_packages: + name: "${{ matrix.build-family }} :: Build ${{ matrix.build-package }} Package" + runs-on: ${{ matrix.runs-on }} + continue-on-error: ${{ matrix.experimental }} + strategy: + fail-fast: false + matrix: + include: + # Ubuntu packages. + - runs-on: ubuntu-20.04 + build-family: linux-x86_64 + build-package: main-dist-linux + experimental: false + - runs-on: + [self-hosted, arm64, os-family=Linux, runner-group=postsubmit] + build-family: linux-aarch64 + build-package: main-dist-linux + experimental: true + - runs-on: ubuntu-20.04 + build-family: linux-x86_64 + build-package: py-compiler-pkg + experimental: false + - runs-on: + [self-hosted, arm64, os-family=Linux, runner-group=postsubmit] + build-family: linux-aarch64 + build-package: py-compiler-pkg + experimental: true + - runs-on: ubuntu-20.04 + build-family: linux-x86_64 + build-package: py-runtime-pkg + experimental: false + - runs-on: + [self-hosted, arm64, os-family=Linux, runner-group=postsubmit] + build-family: linux-aarch64 + build-package: py-runtime-pkg + experimental: true + - runs-on: ubuntu-20.04 + build-family: linux-x86_64 + build-package: py-tf-compiler-tools-pkg + experimental: false + + # MacOS packages. + # TODO(scotttodd): build on larger runner when available (self-hosted or GitHub) + # - runs-on: + # - ${{ github.repository == 'iree-org/iree' && 'self-hosted' || 'macos-14' }} + # - os-family=macOS + # - runner-group=postsubmit + - runs-on: macos-14 + build-family: macos + build-package: py-compiler-pkg + experimental: true + - runs-on: macos-14 + build-family: macos + build-package: py-runtime-pkg + experimental: true + + # Windows packages. + # TODO(scotttodd): build on larger runner when available (self-hosted or GitHub) + # - runs-on: + # - ${{ github.repository == 'iree-org/iree' && 'windows-2022-64core' || 'windows-2022'}} + - runs-on: windows-2022 + build-family: windows + build-package: py-compiler-pkg + experimental: true + - runs-on: windows-2022 + build-family: windows + build-package: py-runtime-pkg + experimental: true + + env: + # These are also set in: build_tools/python_deploy/build_linux_packages.sh + MANYLINUX_X86_64_IMAGE: ghcr.io/nod-ai/manylinux_x86_64:main + MANYLINUX_AARCH64_IMAGE: quay.io/pypa/manylinux_2_28_aarch64 + + steps: + - uses: actions/checkout@v4.1.7 + with: + path: "c" # Windows can hit path length limits, so use a short path. + submodules: true + ref: ${{ github.event.inputs.commit }} + + ########################################################################## + # OS specific setup + ########################################################################## + + - name: Install dependencies (Windows) + if: "matrix.build-family == 'windows'" + shell: powershell + run: ./c/build_tools/python_deploy/install_windows_deps.ps1 + - name: "Configure MSVC (Windows)" + if: "matrix.build-family == 'windows'" + uses: ilammy/msvc-dev-cmd@v1.13.0 + + ########################################################################## + # Write version_info.json + # Various tools will read this in order to embed release information. + ########################################################################## + - name: Write version info (release) + if: "matrix.build-family != 'windows'" + shell: bash + run: | + cat << EOF > ./c/version_info.json + { + "package-suffix": "${{ github.event.inputs.package_suffix }}", + "package-version": "${{ github.event.inputs.package_version }}", + "iree-revision": "$(cd ./c && git rev-parse HEAD)" + } + EOF + cat ./c/version_info.json + + - name: Write version info Windows (release) + if: "matrix.build-family == 'windows'" + shell: powershell + run: | + cd c + $verinfoprop = @{ + 'package-suffix'= "${{ github.event.inputs.package_suffix }}" + 'package-version'= "${{ github.event.inputs.package_version }}" + 'iree-revision'= $(git rev-parse HEAD) + } + + $info = New-Object -TypeName PSObject -Prop $verinfoprop + $info = $info | ConvertTo-JSON + $info | Out-File "version_info.json" -Encoding "ASCII" + + ########################################################################## + # Build the main distribution tarball. + # The main distribution consists of the project being built, installed + # and archived. We have to split it per operating system, and Linux + # is special because we build under a manylinux container which gives + # broad compatibility. We use the Python based manylinux containers + # since they come packaged with all dev tools we need. Note that the + # manylinux containers have a default python 2.x with all supported + # python versions under /opt/python (need to add one to the path). It + # is not enough to just invoke it directly because then pip managed + # console scripts (like cmake, ninja) will not be on the path. + ########################################################################## + - name: Main distribution (Linux for x86_64) + if: "matrix.build-package == 'main-dist-linux' && matrix.build-family == 'linux-x86_64'" + shell: bash + run: | + docker run --rm -w=/work \ + -v $PWD:/work \ + "${MANYLINUX_X86_64_IMAGE}" \ + bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./c/build_tools/github_actions/build_dist.py main-dist' + + - name: Main distribution (Linux for Arm64) + if: "matrix.build-package == 'main-dist-linux' && matrix.build-family == 'linux-aarch64'" + shell: bash + run: | + docker run --rm -w=/work \ + -v $PWD:/work \ + "${MANYLINUX_AARCH64_IMAGE}" \ + bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./c/build_tools/github_actions/build_dist.py main-dist' + + ########################################################################## + # py-runtime-pkg + # Builds the iree-runtime wheels. + # One step per OS. + ########################################################################## + + - name: Build runtime wheels (Linux, x86_64 and Arm64) + if: "matrix.build-package == 'py-runtime-pkg' && contains(matrix.build-family, 'linux')" + shell: bash + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-runtime" + output_dir: "${{ github.workspace }}/bindist" + run: | + [ -e ./bindist/* ] && rm ./bindist/* + ./c/build_tools/python_deploy/build_linux_packages.sh + + - name: Build runtime wheels (MacOS) + if: "matrix.build-package == 'py-runtime-pkg' && matrix.build-family == 'macos'" + shell: bash + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-runtime" + output_dir: "${{ github.workspace }}/bindist" + override_python_versions: "3.11 3.12" + run: | + [ -e ./bindist/* ] && rm ./bindist/* + ./c/build_tools/python_deploy/build_macos_packages.sh + + - name: Build runtime wheels (Windows) + if: "matrix.build-package == 'py-runtime-pkg' && matrix.build-family == 'windows'" + shell: powershell + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-runtime" + output_dir: "${{ github.workspace }}/bindist" + override_python_versions: "3.11 3.12" + run: | + if (Test-Path -Path "${{ github.workspace }}/bindist") { + Remove-Item -Path "${{ github.workspace }}/bindist" -Recurse -Force + } + ./c/build_tools/python_deploy/build_windows_packages.ps1 + + ########################################################################## + # py-compiler-pkg + # Builds the iree-compiler wheel. + # One step per OS. + ########################################################################## + - name: Build compiler wheels (Linux, x86_64 and Arm64) + if: "matrix.build-package == 'py-compiler-pkg' && contains(matrix.build-family, 'linux')" + shell: bash + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-compiler" + output_dir: "${{ github.workspace }}/bindist" + run: | + [ -e ./bindist/* ] && rm ./bindist/* + ./c/build_tools/python_deploy/build_linux_packages.sh + + - name: Build compiler wheels (MacOS) + if: "matrix.build-package == 'py-compiler-pkg' && matrix.build-family == 'macos'" + shell: bash + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-compiler" + output_dir: "${{ github.workspace }}/bindist" + override_python_versions: "3.11 3.12" + run: | + [ -e ./bindist/* ] && rm ./bindist/* + ./c/build_tools/python_deploy/build_macos_packages.sh + + - name: Build compiler wheels (Windows) + if: "matrix.build-package == 'py-compiler-pkg' && matrix.build-family == 'windows'" + shell: powershell + env: + package_suffix: ${{ github.event.inputs.package_suffix }} + packages: "iree-compiler" + output_dir: "${{ github.workspace }}/bindist" + override_python_versions: "3.11 3.12" + run: | + if (Test-Path -Path "${{ github.workspace }}/bindist") { + Remove-Item -Path "${{ github.workspace }}/bindist" -Recurse -Force + } + ./c/build_tools/python_deploy/build_windows_packages.ps1 + + ########################################################################## + # TF Compiler Tools + # These wheels are not Python version specific and include only Python + # code (no C/C++), so just build for one examplar python version on Linux. + ########################################################################## + + - name: Build TF Compiler Tools wheels + if: "matrix.build-package == 'py-tf-compiler-tools-pkg'" + shell: bash + run: | + docker run --rm -w=/work \ + -v $PWD:/work \ + "${MANYLINUX_X86_64_IMAGE}" \ + bash -c 'export PATH=/opt/python/cp39-cp39/bin:$PATH; python ./c/build_tools/github_actions/build_dist.py py-tf-compiler-tools-pkg' + + - uses: actions/upload-artifact@v4.3.3 + with: + name: wheels-${{ matrix.build-family }}-${{ matrix.build-package }} + # We upload all wheels (which includes deps so that subsequent + # steps can run without further fetching). + path: ./bindist/* + retention-days: 5 + + # TODO: Upload the tar.bz2 files too when ready + - name: Upload Release Assets + if: github.event.inputs.release_id != '' + id: upload-release-assets + uses: dwenegar/upload-release-assets@5bc3024cf83521df8ebfadf00ad0c4614fd59148 # v1 + env: + GITHUB_TOKEN: ${{ secrets.WRITE_ACCESS_TOKEN }} + with: + release_id: ${{ github.event.inputs.release_id }} + # Only upload iree artifacts. + assets_path: ./bindist/iree*.* + + validate_and_publish: + name: "Trigger validate and publish release" + needs: build_packages + if: github.event.inputs.release_id != '' + runs-on: ubuntu-20.04 + steps: + - name: "Invoke workflow :: Validate and Publish Release" + uses: benc-uk/workflow-dispatch@798e70c97009500150087d30d9f11c5444830385 # v1.2.2 + with: + workflow: Validate and Publish Release + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + ref: "${{ env.tag_name }}" + inputs: '{"release_id": "${{ github.event.inputs.release_id }}", "package_version": "${{ github.event.inputs.package_version }}", "build_run_id": "${{ github.run_id }}"}' diff --git a/.github/workflows/reserves/iree/bump_torch_mlir.yml b/.github/workflows/reserves/iree/bump_torch_mlir.yml new file mode 100644 index 000000000000..d039ed9ff2c7 --- /dev/null +++ b/.github/workflows/reserves/iree/bump_torch_mlir.yml @@ -0,0 +1,75 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Automatically bumps the torch-mlir submodule to match iree-org/torch-mlir:main. +# Manually triggered for now, but can be scheduled in the future. +# This typically takes ~90s to run. +# Will succeed without creating a PR if there is no newer version for torch-mlir. +name: Bump submodule - torch-mlir + +on: + # TODO: Uncomment the following lines to enable the schedule when this action is stable & useful enough + # schedule: + # - cron: '0 0 * * 0,4' # Run at 00:00 on Sunday and Thursday + workflow_dispatch: + inputs: + bump_target: + description: "Commit hash or branch to bump the submodule to" + required: false + default: "origin/main" + pull_request: + branches: + - main + paths: + - .github/workflows/bump_torch_mlir.yml + +jobs: + bump_submodule: + runs-on: ubuntu-20.04 + steps: + - name: Checkout repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + + - name: Setting git config + run: | + git config --local user.email "iree-github-actions-bot@google.com" + git config --local user.name "Submodule Bump Action" + + - name: Update torch-mlir submodule + run: | + git submodule update --init --depth 1 third_party/torch-mlir + + - name: Bump torch-mlir submodule + id: bump_submodule + run: | + cd third_party/torch-mlir + git fetch origin + if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then + git checkout ${{ github.event.inputs.bump_target }} + else + git checkout origin/main + fi + TORCH_MLIR_COMMIT=$(git rev-parse HEAD) + cd ../.. + git add third_party/torch-mlir + git commit -m "Bump torch-mlir submodule to $TORCH_MLIR_COMMIT" + echo "TORCH_MLIR_COMMIT=$TORCH_MLIR_COMMIT" >> $GITHUB_OUTPUT + + - name: Create Pull Request + uses: peter-evans/create-pull-request@v6.0.5 + with: + token: ${{ secrets.GITHUB_TOKEN }} + base: main + branch: integrates/torch-mlir + delete-branch: true # automatically supercedes the previous bump + title: "Bump torch-mlir submodule to ${{ steps.bump_submodule.outputs.TORCH_MLIR_COMMIT }}" + body: | + This PR updates the torch-mlir submodule to https://github.com/llvm/torch-mlir/commit/${{ steps.bump_submodule.outputs.TORCH_MLIR_COMMIT }}. + + Auto-generated by GitHub Actions using `.github/workflows/bump_torch_mlir.yml`. + commit-message: "Bump torch-mlir submodule to ${{ steps.bump_submodule.outputs.TORCH_MLIR_COMMIT }}" diff --git a/.github/workflows/reserves/iree/ci.yml b/.github/workflows/reserves/iree/ci.yml new file mode 100644 index 000000000000..d0a74bf95c02 --- /dev/null +++ b/.github/workflows/reserves/iree/ci.yml @@ -0,0 +1,408 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI + +# A few notes: +# +# Variables: +# GitHub actions don't have variables or even support normal yaml anchors (they +# are specially disabled because...reasons?): +# See https://github.com/github-community/community/discussions/4501 +# https://github.community/t/support-for-yaml-anchors/16128/92 +# https://github.com/actions/runner/issues/1182 +# Neither does it have any contexts that are available everywhere. The +# top-level `env` field is available in many places, but not all. We already +# have a "setup" job that every other job depends on, so we leverage that +# for variables that every other job can use, since that *is* available in all +# sub-fields of the job. +# See https://docs.github.com/en/actions/learn-github-actions/contexts#context-availability +# and https://github.com/community/community/discussions/27370 +# +# Runner label ordering: +# - self-hosted always has to be listed first in a runs-on block: +# https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow + +on: + workflow_dispatch: + pull_request: + push: + branches: + - main + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +env: + # This needs to be in env instead of the outputs of setup because it contains + # the run attempt and we want that to be the current attempt, not whatever + # attempt the setup step last ran in. + GCS_URL: https://storage.googleapis.com/iree-github-actions-${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }}-artifacts/${{ github.run_id }}/${{ github.run_attempt }} + +# Jobs are organized into groups and topologically sorted by dependencies +jobs: + setup: + uses: ./.github/workflows/setup.yml + + ################################### Basic #################################### + # Jobs that build all of IREE "normally" + ############################################################################## + # TODO(scotttodd): drop this job in favor of ci_linux_x64_clang.yml once no jobs depend on it + build_all: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_all') + uses: ./.github/workflows/build_all.yml + with: + runner-group: ${{ needs.setup.outputs.runner-group }} + runner-env: ${{ needs.setup.outputs.runner-env }} + write-caches: ${{ needs.setup.outputs.write-caches }} + run-tests: true + + build_test_all_bazel: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_test_all_bazel') + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building and testing with Bazel" + env: + IREE_WRITE_REMOTE_BAZEL_CACHE: ${{ needs.setup.outputs.write-caches }} + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_WRITE_REMOTE_BAZEL_CACHE=${IREE_WRITE_REMOTE_BAZEL_CACHE}" \ + gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \ + ./build_tools/bazel/build_test_all.sh + + # Disabled while runner is offline. + # test_nvidia_a100: + # needs: [setup, build_all] + # if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_nvidia_a100') + # env: + # BUILD_DIR: build-tests + # INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }} + # INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }} + # INSTALL_DIR_GCS_URL: ${{ needs.build_all.outputs.install-dir-gcs-url }} + # IREE_CPU_DISABLE: 1 + # IREE_VULKAN_DISABLE: 0 + # IREE_CUDA_DISABLE: 0 + # IREE_HIP_DISABLE: 1 + # runs-on: + # - self-hosted # must come first + # - runner-group=${{ needs.setup.outputs.runner-group }} + # - environment=${{ needs.setup.outputs.runner-env }} + # - a100 + # - os-family=Linux + # steps: + # - name: "Checking out repository" + # uses: actions/checkout@v4.1.7 + # - name: "Checking out runtime submodules" + # run: ./build_tools/scripts/git/update_runtime_submodules.sh + # - name: Querying GPU information + # run: | + # ./build_tools/scripts/check_cuda.sh + # ./build_tools/scripts/check_vulkan.sh + # - name: "Downloading install dir archive" + # run: wget "${INSTALL_DIR_GCS_URL}" -O "${INSTALL_DIR_ARCHIVE}" + # - name: "Extracting install directory" + # run: tar -xf "${INSTALL_DIR_ARCHIVE}" + # - name: "Building tests" + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env IREE_CPU_DISABLE \ + # --env IREE_VULKAN_DISABLE \ + # --env IREE_CUDA_DISABLE \ + # --env IREE_HIP_DISABLE \ + # gcr.io/iree-oss/nvidia@sha256:433e072f075f90fdf07471673626b17091d8d8e2395626f1e6ac6c98803c8807 \ + # ./build_tools/pkgci/build_tests_using_package.sh ${INSTALL_DIR}/bin + # - name: "Running GPU tests" + # env: + # IREE_CTEST_LABEL_REGEX: ^requires-gpu-sm80|^requires-gpu|^driver=vulkan$|^driver=cuda$ + # IREE_NVIDIA_SM80_TESTS_DISABLE: 0 + # IREE_MULTI_DEVICE_TESTS_DISABLE: 1 + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env IREE_VULKAN_DISABLE \ + # --env IREE_CUDA_DISABLE \ + # --env IREE_HIP_DISABLE \ + # --env IREE_CTEST_LABEL_REGEX \ + # --env IREE_NVIDIA_SM80_TESTS_DISABLE \ + # --env IREE_MULTI_DEVICE_TESTS_DISABLE \ + # --env IREE_VULKAN_F16_DISABLE=0 \ + # --env IREE_NVIDIA_GPU_TESTS_DISABLE=0 \ + # --env CTEST_PARALLEL_LEVEL=4 \ + # --env NVIDIA_DRIVER_CAPABILITIES=all \ + # --gpus all \ + # gcr.io/iree-oss/nvidia@sha256:433e072f075f90fdf07471673626b17091d8d8e2395626f1e6ac6c98803c8807 \ + # bash -euo pipefail -c \ + # "./build_tools/scripts/check_cuda.sh + # ./build_tools/scripts/check_vulkan.sh + # ./build_tools/cmake/ctest_all.sh ${BUILD_DIR}" + + ############################### Configurations ############################### + # Jobs that build IREE in some non-default configuration + ############################################################################## + + build_test_runtime: + needs: setup + name: "build_test_runtime :: ${{ matrix.name }}" + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_test_runtime') + runs-on: ${{ matrix.runs-on }} + defaults: + run: + shell: bash + strategy: + fail-fast: false + matrix: + include: + - name: ubuntu-20.04 + runs-on: ubuntu-20.04 + - name: windows-2022 + runs-on: windows-2022 + - name: macos-14 + runs-on: macos-14 + env: + BUILD_DIR: build-runtime + BUILD_PRESET: test + IREE_READ_REMOTE_CCACHE: 0 + IREE_WRITE_REMOTE_CCACHE: 0 + IREE_READ_LOCAL_CCACHE: 1 + IREE_WRITE_LOCAL_CCACHE: ${{ needs.setup.outputs.write-caches }} + steps: + - uses: actions/checkout@v4.1.7 + - uses: actions/setup-python@v5.1.0 + with: + python-version: "3.11" + + - name: (Linux) Install requirements + if: contains(matrix.name, 'ubuntu') + run: | + sudo apt update + sudo apt install -y ninja-build + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + - name: (Windows) Configure MSVC + if: contains(matrix.name, 'windows') + uses: ilammy/msvc-dev-cmd@v1.13.0 + - name: (macOS) Install requirements + if: contains(matrix.name, 'macos') + run: brew install ninja ccache coreutils bash + + - name: Checkout runtime submodules + run: bash ./build_tools/scripts/git/update_runtime_submodules.sh + + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: ${{ github.job }}-${{ matrix.name }} + save: ${{ needs.setup.outputs.write-caches == 1 }} + + - name: Install Python requirements + run: pip install -r ./runtime/bindings/python/iree/runtime/build_requirements.txt + - name: Build runtime + run: bash ./build_tools/cmake/build_runtime.sh "${BUILD_DIR}" + - name: Test runtime + run: bash ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}" + + small_runtime: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'small_runtime') + runs-on: ubuntu-20.04 + env: + BUILD_DIR: build-runtime + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Building size-optimized runtime" + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_runtime_small.sh \ + "${BUILD_DIR}" + - name: "Testing runtime" + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/ctest_all.sh \ + "${BUILD_DIR}" + + tracing: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'tracing') + runs-on: ubuntu-20.04 + env: + BUILD_DIR: build-tracing + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Building IREE runtime with tracing - Tracy" + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "TRACING_PROVIDER=tracy" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_runtime_tracing.sh \ + "${BUILD_DIR}" + - name: "Building IREE runtime with tracing - console" + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "TRACING_PROVIDER=console" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_runtime_tracing.sh \ + "${BUILD_DIR}" + + ############################## Crosscompilation ############################## + # Jobs that cross-compile IREE for other platforms + ############################################################################## + + # Disabled to reduce self-hosted runners needed. See #17957 + # cross_compile_and_test: + # needs: [setup, build_all] + # if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'cross_compile_and_test') + # runs-on: + # - self-hosted # must come first + # - runner-group=${{ needs.setup.outputs.runner-group }} + # - environment=${{ needs.setup.outputs.runner-env }} + # - cpu + # - os-family=Linux + # strategy: + # matrix: + # target: + # - platform: linux + # arch: riscv_64 + # abi: lp64d + # docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + # build_script: "./build_tools/cmake/build_riscv.sh" + # test_script: "./build_tools/cmake/test_riscv.sh" + # - platform: linux + # arch: riscv_32 + # abi: ilp32d + # docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + # build_script: "./build_tools/cmake/build_riscv.sh" + # test_script: "./build_tools/cmake/test_riscv.sh" + # - platform: generic + # arch: riscv_32 + # abi: ilp32 + # docker_image: "gcr.io/iree-oss/riscv@sha256:62e87bad3405d691ddba6f9be0ef44eeb60461a467c8d86f0842c81a1f97da79" + # build_script: "./build_tools/cmake/build_riscv.sh" + # test_script: "./tests/riscv32/smoke.sh" + # - platform: emscripten + # arch: wasm32 + # abi: wasm32 + # docker_image: "gcr.io/iree-oss/emscripten@sha256:2dd4c52f1bb499ab365aad0111fe5538b685d88af38636b409b0cf6a576ab214" + # build_script: "./build_tools/cmake/build_runtime_emscripten.sh" + # # No test script + # env: + # PLATFORM: ${{ matrix.target.platform }} + # ARCH: ${{ matrix.target.arch }} + # ABI: ${{ matrix.target.abi }} + # DOCKER_IMAGE: ${{ matrix.target.docker_image }} + # BUILD_SCRIPT: ${{ matrix.target.build_script }} + # TEST_SCRIPT: ${{ matrix.target.test_script }} + # INSTALL_DIR: ${{ needs.build_all.outputs.install-dir }} + # INSTALL_DIR_ARCHIVE: ${{ needs.build_all.outputs.install-dir-archive }} + # INSTALL_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.install-dir-gcs-artifact }} + # TARGET_BUILD_DIR: build-${{ matrix.target.platform }}-${{ matrix.target.arch }} + # IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + # steps: + # - name: "Checking out repository" + # uses: actions/checkout@v4.1.7 + # - name: "Checking out runtime submodules" + # run: ./build_tools/scripts/git/update_runtime_submodules.sh + # - name: "Downloading install dir archive" + # run: gcloud storage cp "${INSTALL_DIR_GCS_ARTIFACT}" "${INSTALL_DIR_ARCHIVE}" + # - name: "Extracting install directory" + # run: tar -xf "${INSTALL_DIR_ARCHIVE}" + # - name: "Build cross-compiling target" + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + # --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + # --env "CCACHE_NAMESPACE=${DOCKER_IMAGE}" \ + # --env "IREE_TARGET_PLATFORM=${PLATFORM}" \ + # --env "IREE_TARGET_ARCH=${ARCH}" \ + # --env "IREE_TARGET_ABI=${ABI}" \ + # --env "IREE_TARGET_BUILD_DIR=${TARGET_BUILD_DIR}" \ + # --env "BUILD_PRESET=test" \ + # --env "IREE_HOST_BIN_DIR=${INSTALL_DIR}/bin" \ + # "${DOCKER_IMAGE}" \ + # "${BUILD_SCRIPT}" + # - name: "Test cross-compiling target" + # if: matrix.target.test_script + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env "IREE_TARGET_PLATFORM=${PLATFORM}" \ + # --env "IREE_TARGET_ARCH=${ARCH}" \ + # --env "IREE_TARGET_BUILD_DIR=${TARGET_BUILD_DIR}" \ + # --env "BUILD_PRESET=test" \ + # "${DOCKER_IMAGE}" \ + # "${TEST_SCRIPT}" + + ############################################################################## + + # Depends on all the other jobs to provide a single anchor that indicates the + # final status. Status reporting will become more sophisticated in the future + # and we can hopefully avoid the need to explicitly list every single job... + summary: + # Even if you have an explicit if condition, you still need to override + # GitHub's default behavior of not running if any dependencies failed. + if: always() + runs-on: ubuntu-20.04 + needs: + - setup + + # Basic + - build_all + - build_test_all_bazel + + # Accelerators + # - test_nvidia_a100 + + # Configurations + - build_test_runtime + - small_runtime + - tracing + + # Crosscompilation + # - cross_compile_and_test + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: Getting failed jobs + id: failed_jobs + run: | + echo '${{ toJson(needs) }}' + FAILED_JOBS="$(echo '${{ toJson(needs) }}' \ + | jq --raw-output \ + 'map_values(select(.result!="success" and .result!="skipped")) | keys | join(",")' \ + )" + echo "failed-jobs=${FAILED_JOBS}" >> $GITHUB_OUTPUT + if [[ "${FAILED_JOBS}" != "" ]]; then + echo "The following jobs failed: ${FAILED_JOBS}" + exit 1 + fi + - name: Posting to Discord + uses: sarisia/actions-status-discord@v1.14.3 + if: failure() && github.ref_name == 'main' + with: + webhook: ${{ secrets.DISCORD_WEBHOOK }} + description: "The following jobs failed: ${{ steps.failed_jobs.outputs.failed-jobs }}" + url: "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/attempts/${{ github.run_attempt }}" diff --git a/.github/workflows/reserves/iree/ci_linux_arm64_clang.yml b/.github/workflows/reserves/iree/ci_linux_arm64_clang.yml new file mode 100644 index 000000000000..ea0e41e8adfe --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_arm64_clang.yml @@ -0,0 +1,62 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux arm64 clang + +on: + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + # TODO: Switch to GitHub-hosted arm64 runners when available? + # TODO: Switch runs-on labels to use different (non-GCP) self-hosted runners? + linux_arm64_clang: + needs: setup + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - arm64 + - os-family=Linux + env: + BUILD_DIR: build-arm64 + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building IREE" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base-arm64@sha256:9daa1cdbbf12da8527319ece76a64d06219e04ecb99a4cff6e6364235ddf6c59" \ + --env "IREE_BUILD_SETUP_PYTHON_VENV=${BUILD_DIR}/.venv" \ + gcr.io/iree-oss/base-arm64@sha256:9daa1cdbbf12da8527319ece76a64d06219e04ecb99a4cff6e6364235ddf6c59 \ + ./build_tools/cmake/build_all.sh \ + "${BUILD_DIR}" + - name: "Testing IREE" + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_ARM_SME_QEMU_AARCH64_BIN=/usr/bin/qemu-aarch64" \ + gcr.io/iree-oss/base-arm64@sha256:9daa1cdbbf12da8527319ece76a64d06219e04ecb99a4cff6e6364235ddf6c59 \ + ./build_tools/cmake/ctest_all.sh \ + "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/ci_linux_x64_clang.yml b/.github/workflows/reserves/iree/ci_linux_x64_clang.yml new file mode 100644 index 000000000000..049c268928c0 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_clang.yml @@ -0,0 +1,67 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 clang + +on: + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang.yml" + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + linux_x64_clang: + needs: setup + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + env: + BUILD_DIR: full-build-dir + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building IREE" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" \ + --env "IREE_BUILD_SETUP_PYTHON_VENV=${BUILD_DIR}/.venv" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_all.sh "${BUILD_DIR}" + - name: "Testing IREE" + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}" + - name: "Testing iree-dialects" + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/test_iree_dialects.sh "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/ci_linux_x64_clang_asan.yml b/.github/workflows/reserves/iree/ci_linux_x64_clang_asan.yml new file mode 100644 index 000000000000..e291c5cfcd30 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_clang_asan.yml @@ -0,0 +1,78 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 clang ASan + +on: + push: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang_asan.yml" + - "CMakeLists.txt" + - "build_tools/**" + - "compiler/**" + - "llvm-external-projects/**" + - "runtime/**" + - "samples/**" + - "tests/**" + - "third_party/**" + - "tools/**" + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang_asan.yml" + - "CMakeLists.txt" + - "build_tools/**" + - "compiler/**" + - "llvm-external-projects/**" + - "runtime/**" + - "samples/**" + - "tests/**" + - "third_party/**" + - "tools/**" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + linux_x64_clang_asan: + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'linux_x64_clang_asan') + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building and testing with ASan" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + run: | + # Note that this uses the latest version of the clang compiler, etc. + # This gives us access to the latest features and validates that IREE + # builds using the latest versions. + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CCACHE_NAMESPACE=base-bleeding-edge@sha256:14200dacca3a0f3a66f8aa87c6f64729b83a2eeb403b689c24204074ad157418" \ + gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \ + ./build_tools/cmake/build_and_test_asan.sh diff --git a/.github/workflows/reserves/iree/ci_linux_x64_clang_byollvm.yml b/.github/workflows/reserves/iree/ci_linux_x64_clang_byollvm.yml new file mode 100644 index 000000000000..06954adc3855 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_clang_byollvm.yml @@ -0,0 +1,40 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 clang BYO LLVM + +on: + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang_byollvm.yml" + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + linux_x64_clang_byollvm: + runs-on: ubuntu-20.04 + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building and testing with bring-your-own-LLVM" + run: | + ./build_tools/github_actions/docker_run.sh \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_and_test_byo_llvm.sh diff --git a/.github/workflows/reserves/iree/ci_linux_x64_clang_debug.yml b/.github/workflows/reserves/iree/ci_linux_x64_clang_debug.yml new file mode 100644 index 000000000000..1c82415aac78 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_clang_debug.yml @@ -0,0 +1,61 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 clang debug + +on: + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang_debug.yml" + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + # This may run out of memory / disk space on standard GitHub-hosted runners, + # so run on self-hosted CPU build runners instead. + linux_x64_clang_debug: + needs: setup + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + env: + BUILD_DIR: build-debug + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building IREE in Debug configuration" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CMAKE_BUILD_TYPE=Debug" \ + --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" \ + --env "IREE_BUILD_SETUP_PYTHON_VENV=${BUILD_DIR}/.venv" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_all.sh \ + "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/ci_linux_x64_clang_tsan.yml b/.github/workflows/reserves/iree/ci_linux_x64_clang_tsan.yml new file mode 100644 index 000000000000..755dd2a53ec6 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_clang_tsan.yml @@ -0,0 +1,57 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 clang TSan + +on: + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_clang_tsan.yml" + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + linux_x64_clang_tsan: + needs: setup + runs-on: + - self-hosted # must come first + - runner-group=${{ needs.setup.outputs.runner-group }} + - environment=${{ needs.setup.outputs.runner-env }} + - cpu + - os-family=Linux + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building and testing with TSan" + env: + IREE_WRITE_REMOTE_CCACHE: ${{ needs.setup.outputs.write-caches }} + run: | + # Note that this uses the latest version of the clang compiler, etc. + # This gives us access to the latest features and validates that IREE + # builds using the latest versions. + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=$(gcloud auth application-default print-access-token)" \ + --env "IREE_WRITE_REMOTE_CCACHE=${IREE_WRITE_REMOTE_CCACHE}" \ + --env "CCACHE_NAMESPACE=base-bleeding-edge@sha256:14200dacca3a0f3a66f8aa87c6f64729b83a2eeb403b689c24204074ad157418" \ + gcr.io/iree-oss/base-bleeding-edge@sha256:cf2e78194e64fd0166f4141317366261d7a62432b72e9a324cb8c2ff4e1a515a \ + ./build_tools/cmake/build_and_test_tsan.sh diff --git a/.github/workflows/reserves/iree/ci_linux_x64_gcc.yml b/.github/workflows/reserves/iree/ci_linux_x64_gcc.yml new file mode 100644 index 000000000000..b2ccd801354d --- /dev/null +++ b/.github/workflows/reserves/iree/ci_linux_x64_gcc.yml @@ -0,0 +1,48 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Linux x64 gcc + +on: + pull_request: + branches: + - main + paths: + - ".github/workflows/ci_linux_x64_gcc.yml" + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + linux_x64_gcc: + runs-on: ubuntu-20.04 + env: + BUILD_DIR: build-gcc + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Building IREE with gcc" + run: | + ./build_tools/github_actions/docker_run.sh \ + --env CC=/usr/bin/gcc-9 \ + --env CXX=/usr/bin/g++-9 \ + --env CMAKE_BUILD_TYPE=Release \ + --env "IREE_TARGET_BACKEND_WEBGPU_SPIRV=OFF" \ + --env "IREE_BUILD_SETUP_PYTHON_VENV=${BUILD_DIR}/.venv" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./build_tools/cmake/build_all.sh \ + "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/ci_macos_x64_clang.yml b/.github/workflows/reserves/iree/ci_macos_x64_clang.yml new file mode 100644 index 000000000000..c93606f31098 --- /dev/null +++ b/.github/workflows/reserves/iree/ci_macos_x64_clang.yml @@ -0,0 +1,58 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - macOS x64 clang + +on: + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + macos_x64_clang: + runs-on: macos-13 + env: + BUILD_DIR: build-macos + defaults: + run: + shell: bash + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Updating git submodules" + run: git submodule update --init --jobs 8 --depth 1 + # There may be multiple versions of Xcode and SDKs installed. + # Select the latest Xcode app to enable using Metal offline toolchain. + - name: "Update Xcode command line tools path" + run: | + sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer + xcrun metal --version + xcrun metallib --version + - name: "Setting up Python" + uses: actions/setup-python@v5.1.0 + with: + python-version: "3.10" + cache: "pip" + - name: "Installing Python packages" + run: pip install -r runtime/bindings/python/iree/runtime/build_requirements.txt + - name: "Installing requirements" + # We need coreutils for `realpath` used in scripts. + # We need bash because the default one on macOS is fairly old and lacks + # features we use in scripts. + run: brew install ninja ccache coreutils bash + - name: "Building IREE" + run: bash ./build_tools/cmake/build_all.sh "${BUILD_DIR}" + - name: "Testing IREE" + run: bash ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/ci_windows_x64_msvc.yml b/.github/workflows/reserves/iree/ci_windows_x64_msvc.yml new file mode 100644 index 000000000000..b7fa3ebb796d --- /dev/null +++ b/.github/workflows/reserves/iree/ci_windows_x64_msvc.yml @@ -0,0 +1,52 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: CI - Windows x64 MSVC + +on: + schedule: + # Weekday mornings at 09:15 UTC = 01:15 PST (UTC - 8). + - cron: "15 9 * * 1-5" + workflow_dispatch: + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + windows_x64_msvc: + runs-on: windows-2022 + defaults: + run: + shell: bash + env: + BUILD_DIR: build-windows + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: "Setting up Python" + uses: actions/setup-python@v5.1.0 + with: + python-version: "3.10" # Needs pybind >= 2.10.1 for Python >= 3.11 + - name: "Installing Python packages" + run: | + python3 -m venv .venv + .venv/Scripts/activate.bat + python3 -m pip install -r runtime/bindings/python/iree/runtime/build_requirements.txt + - name: "Installing requirements" + run: choco install ccache --yes + - name: "Configuring MSVC" + uses: ilammy/msvc-dev-cmd@v1.13.0 + - name: "Building IREE" + run: ./build_tools/cmake/build_all.sh "${BUILD_DIR}" + - name: "Testing IREE" + run: ./build_tools/cmake/ctest_all.sh "${BUILD_DIR}" diff --git a/.github/workflows/reserves/iree/lint.yml b/.github/workflows/reserves/iree/lint.yml new file mode 100644 index 000000000000..b3501e06d0e2 --- /dev/null +++ b/.github/workflows/reserves/iree/lint.yml @@ -0,0 +1,23 @@ +# Copyright 2019 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: Lint + +on: [pull_request] + +permissions: + contents: read + +jobs: + pre-commit: + runs-on: ubuntu-20.04 + steps: + - name: Checking out repository + uses: actions/checkout@v4.1.7 + - name: Setting up python + uses: actions/setup-python@v5.1.0 + - name: Running pre-commit + uses: pre-commit/action@v3.0.1 diff --git a/.github/workflows/reserves/iree/oneshot_candidate_release.yml b/.github/workflows/reserves/iree/oneshot_candidate_release.yml new file mode 100644 index 000000000000..3629ab9f6ccb --- /dev/null +++ b/.github/workflows/reserves/iree/oneshot_candidate_release.yml @@ -0,0 +1,54 @@ +name: Oneshot candidate release + +on: + workflow_dispatch: + +jobs: + tag_release: + name: "Tag candidate release" + runs-on: ubuntu-20.04 + steps: + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + + - name: Compute version + run: | + git fetch --depth=1 origin +refs/tags/*:refs/tags/* + package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')" + tag_name="candidate-${package_version}" + echo "package_version=${package_version}" >> $GITHUB_ENV + echo "tag_name=${tag_name}" >> $GITHUB_ENV + + - name: Updating candidate tag + run: | + git tag "${tag_name}" + + - name: Pushing changes + uses: ad-m/github-push-action@v0.8.0 + with: + github_token: ${{ secrets.WRITE_ACCESS_TOKEN }} + branch: ${{ github.ref_name }} + tags: true + + - name: Create Release + id: create_release + uses: actions/create-release@0cb9c9b65d5d1901c1f53e5e66eaf4afd303e70e # v1 + env: + GITHUB_TOKEN: ${{ secrets.WRITE_ACCESS_TOKEN }} + with: + tag_name: ${{ env.tag_name }} + release_name: iree candidate ${{ env.tag_name }} + body: | + Automatic candidate release of iree. + draft: true + prerelease: true + + - name: "Invoke workflow :: Build Release Packages" + uses: benc-uk/workflow-dispatch@798e70c97009500150087d30d9f11c5444830385 # v1.2.2 + with: + workflow: Build Release Packages + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + ref: "${{ env.tag_name }}" + inputs: '{"package_suffix": "", "package_version": "${{ env.package_version }}", "release_id": "${{ steps.create_release.outputs.id }}"}' diff --git a/.github/workflows/reserves/iree/pkgci.yml b/.github/workflows/reserves/iree/pkgci.yml new file mode 100644 index 000000000000..2effd7593fee --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci.yml @@ -0,0 +1,87 @@ +# Copyright 2023 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI + +on: + workflow_dispatch: + pull_request: + push: + branches: + - main + # Experimental branches can also get packages and tests automatically. + - exppkg-* + # Short term feature branches: + - sdxl + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + setup: + uses: ./.github/workflows/setup.yml + + build_packages: + name: Build Packages + needs: setup + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'build_packages') + uses: ./.github/workflows/pkgci_build_packages.yml + with: + package_version: 0.dev1 + + unit_test: + name: Unit Test + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'unit_test') + uses: ./.github/workflows/pkgci_unit_test.yml + + regression_test: + name: Regression Test + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'regression_test') + uses: ./.github/workflows/pkgci_regression_test.yml + + test_amd_mi250: + name: Test AMD MI250 + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi250') + uses: ./.github/workflows/pkgci_test_amd_mi250.yml + + test_amd_mi300: + name: Test AMD MI300 + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_mi300') + uses: ./.github/workflows/pkgci_test_amd_mi300.yml + + test_amd_w7900: + name: Test AMD W7900 + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_amd_w7900') + uses: ./.github/workflows/pkgci_test_amd_w7900.yml + + test_nvidia_t4: + name: Test NVIDIA T4 + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_nvidia_t4') + uses: ./.github/workflows/pkgci_test_nvidia_t4.yml + + test_tensorflow_cpu: + name: Test TensorFlow CPU + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_tensorflow_cpu') + uses: ./.github/workflows/pkgci_test_tensorflow_cpu.yml + + test_android: + needs: [setup, build_packages] + if: contains(fromJson(needs.setup.outputs.enabled-jobs), 'test_android') + uses: ./.github/workflows/pkgci_test_android.yml + with: + write-caches: ${{ needs.setup.outputs.write-caches }} diff --git a/.github/workflows/reserves/iree/pkgci_build_packages.yml b/.github/workflows/reserves/iree/pkgci_build_packages.yml new file mode 100644 index 000000000000..a2ceff4c91ed --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_build_packages.yml @@ -0,0 +1,144 @@ +# Copyright 2023 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Build Packages +on: + workflow_call: + inputs: + package_version: + type: string + required: true + +jobs: + linux_x86_64_release_packages: + name: Linux Release (x86_64) + runs-on: + - self-hosted # must come first + - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }} + - environment=prod + - cpu + - os-family=Linux + strategy: + fail-fast: false + env: + CACHE_DIR: ${{ github.workspace }}/.iree-container-cache + MANYLINUX_DOCKER_IMAGE: ghcr.io/nod-ai/manylinux_x86_64:main + PACKAGE_SUFFIX: "" + steps: + - name: Prefetch Docker + run: | + docker pull "$MANYLINUX_DOCKER_IMAGE" & + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + submodules: true + - name: Write version info + shell: bash + run: | + cat << EOF > version_info.json + { + "package-suffix": "${PACKAGE_SUFFIX}", + "package-version": "${{ inputs.package_version }}", + "iree-revision": "$(cd ../iree && git rev-parse HEAD)" + } + EOF + realpath version_info.json + cat version_info.json + - name: Enable cache + uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1 + with: + path: ${{ env.CACHE_DIR }} + key: iree-pkgci-linux-release-x86_64-v1-${{ github.sha }} + restore-keys: | + iree-pkgci-linux-release-x86_64-v1- + - name: Wait for docker pull + run: | + wait + - name: Build + run: | + export cache_dir="${{ env.CACHE_DIR }}" + export output_dir="${{ github.workspace }}/wheelhouse" + export toolchain_suffix=release + export manylinux_docker_image="$MANYLINUX_DOCKER_IMAGE" + export package_suffix="$PACKAGE_SUFFIX" + # If just iterating locally, uncomment this to build a cheap wheel. + # export packages="iree-runtime" + ./build_tools/pkgci/build_linux_packages.sh + # Some things put stuff in cache with weird, root read-only + # permissions. Take them back. + sudo chown -R "$(whoami)" "${cache_dir}" + - name: Upload wheel artifacts + uses: actions/upload-artifact@v4.3.3 + with: + name: linux_x86_64_release_packages + path: | + ${{ github.workspace }}/wheelhouse/iree*.whl + if-no-files-found: error +# TODO: Debug low ccache hit rate and re-enable. +# linux_x86_64_release_asserts_packages: +# name: Linux Release Asserts (x86_64) +# runs-on: +# - self-hosted # must come first +# - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }} +# - environment=prod +# - cpu +# - os-family=Linux +# strategy: +# fail-fast: false +# env: +# CACHE_DIR: ${{ github.workspace }}/.iree-container-cache +# MANYLINUX_DOCKER_IMAGE: ghcr.io/nod-ai/manylinux_x86_64:main +# PACKAGE_SUFFIX: "-asserts" +# steps: +# - name: Prefetch Docker +# run: | +# docker pull "$MANYLINUX_DOCKER_IMAGE" & +# - name: "Checking out repository" +# uses: actions/checkout@v4.1.7 +# with: +# submodules: true +# - name: Write version info +# shell: bash +# run: | +# cat << EOF > version_info.json +# { +# "package-suffix": "${PACKAGE_SUFFIX}", +# "package-version": "${{ inputs.package_version }}", +# "iree-revision": "$(cd ../iree && git rev-parse HEAD)" +# } +# EOF +# realpath version_info.json +# cat version_info.json +# - name: Enable cache +# uses: actions/cache@88522ab9f39a2ea568f7027eddc7d8d8bc9d59c8 # v3.3.1 +# with: +# path: ${{ env.CACHE_DIR }} +# key: iree-pkgci-linux-release-asserts-x86_64-v1-${{ github.sha }} +# restore-keys: | +# iree-pkgci-linux-release-asserts-x86_64-v1- +# - name: Wait for docker pull +# run: | +# wait +# - name: Build +# run: | +# export cache_dir="${{ env.CACHE_DIR }}" +# export output_dir="${{ github.workspace }}/wheelhouse" +# export toolchain_suffix=release_asserts +# export manylinux_docker_image="$MANYLINUX_DOCKER_IMAGE" +# export package_suffix="$PACKAGE_SUFFIX" +# # If just iterating locally, uncomment this to build a cheap wheel. +# # export packages="iree-runtime" +# ./build_tools/pkgci/build_linux_packages.sh +# # Some things put stuff in cache with weird, root read-only +# # permissions. Take them back. +# sudo chown -R "$(whoami)" "${cache_dir}" +# - name: Upload wheel artifacts +# uses: actions/upload-artifact@v4.3.3 +# with: +# name: linux_x86_64_release_asserts_packages +# path: | +# ${{ github.workspace }}/wheelhouse/iree*.whl +# if-no-files-found: error diff --git a/.github/workflows/reserves/iree/pkgci_regression_test.yml b/.github/workflows/reserves/iree/pkgci_regression_test.yml new file mode 100644 index 000000000000..6347f93db575 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_regression_test.yml @@ -0,0 +1,376 @@ +# Copyright 2023 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Regression Test +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + test_onnx: + name: "test_onnx :: ${{ matrix.name }}" + runs-on: ${{ matrix.runs-on }} + strategy: + fail-fast: false + matrix: + include: + # CPU + - name: cpu_llvm_sync + config-file: onnx_cpu_llvm_sync.json + numprocesses: auto + runs-on: ubuntu-20.04 + + # AMD GPU + - name: amdgpu_rocm_rdna3 + numprocesses: 1 + config-file: onnx_gpu_rocm_rdna3.json + runs-on: nodai-amdgpu-w7900-x86-64 + - name: amdgpu_vulkan + numprocesses: 4 + config-file: onnx_gpu_vulkan.json + runs-on: nodai-amdgpu-w7900-x86-64 + + # NVIDIA GPU + - name: nvidiagpu_cuda + config-file: onnx_gpu_cuda.json + numprocesses: 4 + runs-on: + - self-hosted # must come first + - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }} + - environment=prod + - gpu # TODO(scotttodd): qualify further with vendor/model + - os-family=Linux + - name: nvidiagpu_vulkan + config-file: onnx_gpu_vulkan.json + numprocesses: 4 + runs-on: + - self-hosted # must come first + - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }} + - environment=prod + - gpu # TODO(scotttodd): qualify further with vendor/model + - os-family=Linux + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + CONFIG_FILE_PATH: build_tools/pkgci/external_test_suite/${{ matrix.config-file }} + NUMPROCESSES: ${{ matrix.numprocesses }} + LOG_FILE_PATH: /tmp/iree_tests_onnx_${{ matrix.name }}_logs.json + VENV_DIR: ${{ github.workspace }}/venv + steps: + - name: Checking out IREE repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + + - name: Check out external TestSuite repository + uses: actions/checkout@v4.1.7 + with: + repository: nod-ai/SHARK-TestSuite + ref: f5615ab29da491c0047146258dfa3a0c40c735e5 + path: SHARK-TestSuite + submodules: false + lfs: false + - name: Install external TestSuite Python requirements + run: | + source ${VENV_DIR}/bin/activate + python -m pip install -r SHARK-TestSuite/iree_tests/requirements.txt + pip install --no-compile --pre --upgrade -e SHARK-TestSuite/common_tools + + - name: Run external tests - ONNX test suite + run: | + source ${VENV_DIR}/bin/activate + pytest SHARK-TestSuite/iree_tests/onnx/ \ + -rpfE \ + --numprocesses ${NUMPROCESSES} \ + --timeout=30 \ + --durations=20 \ + --no-skip-tests-missing-files \ + --config-files=${CONFIG_FILE_PATH} \ + --report-log=${LOG_FILE_PATH} + - name: "Updating config file with latest XFAIL lists" + if: failure() + run: | + source ${VENV_DIR}/bin/activate + python SHARK-TestSuite/iree_tests/update_config_xfails.py \ + --log-file=${LOG_FILE_PATH} \ + --config-file=${CONFIG_FILE_PATH} + cat ${CONFIG_FILE_PATH} + - name: "Uploading new config file" + if: failure() + uses: actions/upload-artifact@v4.3.3 + with: + name: ${{ matrix.config-file }} + path: ${{ env.CONFIG_FILE_PATH }} + + test_models: + name: "test_models :: ${{ matrix.name }}" + runs-on: ${{ matrix.runs-on }} + strategy: + fail-fast: false + + # Note: these jobs should use persistent runners with local caches. + # Downloading test files (50GB+) without a cache can take 20+ minutes. + matrix: + include: + # CPU + - name: cpu_llvm_task + models-config-file: models_cpu_llvm_task.json + iree-tests-cache: ~/iree_tests_cache + runs-on: nodai-amdgpu-w7900-x86-64 + + # AMD GPU + - name: amdgpu_rocm_mi250_gfx90a + models-config-file: models_gpu_rocm_gfx90a.json + iree-tests-cache: /groups/aig_sharks/iree-tests-cache + runs-on: nodai-amdgpu-mi250-x86-64 + - name: amdgpu_rocm_mi300_gfx942 + models-config-file: models_gpu_rocm_gfx942.json + iree-tests-cache: ~/iree_tests_cache + runs-on: nodai-amdgpu-mi300-x86-64 + - name: amdgpu_vulkan + models-config-file: models_gpu_vulkan.json + iree-tests-cache: ~/iree_tests_cache + runs-on: nodai-amdgpu-w7900-x86-64 + + # NVIDIA GPU + # None at the moment. Could maybe use the persistent a100 runners: + # - self-hosted # must come first + # - runner-group=${{ needs.setup.outputs.runner-group }} + # - environment=${{ needs.setup.outputs.runner-env }} + # - a100 + # - os-family=Linux + # (note: would need to plumb the presubmit/postsubmit runner-group through to here too) + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + IREE_TEST_FILES: ${{ matrix.iree-tests-cache }} + IREE_TEST_PATH_EXTENSION: ${{ github.workspace }}/build_tools/pkgci/external_test_suite + MODELS_CONFIG_FILE_PATH: build_tools/pkgci/external_test_suite/${{ matrix.models-config-file }} + VENV_DIR: ${{ github.workspace }}/venv + LD_LIBRARY_PATH: /home/esaimana/Python-3.11.9 + steps: + # TODO(saienduri): Find alternative to this temporary step that manipulates permission of github actions + # directory to be able to clean after every PR + - name: Pre Checkout MI300 Step + if: contains(matrix.name, 'gfx942') + run: | + sudo chmod -R 777 ~/actions-runner/_work + - name: Checking out IREE repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + + # Out of tree tests + - name: Check out external TestSuite repository + uses: actions/checkout@v4.1.7 + with: + repository: nod-ai/SHARK-TestSuite + ref: f5615ab29da491c0047146258dfa3a0c40c735e5 + path: SHARK-TestSuite + submodules: false + lfs: true + - name: Install external TestSuite Python requirements + run: | + source ${VENV_DIR}/bin/activate + python3 -m pip install -r SHARK-TestSuite/iree_tests/requirements.txt + pip install --no-compile --pre --upgrade -e SHARK-TestSuite/common_tools + - name: Download remote files for real weight model tests + run: | + source ${VENV_DIR}/bin/activate + python SHARK-TestSuite/iree_tests/download_remote_files.py --root-dir iree_tests/pytorch/models + python SHARK-TestSuite/iree_tests/download_remote_files.py --root-dir iree_tests/sharktank + + - name: Run external tests - models with real weights + if: "matrix.models-config-file != '' && !cancelled()" + run: | + source ${VENV_DIR}/bin/activate + pytest \ + SHARK-TestSuite/iree_tests/pytorch/models \ + SHARK-TestSuite/iree_tests/sharktank \ + -rA \ + -k real_weights \ + --no-skip-tests-missing-files \ + --capture=no \ + --log-cli-level=info \ + --timeout=1200 \ + --durations=0 \ + --config-files=${MODELS_CONFIG_FILE_PATH} + + test_regression_suite: + name: "test_regression_suite :: ${{ matrix.name }}" + runs-on: ${{ matrix.runs-on }} + strategy: + fail-fast: false + + # Note: these jobs should use persistent runners with local caches. + # Downloading test files (50GB+) without a cache can take 20+ minutes. + matrix: + include: + # CPU + - name: cpu_llvm_task + models-config-file: models_cpu_llvm_task.json + backend: cpu + iree-tests-cache: ~/iree_tests_cache + runs-on: nodai-amdgpu-w7900-x86-64 + + # AMD GPU + - name: amdgpu_rocm_mi250_gfx90a + rocm-chip: gfx90a + backend: rocm + iree-tests-cache: /groups/aig_sharks/iree-tests-cache + runs-on: nodai-amdgpu-mi250-x86-64 + - name: amdgpu_rocm_mi300_gfx942 + rocm-chip: gfx942 + backend: rocm + iree-tests-cache: ~/iree_tests_cache + runs-on: nodai-amdgpu-mi300-x86-64 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + IREE_TEST_FILES: ${{ matrix.iree-tests-cache }} + IREE_TEST_PATH_EXTENSION: ${{ github.workspace }}/build_tools/pkgci/external_test_suite + VENV_DIR: ${{ github.workspace }}/venv + LD_LIBRARY_PATH: /home/esaimana/Python-3.11.9 + steps: + # TODO(saienduri): Find alternative to this temporary step that manipulates permission of github actions + # directory to be able to clean after every PR + - name: Pre Checkout MI300 Step + if: contains(matrix.name, 'gfx942') + run: | + sudo chmod -R 777 ~/actions-runner/_work + - name: Checking out IREE repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + source ${VENV_DIR}/bin/activate + pip install -e experimental/regression_suite + + # TODO(#17344): regenerate .mlirbc files, test plat_rdna3_rocm on rocm + # # In-tree tests + # - name: Run experimental/regression_suite tests + # run: | + # source ${VENV_DIR}/bin/activate + # pytest \ + # -rA -s -m "plat_host_cpu and presubmit" \ + # experimental/regression_suite + + - name: "Running SDXL special model tests" + if: "!cancelled()" + run: | + source ${VENV_DIR}/bin/activate + pytest ./experimental/regression_suite/shark-test-suite-models/sdxl \ + -k ${{ matrix.backend }} \ + -rpfE \ + --capture=no \ + --log-cli-level=info \ + --timeout=1200 \ + --durations=0 + env: + ROCM_CHIP: ${{ matrix.rocm-chip }} + + - name: "Running SD3 special model tests" + if: "!cancelled()" + run: | + source ${VENV_DIR}/bin/activate + pytest ./experimental/regression_suite/shark-test-suite-models/sd3 \ + -k ${{ matrix.backend }} \ + -rpfE \ + --capture=no \ + --log-cli-level=info \ + --timeout=1200 \ + --durations=0 + env: + ROCM_CHIP: ${{ matrix.rocm-chip }} + + # Note: mi250 benchmark times are more lenient than mi300 (allowing about + # 10% deviation from observed averages), since the mi250 runners we use + # are more unstable and we care most about peak performance on mi300. + - name: "Running SDXL rocm pipeline benchmark (mi250)" + if: contains(matrix.name, 'rocm_mi250_gfx90a') + run: | + source ${VENV_DIR}/bin/activate + pytest ./experimental/benchmarks/sdxl/benchmark_sdxl_rocm.py \ + --goldentime-rocm-e2e-ms 1450.0 \ + --goldentime-rocm-unet-ms 370.0 \ + --goldentime-rocm-clip-ms 18.5 \ + --goldentime-rocm-vae-ms 315.0 \ + --goldendispatch-rocm-unet 1714 \ + --goldendispatch-rocm-clip 1569 \ + --goldendispatch-rocm-vae 248 \ + --goldensize-rocm-unet-bytes 2280000 \ + --goldensize-rocm-clip-bytes 860000 \ + --goldensize-rocm-vae-bytes 840000 \ + --gpu-number 6 \ + --rocm-chip gfx90a \ + --log-cli-level=info \ + --retries 7 + echo "$(> $GITHUB_STEP_SUMMARY + rm job_summary.md + + - name: "Running SDXL rocm pipeline benchmark (mi300)" + if: contains(matrix.name, 'rocm_mi300_gfx942') + run: | + source ${VENV_DIR}/bin/activate + pytest ./experimental/benchmarks/sdxl/benchmark_sdxl_rocm.py \ + --goldentime-rocm-e2e-ms 325.0 \ + --goldentime-rocm-unet-ms 77.0 \ + --goldentime-rocm-clip-ms 15.5 \ + --goldentime-rocm-vae-ms 74.0 \ + --goldendispatch-rocm-unet 1714 \ + --goldendispatch-rocm-clip 1569 \ + --goldendispatch-rocm-vae 248 \ + --goldensize-rocm-unet-bytes 2270000 \ + --goldensize-rocm-clip-bytes 860000 \ + --goldensize-rocm-vae-bytes 840000 \ + --gpu-number 0 \ + --rocm-chip gfx942 \ + --log-cli-level=info \ + --retries 7 + echo "$(> $GITHUB_STEP_SUMMARY diff --git a/.github/workflows/reserves/iree/pkgci_test_amd_mi250.yml b/.github/workflows/reserves/iree/pkgci_test_amd_mi250.yml new file mode 100644 index 000000000000..6253a59301a5 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_amd_mi250.yml @@ -0,0 +1,69 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Test AMD MI250 +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + test: + runs-on: nodai-amdgpu-mi250-x86-64 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + BUILD_DIR: build-tests + VENV_DIR: ${{ github.workspace }}/.venv + IREE_CPU_DISABLE: 1 + IREE_VULKAN_DISABLE: 1 + IREE_CUDA_DISABLE: 1 + IREE_HIP_DISABLE: 0 + IREE_HIP_TEST_TARGET_CHIP: "gfx90a" + steps: + - name: Check out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - name: Check out runtime submodules + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + - name: Install build dependencies + run: | + sudo apt update + sudo apt install -y cmake clang ninja-build libstdc++-12-dev + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + + - name: Build tests + run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin + - name: Run GPU tests + env: + CTEST_PARALLEL_LEVEL: 2 + IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$ + IREE_AMD_RDNA3_TESTS_DISABLE: 1 + IREE_NVIDIA_GPU_TESTS_DISABLE: 0 + IREE_NVIDIA_SM80_TESTS_DISABLE: 1 + IREE_MULTI_DEVICE_TESTS_DISABLE: 0 + run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR} diff --git a/.github/workflows/reserves/iree/pkgci_test_amd_mi300.yml b/.github/workflows/reserves/iree/pkgci_test_amd_mi300.yml new file mode 100644 index 000000000000..23a938dd49c6 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_amd_mi300.yml @@ -0,0 +1,74 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Test AMD MI300 +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + test: + runs-on: nodai-amdgpu-mi300-x86-64 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + BUILD_DIR: build-tests + VENV_DIR: ${{ github.workspace }}/.venv + IREE_CPU_DISABLE: 1 + IREE_VULKAN_DISABLE: 1 + IREE_CUDA_DISABLE: 1 + IREE_HIP_DISABLE: 0 + IREE_HIP_TEST_TARGET_CHIP: "gfx942" + LD_LIBRARY_PATH: /home/esaimana/Python-3.11.9 + steps: + - name: Pre Checkout MI300 Step + if: contains(matrix.name, 'gfx942') + run: | + sudo chmod -R 777 ~/actions-runner/_work + - name: Check out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - name: Check out runtime submodules + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + - name: Install build dependencies + run: | + sudo apt update + sudo apt install -y cmake clang ninja-build libstdc++-12-dev + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + + - name: Build tests + run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin + - name: Run GPU tests + env: + CTEST_PARALLEL_LEVEL: 2 + IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=hip$ + IREE_AMD_RDNA3_TESTS_DISABLE: 1 + IREE_NVIDIA_GPU_TESTS_DISABLE: 0 + IREE_NVIDIA_SM80_TESTS_DISABLE: 1 + IREE_MULTI_DEVICE_TESTS_DISABLE: 0 + run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR} diff --git a/.github/workflows/reserves/iree/pkgci_test_amd_w7900.yml b/.github/workflows/reserves/iree/pkgci_test_amd_w7900.yml new file mode 100644 index 000000000000..574a236ceea1 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_amd_w7900.yml @@ -0,0 +1,63 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Test AMD W7900 +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + test: + runs-on: nodai-amdgpu-w7900-x86-64 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + BUILD_DIR: build-tests + VENV_DIR: ${{ github.workspace }}/.venv + IREE_CPU_DISABLE: 1 + IREE_VULKAN_DISABLE: 0 + IREE_CUDA_DISABLE: 1 + IREE_HIP_DISABLE: 0 + IREE_HIP_TEST_TARGET_CHIP: "gfx1100" + steps: + - name: Check out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - name: Check out runtime submodules + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + + - name: Build tests + run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin + - name: Run GPU tests + env: + CTEST_PARALLEL_LEVEL: 1 + IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=hip$ + IREE_AMD_RDNA3_TESTS_DISABLE: 0 + IREE_NVIDIA_GPU_TESTS_DISABLE: 0 + IREE_NVIDIA_SM80_TESTS_DISABLE: 1 + IREE_MULTI_DEVICE_TESTS_DISABLE: 0 + run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR} diff --git a/.github/workflows/reserves/iree/pkgci_test_android.yml b/.github/workflows/reserves/iree/pkgci_test_android.yml new file mode 100644 index 000000000000..0fb63efe85c8 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_android.yml @@ -0,0 +1,91 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Workflow for Android cross-compilation and test jobs. +# The concurrency of this workflow is controlled by the caller's job. + +name: PkgCI Test Android + +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + write-caches: + required: true + type: string + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + write-caches: + required: true + type: string + +permissions: + contents: read + +jobs: + cross_compile: + runs-on: ubuntu-20.04 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + VENV_DIR: ${{ github.workspace }}/.venv + IREE_TARGET_BUILD_DIR: build-android-arm_64 + steps: + # General setup. + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2 + with: + key: ${{ github.job }} + save: ${{ inputs.write-caches == 1 }} + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + - name: Install build dependencies + run: | + sudo apt update + sudo apt install -y ninja-build + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + + - uses: nttld/setup-ndk@v1 + with: + ndk-version: r25b + - name: Build for Android + env: + IREE_HOST_BIN_DIR: ${{ env.VENV_DIR }}/bin + IREE_READ_REMOTE_CCACHE: 0 + IREE_WRITE_REMOTE_CCACHE: 0 + IREE_READ_LOCAL_CCACHE: 1 + IREE_WRITE_LOCAL_CCACHE: ${{ inputs.write-caches }} + BUILD_PRESET: test + run: build_tools/cmake/build_android.sh + + # TODO: run tests on physical devices or emulators + # * Emulators can run on the same host that ran the build + # * Physical devices could either + # A) be connected to a self-hosted build machine that runs the compiler, + # connects over ADB, and run tests + # B) be connected to a self-hosted run machine that downloads already + # compiled artifacts from a build machine and just connects over ADB + # to run tests diff --git a/.github/workflows/reserves/iree/pkgci_test_nvidia_t4.yml b/.github/workflows/reserves/iree/pkgci_test_nvidia_t4.yml new file mode 100644 index 000000000000..acf63fe04bff --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_nvidia_t4.yml @@ -0,0 +1,77 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Test NVIDIA T4 +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + test: + runs-on: + - self-hosted # must come first + - runner-group=${{ github.event_name == 'pull_request' && 'presubmit' || 'postsubmit' }} + - environment=prod + - gpu + - os-family=Linux + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + BUILD_DIR: build-tests + VENV_DIR: ${{ github.workspace }}/.venv + IREE_CPU_DISABLE: 1 + IREE_VULKAN_DISABLE: 0 + IREE_CUDA_DISABLE: 0 + IREE_HIP_DISABLE: 1 + steps: + - name: Check out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - name: Check out runtime submodules + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: Query GPU information + run: | + ./build_tools/scripts/check_cuda.sh + ./build_tools/scripts/check_vulkan.sh + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + - name: Install build dependencies + run: | + sudo apt update + sudo apt install -y cmake clang ninja-build libstdc++-12-dev + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + + - name: Build tests + run: ./build_tools/pkgci/build_tests_using_package.sh ${VENV_DIR}/bin + - name: Run GPU tests + env: + CTEST_PARALLEL_LEVEL: 2 + IREE_CTEST_LABEL_REGEX: ^requires-gpu|^driver=vulkan$|^driver=cuda$ + IREE_NVIDIA_GPU_TESTS_DISABLE: 0 + IREE_NVIDIA_SM80_TESTS_DISABLE: 1 + IREE_MULTI_DEVICE_TESTS_DISABLE: 1 + IREE_VULKAN_F16_DISABLE: 0 + run: ./build_tools/cmake/ctest_all.sh ${BUILD_DIR} diff --git a/.github/workflows/reserves/iree/pkgci_test_tensorflow_cpu.yml b/.github/workflows/reserves/iree/pkgci_test_tensorflow_cpu.yml new file mode 100644 index 000000000000..221274a9b02d --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_test_tensorflow_cpu.yml @@ -0,0 +1,49 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Test TensorFlow (CPU) +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + linux_x86_64: + name: Linux (x86_64) + runs-on: ubuntu-20.04 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + VENV_DIR: ${{ github.workspace }}/.venv + IREE_VMVX_DISABLE: 0 + steps: + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + - name: Run tests + run: | + source ${VENV_DIR}/bin/activate + build_tools/cmake/run_tf_tests.sh diff --git a/.github/workflows/reserves/iree/pkgci_unit_test.yml b/.github/workflows/reserves/iree/pkgci_unit_test.yml new file mode 100644 index 000000000000..27ed919add08 --- /dev/null +++ b/.github/workflows/reserves/iree/pkgci_unit_test.yml @@ -0,0 +1,62 @@ +# Copyright 2024 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: PkgCI Unit Test +on: + workflow_call: + inputs: + artifact_run_id: + type: string + default: "" + workflow_dispatch: + inputs: + artifact_run_id: + type: string + default: "" + +jobs: + linux_x86_64: + name: Linux (x86_64) + runs-on: ubuntu-20.04 + env: + PACKAGE_DOWNLOAD_DIR: ${{ github.workspace }}/.packages + VENV_DIR: ${{ github.workspace }}/.venv + steps: + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + submodules: false + - uses: actions/setup-python@v5.1.0 + with: + # Must match the subset of versions built in pkgci_build_packages. + python-version: "3.11" + - uses: actions/download-artifact@v4.1.7 + with: + name: linux_x86_64_release_packages + path: ${{ env.PACKAGE_DOWNLOAD_DIR }} + - name: Setup base venv + run: | + ./build_tools/pkgci/setup_venv.py ${VENV_DIR} \ + --artifact-path=${PACKAGE_DOWNLOAD_DIR} \ + --fetch-gh-workflow=${{ inputs.artifact_run_id }} + source ${VENV_DIR}/bin/activate + pip install -r runtime/bindings/python/iree/runtime/build_requirements.txt + - name: Validate runtime wheel + run: | + source ${VENV_DIR}/bin/activate + echo "Testing default runtime package:" + python -m iree.runtime._package_test + echo "Testing tracy runtime package:" + # GH runners don't expose the TSC but we want to make sure the basic packaging + # works, so override the check with TRACY_NO_INVARIANT_CHECK=1 (per instructions + # if this is left off). + TRACY_NO_INVARIANT_CHECK=1 IREE_PY_RUNTIME=tracy \ + python -m iree.runtime._package_test + - name: Validate compiler wheel + run: | + source ${VENV_DIR}/bin/activate + echo "Testing compiler package:" + python -m iree.compiler._package_test diff --git a/.github/workflows/reserves/iree/post_benchmark_comment.yaml b/.github/workflows/reserves/iree/post_benchmark_comment.yaml new file mode 100644 index 000000000000..7d4dfb3c728a --- /dev/null +++ b/.github/workflows/reserves/iree/post_benchmark_comment.yaml @@ -0,0 +1,69 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: Post benchmark comment + +on: + workflow_run: + # TODO(#17001): switch to just "Benchmark" after finishing refactor and + # pending PRs have been rebased + workflows: ["CI", "Benchmark"] + types: [completed] + +env: + PR_CI_STAGE: ${{ github.event.workflow_run.event == 'pull_request' && 'presubmit' || 'postsubmit' }} + PR_CI_RUN_ID: ${{ github.event.workflow_run.id }} + PR_CI_RUN_ATTEMPT: ${{ github.event.workflow_run.run_attempt }} + +jobs: + post_comment: + # Only run the workflow if it's triggered from a pull request. + if: github.event.workflow_run.event == 'pull_request' + runs-on: ubuntu-20.04 + permissions: + issues: write + pull-requests: write + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking benchmark processing job" + id: check + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + BENCHMARK_PROCESS_JOB: process_benchmark_results + run: | + # Assume the workflow has fewer than 100 jobs. + export CONCLUSION=$(gh api \ + "/repos/${GITHUB_REPOSITORY}/actions/runs/${PR_CI_RUN_ID}/attempts/${PR_CI_RUN_ATTEMPT}/jobs?per_page=100" \ + | jq --raw-output --arg job "${BENCHMARK_PROCESS_JOB}" \ + '.jobs | map(select(.name==$job))[0] | .conclusion') + echo "job-conclusion=${CONCLUSION}" >> "${GITHUB_OUTPUT}" + - name: "Downloading artifacts" + if: steps.check.outputs.job-conclusion == 'success' + id: download + env: + # Currently GitHub managed runner doesn't have GCP credential. Fetch + # from public URL directly. + PR_CI_GCS_URL: https://storage.googleapis.com/iree-github-actions-${{ env.PR_CI_STAGE }}-artifacts/${{ env.PR_CI_RUN_ID }}/${{ env.PR_CI_RUN_ATTEMPT }} + BENCHMARK_COMMENT_ARTIFACT: benchmark-comment.json + run: | + wget -O "${BENCHMARK_COMMENT_ARTIFACT}" \ + "${PR_CI_GCS_URL}/${BENCHMARK_COMMENT_ARTIFACT}" + echo "benchmark-comment-artifact=${BENCHMARK_COMMENT_ARTIFACT}" >> "${GITHUB_OUTPUT}" + - name: Posting comments + if: steps.check.outputs.job-conclusion == 'success' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GIST_BOT_TOKEN: ${{ secrets.GIST_BOT_TOKEN }} + COMMENT_BOT_USER: github-actions[bot] + # Get PR number from the event instead of the untrusted comment + # artifact, to make sure we won't incorrectly update other PRs. + BENCHMARK_COMMENT_ARTIFACT: ${{ steps.download.outputs.benchmark-comment-artifact }} + run: | + ./build_tools/benchmarks/post_benchmark_comment.py \ + --verbose \ + --github_event_json="${GITHUB_EVENT_PATH}" \ + "${BENCHMARK_COMMENT_ARTIFACT}" diff --git a/.github/workflows/reserves/iree/publish_website.yml b/.github/workflows/reserves/iree/publish_website.yml new file mode 100644 index 000000000000..adcdd5ce8fd5 --- /dev/null +++ b/.github/workflows/reserves/iree/publish_website.yml @@ -0,0 +1,77 @@ +# Builds the website sourced from docs/website/ using `mkdocs` and pushes +# to the gh-pages branch for publishing on GitHub Pages. +# +# See https://squidfunk.github.io/mkdocs-material/publishing-your-site/ + +name: Publish Website + +on: + workflow_dispatch: + push: + branches: + - main + paths: + # This file itself. + - ".github/workflows/publish_website.yml" + # Directly authored website source files. + - "docs/website/**" + # Python/pip release index page. + - "build_tools/scripts/generate_release_index.py" + # MLIR dialect definitions and .md generation using iree-tblgen. + # Technically this should also include the sources for Tablegen, but + # that rarely changes and we want to run this workflow conservatively. + - "**.td" + - "build_tools/cmake/iree_tablegen_doc.cmake" + # Regenerate the release pip index when a release is created or deleted. + release: + types: [published, unpublished] + +jobs: + publish_website: + # Note: a clean build of `iree-tblgen` takes ~5 minutes on standard runners. + runs-on: ubuntu-20.04 + steps: + - name: Checkout out repository + uses: actions/checkout@v4.1.7 + with: + submodules: true + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + - name: Fetching base gh-pages branch + # We have to explicitly fetch the gh-pages branch as well to preserve history + run: git fetch --no-tags --prune --depth=1 origin "gh-pages:gh-pages" + - name: Setting up Python + uses: actions/setup-python@v5.1.0 + with: + python-version: 3.x + cache: "pip" + - id: "gcp-auth" + name: "Authenticating to Google Cloud" + uses: "google-github-actions/auth@v1" + with: + token_format: "access_token" + credentials_json: "${{ secrets.IREE_OSS_GITHUB_RUNNER_BASIC_TRUST_SERVICE_ACCOUNT_KEY }}" + create_credentials_file: false + - name: Installing dependencies + run: | + pip install -r docs/website/requirements.txt + pip install requests + - name: Generating release index + run: | + ./build_tools/scripts/generate_release_index.py \ + --repo="${GITHUB_REPOSITORY}" \ + --output=docs/website/docs/pip-release-links.html + - name: Building documentation files + run: | + ./build_tools/github_actions/docker_run.sh \ + --env "IREE_CCACHE_GCP_TOKEN=${{ steps.gcp-auth.outputs.access_token }}" \ + --env "IREE_WRITE_REMOTE_CCACHE=1" \ + --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" \ + gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + ./docs/website/generate_extra_files.sh + - name: Setting git config + run: | + git config --local user.email "iree-github-actions-bot@google.com" + git config --local user.name "Website Publish Action" + - name: Deploying to gh-pages + working-directory: docs/website + run: mkdocs gh-deploy diff --git a/.github/workflows/reserves/iree/samples.yml b/.github/workflows/reserves/iree/samples.yml new file mode 100644 index 000000000000..ad83810af72b --- /dev/null +++ b/.github/workflows/reserves/iree/samples.yml @@ -0,0 +1,90 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# Workflow for running Examples of IREE usage against releases periodically. + +name: samples + +on: + schedule: + # Weekdays at 13:00 UTC = 05:00 PST / 06:00 PDT. No one should be dealing + # with this on weekends. + - cron: "5 4 * * 1-5" + workflow_dispatch: + +permissions: + contents: read + +concurrency: + # A PR number if a pull request and otherwise the commit hash. This cancels + # queued and in-progress runs for the same PR (presubmit) or commit + # (postsubmit). The workflow name is prepended to avoid conflicts between + # different workflows. + group: ${{ github.workflow }}-${{ github.event.number || github.sha }} + cancel-in-progress: true + +jobs: + colab: + runs-on: ubuntu-20.04 + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Setting up Python" + uses: actions/setup-python@v5.1.0 + with: + python-version: "3.11" + - name: "Testing Colab Notebooks" + run: ./samples/colab/test_notebooks.py + + samples: + runs-on: ubuntu-20.04 + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + - name: "Checking out runtime submodules" + run: ./build_tools/scripts/git/update_runtime_submodules.sh + - name: "Installing build dependencies" + run: | + sudo apt update + sudo apt install -y cmake clang ninja-build libstdc++-12-dev + echo "CC=clang" >> $GITHUB_ENV + echo "CXX=clang++" >> $GITHUB_ENV + - name: "Setting up Python" + uses: actions/setup-python@v5.1.0 + with: + python-version: "3.11" + - name: "Testing Samples" + run: ./build_tools/testing/test_samples.sh + + # TODO(scotttodd): Convert web test scripts to use installed compiler packages and local runtime + # builds, rather than local compiler builds. The host tools are too slow to build + # on standard sized runners (and for casual users). + + # web: + # runs-on: ubuntu-20.04 + # # TODO: remove dependence on magic matching build dir names + # env: + # HOST_BUILD_DIR: build-host-install + # steps: + # - name: "Checking out repository" + # uses: actions/checkout@v4.1.7 + # with: + # submodules: true + # - name: "Building host tools" + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env "CCACHE_NAMESPACE=gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446" \ + # gcr.io/iree-oss/base@sha256:dc314b4fe30fc1315742512891357bffed4d1b62ffcb46258b1e0761c737b446 \ + # ./build_tools/cmake/build_host_tools.sh "${HOST_BUILD_DIR}" + # - name: "Testing Experimental Web Samples" + # env: + # IREE_EMPSCRIPTEN_BUILD_DIR: build-emscripten + # run: | + # ./build_tools/github_actions/docker_run.sh \ + # --env "IREE_HOST_BUILD_DIR=${HOST_BUILD_DIR}" \ + # --env "IREE_EMPSCRIPTEN_BUILD_DIR=${IREE_EMPSCRIPTEN_BUILD_DIR}" \ + # gcr.io/iree-oss/emscripten@sha256:2dd4c52f1bb499ab365aad0111fe5538b685d88af38636b409b0cf6a576ab214 \ + # ./experimental/web/build_and_test_samples.sh diff --git a/.github/workflows/reserves/iree/schedule_candidate_release.yml b/.github/workflows/reserves/iree/schedule_candidate_release.yml new file mode 100644 index 000000000000..80d17b20d9f4 --- /dev/null +++ b/.github/workflows/reserves/iree/schedule_candidate_release.yml @@ -0,0 +1,78 @@ +name: Schedule candidate release + +on: + schedule: + - cron: "0 10 * * *" + + workflow_dispatch: + inputs: + release-commit: + description: "Explicit commit to release at (overrides green commit check)" + required: false + default: "" + +jobs: + tag_release: + name: "Tag candidate release" + runs-on: ubuntu-20.04 + steps: + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + fetch-depth: 0 + + - name: Checkout the most recent green commit + id: last_green_commit + env: + GH_TOKEN: ${{ github.token }} + run: | + if [ -z "${{ github.event.inputs.release-commit }}" ]; then + release_commit="$(./build_tools/scripts/get_latest_green.sh)" + else + release_commit="${{ github.event.inputs.release-commit }}" + fi + git checkout "${release_commit}" + echo "release-commit=${release_commit}" >> "${GITHUB_OUTPUT}" + + - name: Compute version + run: | + package_version="$(printf '%(%Y%m%d)T.${{ github.run_number }}')" + tag_name="candidate-${package_version}" + echo "package_version=${package_version}" >> $GITHUB_ENV + echo "tag_name=${tag_name}" >> $GITHUB_ENV + + - name: Updating candidate tag + run: | + git tag "${tag_name}" + + - name: Pushing changes + run: git push origin "${tag_name}" + + - name: Create Release + id: create_release + uses: actions/create-release@0cb9c9b65d5d1901c1f53e5e66eaf4afd303e70e # v1 + env: + GITHUB_TOKEN: ${{ secrets.WRITE_ACCESS_TOKEN }} + with: + tag_name: ${{ env.tag_name }} + release_name: iree candidate ${{ env.tag_name }} + commitish: ${{ steps.last_green_commit.outputs.release-commit }} + body: | + Automatic candidate release of iree. + draft: true + prerelease: true + + - name: "Invoke workflow :: Build Release Packages" + uses: benc-uk/workflow-dispatch@798e70c97009500150087d30d9f11c5444830385 # v1.2.2 + with: + workflow: Build Release Packages + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + ref: "${{ env.tag_name }}" + inputs: | + { + "package_suffix": "", + "package_version": "${{ env.package_version }}", + "release_id": "${{ steps.create_release.outputs.id }}", + "commit": "${{ steps.last_green_commit.outputs.release-commit }}" + } diff --git a/.github/workflows/reserves/iree/setup.yml b/.github/workflows/reserves/iree/setup.yml new file mode 100644 index 000000000000..67403efe9646 --- /dev/null +++ b/.github/workflows/reserves/iree/setup.yml @@ -0,0 +1,116 @@ +# Copyright 2022 The IREE Authors +# +# Licensed under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +# Sets up a github workflow. +# It is designed to be called from a parent workflow. +# The concurrency of this workflow is controlled by the caller's job. + +name: Setup + +on: + workflow_call: + outputs: + enabled-jobs: + description: | + Which jobs should run. + value: ${{ jobs.setup.outputs.enabled-jobs }} + is-pr: + description: | + Whether the workflow has been triggered by a pull request. + value: ${{ jobs.setup.outputs.is-pr }} + runner-env: + description: | + The runner environment to use. + value: ${{ jobs.setup.outputs.runner-env }} + runner-group: + description: | + The runner group to use. + value: ${{ jobs.setup.outputs.runner-group }} + write-caches: + description: | + Whether to write to caches. + value: ${{ jobs.setup.outputs.write-caches }} + benchmark-presets: + description: | + A comma-separated string of benchmarks to run. + value: ${{ jobs.setup.outputs.benchmark-presets }} + +permissions: + contents: read + +jobs: + setup: + runs-on: ubuntu-20.04 + env: + # The commit being checked out is the merge commit for the PR. Its first + # parent will be the tip of main. + BASE_REF: HEAD^ + outputs: + enabled-jobs: ${{ steps.configure.outputs.enabled-jobs }} + is-pr: ${{ steps.configure.outputs.is-pr }} + runner-env: ${{ steps.configure.outputs.runner-env }} + runner-group: ${{ steps.configure.outputs.runner-group }} + write-caches: ${{ steps.configure.outputs.write-caches }} + benchmark-presets: ${{ steps.configure.outputs.benchmark-presets }} + steps: + - name: "Checking out repository" + uses: actions/checkout@v4.1.7 + with: + # We need the parent commit to do a diff + fetch-depth: 2 + - name: "Fetching PR description" + # We fetch the latest pull request data (description, labels, ...) from + # API instead of using stale one from pull_request event. This makes it + # possible to update the trailers, labels on the pull request and re-run + # the workflow to make them take effect. + # This is majorly for triggering benchmarks without pushing new commits. + # See https://github.com/iree-org/iree/issues/10042#issuecomment-1449250094 + # for more details. + id: fetch-pr + if: github.event_name == 'pull_request' + env: + PR_NUMBER: ${{ github.event.number }} + PR_JSON: pull_request.json + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + gh api "/repos/${GITHUB_REPOSITORY}/pulls/${PR_NUMBER}" > "${PR_JSON}" + # It requires delimiter to pass multiline strings through + # GITHUB_OUTPUT. Since these are already escaped JSON strings, pass + # the JSON strings and later use fromJSON to decode them. + echo "pr-title=$(jq '.title' ${PR_JSON})" >> "${GITHUB_OUTPUT}" + echo "pr-body=$(jq '.body' ${PR_JSON})" >> "${GITHUB_OUTPUT}" + echo "pr-branch=$(jq '.head.ref' ${PR_JSON})" >> "${GITHUB_OUTPUT}" + # Use --compact-output to avoid multiline JSON. + echo "pr-labels=$(jq --compact-output '.labels | map(.name)' \ + ${PR_JSON})" >> "${GITHUB_OUTPUT}" + - name: "Configuring CI options" + id: configure + env: + PR_TITLE: ${{ fromJSON(steps.fetch-pr.outputs.pr-title || '""') }} + PR_BODY: ${{ fromJSON(steps.fetch-pr.outputs.pr-body || '""') }} + PR_BRANCH: ${{ fromJSON(steps.fetch-pr.outputs.pr-branch || '""') }} + PR_LABELS: ${{ steps.fetch-pr.outputs.pr-labels || '[]' }} + ORIGINAL_PR_TITLE: ${{ github.event.pull_request.title }} + ORIGINAL_PR_BODY: ${{ github.event.pull_request.body }} + ORIGINAL_PR_LABELS: ${{ toJSON(github.event.pull_request.labels.*.name) }} + run: | + # Just informative logging. There should only be two commits in the + # history here, but limiting the depth helps when copying from a local + # repo instead of using checkout, e.g. with + # https://github.com/nektos/act where there will be more. + git log --oneline --graph --max-count=3 + + ./build_tools/github_actions/configure_ci.py + + - name: "Show enabled options" + env: + BENCHMARK_PRESETS: ${{ steps.configure.outputs.benchmark-presets || 'None' }} + ENABLED_JOBS: ${{ join(fromJson(steps.configure.outputs.enabled-jobs)) || 'None' }} + run: | + echo ":green_circle: Enabled jobs: \`${ENABLED_JOBS}\`" \ + >> "${GITHUB_STEP_SUMMARY}" + echo ":stopwatch: Enabled benchmarks: \`${BENCHMARK_PRESETS}\`" \ + >> "${GITHUB_STEP_SUMMARY}" diff --git a/.github/workflows/reserves/iree/validate_and_publish_release.yml b/.github/workflows/reserves/iree/validate_and_publish_release.yml new file mode 100644 index 000000000000..ffcf39f79bc1 --- /dev/null +++ b/.github/workflows/reserves/iree/validate_and_publish_release.yml @@ -0,0 +1,137 @@ +# Publishes a draft snapshot, after performing validation to make sure all linux +# packages are stable. +name: Validate and Publish Release + +on: + workflow_dispatch: + inputs: + release_id: + description: "Release id to publish" + required: true + package_version: + description: "Version of the package" + required: true + build_run_id: + description: "Run ID for the build_package.yml workflow that triggered this workflow" + required: true + +jobs: + validate_packages: + name: "Validate packages" + # TODO(jennik): Look into testing windows and macos builds. + runs-on: ubuntu-20.04 + steps: + - name: Download packages + id: download_packages + uses: dawidd6/action-download-artifact@5e780fc7bbd0cac69fc73271ed86edf5dcb72d67 # v2.26.0 + with: + github_token: ${{secrets.WRITE_ACCESS_TOKEN}} + workflow: build_package.yml + run_id: ${{ github.event.inputs.build_run_id }} + - name: Extract and display downloaded files + run: | + tar -xf wheels-linux-x86_64-main-dist-linux/iree-dist-${{ github.event.inputs.package_version }}-linux-x86_64.tar.xz + pwd + ls -R + - name: Set up python + id: set_up_python + uses: actions/setup-python@v5.1.0 + with: + python-version: "3.9" + - name: Install python packages + id: install_python_packages + run: | + python -m pip install -f file://$PWD/wheels-linux-x86_64-py-compiler-pkg/ iree-compiler[onnx] + python -m pip install -f file://$PWD/wheels-linux-x86_64-py-runtime-pkg/ iree-runtime + python -m pip install -f file://$PWD/wheels-linux-x86_64-py-tf-compiler-tools-pkg/ iree-tools-tflite iree-tools-tf + - name: Validate IREE Compiler Package + id: validate_compiler_package + run: | + echo "Testing compiler package:" + python -m iree.compiler._package_test + - name: Validate IREE Runtime Package + id: validate_runtime_package + run: | + echo "Testing default runtime:" + python -m iree.runtime._package_test + echo "Testing tracy runtime:" + # GH runners don't expose the TSC but we want to make sure the basic packaging + # works, so override the check with TRACY_NO_INVARIANT_CHECK=1 (per instructions + # if this is left off). + TRACY_NO_INVARIANT_CHECK=1 IREE_PY_RUNTIME=tracy \ + python -m iree.runtime._package_test + # Binaries from the tarball + - name: Run iree-benchmark-executable + id: run_iree_benchmark_executable + run: ./bin/iree-benchmark-executable --help + - name: Run iree-benchmark-module + id: run_iree_benchmark_module + run: ./bin/iree-benchmark-module --help + - name: Run iree-dump-module + id: run_iree_dump_module + run: ./bin/iree-dump-module --help + - name: Run iree-cpuinfo + id: run_iree_cpuinfo + run: ./bin/iree-cpuinfo + - name: Run iree-flatcc-cli + id: run_iree_flatcc_cli + run: ./bin/iree-flatcc-cli --help + - name: Run iree-opt + id: run_iree_opt + run: ./bin/iree-opt --help + - name: Run iree-run-mlir + id: run_iree_run_mlir + run: ./bin/iree-run-mlir --help + - name: Run iree-run-module + id: run_iree_run_module + run: ./bin/iree-run-module --help + - name: Run iree-tblgen + id: run_iree_tblgen + run: ./bin/iree-tblgen --help + - name: Run iree-compile + id: run_iree-compile + run: ./bin/iree-compile --help + # Console scripts from the wheels. + - name: Py iree-run-module + id: py_iree-run-module + run: iree-run-module --help + - name: Py iree-benchmark-executable + id: py_iree_benchmark_executable + run: iree-benchmark-executable --help + - name: Py iree-benchmark-module + id: py_iree_benchmark_module + run: iree-benchmark-module --help + - name: Py iree-dump-module + id: py_iree_dump_module + run: iree-dump-module --help + - name: Py iree-cpuinfo + id: py_iree_cpuinfo + run: iree-cpuinfo + + publish_release: + name: "Publish release" + needs: validate_packages + runs-on: ubuntu-20.04 + steps: + - name: Publish Release + id: publish_release + uses: eregon/publish-release@46913fa2b3f7edc7345ae3c17f6d1b093a54916d # v1.0.5 + env: + GITHUB_TOKEN: ${{ secrets.WRITE_ACCESS_TOKEN }} + with: + release_id: ${{ github.event.inputs.release_id }} + + - name: Checking out repository + uses: actions/checkout@v4.1.7 + with: + token: ${{ secrets.WRITE_ACCESS_TOKEN }} + # Get all history. Otherwise the latest-snapshot branch can't be + # fast-forwarded. + fetch-depth: 0 + + - name: Updating latest-snapshot branch + uses: ad-m/github-push-action@v0.8.0 + with: + github_token: ${{ secrets.WRITE_ACCESS_TOKEN }} + branch: latest-snapshot + force: true diff --git a/.github/workflows/reserves/kernel/check-coverage.ps1 b/.github/workflows/reserves/kernel/check-coverage.ps1 new file mode 100644 index 000000000000..39fec480cb40 --- /dev/null +++ b/.github/workflows/reserves/kernel/check-coverage.ps1 @@ -0,0 +1,54 @@ +param ( + [string]$JsonReportPath, + [double]$CoverageThreshold +) + +$jsonContent = Get-Content $JsonReportPath -Raw | ConvertFrom-Json +$coverageBelowThreshold = $false + +function Get-FormattedValue($number) { + $formattedNumber = "{0:N1}" -f $number + $icon = if ($number -ge $CoverageThreshold) { '✅' } else { '❌' } + + return "$formattedNumber% $icon" +} + +$lineCoverage = $jsonContent.summary.linecoverage +$branchCoverage = $jsonContent.summary.branchcoverage + +if ($lineCoverage -lt $CoverageThreshold -or $branchCoverage -lt $CoverageThreshold) { + $coverageBelowThreshold = $true +} + +$totalTableData = [PSCustomObject]@{ + 'Metric' = 'Total Coverage' + 'Line Coverage' = Get-FormattedValue $lineCoverage + 'Branch Coverage' = Get-FormattedValue $branchCoverage +} + +$totalTableData | Format-Table -AutoSize + +$assemblyTableData = @() + +foreach ($assembly in $jsonContent.coverage.assemblies) { + $assemblyName = $assembly.name + $assemblyLineCoverage = $assembly.coverage + $assemblyBranchCoverage = $assembly.branchcoverage + + if ($assemblyLineCoverage -lt $CoverageThreshold -or $assemblyBranchCoverage -lt $CoverageThreshold) { + $coverageBelowThreshold = $true + } + + $assemblyTableData += [PSCustomObject]@{ + 'Assembly Name' = $assemblyName + 'Line' = Get-FormattedValue $assemblyLineCoverage + 'Branch' = Get-FormattedValue $assemblyBranchCoverage + } +} + +$assemblyTableData | Format-Table -AutoSize + +if ($coverageBelowThreshold) { + Write-Host "Code coverage is lower than defined threshold: $CoverageThreshold. Stopping the task." + exit 1 +} diff --git a/.github/workflows/reserves/kernel/close-inactive-issues.yml b/.github/workflows/reserves/kernel/close-inactive-issues.yml new file mode 100644 index 000000000000..10244c7609d9 --- /dev/null +++ b/.github/workflows/reserves/kernel/close-inactive-issues.yml @@ -0,0 +1,22 @@ +name: Close inactive issues +on: + schedule: + - cron: "30 1 * * *" + +jobs: + close-issues: + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + steps: + - uses: actions/stale@v5 + with: + days-before-issue-stale: 90 + days-before-issue-close: 14 + stale-issue-label: "stale" + stale-issue-message: "This issue is stale because it has been open for 90 days with no activity." + close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale." + days-before-pr-stale: -1 + days-before-pr-close: -1 + repo-token: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/kernel/codeql-analysis.yml b/.github/workflows/reserves/kernel/codeql-analysis.yml new file mode 100644 index 000000000000..a0c8cea4b83c --- /dev/null +++ b/.github/workflows/reserves/kernel/codeql-analysis.yml @@ -0,0 +1,79 @@ +# CodeQL is the code analysis engine developed by GitHub to automate security checks. +# The results are shown as code scanning alerts in GitHub. For more details, visit: +# https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql + +name: "CodeQL" + +on: + push: + branches: ["main", "experimental*", "feature*", "*-development"] + schedule: + - cron: "17 11 * * 2" + +jobs: + analyze: + name: Analyze + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + language: ["csharp", "python", "java"] + # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python', 'ruby' ] + # Use only 'java' to analyze code written in Java, Kotlin or both + # Use only 'javascript' to analyze code written in JavaScript, TypeScript or both + # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + # Initializes the CodeQL tools for scanning. + - name: Initialize CodeQL + uses: github/codeql-action/init@v2 + with: + languages: ${{ matrix.language }} + # If you wish to specify custom queries, you can do so here or in a config file. + # By default, queries listed here will override any specified in a config file. + # Prefix the list here with "+" to use these queries and those in the config file. + + # Details on CodeQL's query packs refer to : https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs + # queries: security-extended,security-and-quality + + # Autobuild attempts to build any compiled languages (C/C++, C#, Go, or Java). + # If this step fails, then you should remove it and run the build manually (see below) + - name: Autobuild + if: ${{ matrix.language != 'java' }} + uses: github/codeql-action/autobuild@v2 + + - name: Setup JDK + uses: actions/setup-java@v4 + if: ${{ matrix.language == 'java' }} + with: + java-version: 17 + distribution: microsoft + cache: maven + + - name: Build Java + if: ${{ matrix.language == 'java' }} + run: ./mvnw -B -DskipTests -Pcompile-jdk17 clean install --file pom.xml + working-directory: java + + # ℹ️ Command-line programs to run using the OS shell. + # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun + + # If the Autobuild fails above, remove it and uncomment the following three lines. + # modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance. + + # - run: | + # echo "Run, Build Application using script" + # ./location_of_script_within_repo/buildscript.sh + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v2 + with: + category: "/language:${{matrix.language}}" diff --git a/.github/workflows/reserves/kernel/dotnet-build-and-test.yml b/.github/workflows/reserves/kernel/dotnet-build-and-test.yml new file mode 100644 index 000000000000..6fbc987a6e98 --- /dev/null +++ b/.github/workflows/reserves/kernel/dotnet-build-and-test.yml @@ -0,0 +1,176 @@ +# +# This workflow will build and run all unit tests using dotnet docker containers, +# each targeting a single version of the dotnet SDK. +# + +name: dotnet-build-and-test + +on: + workflow_dispatch: + pull_request: + branches: ["main", "feature*"] + merge_group: + branches: ["main"] + +env: + COVERAGE_THRESHOLD: 80 + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + paths-filter: + runs-on: ubuntu-latest + outputs: + dotnetChanges: ${{ steps.filter.outputs.dotnet }} + steps: + - uses: actions/checkout@v4 + - uses: dorny/paths-filter@v2 + id: filter + with: + filters: | + dotnet: + - 'dotnet/**' + - '**/dotnet/**' + - 'samples/skills/**' + # run only if 'dotnet' files were changed + - name: dotnet tests + if: steps.filter.outputs.dotnet == 'true' + run: echo "Dotnet file" + # run only if not 'dotnet' files were changed + - name: not dotnet tests + if: steps.filter.outputs.dotnet != 'true' + run: echo "NOT dotnet file" + dotnet-build-and-test: + needs: paths-filter + if: needs.paths-filter.outputs.dotnetChanges == 'true' + strategy: + fail-fast: false + matrix: + include: + - { + dotnet: "8.0", + os: "ubuntu-latest", + configuration: Release, + integration-tests: true, + } + - { dotnet: "8.0", os: "windows-latest", configuration: Debug } + - { dotnet: "8.0", os: "windows-latest", configuration: Release } + + runs-on: ${{ matrix.os }} + steps: + - uses: actions/checkout@v4 + - name: Setup dotnet ${{ matrix.dotnet }} + uses: actions/setup-dotnet@v3 + with: + dotnet-version: ${{ matrix.dotnet }} + - name: Build dotnet solutions + shell: bash + run: | + export SOLUTIONS=$(find ./dotnet/ -type f -name "*.sln" | tr '\n' ' ') + for solution in $SOLUTIONS; do + dotnet build $solution -c ${{ matrix.configuration }} --warnaserror + done + + - name: Run Unit Tests + shell: bash + run: | + export UT_PROJECTS=$(find ./dotnet -type f -name "*.UnitTests.csproj" | grep -v -E "(Experimental.Orchestration.Flow.UnitTests.csproj|Experimental.Assistants.UnitTests.csproj)" | tr '\n' ' ') + for project in $UT_PROJECTS; do + dotnet test -c ${{ matrix.configuration }} $project --no-build -v Normal --logger trx --collect:"XPlat Code Coverage" --results-directory:"TestResults/Coverage/" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.ExcludeByAttribute=GeneratedCodeAttribute,CompilerGeneratedAttribute,ExcludeFromCodeCoverageAttribute + done + + - name: Run Integration Tests + shell: bash + if: github.event_name != 'pull_request' && matrix.integration-tests + run: | + export INTEGRATION_TEST_PROJECTS=$(find ./dotnet -type f -name "*IntegrationTests.csproj" | grep -v "Experimental.Orchestration.Flow.IntegrationTests.csproj" | tr '\n' ' ') + for project in $INTEGRATION_TEST_PROJECTS; do + dotnet test -c ${{ matrix.configuration }} $project --no-build -v Normal --logger trx + done + env: + AzureOpenAI__Label: azure-text-davinci-003 + AzureOpenAIEmbedding__Label: azure-text-embedding-ada-002 + AzureOpenAI__DeploymentName: ${{ vars.AZUREOPENAI__DEPLOYMENTNAME }} + AzureOpenAIEmbeddings__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDING__DEPLOYMENTNAME }} + AzureOpenAI__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }} + AzureOpenAIEmbeddings__Endpoint: ${{ secrets.AZUREOPENAI_EASTUS__ENDPOINT }} + AzureOpenAI__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }} + AzureOpenAIEmbeddings__ApiKey: ${{ secrets.AZUREOPENAI_EASTUS__APIKEY }} + Planners__AzureOpenAI__ApiKey: ${{ secrets.PLANNERS__AZUREOPENAI__APIKEY }} + Planners__AzureOpenAI__Endpoint: ${{ secrets.PLANNERS__AZUREOPENAI__ENDPOINT }} + Planners__AzureOpenAI__DeploymentName: ${{ vars.PLANNERS__AZUREOPENAI__DEPLOYMENTNAME }} + Planners__OpenAI__ApiKey: ${{ secrets.PLANNERS__OPENAI__APIKEY }} + Planners__OpenAI__ModelId: ${{ vars.PLANNERS__OPENAI__MODELID }} + OpenAITextToAudio__ApiKey: ${{ secrets.OPENAITEXTTOAUDIO__APIKEY }} + OpenAITextToAudio__ModelId: ${{ vars.OPENAITEXTTOAUDIO__MODELID }} + OpenAIAudioToText__ApiKey: ${{ secrets.OPENAIAUDIOTOTEXT__APIKEY }} + OpenAIAudioToText__ModelId: ${{ vars.OPENAIAUDIOTOTEXT__MODELID }} + AzureOpenAITextToAudio__ApiKey: ${{ secrets.AZUREOPENAITEXTTOAUDIO__APIKEY }} + AzureOpenAITextToAudio__Endpoint: ${{ secrets.AZUREOPENAITEXTTOAUDIO__ENDPOINT }} + AzureOpenAITextToAudio__DeploymentName: ${{ vars.AZUREOPENAITEXTTOAUDIO__DEPLOYMENTNAME }} + AzureOpenAIAudioToText__ApiKey: ${{ secrets.AZUREOPENAIAUDIOTOTEXT__APIKEY }} + AzureOpenAIAudioToText__Endpoint: ${{ secrets.AZUREOPENAIAUDIOTOTEXT__ENDPOINT }} + AzureOpenAIAudioToText__DeploymentName: ${{ vars.AZUREOPENAIAUDIOTOTEXT__DEPLOYMENTNAME }} + Bing__ApiKey: ${{ secrets.BING__APIKEY }} + OpenAI__ApiKey: ${{ secrets.OPENAI__APIKEY }} + + # Generate test reports and check coverage + - name: Generate test reports + uses: danielpalme/ReportGenerator-GitHub-Action@5.2.4 + with: + reports: "./TestResults/Coverage/**/coverage.cobertura.xml" + targetdir: "./TestResults/Reports" + reporttypes: "JsonSummary" + assemblyfilters: "+Microsoft.SemanticKernel.Abstractions;+Microsoft.SemanticKernel.Core;+Microsoft.SemanticKernel.PromptTemplates.Handlebars;+Microsoft.SemanticKernel.Connectors.OpenAI;+Microsoft.SemanticKernel.Yaml;+Microsoft.SemanticKernel.Agents.Abstractions;+Microsoft.SemanticKernel.Agents.Core;+Microsoft.SemanticKernel.Agents.OpenAI" + + - name: Check coverage + shell: pwsh + run: .github/workflows/check-coverage.ps1 -JsonReportPath "TestResults/Reports/Summary.json" -CoverageThreshold $env:COVERAGE_THRESHOLD + + # This final job is required to satisfy the merge queue. It must only run (or succeed) if no tests failed + dotnet-build-and-test-check: + if: always() + runs-on: ubuntu-latest + needs: [dotnet-build-and-test] + steps: + - name: Get Date + shell: bash + run: | + echo "date=$(date +'%m/%d/%Y %H:%M:%S')" >> "$GITHUB_ENV" + + - name: Run Type is Daily + if: ${{ github.event_name == 'schedule' }} + shell: bash + run: | + echo "run_type=Daily" >> "$GITHUB_ENV" + + - name: Run Type is Manual + if: ${{ github.event_name == 'workflow_dispatch' }} + shell: bash + run: | + echo "run_type=Manual" >> "$GITHUB_ENV" + + - name: Run Type is ${{ github.event_name }} + if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch'}} + shell: bash + run: | + echo "run_type=${{ github.event_name }}" >> "$GITHUB_ENV" + + - name: Fail workflow if tests failed + id: check_tests_failed + if: contains(join(needs.*.result, ','), 'failure') + uses: actions/github-script@v6 + with: + script: core.setFailed('Integration Tests Failed!') + + - name: Fail workflow if tests cancelled + id: check_tests_cancelled + if: contains(join(needs.*.result, ','), 'cancelled') + uses: actions/github-script@v6 + with: + script: core.setFailed('Integration Tests Cancelled!') diff --git a/.github/workflows/reserves/kernel/dotnet-ci.yml b/.github/workflows/reserves/kernel/dotnet-ci.yml new file mode 100644 index 000000000000..8a4899735f3f --- /dev/null +++ b/.github/workflows/reserves/kernel/dotnet-ci.yml @@ -0,0 +1,127 @@ +# +# This workflow will build and run all tests using dotnet docker containers, +# each targeting a single version of the dotnet SDK. +# + +name: dotnet-ci + +on: + workflow_dispatch: + schedule: + - cron: '0 7 * * *' + +permissions: + contents: read + +jobs: + build-and-test-docker: + strategy: + fail-fast: false + matrix: + include: + - { os: ubuntu-latest, dotnet: '8.0', configuration: Debug } + - { os: ubuntu-latest, dotnet: '8.0', configuration: Release } + + runs-on: ${{ matrix.os }} + steps: + - uses: actions/checkout@v4 + with: + clean: true + + - name: Find solutions + shell: bash + run: echo "solutions=$(find ./ -type f -name "*.sln" | tr '\n' ' ')" >> $GITHUB_ENV + + - name: Pull container dotnet/sdk:${{ matrix.dotnet }} + run: | + docker pull mcr.microsoft.com/dotnet/sdk:${{ matrix.dotnet }} + + - name: Build + run: | + for solution in ${{ env.solutions }}; do + docker run --rm -v $(pwd):/app -w /app -e GITHUB_ACTIONS='true' mcr.microsoft.com/dotnet/sdk:${{ matrix.dotnet }} /bin/sh -c "dotnet build -c ${{ matrix.configuration }} /app/$solution" + done + + - name: Find test projects + shell: bash + run: echo "testprojects=$(find ./dotnet -type f -name "*UnitTests.csproj" | tr '\n' ' ')" >> $GITHUB_ENV + + - name: Run Tests + shell: bash + run: | + for project in ${{ env.testprojects }}; do + docker run --rm -v $(pwd):/app -w /app mcr.microsoft.com/dotnet/sdk:${{ matrix.dotnet }} /bin/sh -c "dotnet test -c ${{ matrix.configuration }} /app/$project --no-build -v Normal --logger trx" + done + + - name: Upload dotnet test results + uses: actions/upload-artifact@v3 + with: + name: dotnet-testresults-${{ matrix.configuration }} + path: ./TestResults + if: ${{ always() }} + + build-and-test-windows: + strategy: + fail-fast: false + matrix: + os: [windows-latest] + configuration: [Release, Debug] + dotnet-version: ['8.0.x'] + runs-on: ${{ matrix.os }} + env: + NUGET_CERT_REVOCATION_MODE: offline + steps: + - uses: actions/checkout@v4 + with: + clean: true + + - name: Setup .NET SDK ${{ matrix.dotnet-version }} + uses: actions/setup-dotnet@v4 + with: + dotnet-version: ${{ matrix.dotnet-version }} + env: + NUGET_AUTH_TOKEN: ${{ secrets.GPR_READ_TOKEN }} + + - uses: actions/cache@v4 + with: + path: ~/.nuget/packages + # Look to see if there is a cache hit for the corresponding requirements file + key: ${{ runner.os }}-nuget-${{ hashFiles('**/packages.lock.json') }} + restore-keys: | + ${{ runner.os }}-nuget + + - name: Find solutions + shell: bash + run: echo "solutions=$(find ./dotnet -type f -name "*.sln" | tr '\n' ' ')" >> $GITHUB_ENV + + - name: Restore dependencies + shell: bash + run: | + for solution in ${{ env.solutions }}; do + dotnet restore $solution + done + + - name: Build + shell: bash + run: | + for solution in ${{ env.solutions }}; do + dotnet build $solution --no-restore --configuration ${{ matrix.configuration }} + done + + - name: Find test projects + shell: bash + run: echo "testprojects=$(find ./dotnet -type f -name "*Tests.csproj" | tr '\n' ' ')" >> $GITHUB_ENV + + - name: Run Tests + shell: bash + run: | + for project in ${{ env.testprojects }}; do + dotnet test $project --verbosity normal --logger trx --results-directory ./TestResults --configuration ${{ matrix.configuration }} + done + + - name: Upload dotnet test results + uses: actions/upload-artifact@v3 + with: + name: dotnet-testresults-${{ matrix.configuration }} + path: ./TestResults + if: ${{ always() }} diff --git a/.github/workflows/reserves/kernel/dotnet-format.yml b/.github/workflows/reserves/kernel/dotnet-format.yml new file mode 100644 index 000000000000..f23f993dbf19 --- /dev/null +++ b/.github/workflows/reserves/kernel/dotnet-format.yml @@ -0,0 +1,93 @@ +# +# This workflow runs the dotnet formatter on all c-sharp code. +# + +name: dotnet-format + +on: + workflow_dispatch: + pull_request: + branches: ["main", "feature*"] + paths: + - "dotnet/**" + - "samples/dotnet/**" + - "**.cs" + - "**.csproj" + - "**.editorconfig" + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + check-format: + strategy: + fail-fast: false + matrix: + include: + - { dotnet: "8.0", configuration: Release, os: ubuntu-latest } + + runs-on: ${{ matrix.os }} + env: + NUGET_CERT_REVOCATION_MODE: offline + + steps: + - name: Check out code + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Get changed files + id: changed-files + if: github.event_name == 'pull_request' + uses: jitterbit/get-changed-files@v1 + continue-on-error: true + + - name: No C# files changed + id: no-csharp + if: github.event_name == 'pull_request' && steps.changed-files.outputs.added_modified == '' + run: echo "No C# files changed" + + # This step will loop over the changed files and find the nearest .csproj file for each one, then store the unique csproj files in a variable + - name: Find csproj files + id: find-csproj + if: github.event_name != 'pull_request' || steps.changed-files.outputs.added_modified != '' || steps.changed-files.outcome == 'failure' + run: | + csproj_files=() + exclude_files=("Experimental.Orchestration.Flow.csproj" "Experimental.Orchestration.Flow.UnitTests.csproj" "Experimental.Orchestration.Flow.IntegrationTests.csproj") + if [[ ${{ steps.changed-files.outcome }} == 'success' ]]; then + for file in ${{ steps.changed-files.outputs.added_modified }}; do + echo "$file was changed" + dir="./$file" + while [[ $dir != "." && $dir != "/" && $dir != $GITHUB_WORKSPACE ]]; do + if find "$dir" -maxdepth 1 -name "*.csproj" -print -quit | grep -q .; then + csproj_path="$(find "$dir" -maxdepth 1 -name "*.csproj" -print -quit)" + if [[ ! "${exclude_files[@]}" =~ "${csproj_path##*/}" ]]; then + csproj_files+=("$csproj_path") + fi + break + fi + + dir=$(echo ${dir%/*}) + done + done + else + # if the changed-files step failed, run dotnet on the whole sln instead of specific projects + csproj_files=$(find ./ -type f -name "*.sln" | tr '\n' ' '); + fi + csproj_files=($(printf "%s\n" "${csproj_files[@]}" | sort -u)) + echo "Found ${#csproj_files[@]} unique csproj/sln files: ${csproj_files[*]}" + echo "::set-output name=csproj_files::${csproj_files[*]}" + + - name: Pull container dotnet/sdk:${{ matrix.dotnet }} + if: steps.find-csproj.outputs.csproj_files != '' + run: docker pull mcr.microsoft.com/dotnet/sdk:${{ matrix.dotnet }} + + # This step will run dotnet format on each of the unique csproj files and fail if any changes are made + - name: Run dotnet format + if: steps.find-csproj.outputs.csproj_files != '' + run: | + for csproj in ${{ steps.find-csproj.outputs.csproj_files }}; do + echo "Running dotnet format on $csproj" + docker run --rm -v $(pwd):/app -w /app mcr.microsoft.com/dotnet/sdk:${{ matrix.dotnet }} /bin/sh -c "dotnet format $csproj --verify-no-changes --verbosity diagnostic" + done diff --git a/.github/workflows/reserves/kernel/dotnet-integration-tests.yml b/.github/workflows/reserves/kernel/dotnet-integration-tests.yml new file mode 100644 index 000000000000..457e33de1ac2 --- /dev/null +++ b/.github/workflows/reserves/kernel/dotnet-integration-tests.yml @@ -0,0 +1,65 @@ +# +# This workflow will run all dotnet integrations tests. +# + +name: dotnet-integration-tests + +on: + workflow_dispatch: + pull_request: + branches: ["main"] + merge_group: + branches: ["main"] + +permissions: + contents: read + +jobs: + integration-tests: + strategy: + matrix: + os: [ubuntu-latest] + configuration: [Debug] + runs-on: ${{ matrix.os }} + steps: + - uses: actions/checkout@v4 + if: ${{ github.event_name != 'pull_request' }} + with: + clean: true + + - name: Setup .NET + uses: actions/setup-dotnet@v4 + if: ${{ github.event_name != 'pull_request' }} + with: + dotnet-version: 8.0.x + + - name: Find projects + shell: bash + if: ${{ github.event_name != 'pull_request' }} + run: echo "projects=$(find ./dotnet -type f -name "*Tests.csproj" | tr '\n' ' ')" >> $GITHUB_ENV + + - name: Integration Tests + shell: bash + if: ${{ github.event_name != 'pull_request' }} + env: # Set Azure credentials secret as an input + AzureOpenAI__Label: azure-text-davinci-003 + AzureOpenAIEmbedding__Label: azure-text-embedding-ada-002 + AzureOpenAI__DeploymentName: ${{ vars.AZUREOPENAI__DEPLOYMENTNAME }} + AzureOpenAIEmbeddings__DeploymentName: ${{ vars.AZUREOPENAIEMBEDDING__DEPLOYMENTNAME }} + AzureOpenAI__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }} + AzureOpenAIEmbeddings__Endpoint: ${{ secrets.AZUREOPENAI__ENDPOINT }} + AzureOpenAI__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }} + AzureOpenAIEmbeddings__ApiKey: ${{ secrets.AZUREOPENAI__APIKEY }} + Bing__ApiKey: ${{ secrets.BING__APIKEY }} + OpenAI__ApiKey: ${{ secrets.OPENAI__APIKEY }} + run: | + for project in ${{ env.projects }}; do + dotnet test $project --verbosity normal --logger trx --results-directory ./TestResults --configuration ${{ matrix.configuration }} + done + + - name: Upload dotnet test results + uses: actions/upload-artifact@v3 + with: + name: dotnet-testresults-${{ matrix.configuration }} + path: ./TestResults + if: ${{ github.event_name != 'pull_request' && always() }} diff --git a/.github/workflows/reserves/kernel/generate-pr-description.yml b/.github/workflows/reserves/kernel/generate-pr-description.yml new file mode 100644 index 000000000000..70261481efe0 --- /dev/null +++ b/.github/workflows/reserves/kernel/generate-pr-description.yml @@ -0,0 +1,61 @@ +name: Generate PR Description + +on: + issue_comment: + types: [created] + +jobs: + generate-pr-description: + permissions: + pull-requests: write + statuses: write + runs-on: ubuntu-latest + if: github.event.issue.pull_request && contains(github.event.comment.body, '/sk generate-pr-description') + steps: + - name: Get PR branch + uses: xt0rted/pull-request-comment-branch@v1 + id: comment-branch + + - name: Set latest commit status as pending + uses: myrotvorets/set-commit-status-action@master + with: + sha: ${{ steps.comment-branch.outputs.head_sha }} + token: ${{ secrets.GITHUB_TOKEN }} + status: pending + + - name: Generate PR description + uses: mkarle/skonsole-generate-pr-description@v1 + with: + pull-request-number: ${{ github.event.issue.number }} + pull-request-diff-url: ${{ github.event.issue.pull_request.diff_url }} + token: ${{ secrets.GITHUB_TOKEN }} + update-type: ${{ contains(github.event.comment.body, 'replace') && 'replace' || (contains(github.event.comment.body, 'prefix') && 'prefix' || 'suffix') }} + env: # Set Azure credentials secret as an input + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZUREOPENAI__CHAT__DEPLOYMENTNAME }} + AZURE_OPENAI_API_ENDPOINT: ${{ secrets.AZUREOPENAI__ENDPOINT }} + AZURE_OPENAI_API_KEY: ${{ secrets.AZUREOPENAI__APIKEY }} + + - name: Set latest commit status as ${{ job.status }} + uses: myrotvorets/set-commit-status-action@master + if: always() + with: + sha: ${{ steps.comment-branch.outputs.head_sha }} + token: ${{ secrets.GITHUB_TOKEN }} + status: ${{ job.status }} + + - name: Add comment to PR + uses: actions/github-script@v6 + if: always() + with: + script: | + const name = '${{ github.workflow }}'; + const url = '${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}'; + const success = '${{ job.status }}' === 'success'; + const body = `${name}: ${success ? 'succeeded ✅' : 'failed ❌'}\n${url}`; + + await github.rest.issues.createComment({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + body: body + }) diff --git a/.github/workflows/reserves/kernel/java-build.yml b/.github/workflows/reserves/kernel/java-build.yml new file mode 100644 index 000000000000..a27b337512fd --- /dev/null +++ b/.github/workflows/reserves/kernel/java-build.yml @@ -0,0 +1,92 @@ +name: Build Java Semantic Kernel + +# Triggers the workflow on manual dispatch, push, and pull request events +# for the specified branches and paths +on: + workflow_dispatch: + push: + branches: [ "main", "java-development" ] + paths: + - 'java/**' + pull_request: + branches: [ "main", "java-development" ] + paths: + - 'java/**' + +permissions: + contents: read + +jobs: + # Determines if Java files have changed + paths-filter: + runs-on: ubuntu-latest + outputs: + javaChanges: ${{ steps.filter.outputs.java }} + steps: + - uses: actions/checkout@v4 + - uses: dorny/paths-filter@v2 + id: filter + with: + filters: | + java: + - 'java/**' + - '**/java/**' + - name: java tests + if: steps.filter.outputs.java == 'true' + run: echo "Java file" + - name: not java tests + if: steps.filter.outputs.java != 'true' + run: echo "NOT java file" + + # Builds and tests the Java project + java-build: + runs-on: ubuntu-latest + needs: paths-filter + if: needs.paths-filter.outputs.javaChanges == 'true' + strategy: + fail-fast: false + matrix: + # Defines a matrix strategy for JDK versions 8 and 17 + java-versions: [8, 17] + + name: Java CI on JDK${{ matrix.java-versions }} + + steps: + - name: Checkout + uses: actions/checkout@v3 + + # Need to use JDK 11 to build for JDK 8 + - name: Set JDK + id: set-jdk + shell: bash + run: | + if [[ ${{ matrix.java-versions }} == 8 ]]; then + echo "JDK_VERSION=11" >> $GITHUB_OUTPUT + else + version=${{ matrix.java-versions }} + echo "JDK_VERSION=$version" >> $GITHUB_OUTPUT + fi + + # Sets up the specified JDK version from the matrix + - uses: actions/setup-java@v4 + with: + java-version: ${{ steps.set-jdk.outputs.JDK_VERSION }} + distribution: microsoft + cache: maven + + # Builds the project with Maven using the matrix JDK version + - name: Build with Maven + run: ./mvnw -B -Pbug-check -DskipTests -Pcompile-jdk${{ matrix.java-versions }} clean install --file pom.xml + working-directory: java + + # Runs tests with Maven using the matrix JDK version + - name: Run tests + run: ./mvnw -B -Pbug-check -Pcompile-jdk${{ matrix.java-versions }} test --file pom.xml + working-directory: java + + # Uploads test artifacts for each JDK version + - uses: actions/upload-artifact@v2 + if: always() + with: + name: test_output_sk_jdk${{ matrix.java-versions }}u + path: java/**/target/surefire-reports/*Test.txt diff --git a/.github/workflows/reserves/kernel/java-integration-tests.yml b/.github/workflows/reserves/kernel/java-integration-tests.yml new file mode 100644 index 000000000000..e690635444be --- /dev/null +++ b/.github/workflows/reserves/kernel/java-integration-tests.yml @@ -0,0 +1,92 @@ +name: Run Java Integration Tests and Samples + +on: + workflow_dispatch: + push: + branches: [ "main", "java-development" ] + paths: + - 'java/**' + pull_request: + branches: [ "main", "java-development" ] + paths: + - 'java/**' + +permissions: + contents: read + +jobs: + paths-filter: + runs-on: ubuntu-latest + outputs: + javaChanges: ${{ steps.filter.outputs.java }} + steps: + - uses: actions/checkout@v4 + - uses: dorny/paths-filter@v2 + id: filter + with: + filters: | + java: + - 'java/**' + - '**/java/**' + # run only if 'java' files were changed + - name: java tests + if: steps.filter.outputs.java == 'true' + run: echo "Java file" + # run only if not 'java' files were changed + - name: not java tests + if: steps.filter.outputs.java != 'true' + run: echo "NOT java file" + java-integration-tests: + runs-on: ubuntu-latest + needs: paths-filter + if: needs.paths-filter.outputs.javaChanges == 'true' + strategy: + fail-fast: false + matrix: + java-versions: [8, 17] + + name: Java Tests on JDK${{ matrix.java-versions }} + + steps: + - name: Checkout + uses: actions/checkout@v3 + + # Need to use JDK 11 to build for JDK 8 + - name: Set JDK + id: set-jdk + shell: bash + run: | + if [[ ${{ matrix.java-versions }} == 8 ]]; then + echo "JDK_VERSION=11" >> $GITHUB_OUTPUT + else + version=${{ matrix.java-versions }} + echo "JDK_VERSION=$version" >> $GITHUB_OUTPUT + fi + + - uses: actions/setup-java@v4 + with: + java-version: ${{ steps.set-jdk.outputs.JDK_VERSION }} + distribution: microsoft + cache: maven + + - name: Build with Maven + run: ./mvnw -B -Pwith-samples -Pbug-check -DskipTests -Pcompile-jdk${{ matrix.java-versions }} clean install --file pom.xml + working-directory: java + if: ${{ matrix.java-versions >= 17 }} + + # API tests run on JDK 17+ + - name: Run integration tests + run: ../mvnw -B -Pbug-check clean install --file pom.xml + working-directory: java/api-test + env: + OPENAI_API_KEY: "" + AZURE_OPENAI_ENDPOINT: "" + AZURE_OPENAI_API_KEY: "" + AZURE_OPENAI_DEPLOYMENT_NAME: "" + if: ${{ matrix.java-versions >= 17 }} + + # Samples build on JDK 17+ + - name: Build semantic-kernel samples + run: ../mvnw -B clean install --file pom.xml + working-directory: java/samples + if: ${{ matrix.java-versions >= 17 }} diff --git a/.github/workflows/reserves/kernel/label-issues.yml b/.github/workflows/reserves/kernel/label-issues.yml new file mode 100644 index 000000000000..a011f3b4e81f --- /dev/null +++ b/.github/workflows/reserves/kernel/label-issues.yml @@ -0,0 +1,54 @@ +name: Label issues +on: + issues: + types: + - reopened + - opened + +jobs: + label_issues: + name: "Issue: add labels" + if: ${{ github.event.action == 'opened' || github.event.action == 'reopened' }} + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - uses: actions/github-script@v6 + with: + github-token: ${{ secrets.GH_ACTIONS_PR_WRITE }} + script: | + // Get the issue body and title + const body = context.payload.issue.body + let title = context.payload.issue.title + + // Define the labels array + let labels = ["triage"] + + // Check if the body or the title contains the word 'python' (case-insensitive) + if ((body != null && body.match(/python/i)) || (title != null && title.match(/python/i))) { + // Add the 'python' label to the array + labels.push("python") + } + + // Check if the body or the title contains the word 'java' (case-insensitive) + if ((body != null && body.match(/java/i)) || (title != null && title.match(/java/i))) { + // Add the 'java' label to the array + labels.push("java") + } + + // Check if the body or the title contains the words 'dotnet', '.net', 'c#' or 'csharp' (case-insensitive) + if ((body != null && body.match(/.net/i)) || (title != null && title.match(/.net/i)) || + (body != null && body.match(/dotnet/i)) || (title != null && title.match(/dotnet/i)) || + (body != null && body.match(/C#/i)) || (title != null && title.match(/C#/i)) || + (body != null && body.match(/csharp/i)) || (title != null && title.match(/csharp/i))) { + // Add the '.NET' label to the array + labels.push(".NET") + } + + // Add the labels to the issue + github.rest.issues.addLabels({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + labels: labels + }); diff --git a/.github/workflows/reserves/kernel/label-pr.yml b/.github/workflows/reserves/kernel/label-pr.yml new file mode 100644 index 000000000000..61e77b7ff18c --- /dev/null +++ b/.github/workflows/reserves/kernel/label-pr.yml @@ -0,0 +1,21 @@ +# This workflow will triage pull requests and apply a label based on the +# paths that are modified in the pull request. +# +# To use this workflow, you will need to set up a .github/labeler.yml +# file with configuration. For more information, see: +# https://github.com/actions/labeler + +name: Label pull request +on: [pull_request_target] + +jobs: + add_label: + runs-on: ubuntu-latest + permissions: + contents: read + pull-requests: write + + steps: + - uses: actions/labeler@v4 + with: + repo-token: "${{ secrets.GH_ACTIONS_PR_WRITE }}" diff --git a/.github/workflows/reserves/kernel/label-title-prefix.yml b/.github/workflows/reserves/kernel/label-title-prefix.yml new file mode 100644 index 000000000000..b0c01aad3fcd --- /dev/null +++ b/.github/workflows/reserves/kernel/label-title-prefix.yml @@ -0,0 +1,73 @@ +name: Label title prefix +on: + issues: + types: [labeled] + pull_request_target: + types: [labeled] + +jobs: + add_title_prefix: + name: "Issue/PR: add title prefix" + continue-on-error: true + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + + steps: + - uses: actions/github-script@v6 + name: "Issue/PR: update title" + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + let prefixLabels = { + "python": "Python", + "java": "Java", + ".NET": ".Net" + }; + + function addTitlePrefix(title, prefix) + { + // Update the title based on the label and prefix + // Check if the title starts with the prefix (case-sensitive) + if (!title.startsWith(prefix + ": ")) { + // If not, check if the first word is the label (case-insensitive) + if (title.match(new RegExp(`^${prefix}`, 'i'))) { + // If yes, replace it with the prefix (case-sensitive) + title = title.replace(new RegExp(`^${prefix}`, 'i'), prefix); + } else { + // If not, prepend the prefix to the title + title = prefix + ": " + title; + } + } + + return title; + } + + labelAdded = context.payload.label.name + + // Check if the issue or PR has the label + if (labelAdded in prefixLabels) { + let prefix = prefixLabels[labelAdded]; + switch(context.eventName) { + case 'issues': + github.rest.issues.update({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + title: addTitlePrefix(context.payload.issue.title, prefix) + }); + break + + case 'pull_request_target': + github.rest.pulls.update({ + pull_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo, + title: addTitlePrefix(context.payload.pull_request.title, prefix) + }); + break + default: + core.setFailed('Unrecognited eventName: ' + context.eventName); + } + } diff --git a/.github/workflows/reserves/kernel/markdown-link-check-config.json b/.github/workflows/reserves/kernel/markdown-link-check-config.json new file mode 100644 index 000000000000..50ada4911de6 --- /dev/null +++ b/.github/workflows/reserves/kernel/markdown-link-check-config.json @@ -0,0 +1,39 @@ +{ + "ignorePatterns": [ + { + "pattern": "/github/" + }, + { + "pattern": "./actions" + }, + { + "pattern": "./blob" + }, + { + "pattern": "./issues" + }, + { + "pattern": "./discussions" + }, + { + "pattern": "./pulls" + }, + { + "pattern": "^http://localhost" + }, + { + "pattern": "^https://localhost" + }, + { + "pattern": "^https://platform.openai.com" + }, + { + "pattern": "^https://outlook.office.com/bookings" + } + ], + "timeout": "20s", + "retryOn429": true, + "retryCount": 3, + "fallbackRetryDelay": "30s", + "aliveStatusCodes": [200, 206, 429, 500, 503] +} diff --git a/.github/workflows/reserves/kernel/markdown-link-check.yml b/.github/workflows/reserves/kernel/markdown-link-check.yml new file mode 100644 index 000000000000..2cea71de5383 --- /dev/null +++ b/.github/workflows/reserves/kernel/markdown-link-check.yml @@ -0,0 +1,23 @@ +name: Check .md links + +on: + workflow_dispatch: + pull_request: + branches: ["main", "java-development"] + +permissions: + contents: read + +jobs: + markdown-link-check: + runs-on: ubuntu-latest + # check out the latest version of the code + steps: + - uses: actions/checkout@v4 + + # Checks the status of hyperlinks in .md files in verbose mode + - name: Check links + uses: gaurav-nelson/github-action-markdown-link-check@v1 + with: + use-verbose-mode: "yes" + config-file: ".github/workflows/markdown-link-check-config.json" diff --git a/.github/workflows/reserves/kernel/merge-gatekeeper.yml b/.github/workflows/reserves/kernel/merge-gatekeeper.yml new file mode 100644 index 000000000000..9f0e2d22c694 --- /dev/null +++ b/.github/workflows/reserves/kernel/merge-gatekeeper.yml @@ -0,0 +1,32 @@ +name: Merge Gatekeeper + +on: + pull_request: + branches: [ "main", "feature*" ] + merge_group: + branches: ["main"] + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} + cancel-in-progress: true + +jobs: + merge-gatekeeper: + runs-on: ubuntu-latest + # Restrict permissions of the GITHUB_TOKEN. + # Docs: https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs + permissions: + checks: read + statuses: read + steps: + - name: Run Merge Gatekeeper + # NOTE: v1 is updated to reflect the latest v1.x.y. Please use any tag/branch that suits your needs: + # https://github.com/upsidr/merge-gatekeeper/tags + # https://github.com/upsidr/merge-gatekeeper/branches + uses: upsidr/merge-gatekeeper@v1 + if: github.event_name == 'pull_request' + with: + token: ${{ secrets.GITHUB_TOKEN }} + timeout: 3600 + interval: 30 + ignored: "python-tests-coverage" diff --git a/.github/workflows/reserves/kernel/python-build-wheel.yml b/.github/workflows/reserves/kernel/python-build-wheel.yml new file mode 100644 index 000000000000..5752dee8ace9 --- /dev/null +++ b/.github/workflows/reserves/kernel/python-build-wheel.yml @@ -0,0 +1,45 @@ +name: python-build-wheel + +on: + workflow_dispatch: + push: + branches: [ "python_preview" ] + +permissions: + contents: read + +jobs: + build-wheel: + + runs-on: ubuntu-latest + + defaults: + run: + working-directory: python + + steps: + - uses: actions/checkout@v4 + + - run: echo "/root/.local/bin" >> $GITHUB_PATH + + - name: Install poetry + run: pipx install poetry + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.10' + cache: "poetry" + cache-dependency-path: "python/pyproject.toml" + + - name: Install Semantic Kernel + run: poetry install --no-ansi + + - name: Build wheel + run: poetry build + + - name: Upload wheel file to artifacts + uses: actions/upload-artifact@v4 + with: + name: dist + path: python/dist/* diff --git a/.github/workflows/reserves/kernel/python-integration-tests.yml b/.github/workflows/reserves/kernel/python-integration-tests.yml new file mode 100644 index 000000000000..125876de424e --- /dev/null +++ b/.github/workflows/reserves/kernel/python-integration-tests.yml @@ -0,0 +1,269 @@ +# +# This workflow will run all python integrations tests. +# + +name: Python Integration Tests + +on: + workflow_dispatch: + pull_request: + branches: ["main"] + merge_group: + branches: ["main"] + schedule: + - cron: "0 0 * * *" # Run at midnight UTC daily + +permissions: + contents: read + +jobs: + paths-filter: + runs-on: ubuntu-latest + outputs: + pythonChanges: ${{ steps.filter.outputs.python}} + steps: + - uses: actions/checkout@v4 + - uses: dorny/paths-filter@v2 + id: filter + with: + filters: | + python: + - 'python/**' + # run only if 'python' files were changed + - name: python tests + if: steps.filter.outputs.python == 'true' + run: echo "Python file" + # run only if not 'python' files were changed + - name: not python tests + if: steps.filter.outputs.python != 'true' + run: echo "NOT python file" + + python-merge-gate: + name: Python Pre-Merge Integration Tests + needs: paths-filter + if: github.event_name != 'pull_request' && github.event_name != 'schedule' && needs.paths-filter.outputs.pythonChanges == 'true' + runs-on: ${{ matrix.os }} + strategy: + max-parallel: 1 + fail-fast: false + matrix: + python-version: ["3.11"] + os: [ubuntu-latest] + steps: + - uses: actions/checkout@v4 + - name: Install poetry + run: pipx install poetry + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: "poetry" + - name: Install dependencies with hnswlib native disabled + if: matrix.os == 'macos-latest' && matrix.python-version == '3.11' + run: | + export HNSWLIB_NO_NATIVE=1 + python -m pip install --upgrade pip setuptools wheel + cd python && poetry install --with tests + - name: Install dependencies with hnswlib native enabled + if: matrix.os != 'macos-latest' || matrix.python-version != '3.11' + run: | + python -m pip install --upgrade pip setuptools wheel + cd python + poetry install --with tests + - name: Install Ollama + if: matrix.os == 'ubuntu-latest' + run: | + curl -fsSL https://ollama.com/install.sh | sh + ollama serve & + sleep 5 + - name: Pull model in Ollama + if: matrix.os == 'ubuntu-latest' + run: | + ollama pull ${{ vars.OLLAMA_MODEL }} + ollama list + - name: Run Integration Tests + id: run_tests + shell: bash + env: # Set Azure credentials secret as an input + HNSWLIB_NO_NATIVE: 1 + Python_Integration_Tests: Python_Integration_Tests + AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME }} # azure-text-embedding-ada-002 + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }} + AZURE_OPENAI_TEXT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_TEXT_DEPLOYMENT_NAME }} + AZURE_OPENAI_API_VERSION: ${{ vars.AZURE_OPENAI_API_VERSION }} + AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }} + AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }} + BING_API_KEY: ${{ secrets.BING_API_KEY }} + OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }} + OPENAI_TEXT_MODEL_ID: ${{ vars.OPENAI_TEXT_MODEL_ID }} + OPENAI_EMBEDDING_MODEL_ID: ${{ vars.OPENAI_EMBEDDING_MODEL_ID }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + PINECONE_API_KEY: ${{ secrets.PINECONE__APIKEY }} + POSTGRES_CONNECTION_STRING: ${{secrets.POSTGRES__CONNECTIONSTR}} + AZURE_AI_SEARCH_API_KEY: ${{secrets.AZURE_AI_SEARCH_API_KEY}} + AZURE_AI_SEARCH_ENDPOINT: ${{secrets.AZURE_AI_SEARCH_ENDPOINT}} + MONGODB_ATLAS_CONNECTION_STRING: ${{secrets.MONGODB_ATLAS_CONNECTION_STRING}} + AZURE_KEY_VAULT_ENDPOINT: ${{secrets.AZURE_KEY_VAULT_ENDPOINT}} + AZURE_KEY_VAULT_CLIENT_ID: ${{secrets.AZURE_KEY_VAULT_CLIENT_ID}} + AZURE_KEY_VAULT_CLIENT_SECRET: ${{secrets.AZURE_KEY_VAULT_CLIENT_SECRET}} + ACA_POOL_MANAGEMENT_ENDPOINT: ${{secrets.ACA_POOL_MANAGEMENT_ENDPOINT}} + MISTRALAI_API_KEY: ${{secrets.MISTRALAI_API_KEY}} + MISTRALAI_CHAT_MODEL_ID: ${{ vars.MISTRALAI_CHAT_MODEL_ID }} + OLLAMA_MODEL: "${{ matrix.os == 'ubuntu-latest' && vars.OLLAMA_MODEL || '' }}" # phi3 + run: | + if ${{ matrix.os == 'ubuntu-latest' }}; then + docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest + fi + + cd python + poetry run pytest ./tests/integration -v + poetry run pytest ./tests/samples -v + + python-integration-tests: + needs: paths-filter + if: (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch') && needs.paths-filter.outputs.pythonChanges == 'true' + runs-on: ${{ matrix.os }} + strategy: + max-parallel: 1 + fail-fast: false + matrix: + python-version: ["3.10", "3.11", "3.12"] + os: [ubuntu-latest, windows-latest, macos-latest] + steps: + - uses: actions/checkout@v4 + - name: Install poetry + run: pipx install poetry + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: "poetry" + - name: Install dependencies with hnswlib native disabled + if: matrix.os == 'macos-latest' && matrix.python-version == '3.11' + run: | + export HNSWLIB_NO_NATIVE=1 + python -m pip install --upgrade pip setuptools wheel + cd python && poetry install --with tests + + - name: Install dependencies with hnswlib native enabled + if: matrix.os != 'macos-latest' || matrix.python-version != '3.11' + run: | + python -m pip install --upgrade pip setuptools wheel + cd python && poetry install --with tests + + - name: Install Ollama + if: matrix.os == 'ubuntu-latest' + run: | + curl -fsSL https://ollama.com/install.sh | sh + ollama serve & + sleep 5 + + - name: Pull model in Ollama + if: matrix.os == 'ubuntu-latest' + run: | + ollama pull ${{ vars.OLLAMA_MODEL }} + ollama list + + - name: Run Integration Tests + id: run_tests + shell: bash + env: # Set Azure credentials secret as an input + HNSWLIB_NO_NATIVE: 1 + Python_Integration_Tests: Python_Integration_Tests + AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_EMBEDDING_DEPLOYMENT_NAME }} # azure-text-embedding-ada-002 + AZURE_OPENAI_CHAT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_CHAT_DEPLOYMENT_NAME }} + AZURE_OPENAI_TEXT_DEPLOYMENT_NAME: ${{ vars.AZURE_OPENAI_TEXT_DEPLOYMENT_NAME }} + AZURE_OPENAI_API_VERSION: ${{ vars.AZURE_OPENAI_API_VERSION }} + AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }} + AZURE_OPENAI_API_KEY: ${{ secrets.AZURE_OPENAI_API_KEY }} + BING_API_KEY: ${{ secrets.BING_API_KEY }} + OPENAI_CHAT_MODEL_ID: ${{ vars.OPENAI_CHAT_MODEL_ID }} + OPENAI_TEXT_MODEL_ID: ${{ vars.OPENAI_TEXT_MODEL_ID }} + OPENAI_EMBEDDING_MODEL_ID: ${{ vars.OPENAI_EMBEDDING_MODEL_ID }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + PINECONE_API_KEY: ${{ secrets.PINECONE__APIKEY }} + POSTGRES_CONNECTION_STRING: ${{secrets.POSTGRES__CONNECTIONSTR}} + AZURE_AI_SEARCH_API_KEY: ${{secrets.AZURE_AI_SEARCH_API_KEY}} + AZURE_AI_SEARCH_ENDPOINT: ${{secrets.AZURE_AI_SEARCH_ENDPOINT}} + MONGODB_ATLAS_CONNECTION_STRING: ${{secrets.MONGODB_ATLAS_CONNECTION_STRING}} + AZURE_KEY_VAULT_ENDPOINT: ${{secrets.AZURE_KEY_VAULT_ENDPOINT}} + AZURE_KEY_VAULT_CLIENT_ID: ${{secrets.AZURE_KEY_VAULT_CLIENT_ID}} + AZURE_KEY_VAULT_CLIENT_SECRET: ${{secrets.AZURE_KEY_VAULT_CLIENT_SECRET}} + ACA_POOL_MANAGEMENT_ENDPOINT: ${{secrets.ACA_POOL_MANAGEMENT_ENDPOINT}} + MISTRALAI_API_KEY: ${{secrets.MISTRALAI_API_KEY}} + MISTRALAI_CHAT_MODEL_ID: ${{ vars.MISTRALAI_CHAT_MODEL_ID }} + OLLAMA_MODEL: "${{ matrix.os == 'ubuntu-latest' && vars.OLLAMA_MODEL || '' }}" # phi3 + run: | + if ${{ matrix.os == 'ubuntu-latest' }}; then + docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest + fi + + cd python + poetry run pytest ./tests/integration -v + poetry run pytest ./tests/samples -v + + # This final job is required to satisfy the merge queue. It must only run (or succeed) if no tests failed + python-integration-tests-check: + if: always() + runs-on: ubuntu-latest + strategy: + max-parallel: 1 + fail-fast: false + needs: [python-merge-gate, python-integration-tests] + steps: + - name: Get Date + shell: bash + run: | + echo "date=$(date +'%m/%d/%Y %H:%M:%S')" >> "$GITHUB_ENV" + + - name: Run Type is Daily + if: ${{ github.event_name == 'schedule' }} + shell: bash + run: | + echo "run_type=Daily" >> "$GITHUB_ENV" + + - name: Run Type is Manual + if: ${{ github.event_name == 'workflow_dispatch' }} + shell: bash + run: | + echo "run_type=Manual" >> "$GITHUB_ENV" + + - name: Run Type is ${{ github.event_name }} + if: ${{ github.event_name != 'schedule' && github.event_name != 'workflow_dispatch'}} + shell: bash + run: | + echo "run_type=${{ github.event_name }}" >> "$GITHUB_ENV" + + - name: Fail workflow if tests failed + id: check_tests_failed + if: contains(join(needs.*.result, ','), 'failure') + uses: actions/github-script@v6 + with: + script: core.setFailed('Integration Tests Failed!') + + - name: Fail workflow if tests cancelled + id: check_tests_cancelled + if: contains(join(needs.*.result, ','), 'cancelled') + uses: actions/github-script@v6 + with: + script: core.setFailed('Integration Tests Cancelled!') + + - name: Microsoft Teams Notification + uses: skitionek/notify-microsoft-teams@master + if: github.ref == 'refs/heads/main' && github.event_name != 'pull_request' + with: + webhook_url: ${{ secrets.MSTEAMS_WEBHOOK }} + dry_run: ${{ env.run_type != 'Daily' && env.run_type != 'Manual'}} + job: ${{ toJson(job) }} + steps: ${{ toJson(steps) }} + overwrite: "{title: ` ${{ env.run_type }}: ${{ env.date }} `, text: ` ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}`}" + + - name: Microsoft Teams Notification (Dry Run) + uses: skitionek/notify-microsoft-teams@master + if: github.ref != 'refs/heads/main' + with: + webhook_url: NONE + dry_run: ${{ env.run_type != 'Daily' && env.run_type != 'Manual'}} + job: ${{ toJson(job) }} + steps: ${{ toJson(steps) }} + overwrite: "{title: ` ${{ env.run_type }}: ${{ env.date }} `, text: ` ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}`}" diff --git a/.github/workflows/reserves/kernel/python-lint.yml b/.github/workflows/reserves/kernel/python-lint.yml new file mode 100644 index 000000000000..3f20ae2f0d02 --- /dev/null +++ b/.github/workflows/reserves/kernel/python-lint.yml @@ -0,0 +1,28 @@ +name: Python Code Quality Checks +on: + workflow_dispatch: + pull_request: + branches: [ "main", "feature*" ] + paths: + - 'python/**' + +jobs: + pre-commit: + if: '!cancelled()' + strategy: + fail-fast: false + matrix: + python-version: ["3.10"] + runs-on: ubuntu-latest + continue-on-error: true + steps: + - uses: actions/checkout@v4 + - name: Install poetry + run: pipx install poetry + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: "poetry" + - name: Install dependencies + run: cd python && poetry install + - uses: pre-commit/action@v3.0.1 diff --git a/.github/workflows/reserves/kernel/python-test-coverage.yml b/.github/workflows/reserves/kernel/python-test-coverage.yml new file mode 100644 index 000000000000..d61da4f022a2 --- /dev/null +++ b/.github/workflows/reserves/kernel/python-test-coverage.yml @@ -0,0 +1,65 @@ +name: Python Test Coverage + +on: + pull_request_target: + branches: ["main", "feature*"] + paths: + - "python/**" + workflow_run: + workflows: ["Python Unit Tests"] + types: + - in_progress + +env: + PYTHON_VERSION: "3.10" + RUN_OS: ubuntu-latest + +jobs: + python-tests-coverage: + runs-on: ubuntu-latest + continue-on-error: true + permissions: + pull-requests: write + contents: read + actions: read + steps: + - name: Wait for unit tests to succeed + uses: lewagon/wait-on-check-action@v1.3.4 + with: + ref: ${{ github.event.pull_request.head.sha }} + check-name: 'Python Unit Tests (${{ env.PYTHON_VERSION }}, ${{ env.RUN_OS }}, false)' + repo-token: ${{ secrets.GH_ACTIONS_PR_WRITE }} + wait-interval: 90 + allowed-conclusions: success + - uses: actions/checkout@v4 + - name: Setup filename variables + run: echo "FILE_ID=${{ github.event.number }}-${{ env.RUN_OS }}-${{ env.PYTHON_VERSION }}" >> $GITHUB_ENV + - name: Download coverage + uses: dawidd6/action-download-artifact@v3 + with: + name: python-coverage-${{ env.FILE_ID }}.txt + github_token: ${{ secrets.GH_ACTIONS_PR_WRITE }} + workflow: python-unit-tests.yml + search_artifacts: true + if_no_artifact_found: warn + - name: Download pytest + uses: dawidd6/action-download-artifact@v3 + with: + name: pytest-${{ env.FILE_ID }}.xml + github_token: ${{ secrets.GH_ACTIONS_PR_WRITE }} + workflow: python-unit-tests.yml + search_artifacts: true + if_no_artifact_found: warn + - name: Pytest coverage comment + id: coverageComment + uses: MishaKav/pytest-coverage-comment@main + with: + github-token: ${{ secrets.GH_ACTIONS_PR_WRITE }} + pytest-coverage-path: python-coverage.txt + coverage-path-prefix: "python/" + title: "Python ${{ env.PYTHON_VERSION }} Test Coverage Report" + badge-title: "Py${{ env.PYTHON_VERSION }} Test Coverage" + junitxml-title: "Python ${{ env.PYTHON_VERSION }} Unit Test Overview" + junitxml-path: pytest.xml + default-branch: "main" + unique-id-for-comment: python-${{ env.PYTHON_VERSION }} diff --git a/.github/workflows/reserves/kernel/python-unit-tests.yml b/.github/workflows/reserves/kernel/python-unit-tests.yml new file mode 100644 index 000000000000..8e34ad0e9b5f --- /dev/null +++ b/.github/workflows/reserves/kernel/python-unit-tests.yml @@ -0,0 +1,57 @@ +name: Python Unit Tests + +on: + pull_request: + branches: ["main", "feature*"] + paths: + - "python/**" + +jobs: + python-unit-tests: + name: Python Unit Tests + runs-on: ${{ matrix.os }} + continue-on-error: ${{ matrix.experimental }} + strategy: + fail-fast: true + matrix: + python-version: ["3.10", "3.11", "3.12"] + os: [ubuntu-latest, windows-latest, macos-latest] + experimental: [false] + include: + - python-version: "3.13.0-beta.3" + os: "ubuntu-latest" + experimental: true + permissions: + contents: write + defaults: + run: + working-directory: python + steps: + - uses: actions/checkout@v4 + - name: Setup filename variables + run: echo "FILE_ID=${{ github.event.number }}-${{ matrix.os }}-${{ matrix.python-version }}" >> $GITHUB_ENV + - name: Install poetry + run: pipx install poetry + - name: Set up Python ${{ matrix.python-version }} + uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + cache: "poetry" + - name: Install dependencies + run: poetry install --with unit-tests + - name: Test with pytest + run: poetry run pytest -q --junitxml=pytest.xml --cov=semantic_kernel --cov-report=term-missing:skip-covered ./tests/unit | tee python-coverage.txt + - name: Upload coverage + uses: actions/upload-artifact@v4 + with: + name: python-coverage-${{ env.FILE_ID }}.txt + path: python/python-coverage.txt + overwrite: true + retention-days: 1 + - name: Upload pytest.xml + uses: actions/upload-artifact@v4 + with: + name: pytest-${{ env.FILE_ID }}.xml + path: python/pytest.xml + overwrite: true + retention-days: 1 diff --git a/.github/workflows/reserves/kernel/typos.yaml b/.github/workflows/reserves/kernel/typos.yaml new file mode 100644 index 000000000000..e19b7cd61fd8 --- /dev/null +++ b/.github/workflows/reserves/kernel/typos.yaml @@ -0,0 +1,29 @@ +# Check pull requests for typos. +# +# Configuration: .github/_typos.toml +# +# Info: https://github.com/marketplace/actions/typos-action +# Local install: brew install typos-cli +# Local install: conda install typos +# Local run: typos -c .github/_typos.toml + +name: Spell Check + +on: + workflow_dispatch: + pull_request: + branches: ["main", "java-development", "feature*"] + +jobs: + run: + name: Spell Check with Typos + runs-on: ubuntu-latest + steps: + - name: Check out code + uses: actions/checkout@v4 + + - name: Use custom config file + uses: crate-ci/typos@master + with: + config: .github/_typos.toml + write_changes: false diff --git a/.github/workflows/reserves/kernel/update-version.sh b/.github/workflows/reserves/kernel/update-version.sh new file mode 100755 index 000000000000..5db3622f6159 --- /dev/null +++ b/.github/workflows/reserves/kernel/update-version.sh @@ -0,0 +1,107 @@ +#!/bin/bash + +POSITIONAL_ARGS=() + +while [[ $# -gt 0 ]]; do + case $1 in + -f|--file) + file="$2" + shift # past argument + shift # past value + ;; + -p|--propsFile) + propsFile="$2" + shift # past argument + shift # past value + ;; + -b|--buildAndRevisionNumber) + buildAndRevisionNumber="$2" + shift # past argument + shift # past value + ;; + -*|--*) + echo "Unknown option $1" + exit 1 + ;; + *) + POSITIONAL_ARGS+=("$1") # save positional arg + shift # past argument + ;; + esac +done + +set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters + +if [ -z "$file" ]; then + echo "ERROR: Parameter file (-f|--file) not provided" + exit 1; +elif [ ! -f "$file" ]; then + echo "ERROR: file ${file} not found" + exit 1; +fi + +if [ -n "$(cat $file | grep -i "false")" ]; then + echo "Project is marked as NOT packable - skipping." + exit 0; +fi + +if [ -z "$propsFile" ]; then + echo "ERROR: Parameter propsFile (-f|--file) not provided" + exit 1; +elif [ ! -f "$propsFile" ]; then + echo "ERROR: propsFile ${file} not found" + exit 1; +fi + +if [ -z "$buildAndRevisionNumber" ]; then + echo "ERROR: Parameter buildAndRevisionNumber (-b|--buildAndRevisionNumber) not provided" + exit 1; +fi + +propsVersionString=$(cat $propsFile | grep -i ""); +regex="([0-9.]*)<\/Version>" +if [[ $propsVersionString =~ $regex ]]; then + propsVersion=${BASH_REMATCH[1]} +else + echo "ERROR: Version tag not found in propsFile" + exit 1; +fi + +if [ -z "$propsVersion" ]; then + echo "ERROR: Version tag not found in propsFile" + exit 1; +elif [[ ! "$propsVersion" =~ ^0.* ]]; then + echo "ERROR: Version expected to start with 0. Actual: ${propsVersion}" + exit 1; +fi + +fullVersionString="${propsVersion}.${buildAndRevisionNumber}-preview" + +if [[ ! "$fullVersionString" =~ ^0.* ]]; then + echo "ERROR: Version expected to start with 0. Actual: ${fullVersionString}" + exit 1; +fi + +echo "==== Project: ${file} ===="; +echo "propsFile = ${propsFile}" +echo "buildAndRevisionNumber = ${buildAndRevisionNumber}" +echo "version prefix from propsFile = ${propsVersion}" +echo "full version string: ${fullVersionString}" + +versionInProj=$(cat $file | grep -i ""); +if [ -n "$versionInProj" ]; then + # Version tag already exists in the csproj. Let's replace it. + echo "Updating version tag..." + content=$(cat $file | sed --expression="s/\([0-9]*.[0-9]*\)<\/Version>/$fullVersionString<\/Version>/g"); +else + # Version tag not found in the csproj. Let's add it. + echo "Project is packable - adding version tag..." + content=$(cat $file | sed --expression="s/<\/Project>/$fullVersionString<\/Version><\/PropertyGroup><\/Project>/g"); +fi + +if [ $? -ne 0 ]; then exit 1; fi +echo "$content" && echo "$content" > $file; +if [ $? -ne 0 ]; then exit 1; fi + +echo "DONE"; +echo ""; diff --git a/.github/workflows/reserves/licensed.yml b/.github/workflows/reserves/licensed.yml new file mode 100644 index 000000000000..37f1560c3fe6 --- /dev/null +++ b/.github/workflows/reserves/licensed.yml @@ -0,0 +1,15 @@ +name: Licensed + +on: + push: + branches: + - main + pull_request: + branches: + - main + workflow_dispatch: + +jobs: + call-licensed: + name: Licensed + uses: actions/reusable-workflows/.github/workflows/licensed.yml@main diff --git a/.github/workflows/reserves/llvm-project/README.md b/.github/workflows/reserves/llvm-project/README.md new file mode 100644 index 000000000000..ce34d2337e9c --- /dev/null +++ b/.github/workflows/reserves/llvm-project/README.md @@ -0,0 +1 @@ +Github action workflows should be stored in this directory. diff --git a/.github/workflows/reserves/llvm-project/build-ci-container.yml b/.github/workflows/reserves/llvm-project/build-ci-container.yml new file mode 100644 index 000000000000..28fc7de2ee06 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/build-ci-container.yml @@ -0,0 +1,112 @@ +name: Build CI Container + +permissions: + contents: read + +on: + push: + branches: + - main + paths: + - .github/workflows/build-ci-container.yml + - '.github/workflows/containers/github-action-ci/**' + pull_request: + branches: + - main + paths: + - .github/workflows/build-ci-container.yml + - '.github/workflows/containers/github-action-ci/**' + +jobs: + # TODO(boomanaiden154): Switch this back to a single stage build when we can + # run this on the self-hosted runners and don't have to do it this way to + # avoid timeouts. + build-ci-container-stage1: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + steps: + - name: Checkout LLVM + uses: actions/checkout@v4 + with: + sparse-checkout: .github/workflows/containers/github-action-ci/ + - name: Change podman Root Direcotry + run: | + mkdir -p ~/.config/containers + sudo mkdir -p /mnt/podman + sudo chown `whoami`:`whoami` /mnt/podman + cp ./.github/workflows/containers/github-action-ci/storage.conf ~/.config/containers/storage.conf + podman info + - name: Build container stage1 + working-directory: ./.github/workflows/containers/github-action-ci/ + run: | + podman build -t stage1-toolchain --target stage1-toolchain -f stage1.Dockerfile . + - name: Save container image + run: | + podman save stage1-toolchain > stage1-toolchain.tar + - name: Upload container image + uses: actions/upload-artifact@v4 + with: + name: stage1-toolchain + path: stage1-toolchain.tar + retention-days: 1 + build-ci-container-stage2: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + needs: build-ci-container-stage1 + permissions: + packages: write + steps: + - name: Write Variables + id: vars + run: | + tag=`date +%s` + container_name="ghcr.io/$GITHUB_REPOSITORY_OWNER/ci-ubuntu-22.04" + echo "container-name=$container_name" >> $GITHUB_OUTPUT + echo "container-name-tag=$container_name:$tag" >> $GITHUB_OUTPUT + + - name: Checkout LLVM + uses: actions/checkout@v4 + with: + sparse-checkout: .github/workflows/containers/github-action-ci/ + + - name: Change podman Root Direcotry + run: | + mkdir -p ~/.config/containers + sudo mkdir -p /mnt/podman + sudo chown `whoami`:`whoami` /mnt/podman + cp ./.github/workflows/containers/github-action-ci/storage.conf ~/.config/containers/storage.conf + podman info + + # Download the container image into /mnt/podman rather than + # $GITHUB_WORKSPACE to avoid space limitations on the default drive + # and use the permissions setup for /mnt/podman. + - name: Download stage1-toolchain + uses: actions/download-artifact@v4 + with: + name: stage1-toolchain + path: /mnt/podman + + - name: Load stage1-toolchain + run: | + podman load -i /mnt/podman/stage1-toolchain.tar + + - name: Build Container + working-directory: ./.github/workflows/containers/github-action-ci/ + run: | + podman build -t ${{ steps.vars.outputs.container-name-tag }} -f stage2.Dockerfile . + podman tag ${{ steps.vars.outputs.container-name-tag }} ${{ steps.vars.outputs.container-name }}:latest + + - name: Test Container + run: | + for image in ${{ steps.vars.outputs.container-name-tag }} ${{ steps.vars.outputs.container-name }}; do + podman run --rm -it $image /usr/bin/bash -x -c 'printf '\''#include \nint main(int argc, char **argv) { std::cout << "Hello\\n"; }'\'' | clang++ -x c++ - && ./a.out | grep Hello' + done + + - name: Push Container + if: github.event_name == 'push' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + podman login -u ${{ github.actor }} -p $GITHUB_TOKEN ghcr.io + podman push ${{ steps.vars.outputs.container-name-tag }} + podman push ${{ steps.vars.outputs.container-name }}:latest diff --git a/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer-run.py b/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer-run.py new file mode 100644 index 000000000000..e5f52d3b2fa6 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer-run.py @@ -0,0 +1,34 @@ +import json +import multiprocessing +import os +import re +import subprocess +import sys + + +def run_analyzer(data): + os.chdir(data["directory"]) + command = ( + data["command"] + + f" --analyze --analyzer-output html -o analyzer-results -Xclang -analyzer-config -Xclang max-nodes=75000" + ) + print(command) + subprocess.run(command, shell=True, check=True) + + +def pool_error(e): + print("Error analyzing file:", e) + + +def main(): + db_path = sys.argv[1] + database = json.load(open(db_path)) + + with multiprocessing.Pool() as pool: + pool.map_async(run_analyzer, [k for k in database], error_callback=pool_error) + pool.close() + pool.join() + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer.yml b/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer.yml new file mode 100644 index 000000000000..d614dd07b3a4 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/ci-post-commit-analyzer.yml @@ -0,0 +1,95 @@ +name: Post-Commit Static Analyzer + +permissions: + contents: read + +on: + push: + branches: + - 'release/**' + paths: + - 'clang/**' + - 'llvm/**' + - '.github/workflows/ci-post-commit-analyzer.yml' + pull_request: + types: + - opened + - synchronize + - reopened + - closed + paths: + - '.github/workflows/ci-post-commit-analyzer.yml' + - '.github/workflows/ci-post-commit-analyzer-run.py' + schedule: + - cron: '30 0 * * *' + +concurrency: + group: >- + llvm-project-${{ github.workflow }}-${{ github.event_name == 'pull_request' && + ( github.event.pull_request.number || github.ref) }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + post-commit-analyzer: + if: >- + github.repository_owner == 'llvm' && + github.event.action != 'closed' + runs-on: ubuntu-22.04 + container: + image: 'ghcr.io/llvm/ci-ubuntu-22.04:latest' + env: + LLVM_VERSION: 18 + steps: + - name: Checkout Source + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + + - name: Setup ccache + uses: hendrikmuhs/ccache-action@v1 + with: + # A full build of llvm, clang, lld, and lldb takes about 250MB + # of ccache space. There's not much reason to have more than this, + # because we usually won't need to save cache entries from older + # builds. Also, there is an overall 10GB cache limit, and each + # run creates a new cache entry so we want to ensure that we have + # enough cache space for all the tests to run at once and still + # fit under the 10 GB limit. + # Default to 2G to workaround: https://github.com/hendrikmuhs/ccache-action/issues/174 + max-size: 2G + key: post-commit-analyzer + variant: sccache + + - name: Configure + run: | + cmake -B build -S llvm -G Ninja \ + -DLLVM_ENABLE_ASSERTIONS=ON \ + -DLLVM_ENABLE_PROJECTS=clang \ + -DLLVM_BUILD_LLVM_DYLIB=ON \ + -DLLVM_LINK_LLVM_DYLIB=ON \ + -DCMAKE_CXX_COMPILER=clang++ \ + -DCMAKE_C_COMPILER=clang \ + -DCMAKE_CXX_COMPILER_LAUNCHER=sccache \ + -DCMAKE_C_COMPILER_LAUNCHER=sccache \ + -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \ + -DLLVM_INCLUDE_TESTS=OFF \ + -DCLANG_INCLUDE_TESTS=OFF \ + -DCMAKE_BUILD_TYPE=Release + + - name: Build + run: | + # FIXME: We need to build all the generated header files in order to be able to run + # the analyzer on every file. Building libLLVM and libclang is probably overkill for + # this, but it's better than building every target. + ninja -v -C build libLLVM.so libclang.so + + # Run the analyzer. + python3 .github/workflows/ci-post-commit-analyzer-run.py build/compile_commands.json + + scan-build --generate-index-only build/analyzer-results + + - name: Upload Results + uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + if: always() + with: + name: analyzer-results + path: 'build/analyzer-results/*' + diff --git a/.github/workflows/reserves/llvm-project/clang-tests.yml b/.github/workflows/reserves/llvm-project/clang-tests.yml new file mode 100644 index 000000000000..2569ce19518e --- /dev/null +++ b/.github/workflows/reserves/llvm-project/clang-tests.yml @@ -0,0 +1,38 @@ +name: Clang Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'clang/**' + - '.github/workflows/clang-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!llvm/**' + pull_request: + branches: + - 'release/**' + paths: + - 'clang/**' + - '.github/workflows/clang-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!llvm/**' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + check_clang: + if: github.repository_owner == 'llvm' + name: Test clang,lldb,libclc + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-clang + projects: clang;lldb;libclc diff --git a/.github/workflows/reserves/llvm-project/commit-access-review.py b/.github/workflows/reserves/llvm-project/commit-access-review.py new file mode 100644 index 000000000000..97589c138c0b --- /dev/null +++ b/.github/workflows/reserves/llvm-project/commit-access-review.py @@ -0,0 +1,418 @@ +#!/usr/bin/env python3 +# ===-- commit-access-review.py --------------------------------------------===# +# +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception +# +# ===------------------------------------------------------------------------===# +# +# ===------------------------------------------------------------------------===# + +import datetime +import github +import re +import requests +import time +import sys +import re + + +class User: + THRESHOLD = 5 + + def __init__(self, name, triage_list): + self.name = name + self.authored = 0 + self.merged = 0 + self.reviewed = 0 + self.triage_list = triage_list + + def add_authored(self, val=1): + self.authored += val + if self.meets_threshold(): + print(self.name, "meets the threshold with authored commits") + del self.triage_list[self.name] + + def set_authored(self, val): + self.authored = 0 + self.add_authored(val) + + def add_merged(self, val=1): + self.merged += val + if self.meets_threshold(): + print(self.name, "meets the threshold with merged commits") + del self.triage_list[self.name] + + def add_reviewed(self, val=1): + self.reviewed += val + if self.meets_threshold(): + print(self.name, "meets the threshold with reviewed commits") + del self.triage_list[self.name] + + def get_total(self): + return self.authored + self.merged + self.reviewed + + def meets_threshold(self): + return self.get_total() >= self.THRESHOLD + + def __repr__(self): + return "{} : a: {} m: {} r: {}".format( + self.name, self.authored, self.merged, self.reviewed + ) + + +def run_graphql_query( + query: str, variables: dict, token: str, retry: bool = True +) -> dict: + """ + This function submits a graphql query and returns the results as a + dictionary. + """ + s = requests.Session() + retries = requests.adapters.Retry(total=8, backoff_factor=2, status_forcelist=[504]) + s.mount("https://", requests.adapters.HTTPAdapter(max_retries=retries)) + + headers = { + "Authorization": "bearer {}".format(token), + # See + # https://github.blog/2021-11-16-graphql-global-id-migration-update/ + "X-Github-Next-Global-ID": "1", + } + request = s.post( + url="https://api.github.com/graphql", + json={"query": query, "variables": variables}, + headers=headers, + ) + + rate_limit = request.headers.get("X-RateLimit-Remaining") + print(rate_limit) + if rate_limit and int(rate_limit) < 10: + reset_time = int(request.headers["X-RateLimit-Reset"]) + while reset_time - int(time.time()) > 0: + time.sleep(60) + print( + "Waiting until rate limit reset", + reset_time - int(time.time()), + "seconds remaining", + ) + + if request.status_code == 200: + if "data" not in request.json(): + print(request.json()) + sys.exit(1) + return request.json()["data"] + elif retry: + return run_graphql_query(query, variables, token, False) + else: + raise Exception( + "Failed to run graphql query\nquery: {}\nerror: {}".format( + query, request.json() + ) + ) + + +def check_manual_requests(start_date: datetime.datetime, token: str) -> list[str]: + """ + Return a list of users who have been asked since ``start_date`` if they + want to keep their commit access. + """ + query = """ + query ($query: String!) { + search(query: $query, type: ISSUE, first: 100) { + nodes { + ... on Issue { + body + comments (first: 100) { + nodes { + author { + login + } + } + } + } + } + } + } + """ + formatted_start_date = start_date.strftime("%Y-%m-%dT%H:%M:%S") + variables = { + "query": f"type:issue created:>{formatted_start_date} org:llvm repo:llvm-project label:infrastructure:commit-access" + } + + data = run_graphql_query(query, variables, token) + users = [] + for issue in data["search"]["nodes"]: + users.extend([user[1:] for user in re.findall("@[^ ,\n]+", issue["body"])]) + + return users + + +def get_num_commits(user: str, start_date: datetime.datetime, token: str) -> int: + """ + Get number of commits that ``user`` has been made since ``start_date`. + """ + variables = { + "owner": "llvm", + "user": user, + "start_date": start_date.strftime("%Y-%m-%dT%H:%M:%S"), + } + + user_query = """ + query ($user: String!) { + user(login: $user) { + id + } + } + """ + + data = run_graphql_query(user_query, variables, token) + variables["user_id"] = data["user"]["id"] + + query = """ + query ($owner: String!, $user_id: ID!, $start_date: GitTimestamp!){ + organization(login: $owner) { + teams(query: "llvm-committers" first:1) { + nodes { + repositories { + nodes { + ref(qualifiedName: "main") { + target { + ... on Commit { + history(since: $start_date, author: {id: $user_id }) { + totalCount + } + } + } + } + } + } + } + } + } + } + """ + count = 0 + data = run_graphql_query(query, variables, token) + for repo in data["organization"]["teams"]["nodes"][0]["repositories"]["nodes"]: + count += int(repo["ref"]["target"]["history"]["totalCount"]) + if count >= User.THRESHOLD: + break + return count + + +def is_new_committer_query_repo( + user: str, start_date: datetime.datetime, token: str +) -> bool: + """ + Determine if ``user`` is a new committer. A new committer can keep their + commit access even if they don't meet the criteria. + """ + variables = { + "user": user, + } + + user_query = """ + query ($user: String!) { + user(login: $user) { + id + } + } + """ + + data = run_graphql_query(user_query, variables, token) + variables["owner"] = "llvm" + variables["user_id"] = data["user"]["id"] + variables["start_date"] = start_date.strftime("%Y-%m-%dT%H:%M:%S") + + query = """ + query ($owner: String!, $user_id: ID!){ + organization(login: $owner) { + repository(name: "llvm-project") { + ref(qualifiedName: "main") { + target { + ... on Commit { + history(author: {id: $user_id }, first: 5) { + nodes { + committedDate + } + } + } + } + } + } + } + } + """ + + data = run_graphql_query(query, variables, token) + repo = data["organization"]["repository"] + commits = repo["ref"]["target"]["history"]["nodes"] + if len(commits) == 0: + return True + committed_date = commits[-1]["committedDate"] + if datetime.datetime.strptime(committed_date, "%Y-%m-%dT%H:%M:%SZ") < start_date: + return False + return True + + +def is_new_committer(user: str, start_date: datetime.datetime, token: str) -> bool: + """ + Wrapper around is_new_commiter_query_repo to handle exceptions. + """ + try: + return is_new_committer_query_repo(user, start_date, token) + except: + pass + return True + + +def get_review_count(user: str, start_date: datetime.datetime, token: str) -> int: + """ + Return the number of reviews that ``user`` has done since ``start_date``. + """ + query = """ + query ($query: String!) { + search(query: $query, type: ISSUE, first: 5) { + issueCount + } + } + """ + formatted_start_date = start_date.strftime("%Y-%m-%dT%H:%M:%S") + variables = { + "owner": "llvm", + "repo": "llvm-project", + "user": user, + "query": f"type:pr commenter:{user} -author:{user} merged:>{formatted_start_date} org:llvm", + } + + data = run_graphql_query(query, variables, token) + return int(data["search"]["issueCount"]) + + +def count_prs(triage_list: dict, start_date: datetime.datetime, token: str): + """ + Fetch all the merged PRs for the project since ``start_date`` and update + ``triage_list`` with the number of PRs merged for each user. + """ + + query = """ + query ($query: String!, $after: String) { + search(query: $query, type: ISSUE, first: 100, after: $after) { + issueCount, + nodes { + ... on PullRequest { + author { + login + } + mergedBy { + login + } + } + } + pageInfo { + hasNextPage + endCursor + } + } + } + """ + date_begin = start_date + date_end = None + while date_begin < datetime.datetime.now(): + date_end = date_begin + datetime.timedelta(days=7) + formatted_date_begin = date_begin.strftime("%Y-%m-%dT%H:%M:%S") + formatted_date_end = date_end.strftime("%Y-%m-%dT%H:%M:%S") + variables = { + "query": f"type:pr is:merged merged:{formatted_date_begin}..{formatted_date_end} org:llvm", + } + has_next_page = True + while has_next_page: + print(variables) + data = run_graphql_query(query, variables, token) + for pr in data["search"]["nodes"]: + # Users can be None if the user has been deleted. + if not pr["author"]: + continue + author = pr["author"]["login"] + if author in triage_list: + triage_list[author].add_authored() + + if not pr["mergedBy"]: + continue + merger = pr["mergedBy"]["login"] + if author == merger: + continue + if merger not in triage_list: + continue + triage_list[merger].add_merged() + + has_next_page = data["search"]["pageInfo"]["hasNextPage"] + if has_next_page: + variables["after"] = data["search"]["pageInfo"]["endCursor"] + date_begin = date_end + + +def main(): + token = sys.argv[1] + gh = github.Github(login_or_token=token) + org = gh.get_organization("llvm") + repo = org.get_repo("llvm-project") + team = org.get_team_by_slug("llvm-committers") + one_year_ago = datetime.datetime.now() - datetime.timedelta(days=365) + triage_list = {} + for member in team.get_members(): + triage_list[member.login] = User(member.login, triage_list) + + print("Start:", len(triage_list), "triagers") + # Step 0 Check if users have requested commit access in the last year. + for user in check_manual_requests(one_year_ago, token): + if user in triage_list: + print(user, "requested commit access in the last year.") + del triage_list[user] + print("After Request Check:", len(triage_list), "triagers") + + # Step 1 count all PRs authored or merged + count_prs(triage_list, one_year_ago, token) + + print("After PRs:", len(triage_list), "triagers") + + if len(triage_list) == 0: + sys.exit(0) + + # Step 2 check for reviews + for user in list(triage_list.keys()): + review_count = get_review_count(user, one_year_ago, token) + triage_list[user].add_reviewed(review_count) + + print("After Reviews:", len(triage_list), "triagers") + + if len(triage_list) == 0: + sys.exit(0) + + # Step 3 check for number of commits + for user in list(triage_list.keys()): + num_commits = get_num_commits(user, one_year_ago, token) + # Override the total number of commits to not double count commits and + # authored PRs. + triage_list[user].set_authored(num_commits) + + print("After Commits:", len(triage_list), "triagers") + + # Step 4 check for new committers + for user in list(triage_list.keys()): + print("Checking", user) + if is_new_committer(user, one_year_ago, token): + print("Removing new committer: ", user) + del triage_list[user] + + print("Complete:", len(triage_list), "triagers") + + with open("triagers.log", "w") as triagers_log: + for user in triage_list: + print(triage_list[user].__repr__()) + triagers_log.write(user + "\n") + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/reserves/llvm-project/commit-access-review.yml b/.github/workflows/reserves/llvm-project/commit-access-review.yml new file mode 100644 index 000000000000..f9195a1863de --- /dev/null +++ b/.github/workflows/reserves/llvm-project/commit-access-review.yml @@ -0,0 +1,34 @@ +name: Commit Access Review + +on: + workflow_dispatch: + schedule: + # * is a special character in YAML so you have to quote this string + - cron: '0 7 1 * *' + +permissions: + contents: read + +jobs: + commit-access-review: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-22.04 + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + + - name: Install dependencies + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + + - name: Run Script + env: + GITHUB_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + run: | + python3 .github/workflows/commit-access-review.py $GITHUB_TOKEN + + - name: Upload Triage List + uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + with: + name: triagers + path: triagers.log diff --git a/.github/workflows/reserves/llvm-project/containers/github-action-ci/bootstrap.patch b/.github/workflows/reserves/llvm-project/containers/github-action-ci/bootstrap.patch new file mode 100644 index 000000000000..55631c54a396 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/containers/github-action-ci/bootstrap.patch @@ -0,0 +1,13 @@ +diff --git a/clang/cmake/caches/BOLT-PGO.cmake b/clang/cmake/caches/BOLT-PGO.cmake +index 1a04ca9a74e5..d092820e4115 100644 +--- a/clang/cmake/caches/BOLT-PGO.cmake ++++ b/clang/cmake/caches/BOLT-PGO.cmake +@@ -4,6 +4,8 @@ set(CLANG_BOOTSTRAP_TARGETS + stage2-clang-bolt + stage2-distribution + stage2-install-distribution ++ clang ++ lld + CACHE STRING "") + set(BOOTSTRAP_CLANG_BOOTSTRAP_TARGETS + clang-bolt diff --git a/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage1.Dockerfile b/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage1.Dockerfile new file mode 100644 index 000000000000..73828cc05736 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage1.Dockerfile @@ -0,0 +1,42 @@ +FROM docker.io/library/ubuntu:22.04 as base +ENV LLVM_SYSROOT=/opt/llvm + +FROM base as stage1-toolchain +ENV LLVM_VERSION=18.1.8 + +RUN apt-get update && \ + apt-get install -y \ + wget \ + gcc \ + g++ \ + cmake \ + ninja-build \ + python3 \ + git \ + curl + +RUN curl -O -L https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-$LLVM_VERSION.tar.gz && tar -xf llvmorg-$LLVM_VERSION.tar.gz + +WORKDIR /llvm-project-llvmorg-$LLVM_VERSION + +COPY bootstrap.patch / + +# TODO(boomanaiden154): Remove the bootstrap patch once we unsplit the build +# and no longer need to explicitly build the stage2 dependencies. +RUN cat /bootstrap.patch | patch -p1 + +RUN mkdir build + +RUN cmake -B ./build -G Ninja ./llvm \ + -C ./clang/cmake/caches/BOLT-PGO.cmake \ + -DBOOTSTRAP_LLVM_ENABLE_LLD=ON \ + -DBOOTSTRAP_BOOTSTRAP_LLVM_ENABLE_LLD=ON \ + -DPGO_INSTRUMENT_LTO=Thin \ + -DLLVM_ENABLE_RUNTIMES="compiler-rt" \ + -DCMAKE_INSTALL_PREFIX="$LLVM_SYSROOT" \ + -DLLVM_ENABLE_PROJECTS="bolt;clang;lld;clang-tools-extra" \ + -DLLVM_DISTRIBUTION_COMPONENTS="lld;compiler-rt;clang-format;scan-build" \ + -DCLANG_DEFAULT_LINKER="lld" \ + -DBOOTSTRAP_CLANG_PGO_TRAINING_DATA_SOURCE_DIR=/llvm-project-llvmorg-$LLVM_VERSION/llvm + +RUN ninja -C ./build stage2-instrumented-clang stage2-instrumented-lld diff --git a/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage2.Dockerfile b/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage2.Dockerfile new file mode 100644 index 000000000000..0ca0da87734c --- /dev/null +++ b/.github/workflows/reserves/llvm-project/containers/github-action-ci/stage2.Dockerfile @@ -0,0 +1,29 @@ +FROM docker.io/library/ubuntu:22.04 as base +ENV LLVM_SYSROOT=/opt/llvm + +FROM stage1-toolchain AS stage2-toolchain + +RUN ninja -C ./build stage2-clang-bolt stage2-install-distribution && ninja -C ./build install-distribution && rm -rf ./build + +FROM base + +COPY --from=stage2-toolchain $LLVM_SYSROOT $LLVM_SYSROOT + +# Need to install curl for hendrikmuhs/ccache-action +# Need nodejs for some of the GitHub actions. +# Need perl-modules for clang analyzer tests. +# Need git for SPIRV-Tools tests. +RUN apt-get update && \ + apt-get install -y \ + binutils \ + cmake \ + curl \ + git \ + libstdc++-11-dev \ + ninja-build \ + nodejs \ + perl-modules \ + python3-psutil + +ENV LLVM_SYSROOT=$LLVM_SYSROOT +ENV PATH=${LLVM_SYSROOT}/bin:${PATH} diff --git a/.github/workflows/reserves/llvm-project/containers/github-action-ci/storage.conf b/.github/workflows/reserves/llvm-project/containers/github-action-ci/storage.conf new file mode 100644 index 000000000000..60f295ff1e96 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/containers/github-action-ci/storage.conf @@ -0,0 +1,4 @@ +[storage] + driver = "overlay" + runroot = "/mnt/podman/container" + graphroot = "/mnt/podman/image" diff --git a/.github/workflows/reserves/llvm-project/docs.yml b/.github/workflows/reserves/llvm-project/docs.yml new file mode 100644 index 000000000000..800e92915735 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/docs.yml @@ -0,0 +1,169 @@ +# LLVM Documentation CI +# Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions. +# See https://llvm.org/LICENSE.txt for license information. +# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception + +name: "Test documentation build" + +permissions: + contents: read + +on: + push: + branches: + - 'main' + paths: + - 'llvm/docs/**' + - 'clang/docs/**' + - 'clang/include/clang/Basic/AttrDocs.td' + - 'clang/include/clang/Driver/ClangOptionDocs.td' + - 'clang/include/clang/Basic/DiagnosticDocs.td' + - 'clang-tools-extra/docs/**' + - 'lldb/docs/**' + - 'libunwind/docs/**' + - 'libcxx/docs/**' + - 'libc/docs/**' + - 'lld/docs/**' + - 'openmp/docs/**' + - 'polly/docs/**' + - 'flang/docs/**' + - 'flang/include/flang/Optimizer/Dialect/FIROps.td' + - '.github/workflows/docs.yml' + pull_request: + paths: + - 'llvm/docs/**' + - 'clang/docs/**' + - 'clang/include/clang/Basic/AttrDocs.td' + - 'clang/include/clang/Driver/ClangOptionDocs.td' + - 'clang/include/clang/Basic/DiagnosticDocs.td' + - 'clang-tools-extra/docs/**' + - 'lldb/docs/**' + - 'libunwind/docs/**' + - 'libcxx/docs/**' + - 'libc/docs/**' + - 'lld/docs/**' + - 'openmp/docs/**' + - 'polly/docs/**' + - 'flang/docs/**' + - 'flang/include/flang/Optimizer/Dialect/FIROps.td' + - '.github/workflows/docs.yml' + +jobs: + check-docs-build: + name: "Test documentation build" + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + # Don't fetch before checking for file changes to force the file changes + # action to use the Github API in pull requests. If it's a push to a + # branch we can't use the Github API to get the diff, so we need to have + # a local checkout beforehand. + - name: Fetch LLVM sources (Push) + if: ${{ github.event_name == 'push' }} + uses: actions/checkout@v4 + with: + fetch-depth: 1 + - name: Get subprojects that have doc changes + id: docs-changed-subprojects + uses: tj-actions/changed-files@v39 + with: + files_yaml: | + llvm: + - 'llvm/docs/**' + clang: + - 'clang/docs/**' + - 'clang/include/clang/Basic/AttrDocs.td' + - 'clang/include/clang/Driver/ClangOptionDocs.td' + - 'clang/include/clang/Basic/DiagnosticDocs.td' + clang-tools-extra: + - 'clang-tools-extra/docs/**' + lldb: + - 'lldb/docs/**' + libunwind: + - 'libunwind/docs/**' + libcxx: + - 'libcxx/docs/**' + libc: + - 'libc/docs/**' + lld: + - 'lld/docs/**' + openmp: + - 'openmp/docs/**' + polly: + - 'polly/docs/**' + flang: + - 'flang/docs/**' + - 'flang/include/flang/Optimizer/Dialect/FIROps.td' + - name: Fetch LLVM sources (PR) + if: ${{ github.event_name == 'pull_request' }} + uses: actions/checkout@v4 + with: + fetch-depth: 1 + - name: Setup Python env + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: 'pip' + cache-dependency-path: 'llvm/docs/requirements.txt' + - name: Install python dependencies + run: pip install -r llvm/docs/requirements.txt + - name: Install system dependencies + run: | + sudo apt-get update + # swig and graphviz are lldb specific dependencies + sudo apt-get install -y cmake ninja-build swig graphviz + - name: Build LLVM docs + if: steps.docs-changed-subprojects.outputs.llvm_any_changed == 'true' + run: | + cmake -B llvm-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C llvm-build docs-llvm-html docs-llvm-man + - name: Build Clang docs + if: steps.docs-changed-subprojects.outputs.clang_any_changed == 'true' + run: | + cmake -B clang-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C clang-build docs-clang-html docs-clang-man + - name: Build clang-tools-extra docs + if: steps.docs-changed-subprojects.outputs.clang-tools-extra_any_changed == 'true' + run: | + cmake -B clang-tools-extra-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C clang-tools-extra-build docs-clang-tools-html docs-clang-tools-man + - name: Build LLDB docs + if: steps.docs-changed-subprojects.outputs.lldb_any_changed == 'true' + run: | + cmake -B lldb-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;lldb" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C lldb-build docs-lldb-html docs-lldb-man + - name: Build libunwind docs + if: steps.docs-changed-subprojects.outputs.libunwind_any_changed == 'true' + run: | + cmake -B libunwind-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_RUNTIMES="libunwind" -DLLVM_ENABLE_SPHINX=ON ./runtimes + TZ=UTC ninja -C libunwind-build docs-libunwind-html + - name: Build libcxx docs + if: steps.docs-changed-subprojects.outputs.libcxx_any_changed == 'true' + run: | + cmake -B libcxx-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_RUNTIMES="libcxxabi;libcxx;libunwind" -DLLVM_ENABLE_SPHINX=ON ./runtimes + TZ=UTC ninja -C libcxx-build docs-libcxx-html + - name: Build libc docs + if: steps.docs-changed-subprojects.outputs.libc_any_changed == 'true' + run: | + cmake -B libc-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_RUNTIMES="libc" -DLLVM_ENABLE_SPHINX=ON ./runtimes + TZ=UTC ninja -C libc-build docs-libc-html + - name: Build LLD docs + if: steps.docs-changed-subprojects.outputs.lld_any_changed == 'true' + run: | + cmake -B lld-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="lld" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C lld-build docs-lld-html + - name: Build OpenMP docs + if: steps.docs-changed-subprojects.outputs.openmp_any_changed == 'true' + run: | + cmake -B openmp-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C openmp-build docs-openmp-html + - name: Build Polly docs + if: steps.docs-changed-subprojects.outputs.polly_any_changed == 'true' + run: | + cmake -B polly-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="polly" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C polly-build docs-polly-html docs-polly-man + - name: Build Flang docs + if: steps.docs-changed-subprojects.outputs.flang_any_changed == 'true' + run: | + cmake -B flang-build -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;mlir;flang" -DLLVM_ENABLE_SPHINX=ON ./llvm + TZ=UTC ninja -C flang-build docs-flang-html diff --git a/.github/workflows/reserves/llvm-project/email-check.yaml b/.github/workflows/reserves/llvm-project/email-check.yaml new file mode 100644 index 000000000000..8f32d020975f --- /dev/null +++ b/.github/workflows/reserves/llvm-project/email-check.yaml @@ -0,0 +1,46 @@ +name: "Check for private emails used in PRs" + +on: + pull_request: + types: + - opened + +permissions: + contents: read + +jobs: + validate_email: + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + with: + ref: ${{ github.event.pull_request.head.sha }} + + - name: Extract author email + id: author + run: | + git log -1 + echo "EMAIL=$(git show -s --format='%ae' HEAD~0)" >> $GITHUB_OUTPUT + # Create empty comment file + echo "[]" > comments + + - name: Validate author email + if: ${{ endsWith(steps.author.outputs.EMAIL, 'noreply.github.com') }} + env: + COMMENT: >- + ⚠️ We detected that you are using a GitHub private e-mail address to contribute to the repo.
+ Please turn off [Keep my email addresses private](https://github.com/settings/emails) setting in your account.
+ See [LLVM Discourse](https://discourse.llvm.org/t/hidden-emails-on-github-should-we-do-something-about-it) for more information. + run: | + cat << EOF > comments + [{"body" : "$COMMENT"}] + EOF + + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + if: always() + with: + name: workflow-args + path: | + comments diff --git a/.github/workflows/reserves/llvm-project/get-llvm-version/action.yml b/.github/workflows/reserves/llvm-project/get-llvm-version/action.yml new file mode 100644 index 000000000000..2218d926fc13 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/get-llvm-version/action.yml @@ -0,0 +1,26 @@ +name: Get LLVM Version +description: >- + Get the LLVM version from the llvm-project source tree. This action assumes + the llvm-project sources have already been checked out into GITHUB_WORKSPACE. + +outputs: + major: + description: LLVM major version + value: ${{ steps.version.outputs.major }} + minor: + description: LLVM minor version + value: ${{ steps.version.outputs.minor }} + patch: + description: LLVM patch version + value: ${{ steps.version.outputs.patch }} + +runs: + using: "composite" + steps: + - name: Get Version + shell: bash + id: version + run: | + for v in major minor patch; do + echo "$v=`llvm/utils/release/get-llvm-version.sh --$v`" >> $GITHUB_OUTPUT + done diff --git a/.github/workflows/reserves/llvm-project/issue-release-workflow.yml b/.github/workflows/reserves/llvm-project/issue-release-workflow.yml new file mode 100644 index 000000000000..5027d4f3ea6f --- /dev/null +++ b/.github/workflows/reserves/llvm-project/issue-release-workflow.yml @@ -0,0 +1,69 @@ +# This contains the workflow definitions that allow users to test backports +# to the release branch using comments on issues. +# +# /cherry-pick <...> +# +# This comment will attempt to cherry-pick the given commits to the latest +# release branch (release/Y.x) and if successful, push the result to a branch +# on github. +# +# /branch // +# +# This comment will create a pull request from to the latest release +# branch. + +name: Issue Release Workflow + +permissions: + contents: read + +on: + issue_comment: + types: + - created + - edited + issues: + types: + - opened + +env: + COMMENT_BODY: ${{ github.event.action == 'opened' && github.event.issue.body || github.event.comment.body }} + +jobs: + backport-commits: + name: Backport Commits + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + if: >- + (github.repository == 'llvm/llvm-project') && + !startswith(github.event.comment.body, '') && + contains(github.event.action == 'opened' && github.event.issue.body || github.event.comment.body, '/cherry-pick') + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + with: + repository: llvm/llvm-project + # GitHub stores the token used for checkout and uses it for pushes + # too, but we want to use a different token for pushing, so we need + # to disable persist-credentials here. + persist-credentials: false + fetch-depth: 0 + + - name: Setup Environment + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + ./llvm/utils/git/github-automation.py --token ${{ github.token }} setup-llvmbot-git + + - name: Backport Commits + run: | + printf "%s" "$COMMENT_BODY" | + ./llvm/utils/git/github-automation.py \ + --repo "$GITHUB_REPOSITORY" \ + --token "${{ secrets.RELEASE_WORKFLOW_PR_CREATE }}" \ + release-workflow \ + --branch-repo-token ${{ secrets.RELEASE_WORKFLOW_PUSH_SECRET }} \ + --issue-number ${{ github.event.issue.number }} \ + --requested-by ${{ (github.event.action == 'opened' && github.event.issue.user.login) || github.event.comment.user.login }} \ + auto diff --git a/.github/workflows/reserves/llvm-project/issue-subscriber.yml b/.github/workflows/reserves/llvm-project/issue-subscriber.yml new file mode 100644 index 000000000000..ef4fdf441819 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/issue-subscriber.yml @@ -0,0 +1,37 @@ +name: Issue Subscriber + +on: + issues: + types: + - labeled + +permissions: + contents: read + +jobs: + auto-subscribe: + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + - name: Checkout Automation Script + uses: actions/checkout@v4 + with: + sparse-checkout: llvm/utils/git/ + ref: main + + - name: Setup Automation Script + working-directory: ./llvm/utils/git/ + run: | + pip install --require-hashes -r requirements.txt + + - name: Update watchers + working-directory: ./llvm/utils/git/ + # https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-an-intermediate-environment-variable + env: + LABEL_NAME: ${{ github.event.label.name }} + run: | + python3 ./github-automation.py \ + --token '${{ secrets.ISSUE_SUBSCRIBER_TOKEN }}' \ + issue-subscriber \ + --issue-number '${{ github.event.issue.number }}' \ + --label-name "$LABEL_NAME" diff --git a/.github/workflows/reserves/llvm-project/issue-write.yml b/.github/workflows/reserves/llvm-project/issue-write.yml new file mode 100644 index 000000000000..5334157a7fd2 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/issue-write.yml @@ -0,0 +1,157 @@ +name: Comment on an issue + +on: + workflow_run: + workflows: + - "Check code formatting" + - "Check for private emails used in PRs" + - "PR Request Release Note" + types: + - completed + +permissions: + contents: read + +jobs: + pr-comment: + runs-on: ubuntu-latest + permissions: + pull-requests: write + if: > + github.event.workflow_run.event == 'pull_request' && + ( + github.event.workflow_run.conclusion == 'success' || + github.event.workflow_run.conclusion == 'failure' + ) + steps: + - name: Fetch Sources + uses: actions/checkout@v4 + with: + sparse-checkout: | + .github/workflows/unprivileged-download-artifact/action.yml + sparse-checkout-cone-mode: false + - name: 'Download artifact' + uses: ./.github/workflows/unprivileged-download-artifact + id: download-artifact + with: + run-id: ${{ github.event.workflow_run.id }} + artifact-name: workflow-args + + - name: 'Comment on PR' + if: steps.download-artifact.outputs.artifact-id != '' + uses: actions/github-script@v3 + with: + github-token: ${{ secrets.GITHUB_TOKEN }} + script: | + var fs = require('fs'); + const comments = JSON.parse(fs.readFileSync('./comments')); + if (!comments || comments.length == 0) { + return; + } + + let runInfo = await github.actions.getWorkflowRun({ + owner: context.repo.owner, + repo: context.repo.repo, + run_id: context.payload.workflow_run.id + }); + + console.log(runInfo); + + + // Query to find the number of the pull request that triggered this job. + // The associated pull requests are based off of the branch name, so if + // you create a pull request for a branch, close it, and then create + // another pull request with the same branch, then this query will return + // two associated pull requests. This is why we have to fetch all the + // associated pull requests and then iterate through them to find the + // one that is open. + const gql_query = ` + query($repo_owner : String!, $repo_name : String!, $branch: String!) { + repository(owner: $repo_owner, name: $repo_name) { + ref (qualifiedName: $branch) { + associatedPullRequests(first: 100) { + nodes { + baseRepository { + owner { + login + } + } + number + state + } + } + } + } + } + ` + const gql_variables = { + repo_owner: runInfo.data.head_repository.owner.login, + repo_name: runInfo.data.head_repository.name, + branch: runInfo.data.head_branch + } + const gql_result = await github.graphql(gql_query, gql_variables); + console.log(gql_result); + // If the branch for the PR was deleted before this job has a chance + // to run, then the ref will be null. This can happen if someone: + // 1. Rebase the PR, which triggers some workflow. + // 2. Immediately merges the PR and deletes the branch. + // 3. The workflow finishes and triggers this job. + if (!gql_result.repository.ref) { + console.log("Ref has been deleted"); + return; + } + console.log(gql_result.repository.ref.associatedPullRequests.nodes); + + var pr_number = 0; + gql_result.repository.ref.associatedPullRequests.nodes.forEach((pr) => { + + // The largest PR number is the one we care about. The only way + // to have more than one associated pull requests is if all the + // old pull requests are in the closed state. + if (pr.baseRepository.owner.login = context.repo.owner && pr.number > pr_number) { + pr_number = pr.number; + } + }); + if (pr_number == 0) { + console.log("Error retrieving pull request number"); + return; + } + + await comments.forEach(function (comment) { + if (comment.id) { + // Security check: Ensure that this comment was created by + // the github-actions bot, so a malicious input won't overwrite + // a user's comment. + github.issues.getComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: comment.id + }).then((old_comment) => { + console.log(old_comment); + if (old_comment.data.user.login != "github-actions[bot]") { + console.log("Invalid comment id: " + comment.id); + return; + } + github.issues.updateComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: pr_number, + comment_id: comment.id, + body: comment.body + }); + }); + } else { + github.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: pr_number, + body: comment.body + }); + } + }); + + - name: Dump comments file + if: >- + always() && + steps.download-artifact.outputs.artifact-id != '' + run: cat comments diff --git a/.github/workflows/reserves/llvm-project/libclang-abi-tests.yml b/.github/workflows/reserves/llvm-project/libclang-abi-tests.yml new file mode 100644 index 000000000000..9e839ff49e28 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libclang-abi-tests.yml @@ -0,0 +1,168 @@ +name: libclang ABI Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'clang/**' + - '.github/workflows/libclang-abi-tests.yml' + pull_request: + branches: + - 'release/**' + paths: + - 'clang/**' + - '.github/workflows/libclang-abi-tests.yml' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + abi-dump-setup: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + outputs: + BASELINE_REF: ${{ steps.vars.outputs.BASELINE_REF }} + ABI_HEADERS: ${{ steps.vars.outputs.ABI_HEADERS }} + ABI_LIBS: ${{ steps.vars.outputs.ABI_LIBS }} + BASELINE_VERSION_MAJOR: ${{ steps.vars.outputs.BASELINE_VERSION_MAJOR }} + LLVM_VERSION_MAJOR: ${{ steps.version.outputs.major }} + LLVM_VERSION_MINOR: ${{ steps.version.outputs.minor }} + LLVM_VERSION_PATCH: ${{ steps.version.outputs.patch }} + steps: + - name: Checkout source + uses: actions/checkout@v4 + with: + fetch-depth: 250 + + - name: Get LLVM version + id: version + uses: ./.github/workflows/get-llvm-version + + - name: Setup Variables + id: vars + run: | + remote_repo='https://github.com/llvm/llvm-project' + if [ ${{ steps.version.outputs.patch }} -eq 0 ]; then + major_version=$(( ${{ steps.version.outputs.major }} - 1)) + baseline_ref="llvmorg-$major_version.1.0" + + # If there is a minor release, we want to use that as the base line. + minor_ref=$(git ls-remote --refs -t "$remote_repo" llvmorg-"$major_version".[1-9].[0-9] | tail -n1 | grep -o 'llvmorg-.\+' || true) + if [ -n "$minor_ref" ]; then + baseline_ref="$minor_ref" + else + # Check if we have a release candidate + rc_ref=$(git ls-remote --refs -t "$remote_repo" llvmorg-"$major_version".[1-9].[0-9]-rc* | tail -n1 | grep -o 'llvmorg-.\+' || true) + if [ -n "$rc_ref" ]; then + baseline_ref="$rc_ref" + fi + fi + { + echo "BASELINE_VERSION_MAJOR=$major_version" + echo "BASELINE_REF=$baseline_ref" + echo "ABI_HEADERS=clang-c" + echo "ABI_LIBS=libclang.so" + } >> "$GITHUB_OUTPUT" + else + { + echo "BASELINE_VERSION_MAJOR=${{ steps.version.outputs.major }}" + echo "BASELINE_REF=llvmorg-${{ steps.version.outputs.major }}.1.0" + echo "ABI_HEADERS=." + echo "ABI_LIBS=libclang.so libclang-cpp.so" + } >> "$GITHUB_OUTPUT" + fi + + abi-dump: + if: github.repository_owner == 'llvm' + needs: abi-dump-setup + runs-on: ubuntu-latest + strategy: + matrix: + name: + - build-baseline + - build-latest + include: + - name: build-baseline + llvm_version_major: ${{ needs.abi-dump-setup.outputs.BASELINE_VERSION_MAJOR }} + ref: ${{ needs.abi-dump-setup.outputs.BASELINE_REF }} + repo: llvm/llvm-project + - name: build-latest + llvm_version_major: ${{ needs.abi-dump-setup.outputs.LLVM_VERSION_MAJOR }} + ref: ${{ github.sha }} + repo: ${{ github.repository }} + steps: + - name: Install Ninja + uses: llvm/actions/install-ninja@main + - name: Install abi-compliance-checker + run: | + sudo apt-get install abi-dumper autoconf pkg-config + - name: Install universal-ctags + run: | + git clone https://github.com/universal-ctags/ctags.git + cd ctags + ./autogen.sh + ./configure + sudo make install + - name: Download source code + uses: llvm/actions/get-llvm-project-src@main + with: + ref: ${{ matrix.ref }} + repo: ${{ matrix.repo }} + - name: Configure + run: | + mkdir install + cmake -B build -S llvm -G Ninja -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE=Debug -DLLVM_TARGETS_TO_BUILD="" -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_C_FLAGS_DEBUG="-g1 -Og" -DCMAKE_CXX_FLAGS_DEBUG="-g1 -Og" -DCMAKE_INSTALL_PREFIX="$(pwd)"/install llvm + - name: Build + run: ninja -C build/ ${{ needs.abi-dump-setup.outputs.ABI_LIBS }} install-clang-headers + - name: Dump ABI + run: | + parallel abi-dumper -lver ${{ matrix.ref }} -skip-cxx -public-headers ./install/include/${{ needs.abi-dump-setup.outputs.ABI_HEADERS }} -o {}-${{ matrix.ref }}.abi ./build/lib/{} ::: ${{ needs.abi-dump-setup.outputs.ABI_LIBS }} + for lib in ${{ needs.abi-dump-setup.outputs.ABI_LIBS }}; do + # Remove symbol versioning from dumps, so we can compare across major versions. + sed -i 's/LLVM_[0-9]\+/LLVM_NOVERSION/' $lib-${{ matrix.ref }}.abi + done + - name: Upload ABI file + uses: actions/upload-artifact@v3 + with: + name: ${{ matrix.name }} + path: '*${{ matrix.ref }}.abi' + + abi-compare: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + needs: + - abi-dump-setup + - abi-dump + steps: + - name: Download baseline + uses: actions/download-artifact@v3 + with: + name: build-baseline + path: build-baseline + - name: Download latest + uses: actions/download-artifact@v3 + with: + name: build-latest + path: build-latest + + - name: Install abi-compliance-checker + run: sudo apt-get install abi-compliance-checker + - name: Compare ABI + run: | + for lib in ${{ needs.abi-dump-setup.outputs.ABI_LIBS }}; do + abi-compliance-checker -lib $lib -old build-baseline/$lib*.abi -new build-latest/$lib*.abi + done + - name: Upload ABI Comparison + if: always() + uses: actions/upload-artifact@v3 + with: + name: compat-report-${{ github.sha }} + path: compat_reports/ diff --git a/.github/workflows/reserves/llvm-project/libclang-python-tests.yml b/.github/workflows/reserves/llvm-project/libclang-python-tests.yml new file mode 100644 index 000000000000..801a70172478 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libclang-python-tests.yml @@ -0,0 +1,41 @@ +name: Libclang Python Binding Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'main' + paths: + - 'clang/bindings/python/**' + - 'clang/tools/libclang/**' + - 'clang/CMakeList.txt' + - '.github/workflows/libclang-python-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + pull_request: + paths: + - 'clang/bindings/python/**' + - 'clang/tools/libclang/**' + - 'clang/CMakeList.txt' + - '.github/workflows/libclang-python-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + +jobs: + check-clang-python: + # Build libclang and then run the libclang Python binding's unit tests. + name: Build and run Python unit tests + if: github.repository == 'llvm/llvm-project' + strategy: + fail-fast: false + matrix: + python-version: ["3.8", "3.11"] + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-clang-python + projects: clang + # There is an issue running on "windows-2019". + # See https://github.com/llvm/llvm-project/issues/76601#issuecomment-1873049082. + os_list: '["ubuntu-latest"]' + python_version: ${{ matrix.python-version }} diff --git a/.github/workflows/reserves/llvm-project/libclc-tests.yml b/.github/workflows/reserves/llvm-project/libclc-tests.yml new file mode 100644 index 000000000000..23192f776a98 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libclc-tests.yml @@ -0,0 +1,39 @@ +name: libclc Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'libclc/**' + - '.github/workflows/libclc-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!clang/**' + - '!llvm/**' + pull_request: + branches: + - 'release/**' + paths: + - 'libclc/**' + - '.github/workflows/libclc-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!clang/**' + - '!llvm/**' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + check_libclc: + if: github.repository_owner == 'llvm' + name: Test libclc + uses: ./.github/workflows/llvm-project-tests.yml + with: + projects: clang;libclc diff --git a/.github/workflows/reserves/llvm-project/libcxx-build-and-test.yaml b/.github/workflows/reserves/llvm-project/libcxx-build-and-test.yaml new file mode 100644 index 000000000000..b5e60781e000 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libcxx-build-and-test.yaml @@ -0,0 +1,278 @@ +# This file defines pre-commit CI for libc++, libc++abi, and libunwind (on Github). +# +# We split the configurations in multiple stages with the intent of saving compute time +# when a job fails early in the pipeline. This is why the jobs are marked as `continue-on-error: false`. +# We try to run the CI configurations with the most signal in the first stage. +# +# Stages 1 & 2 are meant to be "smoke tests", and are meant to catch most build/test failures quickly and without using +# too many resources. +# Stage 3 is "everything else", and is meant to catch breakages on more niche or unique configurations. +# +# Therefore, we "fail-fast" for any failures during stages 1 & 2, meaning any job failing cancels all other running jobs, +# under the assumption that if the "smoke tests" fail, then the other configurations will likely fail in the same way. +# However, stage 3 does not fail fast, as it's more likely that any one job failing is a flake or a configuration-specific +# +name: Build and Test libc++ +on: + pull_request: + paths: + - 'libcxx/**' + - 'libcxxabi/**' + - 'libunwind/**' + - 'runtimes/**' + - 'cmake/**' + - '.github/workflows/libcxx-build-and-test.yaml' + schedule: + # Run nightly at 08:00 UTC (aka 00:00 Pacific, aka 03:00 Eastern) + - cron: '0 8 * * *' + +permissions: + contents: read # Default everything to read-only + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + + +env: + # LLVM POST-BRANCH bump version + # LLVM POST-BRANCH add compiler test for ToT - 1, e.g. "Clang 17" + # LLVM RELEASE bump remove compiler ToT - 3, e.g. "Clang 15" + LLVM_HEAD_VERSION: "19" # Used compiler, update POST-BRANCH. + LLVM_PREVIOUS_VERSION: "18" + LLVM_OLDEST_VERSION: "17" + GCC_STABLE_VERSION: "13" + LLVM_SYMBOLIZER_PATH: "/usr/bin/llvm-symbolizer-19" + CLANG_CRASH_DIAGNOSTICS_DIR: "crash_diagnostics" + + +jobs: + stage1: + if: github.repository_owner == 'llvm' + runs-on: libcxx-runners-8-set + continue-on-error: false + strategy: + fail-fast: false + matrix: + config: [ + 'generic-cxx03', + 'generic-cxx26', + 'generic-modules' + ] + cc: [ 'clang-19' ] + cxx: [ 'clang++-19' ] + include: + - config: 'generic-gcc' + cc: 'gcc-14' + cxx: 'g++-14' + steps: + - uses: actions/checkout@v4 + - name: ${{ matrix.config }}.${{ matrix.cxx }} + run: libcxx/utils/ci/run-buildbot ${{ matrix.config }} + env: + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0 + if: always() + with: + name: ${{ matrix.config }}-${{ matrix.cxx }}-results + path: | + **/test-results.xml + **/*.abilist + **/CMakeError.log + **/CMakeOutput.log + **/crash_diagnostics/* + stage2: + if: github.repository_owner == 'llvm' + runs-on: libcxx-runners-8-set + needs: [ stage1 ] + continue-on-error: false + strategy: + fail-fast: false + matrix: + config: [ + 'generic-cxx11', + 'generic-cxx14', + 'generic-cxx17', + 'generic-cxx20', + 'generic-cxx23' + ] + cc: [ 'clang-19' ] + cxx: [ 'clang++-19' ] + include: + - config: 'generic-gcc-cxx11' + cc: 'gcc-14' + cxx: 'g++-14' + - config: 'generic-cxx23' + cc: 'clang-17' + cxx: 'clang++-17' + - config: 'generic-cxx26' + cc: 'clang-18' + cxx: 'clang++-18' + steps: + - uses: actions/checkout@v4 + - name: ${{ matrix.config }} + run: libcxx/utils/ci/run-buildbot ${{ matrix.config }} + env: + CC: ${{ matrix.cc }} + CXX: ${{ matrix.cxx }} + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0 + if: always() # Upload artifacts even if the build or test suite fails + with: + name: ${{ matrix.config }}-${{ matrix.cxx }}-results + path: | + **/test-results.xml + **/*.abilist + **/CMakeError.log + **/CMakeOutput.log + **/crash_diagnostics/* + stage3: + if: github.repository_owner == 'llvm' + needs: [ stage1, stage2 ] + continue-on-error: false + strategy: + fail-fast: false + max-parallel: 8 + matrix: + config: [ + 'generic-abi-unstable', + 'generic-hardening-mode-debug', + 'generic-hardening-mode-extensive', + 'generic-hardening-mode-fast', + 'generic-hardening-mode-fast-with-abi-breaks', + 'generic-merged', + 'generic-modules-lsv', + 'generic-no-exceptions', + 'generic-no-experimental', + 'generic-no-filesystem', + 'generic-no-localization', + 'generic-no-terminal', + 'generic-no-random_device', + 'generic-no-threads', + 'generic-no-tzdb', + 'generic-no-unicode', + 'generic-no-wide-characters', + 'generic-no-rtti', + 'generic-optimized-speed', + 'generic-static', + # TODO Find a better place for the benchmark and bootstrapping builds to live. They're either very expensive + # or don't provide much value since the benchmark run results are too noise on the bots. + 'benchmarks', + 'bootstrapping-build' + ] + machine: [ 'libcxx-runners-8-set' ] + include: + - config: 'generic-cxx26' + machine: libcxx-runners-8-set + - config: 'generic-asan' + machine: libcxx-runners-8-set + - config: 'generic-tsan' + machine: libcxx-runners-8-set + - config: 'generic-ubsan' + machine: libcxx-runners-8-set + # Use a larger machine for MSAN to avoid timeout and memory allocation issues. + - config: 'generic-msan' + machine: libcxx-runners-8-set + runs-on: ${{ matrix.machine }} + steps: + - uses: actions/checkout@v4 + - name: ${{ matrix.config }} + run: libcxx/utils/ci/run-buildbot ${{ matrix.config }} + env: + CC: clang-19 + CXX: clang++-19 + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0 + if: always() + with: + name: ${{ matrix.config }}-results + path: | + **/test-results.xml + **/*.abilist + **/CMakeError.log + **/CMakeOutput.log + **/crash_diagnostics/* + + macos: + runs-on: macos-14 + needs: [ stage1 ] + strategy: + fail-fast: true + matrix: + config: [ + generic-cxx03, + generic-cxx23, + generic-modules, + apple-configuration + ] + steps: + - uses: actions/checkout@v4 + - uses: maxim-lobanov/setup-xcode@v1 + with: + xcode-version: 'latest' + - uses: seanmiddleditch/gha-setup-ninja@master + - name: Build and test + run: | + python3 -m venv .venv + source .venv/bin/activate + python -m pip install psutil + bash libcxx/utils/ci/run-buildbot ${{ matrix.config }} + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0 + if: always() # Upload artifacts even if the build or test suite fails + with: + name: macos-${{ matrix.config }}-results + path: | + **/test-results.xml + **/*.abilist + **/CMakeError.log + **/CMakeOutput.log + **/crash_diagnostics/* + + windows: + runs-on: windows-2022 + needs: [ stage1 ] + strategy: + fail-fast: false + matrix: + include: + - { config: clang-cl-dll, mingw: false } + - { config: clang-cl-static, mingw: false } + - { config: clang-cl-no-vcruntime, mingw: false } + - { config: clang-cl-debug, mingw: false } + - { config: clang-cl-static-crt, mingw: false } + - { config: mingw-dll, mingw: true } + - { config: mingw-static, mingw: true } + - { config: mingw-dll-i686, mingw: true } + - { config: mingw-incomplete-sysroot, mingw: true } + steps: + - uses: actions/checkout@v4 + - name: Install dependencies + run: | + choco install -y ninja + pip install psutil + - name: Install a current LLVM + if: ${{ matrix.mingw != true }} + run: | + choco install -y llvm --version=18.1.6 --allow-downgrade + - name: Install llvm-mingw + if: ${{ matrix.mingw == true }} + run: | + curl -LO https://github.com/mstorsjo/llvm-mingw/releases/download/20240606/llvm-mingw-20240606-ucrt-x86_64.zip + powershell Expand-Archive llvm-mingw*.zip -DestinationPath . + del llvm-mingw*.zip + mv llvm-mingw* c:\llvm-mingw + echo "c:\llvm-mingw\bin" | Out-File -FilePath $Env:GITHUB_PATH -Encoding utf8 -Append + - name: Simulate a from-scratch build of llvm-mingw + if: ${{ matrix.config == 'mingw-incomplete-sysroot' }} + run: | + rm -r c:\llvm-mingw\include\c++ + rm -r c:\llvm-mingw\*-w64-mingw32\lib\libc++* + rm -r c:\llvm-mingw\*-w64-mingw32\lib\libunwind* + - name: Add Git Bash to the path + run: | + echo "c:\Program Files\Git\usr\bin" | Out-File -FilePath $Env:GITHUB_PATH -Encoding utf8 -Append + - name: Set up the MSVC dev environment + if: ${{ matrix.mingw != true }} + uses: ilammy/msvc-dev-cmd@v1 + - name: Build and test + run: | + bash libcxx/utils/ci/run-buildbot ${{ matrix.config }} diff --git a/.github/workflows/reserves/llvm-project/libcxx-check-generated-files.yml b/.github/workflows/reserves/llvm-project/libcxx-check-generated-files.yml new file mode 100644 index 000000000000..570055624b2a --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libcxx-check-generated-files.yml @@ -0,0 +1,24 @@ +name: "Check libc++ generated files" +on: + pull_request: + paths: + - 'libcxx/**' + +permissions: + contents: read + +jobs: + check_generated_files: + runs-on: ubuntu-latest + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + + - name: Install dependencies + uses: aminya/setup-cpp@v1 + with: + clangformat: 17.0.1 + ninja: true + + - name: Check generated files + run: libcxx/utils/ci/run-buildbot check-generated-output diff --git a/.github/workflows/reserves/llvm-project/libcxx-restart-preempted-jobs.yaml b/.github/workflows/reserves/llvm-project/libcxx-restart-preempted-jobs.yaml new file mode 100644 index 000000000000..21879ce19c27 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/libcxx-restart-preempted-jobs.yaml @@ -0,0 +1,132 @@ +name: Restart Preempted Libc++ Workflow + +# The libc++ builders run on preemptable VMs, which can be shutdown at any time. +# This workflow identifies when a workflow run was canceled due to the VM being preempted, +# and restarts the workflow run. + +# We identify a canceled workflow run by checking the annotations of the check runs in the check suite, +# which should contain the message "The runner has received a shutdown signal." + +# Note: If a job is both preempted and also contains a non-preemption failure, we do not restart the workflow. + +on: + workflow_run: + workflows: [Build and Test libc\+\+] + types: + - completed + +permissions: + contents: read + +jobs: + restart: + if: github.repository_owner == 'llvm' && (github.event.workflow_run.conclusion == 'failure' || github.event.workflow_run.conclusion == 'cancelled') + name: "Restart Job" + permissions: + statuses: read + checks: write + actions: write + runs-on: ubuntu-latest + steps: + - name: "Restart Job" + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea #v7.0.1 + with: + script: | + const failure_regex = /Process completed with exit code 1./ + const preemption_regex = /The runner has received a shutdown signal/ + + const wf_run = context.payload.workflow_run + core.notice(`Running on "${wf_run.display_title}" by @${wf_run.actor.login} (event: ${wf_run.event})\nWorkflow run URL: ${wf_run.html_url}`) + + + async function create_check_run(conclusion, message) { + // Create a check run on the given workflow run to indicate if + // we are restarting the workflow or not. + if (conclusion != 'success' && conclusion != 'skipped' && conclusion != 'neutral') { + core.setFailed('Invalid conclusion: ' + conclusion) + } + await github.rest.checks.create({ + owner: context.repo.owner, + repo: context.repo.repo, + name: 'Restart Preempted Job', + head_sha: wf_run.head_sha, + status: 'completed', + conclusion: conclusion, + output: { + title: 'Restarted Preempted Job', + summary: message + } + }) + } + + console.log('Listing check runs for suite') + const check_suites = await github.rest.checks.listForSuite({ + owner: context.repo.owner, + repo: context.repo.repo, + check_suite_id: context.payload.workflow_run.check_suite_id, + per_page: 100 // FIXME: We don't have 100 check runs yet, but we should handle this better. + }) + + check_run_ids = []; + for (check_run of check_suites.data.check_runs) { + console.log('Checking check run: ' + check_run.id); + if (check_run.status != 'completed') { + console.log('Check run was not completed. Skipping.'); + continue; + } + if (check_run.conclusion != 'failure' && check_run.conclusion != 'cancelled') { + console.log('Check run had conclusion: ' + check_run.conclusion + '. Skipping.'); + continue; + } + check_run_ids.push(check_run.id); + } + + has_preempted_job = false; + + for (check_run_id of check_run_ids) { + console.log('Listing annotations for check run: ' + check_run_id); + + annotations = await github.rest.checks.listAnnotations({ + owner: context.repo.owner, + repo: context.repo.repo, + check_run_id: check_run_id + }) + + for (annotation of annotations.data) { + if (annotation.annotation_level != 'failure') { + continue; + } + + const preemption_match = annotation.message.match(preemption_regex); + + if (preemption_match != null) { + console.log('Found preemption message: ' + annotation.message); + has_preempted_job = true; + } + + const failure_match = annotation.message.match(failure_regex); + if (failure_match != null) { + // We only want to restart the workflow if all of the failures were due to preemption. + // We don't want to restart the workflow if there were other failures. + core.notice('Choosing not to rerun workflow because we found a non-preemption failure' + + 'Failure message: "' + annotation.message + '"'); + await create_check_run('skipped', 'Choosing not to rerun workflow because we found a non-preemption failure\n' + + 'Failure message: ' + annotation.message) + return; + } + } + } + + if (!has_preempted_job) { + core.notice('No preempted jobs found. Not restarting workflow.'); + await create_check_run('neutral', 'No preempted jobs found. Not restarting workflow.') + return; + } + + core.notice("Restarted workflow: " + context.payload.workflow_run.id); + await github.rest.actions.reRunWorkflowFailedJobs({ + owner: context.repo.owner, + repo: context.repo.repo, + run_id: context.payload.workflow_run.id + }) + await create_check_run('success', 'Restarted workflow run due to preempted job') diff --git a/.github/workflows/reserves/llvm-project/lld-tests.yml b/.github/workflows/reserves/llvm-project/lld-tests.yml new file mode 100644 index 000000000000..599c0975fa68 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/lld-tests.yml @@ -0,0 +1,38 @@ +name: LLD Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'lld/**' + - '.github/workflows/lld-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!llvm/**' + pull_request: + branches: + - 'release/**' + paths: + - 'lld/**' + - '.github/workflows/lld-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!llvm/**' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + check_lld: + if: github.repository_owner == 'llvm' + name: Test lld + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-lld + projects: lld diff --git a/.github/workflows/reserves/llvm-project/lldb-tests.yml b/.github/workflows/reserves/llvm-project/lldb-tests.yml new file mode 100644 index 000000000000..6bb972195625 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/lldb-tests.yml @@ -0,0 +1,39 @@ +name: lldb Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'lldb/**' + - '.github/workflows/lldb-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!clang/**' + - '!llvm/**' + pull_request: + branches: + - 'release/**' + paths: + - 'lldb/**' + - '.github/workflows/lldb-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + - '!clang/**' + - '!llvm/**' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + build_lldb: + if: github.repository_owner == 'llvm' + name: Build lldb + uses: ./.github/workflows/llvm-project-tests.yml + with: + projects: clang;lldb diff --git a/.github/workflows/reserves/llvm-project/llvm-bugs.yml b/.github/workflows/reserves/llvm-project/llvm-bugs.yml new file mode 100644 index 000000000000..c392078fa452 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/llvm-bugs.yml @@ -0,0 +1,63 @@ +name: LLVM Bugs notifier + +permissions: + contents: read + issues: read + +on: + issues: + types: + - opened + +jobs: + auto-subscribe: + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + - uses: actions/setup-node@v4 + with: + node-version: 18 + check-latest: true + - run: npm install mailgun.js form-data + - name: Send notification + uses: actions/github-script@v6 + env: + MAILGUN_API_KEY: ${{ secrets.LLVM_BUGS_KEY }} + with: + script: | + const Mailgun = require('mailgun.js'); + const formData = require('form-data'); + + const mailgun = new Mailgun(formData); + const DOMAIN = 'email.llvm.org'; + + const mg = mailgun.client({ username: 'api', key: process.env.MAILGUN_API_KEY }); + + github.rest.issues.get({ + issue_number: context.issue.number, + owner: context.repo.owner, + repo: context.repo.repo + }) + .then((issue) => { + const payload = { + author : issue.data.user.login, + issue : issue.data.number, + title : issue.data.title, + url : issue.data.html_url, + labels : issue.data.labels.map((label) => label.name), + assignee : issue.data.assignees.map((assignee) => assignee.login), + body : issue.data.body + }; + + const data = { + from: 'LLVM Bugs ', + to: 'llvm-bugs@lists.llvm.org', + subject: `[Bug ${issue.data.number}] ${issue.data.title}`, + template: 'new-github-issue', + 'o:tracking-clicks': 'no', + 'h:X-Mailgun-Variables': JSON.stringify(payload) + }; + + return mg.messages.create(DOMAIN, data); + }) + .then((msg) => console.log(msg)); diff --git a/.github/workflows/reserves/llvm-project/llvm-project-tests.yml b/.github/workflows/reserves/llvm-project/llvm-project-tests.yml new file mode 100644 index 000000000000..95a3890c0d2d --- /dev/null +++ b/.github/workflows/reserves/llvm-project/llvm-project-tests.yml @@ -0,0 +1,148 @@ +name: LLVM Project Tests + +permissions: + contents: read + +on: + workflow_dispatch: + inputs: + build_target: + required: false + projects: + required: false + extra_cmake_args: + required: false + os_list: + required: false + default: '["ubuntu-latest", "windows-2019", "macOS-13"]' + python_version: + required: false + type: string + default: '3.11' + workflow_call: + inputs: + build_target: + required: false + type: string + default: "all" + + projects: + required: true + type: string + + extra_cmake_args: + required: false + type: string + + os_list: + required: false + type: string + # Use windows-2019 due to: + # https://developercommunity.visualstudio.com/t/Prev-Issue---with-__assume-isnan-/1597317 + default: '["ubuntu-latest", "windows-2019", "macOS-13"]' + + python_version: + required: false + type: string + default: '3.11' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + # If the group name here is the same as the group name in the workflow that includes + # this one, then the action will try to wait on itself and get stuck. + group: llvm-project-${{ github.workflow }}-${{ inputs.projects }}-${{ inputs.python_version }}${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + lit-tests: + name: Lit Tests + runs-on: ${{ matrix.os }} + container: + image: ${{(startsWith(matrix.os, 'ubuntu') && 'ghcr.io/llvm/ci-ubuntu-22.04:latest') || null}} + volumes: + - /mnt/:/mnt/ + strategy: + fail-fast: false + matrix: + os: ${{ fromJSON(inputs.os_list) }} + steps: + - name: Setup Windows + if: startsWith(matrix.os, 'windows') + uses: llvm/actions/setup-windows@main + with: + arch: amd64 + # On Windows, starting with win19/20220814.1, cmake choose the 32-bit + # python3.10.6 libraries instead of the 64-bit libraries when building + # lldb. Using this setup-python action to make 3.10 the default + # python fixes this. + - name: Setup Python + uses: actions/setup-python@v5 + with: + python-version: ${{ inputs.python_version }} + - name: Install Ninja + if: runner.os != 'Linux' + uses: llvm/actions/install-ninja@main + # actions/checkout deletes any existing files in the new git directory, + # so this needs to either run before ccache-action or it has to use + # clean: false. + - uses: actions/checkout@v4 + with: + fetch-depth: 250 + - name: Setup ccache + uses: hendrikmuhs/ccache-action@v1 + with: + # A full build of llvm, clang, lld, and lldb takes about 250MB + # of ccache space. There's not much reason to have more than this, + # because we usually won't need to save cache entries from older + # builds. Also, there is an overall 10GB cache limit, and each + # run creates a new cache entry so we want to ensure that we have + # enough cache space for all the tests to run at once and still + # fit under the 10 GB limit. + # Default to 2G to workaround: https://github.com/hendrikmuhs/ccache-action/issues/174 + max-size: 2G + key: ${{ matrix.os }} + variant: sccache + - name: Build and Test + env: + # Workaround for https://github.com/actions/virtual-environments/issues/5900. + # This should be a no-op for non-mac OSes + PKG_CONFIG_PATH: /usr/local/Homebrew/Library/Homebrew/os/mac/pkgconfig//12 + shell: bash + id: build-llvm + run: | + if [ "${{ runner.os }}" == "Linux" ]; then + builddir="/mnt/build/" + mkdir -p $builddir + extra_cmake_args="-DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang" + else + builddir="$(pwd)"/build + fi + if [ "${{ runner.os }}" == "macOS" ]; then + # Workaround test failure on some lld tests on MacOS + # https://github.com/llvm/llvm-project/issues/81967 + extra_cmake_args="-DLLVM_DISABLE_ASSEMBLY_FILES=ON" + fi + echo "llvm-builddir=$builddir" >> "$GITHUB_OUTPUT" + cmake -G Ninja \ + -B "$builddir" \ + -S llvm \ + -DLLVM_ENABLE_PROJECTS="${{ inputs.projects }}" \ + -DCMAKE_BUILD_TYPE=Release \ + -DLLVM_ENABLE_ASSERTIONS=ON \ + -DLLDB_INCLUDE_TESTS=OFF \ + -DLIBCLC_TARGETS_TO_BUILD="amdgcn--;amdgcn--amdhsa;r600--;nvptx--;nvptx64--;nvptx--nvidiacl;nvptx64--nvidiacl" \ + -DCMAKE_C_COMPILER_LAUNCHER=sccache \ + -DCMAKE_CXX_COMPILER_LAUNCHER=sccache \ + $extra_cmake_args \ + ${{ inputs.extra_cmake_args }} + ninja -C "$builddir" '${{ inputs.build_target }}' + + - name: Build and Test libclc + if: "!startsWith(matrix.os, 'windows') && contains(inputs.projects, 'libclc')" + env: + LLVM_BUILDDIR: ${{ steps.build-llvm.outputs.llvm-builddir }} + run: | + # The libclc tests don't have a generated check target so all we can + # do is build it. + ninja -C "$LLVM_BUILDDIR" diff --git a/.github/workflows/reserves/llvm-project/llvm-project-workflow-tests.yml b/.github/workflows/reserves/llvm-project/llvm-project-workflow-tests.yml new file mode 100644 index 000000000000..a2539b279be0 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/llvm-project-workflow-tests.yml @@ -0,0 +1,32 @@ +# This workflow will test the llvm-project-tests workflow in PRs +# targetting the main branch. Since this workflow doesn't normally +# run on main PRs, we need some way to test it to ensure new updates +# don't break it. + +name: LLVM Workflow Test + +permissions: + contents: read + +on: + pull_request: + branches: + - 'main' + paths: + - '.github/workflows/llvm-project-tests.yml' + - '.github/workflows/llvm-project-workflow-tests.yml' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + llvm-test: + if: github.repository_owner == 'llvm' + name: Build and Test + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-all + projects: clang;lld;libclc;lldb diff --git a/.github/workflows/reserves/llvm-project/llvm-tests.yml b/.github/workflows/reserves/llvm-project/llvm-tests.yml new file mode 100644 index 000000000000..26e644229aaa --- /dev/null +++ b/.github/workflows/reserves/llvm-project/llvm-tests.yml @@ -0,0 +1,192 @@ +name: LLVM Tests + +permissions: + contents: read + +on: + workflow_dispatch: + push: + branches: + - 'release/**' + paths: + - 'llvm/**' + - '.github/workflows/llvm-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + pull_request: + branches: + - 'release/**' + paths: + - 'llvm/**' + - '.github/workflows/llvm-tests.yml' + - '.github/workflows/llvm-project-tests.yml' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + check-all: + if: github.repository_owner == 'llvm' + name: Build and Test + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-all + projects: clang;lld;libclc;lldb + + abi-dump-setup: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + outputs: + BASELINE_REF: ${{ steps.vars.outputs.BASELINE_REF }} + ABI_HEADERS: ${{ steps.vars.outputs.ABI_HEADERS }} + BASELINE_VERSION_MAJOR: ${{ steps.vars.outputs.BASELINE_VERSION_MAJOR }} + BASELINE_VERSION_MINOR: ${{ steps.vars.outputs.BASELINE_VERSION_MINOR }} + LLVM_VERSION_MAJOR: ${{ steps.version.outputs.major }} + LLVM_VERSION_MINOR: ${{ steps.version.outputs.minor }} + LLVM_VERSION_PATCH: ${{ steps.version.outputs.patch }} + steps: + - name: Checkout source + uses: actions/checkout@v4 + with: + fetch-depth: 250 + + - name: Get LLVM version + id: version + uses: ./.github/workflows/get-llvm-version + + - name: Setup Variables + id: vars + run: | + # C++ ABI: + # 18.1.0 we aren't doing ABI checks. + # 18.1.1 We want to check 18.1.0. + # C ABI: + # 18.1.0 We want to check 17.0.x + # 18.1.1 We want to check 18.1.0 + echo "BASELINE_VERSION_MINOR=1" >> "$GITHUB_OUTPUT" + if [ ${{ steps.version.outputs.patch }} -eq 0 ]; then + { + echo "BASELINE_VERSION_MAJOR=$(( ${{ steps.version.outputs.major }} - 1))" + echo "ABI_HEADERS=llvm-c" + } >> "$GITHUB_OUTPUT" + else + { + echo "BASELINE_VERSION_MAJOR=${{ steps.version.outputs.major }}" + echo "ABI_HEADERS=." + } >> "$GITHUB_OUTPUT" + fi + + abi-dump: + if: github.repository_owner == 'llvm' + needs: abi-dump-setup + runs-on: ubuntu-latest + strategy: + matrix: + name: + - build-baseline + - build-latest + include: + - name: build-baseline + llvm_version_major: ${{ needs.abi-dump-setup.outputs.BASELINE_VERSION_MAJOR }} + ref: llvmorg-${{ needs.abi-dump-setup.outputs.BASELINE_VERSION_MAJOR }}.${{ needs.abi-dump-setup.outputs.BASELINE_VERSION_MINOR }}.0 + repo: llvm/llvm-project + - name: build-latest + llvm_version_major: ${{ needs.abi-dump-setup.outputs.LLVM_VERSION_MAJOR }} + ref: ${{ github.sha }} + repo: ${{ github.repository }} + steps: + - name: Install Ninja + uses: llvm/actions/install-ninja@main + - name: Install abi-compliance-checker + run: | + sudo apt-get install abi-dumper autoconf pkg-config + - name: Install universal-ctags + run: | + git clone https://github.com/universal-ctags/ctags.git + cd ctags + ./autogen.sh + ./configure + sudo make install + - name: Download source code + uses: llvm/actions/get-llvm-project-src@main + with: + ref: ${{ matrix.ref }} + repo: ${{ matrix.repo }} + - name: Configure + run: | + mkdir install + cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Debug -DLLVM_TARGETS_TO_BUILD="" -DLLVM_BUILD_LLVM_DYLIB=ON -DCMAKE_C_FLAGS_DEBUG="-g1 -Og" -DCMAKE_CXX_FLAGS_DEBUG="-g1 -Og" -DCMAKE_INSTALL_PREFIX="$(pwd)"/install llvm + - name: Build + # Need to run install-LLVM twice to ensure the symlink is installed (this is a bug). + run: | + ninja -C build install-LLVM + ninja -C build install-LLVM + ninja -C build install-llvm-headers + - name: Dump ABI + run: | + if [ "${{ needs.abi-dump-setup.outputs.ABI_HEADERS }}" = "llvm-c" ]; then + nm ./install/lib/libLLVM.so | awk "/T _LLVM/ || /T LLVM/ { print $3 }" | sort -u | sed -e "s/^_//g" | cut -d ' ' -f 3 > llvm.symbols + # Even though the -symbols-list option doesn't seem to filter out the symbols, I believe it speeds up processing, so I'm leaving it in. + export EXTRA_ARGS="-symbols-list llvm.symbols" + else + touch llvm.symbols + fi + abi-dumper $EXTRA_ARGS -lver ${{ matrix.ref }} -skip-cxx -public-headers ./install/include/${{ needs.abi-dump-setup.outputs.ABI_HEADERS }} -o ${{ matrix.ref }}.abi ./install/lib/libLLVM.so + # Remove symbol versioning from dumps, so we can compare across major versions. + sed -i 's/LLVM_${{ matrix.llvm_version_major }}/LLVM_NOVERSION/' ${{ matrix.ref }}.abi + - name: Upload ABI file + uses: actions/upload-artifact@v3 + with: + name: ${{ matrix.name }} + path: ${{ matrix.ref }}.abi + + - name: Upload symbol list file + if: matrix.name == 'build-baseline' + uses: actions/upload-artifact@v3 + with: + name: symbol-list + path: llvm.symbols + + abi-compare: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + needs: + - abi-dump-setup + - abi-dump + steps: + - name: Download baseline + uses: actions/download-artifact@v3 + with: + name: build-baseline + path: build-baseline + - name: Download latest + uses: actions/download-artifact@v3 + with: + name: build-latest + path: build-latest + - name: Download symbol list + uses: actions/download-artifact@v3 + with: + name: symbol-list + path: symbol-list + + - name: Install abi-compliance-checker + run: sudo apt-get install abi-compliance-checker + - name: Compare ABI + run: | + if [ -s symbol-list/llvm.symbols ]; then + # This option doesn't seem to work with the ABI dumper, so passing it here. + export EXTRA_ARGS="-symbols-list symbol-list/llvm.symbols" + fi + # FIXME: Reading of gzip'd abi files on the GitHub runners stop + # working some time in March of 2021, likely due to a change in the + # runner's environment. + abi-compliance-checker $EXTRA_ARGS -l libLLVM.so -old build-baseline/*.abi -new build-latest/*.abi || test "${{ needs.abi-dump-setup.outputs.ABI_HEADERS }}" = "llvm-c" + - name: Upload ABI Comparison + if: always() + uses: actions/upload-artifact@v3 + with: + name: compat-report-${{ github.sha }} + path: compat_reports/ diff --git a/.github/workflows/reserves/llvm-project/merged-prs.yml b/.github/workflows/reserves/llvm-project/merged-prs.yml new file mode 100644 index 000000000000..e29afd4097f9 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/merged-prs.yml @@ -0,0 +1,41 @@ +name: "Add buildbot information to first PRs from new contributors" + +permissions: + contents: read + +on: + # It's safe to use pull_request_target here, because we aren't checking out + # code from the pull request branch. + # See https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ + pull_request_target: + types: + - closed + +jobs: + buildbot_comment: + runs-on: ubuntu-latest + permissions: + pull-requests: write + if: >- + (github.repository == 'llvm/llvm-project') && + (github.event.pull_request.merged == true) + steps: + - name: Checkout Automation Script + uses: actions/checkout@v4 + with: + sparse-checkout: llvm/utils/git/ + ref: main + + - name: Setup Automation Script + working-directory: ./llvm/utils/git/ + run: | + pip install --require-hashes -r requirements.txt + + - name: Add Buildbot information comment + working-directory: ./llvm/utils/git/ + run: | + python3 ./github-automation.py \ + --token '${{ secrets.GITHUB_TOKEN }}' \ + pr-buildbot-information \ + --issue-number "${{ github.event.pull_request.number }}" \ + --author "${{ github.event.pull_request.user.login }}" diff --git a/.github/workflows/reserves/llvm-project/new-issues.yml b/.github/workflows/reserves/llvm-project/new-issues.yml new file mode 100644 index 000000000000..ed15fdb9fba6 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/new-issues.yml @@ -0,0 +1,23 @@ +name: Labeling new issues +on: + issues: + types: ['opened'] + +permissions: + contents: read + +jobs: + automate-issues-labels: + permissions: + issues: write + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + - uses: llvm/actions/issue-labeler@main + with: + repo-token: ${{ secrets.GITHUB_TOKEN }} + configuration-path: .github/new-issues-labeler.yml + include-title: 1 + include-body: 0 + sync-labels: 0 + enable-versioned-regex: 0 diff --git a/.github/workflows/reserves/llvm-project/new-prs.yml b/.github/workflows/reserves/llvm-project/new-prs.yml new file mode 100644 index 000000000000..88175d6f8d64 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/new-prs.yml @@ -0,0 +1,75 @@ +name: "Labelling new pull requests" + +permissions: + contents: read + +on: + # It's safe to use pull_request_target here, because we aren't checking out + # code from the pull request branch. + # See https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ + pull_request_target: + types: + - opened + - reopened + - ready_for_review + - synchronize + +jobs: + greeter: + runs-on: ubuntu-latest + permissions: + pull-requests: write + # Only comment on PRs that have been opened for the first time, by someone + # new to LLVM or to GitHub as a whole. Ideally we'd look for FIRST_TIMER + # or FIRST_TIME_CONTRIBUTOR, but this does not appear to work. Instead check + # that we do not have any of the other author associations. + # See https://docs.github.com/en/webhooks/webhook-events-and-payloads?actionType=opened#pull_request + # for all the possible values. + if: >- + (github.repository == 'llvm/llvm-project') && + (github.event.action == 'opened') && + (github.event.pull_request.author_association != 'COLLABORATOR') && + (github.event.pull_request.author_association != 'CONTRIBUTOR') && + (github.event.pull_request.author_association != 'MANNEQUIN') && + (github.event.pull_request.author_association != 'MEMBER') && + (github.event.pull_request.author_association != 'OWNER') + steps: + - name: Checkout Automation Script + uses: actions/checkout@v4 + with: + sparse-checkout: llvm/utils/git/ + ref: main + + - name: Setup Automation Script + working-directory: ./llvm/utils/git/ + run: | + pip install --require-hashes -r requirements.txt + + - name: Greet Author + working-directory: ./llvm/utils/git/ + run: | + python3 ./github-automation.py \ + --token '${{ secrets.GITHUB_TOKEN }}' \ + pr-greeter \ + --issue-number "${{ github.event.pull_request.number }}" + + automate-prs-labels: + # Greet first so that only the author gets that notification. + needs: greeter + runs-on: ubuntu-latest + # Ignore PRs with more than 10 commits. Pull requests with a lot of + # commits tend to be accidents usually when someone made a mistake while trying + # to rebase. We want to ignore these pull requests to avoid excessive + # notifications. + # always() means that even if greeter is skipped, this job will run. + if: > + always() && github.repository == 'llvm/llvm-project' && + github.event.pull_request.draft == false && + github.event.pull_request.commits < 10 + steps: + - uses: actions/labeler@v4 + with: + configuration-path: .github/new-prs-labeler.yml + # workaround for https://github.com/actions/labeler/issues/112 + sync-labels: '' + repo-token: ${{ secrets.ISSUE_SUBSCRIBER_TOKEN }} diff --git a/.github/workflows/reserves/llvm-project/pr-code-format.yml b/.github/workflows/reserves/llvm-project/pr-code-format.yml new file mode 100644 index 000000000000..f2bb37316d3a --- /dev/null +++ b/.github/workflows/reserves/llvm-project/pr-code-format.yml @@ -0,0 +1,101 @@ +name: "Check code formatting" + +permissions: + contents: read + +on: + pull_request: + branches: + - main + - 'users/**' + +jobs: + code_formatter: + runs-on: ubuntu-latest + timeout-minutes: 30 + concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number }} + cancel-in-progress: true + if: github.repository == 'llvm/llvm-project' + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + with: + ref: ${{ github.event.pull_request.head.sha }} + + - name: Checkout through merge base + uses: rmacklin/fetch-through-merge-base@v0 + with: + base_ref: ${{ github.event.pull_request.base.ref }} + head_ref: ${{ github.event.pull_request.head.sha }} + deepen_length: 500 + + - name: Get changed files + id: changed-files + uses: tj-actions/changed-files@v39 + with: + separator: "," + skip_initial_fetch: true + + # We need to pull the script from the main branch, so that we ensure + # we get the latest version of this script. + - name: Fetch code formatting utils + uses: actions/checkout@v4 + with: + repository: ${{ github.repository }} + ref: ${{ github.base_ref }} + sparse-checkout: | + llvm/utils/git/requirements_formatting.txt + llvm/utils/git/code-format-helper.py + sparse-checkout-cone-mode: false + path: code-format-tools + + - name: "Listed files" + env: + CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }} + run: | + echo "Formatting files:" + echo "$CHANGED_FILES" + + - name: Install clang-format + uses: aminya/setup-cpp@v1 + with: + clangformat: 18.1.7 + + - name: Setup Python env + uses: actions/setup-python@v5 + with: + python-version: '3.11' + cache: 'pip' + cache-dependency-path: 'code-format-tools/llvm/utils/git/requirements_formatting.txt' + + - name: Install python dependencies + run: pip install -r code-format-tools/llvm/utils/git/requirements_formatting.txt + + - name: Run code formatter + env: + GITHUB_PR_NUMBER: ${{ github.event.pull_request.number }} + START_REV: ${{ github.event.pull_request.base.sha }} + END_REV: ${{ github.event.pull_request.head.sha }} + CHANGED_FILES: ${{ steps.changed-files.outputs.all_changed_files }} + # TODO(boomanaiden154): Once clang v18 is released, we should be able + # to take advantage of the new --diff_from_common_commit option + # explicitly in code-format-helper.py and not have to diff starting at + # the merge base. + # Create an empty comments file so the pr-write job doesn't fail. + run: | + echo "[]" > comments && + python ./code-format-tools/llvm/utils/git/code-format-helper.py \ + --write-comment-to-file \ + --token ${{ secrets.GITHUB_TOKEN }} \ + --issue-number $GITHUB_PR_NUMBER \ + --start-rev $(git merge-base $START_REV $END_REV) \ + --end-rev $END_REV \ + --changed-files "$CHANGED_FILES" + + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + if: always() + with: + name: workflow-args + path: | + comments diff --git a/.github/workflows/reserves/llvm-project/pr-request-release-note.yml b/.github/workflows/reserves/llvm-project/pr-request-release-note.yml new file mode 100644 index 000000000000..2fa501dda16b --- /dev/null +++ b/.github/workflows/reserves/llvm-project/pr-request-release-note.yml @@ -0,0 +1,49 @@ +name: PR Request Release Note + +permissions: + contents: read + +on: + pull_request: + types: + - closed + +jobs: + request-release-note: + if: >- + github.repository_owner == 'llvm' && + startsWith(github.ref, 'refs/heads/release') + + runs-on: ubuntu-latest + steps: + # We need to pull the script from the main branch, so that we ensure + # we get the latest version of this script. + - name: Checkout Scripts + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + sparse-checkout: | + llvm/utils/git/requirements.txt + llvm/utils/git/github-automation.py + sparse-checkout-cone-mode: false + + - name: Install Dependencies + run: | + pip install --require-hashes -r llvm/utils/git/requirements.txt + + - name: Request Release Note + env: + # We need to use an llvmbot token here, because we are mentioning a user. + GITHUB_TOKEN: ${{ github.token }} + run: | + python3 llvm/utils/git/github-automation.py \ + --repo "$GITHUB_REPOSITORY" \ + --token "$GITHUB_TOKEN" \ + request-release-note \ + --pr-number ${{ github.event.pull_request.number}} + + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + if: always() + with: + name: workflow-args + path: | + comments diff --git a/.github/workflows/reserves/llvm-project/pr-subscriber.yml b/.github/workflows/reserves/llvm-project/pr-subscriber.yml new file mode 100644 index 000000000000..272d3e2f9ef8 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/pr-subscriber.yml @@ -0,0 +1,34 @@ +name: PR Subscriber + +on: + pull_request_target: + types: + - labeled + +permissions: + contents: read + +jobs: + auto-subscribe: + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + steps: + - name: Checkout Automation Script + uses: actions/checkout@v4 + with: + sparse-checkout: llvm/utils/git/ + ref: main + + - name: Setup Automation Script + working-directory: ./llvm/utils/git/ + run: | + pip install --require-hashes -r requirements.txt + + - name: Update watchers + working-directory: ./llvm/utils/git/ + run: | + python3 ./github-automation.py \ + --token '${{ secrets.ISSUE_SUBSCRIBER_TOKEN }}' \ + pr-subscriber \ + --issue-number "${{ github.event.number }}" \ + --label-name "${{ github.event.label.name }}" diff --git a/.github/workflows/reserves/llvm-project/release-asset-audit.py b/.github/workflows/reserves/llvm-project/release-asset-audit.py new file mode 100644 index 000000000000..cf6ad7fbbe14 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-asset-audit.py @@ -0,0 +1,66 @@ +import github +import sys + +_SPECIAL_CASE_BINARIES = { + "keith": {"clang+llvm-18.1.8-arm64-apple-macos11.tar.xz"}, +} + + +def _is_valid(uploader_name, valid_uploaders, asset_name): + if uploader_name in valid_uploaders: + return True + + if uploader_name in _SPECIAL_CASE_BINARIES: + return asset_name in _SPECIAL_CASE_BINARIES[uploader_name] + + return False + + +def main(): + token = sys.argv[1] + + gh = github.Github(login_or_token=token) + repo = gh.get_repo("llvm/llvm-project") + + uploaders = set( + [ + "DimitryAndric", + "stefanp-ibm", + "lei137", + "omjavaid", + "nicolerabjohn", + "amy-kwan", + "mandlebug", + "zmodem", + "androm3da", + "tru", + "rovka", + "rorth", + "quinnlp", + "kamaub", + "abrisco", + "jakeegan", + "maryammo", + "tstellar", + "github-actions[bot]", + ] + ) + + for release in repo.get_releases(): + print("Release:", release.title) + for asset in release.get_assets(): + created_at = asset.created_at + updated_at = ( + "" if asset.created_at == asset.updated_at else asset.updated_at + ) + print( + f"{asset.name} : {asset.uploader.login} [{created_at} {updated_at}] ( {asset.download_count} )" + ) + if not _is_valid(asset.uploader.login, uploaders, asset.name): + with open('comment', 'w') as file: + file.write(f'@{asset.uploader.login} is not a valid uploader.') + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/reserves/llvm-project/release-asset-audit.yml b/.github/workflows/reserves/llvm-project/release-asset-audit.yml new file mode 100644 index 000000000000..018c5d542f32 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-asset-audit.yml @@ -0,0 +1,54 @@ +name: Release Asset Audit + +on: + workflow_dispatch: + release: + schedule: + # * is a special character in YAML so you have to quote this string + # Run once an hour + - cron: '5 * * * *' + + pull_request: + paths: + - ".github/workflows/release-asset-audit.py" + - ".github/workflows/release-asset-audit.yml" + +permissions: + contents: read # Default everything to read-only + +jobs: + audit: + name: "Release Asset Audit" + runs-on: ubuntu-22.04 + if: github.repository == 'llvm/llvm-project' + steps: + - uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 #v4.1.6 + - name: "Run Audit Script" + env: + GITHUB_TOKEN: ${{ github.token }} + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + python3 ./.github/workflows/release-asset-audit.py $GITHUB_TOKEN + - name: "File Issue" + if: >- + github.event_name != 'pull_request' && + failure() + uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea #v7.0.1 + with: + github-token: ${{ secrets.ISSUE_SUBSCRIBER_TOKEN }} + script: | + var fs = require('fs'); + var body = '' + if (fs.existsSync('./comment')) { + body = fs.readFileSync('./comment') + "\n\n"; + } + body = body + `\n\nhttps://github.com/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}` + + const issue = await github.rest.issues.create({ + owner: context.repo.owner, + repo: context.repo.repo, + title: "Release Asset Audit Failed", + labels: ['infrastructure'], + body: body + }); + console.log(issue); diff --git a/.github/workflows/reserves/llvm-project/release-binaries-all.yml b/.github/workflows/reserves/llvm-project/release-binaries-all.yml new file mode 100644 index 000000000000..394b0c74d24e --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-binaries-all.yml @@ -0,0 +1,98 @@ +name: Release Binaries All + +permissions: + contents: read # Default everything to read-only + +on: + workflow_dispatch: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload binaries to the release page' + required: true + default: false + type: boolean + + workflow_call: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload binaries to the release page' + required: true + default: false + type: boolean + + pull_request: + types: + - opened + - synchronize + - reopened + # When a PR is closed, we still start this workflow, but then skip + # all the jobs, which makes it effectively a no-op. The reason to + # do this is that it allows us to take advantage of concurrency groups + # to cancel in progress CI jobs whenever the PR is closed. + - closed + paths: + - '.github/workflows/release-binaries-all.yml' + - '.github/workflows/release-binaries.yml' + - '.github/workflows/release-binaries-setup-stage/*' + - '.github/workflows/release-binaries-save-stage/*' + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || 'dispatch' }} + cancel-in-progress: True + +jobs: + setup-variables: + if: >- + (github.event_name != 'pull_request' || github.event.action != 'closed') + runs-on: ubuntu-22.04 + outputs: + release-version: ${{ steps.vars.outputs.release-version }} + upload: ${{ steps.vars.outputs.upload }} + steps: + - shell: bash + id: vars + run: | + upload="${{ inputs.upload }}" + release_version="${{ inputs.release-version }}" + if [ "${{ github.event_name }}" = "pull_request" ]; then + upload="false" + release_version="" + fi + echo "release-version=$release_version" >> "$GITHUB_OUTPUT" + echo "upload=$upload" >> "$GITHUB_OUTPUT" + + release-binaries-all: + name: Build Release Binaries + needs: + - setup-variables + permissions: + contents: write # For release uploads + id-token: write # For artifact attestations + attestations: write # For artifact attestations + strategy: + fail-fast: false + matrix: + runs-on: + - ubuntu-22.04 + - windows-2022 + - macos-13 + - macos-14 + + uses: ./.github/workflows/release-binaries.yml + with: + release-version: "${{ needs.setup-variables.outputs.release-version }}" + upload: ${{ needs.setup-variables.outputs.upload == 'true'}} + runs-on: "${{ matrix.runs-on }}" + secrets: + # This will be empty for pull_request events, but that's fine, because + # the release-binaries workflow does not use this secret for the + # pull_request event. + RELEASE_TASKS_USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} diff --git a/.github/workflows/reserves/llvm-project/release-binaries-save-stage/action.yml b/.github/workflows/reserves/llvm-project/release-binaries-save-stage/action.yml new file mode 100644 index 000000000000..f08088c7bc56 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-binaries-save-stage/action.yml @@ -0,0 +1,44 @@ +name: Save Stage +description: >- + Upload the source and binary directories from a build stage so that they + can be re-used in the next stage. This action is used to the release + binaries workflow into multiple stages to avoid the 6 hour timeout on + the GitHub hosted runners. +inputs: + build-prefix: + description: "Directory containing the build directory." + required: true + type: 'string' + +permissions: + contents: read + +runs: + using: "composite" + steps: + # We need to create an archive of the build directory, because it has too + # many files to upload. + - name: Package Build and Source Directories + shell: bash + run: | + # Remove .git/config to avoid leaking GITHUB_TOKEN stored there. + # See https://unit42.paloaltonetworks.com/github-repo-artifacts-leak-tokens/ + rm -Rf .git/config + # Windows does not support symlinks, so we need to dereference them. + tar --exclude build/ ${{ (runner.os == 'Windows' && '-h') || '' }} -c . | zstd -T0 -c > ../llvm-project.tar.zst + mv ../llvm-project.tar.zst . + tar -C ${{ inputs.build-prefix }} -c build/ | zstd -T0 -c > build.tar.zst + + - name: Upload Stage 1 Source + uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + with: + name: ${{ runner.os }}-${{ runner.arch }}-${{ github.job }}-source + path: llvm-project.tar.zst + retention-days: 2 + + - name: Upload Stage 1 Build Dir + uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + with: + name: ${{ runner.os}}-${{ runner.arch }}-${{ github.job }}-build + path: build.tar.zst + retention-days: 2 diff --git a/.github/workflows/reserves/llvm-project/release-binaries-setup-stage/action.yml b/.github/workflows/reserves/llvm-project/release-binaries-setup-stage/action.yml new file mode 100644 index 000000000000..f5e5db27e659 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-binaries-setup-stage/action.yml @@ -0,0 +1,59 @@ +name: Setup Stage +description: >- + Setup the next stage of the release binaries workflow. This sets up the + environment correctly for a new stage of the release binaries workflow + and also restores the source and build directory from the previous stage. + +inputs: + previous-artifact: + description: >- + A unique descriptor for the artifact from the previous stage. This will + be used to construct the final artifact pattern, which is: + $RUNNER_OS-$RUNNER_ARCH-$PREVIOUS_ARTIFACT-* + required: false + type: 'string' + +outputs: + build-prefix: + description: "Directory containing the build directory." + value: ${{ steps.build-prefix.outputs.build-prefix }} + +runs: + using: "composite" + steps: + - name: Install Ninja + uses: llvm/actions/install-ninja@22e9f909d35b50bd1181709564bfe816eaeaae81 # main + + - name: Setup Windows + if: startsWith(runner.os, 'Windows') + uses: llvm/actions/setup-windows@main + with: + arch: amd64 + + - name: Set Build Prefix + id: build-prefix + shell: bash + run: | + build_prefix=`pwd` + if [ "${{ runner.os }}" = "Linux" ]; then + sudo chown $USER:$USER /mnt/ + build_prefix=/mnt/ + fi + echo "build-prefix=$build_prefix" >> $GITHUB_OUTPUT + + - name: Download Previous Stage Artifact + if: ${{ inputs.previous-artifact }} + id: download + uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1 + with: + pattern: ${{ runner.os }}-${{ runner.arch }}-${{ inputs.previous-artifact }}-* + merge-multiple: true + + - name: Unpack Artifact + if: ${{ steps.download.outputs.download-path }} + shell: bash + run: | + tar --zstd -xf llvm-project.tar.zst + rm llvm-project.tar.zst + tar --zstd -C ${{ steps.build-prefix.outputs.build-prefix}} -xf build.tar.zst + rm build.tar.zst diff --git a/.github/workflows/reserves/llvm-project/release-binaries.yml b/.github/workflows/reserves/llvm-project/release-binaries.yml new file mode 100644 index 000000000000..672dd7517d23 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-binaries.yml @@ -0,0 +1,481 @@ +name: Release Binaries + +on: + workflow_dispatch: + inputs: + release-version: + description: 'Release Version' + required: false + type: string + upload: + description: 'Upload binaries to the release page' + required: true + default: false + type: boolean + runs-on: + description: "Runner to use for the build" + required: true + type: choice + options: + - ubuntu-22.04 + - windows-2022 + - macos-13 + - macos-14 + + workflow_call: + inputs: + release-version: + description: 'Release Version' + required: false + type: string + upload: + description: 'Upload binaries to the release page' + required: true + default: false + type: boolean + runs-on: + description: "Runner to use for the build" + required: true + type: string + secrets: + RELEASE_TASKS_USER_TOKEN: + description: "Secret used to check user permissions." + required: false + + +permissions: + contents: read # Default everything to read-only + +jobs: + prepare: + name: Prepare to build binaries + runs-on: ${{ inputs.runs-on }} + if: github.repository == 'llvm/llvm-project' + outputs: + release-version: ${{ steps.vars.outputs.release-version }} + ref: ${{ steps.vars.outputs.ref }} + upload: ${{ steps.vars.outputs.upload }} + target-cmake-flags: ${{ steps.vars.outputs.target-cmake-flags }} + build-flang: ${{ steps.vars.outputs.build-flang }} + enable-pgo: ${{ steps.vars.outputs.enable-pgo }} + release-binary-basename: ${{ steps.vars.outputs.release-binary-basename }} + release-binary-filename: ${{ steps.vars.outputs.release-binary-filename }} + + steps: + # It's good practice to use setup-python, but this is also required on macos-14 + # due to https://github.com/actions/runner-images/issues/10385 + - uses: actions/setup-python@39cd14951b08e74b54015e9e001cdefcf80e669f + with: + python-version: '3.12' + + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + + - name: Install Dependencies + shell: bash + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + + - name: Check Permissions + if: github.event_name != 'pull_request' + env: + GITHUB_TOKEN: ${{ github.token }} + USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + shell: bash + run: | + ./llvm/utils/release/./github-upload-release.py --token "$GITHUB_TOKEN" --user ${{ github.actor }} --user-token "$USER_TOKEN" check-permissions + + - name: Collect Variables + id: vars + shell: bash + # In order for the test-release.sh script to run correctly, the LLVM + # source needs to be at the following location relative to the build dir: + # | X.Y.Z-rcN | ./rcN/llvm-project + # | X.Y.Z | ./final/llvm-project + # + # We also need to set divergent flags based on the release version: + # | X.Y.Z-rcN | -rc N -test-asserts + # | X.Y.Z | -final + run: | + trimmed=$(echo ${{ inputs.release-version }} | xargs) + if [ -n "$trimmed" ]; then + release_version="$trimmed" + ref="llvmorg-$release_version" + else + release_version="${{ (github.event_name == 'pull_request' && format('PR{0}', github.event.pull_request.number)) || 'CI'}}-${{ github.sha }}" + ref=${{ github.sha }} + fi + if [ -n "${{ inputs.upload }}" ]; then + upload="${{ inputs.upload }}" + else + upload="false" + fi + echo "release-version=$release_version">> $GITHUB_OUTPUT + echo "ref=$ref" >> $GITHUB_OUTPUT + echo "upload=$upload" >> $GITHUB_OUTPUT + + release_binary_basename="LLVM-$release_version-${{ runner.os }}-${{ runner.arch }}" + echo "release-binary-basename=$release_binary_basename" >> $GITHUB_OUTPUT + echo "release-binary-filename=$release_binary_basename.tar.xz" >> $GITHUB_OUTPUT + + # Detect necessary CMake flags + target="${{ runner.os }}-${{ runner.arch }}" + echo "enable-pgo=false" >> $GITHUB_OUTPUT + target_cmake_flags="-DLLVM_RELEASE_ENABLE_PGO=OFF" + # The macOS builds try to cross compile some libraries so we need to + # add extra CMake args to disable them. + # See https://github.com/llvm/llvm-project/issues/99767 + if [ "${{ runner.os }}" = "macOS" ]; then + target_cmake_flags="$target_cmake_flags -DBOOTSTRAP_COMPILER_RT_ENABLE_IOS=OFF" + if [ "${{ runner.arch }}" = "ARM64" ]; then + arches=arm64 + else + arches=x86_64 + fi + target_cmake_flags="$target_cmake_flags -DBOOTSTRAP_DARWIN_osx_ARCHS=$arches -DBOOTSTRAP_DARWIN_osx_BUILTIN_ARCHS=$arches" + fi + + build_flang="true" + + if [ "${{ runner.os }}" = "Windows" ]; then + # The build times out on Windows, so we need to disable LTO. + target_cmake_flags="$target_cmake_flags -DLLVM_RELEASE_ENABLE_LTO=OFF" + fi + + echo "target-cmake-flags=$target_cmake_flags" >> $GITHUB_OUTPUT + echo "build-flang=$build_flang" >> $GITHUB_OUTPUT + + build-stage1: + name: "Build Stage 1" + needs: prepare + if: github.repository == 'llvm/llvm-project' + runs-on: ${{ inputs.runs-on }} + steps: + + - name: Checkout Actions + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + # Check out outside of working directory so the source checkout doesn't + # remove it. + path: workflows + + # actions/checkout does not support paths outside of the GITHUB_WORKSPACE. + # Also, anything that we put inside of GITHUB_WORKSPACE will be overwritten + # by future actions/checkout steps. Therefore, in order to checkout the + # latest actions from main, we need to first checkout out the actions inside of + # GITHUB_WORKSPACE (see previous step), then use actions/checkout to checkout + # the code being built and the move the actions from main back into GITHUB_WORKSPACE, + # becasue the uses on composite actions only reads workflows from inside GITHUB_WORKSPACE. + - shell: bash + run: mv workflows ../workflows-main + + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + ref: ${{ needs.prepare.outputs.ref }} + + - name: Copy main workflows + shell: bash + run: | + mv ../workflows-main . + + - name: Setup Stage + id: setup-stage + uses: ./workflows-main/.github/workflows/release-binaries-setup-stage + + - name: Setup sccache + uses: hendrikmuhs/ccache-action@ca3acd2731eef11f1572ccb126356c2f9298d35e # v1.2.9 + with: + # Default to 2G to workaround: https://github.com/hendrikmuhs/ccache-action/issues/174 + max-size: 2G + key: sccache-${{ runner.os }}-${{ runner.arch }}-release + variant: sccache + + - name: Build Stage 1 Clang + id: build + shell: bash + run: | + # There were some issues on the ARM64 MacOS runners with trying to build x86 object, + # so we need to set some extra cmake flags to disable this. + cmake -G Ninja -S llvm -B ${{ steps.setup-stage.outputs.build-prefix }}/build \ + ${{ needs.prepare.outputs.target-cmake-flags }} \ + -C clang/cmake/caches/Release.cmake \ + -DBOOTSTRAP_LLVM_PARALLEL_LINK_JOBS=1 \ + -DBOOTSTRAP_CPACK_PACKAGE_FILE_NAME="${{ needs.prepare.outputs.release-binary-basename }}" \ + -DCMAKE_C_COMPILER_LAUNCHER=sccache \ + -DCMAKE_CXX_COMPILER_LAUNCHER=sccache + ninja -v -C ${{ steps.setup-stage.outputs.build-prefix }}/build + # There is a race condition on the MacOS builders and this command is here + # to help debug that when it happens. + ls -ltr ${{ steps.setup-stage.outputs.build-prefix }}/build + + - name: Save Stage + uses: ./workflows-main/.github/workflows/release-binaries-save-stage + with: + build-prefix: ${{ steps.setup-stage.outputs.build-prefix }} + + build-stage2: + name: "Build Stage 2" + needs: + - prepare + - build-stage1 + if: github.repository == 'llvm/llvm-project' + runs-on: ${{ inputs.runs-on }} + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + path: workflows + - name: Setup Stage + id: setup-stage + uses: ./workflows/.github/workflows/release-binaries-setup-stage + with: + previous-artifact: build-stage1 + + - name: Build Stage 2 + # Re-enable once PGO builds are supported. + if: needs.prepare.outputs.enable-pgo == 'true' + shell: bash + run: | + ninja -C ${{ steps.setup-stage.outputs.build-prefix}}/build stage2-instrumented + + - name: Save Stage + uses: ./workflows/.github/workflows/release-binaries-save-stage + with: + build-prefix: ${{ steps.setup-stage.outputs.build-prefix }} + + build-stage3-clang: + name: "Build Stage 3 LLVM/Clang" + needs: + - prepare + - build-stage2 + if: github.repository == 'llvm/llvm-project' + runs-on: ${{ inputs.runs-on }} + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + path: workflows + - name: Setup Stage + id: setup-stage + uses: ./workflows/.github/workflows/release-binaries-setup-stage + with: + previous-artifact: build-stage2 + + - name: Build LLVM/Clang + shell: bash + run: | + # There is a race condition on the MacOS builders and this command is here + # to help debug that when it happens. + ls -ltr ${{ steps.setup-stage.outputs.build-prefix }}/build + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build stage2-clang + # Build some of the larger binaries here too. + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build/tools/clang/stage2-bins/ \ + clang-scan-deps \ + modularize clangd \ + clangd-indexer \ + clang-check \ + ${{ (runner.os == 'Linux' && 'clangd-fuzzer') || '' }} \ + clang-tidy \ + llc \ + lli \ + llvm-exegesis \ + llvm-opt-fuzzer \ + llvm-reduce \ + llvm-lto \ + dsymutil + + - name: Save Stage + uses: ./workflows/.github/workflows/release-binaries-save-stage + with: + build-prefix: ${{ steps.setup-stage.outputs.build-prefix }} + + build-stage3-flang: + name: "Build Stage 3 Flang/MLIR/Bolt" + needs: + - prepare + - build-stage3-clang + runs-on: ${{ inputs.runs-on }} + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + path: workflows + - name: Setup Stage + id: setup-stage + uses: ./workflows/.github/workflows/release-binaries-setup-stage + with: + previous-artifact: build-stage3-clang + + - name: Build Flang / MLIR / Bolt + shell: bash + run: | + # Build some of the mlir tools that take a long time to link + if [ "${{ needs.prepare.outputs.build-flang }}" = "true" ]; then + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build/tools/clang/stage2-bins/ -j2 flang-new bbc + fi + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build/tools/clang/stage2-bins/ \ + mlir-bytecode-parser-fuzzer \ + mlir-cpu-runner \ + mlir-lsp-server \ + mlir-opt \ + mlir-query \ + mlir-reduce \ + mlir-text-parser-fuzzer \ + mlir-translate \ + mlir-transform-opt \ + mlir-cat \ + mlir-minimal-opt \ + mlir-minimal-opt-canonicalize \ + mlir-pdll-lsp-server \ + llvm-bolt \ + llvm-bolt-heatmap + + - name: Save Stage + uses: ./workflows/.github/workflows/release-binaries-save-stage + with: + build-prefix: ${{ steps.setup-stage.outputs.build-prefix }} + + build-stage3-all: + name: "Build Stage 3" + needs: + - prepare + - build-stage3-flang + runs-on: ${{ inputs.runs-on }} + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + path: workflows + - name: Setup Stage + id: setup-stage + uses: ./workflows/.github/workflows/release-binaries-setup-stage + with: + previous-artifact: build-stage3-flang + + - name: Build Release Package + shell: bash + run: | + which cmake + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build stage2-package + # Copy Release artifact to the workspace so it is easier to upload. + # This is necessary, because on Windows, the build-prefix path can + # only be used on bash steps, because it uses the form of /d/files/ + # and other steps expect D:\files. + mv ${{ steps.setup-stage.outputs.build-prefix }}/build/tools/clang/stage2-bins/${{ needs.prepare.outputs.release-binary-filename }} . + + - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 #v4.3.0 + with: + name: ${{ runner.os }}-${{ runner.arch }}-release-binary + # Due to path differences on Windows when running in bash vs running on node, + # we need to search for files in the current workspace. + path: | + ${{ needs.prepare.outputs.release-binary-filename }} + + # Clean up some build files to reduce size of artifact. + - name: Clean Up Build Directory + shell: bash + run: | + find ${{ steps.setup-stage.outputs.build-prefix }}/build -iname ${{ needs.prepare.outputs.release-binary-filename }} -delete + rm -Rf ${{ steps.setup-stage.outputs.build-prefix }}/build/tools/clang/stage2-bins/_CPack_Packages + + - name: Save Stage + uses: ./workflows/.github/workflows/release-binaries-save-stage + with: + build-prefix: ${{ steps.setup-stage.outputs.build-prefix }} + + upload-release-binaries: + name: "Upload Release Binaries" + needs: + - prepare + - build-stage3-all + if: >- + always() && + github.event_name != 'pull_request' && + needs.prepare.outputs.upload == 'true' + runs-on: ubuntu-22.04 + permissions: + contents: write # For release uploads + id-token: write # For artifact attestations + attestations: write # For artifact attestations + + steps: + - name: 'Download artifact' + uses: actions/download-artifact@6b208ae046db98c579e8a3aa621ab581ff575935 # v4.1.1 + with: + pattern: '*-release-binary' + merge-multiple: true + + - name: Attest Build Provenance + id: provenance + uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0 + with: + subject-path: ${{ needs.prepare.outputs.release-binary-filename }} + + - name: Rename attestation file + run: + mv ${{ steps.provenance.outputs.bundle-path }} ${{ needs.prepare.outputs.release-binary-filename }}.jsonl + + - name: Upload Build Provenance + uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 #v4.3.3 + with: + name: ${{ needs.prepare.outputs.release-binary-filename }}-attestation + path: ${{ needs.prepare.outputs.release-binary-filename }}.jsonl + + - name: Upload Release + shell: bash + run: | + sudo apt install python3-github + ./llvm-project/llvm/utils/release/github-upload-release.py \ + --token ${{ github.token }} \ + --release ${{ needs.prepare.outputs.release-version }} \ + upload \ + --files ${{ needs.prepare.outputs.release-binary-filename }}* + + test-stage3: + name: "Test Stage 3" + needs: + - prepare + - build-stage3-all + if: >- + github.repository == 'llvm/llvm-project' + runs-on: ${{ inputs.runs-on }} + steps: + - name: Checkout Actions + uses: actions/checkout@v4 + with: + ref: ${{ (github.event_name == 'pull_request' && github.sha) || 'main' }} + sparse-checkout: | + .github/workflows/ + sparse-checkout-cone-mode: false + path: workflows + - name: Setup Stage + id: setup-stage + uses: ./workflows/.github/workflows/release-binaries-setup-stage + with: + previous-artifact: build-stage3-all + + - name: Run Tests + shell: bash + run: | + ninja -C ${{ steps.setup-stage.outputs.build-prefix }}/build stage2-check-all diff --git a/.github/workflows/reserves/llvm-project/release-documentation.yml b/.github/workflows/reserves/llvm-project/release-documentation.yml new file mode 100644 index 000000000000..70e5f08b6f72 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-documentation.yml @@ -0,0 +1,88 @@ +name: Release Documentation + +permissions: + contents: read + +on: + workflow_dispatch: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload documentation' + required: false + type: boolean + + workflow_call: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload documentation' + required: false + type: boolean + +jobs: + release-documentation: + name: Build and Upload Release Documentation + runs-on: ubuntu-latest + env: + upload: ${{ inputs.upload && !contains(inputs.release-version, 'rc') }} + steps: + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + + - name: Setup Python env + uses: actions/setup-python@v5 + with: + cache: 'pip' + cache-dependency-path: './llvm/docs/requirements.txt' + + - name: Install Dependencies + run: | + sudo apt-get update + sudo apt-get install -y \ + graphviz \ + python3-github \ + ninja-build \ + texlive-font-utils + pip3 install --user -r ./llvm/docs/requirements.txt + + - name: Build Documentation + env: + GITHUB_TOKEN: ${{ github.token }} + run: | + ./llvm/utils/release/build-docs.sh -release "${{ inputs.release-version }}" -no-doxygen + + - name: Create Release Notes Artifact + uses: actions/upload-artifact@v3 + with: + name: release-notes + path: docs-build/html-export/ + + - name: Clone www-releases + if: env.upload + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + repository: ${{ github.repository_owner }}/www-releases + ref: main + fetch-depth: 0 + path: www-releases + + - name: Upload Release Notes + if: env.upload + env: + WWW_RELEASES_TOKEN: ${{ secrets.WWW_RELEASES_TOKEN }} + run: | + mkdir -p ../www-releases/${{ inputs.release-version }} + mv ./docs-build/html-export/* ../www-releases/${{ inputs.release-version }} + cd ../www-releases + git add ${{ inputs.release-version }} + git config user.email "llvmbot@llvm.org" + git config user.name "llvmbot" + git commit -a -m "Add ${{ inputs.release-version }} documentation" + git push "https://$WWW_RELEASES_TOKEN@github.com/${{ github.repository_owner }}/www-releases" main:main diff --git a/.github/workflows/reserves/llvm-project/release-doxygen.yml b/.github/workflows/reserves/llvm-project/release-doxygen.yml new file mode 100644 index 000000000000..ea95e5bb12b2 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-doxygen.yml @@ -0,0 +1,72 @@ +name: Release Doxygen + +permissions: + contents: read + +on: + workflow_dispatch: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload documentation' + required: false + type: boolean + + workflow_call: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + upload: + description: 'Upload documentation' + required: false + type: boolean + secrets: + RELEASE_TASKS_USER_TOKEN: + description: "Secret used to check user permissions." + required: false + +jobs: + release-doxygen: + name: Build and Upload Release Doxygen + runs-on: ubuntu-latest + permissions: + contents: write + env: + upload: ${{ inputs.upload && !contains(inputs.release-version, 'rc') }} + steps: + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + + - name: Setup Python env + uses: actions/setup-python@v5 + with: + cache: 'pip' + cache-dependency-path: './llvm/docs/requirements.txt' + + - name: Install Dependencies + run: | + sudo apt-get update + sudo apt-get install -y \ + doxygen \ + graphviz \ + python3-github \ + ninja-build \ + texlive-font-utils + pip3 install --user -r ./llvm/docs/requirements.txt + + - name: Build Doxygen + run: | + ./llvm/utils/release/build-docs.sh -release "${{ inputs.release-version }}" -no-sphinx + + - name: Upload Doxygen + if: env.upload + env: + GITHUB_TOKEN: ${{ github.token }} + USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + run: | + ./llvm/utils/release/github-upload-release.py --token "$GITHUB_TOKEN" --release "${{ inputs.release-version }}" --user "${{ github.actor }}" --user-token "$USER_TOKEN" upload --files ./*doxygen*.tar.xz diff --git a/.github/workflows/reserves/llvm-project/release-lit.yml b/.github/workflows/reserves/llvm-project/release-lit.yml new file mode 100644 index 000000000000..9d6f3140e688 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-lit.yml @@ -0,0 +1,79 @@ +name: Release Lit + +permissions: + contents: read + +on: + workflow_dispatch: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + + workflow_call: + inputs: + release-version: + description: 'Release Version' + required: true + type: string + secrets: + RELEASE_TASKS_USER_TOKEN: + description: "Secret used to check user permissions." + required: false + +jobs: + release-lit: + name: Release Lit + runs-on: ubuntu-latest + steps: + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + ref: "llvmorg-${{ inputs.release-version }}" + + - name: Install dependencies + run: | + sudo apt-get update + sudo apt-get install -y python3-setuptools python3-psutil python3-github + + - name: Check Permissions + env: + GITHUB_TOKEN: ${{ github.token }} + USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + run: | + ./llvm/utils/release/./github-upload-release.py --token "$GITHUB_TOKEN" --user ${{ github.actor }} --user-token "$USER_TOKEN" check-permissions + + - name: Setup Cpp + uses: aminya/setup-cpp@v1 + with: + compiler: llvm-16.0.6 + cmake: true + ninja: true + + - name: Test lit + run: | + mkdir build && cd build + export FILECHECK_OPTS='-dump-input-filter=all -vv -color' + cmake ../llvm -DCMAKE_BUILD_TYPE=Release -G Ninja + ninja -v -j $(nproc) check-lit + + - name: Package lit + run: | + cd llvm/utils/lit + # Remove 'dev' suffix from lit version. + sed -i 's/ + "dev"//g' lit/__init__.py + python3 setup.py sdist bdist_wheel + + - name: Upload lit to test.pypi.org + uses: pypa/gh-action-pypi-publish@release/v1 + with: + password: ${{ secrets.LLVM_LIT_TEST_PYPI_API_TOKEN }} + repository-url: https://test.pypi.org/legacy/ + packages-dir: llvm/utils/lit/dist/ + + - name: Upload lit to pypi.org + uses: pypa/gh-action-pypi-publish@release/v1 + with: + password: ${{ secrets.LLVM_LIT_PYPI_API_TOKEN }} + packages-dir: llvm/utils/lit/dist/ diff --git a/.github/workflows/reserves/llvm-project/release-sources.yml b/.github/workflows/reserves/llvm-project/release-sources.yml new file mode 100644 index 000000000000..a6c86823f99d --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-sources.yml @@ -0,0 +1,108 @@ +name: Release Sources + +permissions: + contents: read + +on: + workflow_dispatch: + inputs: + release-version: + description: Release Version + required: true + type: string + workflow_call: + inputs: + release-version: + description: Release Version + required: true + type: string + secrets: + RELEASE_TASKS_USER_TOKEN: + description: "Secret used to check user permissions." + required: false + # Run on pull_requests for testing purposes. + pull_request: + paths: + - '.github/workflows/release-sources.yml' + types: + - opened + - synchronize + - reopened + # When a PR is closed, we still start this workflow, but then skip + # all the jobs, which makes it effectively a no-op. The reason to + # do this is that it allows us to take advantage of concurrency groups + # to cancel in progress CI jobs whenever the PR is closed. + - closed + +concurrency: + group: ${{ github.workflow }}-${{ inputs.release-version || github.event.pull_request.number }} + cancel-in-progress: True + +jobs: + inputs: + name: Collect Job Inputs + if: >- + github.repository_owner == 'llvm' && + github.event.action != 'closed' + outputs: + ref: ${{ steps.inputs.outputs.ref }} + export-args: ${{ steps.inputs.outputs.export-args }} + runs-on: ubuntu-latest + steps: + - id: inputs + run: | + ref=${{ (inputs.release-version && format('llvmorg-{0}', inputs.release-version)) || github.sha }} + if [ -n "${{ inputs.release-version }}" ]; then + export_args="-release ${{ inputs.release-version }} -final" + else + export_args="-git-ref ${{ github.sha }}" + fi + echo "ref=$ref" >> $GITHUB_OUTPUT + echo "export-args=$export_args" >> $GITHUB_OUTPUT + + release-sources: + name: Package Release Sources + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + needs: + - inputs + permissions: + id-token: write + attestations: write + steps: + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + with: + ref: ${{ needs.inputs.outputs.ref }} + fetch-tags: true + - name: Install Dependencies + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + + - name: Check Permissions + if: github.event_name != 'pull_request' + env: + GITHUB_TOKEN: ${{ github.token }} + USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + run: | + ./llvm/utils/release/./github-upload-release.py --token "$GITHUB_TOKEN" --user ${{ github.actor }} --user-token "$USER_TOKEN" check-permissions + - name: Create Tarballs + run: | + ./llvm/utils/release/export.sh ${{ needs.inputs.outputs.export-args }} + - name: Attest Build Provenance + if: github.event_name != 'pull_request' + id: provenance + uses: actions/attest-build-provenance@897ed5eab6ed058a474202017ada7f40bfa52940 # v1.0.0 + with: + subject-path: "*.xz" + - if: github.event_name != 'pull_request' + run: | + mv ${{ steps.provenance.outputs.bundle-path }} . + - name: Create Tarball Artifacts + uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 #v4.3.3 + with: + path: | + *.xz + attestation.jsonl + + diff --git a/.github/workflows/reserves/llvm-project/release-tasks.yml b/.github/workflows/reserves/llvm-project/release-tasks.yml new file mode 100644 index 000000000000..780dd0ff6325 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/release-tasks.yml @@ -0,0 +1,123 @@ +name: Release Task + +permissions: + contents: read + +on: + push: + tags: + # The regex support here is limited, so just match everything that starts with llvmorg- and filter later. + - 'llvmorg-*' + +jobs: + validate-tag: + name: Validate Tag + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + outputs: + release-version: ${{ steps.validate-tag.outputs.release-version }} + steps: + - name: Validate Tag + id: validate-tag + run: | + echo "${{ github.ref_name }}" | grep -e '^llvmorg-[0-9]\+\.[0-9]\+\.[0-9]\+\(-rc[0-9]\+\)\?$' + release_version=$(echo "${{ github.ref_name }}" | sed 's/llvmorg-//g') + echo "release-version=$release_version" >> "$GITHUB_OUTPUT" + + release-create: + name: Create a New Release + runs-on: ubuntu-latest + permissions: + contents: write # For creating the release. + needs: validate-tag + + steps: + - name: Install Dependencies + run: | + sudo apt-get update + sudo apt-get install python3-github + + - name: Checkout LLVM + uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1 + + - name: Create Release + env: + GITHUB_TOKEN: ${{ github.token }} + USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + run: | + ./llvm/utils/release/./github-upload-release.py --token "$GITHUB_TOKEN" --release ${{ needs.validate-tag.outputs.release-version }} --user ${{ github.actor }} --user-token "$USER_TOKEN" create + release-documentation: + name: Build and Upload Release Documentation + needs: + - validate-tag + uses: ./.github/workflows/release-documentation.yml + with: + release-version: ${{ needs.validate-tag.outputs.release-version }} + upload: true + + release-doxygen: + name: Build and Upload Release Doxygen + permissions: + contents: write + needs: + - validate-tag + - release-create + uses: ./.github/workflows/release-doxygen.yml + with: + release-version: ${{ needs.validate-tag.outputs.release-version }} + upload: true + # Called workflows don't have access to secrets by default, so we need to explicitly pass secrets that we use. + secrets: + RELEASE_TASKS_USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + + release-lit: + name: Release Lit + needs: validate-tag + uses: ./.github/workflows/release-lit.yml + with: + release-version: ${{ needs.validate-tag.outputs.release-version }} + # Called workflows don't have access to secrets by default, so we need to explicitly pass secrets that we use. + secrets: + RELEASE_TASKS_USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + + release-binaries: + name: Build Release Binaries + permissions: + contents: write + id-token: write + attestations: write + needs: + - validate-tag + - release-create + strategy: + fail-fast: false + matrix: + runs-on: + - ubuntu-22.04 + - windows-2022 + - macos-13 + - macos-14 + + uses: ./.github/workflows/release-binaries.yml + with: + release-version: ${{ needs.validate-tag.outputs.release-version }} + upload: true + runs-on: ${{ matrix.runs-on }} + # Called workflows don't have access to secrets by default, so we need to explicitly pass secrets that we use. + secrets: + RELEASE_TASKS_USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} + + release-sources: + name: Package Release Sources + permissions: + contents: read + id-token: write + attestations: write + needs: + - validate-tag + uses: ./.github/workflows/release-sources.yml + with: + release-version: ${{ needs.validate-tag.outputs.release-version }} + # Called workflows don't have access to secrets by default, so we need to explicitly pass secrets that we use. + secrets: + RELEASE_TASKS_USER_TOKEN: ${{ secrets.RELEASE_TASKS_USER_TOKEN }} diff --git a/.github/workflows/reserves/llvm-project/scorecard.yml b/.github/workflows/reserves/llvm-project/scorecard.yml new file mode 100644 index 000000000000..ff61cf83a6af --- /dev/null +++ b/.github/workflows/reserves/llvm-project/scorecard.yml @@ -0,0 +1,62 @@ +# This workflow uses actions that are not certified by GitHub. They are provided +# by a third-party and are governed by separate terms of service, privacy +# policy, and support documentation. + +# Check current LLVM-Project results here: https://securityscorecards.dev/viewer/?uri=github.com/llvm/llvm-project + +name: Scorecard supply-chain security +on: + # For Branch-Protection check. Only the default branch is supported. See + # https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection + branch_protection_rule: + # To guarantee Maintained check is occasionally updated. See + # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained + schedule: + - cron: '38 20 * * *' + +# Declare default permissions as read only. +permissions: + contents: read + +jobs: + analysis: + name: Scorecard analysis + runs-on: ubuntu-latest + if: github.repository == 'llvm/llvm-project' + permissions: + # Needed to upload the results to code-scanning dashboard. + security-events: write + # Needed to publish results and get a badge (see publish_results below). + id-token: write + + steps: + - name: "Checkout code" + uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0 + with: + persist-credentials: false + + - name: "Run analysis" + uses: ossf/scorecard-action@0864cf19026789058feabb7e87baa5f140aac736 # v2.3.1 + with: + results_file: results.sarif + results_format: sarif + + # - Publish results to OpenSSF REST API for easy access by consumers + # - Allows the repository to include the Scorecard badge. + # - See https://github.com/ossf/scorecard-action#publishing-results. + publish_results: true + + # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF + # format to the repository Actions tab. + - name: "Upload artifact" + uses: actions/upload-artifact@3cea5372237819ed00197afe530f5a7ea3e805c8 # v3.1.0 + with: + name: SARIF file + path: results.sarif + retention-days: 5 + + # Upload the results to GitHub's code scanning dashboard. + - name: "Upload to code-scanning" + uses: github/codeql-action/upload-sarif@17573ee1cc1b9d061760f3a006fc4aac4f944fd5 # v2.2.4 + with: + sarif_file: results.sarif diff --git a/.github/workflows/reserves/llvm-project/set-release-binary-outputs.sh b/.github/workflows/reserves/llvm-project/set-release-binary-outputs.sh new file mode 100644 index 000000000000..14d0798364e9 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/set-release-binary-outputs.sh @@ -0,0 +1,34 @@ +# Usage: set-release-binary-outputs.sh + +set -e + +if [ -z "$GITHUB_OUTPUT" ]; then + export GITHUB_OUTPUT=`mktemp` + echo "Warning: Environment variable GITHUB_OUTPUT is not set." + echo "Writing output variables to $GITHUB_OUTPUT" +fi + +tag=$1 +upload=$2 + +if echo $tag | grep -e '^[0-9a-f]\+$'; then + # This is a plain commit. + # TODO: Don't hardcode this. + release_version="18" + upload='false' + ref="$tag" + +else + + pattern='^llvmorg-[0-9]\+\.[0-9]\+\.[0-9]\+\(-rc[0-9]\+\)\?$' + echo "$tag" | grep -e $pattern + if [ $? != 0 ]; then + echo "ERROR: Tag '$tag' doesn't match pattern: $pattern" + exit 1 + fi + release_version=`echo "$tag" | sed 's/llvmorg-//g'` + release=`echo "$release_version" | sed 's/-.*//g'` +fi +echo "release-version=$release_version" >> $GITHUB_OUTPUT +echo "upload=$upload" >> $GITHUB_OUTPUT +echo "ref=$tag" >> $GITHUB_OUTPUT diff --git a/.github/workflows/reserves/llvm-project/spirv-tests.yml b/.github/workflows/reserves/llvm-project/spirv-tests.yml new file mode 100644 index 000000000000..75918e73e897 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/spirv-tests.yml @@ -0,0 +1,29 @@ +name: SPIR-V Tests + +permissions: + contents: read + +on: + workflow_dispatch: + pull_request: + paths: + - 'llvm/lib/Target/SPIRV/**' + - 'llvm/test/CodeGen/SPIRV/**' + - '.github/workflows/spirv-tests.yml' + +concurrency: + # Skip intermediate builds: always. + # Cancel intermediate builds: only if it is a pull request build. + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }} + +jobs: + check_spirv: + if: github.repository_owner == 'llvm' + name: Test SPIR-V + uses: ./.github/workflows/llvm-project-tests.yml + with: + build_target: check-llvm-codegen-spirv + projects: + extra_cmake_args: '-DLLVM_TARGETS_TO_BUILD="" -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="SPIRV" -DLLVM_INCLUDE_SPIRV_TOOLS_TESTS=ON' + os_list: '["ubuntu-latest"]' diff --git a/.github/workflows/reserves/llvm-project/unprivileged-download-artifact/action.yml b/.github/workflows/reserves/llvm-project/unprivileged-download-artifact/action.yml new file mode 100644 index 000000000000..9d8fb59a67c0 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/unprivileged-download-artifact/action.yml @@ -0,0 +1,81 @@ +name: Unprivileged Download Artifact +description: >- + Download artifacts from another workflow run without using an access token. +inputs: + run-id: + description: >- + The run-id for the workflow run that you want to download the artifact + from. If ommitted it will download the most recently created artifact + from the repo with the artifact-name. + required: false + artifact-name: + desciption: The name of the artifact to download. + required: true + + +outputs: + filename: + description: >- + The filename of the downloaded artifact or the empty string if the + artifact was not found. + value: ${{ steps.download-artifact.outputs.filename }} + artifact-id: + description: "The id of the artifact being downloaded." + value: ${{ steps.artifact-url.outputs.id }} + + +runs: + using: "composite" + steps: + - uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea #v7.0.1 + id: artifact-url + with: + script: | + var response; + if (!"${{ inputs.run-id }}") { + response = await github.rest.actions.listArtifactsForRepo({ + owner: context.repo.owner, + repo: context.repo.repo, + name: "${{ inputs.artifact-name }}" + }) + } else { + response = await github.rest.actions.listWorkflowRunArtifacts({ + owner: context.repo.owner, + repo: context.repo.repo, + run_id: "${{ inputs.run-id }}", + name: "${{ inputs.artifact-name }}" + }) + } + + console.log(response) + + for (artifact of response.data.artifacts) { + console.log(artifact); + } + + if (response.data.artifacts.length == 0) { + console.log("Could not find artifact ${{ inputs.artifact-name }} for workflow run ${{ inputs.run-id }}") + return; + } + + const url_response = await github.rest.actions.downloadArtifact({ + owner: context.repo.owner, + repo: context.repo.repo, + artifact_id: response.data.artifacts[0].id, + archive_format: "zip" + }) + + core.setOutput("url", url_response.url); + core.setOutput("id", response.data.artifacts[0].id); + + - shell: bash + if: steps.artifact-url.outputs.url != '' + id: download-artifact + run: | + curl -L -o ${{ inputs.artifact-name }}.zip "${{ steps.artifact-url.outputs.url }}" + echo "filename=${{ inputs.artifact-name }}.zip" >> $GITHUB_OUTPUT + + - shell: bash + if: steps.download-artifact.outputs.filename != '' + run: | + unzip ${{ steps.download-artifact.outputs.filename }} diff --git a/.github/workflows/reserves/llvm-project/version-check.py b/.github/workflows/reserves/llvm-project/version-check.py new file mode 100755 index 000000000000..f75fd5030088 --- /dev/null +++ b/.github/workflows/reserves/llvm-project/version-check.py @@ -0,0 +1,36 @@ +#!/usr/bin/python3 + +from git import Repo +import re +import sys + + +def get_version_from_tag(tag): + m = re.match("llvmorg-([0-9]+)\.([0-9]+)\.([0-9]+)(-rc[0-9]+)?$", tag) + if m: + if m.lastindex == 4: + # We have an rc tag. + return m.group(1, 2, 3) + # We have a final release tag. + return (m.group(1), m.group(2), str(int(m.group(3)) + 1)) + + m = re.match("llvmorg-([0-9]+)-init", tag) + if m: + return (m.group(1), "1", "0") + + raise Exception(f"error: Tag is not valid: {tag}") + + +version = sys.argv[1] + +repo = Repo() + +tag = repo.git.describe(tags=True, abbrev=0) +expected_version = ".".join(get_version_from_tag(tag)) + +if version != expected_version: + print("error: Expected version", expected_version, "but found version", version) + sys.exit(1) + +print("Versions match:", version, expected_version) +sys.exit(0) diff --git a/.github/workflows/reserves/llvm-project/version-check.yml b/.github/workflows/reserves/llvm-project/version-check.yml new file mode 100644 index 000000000000..894e07d323ca --- /dev/null +++ b/.github/workflows/reserves/llvm-project/version-check.yml @@ -0,0 +1,31 @@ +name: LLVM Project Version Check + +on: + push: + branches: + - 'release/**' + pull_request: + branches: + - 'release/**' + +permissions: + contents: read + +jobs: + version_check: + if: github.repository_owner == 'llvm' + runs-on: ubuntu-latest + steps: + - name: Fetch LLVM sources + uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - name: Install dependencies + run: | + pip install --require-hashes -r ./llvm/utils/git/requirements.txt + + - name: Version Check + run: | + version=$(grep -o 'LLVM_VERSION_\(MAJOR\|MINOR\|PATCH\) [0-9]\+' cmake/Modules/LLVMVersion.cmake | cut -d ' ' -f 2 | tr "\n" "." | sed 's/.$//g') + .github/workflows/version-check.py "$version" diff --git a/.github/workflows/reserves/main.yml b/.github/workflows/reserves/main.yml new file mode 100644 index 000000000000..071d77be2241 --- /dev/null +++ b/.github/workflows/reserves/main.yml @@ -0,0 +1,44 @@ +# Deploy Jekyll to GitHub Pages +name: Build and deploy Jekyll site + +# 💎 Runs on workflow_run.conclusion +on: # push + workflow_run: + types: [completed] #requested + workflows: ["pages-build-deployment"] + +# 🪂 Allow only one concurrent deployment across the branches +concurrency: + group: "pages" + cancel-in-progress: true + +# Sets global environtment variables +env: + OWNER: ${{ github.repository_owner }} + JEKYLL_CFG: /maps/_config.yml + +jobs: + # Build job + github-pages: + if: github.event.workflow_run.conclusion == 'success' + runs-on: ${{ github.ref_name != 'main' && 'ubuntu-latest' || 'windows-latest' }} + steps: + - name: 📂 Checkout + uses: actions/checkout@v3 + with: + submodules: recursive + token: ${{ secrets.JEKYLL_GITHUB_TOKEN }} + + - name: 💎 Build on Linux + if: runner.os == 'Linux' + uses: eq19/feed@v3 + with: + pre_build_commands: 'make build' + token: ${{ secrets.JEKYLL_GITHUB_TOKEN }} + + - name: 💎 Build on Windows + if: runner.os == 'Windows' + uses: eq19/maps@v1 + with: + dotnet-version: '3.1.x' + token: ${{ secrets.JEKYLL_GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/major-release-num.yml b/.github/workflows/reserves/major-release-num.yml new file mode 100644 index 000000000000..9a3297a43488 --- /dev/null +++ b/.github/workflows/reserves/major-release-num.yml @@ -0,0 +1,23 @@ +name: Move Major Release Tag + +on: + release: + types: [created] + +jobs: + movetag: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Get major version num and update tag + run: | + VERSION=${GITHUB_REF#refs/tags/} + MAJOR=${VERSION%%.*} + git config --global user.name 'Vincent A Cicirello' + git config --global user.email 'cicirello@users.noreply.github.com' + git tag -fa ${MAJOR} -m "Update major version tag" + git push origin ${MAJOR} --force + + diff --git a/.github/workflows/reserves/marty.yml b/.github/workflows/reserves/marty.yml new file mode 100644 index 000000000000..e29227bfbe4b --- /dev/null +++ b/.github/workflows/reserves/marty.yml @@ -0,0 +1,140 @@ +name: build + +on: + push: + branches: + - master + pull_request: + branches: + - master + types: + - ready_for_review + - review_requested + +jobs: + build-and-test-ubuntu-gcc12: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: dependencies + run: sudo apt-get install coreutils build-essential gcc-12 g++-12 gfortran-12 libgsl-dev libgslcblas0 qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools dvipng lcov cmake texlive texlive-latex-extra texlive-luatex -y + - name: configure + run: | + mkdir build + cd build + export CXX=g++-12 + export CC=gcc-12 + export FC=gfortran-12 + cmake .. + - name: make + run: | + cd build + make + - name: test + run: | + cd build + ctest --output-on-failure + build-and-test-ubuntu-gcc9: + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@v3 + - name: dependencies + run: sudo apt-get install coreutils build-essential gcc-9 g++-9 gfortran-9 libgsl-dev libgslcblas0 qt5-default dvipng lcov cmake texlive texlive-latex-extra texlive-luatex -y + - name: configure + run: | + mkdir build + cd build + export CXX=g++-9 + export CC=gcc-9 + export FC=gfortran-9 + cmake .. + - name: make + run: | + cd build + make + - name: install + run: | + cd build + sudo make install + - name: test + run: | + cd build + export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib + ctest --output-on-failure + build-and-test-ubuntu-clang: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + - name: dependencies + run: sudo apt-get install coreutils build-essential clang gfortran libgsl-dev libgslcblas0 qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools dvipng cmake lcov texlive texlive-latex-extra texlive-luatex -y + - name: configure + run: | + mkdir build + cd build + export CXX=clang++ + export CC=clang + export FC=gfortran + cmake .. + - name: make + run: | + cd build + make + - name: install + run: | + cd build + sudo make install + - name: test + run: | + cd build + export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib + ctest --output-on-failure + build-and-test-macos: + runs-on: macos-latest + steps: + - uses: actions/checkout@v3 + - name: dependencies + run: | + brew install coreutils gcc@12 gsl qt5 cmake lcov imagemagick texlive octave pkg-config + brew info qt5 + brew info gsl + - name: configure + run: | + mkdir build + cd build + echo "GSL prefix: $(brew --prefix gsl)" + echo "Content of GSL prefix:" + ls $(brew --prefix gsl) + export PATH=$PATH:/usr/local/bin + export CPATH=$CPATH:/usr/local/include:$(brew --prefix gsl)/include + export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include:$(brew --prefix gsl)/include + export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export CXX=g++-12 + export CC=gcc-12 + export FC=gfortran-12 + cmake .. -DQt5_DIR=$(brew --prefix qt5)/lib/cmake/Qt5 -DCMAKE_PREFIX_PATH=$(brew --prefix qt5)/lib/cmake + - name: make + run: | + cd build + export PATH=$PATH:/usr/local/bin + export CPATH=$CPATH:/usr/local/include:$(brew --prefix gsl)/include + export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include:$(brew --prefix gsl)/include + export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export CXX=g++-12 + export CC=gcc-12 + export FC=gfortran-12 + make + - name: install + run: | + cd build + sudo make install + - name: test + run: | + export PATH=$PATH:/usr/local/bin + export CPATH=$CPATH:/usr/local/include:$(brew --prefix gsl)/include + export C_INCLUDE_PATH=$C_INCLUDE_PATH:/usr/local/include:$(brew --prefix gsl)/include + export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + export DYLD_FALLBACK_LIBRARY_PATH=$DYLD_FALLBACK_LIBRARY_PATH:/usr/local/lib:$(brew --prefix gsl)/lib + cd build + ctest --output-on-failure diff --git a/.github/workflows/reserves/matching.yml b/.github/workflows/reserves/matching.yml new file mode 100644 index 000000000000..940f7a924514 --- /dev/null +++ b/.github/workflows/reserves/matching.yml @@ -0,0 +1,83 @@ +name: CI + +on: + push: + branches: [develop] + pull_request: + branches: [develop] + +jobs: + checkout: + runs-on: ubuntu-latest + outputs: + version: ${{ steps.matching.outputs.version }} + steps: + - uses: actions/checkout@main + - uses: rundocs/checkout@master + - name: Match the commit message whether to Release! + run: | + MESSAGE=$(git log -1 --format="%s") + if [[ ${MESSAGE} =~ ^v[0-9]+\.[0-9]+\.[0-9]+ ]]; then + echo "::set-output name=version::${MESSAGE:1}" + fi + id: matching + if: ${{ github.event_name == 'push' }} + release: + runs-on: ubuntu-latest + needs: checkout + if: ${{ needs.checkout.outputs.version }} + env: + VERSION: ${{ needs.checkout.outputs.version }} + steps: + - uses: actions/checkout@main + with: + fetch-depth: 0 + - uses: actions/setup-node@main + - uses: actions/setup-ruby@main + + - name: Generate changelog and Auto update version! + run: | + LAST_TAG=$(git describe --tags --abbrev=0) + git log ${LAST_TAG}..HEAD --no-merges --format="- %h %s @%an" > action.changelog.md + sed -i "s/${LAST_TAG:1}/${VERSION}/g" jekyll-rtd-theme.gemspec + sed -i "s/${LAST_TAG:1}/${VERSION}/g" _includes/rest/defaults.liquid + + - name: Prettier check and Build theme assets! + run: | + npm install + npm run format && npm run build + + - name: Save changes! + run: | + git remote set-url origin "https://${GITHUB_ACTOR}:${{ secrets.AUTH_TOKEN }}@github.com/${GITHUB_REPOSITORY}.git" + git config user.name "${GITHUB_ACTOR}" + git config user.email "${GITHUB_ACTOR}@users.noreply.github.com" + git add . + git commit -m "💎 v${VERSION}" && git push + + - name: Create release! + uses: actions/create-release@main + with: + release_name: ${{ needs.checkout.outputs.version }} + tag_name: v${{ needs.checkout.outputs.version }} + commitish: ${{ github.ref }} + body_path: action.changelog.md + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Configure gem credentials and Build gemspec! + run: | + CONFIG=${HOME}/.gem/credentials + mkdir -p ${HOME}/.gem && touch ${CONFIG} && chmod 0600 ${CONFIG} + echo "---" > ${CONFIG} + echo ":github: Bearer ${GITHUB_TOKEN}" >> ${CONFIG} + echo ":rubygems_api_key: ${RUBYGEMS_AUTH_TOKEN}" >> ${CONFIG} + gem build *.gemspec + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + RUBYGEMS_AUTH_TOKEN: ${{ secrets.RUBYGEMS_AUTH_TOKEN }} + + - name: Publish rubygems! + run: | + gem push *.gem + gem push --KEY github --host https://rubygems.pkg.github.com/${{ github.repository_owner }} *.gem diff --git a/.github/workflows/reserves/math/Benchmark.yml b/.github/workflows/reserves/math/Benchmark.yml new file mode 100644 index 000000000000..6bc1978a5dfc --- /dev/null +++ b/.github/workflows/reserves/math/Benchmark.yml @@ -0,0 +1,32 @@ +name: 'Kernel Benchmark' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Benchmark: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v2 + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: 'Inter version CPU benchmark' + run: | + cd Sources/Tests/DotnetBenchmark + dotnet run -c Release CommonFunctionsInterVersion + + - name: 'Inter version RAM benchmark' + run: | + cd Sources/Tests/DotnetBenchmark + dotnet run -c Release RAMUsageTest + diff --git a/.github/workflows/reserves/math/CPPBuild.yml b/.github/workflows/reserves/math/CPPBuild.yml new file mode 100644 index 000000000000..a885c2875e40 --- /dev/null +++ b/.github/workflows/reserves/math/CPPBuild.yml @@ -0,0 +1,94 @@ +name: 'C++ Build' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + AngouriMathCpp: + + strategy: + matrix: + include: + - os: windows-latest + flag: win-x64 + - os: ubuntu-latest + flag: linux-x64 + - os: macos-latest + flag: osx-x64 + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + + - name: Setup .NET 5 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '5.0.x' + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + include-prerelease: true + + - name: 'Generating exports' + run: | + cd Sources/Utils + dotnet run -p Utils ExportsGenerator + + - name: 'Install some libs for Linux' + if: ${{ matrix.os == 'ubuntu-latest' }} + run: | + sudo apt-get install zlib1g-dev + sudo apt-get install libkrb5-dev + + - name: 'Building the library into native for Windows' + if: ${{ matrix.os == 'windows-latest' }} + shell: cmd + run: | + cd Sources/Wrappers/AngouriMath.CPP.Exporting + mkdir ../AngouriMath.CPP.Importing/out-x64 + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" + dotnet publish -p:NativeLib=Shared -p:SelfContained=true -r ${{ matrix.flag }} -c release + + - name: 'Building the library into native for Linux & MacOS' + if: ${{ matrix.os != 'windows-latest' }} + run: | + cd Sources/Wrappers/AngouriMath.CPP.Exporting + mkdir ../AngouriMath.CPP.Importing/out-x64 + dotnet publish -p:NativeLib=Shared -p:SelfContained=true -r ${{ matrix.flag }} -c release + + + - name: 'Renaming the library for Linux' + if: ${{ matrix.os == 'ubuntu-latest' }} + run: | + cd Sources/Wrappers/AngouriMath.CPP.Importing/out-x64 + mv AngouriMath.CPP.Exporting.so libAngouriMath.CPP.Exporting.so + + - name: 'Renaming the library for OS X' + if: ${{ matrix.os == 'macos-latest' }} + run: | + cd Sources/Wrappers/AngouriMath.CPP.Importing/out-x64 + mv AngouriMath.CPP.Exporting.dylib libAngouriMath.CPP.Exporting.dylib + install_name_tool -id "@rpath/libAngouriMath.CPP.Exporting.dylib" libAngouriMath.CPP.Exporting.dylib + + + - name: 'Uploading artifacts' + uses: actions/upload-artifact@v2 + with: + name: AngouriMath.CPP-${{ matrix.os }}-x64 + path: ./Sources/Wrappers/AngouriMath.CPP.Importing + retention-days: 3 + + - name: 'Building AngouriMath.Importing' + run: | + cd Sources/Wrappers/AngouriMath.CPP.Importing + cmake -S . -B build + cmake --build build + diff --git a/.github/workflows/reserves/math/CPPTest.yml b/.github/workflows/reserves/math/CPPTest.yml new file mode 100644 index 000000000000..08cf7506244a --- /dev/null +++ b/.github/workflows/reserves/math/CPPTest.yml @@ -0,0 +1,99 @@ +name: 'C++ Test' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + + +env: + tests_path: Sources/Tests/CPPWrapperUnitTests/tests + +jobs: + Testing: + strategy: + matrix: + include: + - os: windows-latest + flag: win-x64 + - os: ubuntu-latest + flag: linux-x64 + - os: macos-latest + flag: osx-x64 + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + include-prerelease: true + + - name: 'Install some libs for Linux' + if: ${{ matrix.os == 'ubuntu-latest' }} + run: | + sudo apt-get install zlib1g-dev + sudo apt-get install libkrb5-dev + + - name: 'Building the library into native for Windows' + if: ${{ matrix.os == 'windows-latest' }} + shell: cmd + run: | + cd Sources/Wrappers/AngouriMath.CPP.Exporting + mkdir ../AngouriMath.CPP.Importing/out-x64 + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvars64.bat" + dotnet publish -p:NativeLib=Shared -p:SelfContained=true -r ${{ matrix.flag }} -c release + + - name: 'Building the library into native for Linux & MacOS' + if: ${{ matrix.os != 'windows-latest' }} + run: | + cd Sources/Wrappers/AngouriMath.CPP.Exporting + mkdir ../AngouriMath.CPP.Importing/out-x64 + dotnet publish -p:NativeLib=Shared -p:SelfContained=true -r ${{ matrix.flag }} -c release + + - name: 'Preparing tests' + run: | + cd ${{ env.tests_path }} + cmake -S . -B build + mkdir "build/Debug" + + - name: 'Copying the library for Windows' + if: ${{ matrix.os == 'windows-latest' }} + run: | + cd ${{ env.tests_path }} + copy "..\..\..\Wrappers\AngouriMath.CPP.Importing\out-x64\AngouriMath.CPP.Exporting.dll" "build\Debug\AngouriMath.CPP.Exporting.dll" + + - name: 'Copying the library for Linux' + if: ${{ matrix.os == 'ubuntu-latest' }} + run: | + cd ${{ env.tests_path }} + cp -a "../../../Wrappers/AngouriMath.CPP.Importing/out-x64/AngouriMath.CPP.Exporting.so" "build/Debug/libAngouriMath.CPP.Exporting.so" + + - name: 'Copying the library for OS X' + if: ${{ matrix.os == 'macos-latest' }} + run: | + cd ${{ env.tests_path }} + cp -r "../../../Wrappers/AngouriMath.CPP.Importing/out-x64/AngouriMath.CPP.Exporting.dylib" "build/Debug/libAngouriMath.CPP.Exporting.dylib" + cd build/Debug + path=$(pwd) + install_name_tool -id "$path/libAngouriMath.CPP.Exporting.dylib" libAngouriMath.CPP.Exporting.dylib + + - name: 'Building and running tests' + run: | + cd ${{ env.tests_path }} + cmake --build build + cd build + ctest + + - name: 'Upload logs' + if: failure() + uses: actions/upload-artifact@v2 + with: + name: Logs + path: ./Sources/Tests/CPPWrapperUnitTests/tests/build/Testing/Temporary diff --git a/.github/workflows/reserves/math/CSharpBuild.yml b/.github/workflows/reserves/math/CSharpBuild.yml new file mode 100644 index 000000000000..a927b62cee77 --- /dev/null +++ b/.github/workflows/reserves/math/CSharpBuild.yml @@ -0,0 +1,30 @@ +name: 'C#/Kernel Build' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + AngouriMath: + runs-on: ${{ matrix.os }} + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + include-prerelease: true + + - name: Build AngouriMath + run: | + cd Sources/AngouriMath/AngouriMath + dotnet build diff --git a/.github/workflows/reserves/math/CSharpTest.yml b/.github/workflows/reserves/math/CSharpTest.yml new file mode 100644 index 000000000000..3e2619dc368b --- /dev/null +++ b/.github/workflows/reserves/math/CSharpTest.yml @@ -0,0 +1,75 @@ +name: 'C# Test' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Test: + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + + - name: Setup .NET Core 3 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 3.1.202 + + - name: Setup .NET Core 5 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 5.0.202 + + - name: Setup .NET 7 # The dotnet CLI above is replaced by this but the runtime is kept + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + include-prerelease: true + + - name: Build Tests # We need it to be able to retry + run: | + dotnet add "Sources/Tests/UnitTests" package GitHubActionsTestLogger + dotnet build Sources/Tests/UnitTests + + - name: Test if not to send + if: ${{ matrix.os != 'windows-latest' }} + run: | + dotnet test "Sources/Tests/UnitTests" -c Release -l GitHubActions /p:CollectCoverage=true /p:CoverletOutputFormat=opencover + + - name: Test if to send + if: ${{ matrix.os == 'windows-latest' }} + run: | + cd Sources/Tests/UnitTests + dotnet test --collect:"XPlat Code Coverage" -- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover + cd ../../Utils/Utils + dotnet run CopyCovReport + + - name: Info about the coverage report + if: ${{ matrix.os == 'windows-latest' }} + run: dir ./Sources/Tests/UnitTests/coverage.opencover.xml + + - name: 'Upload report as artifact' + if: ${{ matrix.os == 'windows-latest' }} + uses: actions/upload-artifact@v2 + with: + name: Report + path: ./Sources/Tests/UnitTests/coverage.opencover.xml + retention-days: 3 + + - name: 'Send to codecov' + if: ${{ matrix.os == 'windows-latest' }} + uses: codecov/codecov-action@v1 + with: + file: ./Sources/Tests/UnitTests/coverage.opencover.xml diff --git a/.github/workflows/reserves/math/EverythingBuild.yml b/.github/workflows/reserves/math/EverythingBuild.yml new file mode 100644 index 000000000000..af7f7e11794f --- /dev/null +++ b/.github/workflows/reserves/math/EverythingBuild.yml @@ -0,0 +1,39 @@ +name: 'Solution Build' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Everything: + runs-on: windows-latest # Window application cannot be run on other OS + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: Build AM + run: dotnet build Sources/AngouriMath/AngouriMath/AngouriMath.csproj + + - name: Build FSharp + run: dotnet build Sources/Wrappers/AngouriMath.FSharp/AngouriMath.FSharp.fsproj + + - name: Build Interactive + run: dotnet build Sources/Wrappers/AngouriMath.Interactive/AngouriMath.Interactive.fsproj + + - name: Build Analyzers + run: dotnet build Sources/Analyzers/Analyzers/Analyzers.csproj + + - name: Build Utils + run: dotnet build Sources/Utils/Utils/Utils.csproj diff --git a/.github/workflows/reserves/math/FSharpBuild.yml b/.github/workflows/reserves/math/FSharpBuild.yml new file mode 100644 index 000000000000..bd26da370d64 --- /dev/null +++ b/.github/workflows/reserves/math/FSharpBuild.yml @@ -0,0 +1,38 @@ +name: 'F# Build' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Test: + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET Core 3.1 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 3.1.302 + + - name: Setup .NET 7 # The dotnet CLI above is replaced by this but the runtime is kept + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: Build + run: | + cd Sources/Wrappers + dotnet build AngouriMath.FSharp diff --git a/.github/workflows/reserves/math/FSharpTest.yml b/.github/workflows/reserves/math/FSharpTest.yml new file mode 100644 index 000000000000..4d89b32c43f8 --- /dev/null +++ b/.github/workflows/reserves/math/FSharpTest.yml @@ -0,0 +1,43 @@ +name: 'F# Test' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Test: + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 5.0.202 + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: Build Tests # We need it to be able to retry + run: | + dotnet add "Sources/Tests/FSharpWrapperUnitTests" package Microsoft.NET.Test.Sdk # Update is required for GitHubActionsTestLogger to print anything + dotnet add "Sources/Tests/FSharpWrapperUnitTests" package GitHubActionsTestLogger + dotnet build Sources/Tests/FSharpWrapperUnitTests + + - name: Test + run: | + dotnet test "Sources/Tests/FSharpWrapperUnitTests" -c Release -l GitHubActions diff --git a/.github/workflows/reserves/math/IntegrationTest.yml b/.github/workflows/reserves/math/IntegrationTest.yml new file mode 100644 index 000000000000..523befdd20e1 --- /dev/null +++ b/.github/workflows/reserves/math/IntegrationTest.yml @@ -0,0 +1,116 @@ +name: 'Integration Test' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +env: + package_version: 99.99.99 + packed_path: D:\a\AngouriMath\AngouriMath\Sources\AngouriMath\AngouriMath\bin\release + + library_version: 33.33.33 + library_packed_path: D:\a\AngouriMath\AngouriMath\testlib\bin\release + + +jobs: + ConsoleApp: + runs-on: windows-latest + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: 'Pack AngouriMath' + run: | + cd Sources/AngouriMath/AngouriMath + dotnet restore AngouriMath.csproj + dotnet build AngouriMath.csproj -c release + dotnet pack AngouriMath.csproj -c release -p:PackageVersion=${{ env.package_version }} + + - name: 'Create hello world' + run: | + mkdir IntTest + cd IntTest + dotnet new console + "using System; using AngouriMath; Console.WriteLine(MathS.I_3);" | Out-File -FilePath Program.cs + + - name: 'Add it as a source' + run: | + cd IntTest + dotnet new nugetconfig + dotnet nuget add source --configfile nuget.config "${{ env.packed_path }}" + dir ${{ env.packed_path }} + + - name: 'Add AngouriMath as a package' + run: | + cd IntTest + dotnet add package AngouriMath -v ${{ env.package_version }} + dotnet restore + + - name: 'Build and run' + run: | + cd IntTest + dotnet build + dotnet run + + ClassLibConsoleApp: + runs-on: windows-latest + steps: + - uses: actions/checkout@v2 + - uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: 'Pack AngouriMath' + run: | + cd Sources/AngouriMath/AngouriMath + dotnet restore AngouriMath.csproj + dotnet build AngouriMath.csproj -c release + dotnet pack AngouriMath.csproj -c release -p:PackageVersion=${{ env.package_version }} + + - name: 'Create library' + run: | + dotnet new classlib --name testlib --framework netstandard2.0 + cd testlib + "using AngouriMath; namespace TestLib { public class TestType { public static void TestMethod() => System.Console.WriteLine(MathS.I_3); } }" | Out-File -FilePath Class1.cs + + - name: 'Add AngouriMath to library' + run: | + cd testlib + dotnet new nugetconfig + dotnet nuget add source --configfile nuget.config "${{ env.packed_path }}" + dotnet add package AngouriMath -v ${{ env.package_version }} + dotnet restore + + - name: 'Pack library' + run: | + cd testlib + dotnet pack -c release -p:PackageVersion=${{ env.library_version }} + + - name: 'Create console app' + run: | + dotnet new console --name testconsole --framework net7.0 + cd testconsole + "using TestLib; TestType.TestMethod();" | Out-File -FilePath Program.cs + + - name: 'Add library to console app' + run: | + cd testconsole + dotnet new nugetconfig + dotnet nuget add source --configfile nuget.config "${{ env.library_packed_path }}" + dotnet add package testlib -v ${{ env.library_version }} + dotnet restore + + - name: 'Build and run app' + run: | + cd testconsole + dotnet build + dotnet run + diff --git a/.github/workflows/reserves/math/InteractiveBuild.yml b/.github/workflows/reserves/math/InteractiveBuild.yml new file mode 100644 index 000000000000..fd5fb7fa8794 --- /dev/null +++ b/.github/workflows/reserves/math/InteractiveBuild.yml @@ -0,0 +1,38 @@ +name: 'Interactive Build' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Test: + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET Core 3.1 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 3.1.302 + + - name: Setup .NET 7 # The dotnet CLI above is replaced by this but the runtime is kept + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: Build + run: | + cd Sources/Wrappers + dotnet build AngouriMath.Interactive diff --git a/.github/workflows/reserves/math/InteractiveTest.yml b/.github/workflows/reserves/math/InteractiveTest.yml new file mode 100644 index 000000000000..2bff082eac42 --- /dev/null +++ b/.github/workflows/reserves/math/InteractiveTest.yml @@ -0,0 +1,45 @@ +name: 'Interactive Test' + +on: + push: + branches: + - master + pull_request: + branches: + - '*' + +jobs: + Test: + strategy: + matrix: + os: [windows-latest, ubuntu-latest, macos-latest] + + runs-on: ${{ matrix.os }} + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET Core 5 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: 5.0.202 + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + + - name: Build Tests # We need it to be able to retry + run: | + dotnet build Sources/Tests/InteractiveWrapperUnitTests -c Release + dotnet build Sources/Tests/TerminalUnitTests -c Release + + - name: Test Interactive Wrapper + run: | + dotnet test "Sources/Tests/InteractiveWrapperUnitTests" -c Release + + - name: Test Terminal Lib + run: | + dotnet test "Sources/Tests/TerminalUnitTests" -c Release diff --git a/.github/workflows/reserves/math/Nuget.yml b/.github/workflows/reserves/math/Nuget.yml new file mode 100644 index 000000000000..e06b46306cab --- /dev/null +++ b/.github/workflows/reserves/math/Nuget.yml @@ -0,0 +1,57 @@ +name: 'Upload last-master versions to MyGet' + +on: + push: + branches: + - master + +jobs: + main: + runs-on: ubuntu-latest + steps: + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.100' + include-prerelease: true + + - uses: actions/checkout@v2 + + - name: 'Pack AngouriMath' + run: | + cd Sources + + # versioning + commithash=$(git rev-parse --short HEAD) + currtime=$(date +%s) + echo "commit hash is $commithash" + echo "time is $currtime" + name=10.0.0-master-$currtime-$commithash + echo "name is $name" + + # AngouriMath + cd AngouriMath/AngouriMath + dotnet restore AngouriMath.csproj + dotnet build AngouriMath.csproj -c release + dotnet pack AngouriMath.csproj -c release -p:PackageVersion=$name + cd bin/release + dotnet nuget push AngouriMath.$name.nupkg --api-key ${{ secrets.MYGET_KEY }} --source "myget" + cd ../../../.. + + # AngouriMath.FSharp + cd Wrappers/AngouriMath.FSharp + dotnet restore + dotnet build -c Release + dotnet pack -c Release -p:PackageVersion=$name + cd bin/Release + dotnet nuget push AngouriMath.FSharp.$name.nupkg --api-key ${{ secrets.MYGET_KEY }} --source "myget" + cd ../../../.. + + # AngouriMath.Interactive + cd Wrappers/AngouriMath.Interactive + dotnet restore + dotnet build -c Release + dotnet pack -c Release -p:PackageVersion=$name + cd bin/Release + dotnet nuget push AngouriMath.Interactive.$name.nupkg --api-key ${{ secrets.MYGET_KEY }} --source "myget" + cd ../../../.. diff --git a/.github/workflows/reserves/math/TerminalNightly.yml b/.github/workflows/reserves/math/TerminalNightly.yml new file mode 100644 index 000000000000..29e5f1d08b80 --- /dev/null +++ b/.github/workflows/reserves/math/TerminalNightly.yml @@ -0,0 +1,61 @@ +name: 'Terminal nightly builds' + +on: + push: + branches: + - master + paths: + - 'Sources/Terminal/VERSION/**' + +jobs: + TerminalBuild: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + with: + submodules: 'recursive' + + - name: Setup .NET 7 + uses: actions/setup-dotnet@v1 + with: + dotnet-version: '7.0.x' + include-prerelease: true + + - name: 'Build terminal' + run: | + cd Sources/Terminal/AngouriMath.Terminal + ./publish.sh + ./pack-deb-linux.sh + + - name: 'Pack' + run: | + for arch in linux-x64 linux-arm linux-arm64 win-x64 win-x86 osx-x64 osx.12-arm64 + do + zip -r "angourimath-terminal-${arch}.zip" ./Sources/Terminal/AngouriMath.Terminal/publish-output/$arch + done + + - name: 'Release' + run: | + name=$(cat ./Sources/Terminal/VERSION/VERSION) + echo "Version: $name" + + cp ./Sources/Terminal/AngouriMath.Terminal/publish-output/angourimath-terminal-*.deb . + + echo ${{ secrets.LAB_ACCESS_TOKEN }} > token.txt + gh auth login --with-token < token.txt + gh release create v$name \ + 'angourimath-terminal-linux-x64.zip' \ + 'angourimath-terminal-linux-arm.zip' \ + 'angourimath-terminal-linux-arm64.zip' \ + 'angourimath-terminal-win-x64.zip' \ + 'angourimath-terminal-win-x86.zip' \ + 'angourimath-terminal-osx-x64.zip' \ + 'angourimath-terminal-osx.12-arm64.zip' \ + 'angourimath-terminal-amd64.deb' \ + 'angourimath-terminal-arm.deb' \ + 'angourimath-terminal-arm64.deb' \ + -R asc-community/AngouriMathLab \ + -t 'AngouriMath.Terminal' \ + -n "Expand 'assets' and choose the one for your OS" + diff --git a/.github/workflows/reserves/matrix-go.yml b/.github/workflows/reserves/matrix-go.yml new file mode 100644 index 000000000000..152c7918ccc0 --- /dev/null +++ b/.github/workflows/reserves/matrix-go.yml @@ -0,0 +1,128 @@ +name: distribution + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + push: + branches: + - 'main' + - 'release/*' + tags: + - 'v*' + pull_request: + +env: + DOCKERHUB_SLUG: distribution/distribution + +jobs: + test: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + go: + - 1.18 + - 1.19 + steps: + - + name: Checkout + uses: actions/checkout@v3 + - + name: Set up Go + uses: actions/setup-go@v3 + with: + go-version: ${{ matrix.go }} + - + name: Test + run: | + make coverage + - + name: Codecov + uses: codecov/codecov-action@v3 + with: + directory: ./ + + build: + runs-on: ubuntu-latest + needs: + - test + steps: + - + name: Checkout + uses: actions/checkout@v3 + with: + fetch-depth: 0 + - + name: Docker meta + id: meta + uses: docker/metadata-action@v4 + with: + images: | + ${{ env.DOCKERHUB_SLUG }} + ### versioning strategy + ### push semver tag v3.2.1 on main (default branch) + # distribution/distribution:3.2.1 + # distribution/distribution:3.2 + # distribution/distribution:3 + # distribution/distribution:latest + ### push semver prelease tag v3.0.0-beta.1 on main (default branch) + # distribution/distribution:3.0.0-beta.1 + ### push on main + # distribution/distribution:edge + tags: | + type=semver,pattern={{version}} + type=semver,pattern={{major}}.{{minor}} + type=semver,pattern={{major}} + type=ref,event=pr + type=edge + labels: | + org.opencontainers.image.title=Distribution + org.opencontainers.image.description=The toolkit to pack, ship, store, and deliver container content + - + name: Set up Docker Buildx + uses: docker/setup-buildx-action@v2 + - + name: Login to DockerHub + if: github.event_name != 'pull_request' + uses: docker/login-action@v2 + with: + username: ${{ secrets.DOCKERHUB_USERNAME }} + password: ${{ secrets.DOCKERHUB_TOKEN }} + - + name: Build artifacts + uses: docker/bake-action@v2 + with: + targets: artifact-all + - + name: Move artifacts + run: | + mv ./bin/**/* ./bin/ + - + name: Upload artifacts + uses: actions/upload-artifact@v3 + with: + name: registry + path: ./bin/* + if-no-files-found: error + - + name: Build image + uses: docker/bake-action@v2 + with: + files: | + ./docker-bake.hcl + ${{ steps.meta.outputs.bake-file }} + targets: image-all + push: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }} + - + name: GitHub Release + uses: softprops/action-gh-release@v1 + if: startsWith(github.ref, 'refs/tags/') + with: + draft: true + files: | + bin/*.tar.gz + bin/*.sha256 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/matrix-ruby.yml b/.github/workflows/reserves/matrix-ruby.yml new file mode 100644 index 000000000000..608a834baad9 --- /dev/null +++ b/.github/workflows/reserves/matrix-ruby.yml @@ -0,0 +1,90 @@ +name: Build +on: + - push + - pull_request + +jobs: + build: + name: Ruby ${{ matrix.ruby }} / Rails ${{ matrix.rails }} + if: github.event_name == 'push' || github.event.pull_request.head.repo.full_name != github.repository + strategy: + fail-fast: false + matrix: + ruby: + - jruby-9.2.16.0 + - truffleruby + - "3.1" + - "3.0" + - "2.7" + - "2.6" + - "2.5" + rails: + - "5.0" + - "5.1" + - "5.2" + - "6.0" + - "6.1" + - "7.0" + - main + exclude: + - ruby: 2.5 + rails: "7.0" + - ruby: 2.5 + rails: main + - ruby: 2.6 + rails: "7.0" + - ruby: 2.6 + rails: main + - ruby: jruby-9.2.16.0 + rails: "7.0" + - ruby: jruby-9.2.16.0 + rails: main + - ruby: "3.0" + rails: "5.0" + - ruby: "3.0" + rails: "5.1" + - ruby: "3.0" + rails: "5.2" + - ruby: truffleruby + rails: "5.0" + - ruby: truffleruby + rails: "5.1" + - ruby: truffleruby + rails: "5.2" + - ruby: "3.1" + rails: "5.0" + - ruby: "3.1" + rails: "5.1" + - ruby: "3.1" + rails: "5.2" + - ruby: "3.1" + rails: "6.0" + + runs-on: 'ubuntu-latest' + + env: + BUNDLE_GEMFILE: gemfiles/${{ matrix.rails }}.gemfile + RUBYOPT: "--disable-error_highlight" + + steps: + - uses: actions/checkout@v2 + - uses: ruby/setup-ruby@v1 + with: + ruby-version: ${{ matrix.ruby }} + - name: Setup project + run: bundle install + - name: Run test + run: bundle exec rake all_specs + + standard: + name: Run standard + runs-on: 'ubuntu-latest' + steps: + - uses: actions/checkout@v3 + - uses: ruby/setup-ruby@v1 + with: + ruby-version: "3.1" + - name: Setup project + run: bundle install + - name: Run test + run: bundle exec rake standard diff --git a/.github/workflows/reserves/no-response.yml b/.github/workflows/reserves/no-response.yml new file mode 100644 index 000000000000..7193eaa3b20b --- /dev/null +++ b/.github/workflows/reserves/no-response.yml @@ -0,0 +1,13 @@ +# Configuration for probot-no-response - https://github.com/probot/no-response + +# Number of days of inactivity before an Issue is closed for lack of response +daysUntilClose: 14 +# Label requiring a response +responseRequiredLabel: more-information-needed +# Comment to post when closing an Issue for lack of response. Set to `false` to disable +closeComment: > + This issue has been automatically closed because there has been no response + to our request for more information from the original author. With only the + information that is currently in the issue, we don't have enough information + to take action. Please reach out if you have or find the answers we need so + that we can investigate further. diff --git a/.github/workflows/reserves/project-euler/build.yml b/.github/workflows/reserves/project-euler/build.yml new file mode 100644 index 000000000000..a113b4608678 --- /dev/null +++ b/.github/workflows/reserves/project-euler/build.yml @@ -0,0 +1,34 @@ +name: "build" + +on: + pull_request: + schedule: + - cron: "0 0 * * *" # Run everyday + +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: 3.12 + allow-prereleases: true + - uses: actions/cache@v4 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('requirements.txt') }} + - name: Install dependencies + run: | + python -m pip install --upgrade pip setuptools six wheel + python -m pip install pytest-cov -r requirements.txt + - name: Run tests + # TODO: #8818 Re-enable quantum tests + run: pytest + --ignore=quantum/q_fourier_transform.py + --ignore=project_euler/ + --ignore=scripts/validate_solutions.py + --cov-report=term-missing:skip-covered + --cov=. . + - if: ${{ success() }} + run: scripts/build_directory_md.py 2>&1 | tee DIRECTORY.md diff --git a/.github/workflows/reserves/project-euler/directory_writer.yml b/.github/workflows/reserves/project-euler/directory_writer.yml new file mode 100644 index 000000000000..55d89f455a25 --- /dev/null +++ b/.github/workflows/reserves/project-euler/directory_writer.yml @@ -0,0 +1,25 @@ +# The objective of this GitHub Action is to update the DIRECTORY.md file (if needed) +# when doing a git push +name: directory_writer +on: [push] +jobs: + build: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: 3.x + - name: Write DIRECTORY.md + run: | + scripts/build_directory_md.py 2>&1 | tee DIRECTORY.md + git config --global user.name "$GITHUB_ACTOR" + git config --global user.email "$GITHUB_ACTOR@users.noreply.github.com" + git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/$GITHUB_REPOSITORY + - name: Update DIRECTORY.md + run: | + git add DIRECTORY.md + git commit -am "updating DIRECTORY.md" || true + git push --force origin HEAD:$GITHUB_REF || true diff --git a/.github/workflows/reserves/project-euler/project_euler.yml b/.github/workflows/reserves/project-euler/project_euler.yml new file mode 100644 index 000000000000..59e1208a650d --- /dev/null +++ b/.github/workflows/reserves/project-euler/project_euler.yml @@ -0,0 +1,39 @@ +on: + pull_request: + # Run only if a file is changed within the project_euler directory and related files + paths: + - "project_euler/**" + - ".github/workflows/project_euler.yml" + - "scripts/validate_solutions.py" + schedule: + - cron: "0 0 * * *" # Run everyday + +name: "Project Euler" + +jobs: + project-euler: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: 3.x + - name: Install pytest and pytest-cov + run: | + python -m pip install --upgrade pip + python -m pip install --upgrade numpy pytest pytest-cov + - run: pytest --doctest-modules --cov-report=term-missing:skip-covered --cov=project_euler/ project_euler/ + validate-solutions: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: 3.x + - name: Install pytest and requests + run: | + python -m pip install --upgrade pip + python -m pip install --upgrade numpy pytest requests + - run: pytest scripts/validate_solutions.py + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.github/workflows/reserves/project-euler/ruff.yml b/.github/workflows/reserves/project-euler/ruff.yml new file mode 100644 index 000000000000..d354eba672ae --- /dev/null +++ b/.github/workflows/reserves/project-euler/ruff.yml @@ -0,0 +1,16 @@ +# https://beta.ruff.rs +name: ruff +on: + push: + branches: + - master + pull_request: + branches: + - master +jobs: + ruff: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - run: pip install --user ruff + - run: ruff check --output-format=github . diff --git a/.github/workflows/reserves/release-new-action-version.yml b/.github/workflows/reserves/release-new-action-version.yml new file mode 100644 index 000000000000..e584e46f6479 --- /dev/null +++ b/.github/workflows/reserves/release-new-action-version.yml @@ -0,0 +1,29 @@ +name: Release new action version + +on: + release: + types: [released] + workflow_dispatch: + inputs: + TAG_NAME: + description: 'Tag name that the major tag will point to' + required: true + +env: + TAG_NAME: ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} +permissions: + contents: write + +jobs: + update_tag: + name: Update the major tag to include the ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} changes + environment: + name: releaseNewActionVersion + runs-on: ubuntu-latest + steps: + - name: Update the ${{ env.TAG_NAME }} tag + id: update-major-tag + uses: actions/publish-action@v0.2.2 + with: + source-tag: ${{ env.TAG_NAME }} + slack-webhook: ${{ secrets.SLACK_WEBHOOK }} diff --git a/.github/workflows/reserves/settings.yml b/.github/workflows/reserves/settings.yml new file mode 100644 index 000000000000..7c1b11592e1f --- /dev/null +++ b/.github/workflows/reserves/settings.yml @@ -0,0 +1,31 @@ +# Repository settings set via https://github.com/probot/settings + +repository: + has_issues: true + has_wiki: false + has_projects: false + has_downloads: false + +labels: + - name: help wanted + oldname: help-wanted + color: 0e8a16 + - name: more-information-needed + color: d93f0b + - name: bug + color: b60205 + - name: feature + color: 1d76db + - name: good first issue + color: "5319e7" + +# Not currently implemented by probot/settings, but manually implemented in script/deploy +branch_protection: + restrictions: null + enforce_admins: false + required_status_checks: + strict: true + contexts: + - "script/cibuild" # GitHub Actions CI workflow + required_pull_request_reviews: + require_code_owner_reviews: true diff --git a/.github/workflows/reserves/setup-dotnet/basic-validation.yml b/.github/workflows/reserves/setup-dotnet/basic-validation.yml new file mode 100644 index 000000000000..4bcf16ca53bb --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/basic-validation.yml @@ -0,0 +1,19 @@ +name: Basic validation + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + call-basic-validation: + name: Basic validation + uses: actions/reusable-workflows/.github/workflows/basic-validation.yml@main + with: + node-version: '20' diff --git a/.github/workflows/reserves/setup-dotnet/check-dist.yml b/.github/workflows/reserves/setup-dotnet/check-dist.yml new file mode 100644 index 000000000000..30aa2210da5f --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/check-dist.yml @@ -0,0 +1,19 @@ +name: Check dist/ + +on: + push: + branches: + - main + paths-ignore: + - '**.md' + pull_request: + paths-ignore: + - '**.md' + workflow_dispatch: + +jobs: + call-check-dist: + name: Check dist/ + uses: actions/reusable-workflows/.github/workflows/check-dist.yml@main + with: + node-version: '20' diff --git a/.github/workflows/reserves/setup-dotnet/codeql-analysis.yml b/.github/workflows/reserves/setup-dotnet/codeql-analysis.yml new file mode 100644 index 000000000000..7a82612387a6 --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/codeql-analysis.yml @@ -0,0 +1,14 @@ +name: CodeQL analysis + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: '0 3 * * 0' + +jobs: + call-codeQL-analysis: + name: CodeQL analysis + uses: actions/reusable-workflows/.github/workflows/codeql-analysis.yml@main diff --git a/.github/workflows/reserves/setup-dotnet/e2e-tests.yml b/.github/workflows/reserves/setup-dotnet/e2e-tests.yml new file mode 100644 index 000000000000..332eebccb62f --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/e2e-tests.yml @@ -0,0 +1,483 @@ +name: e2e tests + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + test-setup-multiple-versions: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 2.2.402, 3.1.404 and 3.0.x + uses: ./ + with: + dotnet-version: | + 2.2.402 + 3.1.404 + 3.0.x + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2.402$", "^3.1.404$", "^3.0" + + test-setup-full-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # Side-by-side install of 2.2 and 3.1 used for the test project + - name: Setup dotnet 2.2.402 + uses: ./ + with: + dotnet-version: 2.2.402 + - name: Setup dotnet 3.1.201 + uses: ./ + with: + dotnet-version: 3.1.201 + # We are including this variable to force the generation of the nuget config file to verify that it is created in the correct place + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1.201$", "^2.2.402$" -CheckNugetConfig + + test-setup-without-patch-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # 2.0, 3.0, 5.0 needs to be in single quotes to interpret as a string instead of as an integer + - name: Setup dotnet '3.1' + uses: ./ + with: + dotnet-version: '3.1' + - name: Setup dotnet '2.2' + uses: ./ + with: + dotnet-version: '2.2' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1", "^2.2" + + test-setup-prerelease-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet '3.1.100-preview1-014459' + uses: ./ + with: + dotnet-version: '3.1.100-preview1-014459' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "3.1.100-preview1-014459" + + test-setup-latest-patch-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.x + uses: ./ + with: + dotnet-version: 3.1.x + - name: Setup dotnet 2.2.X + uses: ./ + with: + dotnet-version: 2.2.X + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2", "^3.1" + + test-ABCxx-syntax: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.4xx + uses: ./ + with: + dotnet-version: '6.0.4xx' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^6\.0\.4\d{2}" + + test-setup-with-wildcard: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.* + uses: ./ + with: + dotnet-version: 3.1.* + - name: Setup dotnet 2.2.* + uses: ./ + with: + dotnet-version: 2.2.* + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1", "^2.2" + + test-setup-global-json-specified-and-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '{"sdk":{"version": "2.2.207","rollForward": "latestFeature"}}' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + dotnet-version: 3.1 + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2", "^3.1" + + test-setup-global-json-only: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '{"sdk":{"version": "2.2.207","rollForward": "latestFeature"}}' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2" + + test-global-json-with-comments: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Write global.json + shell: bash + run: | + mkdir subdirectory + echo '/* should support comments */ {"sdk":{"version": "2.2.207","rollForward": "latestFeature"}} // should support comments' > ./subdirectory/global.json + - name: Setup dotnet + uses: ./ + with: + global-json-file: ./subdirectory/global.json + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^2.2" + + test-setup-with-dotnet-quality: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 7.0 with preview quality + uses: ./ + with: + dotnet-version: '7.0' + dotnet-quality: 'preview' + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^7\.0\.\d+-" + + test-setup-with-cache: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macos-latest] + env: + NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Copy NuGet lock file to root + shell: bash + run: cp ./__tests__/e2e-test-csproj/packages.lock.json ./packages.lock.json + - name: Setup .NET Core 3.1 + id: setup-dotnet + uses: ./ + with: + dotnet-version: 3.1 + cache: true + - name: Verify Cache + if: steps.setup-dotnet.outputs.cache-hit == 'true' + shell: bash + run: if [[ -e ${NUGET_PACKAGES} ]]; then exit 0; else exit 1; fi + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1" + + test-setup-with-cache-dependency-path: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macos-latest] + env: + NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup .NET Core 3.1 + id: setup-dotnet + uses: ./ + with: + dotnet-version: 3.1 + cache: true + cache-dependency-path: './__tests__/e2e-test-csproj/packages.lock.json' + - name: Verify Cache + if: steps.setup-dotnet.outputs.cache-hit == 'true' + shell: bash + run: if [[ -e ${NUGET_PACKAGES} ]]; then exit 0; else exit 1; fi + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1" + + test-dotnet-version-output-during-single-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.401 + uses: ./ + id: step1 + with: + dotnet-version: '6.0.401' + + - name: Verify value of the dotnet-version output + shell: pwsh + run: | + $version = & dotnet --version + Write-Host "Installed version: $version" + if (-not ($version -eq '${{steps.step1.outputs.dotnet-version}}')) { throw "Unexpected output value" } + + test-dotnet-version-output-during-multiple-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + + - name: Setup dotnet 6.0.401, 5.0.408, 7.0.100-rc.1.22431.12 + uses: ./ + id: step2 + with: + dotnet-version: | + 7.0.100-rc.1.22431.12 + 6.0.401 + 5.0.408 + + - name: Verify value of the dotnet-version output + shell: pwsh + run: | + $version = "7.0.100-rc.1.22431.12" + if (-not ($version -eq '${{steps.step2.outputs.dotnet-version}}')) { throw "Unexpected output value" } + + test-proxy: + runs-on: ubuntu-latest + container: + image: ubuntu:latest + options: --dns 127.0.0.1 + services: + squid-proxy: + image: ubuntu/squid:latest + ports: + - 3128:3128 + env: + https_proxy: http://squid-proxy:3128 + http_proxy: http://squid-proxy:3128 + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Install Powershell + run: | + apt-get update + apt-get install -y wget apt-transport-https software-properties-common + wget -q "https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/packages-microsoft-prod.deb" + dpkg -i packages-microsoft-prod.deb + rm packages-microsoft-prod.deb + apt-get update + apt-get install -y powershell + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 6.0 + uses: ./ + with: + dotnet-version: 6.0 + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: | + __tests__/verify-dotnet.ps1 -Patterns "^6.0" -CheckNugetConfig + + test-bypass-proxy: + runs-on: ubuntu-latest + env: + https_proxy: http://no-such-proxy:3128 + no_proxy: github.com,dotnetcli.blob.core.windows.net,download.visualstudio.microsoft.com,api.nuget.org,dotnetcli.azureedge.net + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet 3.1.201 + uses: ./ + with: + dotnet-version: 3.1.201 + source-url: https://api.nuget.org/v3/index.json + env: + NUGET_AUTH_TOKEN: NOTATOKEN + - name: Verify dotnet + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^3.1.201$" -CheckNugetConfig + + test-sequential-version-installation: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + lower-version: ['3.1.426'] + higher-version: ['7.0.203'] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + # Install one version, use it for something, then switch to next version + - name: Setup dotnet (lower version) + uses: ./ + with: + dotnet-version: ${{ matrix.lower-version }} + - name: Verify dotnet (lower version) + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.lower-version }}$" + - name: Setup dotnet (higher version) + uses: ./ + with: + dotnet-version: ${{ matrix.higher-version }} + - name: Verify dotnet (higher version) + shell: pwsh + run: __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.lower-version }}$", "^${{ matrix.higher-version }}$" diff --git a/.github/workflows/reserves/setup-dotnet/licensed.yml b/.github/workflows/reserves/setup-dotnet/licensed.yml new file mode 100644 index 000000000000..37f1560c3fe6 --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/licensed.yml @@ -0,0 +1,15 @@ +name: Licensed + +on: + push: + branches: + - main + pull_request: + branches: + - main + workflow_dispatch: + +jobs: + call-licensed: + name: Licensed + uses: actions/reusable-workflows/.github/workflows/licensed.yml@main diff --git a/.github/workflows/reserves/setup-dotnet/release-new-action-version.yml b/.github/workflows/reserves/setup-dotnet/release-new-action-version.yml new file mode 100644 index 000000000000..e584e46f6479 --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/release-new-action-version.yml @@ -0,0 +1,29 @@ +name: Release new action version + +on: + release: + types: [released] + workflow_dispatch: + inputs: + TAG_NAME: + description: 'Tag name that the major tag will point to' + required: true + +env: + TAG_NAME: ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} +permissions: + contents: write + +jobs: + update_tag: + name: Update the major tag to include the ${{ github.event.inputs.TAG_NAME || github.event.release.tag_name }} changes + environment: + name: releaseNewActionVersion + runs-on: ubuntu-latest + steps: + - name: Update the ${{ env.TAG_NAME }} tag + id: update-major-tag + uses: actions/publish-action@v0.2.2 + with: + source-tag: ${{ env.TAG_NAME }} + slack-webhook: ${{ secrets.SLACK_WEBHOOK }} diff --git a/.github/workflows/reserves/setup-dotnet/test-dotnet.yml b/.github/workflows/reserves/setup-dotnet/test-dotnet.yml new file mode 100644 index 000000000000..03b9a294bc60 --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/test-dotnet.yml @@ -0,0 +1,35 @@ +name: Validate dotnet + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + setup-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + dotnet-version: ['2.1', '2.2', '3.0', '3.1', '5.0', '6.0', '7.0', '8.0'] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet ${{ matrix.dotnet-version }} + uses: ./ + with: + dotnet-version: ${{ matrix.dotnet-version }} + - name: Verify installed version + shell: pwsh + run: | + __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.dotnet-version }}" diff --git a/.github/workflows/reserves/setup-dotnet/update-config-files.yml b/.github/workflows/reserves/setup-dotnet/update-config-files.yml new file mode 100644 index 000000000000..87af500426bc --- /dev/null +++ b/.github/workflows/reserves/setup-dotnet/update-config-files.yml @@ -0,0 +1,11 @@ +name: Update configuration files + +on: + schedule: + - cron: '0 3 * * 0' + workflow_dispatch: + +jobs: + call-update-configuration-files: + name: Update configuration files + uses: actions/reusable-workflows/.github/workflows/update-config-files.yml@main diff --git a/.github/workflows/reserves/setup-gcloud/dependabot.yml b/.github/workflows/reserves/setup-gcloud/dependabot.yml new file mode 100644 index 000000000000..3a92eae4e726 --- /dev/null +++ b/.github/workflows/reserves/setup-gcloud/dependabot.yml @@ -0,0 +1,10 @@ +version: 2 +updates: + - package-ecosystem: 'npm' + directory: '/' + rebase-strategy: 'disabled' + schedule: + interval: 'daily' + commit-message: + prefix: 'security: ' + open-pull-requests-limit: 0 # only check security updates diff --git a/.github/workflows/reserves/setup-gcloud/draft-release.yml b/.github/workflows/reserves/setup-gcloud/draft-release.yml new file mode 100644 index 000000000000..6c47e88c3520 --- /dev/null +++ b/.github/workflows/reserves/setup-gcloud/draft-release.yml @@ -0,0 +1,24 @@ +name: 'Draft release' + +on: + workflow_dispatch: + inputs: + version_strategy: + description: 'Version strategy: The strategy to used to update the version based on semantic versioning (more info at https://semver.org/).' + required: true + default: 'patch' + type: 'choice' + options: + - 'major' + - 'minor' + - 'patch' + +jobs: + draft-release: + name: 'Draft release' + uses: 'google-github-actions/.github/.github/workflows/draft-release.yml@v0' + with: + version_strategy: '${{ github.event.inputs.version_strategy }}' + # secrets must be explicitly passed to reusable workflows https://docs.github.com/en/enterprise-cloud@latest/actions/using-workflows/reusing-workflows#using-inputs-and-secrets-in-a-reusable-workflow + secrets: + ACTIONS_BOT_TOKEN: '${{ secrets.ACTIONS_BOT_TOKEN }}' diff --git a/.github/workflows/reserves/setup-gcloud/integration.yml b/.github/workflows/reserves/setup-gcloud/integration.yml new file mode 100644 index 000000000000..efed5320e079 --- /dev/null +++ b/.github/workflows/reserves/setup-gcloud/integration.yml @@ -0,0 +1,125 @@ +name: 'Integration' + +on: + push: + branches: + - 'main' + - 'release/**/*' + pull_request: + branches: + - 'main' + - 'release/**/*' + workflow_dispatch: + +concurrency: + group: '${{ github.workflow }}-${{ github.head_ref || github.ref }}' + cancel-in-progress: true + +jobs: + integration: + if: ${{ github.event_name == 'push' || github.repository == github.event.pull_request.head.repo.full_name && github.actor != 'dependabot[bot]' }} + strategy: + fail-fast: false + matrix: + os: + - 'ubuntu-latest' + - 'windows-latest' + - 'macos-latest' + runs-on: '${{ matrix.os }}' + + permissions: + id-token: 'write' + + steps: + - uses: 'actions/checkout@v4' + + - uses: 'actions/setup-node@v4' + with: + node-version: '20.x' + + - name: 'Build' + run: 'npm ci && npm run build' + + # No installation + - name: 'Skip install' + uses: './' + with: + skip_install: true + + # Constraint installation + - name: 'Install constraint' + uses: './' + with: + version: '>= 1.0.0' + + # Default installation + - name: 'Install version' + uses: './' + with: + version: '374.0.0' + + # Latest installation + - name: 'Install latest' + uses: './' + with: + version: 'latest' + + # By default, there is no configuration + - name: 'Check defaults' + run: 'npm run integration' + env: + TEST_ACCOUNT: '(unset)' + TEST_PROJECT_ID: '(unset)' + + # Install components + - name: 'Install components' + uses: './' + with: + install_components: 'cloud-run-proxy' + + - name: 'Check components' + run: 'npm run integration' + env: + TEST_COMPONENTS: 'cloud-run-proxy' + + # Set a project ID + - name: 'Set project ID' + uses: './' + with: + project_id: '${{ vars.PROJECT_ID }}' + + - name: 'Check project ID' + run: 'npm run integration' + env: + TEST_PROJECT_ID: '${{ vars.PROJECT_ID }}' + + # Authenticate via WIF + - name: 'Authenticate via WIF' + uses: 'google-github-actions/auth@v2' + with: + workload_identity_provider: '${{ vars.WIF_PROVIDER_NAME }}' + service_account: '${{ vars.SERVICE_ACCOUNT_EMAIL }}' + + - name: 'Setup gcloud with WIF' + uses: './' + + - name: 'Check WIF authentication' + run: 'npm run integration' + env: + TEST_ACCOUNT: '${{ vars.SERVICE_ACCOUNT_EMAIL }}' + TEST_PROJECT_ID: '${{ vars.PROJECT_ID }}' + + # Authenticate via SAKE + - name: 'Authenticate via SAKE' + uses: 'google-github-actions/auth@v2' + with: + credentials_json: '${{ secrets.SERVICE_ACCOUNT_KEY_JSON }}' + + - name: 'Setup gcloud with SAKE' + uses: './' + + - name: 'Check SAKE authentication' + run: 'npm run integration' + env: + TEST_ACCOUNT: '${{ vars.SERVICE_ACCOUNT_EMAIL }}' + TEST_PROJECT_ID: '${{ vars.PROJECT_ID }}' diff --git a/.github/workflows/reserves/setup-gcloud/release.yml b/.github/workflows/reserves/setup-gcloud/release.yml new file mode 100644 index 000000000000..15eea04ad2ce --- /dev/null +++ b/.github/workflows/reserves/setup-gcloud/release.yml @@ -0,0 +1,13 @@ +name: 'Release' + +on: + push: + branches: + - 'main' + - 'release/**/*' + +jobs: + release: + if: "startsWith(github.event.head_commit.message, 'Release: v')" + name: 'Release' + uses: 'google-github-actions/.github/.github/workflows/release.yml@v0' diff --git a/.github/workflows/reserves/setup-gcloud/unit.yml b/.github/workflows/reserves/setup-gcloud/unit.yml new file mode 100644 index 000000000000..3585ca1064fd --- /dev/null +++ b/.github/workflows/reserves/setup-gcloud/unit.yml @@ -0,0 +1,49 @@ +name: 'Unit' + +on: + push: + branches: + - 'main' + - 'release/**/*' + pull_request: + branches: + - 'main' + - 'release/**/*' + workflow_dispatch: + +concurrency: + group: '${{ github.workflow }}-${{ github.head_ref || github.ref }}' + cancel-in-progress: true + +jobs: + unit: + strategy: + fail-fast: false + matrix: + os: + - 'ubuntu-latest' + - 'windows-latest' + - 'macos-latest' + runs-on: '${{ matrix.os }}' + + steps: + - uses: 'actions/checkout@v4' + + - uses: 'actions/setup-node@v4' + with: + node-version: '20.x' + + - name: 'npm ci' + run: 'npm ci' + + - name: 'npm build' + run: 'npm run build' + + - name: 'run lint' + run: 'npm run lint' + # There's no need to run the linter for each operating system, since it + # will find the same thing 3x and clog up the PR review. + if: ${{matrix.os == 'ubuntu-latest'}} + + - name: 'test' + run: 'npm test' diff --git a/.github/workflows/reserves/setup-wsl/check-action-typing.main.kts b/.github/workflows/reserves/setup-wsl/check-action-typing.main.kts new file mode 100755 index 000000000000..2631d1139813 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/check-action-typing.main.kts @@ -0,0 +1,53 @@ +#!/usr/bin/env kotlin + +/* + * Copyright 2020-2023 Björn Kautler + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:Import("workflow-with-copyright.main.kts") + +import io.github.typesafegithub.workflows.actions.actions.CheckoutV3 +import io.github.typesafegithub.workflows.actions.typesafegithub.GithubActionsTypingV1 +import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest +import io.github.typesafegithub.workflows.domain.triggers.PullRequest +import io.github.typesafegithub.workflows.domain.triggers.Push + +workflowWithCopyright( + name = "Check Action Typing", + on = listOf( + Push(), + PullRequest() + ), + sourceFile = __FILE__ +) { + job( + id = "check_action_typing", + name = "Check Action Typing", + runsOn = UbuntuLatest + ) { + run( + name = "Configure Git", + command = "git config --global core.autocrlf input" + ) + uses( + name = "Checkout", + action = CheckoutV3() + ) + uses( + name = "Check Action Typing", + action = GithubActionsTypingV1() + ) + } +} diff --git a/.github/workflows/reserves/setup-wsl/check-action-typing.yaml b/.github/workflows/reserves/setup-wsl/check-action-typing.yaml new file mode 100644 index 000000000000..e8afd04201a4 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/check-action-typing.yaml @@ -0,0 +1,51 @@ +# Copyright 2020-2023 Björn Kautler +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file was generated using Kotlin DSL (.github/workflows/check-action-typing.main.kts). +# If you want to modify the workflow, please change the Kotlin file and regenerate this YAML file. +# Generated with https://github.com/typesafegithub/github-workflows-kt + +name: 'Check Action Typing' +on: + push: {} + pull_request: {} +jobs: + check_yaml_consistency: + name: 'Check YAML consistency' + runs-on: 'ubuntu-latest' + steps: + - id: 'step-0' + name: 'Check out' + uses: 'actions/checkout@v4' + - id: 'step-1' + name: 'Execute script' + run: 'rm ''.github/workflows/check-action-typing.yaml'' && ''.github/workflows/check-action-typing.main.kts''' + - id: 'step-2' + name: 'Consistency check' + run: 'git diff --exit-code ''.github/workflows/check-action-typing.yaml''' + check_action_typing: + name: 'Check Action Typing' + runs-on: 'ubuntu-latest' + needs: + - 'check_yaml_consistency' + steps: + - id: 'step-0' + name: 'Configure Git' + run: 'git config --global core.autocrlf input' + - id: 'step-1' + name: 'Checkout' + uses: 'actions/checkout@v3' + - id: 'step-2' + name: 'Check Action Typing' + uses: 'typesafegithub/github-actions-typing@v1' diff --git a/.github/workflows/reserves/setup-wsl/check-dependency-versions.main.kts b/.github/workflows/reserves/setup-wsl/check-dependency-versions.main.kts new file mode 100755 index 000000000000..27705d447853 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/check-dependency-versions.main.kts @@ -0,0 +1,73 @@ +#!/usr/bin/env kotlin + +/* + * Copyright 2020-2023 Björn Kautler + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:Import("workflow-with-copyright.main.kts") + +import io.github.typesafegithub.workflows.actions.actions.CheckoutV3 +import io.github.typesafegithub.workflows.actions.actions.SetupJavaV3 +import io.github.typesafegithub.workflows.actions.actions.SetupJavaV3.Distribution.Temurin +import io.github.typesafegithub.workflows.actions.burrunan.GradleCacheActionV1 +import io.github.typesafegithub.workflows.domain.RunnerType.WindowsLatest +import io.github.typesafegithub.workflows.domain.triggers.Cron +import io.github.typesafegithub.workflows.domain.triggers.Schedule + +workflowWithCopyright( + name = "Check Dependency Versions", + on = listOf( + Schedule( + listOf( + Cron( + minute = "0", + hour = "0", + dayWeek = "FRI" + ) + ) + ) + ), + sourceFile = __FILE__ +) { + job( + id = "check_dependency_versions", + name = "Check Dependency Versions", + runsOn = WindowsLatest + ) { + run( + name = "Configure Git", + command = "git config --global core.autocrlf input" + ) + uses( + name = "Checkout", + action = CheckoutV3() + ) + uses( + name = "Setup Java 11", + action = SetupJavaV3( + javaVersion = "11", + distribution = Temurin + ) + ) + uses( + name = "Check Dependency Versions", + action = GradleCacheActionV1( + arguments = listOf("--show-version", "dependencyUpdates"), + debug = false, + concurrent = true + ) + ) + } +} diff --git a/.github/workflows/reserves/setup-wsl/check-dependency-versions.yaml b/.github/workflows/reserves/setup-wsl/check-dependency-versions.yaml new file mode 100644 index 000000000000..8704107e23b7 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/check-dependency-versions.yaml @@ -0,0 +1,63 @@ +# Copyright 2020-2023 Björn Kautler +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file was generated using Kotlin DSL (.github/workflows/check-dependency-versions.main.kts). +# If you want to modify the workflow, please change the Kotlin file and regenerate this YAML file. +# Generated with https://github.com/typesafegithub/github-workflows-kt + +name: 'Check Dependency Versions' +on: + schedule: + - cron: '0 0 * * FRI' +jobs: + check_yaml_consistency: + name: 'Check YAML consistency' + runs-on: 'ubuntu-latest' + steps: + - id: 'step-0' + name: 'Check out' + uses: 'actions/checkout@v4' + - id: 'step-1' + name: 'Execute script' + run: 'rm ''.github/workflows/check-dependency-versions.yaml'' && ''.github/workflows/check-dependency-versions.main.kts''' + - id: 'step-2' + name: 'Consistency check' + run: 'git diff --exit-code ''.github/workflows/check-dependency-versions.yaml''' + check_dependency_versions: + name: 'Check Dependency Versions' + runs-on: 'windows-latest' + needs: + - 'check_yaml_consistency' + steps: + - id: 'step-0' + name: 'Configure Git' + run: 'git config --global core.autocrlf input' + - id: 'step-1' + name: 'Checkout' + uses: 'actions/checkout@v3' + - id: 'step-2' + name: 'Setup Java 11' + uses: 'actions/setup-java@v3' + with: + java-version: '11' + distribution: 'temurin' + - id: 'step-3' + name: 'Check Dependency Versions' + uses: 'burrunan/gradle-cache-action@v1' + with: + debug: 'false' + concurrent: 'true' + arguments: |- + --show-version + dependencyUpdates diff --git a/.github/workflows/reserves/setup-wsl/test.main.kts b/.github/workflows/reserves/setup-wsl/test.main.kts new file mode 100755 index 000000000000..b3e80f1d506e --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/test.main.kts @@ -0,0 +1,1084 @@ +#!/usr/bin/env kotlin + +/* + * Copyright 2020-2023 Björn Kautler + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:Import("workflow-with-copyright.main.kts") + +import io.github.typesafegithub.workflows.actions.actions.CacheRestoreV4 +import io.github.typesafegithub.workflows.actions.actions.CacheSaveV4 +import io.github.typesafegithub.workflows.actions.actions.CheckoutV4 +import io.github.typesafegithub.workflows.actions.actions.SetupJavaV4 +import io.github.typesafegithub.workflows.actions.actions.SetupJavaV4.Distribution.Temurin +import io.github.typesafegithub.workflows.actions.burrunan.GradleCacheActionV1 +import io.github.typesafegithub.workflows.actions.vampire.SetupWslV3 +import io.github.typesafegithub.workflows.actions.vampire.SetupWslV3.Distribution +import io.github.typesafegithub.workflows.domain.CommandStep +import io.github.typesafegithub.workflows.domain.ActionStep +import io.github.typesafegithub.workflows.domain.JobOutputs.EMPTY +import io.github.typesafegithub.workflows.domain.AbstractResult.Status.Success +import io.github.typesafegithub.workflows.domain.RunnerType +import io.github.typesafegithub.workflows.domain.RunnerType.WindowsLatest +import io.github.typesafegithub.workflows.domain.Shell +import io.github.typesafegithub.workflows.domain.Shell.Cmd +import io.github.typesafegithub.workflows.domain.Step +import io.github.typesafegithub.workflows.domain.triggers.Cron +import io.github.typesafegithub.workflows.domain.triggers.PullRequest +import io.github.typesafegithub.workflows.domain.triggers.Push +import io.github.typesafegithub.workflows.domain.triggers.Schedule +import io.github.typesafegithub.workflows.dsl.JobBuilder +import io.github.typesafegithub.workflows.dsl.WorkflowBuilder +import io.github.typesafegithub.workflows.dsl.expressions.expr +import kotlin.math.min + +val environments = listOf( + "windows-2019", + "windows-2022", + "windows-latest" +) + +val debian = mapOf( + "wsl-id" to "Debian", + "user-id" to "Debian", + "match-pattern" to "*Debian*", + "default-absent-tool" to "dos2unix" +) + +val alpine = mapOf( + "wsl-id" to "Alpine", + "user-id" to "Alpine", + "match-pattern" to "*Alpine*", + "default-absent-tool" to "dos2unix" +) + +val kali = mapOf( + "wsl-id" to "kali-linux", + "user-id" to "kali-linux", + "match-pattern" to "*Kali*", + "default-absent-tool" to "dos2unix" +) + +val openSuseLeap15_2 = mapOf( + "wsl-id" to "openSUSE-Leap-15.2", + "user-id" to "openSUSE-Leap-15.2", + "match-pattern" to "*openSUSE*Leap*15.2*", + "default-absent-tool" to "which" +) + +val ubuntu2404 = mapOf( + "wsl-id" to "Ubuntu-24.04", + "user-id" to "Ubuntu-24.04", + "match-pattern" to "*Ubuntu*24.04*", + "default-absent-tool" to "dos2unix" +) + +val ubuntu2204 = mapOf( + "wsl-id" to "Ubuntu", + "user-id" to "Ubuntu-22.04", + "match-pattern" to "*Ubuntu*22.04*", + "default-absent-tool" to "dos2unix" +) + +val ubuntu2004 = mapOf( + "wsl-id" to "Ubuntu", + "user-id" to "Ubuntu-20.04", + "match-pattern" to "*Ubuntu*20.04*", + "default-absent-tool" to "dos2unix" +) + +val ubuntu1804 = mapOf( + "wsl-id" to "Ubuntu-18.04", + "user-id" to "Ubuntu-18.04", + "match-pattern" to "*Ubuntu*18.04*", + "default-absent-tool" to "dos2unix" +) + +val ubuntu1604 = mapOf( + "wsl-id" to "Ubuntu-16.04", + "user-id" to "Ubuntu-16.04", + "match-pattern" to "*Ubuntu*16.04*", + "default-absent-tool" to "dos2unix" +) + +val distributions = listOf( + debian, + alpine, + kali, + openSuseLeap15_2, + ubuntu2404, + ubuntu2204, + ubuntu2004, + ubuntu1804, + ubuntu1604 +) + +val wslBash = Shell.Custom("wsl-bash {0}") + +val wslSh = Shell.Custom("wsl-sh {0}") + +lateinit var executeActionStep: ActionStep + +workflowWithCopyright( + name = "Build and Test", + on = listOf( + Push(), + PullRequest(), + Schedule(listOf(Cron(minute = "0", hour = "0"))) + ), + sourceFile = __FILE__ +) { + val builtArtifacts = listOf( + "action.yml", + "build/distributions/" + ) + + val executeAction = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distribution.user-id")) + ) + + val build = job( + id = "build", + name = "Build", + runsOn = WindowsLatest + ) { + run( + name = "Configure Git", + command = "git config --global core.autocrlf input" + ) + uses( + name = "Checkout", + action = CheckoutV4() + ) + uses( + name = "Setup Java 11", + action = SetupJavaV4( + javaVersion = "11", + distribution = Temurin + ) + ) + uses( + name = "Build", + action = GradleCacheActionV1( + arguments = listOf( + "--show-version", + "build", + "--info", + "--stacktrace", + "--scan" + ), + debug = false, + concurrent = true + ) + ) + uses( + name = "Save built artifacts to cache", + action = CacheSaveV4( + path = builtArtifacts, + key = expr { github.run_id } + ) + ) + } + + fun WorkflowBuilder.testJob( + id: String, + name: String, + _customArguments: Map = mapOf(), + block: JobBuilder.() -> Unit, + ) = job( + id = id, + name = name, + needs = listOf(build), + runsOn = RunnerType.Custom(expr("matrix.environment")), + _customArguments = _customArguments + ) { + uses( + name = "Restore built artifacts from cache", + action = CacheRestoreV4( + path = builtArtifacts, + key = expr { github.run_id }, + failOnCacheMiss = true + ) + ) + block() + } + + testJob( + id = "test_invalid_distribution", + name = """Test "${expr("matrix.distribution.label")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to listOf( + mapOf( + "user-id" to "invalid", + "label" to "invalid" + ), + mapOf( + "user-id" to "", + "label" to "" + ), + mapOf( + "user-id" to null, + "label" to "null" + ) + ) + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = executeAction, + continueOnError = true + ) + run( + name = "Test - action should fail if an invalid distribution is given", + shell = Cmd, + command = "if '${expr("${executeActionStep.outcome}")}' NEQ 'failure' exit 1" + ) + } + + testJob( + id = "test_default_distribution", + name = "Test default distribution on ${expr("matrix.environment")}", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to listOf(debian) + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = SetupWslV3( + update = true + ) + ) + commonTests() + verifyFailure( + name = "Test - wsl-bash should fail if no script file is given", + provocationShell = Shell.Custom("wsl-bash") + ) + verifyFailure( + name = "Test - wsl-bash should fail if more than one parameter is given and first is not -u", + provocationShell = Shell.Custom("wsl-bash user {0}") + ) + verifyFailure( + name = "Test - wsl-bash should fail if only user is given", + provocationShell = Shell.Custom("wsl-bash -u {0}") + ) + verifyFailure( + name = "Test - wsl-bash should fail if excess argument is given", + provocationShell = Shell.Custom("wsl-bash -u user {0} foo") + ) + verifyFailure( + name = "Test - wsl-bash should fail if given script file does not exist", + provocationShell = Shell.Custom("wsl-bash -u user {0}foo") + ) + } + + testJob( + id = "test", + name = """Test "${expr("matrix.distribution.user-id")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to distributions + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = executeAction.copy( + useCache = false + ) + ) + verifyFailure( + name = "Test - wsl-bash should fail if bash is not present by default", + conditionTransformer = { executeActionStep.successOnAlpineCondition }, + verificationShell = null, + verificationTransformer = { _, command -> + """wsl sh -euc "${command.replace("==", "=")}"""" + } + ) + deleteWslBashOnAlpine() + usesSelf( + name = "Install Bash on Alpine", + action = executeAction.copy( + additionalPackages = listOf("bash") + ), + condition = executeActionStep.successOnAlpineCondition + ) + commonTests() + verifyFailure( + name = "Test - ${expr("matrix.distribution.default-absent-tool")} should not be installed by default", + provocationCommand = "${expr("matrix.distribution.default-absent-tool")} --version" + ) + runAfterSuccess( + name = "Test - bash should be installed by default", + command = "bash -c true" + ) + runAfterSuccess( + name = "Test - sh should be installed by default", + command = "sh -c true" + ) + verifyFailure( + name = "Test - wsl-sh should not be present", + provocationShell = wslSh + ) + val wslBashPath = executeActionStep.outputs.wslShellWrapperPath + executeActionStep = usesSelfAfterSuccess( + name = "Add wsl-sh wrapper", + action = executeAction.copy( + wslShellCommand = "sh -eu" + ) + ) + runAfterSuccess( + name = "Test - wsl-sh should be present", + shell = wslSh + ) + runAfterSuccess( + name = "Test - wsl-bash should use bash", + command = """ + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" == 'bash' ] + """ + ) + runAfterSuccess( + name = "Test - wsl-sh should use sh", + shell = wslSh, + command = """ + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" = 'sh' ] + """ + ) + deleteWslBash( + wslBashPathExpression = wslBashPath + ) + verifyFailure( + name = "Test - wsl-bash should not be present", + verificationShell = wslSh, + verificationTransformer = { _, command -> + command.replace("==", "=") + } + ) + executeActionStep = usesSelfAfterSuccess( + name = "Re-add wsl-bash wrapper", + action = executeAction + ) + runAfterSuccess( + name = "Test - wsl-bash should be present" + ) + runAfterSuccess( + name = "Test - wsl-bash should use bash", + command = """ + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" == 'bash' ] + """ + ) + verifyCommandResult( + name = "Test - wsl-bash should use root as default user", + actualCommand = "whoami", + expected = "root" + ) + runAfterSuccess( + name = "Add user test", + command = "useradd -m -p 4qBD5NWD3IkbU test" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Set wsl-bash wrapper to use user test by default", + action = executeAction.copy( + additionalPackages = listOf("sudo"), + wslShellCommand = """bash -c "sudo -u test bash --noprofile --norc -euo pipefail "\""" + ) + ) + verifyCommandResult( + name = "Test - wsl-bash should use test as default user", + actualCommand = "whoami", + expected = "test" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Set wsl-bash wrapper to use user test by default with inline script usage", + action = executeAction.copy( + wslShellCommand = """bash -c "sudo -u test bash --noprofile --norc -euo pipefail '{0}'"""" + ) + ) + verifyCommandResult( + name = "Test - wsl-bash should use test as default user with inline script usage", + actualCommand = "whoami", + expected = "test" + ) + deleteWslBash() + executeActionStep = usesSelfAfterSuccess( + name = "Set wsl-bash wrapper to use default user by default", + action = executeAction + ) + verifyCommandResult( + name = "Test - wsl-bash should use root as default user", + actualCommand = "whoami", + expected = "root" + ) + runAfterSuccess( + name = "Test - test user does already exist", + command = "id -u test" + ) + deleteWslBash() + executeActionStep = usesSelfAfterSuccess( + name = "Set wsl-bash wrapper to use existing user test by default with extra parameter", + action = executeAction.copy( + wslShellUser = "test" + ) + ) + verifyCommandResult( + name = "Test - wsl-bash should use existing user test as default user with extra parameter", + actualCommand = "whoami", + expected = "test" + ) + runAfterSuccess( + name = "Test - test2 user does not exist", + command = "! id -u test2" + ) + deleteWslBash() + executeActionStep = usesSelfAfterSuccess( + name = "Set wsl-bash wrapper to use non-existing user test2 by default with extra parameter", + action = executeAction.copy( + wslShellUser = "test2" + ) + ) + verifyCommandResult( + name = "Test - wsl-bash should use auto-generated user test2 as default user", + actualCommand = "whoami", + expected = "test2" + ) + verifyCommandResult( + name = "Test - wsl-bash should use ad-hoc user test", + actualCommand = "whoami", + shell = Shell.Custom("wsl-bash -u test {0}"), + expected = "test" + ) + verifyCommandResult( + name = "Test - wsl-bash should use ad-hoc user root", + actualCommand = "whoami", + shell = Shell.Custom("wsl-bash -u root {0}"), + expected = "root" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Make a no-op execution of the action", + action = executeAction + ) + verifyCommandResult( + name = "Test - wsl-bash should still use test2 as default user", + actualCommand = "whoami", + expected = "test2" + ) + } + + testJob( + id = "test_wsl-conf_on_initial_execution", + name = """Test /etc/wsl.conf handling on initial execution for "${expr("matrix.distribution.user-id")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to distributions + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = executeAction.copy( + wslConf = """ + [automount] + options = uid=1000 + """.trimIndent() + ) + ) + deleteWslBashOnAlpine() + usesSelf( + name = "Install Bash on Alpine", + action = executeAction.copy( + additionalPackages = listOf("bash") + ), + condition = executeActionStep.successOnAlpineCondition + ) + runAfterSuccess( + name = "Test - /etc/wsl.conf should exist", + command = """ + [ -f /etc/wsl.conf ] + cat /etc/wsl.conf + """ + ) + runAfterSuccess( + name = "Test - /mnt/c should be mounted with uid 1000", + command = """ + ls -alh /mnt + [[ "$(stat -c %u /mnt/c)" == 1000 ]] + """ + ) + } + + testJob( + id = "test_wsl-conf_on_subsequent_execution", + name = """Test /etc/wsl.conf handling on subsequent execution for "${expr("matrix.distribution.user-id")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to distributions + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = executeAction + ) + deleteWslBashOnAlpine() + usesSelf( + name = "Install Bash on Alpine", + action = executeAction.copy( + additionalPackages = listOf("bash") + ), + condition = executeActionStep.successOnAlpineCondition + ) + runAfterSuccess( + name = "Test - /etc/wsl.conf should not exist", + command = "[ ! -f /etc/wsl.conf ] || { cat /etc/wsl.conf; false; }", + conditionTransformer = { executeActionStep.successNotOnUbuntu2404Condition } + ) + runAfterSuccess( + name = "Test - C: should be mounted at /mnt/c", + command = """ + mount + mount | grep 'C:.* on /mnt/c' + """ + ) + runAfterSuccess( + name = "Test - /mnt/c should be mounted with uid 0", + command = """ + ls -alh /mnt + [[ "$(stat -c %u /mnt/c)" == 0 ]] + """ + ) + executeActionStep = usesSelfAfterSuccess( + action = executeAction.copy( + wslConf = """ + [automount] + root = / + """.trimIndent() + ) + ) + runAfterSuccess( + name = "Test - /etc/wsl.conf should exist", + command = """ + [ -f /etc/wsl.conf ] + cat /etc/wsl.conf + """ + ) + runAfterSuccess( + name = "Test - C: should be mounted at /c", + command = """ + mount + mount | grep 'C:.* on /c' + """ + ) + } + + testJob( + id = "test_additional_packages", + name = """Test additional packages for "${expr("matrix.distribution.user-id")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to distributions + ) + ) + ) + ) { + executeActionStep = usesSelf( + action = executeAction.copy( + additionalPackages = listOf( + expr("matrix.distribution.default-absent-tool"), + "bash" + ) + ) + ) + runAfterSuccess( + name = "Test - ${expr("matrix.distribution.default-absent-tool")} should be installed", + command = "${expr("matrix.distribution.default-absent-tool")} --version" + ) + runAfterSuccess( + name = "Test - bash should be installed", + command = "bash -c true" + ) + } + + testJob( + id = "test_multiple_usage_with_different_distributions", + name = """ + Test multiple usage with different distributions + ("${expr("matrix.distributions.distribution1.user-id")}" + / "${expr("matrix.distributions.distribution2.user-id")}" + / "${expr("matrix.distributions.distribution3.user-id")}") + on ${expr("matrix.environment")} + """.trimIndent().replace("\n", " "), + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distributions" to listOf( + mapOf( + "distribution1" to debian, + "distribution2" to ubuntu2004, + "distribution3" to ubuntu1804 + ), + mapOf( + "distribution1" to debian, + "distribution2" to ubuntu1804, + "distribution3" to ubuntu2004 + ), + mapOf( + "distribution1" to ubuntu2004, + "distribution2" to debian, + "distribution3" to ubuntu1804 + ), + mapOf( + "distribution1" to ubuntu2004, + "distribution2" to ubuntu1804, + "distribution3" to debian + ), + mapOf( + "distribution1" to ubuntu1804, + "distribution2" to debian, + "distribution3" to ubuntu2004 + ), + mapOf( + "distribution1" to ubuntu1804, + "distribution2" to ubuntu2004, + "distribution3" to debian + ) + ) + ) + ) + ) + ) { + usesSelf( + name = "Execute action for ${expr("matrix.distributions.distribution1.user-id")}", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distributions.distribution1.user-id")) + ) + ) + usesSelf( + name = "Execute action for ${expr("matrix.distributions.distribution2.user-id")}", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distributions.distribution2.user-id")) + ) + ) + usesSelf( + name = "Execute action for ${expr("matrix.distributions.distribution3.user-id")}", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distributions.distribution3.user-id")), + setAsDefault = false + ) + ) + executeActionStep = usesSelf( + name = "Execute action for ${expr("matrix.distributions.distribution1.user-id")} again", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distributions.distribution1.user-id")) + ) + ) + verifyCommandResult( + name = "Test - the default distribution should be the last installed distribution with set-as-default true", + actualCommand = """ + cat + <(wsl.exe --list || true) + <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) + <(wslconfig.exe /list || true) + <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8) + """.trimIndent().replace("\n", " "), + expectedPattern = """*${expr("matrix.distributions.distribution2.wsl-id")}\ \(Default\)*""" + ) + verifyInstalledDistribution( + name = "Test - wsl-bash should use the last installed distribution with set-as-default true", + expectedPatternExpression = "matrix.distributions.distribution2.match-pattern" + ) + } + + testJob( + id = "test_multiple_usage_with_same_distribution", + name = """Test multiple usage with "${expr("matrix.distribution.user-id")}" distribution on ${expr("matrix.environment")}""", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + "distribution" to distributions, + "distribution2" to listOf(debian), + "exclude" to environments.map { + mapOf( + "environment" to it, + "distribution" to debian, + "distribution2" to debian + ) + }, + "include" to environments.map { + mapOf( + "environment" to it, + "distribution" to debian, + "distribution2" to ubuntu2004 + ) + } + ) + ) + ) + ) { + usesSelf( + action = executeAction.copy( + additionalPackages = listOf("bash") + ) + ) + usesSelf( + name = "Update distribution", + action = executeAction.copy( + update = true + ), + // work-around for https://bugs.kali.org/view.php?id=6672 + condition = "matrix.distribution.user-id != 'kali-linux'" + ) + executeActionStep = usesSelf( + name = "Install default absent tool", + action = executeAction.copy( + additionalPackages = listOf(expr("matrix.distribution.default-absent-tool")) + ) + ) + runAfterSuccess( + name = "Test - ${expr("matrix.distribution.default-absent-tool")} should be installed", + command = "${expr("matrix.distribution.default-absent-tool")} --version" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Execute action for ${expr("matrix.distribution2.user-id")}", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distribution2.user-id")) + ) + ) + verifyInstalledDistribution( + name = """Test - "${expr("matrix.distribution2.user-id")}" should be the default distribution after installation""", + expectedPatternExpression = "matrix.distribution2.match-pattern" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Re-execute action", + action = executeAction + ) + verifyInstalledDistribution( + name = """Test - "${expr("matrix.distribution2.user-id")}" should still be the default distribution after re-running for "${expr("matrix.distribution.user-id")}"""", + expectedPatternExpression = "matrix.distribution2.match-pattern" + ) + executeActionStep = usesSelfAfterSuccess( + name = "Set as default", + action = executeAction.copy( + setAsDefault = true + ) + ) + verifyInstalledDistribution( + name = """Test - "${expr("matrix.distribution.user-id")}" should be the default distribution after re-running with set-as-default true""" + ) + } + + testJob( + id = "test_distribution_specific_wsl_bash_scripts", + name = "Test distribution specific wsl-bash scripts on ${expr("matrix.environment")} (without ${expr("matrix.distributions.incompatibleUbuntu")})", + _customArguments = mapOf( + "strategy" to mapOf( + "fail-fast" to false, + "matrix" to mapOf( + "environment" to environments, + // ubuntu2004 and ubuntu2204 currently have the same wsl-id + // so their distribution specific wsl_bash scripts will clash + // and thus cannot be tested together + "distributions" to listOf(ubuntu2204, ubuntu2004) + .map { incompatibleUbuntu -> + distributions + .filter { it != incompatibleUbuntu } + .mapIndexed, Pair> { i, distribution -> + "distribution${i + 1}" to distribution + } + .toMutableList() + .apply { + add(0, "incompatibleUbuntu" to incompatibleUbuntu["user-id"]!!) + } + .toMap() + } + ) + ) + ) + ) { + (1 until distributions.size) + .associateWith { + usesSelf( + name = "Execute action for ${expr("matrix.distributions.distribution$it.user-id")}", + action = SetupWslV3( + distribution = Distribution.Custom(expr("matrix.distributions.distribution$it.user-id")), + additionalPackages = if (it == 2) listOf("bash") else null, + setAsDefault = if (it >= 3) false else null + ) + ) + } + .forEach { (i, localExecuteActionStep) -> + executeActionStep = localExecuteActionStep + verifyInstalledDistribution( + name = "Test - wsl-bash_${expr("matrix.distributions.distribution$i.user-id")} should use the correct distribution", + conditionTransformer = if (distributions[i] == ubuntu2004) { + { executeActionStep.getSuccessNotOnDistributionCondition(i, "Ubuntu-20.04") } + } else { + { it } + }, + // the formula adds 1 to the indices from ubuntu2004 on + // to mitigate the double entry for the previous index + shell = Shell.Custom("wsl-bash_${distributions[min(1, i / (distributions.indexOf(ubuntu2004) + 1)) + i - 1]["user-id"]} {0}"), + expectedPatternExpression = "matrix.distributions.distribution$i.match-pattern" + ) + if (distributions[i] == ubuntu2004) { + verifyInstalledDistribution( + name = "Test - wsl-bash_${expr("matrix.distributions.distribution$i.user-id")} should use the correct distribution", + conditionTransformer = { executeActionStep.getSuccessNotOnDistributionCondition(i, "Ubuntu-22.04") }, + shell = Shell.Custom("wsl-bash_${distributions[i]["user-id"]} {0}"), + expectedPatternExpression = "matrix.distributions.distribution$i.match-pattern" + ) + } + } + + } +} + +fun JobBuilder<*>.commonTests() { + runAfterSuccess( + name = "Test - wsl-bash should be available as custom shell" + ) + verifyFailure( + name = "Test - wsl-bash should fail if the script fails", + provocationCommand = "false", + verificationShell = Cmd, + verificationTransformer = { provocationStep, _ -> + // do not just rely on false here, but explicitly use exit + // in case failing commands do not make the script fail + // and use "shell = Cmd" to capture that the wrapper script is hiding errors + "IF '${expr("${provocationStep.outcome}") }' NEQ 'failure' EXIT /B 1" + } + ) + verifyFailure( + name = "Test - wsl-bash should fail if one of the commands fails", + provocationCommand = """ + false + : + """, + // do not just rely on false here, but explicitly use exit + // in case failing commands do not make the script fail + verificationTransformer = { _, command -> "$command || exit 1" } + ) + verifyFailure( + name = "Test - wsl-bash should fail if an undefined variable is used", + provocationCommand = "\$foo" + ) + verifyFailure( + name = "Test - wsl-bash should fail if any command in a pipe fails", + provocationCommand = "false | true" + ) + verifyCommandResult( + name = "Test - the default distribution should be correct", + actualCommand = """ + cat + <(wsl.exe --list || true) + <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) + <(wslconfig.exe /list || true) + <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8) + """.trimIndent().replace("\n", " "), + expectedPattern = """*${expr("matrix.distribution.wsl-id")}\ \(Default\)*""" + ) + verifyInstalledDistribution( + name = "Test - wsl-bash should use the correct distribution" + ) + runAfterSuccess( + name = "Test - multi-line commands should not be disturbed by CRLF line endings", + command = """ + : # this comment catches the CR if present + ! grep -q $'\r' "$0" # this comment catches the CR if present + """ + ) +} + +fun JobBuilder<*>.usesSelfAfterSuccess( + name: String = "Execute action", + action: SetupWslV3 +) = usesSelf( + name = name, + action = action, + condition = executeActionStep.successCondition +) + +fun JobBuilder<*>.usesSelf( + name: String = "Execute action", + action: SetupWslV3, + condition: String? = null, + continueOnError: Boolean? = null +) = uses( + name = name, + action = action, + condition = condition, + continueOnError = continueOnError, + _customArguments = mapOf( + "uses" to "./" + ) +) + +fun JobBuilder<*>.deleteWslBashOnAlpine() = deleteWslBash( + conditionTransformer = { executeActionStep.successOnAlpineCondition } +) + +fun JobBuilder<*>.deleteWslBash( + wslBashPathExpression: String = executeActionStep.outputs.wslShellWrapperPath, + conditionTransformer: (String) -> String = { it } +) = run( + name = "Delete wsl-bash", + condition = conditionTransformer(executeActionStep.successCondition).trimIndent(), + shell = Cmd, + command = """DEL /F "${expr(wslBashPathExpression)}"""" +) + +fun JobBuilder<*>.runAfterSuccess( + name: String, + conditionTransformer: (String) -> String = { it }, + shell: Shell? = wslBash, + command: String = ":", + continueOnError: Boolean? = null +) = run( + name = name, + condition = conditionTransformer(executeActionStep.successCondition).trimIndent(), + shell = shell, + command = command.trimIndent(), + continueOnError = continueOnError +) + +fun JobBuilder<*>.verifyFailure( + name: String, + conditionTransformer: (String) -> String = { it }, + provocationShell: Shell = wslBash, + provocationCommand: String = ":", + verificationShell: Shell? = wslBash, + verificationTransformer: (CommandStep, String) -> String = { _, command -> command } +) { + val provocationStep = runAfterSuccess( + name = "$name (provocation)", + conditionTransformer = conditionTransformer, + shell = provocationShell, + command = provocationCommand, + continueOnError = true + ) + runAfterSuccess( + name = "$name (verification)", + conditionTransformer = conditionTransformer, + shell = verificationShell, + command = verificationTransformer( + provocationStep, + "[ '${expr("${provocationStep.outcome}")}' == 'failure' ]" + ) + ) +} + +fun JobBuilder<*>.verifyInstalledDistribution( + name: String, + conditionTransformer: (String) -> String = { it }, + shell: Shell = wslBash, + expectedPatternExpression: String = "matrix.distribution.match-pattern" +) = verifyCommandResult( + name = name, + conditionTransformer = conditionTransformer, + actualCommand = """ + cat + <(lsb_release -a || true) + <(uname -a || true) + <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) + <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) + <([ -f /proc/version ] && cat /proc/version || true) + """.trimIndent().replace("\n", " "), + shell = shell, + expectedPattern = expr(expectedPatternExpression) +) + +fun JobBuilder<*>.verifyCommandResult( + name: String, + conditionTransformer: (String) -> String = { it }, + actualCommand: String, + shell: Shell = wslBash, + expected: String? = null, + expectedPattern: String? = null +) { + require((expected != null) || (expectedPattern != null)) { + "Either expected or expectedPattern must be non-null" + } + require( + ((expected != null) && (expectedPattern == null)) + || ((expected == null) && (expectedPattern != null)) + ) { + "Either expected or expectedPattern must be non-null, but not both" + } + runAfterSuccess( + name = name, + conditionTransformer = conditionTransformer, + shell = shell, + command = if (expected != null) """ + ${actualCommand.trimIndent()} + [ "$(${actualCommand.trimIndent()})" == '${expected.trimIndent()}' ] + """ else if (expectedPattern != null) """ + ${actualCommand.trimIndent()} + [[ "$(${actualCommand.trimIndent()})" == ${expectedPattern.trimIndent()} ]] + """ else error("Erroneous method input validation") + ) +} + +val Step.successCondition + get() = """ + always() + && (${outcome.eq(Success)}) + """.trimIndent() + +val Step.successOnAlpineCondition + get() = """ + always() + && (${outcome.eq(Success)}) + && (matrix.distribution.user-id == 'Alpine') + """.trimIndent() + +val Step.successNotOnUbuntu2404Condition + get() = """ + always() + && (${outcome.eq(Success)}) + && (matrix.distribution.user-id != 'Ubuntu-24.04') + """.trimIndent() + +fun Step.getSuccessNotOnDistributionCondition(i: Int, distribution: String) = """ + always() + && (${outcome.eq(Success)}) + && (matrix.distributions.distribution$i.user-id != '$distribution') +""".trimIndent() diff --git a/.github/workflows/reserves/setup-wsl/test.yaml b/.github/workflows/reserves/setup-wsl/test.yaml new file mode 100644 index 000000000000..8c1fec5f45e8 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/test.yaml @@ -0,0 +1,1738 @@ +# Copyright 2020-2023 Björn Kautler +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file was generated using Kotlin DSL (.github/workflows/test.main.kts). +# If you want to modify the workflow, please change the Kotlin file and regenerate this YAML file. +# Generated with https://github.com/typesafegithub/github-workflows-kt + +name: 'Build and Test' +on: + push: {} + pull_request: {} + schedule: + - cron: '0 0 * * *' +jobs: + check_yaml_consistency: + name: 'Check YAML consistency' + runs-on: 'ubuntu-latest' + steps: + - id: 'step-0' + name: 'Check out' + uses: 'actions/checkout@v4' + - id: 'step-1' + name: 'Execute script' + run: 'rm ''.github/workflows/test.yaml'' && ''.github/workflows/test.main.kts''' + - id: 'step-2' + name: 'Consistency check' + run: 'git diff --exit-code ''.github/workflows/test.yaml''' + build: + name: 'Build' + runs-on: 'windows-latest' + needs: + - 'check_yaml_consistency' + steps: + - id: 'step-0' + name: 'Configure Git' + run: 'git config --global core.autocrlf input' + - id: 'step-1' + name: 'Checkout' + uses: 'actions/checkout@v4' + - id: 'step-2' + name: 'Setup Java 11' + uses: 'actions/setup-java@v4' + with: + java-version: '11' + distribution: 'temurin' + - id: 'step-3' + name: 'Build' + uses: 'burrunan/gradle-cache-action@v1' + with: + debug: 'false' + concurrent: 'true' + arguments: |- + --show-version + build + --info + --stacktrace + --scan + - id: 'step-4' + name: 'Save built artifacts to cache' + uses: 'actions/cache/save@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + test_invalid_distribution: + name: 'Test "${{ matrix.distribution.label }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - user-id: 'invalid' + label: 'invalid' + - user-id: '' + label: '' + - user-id: null + label: 'null' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + continue-on-error: true + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + - id: 'step-2' + name: 'Test - action should fail if an invalid distribution is given' + shell: 'cmd' + run: 'if ''${{ steps.step-1.outcome }}'' NEQ ''failure'' exit 1' + test_default_distribution: + name: 'Test default distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + update: 'true' + - id: 'step-2' + name: 'Test - wsl-bash should be available as custom shell' + shell: 'wsl-bash {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-3' + name: 'Test - wsl-bash should fail if the script fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: 'false' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-4' + name: 'Test - wsl-bash should fail if the script fails (verification)' + shell: 'cmd' + run: 'IF ''${{ steps.step-3.outcome }}'' NEQ ''failure'' EXIT /B 1' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-5' + name: 'Test - wsl-bash should fail if one of the commands fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: |- + false + : + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-6' + name: 'Test - wsl-bash should fail if one of the commands fails (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-5.outcome }}'' == ''failure'' ] || exit 1' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-7' + name: 'Test - wsl-bash should fail if an undefined variable is used (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: '$foo' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-8' + name: 'Test - wsl-bash should fail if an undefined variable is used (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-7.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-9' + name: 'Test - wsl-bash should fail if any command in a pipe fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: 'false | true' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-10' + name: 'Test - wsl-bash should fail if any command in a pipe fails (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-9.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-11' + name: 'Test - the default distribution should be correct' + shell: 'wsl-bash {0}' + run: |- + cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8) + [[ "$(cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8))" == *${{ matrix.distribution.wsl-id }}\ \(Default\)* ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-12' + name: 'Test - wsl-bash should use the correct distribution' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distribution.match-pattern }} ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-13' + name: 'Test - multi-line commands should not be disturbed by CRLF line endings' + shell: 'wsl-bash {0}' + run: |- + : # this comment catches the CR if present + ! grep -q $'\r' "$0" # this comment catches the CR if present + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-14' + name: 'Test - wsl-bash should fail if no script file is given (provocation)' + continue-on-error: true + shell: 'wsl-bash' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-15' + name: 'Test - wsl-bash should fail if no script file is given (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-14.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-16' + name: 'Test - wsl-bash should fail if more than one parameter is given and first is not -u (provocation)' + continue-on-error: true + shell: 'wsl-bash user {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-17' + name: 'Test - wsl-bash should fail if more than one parameter is given and first is not -u (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-16.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-18' + name: 'Test - wsl-bash should fail if only user is given (provocation)' + continue-on-error: true + shell: 'wsl-bash -u {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-19' + name: 'Test - wsl-bash should fail if only user is given (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-18.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-20' + name: 'Test - wsl-bash should fail if excess argument is given (provocation)' + continue-on-error: true + shell: 'wsl-bash -u user {0} foo' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-21' + name: 'Test - wsl-bash should fail if excess argument is given (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-20.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-22' + name: 'Test - wsl-bash should fail if given script file does not exist (provocation)' + continue-on-error: true + shell: 'wsl-bash -u user {0}foo' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-23' + name: 'Test - wsl-bash should fail if given script file does not exist (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-22.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + test: + name: 'Test "${{ matrix.distribution.user-id }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + - wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + - wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + - wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + use-cache: 'false' + - id: 'step-2' + name: 'Test - wsl-bash should fail if bash is not present by default (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-3' + name: 'Test - wsl-bash should fail if bash is not present by default (verification)' + run: 'wsl sh -euc "[ ''${{ steps.step-2.outcome }}'' = ''failure'' ]"' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-4' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-1.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-5' + name: 'Install Bash on Alpine' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: 'bash' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-6' + name: 'Test - wsl-bash should be available as custom shell' + shell: 'wsl-bash {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-7' + name: 'Test - wsl-bash should fail if the script fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: 'false' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-8' + name: 'Test - wsl-bash should fail if the script fails (verification)' + shell: 'cmd' + run: 'IF ''${{ steps.step-7.outcome }}'' NEQ ''failure'' EXIT /B 1' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-9' + name: 'Test - wsl-bash should fail if one of the commands fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: |- + false + : + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-10' + name: 'Test - wsl-bash should fail if one of the commands fails (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-9.outcome }}'' == ''failure'' ] || exit 1' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-11' + name: 'Test - wsl-bash should fail if an undefined variable is used (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: '$foo' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-12' + name: 'Test - wsl-bash should fail if an undefined variable is used (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-11.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-13' + name: 'Test - wsl-bash should fail if any command in a pipe fails (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: 'false | true' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-14' + name: 'Test - wsl-bash should fail if any command in a pipe fails (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-13.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-15' + name: 'Test - the default distribution should be correct' + shell: 'wsl-bash {0}' + run: |- + cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8) + [[ "$(cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8))" == *${{ matrix.distribution.wsl-id }}\ \(Default\)* ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-16' + name: 'Test - wsl-bash should use the correct distribution' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distribution.match-pattern }} ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-17' + name: 'Test - multi-line commands should not be disturbed by CRLF line endings' + shell: 'wsl-bash {0}' + run: |- + : # this comment catches the CR if present + ! grep -q $'\r' "$0" # this comment catches the CR if present + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-18' + name: 'Test - ${{ matrix.distribution.default-absent-tool }} should not be installed by default (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: '${{ matrix.distribution.default-absent-tool }} --version' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-19' + name: 'Test - ${{ matrix.distribution.default-absent-tool }} should not be installed by default (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-18.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-20' + name: 'Test - bash should be installed by default' + shell: 'wsl-bash {0}' + run: 'bash -c true' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-21' + name: 'Test - sh should be installed by default' + shell: 'wsl-bash {0}' + run: 'sh -c true' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-22' + name: 'Test - wsl-sh should not be present (provocation)' + continue-on-error: true + shell: 'wsl-sh {0}' + run: ':' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-23' + name: 'Test - wsl-sh should not be present (verification)' + shell: 'wsl-bash {0}' + run: '[ ''${{ steps.step-22.outcome }}'' == ''failure'' ]' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-24' + name: 'Add wsl-sh wrapper' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-shell-command: 'sh -eu' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-25' + name: 'Test - wsl-sh should be present' + shell: 'wsl-sh {0}' + run: ':' + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-26' + name: 'Test - wsl-bash should use bash' + shell: 'wsl-bash {0}' + run: |- + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" == 'bash' ] + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-27' + name: 'Test - wsl-sh should use sh' + shell: 'wsl-sh {0}' + run: |- + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" = 'sh' ] + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-28' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-1.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-29' + name: 'Test - wsl-bash should not be present (provocation)' + continue-on-error: true + shell: 'wsl-bash {0}' + run: ':' + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-30' + name: 'Test - wsl-bash should not be present (verification)' + shell: 'wsl-sh {0}' + run: '[ ''${{ steps.step-29.outcome }}'' = ''failure'' ]' + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-31' + name: 'Re-add wsl-bash wrapper' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + if: |- + always() + && (steps.step-24.outcome == 'success') + - id: 'step-32' + name: 'Test - wsl-bash should be present' + shell: 'wsl-bash {0}' + run: ':' + if: |- + always() + && (steps.step-31.outcome == 'success') + - id: 'step-33' + name: 'Test - wsl-bash should use bash' + shell: 'wsl-bash {0}' + run: |- + ps -o pid='' -o comm='' | grep "^\s\+$$\s\+" | grep -o '\S\+$' + [ "$(ps -o pid='' -o comm='' 2>/dev/null | grep "^\s\+$$\s\+" | grep -o '\S\+$')" == 'bash' ] + if: |- + always() + && (steps.step-31.outcome == 'success') + - id: 'step-34' + name: 'Test - wsl-bash should use root as default user' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'root' ] + if: |- + always() + && (steps.step-31.outcome == 'success') + - id: 'step-35' + name: 'Add user test' + shell: 'wsl-bash {0}' + run: 'useradd -m -p 4qBD5NWD3IkbU test' + if: |- + always() + && (steps.step-31.outcome == 'success') + - id: 'step-36' + name: 'Set wsl-bash wrapper to use user test by default' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: 'sudo' + wsl-shell-command: 'bash -c "sudo -u test bash --noprofile --norc -euo pipefail "\' + if: |- + always() + && (steps.step-31.outcome == 'success') + - id: 'step-37' + name: 'Test - wsl-bash should use test as default user' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'test' ] + if: |- + always() + && (steps.step-36.outcome == 'success') + - id: 'step-38' + name: 'Set wsl-bash wrapper to use user test by default with inline script usage' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-shell-command: 'bash -c "sudo -u test bash --noprofile --norc -euo pipefail ''{0}''"' + if: |- + always() + && (steps.step-36.outcome == 'success') + - id: 'step-39' + name: 'Test - wsl-bash should use test as default user with inline script usage' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'test' ] + if: |- + always() + && (steps.step-38.outcome == 'success') + - id: 'step-40' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-38.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-38.outcome == 'success') + - id: 'step-41' + name: 'Set wsl-bash wrapper to use default user by default' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + if: |- + always() + && (steps.step-38.outcome == 'success') + - id: 'step-42' + name: 'Test - wsl-bash should use root as default user' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'root' ] + if: |- + always() + && (steps.step-41.outcome == 'success') + - id: 'step-43' + name: 'Test - test user does already exist' + shell: 'wsl-bash {0}' + run: 'id -u test' + if: |- + always() + && (steps.step-41.outcome == 'success') + - id: 'step-44' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-41.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-41.outcome == 'success') + - id: 'step-45' + name: 'Set wsl-bash wrapper to use existing user test by default with extra parameter' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-shell-user: 'test' + if: |- + always() + && (steps.step-41.outcome == 'success') + - id: 'step-46' + name: 'Test - wsl-bash should use existing user test as default user with extra parameter' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'test' ] + if: |- + always() + && (steps.step-45.outcome == 'success') + - id: 'step-47' + name: 'Test - test2 user does not exist' + shell: 'wsl-bash {0}' + run: '! id -u test2' + if: |- + always() + && (steps.step-45.outcome == 'success') + - id: 'step-48' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-45.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-45.outcome == 'success') + - id: 'step-49' + name: 'Set wsl-bash wrapper to use non-existing user test2 by default with extra parameter' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-shell-user: 'test2' + if: |- + always() + && (steps.step-45.outcome == 'success') + - id: 'step-50' + name: 'Test - wsl-bash should use auto-generated user test2 as default user' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'test2' ] + if: |- + always() + && (steps.step-49.outcome == 'success') + - id: 'step-51' + name: 'Test - wsl-bash should use ad-hoc user test' + shell: 'wsl-bash -u test {0}' + run: |- + whoami + [ "$(whoami)" == 'test' ] + if: |- + always() + && (steps.step-49.outcome == 'success') + - id: 'step-52' + name: 'Test - wsl-bash should use ad-hoc user root' + shell: 'wsl-bash -u root {0}' + run: |- + whoami + [ "$(whoami)" == 'root' ] + if: |- + always() + && (steps.step-49.outcome == 'success') + - id: 'step-53' + name: 'Make a no-op execution of the action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + if: |- + always() + && (steps.step-49.outcome == 'success') + - id: 'step-54' + name: 'Test - wsl-bash should still use test2 as default user' + shell: 'wsl-bash {0}' + run: |- + whoami + [ "$(whoami)" == 'test2' ] + if: |- + always() + && (steps.step-53.outcome == 'success') + test_wsl-conf_on_initial_execution: + name: 'Test /etc/wsl.conf handling on initial execution for "${{ matrix.distribution.user-id }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + - wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + - wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + - wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-conf: |- + [automount] + options = uid=1000 + - id: 'step-2' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-1.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-3' + name: 'Install Bash on Alpine' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: 'bash' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-4' + name: 'Test - /etc/wsl.conf should exist' + shell: 'wsl-bash {0}' + run: |- + [ -f /etc/wsl.conf ] + cat /etc/wsl.conf + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-5' + name: 'Test - /mnt/c should be mounted with uid 1000' + shell: 'wsl-bash {0}' + run: |- + ls -alh /mnt + [[ "$(stat -c %u /mnt/c)" == 1000 ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + test_wsl-conf_on_subsequent_execution: + name: 'Test /etc/wsl.conf handling on subsequent execution for "${{ matrix.distribution.user-id }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + - wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + - wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + - wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + - id: 'step-2' + name: 'Delete wsl-bash' + shell: 'cmd' + run: 'DEL /F "${{ steps.step-1.outputs.wsl-shell-wrapper-path }}"' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-3' + name: 'Install Bash on Alpine' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: 'bash' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id == 'Alpine') + - id: 'step-4' + name: 'Test - /etc/wsl.conf should not exist' + shell: 'wsl-bash {0}' + run: '[ ! -f /etc/wsl.conf ] || { cat /etc/wsl.conf; false; }' + if: |- + always() + && (steps.step-1.outcome == 'success') + && (matrix.distribution.user-id != 'Ubuntu-24.04') + - id: 'step-5' + name: 'Test - C: should be mounted at /mnt/c' + shell: 'wsl-bash {0}' + run: |- + mount + mount | grep 'C:.* on /mnt/c' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-6' + name: 'Test - /mnt/c should be mounted with uid 0' + shell: 'wsl-bash {0}' + run: |- + ls -alh /mnt + [[ "$(stat -c %u /mnt/c)" == 0 ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-7' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + wsl-conf: |- + [automount] + root = / + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-8' + name: 'Test - /etc/wsl.conf should exist' + shell: 'wsl-bash {0}' + run: |- + [ -f /etc/wsl.conf ] + cat /etc/wsl.conf + if: |- + always() + && (steps.step-7.outcome == 'success') + - id: 'step-9' + name: 'Test - C: should be mounted at /c' + shell: 'wsl-bash {0}' + run: |- + mount + mount | grep 'C:.* on /c' + if: |- + always() + && (steps.step-7.outcome == 'success') + test_additional_packages: + name: 'Test additional packages for "${{ matrix.distribution.user-id }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + - wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + - wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + - wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: '${{ matrix.distribution.default-absent-tool }} bash' + - id: 'step-2' + name: 'Test - ${{ matrix.distribution.default-absent-tool }} should be installed' + shell: 'wsl-bash {0}' + run: '${{ matrix.distribution.default-absent-tool }} --version' + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-3' + name: 'Test - bash should be installed' + shell: 'wsl-bash {0}' + run: 'bash -c true' + if: |- + always() + && (steps.step-1.outcome == 'success') + test_multiple_usage_with_different_distributions: + name: 'Test multiple usage with different distributions ("${{ matrix.distributions.distribution1.user-id }}" / "${{ matrix.distributions.distribution2.user-id }}" / "${{ matrix.distributions.distribution3.user-id }}") on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distributions: + - distribution1: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - distribution1: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - distribution1: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - distribution1: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - distribution1: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - distribution1: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action for ${{ matrix.distributions.distribution1.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution1.user-id }}' + - id: 'step-2' + name: 'Execute action for ${{ matrix.distributions.distribution2.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution2.user-id }}' + - id: 'step-3' + name: 'Execute action for ${{ matrix.distributions.distribution3.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution3.user-id }}' + set-as-default: 'false' + - id: 'step-4' + name: 'Execute action for ${{ matrix.distributions.distribution1.user-id }} again' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution1.user-id }}' + - id: 'step-5' + name: 'Test - the default distribution should be the last installed distribution with set-as-default true' + shell: 'wsl-bash {0}' + run: |- + cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8) + [[ "$(cat <(wsl.exe --list || true) <(wsl.exe --list || true | iconv -f UTF-16LE -t UTF-8) <(wslconfig.exe /list || true) <(wslconfig.exe /list || true | iconv -f UTF-16LE -t UTF-8))" == *${{ matrix.distributions.distribution2.wsl-id }}\ \(Default\)* ]] + if: |- + always() + && (steps.step-4.outcome == 'success') + - id: 'step-6' + name: 'Test - wsl-bash should use the last installed distribution with set-as-default true' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution2.match-pattern }} ]] + if: |- + always() + && (steps.step-4.outcome == 'success') + test_multiple_usage_with_same_distribution: + name: 'Test multiple usage with "${{ matrix.distribution.user-id }}" distribution on ${{ matrix.environment }}' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distribution: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + - wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + - wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + - wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + - wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + distribution2: + - wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + exclude: + - environment: 'windows-2019' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - environment: 'windows-2022' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + - environment: 'windows-latest' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + include: + - environment: 'windows-2019' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - environment: 'windows-2022' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + - environment: 'windows-latest' + distribution: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: 'bash' + - id: 'step-2' + name: 'Update distribution' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + update: 'true' + if: 'matrix.distribution.user-id != ''kali-linux''' + - id: 'step-3' + name: 'Install default absent tool' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + additional-packages: '${{ matrix.distribution.default-absent-tool }}' + - id: 'step-4' + name: 'Test - ${{ matrix.distribution.default-absent-tool }} should be installed' + shell: 'wsl-bash {0}' + run: '${{ matrix.distribution.default-absent-tool }} --version' + if: |- + always() + && (steps.step-3.outcome == 'success') + - id: 'step-5' + name: 'Execute action for ${{ matrix.distribution2.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distribution2.user-id }}' + if: |- + always() + && (steps.step-3.outcome == 'success') + - id: 'step-6' + name: 'Test - "${{ matrix.distribution2.user-id }}" should be the default distribution after installation' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distribution2.match-pattern }} ]] + if: |- + always() + && (steps.step-5.outcome == 'success') + - id: 'step-7' + name: 'Re-execute action' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + if: |- + always() + && (steps.step-5.outcome == 'success') + - id: 'step-8' + name: 'Test - "${{ matrix.distribution2.user-id }}" should still be the default distribution after re-running for "${{ matrix.distribution.user-id }}"' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distribution2.match-pattern }} ]] + if: |- + always() + && (steps.step-7.outcome == 'success') + - id: 'step-9' + name: 'Set as default' + uses: './' + with: + distribution: '${{ matrix.distribution.user-id }}' + set-as-default: 'true' + if: |- + always() + && (steps.step-7.outcome == 'success') + - id: 'step-10' + name: 'Test - "${{ matrix.distribution.user-id }}" should be the default distribution after re-running with set-as-default true' + shell: 'wsl-bash {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distribution.match-pattern }} ]] + if: |- + always() + && (steps.step-9.outcome == 'success') + test_distribution_specific_wsl_bash_scripts: + name: 'Test distribution specific wsl-bash scripts on ${{ matrix.environment }} (without ${{ matrix.distributions.incompatibleUbuntu }})' + runs-on: '${{ matrix.environment }}' + needs: + - 'build' + - 'check_yaml_consistency' + strategy: + fail-fast: false + matrix: + environment: + - 'windows-2019' + - 'windows-2022' + - 'windows-latest' + distributions: + - incompatibleUbuntu: 'Ubuntu-22.04' + distribution1: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + distribution4: + wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + distribution5: + wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + distribution6: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-20.04' + match-pattern: '*Ubuntu*20.04*' + default-absent-tool: 'dos2unix' + distribution7: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution8: + wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + - incompatibleUbuntu: 'Ubuntu-20.04' + distribution1: + wsl-id: 'Debian' + user-id: 'Debian' + match-pattern: '*Debian*' + default-absent-tool: 'dos2unix' + distribution2: + wsl-id: 'Alpine' + user-id: 'Alpine' + match-pattern: '*Alpine*' + default-absent-tool: 'dos2unix' + distribution3: + wsl-id: 'kali-linux' + user-id: 'kali-linux' + match-pattern: '*Kali*' + default-absent-tool: 'dos2unix' + distribution4: + wsl-id: 'openSUSE-Leap-15.2' + user-id: 'openSUSE-Leap-15.2' + match-pattern: '*openSUSE*Leap*15.2*' + default-absent-tool: 'which' + distribution5: + wsl-id: 'Ubuntu-24.04' + user-id: 'Ubuntu-24.04' + match-pattern: '*Ubuntu*24.04*' + default-absent-tool: 'dos2unix' + distribution6: + wsl-id: 'Ubuntu' + user-id: 'Ubuntu-22.04' + match-pattern: '*Ubuntu*22.04*' + default-absent-tool: 'dos2unix' + distribution7: + wsl-id: 'Ubuntu-18.04' + user-id: 'Ubuntu-18.04' + match-pattern: '*Ubuntu*18.04*' + default-absent-tool: 'dos2unix' + distribution8: + wsl-id: 'Ubuntu-16.04' + user-id: 'Ubuntu-16.04' + match-pattern: '*Ubuntu*16.04*' + default-absent-tool: 'dos2unix' + steps: + - id: 'step-0' + name: 'Restore built artifacts from cache' + uses: 'actions/cache/restore@v4' + with: + path: |- + action.yml + build/distributions/ + key: '${{ github.run_id }}' + fail-on-cache-miss: 'true' + - id: 'step-1' + name: 'Execute action for ${{ matrix.distributions.distribution1.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution1.user-id }}' + - id: 'step-2' + name: 'Execute action for ${{ matrix.distributions.distribution2.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution2.user-id }}' + additional-packages: 'bash' + - id: 'step-3' + name: 'Execute action for ${{ matrix.distributions.distribution3.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution3.user-id }}' + set-as-default: 'false' + - id: 'step-4' + name: 'Execute action for ${{ matrix.distributions.distribution4.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution4.user-id }}' + set-as-default: 'false' + - id: 'step-5' + name: 'Execute action for ${{ matrix.distributions.distribution5.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution5.user-id }}' + set-as-default: 'false' + - id: 'step-6' + name: 'Execute action for ${{ matrix.distributions.distribution6.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution6.user-id }}' + set-as-default: 'false' + - id: 'step-7' + name: 'Execute action for ${{ matrix.distributions.distribution7.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution7.user-id }}' + set-as-default: 'false' + - id: 'step-8' + name: 'Execute action for ${{ matrix.distributions.distribution8.user-id }}' + uses: './' + with: + distribution: '${{ matrix.distributions.distribution8.user-id }}' + set-as-default: 'false' + - id: 'step-9' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution1.user-id }} should use the correct distribution' + shell: 'wsl-bash_Debian {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution1.match-pattern }} ]] + if: |- + always() + && (steps.step-1.outcome == 'success') + - id: 'step-10' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution2.user-id }} should use the correct distribution' + shell: 'wsl-bash_Alpine {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution2.match-pattern }} ]] + if: |- + always() + && (steps.step-2.outcome == 'success') + - id: 'step-11' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution3.user-id }} should use the correct distribution' + shell: 'wsl-bash_kali-linux {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution3.match-pattern }} ]] + if: |- + always() + && (steps.step-3.outcome == 'success') + - id: 'step-12' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution4.user-id }} should use the correct distribution' + shell: 'wsl-bash_openSUSE-Leap-15.2 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution4.match-pattern }} ]] + if: |- + always() + && (steps.step-4.outcome == 'success') + - id: 'step-13' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution5.user-id }} should use the correct distribution' + shell: 'wsl-bash_Ubuntu-24.04 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution5.match-pattern }} ]] + if: |- + always() + && (steps.step-5.outcome == 'success') + - id: 'step-14' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution6.user-id }} should use the correct distribution' + shell: 'wsl-bash_Ubuntu-22.04 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution6.match-pattern }} ]] + if: |- + always() + && (steps.step-6.outcome == 'success') + && (matrix.distributions.distribution6.user-id != 'Ubuntu-20.04') + - id: 'step-15' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution6.user-id }} should use the correct distribution' + shell: 'wsl-bash_Ubuntu-20.04 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution6.match-pattern }} ]] + if: |- + always() + && (steps.step-6.outcome == 'success') + && (matrix.distributions.distribution6.user-id != 'Ubuntu-22.04') + - id: 'step-16' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution7.user-id }} should use the correct distribution' + shell: 'wsl-bash_Ubuntu-18.04 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution7.match-pattern }} ]] + if: |- + always() + && (steps.step-7.outcome == 'success') + - id: 'step-17' + name: 'Test - wsl-bash_${{ matrix.distributions.distribution8.user-id }} should use the correct distribution' + shell: 'wsl-bash_Ubuntu-16.04 {0}' + run: |- + cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true) + [[ "$(cat <(lsb_release -a || true) <(uname -a || true) <([ -d /etc ] && find /etc -maxdepth 1 -type f \( -name '*release' -or -name 'issue*' \) -exec cat {} + || true) <([ -d /etc/products.d ] && find /etc/products.d -maxdepth 1 -type f -name '*.prod' -exec cat {} + || true) <([ -f /proc/version ] && cat /proc/version || true))" == ${{ matrix.distributions.distribution8.match-pattern }} ]] + if: |- + always() + && (steps.step-8.outcome == 'success') diff --git a/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.main.kts b/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.main.kts new file mode 100755 index 000000000000..91fa1ad48d3e --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.main.kts @@ -0,0 +1,53 @@ +#!/usr/bin/env kotlin + +/* + * Copyright 2020-2023 Björn Kautler + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:Import("workflow-with-copyright.main.kts") + +import io.github.typesafegithub.workflows.actions.actions.CheckoutV3 +import io.github.typesafegithub.workflows.actions.gradle.WrapperValidationActionV1 +import io.github.typesafegithub.workflows.domain.RunnerType.UbuntuLatest +import io.github.typesafegithub.workflows.domain.triggers.PullRequest +import io.github.typesafegithub.workflows.domain.triggers.Push + +workflowWithCopyright( + name = "Validate Gradle Wrapper", + on = listOf( + Push(), + PullRequest() + ), + sourceFile = __FILE__ +) { + job( + id = "validate_gradle_wrapper", + name = "Validate Gradle Wrapper", + runsOn = UbuntuLatest + ) { + run( + name = "Configure Git", + command = "git config --global core.autocrlf input" + ) + uses( + name = "Checkout", + action = CheckoutV3() + ) + uses( + name = "Validate Gradle Wrapper", + action = WrapperValidationActionV1() + ) + } +} diff --git a/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.yaml b/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.yaml new file mode 100644 index 000000000000..9c956b95324d --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/validate-gradle-wrapper.yaml @@ -0,0 +1,51 @@ +# Copyright 2020-2023 Björn Kautler +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This file was generated using Kotlin DSL (.github/workflows/validate-gradle-wrapper.main.kts). +# If you want to modify the workflow, please change the Kotlin file and regenerate this YAML file. +# Generated with https://github.com/typesafegithub/github-workflows-kt + +name: 'Validate Gradle Wrapper' +on: + push: {} + pull_request: {} +jobs: + check_yaml_consistency: + name: 'Check YAML consistency' + runs-on: 'ubuntu-latest' + steps: + - id: 'step-0' + name: 'Check out' + uses: 'actions/checkout@v4' + - id: 'step-1' + name: 'Execute script' + run: 'rm ''.github/workflows/validate-gradle-wrapper.yaml'' && ''.github/workflows/validate-gradle-wrapper.main.kts''' + - id: 'step-2' + name: 'Consistency check' + run: 'git diff --exit-code ''.github/workflows/validate-gradle-wrapper.yaml''' + validate_gradle_wrapper: + name: 'Validate Gradle Wrapper' + runs-on: 'ubuntu-latest' + needs: + - 'check_yaml_consistency' + steps: + - id: 'step-0' + name: 'Configure Git' + run: 'git config --global core.autocrlf input' + - id: 'step-1' + name: 'Checkout' + uses: 'actions/checkout@v3' + - id: 'step-2' + name: 'Validate Gradle Wrapper' + uses: 'gradle/wrapper-validation-action@v1' diff --git a/.github/workflows/reserves/setup-wsl/workflow-with-copyright.main.kts b/.github/workflows/reserves/setup-wsl/workflow-with-copyright.main.kts new file mode 100644 index 000000000000..b636366e73f4 --- /dev/null +++ b/.github/workflows/reserves/setup-wsl/workflow-with-copyright.main.kts @@ -0,0 +1,59 @@ +/* + * Copyright 2020-2023 Björn Kautler + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +@file:DependsOn("io.github.typesafegithub:github-workflows-kt:2.0.0") + +import io.github.typesafegithub.workflows.domain.Concurrency +import io.github.typesafegithub.workflows.domain.triggers.Trigger +import io.github.typesafegithub.workflows.dsl.WorkflowBuilder +import io.github.typesafegithub.workflows.dsl.workflow +import io.github.typesafegithub.workflows.yaml.Preamble.WithOriginalAfter +import java.io.File + +fun workflowWithCopyright( + name: String, + on: List, + env: Map = mapOf(), + sourceFile: File, + concurrency: Concurrency? = null, + block: WorkflowBuilder.() -> Unit +) { + workflow( + name = name, + on = on, + env = env, + sourceFile = sourceFile, + concurrency = concurrency, + preamble = WithOriginalAfter( + """ + Copyright 2020-2023 Björn Kautler + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + """.trimIndent() + ), + block = block + ) +} diff --git a/.github/workflows/reserves/stale.yml b/.github/workflows/reserves/stale.yml new file mode 100644 index 000000000000..c6d0c2fb173f --- /dev/null +++ b/.github/workflows/reserves/stale.yml @@ -0,0 +1,18 @@ +name: 'Close stale PRs' +on: + schedule: + - cron: '30 1 * * *' + +permissions: + pull-requests: write + +jobs: + stale: + runs-on: ubuntu-latest + steps: + - uses: actions/stale@v9 + with: + stale-pr-message: 'This PR has been automarked as stale for 30 days of inactivity, and will autoclose if still inactive in 5 days.' + close-pr-message: 'Autoclosing this stale PR.' + days-before-stale: 30 + days-before-close: 5 diff --git a/.github/workflows/reserves/tensorflow-io/api.yml b/.github/workflows/reserves/tensorflow-io/api.yml new file mode 100644 index 000000000000..60b0f61c78e7 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/api.yml @@ -0,0 +1,150 @@ +name: API Compatibility + +on: + push: + branches: + - master + schedule: + - cron: "0 12 * * *" + +permissions: + contents: read + +jobs: + + macos: + if: github.repository == 'tensorflow/io' # Don't do this in forks + name: macOS ${{ matrix.python }} + ${{ matrix.version }} + runs-on: macos-latest + strategy: + fail-fast: false + matrix: + python: ['3.8'] + version: ['tensorflow:tensorflow-io-nightly', 'tf-nightly:tensorflow-io', 'tf-nightly:tensorflow-io-nightly'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - run: | + set -x -e + python -m pip install -U wheel setuptools + python --version + - name: Setup macOS + run: | + set -x -e + bash -x -e tests/test_azure/start_azure.sh + - name: Test macOS + run: | + set -x -e + python --version + df -h + rm -rf tensorflow_io + rm -rf tensorflow_io_gcs_filesystem + export TF_USE_MODULAR_FILESYSTEM=1 + echo ${{ matrix.version }} | awk -F: '{print $1}' | xargs python -m pip install -U + echo ${{ matrix.version }} | awk -F: '{print $2}' | xargs python -m pip install --no-deps -U + python -m pip install tensorflow-io-gcs-filesystem-nightly + python -m pip install pytest-benchmark boto3 google-cloud-storage==1.32.0 azure-storage-blob==12.7.1 pyarrow==3.0.0 + python -m pip freeze + python -c 'import tensorflow as tf; print(tf.version.VERSION)' + python -c 'import tensorflow_io as tfio; print(tfio.version.VERSION)' + python -m pytest -s -v tests/test_http.py + python -m pytest -s -v tests/test_s3.py + python -m pytest -s -v tests/test_azure.py + python -m pytest -s -v tests/test_gcs.py + python -m pytest -s -v tests/test_standalone_filesystem.py + + linux: + if: github.repository == 'tensorflow/io' # Don't do this in forks + name: Linux ${{ matrix.python }} + ${{ matrix.version }} + runs-on: ubuntu-20.04 + strategy: + fail-fast: false + matrix: + python: ['3.8'] + version: ['tensorflow:tensorflow-io-nightly', 'tf-nightly:tensorflow-io', 'tf-nightly:tensorflow-io-nightly'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Setup Linux + run: | + set -x -e + bash -x -e .github/workflows/build.space.sh + bash -x -e tests/test_aws/aws_test.sh + bash -x -e tests/test_azure/start_azure.sh + bash -x -e tests/test_gcloud/test_gcs.sh gcs-emulator + bash -x -e tests/test_hdfs/hdfs_test.sh + - name: Test Linux + run: | + set -x -e + python --version + df -h + rm -rf tensorflow_io + rm -rf tensorflow_io_gcs_filesystem + export TF_USE_MODULAR_FILESYSTEM=1 + echo ${{ matrix.version }} | awk -F: '{print $1}' | xargs python -m pip install -U + echo ${{ matrix.version }} | awk -F: '{print $2}' | xargs python -m pip install --no-deps -U + python -m pip install tensorflow-io-gcs-filesystem-nightly + python -m pip install pytest-benchmark boto3 google-cloud-storage==1.32.0 azure-storage-blob==12.7.1 pyarrow==3.0.0 + python -m pip freeze + + sudo apt-get -y -qq update + sudo apt-get -y -qq install openjdk-8-jdk + sudo update-alternatives --config java + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + curl -OL https://archive.apache.org/dist/hadoop/common/hadoop-2.7.0/hadoop-2.7.0.tar.gz + sudo tar -xzf hadoop-2.7.0.tar.gz -C /usr/local + sudo chmod 755 -R /usr/local/hadoop-2.7.0 + export HADOOP_HOME=/usr/local/hadoop-2.7.0 + export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${JAVA_HOME}/jre/lib/amd64/server:${HADOOP_HOME}/lib/native + export CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob) + + python -c 'import tensorflow as tf; print(tf.version.VERSION)' + python -c 'import tensorflow_io as tfio; print(tfio.version.VERSION)' + python -m pytest -s -v tests/test_http.py + python -m pytest -s -v tests/test_s3.py + python -m pytest -s -v tests/test_azure.py + python -m pytest -s -v tests/test_gcs.py + python -m pytest -s -v tests/test_standalone_filesystem.py + + windows: + if: github.repository == 'tensorflow/io' # Don't do this in forks + name: Windows ${{ matrix.python }} + ${{ matrix.version }} + runs-on: windows-latest + strategy: + fail-fast: false + matrix: + python: ['3.8'] + version: ['tensorflow:tensorflow-io-nightly', 'tf-nightly:tensorflow-io', 'tf-nightly:tensorflow-io-nightly'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2 + with: + node-version: '8.x' + - name: Setup Windows + shell: cmd + run: | + @echo on + bash -x -e tests/test_azure/start_azure.sh + - name: Test Windows + shell: cmd + run: | + @echo on + python --version + rm -rf tensorflow_io + rm -rf tensorflow_io_gcs_filesystem + export TF_USE_MODULAR_FILESYSTEM=1 + echo ${{ matrix.version }} | awk -F: '{print $1}' | xargs python -m pip install -U + echo ${{ matrix.version }} | awk -F: '{print $2}' | xargs python -m pip install --no-deps -U + python -m pip install tensorflow-io-gcs-filesystem-nightly + python -m pip install pytest-benchmark + python -m pip freeze + python -c 'import tensorflow as tf; print(tf.version.VERSION)' + python -c 'import tensorflow_io as tfio; print(tfio.version.VERSION)' + python -m pytest -s -v tests/test_http.py -k remote diff --git a/.github/workflows/reserves/tensorflow-io/benchmarks.yml b/.github/workflows/reserves/tensorflow-io/benchmarks.yml new file mode 100644 index 000000000000..5ce00d202525 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/benchmarks.yml @@ -0,0 +1,96 @@ +name: API Performance Benchmarks + +on: + push: + branches: + - master + +permissions: + contents: read + +jobs: + macos: + if: github.repository == 'tensorflow/io' # Don't do this in forks + name: macOS ${{ matrix.python }} + ${{ matrix.version }} + runs-on: macos-latest + strategy: + fail-fast: false + matrix: + python: ['3.8'] + version: ['tensorflow:tensorflow-io-nightly', 'tensorflow:tensorflow-io'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Setup macOS + run: | + set -x -e + python -m pip install -U wheel setuptools + python --version + - name: Benchmark on macOS + run: | + set -x -e + python --version + df -h + rm -rf tensorflow_io + rm -rf tensorflow_io_gcs_filesystem + export TF_USE_MODULAR_FILESYSTEM=1 + echo ${{ matrix.version }} | awk -F: '{print $1}' | xargs python -m pip install -U + echo ${{ matrix.version }} | awk -F: '{print $2}' | xargs python -m pip install --no-deps -U + python -m pip install tensorflow-io-gcs-filesystem-nightly + python -m pip install -q scikit-image pytest pytest-benchmark boto3 fastavro avro-python3 scikit-image pandas pyarrow==2.0.0 google-cloud-pubsub==2.1.0 google-cloud-bigquery-storage==1.1.0 google-cloud-bigquery==2.3.1 google-cloud-storage==1.32.0 google-cloud-bigtable==1.6.0 azure-storage-blob==12.7.1 + python -m pip freeze + python -c 'import tensorflow as tf; print(tf.version.VERSION)' + python -c 'import tensorflow_io as tfio; print(tfio.version.VERSION)' + python -c 'from tensorflow_io_gcs_filesystem.core.python.ops import plugin_gs; print(plugin_gs)' + python -m pytest --benchmark-only -v --import-mode=append $(find . -type f \( -iname "test_*.py" ! \( -iname "test_*_v1.py" -o -iname "test_bigquery.py" \) \)) + + linux: + if: github.repository == 'tensorflow/io' # Don't do this in forks + name: Linux ${{ matrix.python }} + ${{ matrix.version }} + runs-on: ubuntu-20.04 + permissions: + contents: write # to allow sending a commit comment for the benchmark action + strategy: + fail-fast: false + matrix: + python: ['3.8'] + version: ['tensorflow:tensorflow-io-nightly', 'tensorflow:tensorflow-io'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + with: + fetch-depth: 0 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Setup Linux + run: | + set -x -e + bash -x -e .github/workflows/build.space.sh + bash -x -e tests/test_sql/sql_test.sh + - name: Benchmark on Linux + run: | + set -x -e + python --version + df -h + rm -rf tensorflow_io + rm -rf tensorflow_io_gcs_filesystem + export TF_USE_MODULAR_FILESYSTEM=1 + echo ${{ matrix.version }} | awk -F: '{print $1}' | xargs python -m pip install -U + echo ${{ matrix.version }} | awk -F: '{print $2}' | xargs python -m pip install --no-deps -U + python -m pip install tensorflow-io-gcs-filesystem-nightly + python -m pip install -q scikit-image pytest pytest-benchmark boto3 fastavro avro-python3 scikit-image pandas pyarrow==2.0.0 google-cloud-pubsub==2.1.0 google-cloud-bigquery-storage==1.1.0 google-cloud-bigquery==2.3.1 google-cloud-storage==1.32.0 google-cloud-bigtable==1.6.0 azure-storage-blob==12.7.1 + python -m pip freeze + python -c 'import tensorflow as tf; print(tf.version.VERSION)' + python -c 'import tensorflow_io as tfio; print(tfio.version.VERSION)' + python -c 'from tensorflow_io_gcs_filesystem.core.python.ops import plugin_gs; print(plugin_gs)' + python -m pytest --benchmark-only --benchmark-json benchmark.json -v --import-mode=append $(find . -type f \( -iname "test_*.py" ! \( -iname "test_*_v1.py" -o -iname "test_bigquery.py" \) \)) + - name: Store benchmark result + uses: rhysd/github-action-benchmark@4de1bed97a47495fc4c5404952da0499e31f5c29 # v1.20.3 + with: + name: Tensorflow-IO Benchmarks + tool: 'pytest' + output-file-path: benchmark.json + github-token: ${{ secrets.GITHUB_TOKEN }} + auto-push: true diff --git a/.github/workflows/reserves/tensorflow-io/build.arm64.yml b/.github/workflows/reserves/tensorflow-io/build.arm64.yml new file mode 100644 index 000000000000..e638756fcce9 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.arm64.yml @@ -0,0 +1,145 @@ +name: Release arm64 on macOS + +on: + workflow_dispatch: + +permissions: + contents: read + +jobs: + macos-arm64-bazel: + name: Bazel arm64 on macOS + runs-on: macos-13-xlarge + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + with: + fetch-depth: 0 + - name: Bazel on macOS + run: | + set -x -e + bash -e .github/workflows/build.bazel.sh python3 + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-arm64-bazel-bin + path: | + build/tensorflow_io + build/tensorflow_io_gcs_filesystem + - run: | + set -x -e + echo "Successfully completely macOS arm64 release" + + macos-arm64-wheel: + name: Wheel ${{ matrix.python }} macOS arm64 + needs: macos-arm64-bazel + runs-on: macos-13-xlarge + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-arm64-bazel-bin + path: bazel-bin + #- uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5.0.0 + # with: + # python-version: "${{ matrix.python }}" + - name: Wheel ${{ matrix.python }} macOS + run: | + set -x -e + python3 --version + python --version + brew search python + brew install python@${{ matrix.python }} + python${{ matrix.python }} --version + python${{ matrix.python }} -m pip install --break-system-packages -U wheel setuptools + python${{ matrix.python }} setup.py --data bazel-bin -q bdist_wheel --plat-name macosx_12_0_arm64 + rm -rf build + python${{ matrix.python }} setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel --plat-name macosx_12_0_arm64 + ls dist/* + mkdir -p wheelhouse + for f in dist/*.whl; do + echo "disable delocate due to build breaks: delocate-wheel -w wheelhouse $f" + cp $f wheelhouse + done + ls wheelhouse/* + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-arm64-${{ matrix.python }}-wheel + path: wheelhouse + + release-macos-arm64: + name: Release (MacOS arm64) + needs: macos-arm64-wheel + runs-on: ubuntu-20.04 + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-arm64-3.9-wheel + path: macOS-arm64-3.9-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-arm64-3.10-wheel + path: macOS-arm64-3.10-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-arm64-3.11-wheel + path: macOS-arm64-3.11-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-arm64-3.12-wheel + path: macOS-arm64-3.12-wheel + - run: | + set -e -x + mkdir -p wheelhouse + cp macOS-arm64-3.9-wheel/*.whl wheelhouse/ + cp macOS-arm64-3.10-wheel/*.whl wheelhouse/ + cp macOS-arm64-3.11-wheel/*.whl wheelhouse/ + cp macOS-arm64-3.12-wheel/*.whl wheelhouse/ + ls -la wheelhouse/ + sha256sum wheelhouse/*.whl + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: tensorflow-io-release-macos-arm64 + path: wheelhouse + + release-candidate-macos-arm64: + name: Release Candidate (MacOS arm64) + needs: release-macos-arm64 + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: tensorflow-io-release-macos-arm64 + path: wheelhouse + - run: | + set -x -e + sha256sum wheelhouse/*.whl | sort -u > wheelhouse.sha256 + git rev-parse --verify HEAD > wheelhouse.commit + grep 'version = ' tensorflow_io/python/ops/version_ops.py | sed -E 's@^.*version = "@@g' | sed -E 's@".*$@@g' > wheelhouse.version + cat wheelhouse.sha256 + cat wheelhouse.commit + cat wheelhouse.version + echo "::set-output name=version::$(cat wheelhouse.version)" + id: info + - run: |- + cat < Dockerfile + FROM gcr.io/distroless/base-debian11 + COPY wheelhouse /wheelhouse + COPY wheelhouse.sha256 /wheelhouse.sha256 + COPY wheelhouse.commit /wheelhouse.commit + COPY wheelhouse.version /wheelhouse.version + EOF + - uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0 + - uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0 + with: + username: tfsigio + password: ${{ secrets.DOCKER_PASSWORD }} + - uses: docker/build-push-action@15560696de535e4014efeff63c48f16952e52dd1 # v6.2.0 + with: + context: . + push: true + tags: tfsigio/candidate:macos-arm64-${{ steps.info.outputs.version }} + + diff --git a/.github/workflows/reserves/tensorflow-io/build.bazel.sh b/.github/workflows/reserves/tensorflow-io/build.bazel.sh new file mode 100755 index 000000000000..de88d41373ba --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.bazel.sh @@ -0,0 +1,101 @@ +#!/usr/bin/env bash +# Copyright 2018 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ============================================================================== +set -x + + +PYTHON=python3 +if [[ $# == 1 ]]; then + PYTHON=$1 +fi +$PYTHON --version + + +export PYTHON_BIN_PATH=`which $PYTHON` + +if [[ $(uname) == "Linux" ]]; then + curl -sSOL https://github.com/bazelbuild/bazelisk/releases/download/v1.11.0/bazelisk-linux-amd64 + mv bazelisk-linux-amd64 /usr/local/bin/bazel + chmod +x /usr/local/bin/bazel + cp -f $(which $PYTHON) /usr/bin/python3 +fi + +bazel version + +$PYTHON -m pip --version + +if [[ $(uname) == "Darwin" && $(uname -m) == "arm64" ]]; then +$PYTHON -m pip install --upgrade --break-system-packages pip +$PYTHON -m pip install --upgrade --break-system-packages setuptools +$PYTHON -m pip --version + +export TENSORFLOW_INSTALL="$($PYTHON setup.py --install-require)" + +$PYTHON -m pip install --break-system-packages ${TENSORFLOW_INSTALL} +$PYTHON -m pip install --break-system-packages "urllib3 <2" +$PYTHON -m pip uninstall --break-system-packages -y tensorflow-io-gcs-filesystem +else +$PYTHON -m pip install --upgrade pip +$PYTHON -m pip install --upgrade setuptools +$PYTHON -m pip --version + +export TENSORFLOW_INSTALL="$($PYTHON setup.py --install-require)" + +$PYTHON -m pip install -q ${TENSORFLOW_INSTALL} +$PYTHON -m pip install -q "urllib3 <2" +$PYTHON -m pip uninstall -y tensorflow-io-gcs-filesystem +fi + + +$PYTHON tools/build/configure.py + +cat .bazelrc + +if [[ $(uname -m) != "arm64" && $(uname) == "Darwin" ]]; then + +bazel build \ + ${BAZEL_OPTIMIZATION} \ + -- //tensorflow_io:python/ops/libtensorflow_io.so //tensorflow_io:python/ops/libtensorflow_io_plugins.so //tensorflow_io_gcs_filesystem/... + +elif [[ $(uname -m) == "arm64" && $(uname) == "Darwin" ]]; then + +bazel build \ + ${BAZEL_OPTIMIZATION} \ + -- //tensorflow_io_gcs_filesystem/... //tensorflow_io:python/ops/libtensorflow_io.so //tensorflow_io:python/ops/libtensorflow_io_plugins.so + +else + +bazel build \ + ${BAZEL_OPTIMIZATION} \ + -- //tensorflow_io/... //tensorflow_io_gcs_filesystem/... + +fi + +rm -rf build && mkdir -p build + +if [[ $(uname) == "Linux" ]]; then +cp -r -L bazel-bin/tensorflow_io build/tensorflow_io +cp -r -L bazel-bin/tensorflow_io_gcs_filesystem build/tensorflow_io_gcs_filesystem +else +cp -r bazel-bin/tensorflow_io build/tensorflow_io +cp -r bazel-bin/tensorflow_io_gcs_filesystem build/tensorflow_io_gcs_filesystem +fi + +echo chown -R $(id -nu):$(id -ng) build/tensorflow_io/ +echo chown -R $(id -nu):$(id -ng) build/tensorflow_io_gcs_filesystem/ +find build/tensorflow_io -name '*runfiles*' | xargs rm -rf +find build/tensorflow_io_gcs_filesystem -name '*runfiles*' | xargs rm -rf + +exit 0 diff --git a/.github/workflows/reserves/tensorflow-io/build.gpu.sh b/.github/workflows/reserves/tensorflow-io/build.gpu.sh new file mode 100644 index 000000000000..06ca5331b389 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.gpu.sh @@ -0,0 +1,82 @@ +#!/usr/bin/env bash +# Copyright 2019 The TensorFlow Authors. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# ============================================================================== +# Make sure we're in the project root path. +SCRIPT_DIR=$( cd ${0%/*} && pwd -P ) +ROOT_DIR=$( cd "$SCRIPT_DIR/../.." && pwd -P ) +if [[ ! -d "tensorflow_io" ]]; then + echo "ERROR: PWD: $PWD is not project root" + exit 1 +fi + +set -x + +PLATFORM="$(uname -s | tr 'A-Z' 'a-z')" + +if [[ ${PLATFORM} == "darwin" ]]; then + N_JOBS=$(sysctl -n hw.ncpu) +else + N_JOBS=$(grep -c ^processor /proc/cpuinfo) +fi + +echo "" +echo "Bazel will use ${N_JOBS} concurrent job(s)." +echo "" + +export CC_OPT_FLAGS='-mavx' +export TF_NEED_CUDA=0 # TODO: Verify this is used in GPU custom-op + +export PYTHON_BIN_PATH=`which python` + +lsb_release -a +ls -la /usr/bin/python* +python --version +python -m pip --version +docker --version + +echo Disabled for now. +exit 0 +if [[ $(python -c "import sys;print(sys.version[0])") == "2" ]]; then + echo Python 2 has been deprecated. + exit 0 +fi + +if [[ $(python3 -c 'import sys;print("{}.{}".format(sys.version_info[0],sys.version_info[1]))') == "3.4" ]]; then + echo Python 3.4 has been deprecated. + exit 0 +fi + +export CC_OPT_FLAGS='-mavx' +export TF_NEED_CUDA="1" +export TF_CUDA_VERSION="10.1" +export CUDA_TOOLKIT_PATH="/usr/local/cuda" +export TF_CUDNN_VERSION="7" +export CUDNN_INSTALL_PATH="/usr/lib/x86_64-linux-gnu" + +python3 --version +python3 -m pip --version + +python3 -m pip install $(python3 setup.py --install-require) +python3 tools/build/configure.py --cuda + +cat .bazelrc + +bazel build -s --verbose_failures -c opt -k \ + --jobs=${N_JOBS} \ + --config=linux_ci_gpu \ + //tensorflow_io:python/ops/libtensorflow_io.so + +exit $? diff --git a/.github/workflows/reserves/tensorflow-io/build.instruction.py b/.github/workflows/reserves/tensorflow-io/build.instruction.py new file mode 100644 index 000000000000..75e8aedadd41 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.instruction.py @@ -0,0 +1,24 @@ +import sys + +sudo = False +if sys.argv[1].startswith("--sudo="): + sudo = sys.argv[1][7:].lower() == "true" + +source = sys.argv[len(sys.argv) - 2] +section = sys.argv[len(sys.argv) - 1] +with open(source, "r") as f: + lines = [line.rstrip() for line in list(f)] + +# Remove lines before section title +lines = lines[lines.index(section) :] + +# Remove lines outside (including) "```sh" and "```" +lines = lines[lines.index("```sh") + 1 : lines.index("```")] + +# Remove sudo +if not sudo: + lines = [ + (line[len("sudo ") :] if line.startswith("sudo ") else line) for line in lines + ] + +print("\n".join(lines)) diff --git a/.github/workflows/reserves/tensorflow-io/build.space.sh b/.github/workflows/reserves/tensorflow-io/build.space.sh new file mode 100644 index 000000000000..4f47898c5c6b --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.space.sh @@ -0,0 +1,9 @@ +# Free disk space on Linux +#sudo swapoff /swapfile +#sudo rm -rf /swapfile +sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc +#sudo apt-get remove php* ruby-* subversion mongodb-org -yq >/dev/null 2>&1 +sudo apt-get autoremove -y >/dev/null 2>&1 +sudo apt-get autoclean -y >/dev/null 2>&1 +sudo rm -rf /usr/local/lib/android >/dev/null 2>&1 +docker rmi $(docker image ls -aq) >/dev/null 2>&1 diff --git a/.github/workflows/reserves/tensorflow-io/build.wheel.sh b/.github/workflows/reserves/tensorflow-io/build.wheel.sh new file mode 100755 index 000000000000..9296f877c5d9 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.wheel.sh @@ -0,0 +1,54 @@ +set -e + +export TF_AZURE_USE_DEV_STORAGE=1 +export TF_USE_MODULAR_FILESYSTEM=1 + +run_test() { + entry=$1 + CPYTHON_VERSION=$($entry -c 'import sys; print(str(sys.version_info[0])+str(sys.version_info[1]))') + TF_VERSION=$(/usr/bin/grep tensorflow tensorflow_io/python/ops/version_ops.py | /usr/bin/cut -d '"' -f 2) + $entry -m pip install $TF_VERSION pytest pytest-benchmark pytest-xdist==2.5.0 boto3 fastavro avro-python3 scikit-image pandas pyarrow==3.0.0 google-cloud-pubsub==2.1.0 google-cloud-bigtable==1.6.0 google-cloud-bigquery-storage==1.1.0 google-cloud-bigquery==2.3.1 google-cloud-storage==1.32.0 PyYAML==5.3.1 azure-storage-blob==12.8.1 azure-cli==2.29.0 + $entry -m pip uninstall -y tensorflow_io_gcs_filesystem + (cd wheelhouse && $entry -m pip install tensorflow_io_gcs_filesystem-*-cp${CPYTHON_VERSION}-*.whl) + (cd wheelhouse && $entry -m pip install tensorflow_io-*-cp${CPYTHON_VERSION}-*.whl) + $entry -m pip freeze + (cd tests && $entry -m pytest --benchmark-disable -v --import-mode=append --forked --numprocesses=auto --dist loadfile $(find . -type f \( -iname "test_*.py" ! \( -iname "test_standalone_*.py" \) \))) + (cd tests && $entry -m pytest --benchmark-disable -v --import-mode=append $(find . -type f \( -iname "test_standalone_*.py" \))) +} + +PYTHON_VERSION=python +if [[ "$#" -gt 0 ]]; then + PYTHON_VERSION="${1}" + shift +fi + +if [[ $(uname) == "Linux" ]]; then + apt-get -y -qq update + if [[ "${PYTHON_VERSION}" == "python3.9" ]]; then + apt-get install -y -qq software-properties-common + add-apt-repository -y ppa:deadsnakes/ppa + apt-get -y -qq update + apt-get -y -qq install python3.9-distutils + fi + apt-get -y -qq install $PYTHON_VERSION ffmpeg dnsutils libmp3lame0 + curl -sSOL https://bootstrap.pypa.io/get-pip.py + $PYTHON_VERSION get-pip.py -q + + # Install Java + apt-get -y -qq install openjdk-8-jdk + update-alternatives --config java + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + + # Install Hadoop + curl -OL https://archive.apache.org/dist/hadoop/common/hadoop-2.7.0/hadoop-2.7.0.tar.gz + tar -xzf hadoop-2.7.0.tar.gz -C /usr/local + export HADOOP_HOME=/usr/local/hadoop-2.7.0 + + # Update environmental variable + export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${JAVA_HOME}/jre/lib/amd64/server:${HADOOP_HOME}/lib/native + export CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob) + export +elif [[ $(uname) == "Darwin" ]]; then + brew install ffmpeg +fi +run_test $PYTHON_VERSION diff --git a/.github/workflows/reserves/tensorflow-io/build.yml b/.github/workflows/reserves/tensorflow-io/build.yml new file mode 100644 index 000000000000..c29b16f64fdb --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/build.yml @@ -0,0 +1,873 @@ +name: GitHub CI + +on: + push: + branches: + - master + pull_request: + branches: + - master + +permissions: + contents: read + +env: + REPO_NAME: ${{ github.repository }} + EVENT_NAME: ${{ github.event_name }} + +jobs: + lint: + name: Lint + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: Run Lint Script for Bazel/Pyupgrade/Black/Clang + run: | + set -x -e + git log --pretty -1 + sudo apt update -y -qq && sudo apt install -y -qq libtinfo5 + TF_PYTHON_VERSION=3.10 bazel run -s --verbose_failures --experimental_repo_remote_exec //tools/lint:check -- bazel clang + + lint-docs: + name: Lint for Docs + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: Run Lint Script for Docs + run: | + set -x -e + git log --pretty -1 + sudo python3 --version + sudo python3 -m pip install dataclasses + sudo python3 -m pip install setuptools + sudo python3 -m pip install -U git+https://github.com/tensorflow/docs + find docs -name '*.ipynb' | xargs python3 -m tensorflow_docs.tools.nbfmt + echo "Check for failed fmt: " + git status --untracked-files=no --porcelain + [ -z "$(git status --untracked-files=no --porcelain)" ] || exit 1 + find docs -name '*.ipynb' | xargs python3 -m tensorflow_docs.tools.nblint --arg=repo:tensorflow/io + echo "Check for failed lint: " + git status --untracked-files=no --porcelain + [ -z "$(git status --untracked-files=no --porcelain)" ] || exit 1 + + macos: + name: macOS + runs-on: macOS-12 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: GCP + run: | + cat > service_account_creds.json << EOF + ${{ secrets.GCP_CREDS }} + EOF + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: "3.9" + - name: macOS + run: | + set -x -e + export BAZEL_OPTIMIZATION="--config=cache" + if [[ "${EVENT_NAME}" == "push" && "${REPO_NAME}" == "tensorflow/io" ]]; then + export BAZEL_OPTIMIZATION="$BAZEL_OPTIMIZATION --remote_upload_local_results=true --google_credentials=service_account_creds.json" + fi + echo $PATH + which python3 + python3 --version + python3 -c 'import site; print(site.getsitepackages())' + python3 .github/workflows/build.instruction.py --sudo=true docs/development.md "#### macOS" > source.sh + bash -x -e source.sh + python3 -c 'import tensorflow as tf; print(tf.version.VERSION)' + + linux: + name: Linux + runs-on: ubuntu-22.04 + container: + image: ubuntu:22.04 + env: + REPO_NAME: ${{ env.REPO_NAME }} + EVENT_NAME: ${{ env.EVENT_NAME }} + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: GCP + run: | + cat > service_account_creds.json << EOF + ${{ secrets.GCP_CREDS }} + EOF + - name: Ubuntu 22.04 + shell: bash + run: | + set -x -e + export BAZEL_OPTIMIZATION="--config=cache" + if [[ "${EVENT_NAME}" == "push" && "${REPO_NAME}" == "tensorflow/io" ]]; then + export BAZEL_OPTIMIZATION="$BAZEL_OPTIMIZATION --remote_upload_local_results=true --google_credentials=service_account_creds.json" + fi + apt update + apt-get install -y python3 + python3 .github/workflows/build.instruction.py docs/development.md "##### Ubuntu 22.04" > source.sh + cat source.sh + bash -x -e source.sh + + macos-bazel: + name: Bazel macOS + runs-on: macOS-12 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: GCP + run: | + cat > service_account_creds.json << EOF + ${{ secrets.GCP_CREDS }} + EOF + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: "3.10" + - name: Bazel on macOS + run: | + set -x -e + export BAZEL_OPTIMIZATION="--config=optimization --config=cache" + if [[ "${EVENT_NAME}" == "push" && "${REPO_NAME}" == "tensorflow/io" ]]; then + export BAZEL_OPTIMIZATION="$BAZEL_OPTIMIZATION --remote_upload_local_results=true --google_credentials=service_account_creds.json" + fi + sudo -E -H bash -e .github/workflows/build.bazel.sh python3 + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-bazel-bin + path: | + build/tensorflow_io + build/tensorflow_io_gcs_filesystem + - name: Package pip on macOS + run: | + set -x -e + which python3 + + macos-wheel: + name: Wheel ${{ matrix.python }} macOS + needs: macos-bazel + runs-on: macOS-12 + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Wheel ${{ matrix.python }} macOS + run: | + set -x -e + python -m pip install -U wheel setuptools + python --version + python setup.py --data bazel-bin -q bdist_wheel --plat-name macosx_10_14_x86_64 + rm -rf build + python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel --plat-name macosx_10_14_x86_64 + - name: Auditwheel ${{ matrix.python }} macOS + run: | + set -x -e + python -m pip install twine delocate==0.10.3 + delocate-wheel --version + ls dist/* + mkdir -p wheelhouse + for f in dist/*.whl; do + echo "disable delocate due to build breaks: delocate-wheel -w wheelhouse $f" + cp $f wheelhouse + done + ls wheelhouse/* + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: wheelhouse + + macos-test: + name: Test ${{ matrix.python }} macOS + needs: macos-wheel + runs-on: macOS-12 + strategy: + matrix: + python: ['3.9'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: wheelhouse + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - run: | + set -x -e + python -m pip install pip==21.3.1 + python -m pip install -U wheel setuptools + python --version + - name: Setup ${{ matrix.python }} macOS + run: | + set -x -e + bash -x -e tests/test_kafka/kafka_test.sh + bash -x -e tests/test_azure/start_azure.sh + bash -x -e tests/test_gcloud/test_pubsub_bigtable.sh + bash -x -e tests/test_pulsar/pulsar_test.sh + - name: Install ${{ matrix.python }} macOS + run: | + set -x -e + python --version + df -h + (cd wheelhouse && python -m pip install tensorflow_io_gcs_filesystem-*.whl) + (cd wheelhouse && python -m pip install tensorflow_io-*.whl) + - name: Test ${{ matrix.python }} macOS + run: | + set -x -e + python --version + bash -x -e .github/workflows/build.wheel.sh python + + linux-bazel: + name: Bazel Linux + runs-on: ubuntu-22.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - name: GCP + run: | + cat > service_account_creds.json << EOF + ${{ secrets.GCP_CREDS }} + EOF + - name: Bazel on Linux + shell: bash + run: | + set -x + docker system prune --all --force + docker images + docker ps -a + df -h + # See https://github.com/actions/runner-images/issues/2840 + sudo rm -rf /usr/share/dotnet + sudo rm -rf /opt/ghc + sudo rm -rf "/usr/local/share/boost" + sudo rm -rf "$AGENT_TOOLSDIRECTORY" + df -h + # See https://github.com/actions/runner-images/issues/2606 + sudo rm -rf /usr/local/lib/android + df -h + sudo apt-get autoremove -y + sudo apt-get autoclean -y + df -h + #export BAZEL_OPTIMIZATION="--config=optimization --config=linux_ci --config=cache" + #if [[ "${{ env.EVENT_NAME }}" == "push" && "${{ env.REPO_NAME }}" == "tensorflow/io" ]]; then + # export BAZEL_OPTIMIZATION="${BAZEL_OPTIMIZATION} --remote_upload_local_results=true --google_credentials=service_account_creds.json" + #fi + TENSORFLOW_VERSION=$(grep tensorflow tensorflow_io/python/ops/version_ops.py | sed -e "s/require = //g" | awk -F, '{print $1}' | awk -F= '{print $2}' | awk -F. '{print $1"."$2}') + DOCKER_BUILDKIT=0 docker build -t io -f tools/docker/build.Dockerfile --build-arg PYTHON_VERSION=3.10 --build-arg TENSORFLOW_VERSION="${TENSORFLOW_VERSION}" --build-arg BAZEL_OPTIMIZATION="${BAZEL_OPTIMIZATION}" . + docker create -it --name io io bash + sleep 5 + docker cp -L io:/opt/io/bazel-bin build + docker cp -L io:/opt/io/.bazelrc build/tensorflow_io/ + sudo chown -R $(id -nu):$(id -ng) . + find build/tensorflow_io -type f + find build/tensorflow_io_gcs_filesystem -type f + find build/tensorflow_io -type l + find build/tensorflow_io_gcs_filesystem -type l + find build/tensorflow_io -name '*runfiles*' + find build/tensorflow_io_gcs_filesystem -name '*runfiles*' + find build/tensorflow_io -name '*runfiles*' | xargs rm -rf + find build/tensorflow_io_gcs_filesystem -name '*runfiles*' | xargs rm -rf + sudo rm -rf build/tensorflow_io/core/golang_ops.h + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-bazel-bin + path: | + build/tensorflow_io + build/tensorflow_io_gcs_filesystem + + linux-wheel: + name: Wheel ${{ matrix.python }} Linux + needs: linux-bazel + runs-on: ubuntu-22.04 + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - name: Wheel ${{ matrix.python }} Linux + run: | + set -x -e + mv bazel-bin/tensorflow_io/.bazelrc . + docker run -i --rm --user $(id -u):$(id -g) -v /etc/password:/etc/password -v $PWD:/v -w /v --net=host python:${{ matrix.python }}-slim python setup.py --data bazel-bin -q bdist_wheel + rm -rf build + docker run -i --rm --user $(id -u):$(id -g) -v /etc/password:/etc/password -v $PWD:/v -w /v --net=host python:${{ matrix.python }}-slim python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel + - name: Auditwheel ${{ matrix.python }} Linux + run: | + set -x -e + ls dist/* + for f in dist/*.whl; do + docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2014_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2014_x86_64 $f + done + sudo chown -R $(id -nu):$(id -ng) . + ls wheelhouse/* + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: wheelhouse + + linux-test: + name: Test ${{ matrix.python }} Linux + needs: linux-wheel + runs-on: ubuntu-22.04 + strategy: + matrix: + python: ['3.9'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: wheelhouse + - name: Setup Linux + run: | + set -x -e + bash -x -e .github/workflows/build.space.sh + bash -x -e tests/test_pulsar/pulsar_test.sh + bash -x -e tests/test_kafka/kafka_test.sh + bash -x -e tests/test_aws/aws_test.sh + echo bash -x -e tests/test_gcloud/test_pubsub_bigtable.sh + echo bash -x -e tests/test_prometheus/prometheus_test.sh start + bash -x -e tests/test_elasticsearch/elasticsearch_test.sh start + bash -x -e tests/test_mongodb/mongodb_test.sh start + bash -x -e tests/test_azure/start_azure.sh + bash -x -e tests/test_sql/sql_test.sh postgresql + bash -x -e tests/test_gcloud/test_gcs.sh gcs-emulator + bash -x -e tests/test_hdfs/hdfs_test.sh + - name: Test Linux + run: | + set -x -e + df -h + docker run -i --rm -v $PWD:/v -w /v --net=host \ + buildpack-deps:22.04 \ + bash -x -e .github/workflows/build.wheel.sh python${{ matrix.python }} + + windows-bazel: + name: Bazel Windows + runs-on: windows-2022 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: egor-tensin/vs-shell@9a932a62d05192eae18ca370155cf877eecc2202 # v2 + with: + arch: x64 + - name: GCP + shell: bash + run: | + cat > service_account_creds.json << EOF + ${{ secrets.GCP_CREDS }} + EOF + - name: Bazel on Windows + env: + BAZEL_VC: "C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/VC/" + shell: bash + run: | + set -x -e + export BAZEL_OPTIMIZATION="--config=cache" + if [[ "${EVENT_NAME}" == "push" && "${REPO_NAME}" == "tensorflow/io" ]]; then + export BAZEL_OPTIMIZATION="$BAZEL_OPTIMIZATION --remote_upload_local_results=true --google_credentials=service_account_creds.json" + fi + bazel version + python3 --version + python3 -m pip install wheel setuptools + python3 -m pip --version + python3 setup.py --install-require | xargs python3 -m pip install + python3 tools/build/configure.py + cat .bazelrc + bazel build -s ${BAZEL_OPTIMIZATION} //tensorflow_io:python/ops/libtensorflow_io.so //tensorflow_io:python/ops/libtensorflow_io_plugins.so //tensorflow_io_gcs_filesystem/... + mkdir -p build + cp -r bazel-bin/tensorflow_io build + cp -r bazel-bin/tensorflow_io_gcs_filesystem build + bash -c "find build -not -name '*.so' -type f | xargs rm -rf" + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-bazel-bin + path: | + build/tensorflow_io + build/tensorflow_io_gcs_filesystem + + windows-wheel: + name: Wheel ${{ matrix.python }} Windows + needs: windows-bazel + runs-on: windows-latest + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Wheel ${{ matrix.python }} Windows + shell: cmd + run: | + @echo on + python --version + python -m pip install -U wheel setuptools + python setup.py --data bazel-bin -q bdist_wheel + rm -rf build + python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel + ls -la dist + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: dist + + windows-test: + name: Test ${{ matrix.python }} Windows + needs: windows-wheel + runs-on: windows-latest + strategy: + matrix: + python: ['3.9'] + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-${{ matrix.python }}-wheel + path: wheel + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - uses: actions/setup-node@60edb5dd545a775178f52524783378180af0d1f8 # v4.0.2 + with: + node-version: '8.x' + - name: Setup ${{ matrix.python }} Windows + shell: cmd + run: | + @echo on + bash -x -e tests/test_azure/start_azure.sh + - name: Install ${{ matrix.python }} Windows + shell: cmd + run: | + @echo on + python --version + (cd wheel && ls tensorflow_io_gcs_filesystem-*.whl | xargs python -m pip install && cd ..) + (cd wheel && ls tensorflow_io-*.whl | xargs python -m pip install && cd ..) + (grep tensorflow tensorflow_io/python/ops/version_ops.py | sed -e "s/require = //g" | xargs python -m pip install) + - name: Test ${{ matrix.python }} Windows + shell: cmd + run: | + @echo on + python --version + python -m pip install -U pytest-benchmark + python -m pip install google-cloud-bigquery-storage==0.7.0 google-cloud-bigquery==1.22.0 fastavro + rm -rf tensorflow_io tensorflow_io_gcs_filesystem + cd tests + python -m pytest -s -v test_lmdb.py + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_image.py -k "webp or ppm or bmp or bounding or exif or hdr or openexr or tiff or avif" + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_serialization.py + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_io_dataset.py -k "numpy or hdf5 or audio or to_file" + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_http.py + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_bigquery.py + if %errorlevel% neq 0 exit /b %errorlevel% + python -m pytest -s -v test_dicom.py + if %errorlevel% neq 0 exit /b %errorlevel% + + release: + name: Release + if: github.event_name == 'push' + needs: [linux-wheel, macos-wheel] #, windows-wheel] + runs-on: ubuntu-20.04 + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.9-wheel + path: macOS-3.9-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.10-wheel + path: macOS-3.10-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.11-wheel + path: macOS-3.11-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.12-wheel + path: macOS-3.12-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.9-wheel + path: Linux-3.9-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.10-wheel + path: Linux-3.10-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.11-wheel + path: Linux-3.11-wheel + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.12-wheel + path: Linux-3.12-wheel + #- uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + # with: + # name: Windows-3.9-wheel + # path: Windows-3.9-wheel + #- uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + # with: + # name: Windows-3.10-wheel + # path: Windows-3.10-wheel + #- uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + # with: + # name: Windows-3.11-wheel + # path: Windows-3.11-wheel + #- uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + # with: + # name: Windows-3.12-wheel + # path: Windows-3.12-wheel + - run: | + set -e -x + mkdir -p wheelhouse + cp macOS-3.9-wheel/*.whl wheelhouse/ + cp macOS-3.10-wheel/*.whl wheelhouse/ + cp macOS-3.11-wheel/*.whl wheelhouse/ + cp macOS-3.12-wheel/*.whl wheelhouse/ + cp Linux-3.9-wheel/*.whl wheelhouse/ + cp Linux-3.10-wheel/*.whl wheelhouse/ + cp Linux-3.11-wheel/*.whl wheelhouse/ + cp Linux-3.12-wheel/*.whl wheelhouse/ + # cp Windows-3.9-wheel/*.whl wheelhouse/ + # cp Windows-3.10-wheel/*.whl wheelhouse/ + # cp Windows-3.11-wheel/*.whl wheelhouse/ + # cp Windows-3.12-wheel/*.whl wheelhouse/ + ls -la wheelhouse/ + sha256sum wheelhouse/*.whl + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: tensorflow-io-release + path: wheelhouse + + release-candidate: + name: Release Candidate + if: github.event_name == 'push' && github.ref == 'refs/heads/master' + needs: [release] + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: tensorflow-io-release + path: wheelhouse + - run: | + set -x -e + sha256sum wheelhouse/*.whl | sort -u > wheelhouse.sha256 + git rev-parse --verify HEAD > wheelhouse.commit + grep 'version = ' tensorflow_io/python/ops/version_ops.py | sed -E 's@^.*version = "@@g' | sed -E 's@".*$@@g' > wheelhouse.version + cat wheelhouse.sha256 + cat wheelhouse.commit + cat wheelhouse.version + echo "::set-output name=version::$(cat wheelhouse.version)" + id: info + - run: |- + cat < Dockerfile + FROM gcr.io/distroless/base-debian11 + COPY wheelhouse /wheelhouse + COPY wheelhouse.sha256 /wheelhouse.sha256 + COPY wheelhouse.commit /wheelhouse.commit + COPY wheelhouse.version /wheelhouse.version + EOF + - uses: docker/setup-buildx-action@d70bba72b1f3fd22344832f00baa16ece964efeb # v3.3.0 + - uses: docker/login-action@0d4c9c5ea7693da7b068278f7b52bda2a190a446 # v3.2.0 + with: + username: tfsigio + password: ${{ secrets.DOCKER_PASSWORD }} + - uses: docker/build-push-action@15560696de535e4014efeff63c48f16952e52dd1 # v6.2.0 + with: + context: . + push: true + tags: tfsigio/candidate:${{ steps.info.outputs.version }} + + docker-release: + name: Docker Release + if: github.event_name == 'push' && github.ref == 'refs/heads/master' + needs: [lint, linux-test, macos-test, windows-test] + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: 3.8 + - run: | + set -e -x + docker login --username tfsigio --password ${{ secrets.DOCKER_PASSWORD }} + bash -x -e tools/docker/tests/dockerfile_cpu_test.sh + docker push tfsigio/tfio:latest + python --version + TFIO_VERSION=$(python setup.py --version | tail -1) + docker tag tfsigio/tfio:latest tfsigio/tfio:${TFIO_VERSION} + docker push tfsigio/tfio:${TFIO_VERSION} + bash -x -e tools/docker/tests/dockerfile_devel_test.sh + docker push tfsigio/tfio:latest-devel + + build-number: + name: Build Number + if: github.event_name == 'push' + runs-on: ubuntu-20.04 + steps: + - run: | + set -e -x + BUILD_NUMBER=$(date "+%Y%m%d%H%M%S") + echo ${BUILD_NUMBER} > BUILD_NUMBER + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: BUILD_NUMBER + path: BUILD_NUMBER + + macos-nightly: + name: Nightly ${{ matrix.python }} macOS + if: github.event_name == 'push' + needs: [build-number, macos-wheel] + runs-on: macOS-12 + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: BUILD_NUMBER + - uses: einaregilsson/build-number@46decf22c413b48c8923e98b2b5836f8aaf7781e # v3 + - run: echo "Build number is $BUILD_NUMBER" + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Wheel ${{ matrix.python }} macOS + run: | + set -x -e + python -m pip install -U wheel setuptools + python --version + python setup.py --data bazel-bin -q bdist_wheel --plat-name macosx_10_14_x86_64 --nightly $BUILD_NUMBER + rm -rf build + python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel --plat-name macosx_10_14_x86_64 --nightly $BUILD_NUMBER + - name: Auditwheel ${{ matrix.python }} macOS + run: | + set -x -e + python -m pip install twine delocate==0.10.3 + delocate-wheel --version + ls dist/* + mkdir -p wheelhouse + for f in dist/*.whl; do + echo "disable delocate due to build breaks: delocate-wheel -w wheelhouse $f" + cp $f wheelhouse + done + ls wheelhouse/* + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-nightly + path: wheelhouse + + linux-nightly: + name: Nightly ${{ matrix.python }} Linux + if: github.event_name == 'push' + needs: [build-number, linux-wheel] + runs-on: ubuntu-20.04 + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: BUILD_NUMBER + - uses: einaregilsson/build-number@46decf22c413b48c8923e98b2b5836f8aaf7781e # v3 + - run: echo "Build number is $BUILD_NUMBER" + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - name: Wheel ${{ matrix.python }} Linux + run: | + set -x -e + mv bazel-bin/tensorflow_io/.bazelrc . + docker run -i --rm --user $(id -u):$(id -g) -v /etc/password:/etc/password -v $PWD:/v -w /v --net=host python:${{ matrix.python }}-slim python setup.py --data bazel-bin -q bdist_wheel --nightly $BUILD_NUMBER + rm -rf build + docker run -i --rm --user $(id -u):$(id -g) -v /etc/password:/etc/password -v $PWD:/v -w /v --net=host python:${{ matrix.python }}-slim python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel --nightly $BUILD_NUMBER + - name: Auditwheel ${{ matrix.python }} Linux + run: | + set -x -e + ls dist/* + for f in dist/*.whl; do + docker run -i --rm -v $PWD:/v -w /v --net=host quay.io/pypa/manylinux2014_x86_64 bash -x -e /v/tools/build/auditwheel repair --plat manylinux2014_x86_64 $f + done + sudo chown -R $(id -nu):$(id -ng) . + ls wheelhouse/* + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-nightly + path: wheelhouse + + windows-nightly: + name: Nightly ${{ matrix.python }} Windows + if: github.event_name == 'push' + needs: [build-number, windows-wheel] + runs-on: windows-latest + strategy: + matrix: + python: ['3.9', '3.10', '3.11', '3.12'] + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: BUILD_NUMBER + - uses: einaregilsson/build-number@46decf22c413b48c8923e98b2b5836f8aaf7781e # v3 + - run: echo "Build number is $BUILD_NUMBER" + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: ${{ runner.os }}-bazel-bin + path: bazel-bin + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: ${{ matrix.python }} + - name: Wheel ${{ matrix.python }} Windows + shell: cmd + run: | + @echo on + python --version + python -m pip install -U wheel setuptools + python setup.py --data bazel-bin -q bdist_wheel --nightly %BUILD_NUMBER% + rm -rf build + python setup.py --project tensorflow-io-gcs-filesystem --data bazel-bin -q bdist_wheel --nightly %BUILD_NUMBER% + ls -la dist + - uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: ${{ runner.os }}-${{ matrix.python }}-nightly + path: dist + + nightly: + name: Nightly + if: github.event_name == 'push' + needs: [linux-nightly, macos-nightly, windows-nightly] + runs-on: ubuntu-20.04 + steps: + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.9-nightly + path: macOS-3.9-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.10-nightly + path: macOS-3.10-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.11-nightly + path: macOS-3.11-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: macOS-3.12-nightly + path: macOS-3.12-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.9-nightly + path: Linux-3.9-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.10-nightly + path: Linux-3.10-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.11-nightly + path: Linux-3.11-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Linux-3.12-nightly + path: Linux-3.12-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Windows-3.9-nightly + path: Windows-3.9-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Windows-3.10-nightly + path: Windows-3.10-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Windows-3.11-nightly + path: Windows-3.11-nightly + - uses: actions/download-artifact@65a9edc5881444af0b9093a5e628f2fe47ea3b2e # v4.1.7 + with: + name: Windows-3.12-nightly + path: Windows-3.12-nightly + - run: | + set -e -x + mkdir -p dist + cp macOS-3.9-nightly/tensorflow_io_nightly*.whl dist/ + cp macOS-3.10-nightly/tensorflow_io_nightly*.whl dist/ + cp macOS-3.11-nightly/tensorflow_io_nightly*.whl dist/ + cp macOS-3.12-nightly/tensorflow_io_nightly*.whl dist/ + cp Linux-3.9-nightly/tensorflow_io_nightly*.whl dist/ + cp Linux-3.10-nightly/tensorflow_io_nightly*.whl dist/ + cp Linux-3.11-nightly/tensorflow_io_nightly*.whl dist/ + cp Linux-3.12-nightly/tensorflow_io_nightly*.whl dist/ + cp Windows-3.9-nightly/tensorflow_io_nightly*.whl dist/ + cp Windows-3.10-nightly/tensorflow_io_nightly*.whl dist/ + cp Windows-3.11-nightly/tensorflow_io_nightly*.whl dist/ + cp Windows-3.12-nightly/tensorflow_io_nightly*.whl dist/ + ls -la dist/ + sha256sum dist/*.whl + - uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # v1.9.0 + with: + user: __token__ + password: ${{ secrets.github_tensorflow_io_nightly }} + - run: | + set -e -x + rm -rf dist + mkdir -p dist + cp macOS-3.9-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp macOS-3.10-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp macOS-3.11-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp macOS-3.12-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Linux-3.9-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Linux-3.10-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Linux-3.11-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Linux-3.12-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Windows-3.9-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Windows-3.10-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Windows-3.11-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + cp Windows-3.12-nightly/tensorflow_io_gcs_filesystem_nightly*.whl dist/ + ls -la dist/ + sha256sum dist/*.whl + - uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # v1.9.0 + with: + user: __token__ + password: ${{ secrets.tensorflow_io_gcs_filesystem_nightly }} + + docker-nightly: + name: Docker Nightly + if: github.event_name == 'push' && github.ref == 'refs/heads/master' + needs: [linux-nightly, macos-nightly, windows-nightly] + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - uses: actions/setup-python@82c7e631bb3cdc910f68e0081d67478d79c6982d # v5.1.0 + with: + python-version: 3.8 + - run: | + set -e -x + docker login --username tfsigio --password ${{ secrets.DOCKER_PASSWORD }} + python --version + bash -x -e tools/docker/tests/dockerfile_nightly_test.sh + docker push tfsigio/tfio:nightly diff --git a/.github/workflows/reserves/tensorflow-io/release.note.yml b/.github/workflows/reserves/tensorflow-io/release.note.yml new file mode 100644 index 000000000000..3533732b7e53 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/release.note.yml @@ -0,0 +1,34 @@ +name: Release Note + +on: + push: + branches: + - master + +permissions: + contents: read + +jobs: + release-note: + permissions: + contents: write # for peter-evans/create-pull-request to create branch + pull-requests: write # for peter-evans/create-pull-request to create a PR + name: README.md + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + with: + fetch-depth: 0 + - run: git tag + - run: python3 tools/release/note_create.py + - run: git diff + - run: python3 tools/release/note_update.py + - run: git diff + - uses: peter-evans/create-pull-request@c5a7806660adbe173f04e3e038b0ccdcd758773c # v6.1.0 + with: + commit-message: Update RELEASE.md [bot] + branch: bot-RELEASE.md + delete-branch: true + title: 'Update RELEASE.md [bot]' + body: | + README.md: auto-updated by .github/workflows/release.note.yml diff --git a/.github/workflows/reserves/tensorflow-io/release.yml b/.github/workflows/reserves/tensorflow-io/release.yml new file mode 100644 index 000000000000..33f2039819ca --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/release.yml @@ -0,0 +1,82 @@ +name: Release + +on: + workflow_dispatch: + inputs: + version: + description: "Version (e.g., v0.24.0)" + required: true + commit: + description: "Commit (e.g., 92b44e1)" + required: true + +permissions: + contents: read + +jobs: + release: + permissions: + contents: write + name: Release + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + with: + fetch-depth: 0 + - run: | + set -x -e + COMMIT=$(git rev-parse --quiet --verify ${{ github.event.inputs.commit }}) + if [[ "$(git tag -l ${{ github.event.inputs.version }})" == "${{ github.event.inputs.version }}" ]]; then + echo "${{ github.event.inputs.version }} already released" + exit 1 + fi + VERSION=${{ github.event.inputs.version }} + + docker pull tfsigio/candidate:${VERSION:1} + docker create -it --name storage tfsigio/candidate:${VERSION:1} bash + docker cp storage:/wheelhouse . + docker cp storage:/wheelhouse.sha256 . + docker cp storage:/wheelhouse.commit . + docker cp storage:/wheelhouse.version . + + sha256sum wheelhouse/*.whl | sort -u | diff wheelhouse.sha256 - + mv wheelhouse dist + + if [[ "${COMMIT}" != "$(cat wheelhouse.commit)" ]]; then + echo "${COMMIT} != $(cat wheelhouse.commit)" + exit 1 + fi + if [[ "${VERSION}" != "v$(cat wheelhouse.version)" ]]; then + echo "${VERSION} != v$(cat wheelhouse.version)" + exit 1 + fi + + python3 tools/release/note_take.py ${VERSION} + + echo "::set-output name=tag::${VERSION}" + echo "::set-output name=name::TensorFlow I/O ${VERSION:1}" + echo "::set-output name=commit::${COMMIT}" + id: info + - run: | + set -x -e + echo ${{ steps.info.outputs.tag }} + echo ${{ steps.info.outputs.name }} + echo ${{ steps.info.outputs.commit }} + cat CURRENT.md + - uses: softprops/action-gh-release@a74c6b72af54cfa997e81df42d94703d6313a2d0 # v2.0.6 + with: + body_path: CURRENT.md + name: ${{ steps.info.outputs.name }} + tag_name: ${{ steps.info.outputs.tag }} + target_commitish: ${{ steps.info.outputs.commit }} + draft: true + - uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # master + with: + user: __token__ + password: ${{ secrets.TEST_PYPI_TOKEN }} + repository_url: https://test.pypi.org/legacy/ + #- uses: pypa/gh-action-pypi-publish@ec4db0b4ddc65acdf4bff5fa45ac92d78b56bdf0 # master + # with: + # user: __token__ + # password: ${{ secrets.PYPI_TOKEN }} + diff --git a/.github/workflows/reserves/tensorflow-io/reviewdog.yml b/.github/workflows/reserves/tensorflow-io/reviewdog.yml new file mode 100644 index 000000000000..298f49e97766 --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/reviewdog.yml @@ -0,0 +1,59 @@ +name: Reviewdog + +on: + pull_request: + branches: + - master + +permissions: + contents: read + +jobs: + bazel: + permissions: + checks: write # for reviewdog/action-suggester to report issues using checks + contents: read # for actions/checkout to fetch code + name: Bazel Buildifier + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - run: | + sudo python3 -m pip install -U numpy pip black pyupgrade + bazel run -s --verbose_failures --experimental_repo_remote_exec //tools/lint:lint -- bazel + - uses: reviewdog/action-suggester@a1d57ff096639094e0ba35ef3039e79316364796 # v1.15.0 + black: + permissions: + checks: write # for reviewdog/action-suggester to report issues using checks + contents: read # for actions/checkout to fetch code + name: Python Black + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - run: | + sudo python3 -m pip install -U numpy pip black pyupgrade + bazel run -s --verbose_failures --experimental_repo_remote_exec //tools/lint:lint -- black + - uses: reviewdog/action-suggester@a1d57ff096639094e0ba35ef3039e79316364796 # v1.15.0 + clang: + permissions: + checks: write # for reviewdog/action-suggester to report issues using checks + contents: read # for actions/checkout to fetch code + name: Clang Format + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - run: | + sudo python3 -m pip install -U numpy pip black pyupgrade + bazel run -s --verbose_failures --experimental_repo_remote_exec //tools/lint:lint -- clang + - uses: reviewdog/action-suggester@a1d57ff096639094e0ba35ef3039e79316364796 # v1.15.0 + pyupgrade: + permissions: + checks: write # for reviewdog/action-suggester to report issues using checks + contents: read # for actions/checkout to fetch code + name: Python Pyupgrade + runs-on: ubuntu-20.04 + steps: + - uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + - run: | + sudo python3 -m pip install -U numpy pip black pyupgrade + bazel run -s --verbose_failures --experimental_repo_remote_exec //tools/lint:lint -- pyupgrade + - uses: reviewdog/action-suggester@a1d57ff096639094e0ba35ef3039e79316364796 # v1.15.0 diff --git a/.github/workflows/reserves/tensorflow-io/scorecard.yml b/.github/workflows/reserves/tensorflow-io/scorecard.yml new file mode 100644 index 000000000000..bd84cd48dd0e --- /dev/null +++ b/.github/workflows/reserves/tensorflow-io/scorecard.yml @@ -0,0 +1,64 @@ +# This workflow uses actions that are not certified by GitHub. They are provided +# by a third-party and are governed by separate terms of service, privacy +# policy, and support documentation. + +name: Scorecard supply-chain security +on: + # For Branch-Protection check. Only the default branch is supported. See + # https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection + branch_protection_rule: + # To guarantee Maintained check is occasionally updated. See + # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained + schedule: + - cron: '42 22 * * 1' + push: + branches: [ "master" ] + +# Declare default permissions as read only. +permissions: read-all + +jobs: + analysis: + name: Scorecard analysis + runs-on: ubuntu-latest + permissions: + # Needed to upload the results to code-scanning dashboard. + security-events: write + # Needed to publish results and get a badge (see publish_results below). + id-token: write + + steps: + - name: "Checkout code" + uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 + with: + persist-credentials: false + + - name: "Run analysis" + uses: ossf/scorecard-action@dc50aa9510b46c811795eb24b2f1ba02a914e534 # v2.3.3 + with: + results_file: results.sarif + results_format: sarif + # (Optional) "write" PAT token. Uncomment the `repo_token` line below if you want to enable the Branch-Protection check + # To create the PAT, follow the steps in https://github.com/ossf/scorecard-action#authentication-with-fine-grained-pat-optional. + # repo_token: ${{ secrets.SCORECARD_TOKEN }} + + # Publish results enabled: + # - Publish results to OpenSSF REST API for easy access by consumers + # - Allows the repository to include the Scorecard badge. + # - See https://github.com/ossf/scorecard-action#publishing-results. + publish_results: true + + # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF + # format to the repository Actions tab. + - name: "Upload artifact" + uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3 + with: + name: SARIF file + path: results.sarif + retention-days: 5 + + # Upload the results to GitHub's code scanning dashboard. + - name: "Upload to code-scanning" + uses: github/codeql-action/upload-sarif@b611370bb5703a7efb587f9d136a52ea24c5c38c # v3.25.11 + with: + sarif_file: results.sarif diff --git a/.github/workflows/reserves/test-dotnet.yml b/.github/workflows/reserves/test-dotnet.yml new file mode 100644 index 000000000000..03b9a294bc60 --- /dev/null +++ b/.github/workflows/reserves/test-dotnet.yml @@ -0,0 +1,35 @@ +name: Validate dotnet + +on: + pull_request: + paths-ignore: + - '**.md' + push: + branches: + - main + - releases/* + paths-ignore: + - '**.md' + +jobs: + setup-version: + runs-on: ${{ matrix.operating-system }} + strategy: + fail-fast: false + matrix: + operating-system: [ubuntu-latest, windows-latest, macOS-latest] + dotnet-version: ['2.1', '2.2', '3.0', '3.1', '5.0', '6.0', '7.0', '8.0'] + steps: + - name: Checkout + uses: actions/checkout@v3 + - name: Clear toolcache + shell: pwsh + run: __tests__/clear-toolcache.ps1 ${{ runner.os }} + - name: Setup dotnet ${{ matrix.dotnet-version }} + uses: ./ + with: + dotnet-version: ${{ matrix.dotnet-version }} + - name: Verify installed version + shell: pwsh + run: | + __tests__/verify-dotnet.ps1 -Patterns "^${{ matrix.dotnet-version }}" diff --git a/.github/workflows/reserves/test_of_push_and_pull.yml b/.github/workflows/reserves/test_of_push_and_pull.yml new file mode 100644 index 000000000000..4ee1316f9302 --- /dev/null +++ b/.github/workflows/reserves/test_of_push_and_pull.yml @@ -0,0 +1,37 @@ +name: "Tests of push & pull" + +on: + push: + branches: [ master ] + pull_request: + branches: [ master ] + +env: + FILENAME: README.md + +jobs: + tests: + name: 'Validate README.md changes' + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: '3.8' + + - name: Install dependencies + run: python -m pip install -r scripts/requirements.txt + + - name: Validate Markdown format + run: python scripts/validate/format.py ${FILENAME} + + - name: Validate pull request changes + run: scripts/github_pull_request.sh ${{ github.repository }} ${{ github.event.pull_request.number }} ${FILENAME} + if: github.event_name == 'pull_request' + + - name: Checking if push changes are duplicated + run: python scripts/validate/links.py ${FILENAME} --only_duplicate_links_checker + if: github.event_name == 'push' diff --git a/.github/workflows/reserves/test_of_validate_package.yml b/.github/workflows/reserves/test_of_validate_package.yml new file mode 100644 index 000000000000..f739df81d759 --- /dev/null +++ b/.github/workflows/reserves/test_of_validate_package.yml @@ -0,0 +1,29 @@ +name: "Tests of validate package" + +on: + push: + branches: [ master ] + pull_request: + branches: [ master ] + +jobs: + unittest: + name: 'Run tests of validate package' + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: '3.8' + + - name: Install dependencies + run: python -m pip install -r scripts/requirements.txt + + - name: Run Unittest + run: | + cd scripts + python -m unittest discover tests/ --verbose diff --git a/.github/workflows/reserves/ubuntu-wsl/build-pr.yaml b/.github/workflows/reserves/ubuntu-wsl/build-pr.yaml new file mode 100644 index 000000000000..35d9e080e173 --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/build-pr.yaml @@ -0,0 +1,64 @@ +name: Build windows package +on: + pull_request: + types: + - opened + - reopened + - synchronize + - ready_for_review + paths-ignore: + - docs/** + +jobs: + build-wsl: + name: Build + runs-on: windows-latest + if: ${{ !github.event.pull_request.draft }} + env: + rootfs64: 'http://cloud-images.ubuntu.com/wsl/jammy/current/ubuntu-jammy-wsl-amd64-wsl.rootfs.tar.gz' + workDir: 'C:/Temp/builddir' + steps: + - name: Checkout WSL + uses: actions/checkout@v3 + with: + # we need to use a subdirectory as we can't move back base GITHUB_WORKSPACE directory + path: repo + - name: Copy under workDir (which has more space to build the package) + shell: bash + run: | + mkdir -p $(dirname ${{ env.workDir }}) + mv ${GITHUB_WORKSPACE}/repo ${{ env.workDir }} + cd ${{ env.workDir }} + git checkout "${GITHUB_SHA}" + git submodule update --init --recursive + - name: Download rootfs and configure project to only build x64 + working-directory: ${{ env.workDir }} + shell: powershell + run: | + Set-Content -Path "DistroLauncher.sln" -Value (get-content -Path "DistroLauncher.sln" | Select-String -Pattern 'ARM64' -NotMatch) + mkdir x64/ + $client = new-object System.Net.WebClient + $client.DownloadFile("${{ env.rootfs64}}","x64/install.tar.gz") + - name: Setup MSBuild (PATH) + uses: microsoft/setup-msbuild@v1.0.2 + - name: Install certificate + shell: powershell + working-directory: ${{ env.workDir }} + run: | + New-Item -ItemType directory -Path certificate + Set-Content -Path certificate\certificate.txt -Value '${{ secrets.CERTIFICATE }}' + certutil -decode certificate\certificate.txt certificate\certificate.pfx + + $pwd = ConvertTo-SecureString '${{ secrets.CERTIFICATE_PASSWORD }}' -AsPlainText -Force + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:LocalMachine\Trust -FilePath certificate\certificate.pfx + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:CurrentUser\My -FilePath certificate\certificate.pfx + - name: Build Bundle + working-directory: ${{ env.workDir }} + run: msbuild .\DistroLauncher.sln /t:Build /m /nr:false /p:Configuration=Release /p:AppxBundle=Always /p:AppxBundlePlatforms="x64" /p:UapAppxPackageBuildMode=SideloadOnly -verbosity:normal + - name: Allow downloading sideload appxbundle + uses: actions/upload-artifact@v2 + with: + name: sideload-build + path: | + ${{ env.workDir }}/AppPackages/Ubuntu/Ubuntu_*/* + retention-days: 7 diff --git a/.github/workflows/reserves/ubuntu-wsl/build-wsl.yaml b/.github/workflows/reserves/ubuntu-wsl/build-wsl.yaml new file mode 100644 index 000000000000..9bc365cc0d87 --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/build-wsl.yaml @@ -0,0 +1,359 @@ +name: Build WSL +on: + workflow_dispatch: + inputs: + appID: + description: 'Release name to use for the bundle' + required: true + default: 'UbuntuPreview' + rootfses: + description: 'Ubuntu WSL rootfs urls, separated by a colon. Direct set of "tar.gz::arch" if arch is not in the filename' + required: true + default: '' + rootfseschecksum: + description: 'Should download a SHA256SUMS file to check the rootfs' + required: true + default: 'yes' + upload: + description: 'Should we upload the appxbundle to the store' + required: true + default: 'yes' + schedule: + - cron: '0 10 * * *' + +env: + goversion: '1.21.4' + +jobs: + build-matrix: + name: Build Matrix for AppIDs to run on with rootfses, which can be manually supplied or automatically. + runs-on: ubuntu-latest + outputs: + matrix: ${{ steps.build-matrix-release.outputs.matrix }} + steps: + - name: Install dependencies + run: | + sudo DEBIAN_FRONTEND=noninteractive apt update + sudo DEBIAN_FRONTEND=noninteractive apt install -y jq + - uses: actions/checkout@v3 + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.goversion }} + - name: Build Matrix for AppIDs to run on with rootfses, which can be manually supplied or automatically + id: build-matrix-release + run: | + set -eux + # Manual build + if [ ${{ github.event_name }} = 'workflow_dispatch' ]; then + appID="${{ github.event.inputs.appID }}" + if [ -z "${appID}" ]; then + appID="UbuntuPreview" + fi + + builds="$(cat <<-EOF|jq -c + {"include": + [ + { + "AppID": "${appID}", + "Rootfses": "${{ github.event.inputs.rootfses }}", + "RootfsesChecksum": "${{ github.event.inputs.rootfseschecksum }}", + "Upload": "${{ github.event.inputs.upload }}" + } + ] + } + EOF + )" + else + wsl-builder/lp-distro-info > /tmp/all-releases.csv + go build ./wsl-builder/prepare-build + builds_config="$(./prepare-build build-github-matrix /tmp/all-releases.csv)" + if [ "${builds_config}" == "null" ]; then + echo "No active application to build" + exit 0 + fi + builds="{\"include\":${builds_config}}" + fi + + echo "${builds}" + echo "::set-output name=matrix::${builds}" + echo "::notice::Building for: $(echo "${builds}" | jq '.include[] | "\(.AppID): \(.Rootfses). RootfsesChecksum: \(.RootfsesChecksum). Upload to store: \(.Upload)"')" + + build-wsl: + name: Build ${{ matrix.AppID }} + runs-on: windows-latest + needs: build-matrix + strategy: + matrix: ${{fromJson(needs.build-matrix.outputs.matrix)}} + fail-fast: false + env: + buildInfoPath: 'wiki/build-info' + workDir: 'C:/Temp/builddir' + outputs: + has-changes: ${{ steps.detect-changes.outputs.has-changes }} + steps: + - name: Checkout WSL + shell: bash + run: | + mkdir -p ${{ env.workDir }} + git clone https://github.com/ubuntu/wsl.git ${{ env.workDir }} + cd ${{ env.workDir }} + git checkout "${GITHUB_SHA}" + git submodule update --init --recursive + - name: Checkout build-info + shell: bash + run: | + mkdir -p ${{ env.workDir }}/wiki/ + git clone https://github.com/ubuntu/wsl.wiki ${{ env.workDir }}/wiki --depth 1 + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.goversion }} + - name: Prepare project metadata, assets and download rootfses + working-directory: ${{ env.workDir }} + shell: bash + run: | + set -eu + # Download rootfses, checksum and place them at the correct place + go build ./wsl-builder/prepare-build + extraArgs="" + if [ ${{ matrix.RootfsesChecksum }} != "yes" ]; then + extraArgs="--no-checksum" + fi + archsBundle="$(./prepare-build prepare ${{ env.buildInfoPath }}/${{ matrix.AppID }}-buildid.md ${{ matrix.AppID }} ${{ matrix.Rootfses }} ${extraArgs})" + echo "AppxBundlePlatforms=${archsBundle}" >> $GITHUB_ENV + + # Always StoreUpload mode to get appxupload file + buildMode="StoreUpload" + echo "UapAppxPackageBuildMode=${buildMode}" >> $GITHUB_ENV + - name: Setup MSBuild (PATH) + uses: microsoft/setup-msbuild@v1.0.2 + - name: Install certificate + shell: powershell + working-directory: ${{ env.workDir }} + run: | + New-Item -ItemType directory -Path certificate + Set-Content -Path certificate\certificate.txt -Value '${{ secrets.CERTIFICATE }}' + certutil -decode certificate\certificate.txt certificate\certificate.pfx + + $pwd = ConvertTo-SecureString '${{ secrets.CERTIFICATE_PASSWORD }}' -AsPlainText -Force + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:LocalMachine\Trust -FilePath certificate\certificate.pfx + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:CurrentUser\My -FilePath certificate\certificate.pfx + - name: Build Bundle + working-directory: ${{ env.workDir }} + run: msbuild .\DistroLauncher.sln /t:Build /m /nr:false /p:Configuration=Release /p:AppxBundle=Always /p:AppxBundlePlatforms="${{ env.AppxBundlePlatforms }}" /p:UapAppxPackageBuildMode=${{ env.UapAppxPackageBuildMode }} -verbosity:normal + - name: Collect the PDB files + working-directory: ${{ env.workDir }} + shell: bash + run: | + set -eu + + # Launcher PDBs + + for arch in "x64" "ARM64"; do + collectTo="debug-database-${{ matrix.AppID }}/launcher/$arch/" + mkdir -p "$collectTo" + + findDir="ARM64/Release/" + if [ $arch == "x64" ]; then + findDir="ARM64/Release/DistroLauncher-Appx/x64/" + fi + + find "$findDir" -maxdepth 1 -name "*.pdb" -exec cp '{}' "$collectTo" \; + + done + + - name: Allow downloading sideload appxbundle + uses: actions/upload-artifact@v2 + with: + name: sideload-${{ matrix.AppID }} + path: | + ${{ env.workDir }}/AppPackages/Ubuntu/Ubuntu_*/* + retention-days: 7 + - name: Allow downloading store appxupload + uses: actions/upload-artifact@v2 + with: + name: store-${{ matrix.AppID }} + path: | + ${{ env.workDir }}/AppPackages/Ubuntu/Ubuntu_*.appxupload + retention-days: 7 + - name: Allow downloading the program debug artifacts + uses: actions/upload-artifact@v2 + with: + name: debug-database-${{ matrix.AppID }} + path: | + ${{ env.workDir }}/debug-database-${{ matrix.AppID }}/ + retention-days: 7 + - name: Detect any potential changes and if we should upload automatically to the Store + id: detect-changes + working-directory: ${{ env.workDir }} + shell: bash + run: | + set -eux + + build_id=$(cat "${{ env.buildInfoPath }}/${{ matrix.AppID }}-buildid.md") + + echo "::set-output name=has-changes::false" + echo "::set-output name=should-upload::false" + + # Store md5sum of rootfs, launcher and assets related code + fingerprint_file="${{ matrix.AppID }}-fingerprint.md" + fingerprint_filepath="${{ env.buildInfoPath }}/${fingerprint_file}" + + first_upload="false" + if [ ! -f "${fingerprint_filepath}" ]; then + first_upload="true" + fi + + # First. the rootfses + echo '```' > "${fingerprint_filepath}" + md5sum */install.tar.gz | sort -k2 >> "${fingerprint_filepath}" + # Launcher code + echo "$(find DistroLauncher -type f -not -path "*/ARM64/*" -not -path "*/x64/*" -exec md5sum {} \; | sort -k 2 | md5sum)DistroLauncher" >> "${fingerprint_filepath}" + # Build info and assets (without specific build number) + sed -i "s/\.${build_id}\./XXX/" DistroLauncher-Appx/MyDistro.appxmanifest + echo "$(find DistroLauncher-Appx -type f -not -path "*/ARM64/*" -not -path "*/x64/*" -not -path "*/BundleArtifacts/*" -not -name "Generated Files" -exec md5sum {} \; | sort -k 2 | md5sum)DistroLauncher-Appx" >> "${fingerprint_filepath}" + echo '```' >> "${fingerprint_filepath}" + + cd "${{ env.buildInfoPath }}" + git add "${fingerprint_file}" + + hasChanges="$(git diff --staged ${fingerprint_file})" + cd - + if [ -z "${hasChanges}" ]; then + exit 0 + fi + + echo "::set-output name=has-changes::true" + + if [ ${{ matrix.Upload }} != "yes" ]; then + echo "::notice::${{ matrix.AppID }} build ${build_id} ready for sideload or manual submission to the Microsoft Store" + exit 0 + fi + + # If we are doing a first build for this distro, we don’t want to submit it to the store as first submission + # is manual. We let the other steps and jobs happening to save artifacts (build-id, fingerprints) of + # the first build. + if [ "${first_upload}" = "true" ]; then + echo "::notice::This is the first build for ${{ matrix.AppID }}. It needs to be submitted manually to the Microsoft Store" + exit 0 + fi + echo "::notice::Uploading to the store ${{ matrix.AppID }} build ${build_id}" + echo "::set-output name=should-upload::true" + + echo "Uploading new version as some files have changed:" + echo "${hasChanges}" + + - name: Install Store Broker + if: ${{ steps.detect-changes.outputs.should-upload == 'true' }} + working-directory: ${{ env.workDir }} + shell: powershell + run: | + Install-Module -Name StoreBroker -AcceptLicense -Force -Scope CurrentUser -Verbose + - name: Submit to Microsoft Store + if: ${{ steps.detect-changes.outputs.should-upload == 'true' }} + working-directory: ${{ env.workDir }} + shell: powershell + run: | + New-Item -ItemType directory -Path store -Force + + # Authenticate against the store + $pass = ConvertTo-SecureString -String '${{ secrets.AZUREAD_APPKEY }}' -AsPlainText -Force + $cred = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList ${{ secrets.AZUREAD_APPID }},$pass + Set-StoreBrokerAuthentication -TenantId '${{ secrets.AZUREAD_TENANTID }}' -Credential $cred + + # Get store app ID for AppID + $appid = (Get-Applications -GetAll | Where-Object PackageIdentityName -like '*${{ matrix.AppID }}').id + If($appid -eq '') { + echo "No app found for AppID: ${{ matrix.AppID }}" + exit 1 + } + + # We want to publish UbuntuPreview automatically + If('${{ matrix.AppID }}' -eq 'UbuntuPreview') { + ((Get-Content -path meta\SBConfig.json -Raw) -replace 'Manual','Immediate') | Set-Content -Path meta\SBConfig.json + } + + # Prepare and submit to the Store + cd store\ + New-SubmissionPackage -ConfigPath ..\meta\SBConfig.json + Update-ApplicationSubmission -AppId $appid -SubmissionDataPath "out\appstore-submission.json" -PackagePath "out\appstore-submission.zip" -Force -Autocommit -ReplacePackages -UpdateListings -UpdatePublishModeAndVisibility -UpdatePricingAndAvailability -UpdateAppProperties -UpdateGamingOptions -UpdateTrailers -UpdateNotesForCertification + + - name: Upload build artifacts + if: ${{ steps.detect-changes.outputs.has-changes == 'true' }} + uses: actions/upload-artifact@v2 + with: + name: build-artifacts-${{ matrix.AppID }} + path: ${{ env.workDir }}/${{ env.buildInfoPath }}/${{ matrix.AppID }}-* + + update-build-artifacts: + # TODO: download only artefacts for wiki (pattern not supported yet: https://github.com/actions/download-artifact/issues/103) + name: Store updated build artifacts in wiki for reuse in future workflow runs + runs-on: ubuntu-latest + if: always() + needs: build-wsl + env: + artifactsPath: '/tmp/artifacts' + codeDir: '/tmp/update-codedir' + steps: + - name: Checkout wiki + uses: actions/checkout@v3 + with: + repository: ubuntu/wsl.wiki + - name: Download artifacts from all previous matrix runs + uses: actions/download-artifact@v2 + with: + path: ${{ env.artifactsPath }} + - name: Copy modified artifacts to base wiki + id: modified-artifacts + run: | + set -eu + + mkdir -p build-info/ + + cp -a ${{ env.artifactsPath }}/build-artifacts-*/*.md build-info/ || exit 0 + echo "::set-output name=needs-wiki-update::true" + + # Pushing PDB's to the wiki only makes sense if we uploaded new app versions to the store. + - name: Copy debug databases to base wiki + if: ${{ needs.build-wsl.outputs.has-changes == 'true' }} + id: pdb-artifacts + run: | + set -eu + + mkdir -p debug-databases/ + + find ${{ env.artifactsPath }} -name "debug-database-*" -maxdepth 1 -exec sh -c 'tar -cavf debug-databases/"$(basename $1)".tar.zst' sh '{}' \; || \ + exit 0 + + echo "::set-output name=needs-wiki-update::true" + + - name: Sync wiki to repository documentation + if: ${{ steps.modified-artifacts.outputs.needs-wiki-update == 'true' }} + run: | + set -eux + git config user.name "github-actions[bot]" + git config user.email "41898282+github-actions[bot]@users.noreply.github.com" + git add . + git commit -m "Auto-update build info" + git push origin master + + - name: Checkout repositories required by ISO Tracker notification + if: ${{ steps.modified-artifacts.outputs.needs-wiki-update == 'true' }} + run: | + mkdir -p ${{ env.codeDir }} + git clone https://github.com/ubuntu/wsl.git ${{ env.codeDir }} --depth 1 + mkdir -p ${{ env.codeDir }}/uat + git clone https://git.launchpad.net/ubuntu-archive-tools ${{ env.codeDir }}/uat --depth 1 + + - name: Notify ISO Tracker + if: ${{ steps.modified-artifacts.outputs.needs-wiki-update == 'true' }} + env: + ISOTRACKER_USERNAME: ${{ secrets.ISOTRACKER_USERNAME }} + ISOTRACKER_PASSWORD: ${{ secrets.ISOTRACKER_PASSWORD }} + run: | + [ -f /tmp/all-releases.csv ] || ${{ env.codeDir }}/wsl-builder/lp-distro-info > /tmp/all-releases.csv + # There might have been more than one build in the latest commit + for build in $(git diff-tree --no-commit-id --name-only -r HEAD | grep buildid); do + AppID=$(basename $build); + AppID=${AppID%-buildid.md}; + PYTHONPATH=${{ env.codeDir }}/uat ${{ env.codeDir }}/wsl-builder/notify-isotracker --debug $AppID $GITHUB_RUN_ID + done diff --git a/.github/workflows/reserves/ubuntu-wsl/detect-update-releases.yaml b/.github/workflows/reserves/ubuntu-wsl/detect-update-releases.yaml new file mode 100644 index 000000000000..7fe6535dda0f --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/detect-update-releases.yaml @@ -0,0 +1,62 @@ +name: Update and create asset releases +on: + push: + branches: + - '**' # Ignore tag push, but take any branch + paths: + - 'meta/src/**' + - 'meta/*/src/**' + - 'DistroLauncher/DistributionInfo.h' + - 'DistroLauncher-Appx/**' + - '.github/workflows/detect-update-releases.yaml' + - 'wsl-builder/**' + schedule: + - cron: '42 0 * * 6' + +env: + goversion: '1.21.4' + +jobs: + update-releases: + name: Update and create metadata and assets for releases + runs-on: ubuntu-latest + container: + image: ubuntu:rolling + steps: + - name: Install git and dependencies + run: | + export DEBIAN_FRONTEND=noninteractive + apt update + apt install -y python3-launchpadlib libmagickwand-dev git pngquant fonts-ubuntu + - name: work around permission issue with git vulnerability (we are local here). TO REMOVE + run: git config --global --add safe.directory /__w/WSL/WSL + - uses: actions/checkout@v3 + - uses: actions/setup-go@v3 + with: + go-version: ${{ env.goversion }} + - name: Fetch current available releases from launchpad + run: | + wsl-builder/lp-distro-info > /tmp/all-releases.csv + echo "Releases fetched from Launchpad" + cat /tmp/all-releases.csv + - name: Update current metadata and asset for all releases + run: | + go build ./wsl-builder/prepare-assets + ./prepare-assets /tmp/all-releases.csv + - name: Add any changes to git + run: | + git add . + - name: Get destination branch for branch name + id: get-branch-name + shell: bash + run: echo "::set-output name=branch::${GITHUB_REF#refs/heads/}" + - name: Create Pull Request + uses: peter-evans/create-pull-request@v3 + with: + commit-message: Refreshes releases and assets + title: Refresh releases and assets + labels: automated pr + body: "[Auto-generated pull request](https://github.com/ubuntu/WSL/actions/workflows/detect-update-releases.yaml) by GitHub Action" + branch: auto-update-releases-${{ steps.get-branch-name.outputs.branch }} + token: ${{ secrets.GITHUB_TOKEN }} + delete-branch: true diff --git a/.github/workflows/reserves/ubuntu-wsl/e2e.yaml b/.github/workflows/reserves/ubuntu-wsl/e2e.yaml new file mode 100644 index 000000000000..e9c442d4929c --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/e2e.yaml @@ -0,0 +1,112 @@ +name: End-to-End Tests +on: + workflow_dispatch: + pull_request: + paths-ignore: + - docs/** +concurrency: azure-vm + +env: + az_name: wsl-ci + az_resource_group: wsl + +jobs: + vm-setup: + runs-on: ubuntu-latest + steps: + - uses: azure/login@v1 + with: + creds: ${{ secrets.AZURE_VM_CREDS }} + - name: Start the Runner + shell: bash + run: | + az vm start --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }} + + end-to-end-tests: + runs-on: self-hosted + needs: vm-setup + env: + rootFsCache: "${env:USERPROFILE}\\Downloads\\rootfs" + distroName: Ubuntu-Preview + appID: UbuntuPreview + launcher: ubuntupreview.exe + steps: + - name: Checkout + uses: actions/checkout@v3 + with: + submodules: 'recursive' + - name: Set up MSBuild (PATH) + uses: microsoft/setup-msbuild@v1.0.2 + - name: Install certificate + shell: powershell + run: | + New-Item -ItemType directory -Path certificate + Set-Content -Path certificate\certificate.txt -Value '${{ secrets.CERTIFICATE }}' + certutil -decode certificate\certificate.txt certificate\certificate.pfx + + $pwd = ConvertTo-SecureString '${{ secrets.CERTIFICATE_PASSWORD }}' -AsPlainText -Force + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:LocalMachine\Trust -FilePath certificate\certificate.pfx + Import-PfxCertificate -Password $pwd -CertStoreLocation Cert:CurrentUser\My -FilePath certificate\certificate.pfx + - name: Set up Go + uses: actions/setup-go@v3 + with: + go-version-file: ./e2e/go.work + - name: Install or update WSL + uses: ./.github/actions/wsl-install + - name: Download rootfs + uses: ./.github/actions/download-rootfs + with: + distros: ${{ env.distroName }} + path: ${{ env.rootFsCache }} + - name: Build and install the appxbundle for testing + shell: powershell + run: | + Write-Output "::group::Clearing leftovers from previous runs" + $package=$(Get-AppxPackage | Where-Object PackageFullName -like *${{ env.appID }}*) + If ( $package -ne $null -And $package -ne '' ) { + Remove-AppxPackage $package + } + Write-Output "::endgroup::" + + Write-Output "::group::Building and installing appbundle" + .\e2e\pre-e2e-testing.ps1 -AppID ${{ env.appID }} -RootfsX64 "${{ env.rootFsCache }}/${{ env.distroName }}.tar.gz" + Write-Output "::endgroup::" + - name: Run tests + shell: powershell + run: | + Write-Output "::group::Environment setup" + $env:LAUNCHER_REPO_ROOT=$PWD + $env:WSL_UTF8 = 1 + + # Shutting down WSL. This is sometimes necessary as it prevents "Access is denied" later. + wsl.exe --shutdown + wsl.exe --version + wsl.exe --unregister '${{ env.distroName }}' 2>&1 | Out-Null + + cd e2e + Write-Output "::endgroup::" + + Write-Output "::group::Tests" + go test .\launchertester\ -timeout 15m -run TestBasicSetup --distro-name '${{ env.distroName }}' --launcher-name '${{ env.launcher }}' + $exitStatus=$? + Write-Output "::endgroup::" + if ( ! $exitStatus ) { Exit(1) } + + - name: Remove the installed package + if: always() + shell: powershell + run: | + Remove-AppxPackage "$(Get-AppxPackage | Where-Object PackageFullName -like *${{ env.appID }}*)" + + stop-vm: + runs-on: ubuntu-latest + needs: [vm-setup, end-to-end-tests] + if: always() + steps: + - uses: azure/login@v1 + with: + creds: ${{ secrets.AZURE_VM_CREDS }} + - name: Deallocate the Runner + shell: bash + run: | + az vm deallocate --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }} diff --git a/.github/workflows/reserves/ubuntu-wsl/sync-gh-jira.yaml b/.github/workflows/reserves/ubuntu-wsl/sync-gh-jira.yaml new file mode 100644 index 000000000000..c051038e9f90 --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/sync-gh-jira.yaml @@ -0,0 +1,12 @@ +name: Sync GitHub issues to Jira +on: [issues, issue_comment] + +jobs: + sync-issues: + name: Sync issues to Jira + runs-on: ubuntu-latest + steps: + - uses: canonical/sync-issues-github-jira@v1 + with: + webhook-url: ${{ secrets.JIRA_WEBHOOK_URL }} + component: 'WSL' diff --git a/.github/workflows/reserves/ubuntu-wsl/wsl-example.yaml b/.github/workflows/reserves/ubuntu-wsl/wsl-example.yaml new file mode 100644 index 000000000000..84bf38442c3d --- /dev/null +++ b/.github/workflows/reserves/ubuntu-wsl/wsl-example.yaml @@ -0,0 +1,75 @@ +name: Example action using WSL + +concurrency: azure-vm + +on: + pull_request: + paths: + # We run it when changing one if its dependencies + - .github/workflows/wsl-example.yaml + - .github/actions/** + push: + branches: [main] + workflow_dispatch: + +env: + az_name: wsl-ci + az_resource_group: wsl + +jobs: + vm-setup: + runs-on: ubuntu-latest + steps: + - uses: azure/login@v1 + with: + creds: ${{ secrets.AZURE_VM_CREDS }} + - name: Start the Runner + shell: bash + run: | + az vm start --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }} + + run-script: + needs: vm-setup + runs-on: self-hosted + steps: + - name: Checkout + uses: actions/checkout@v3 + + - name: Install or update WSL + uses: ./.github/actions/wsl-install + with: + distro: Ubuntu + + - name: Checkout your repo into WSL + uses: ./.github/actions/wsl-checkout + with: + distro: Ubuntu + working-dir: "~/myrepo" + + # You'll probably use this action if you program runs entirely inside WSL + - name: Run bash inside WSL + uses: ./.github/actions/wsl-bash + with: + distro: Ubuntu + working-dir: "~/myrepo" + exec: | + echo "Hello from $(lsb_release -ds)" + + - name: Run powershell on Windows + shell: powershell + env: + WSL_UTF8: "1" # Recommended otherwise it's hard to read on Github + run: wsl --list --verbose + + stop-vm: + runs-on: ubuntu-latest + needs: [vm-setup, run-script] + if: always() + steps: + - uses: azure/login@v1 + with: + creds: ${{ secrets.AZURE_VM_CREDS }} + - name: Deallocate the Runner + shell: bash + run: | + az vm deallocate --name ${{ env.az_name }} --resource-group ${{ env.az_resource_group }} \ No newline at end of file diff --git a/.github/workflows/reserves/update-config-files.yml b/.github/workflows/reserves/update-config-files.yml new file mode 100644 index 000000000000..87af500426bc --- /dev/null +++ b/.github/workflows/reserves/update-config-files.yml @@ -0,0 +1,11 @@ +name: Update configuration files + +on: + schedule: + - cron: '0 3 * * 0' + workflow_dispatch: + +jobs: + call-update-configuration-files: + name: Update configuration files + uses: actions/reusable-workflows/.github/workflows/update-config-files.yml@main diff --git a/.github/workflows/reserves/update.yml b/.github/workflows/reserves/update.yml new file mode 100644 index 000000000000..642beb674be5 --- /dev/null +++ b/.github/workflows/reserves/update.yml @@ -0,0 +1,34 @@ +name: Update dependencies + +on: + schedule: + - cron: "0 0 * * 0" + workflow_dispatch: {} + +jobs: + update: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v2 + + - uses: ruby/setup-ruby@v1 + with: + bundler-cache: true + + - uses: actions/setup-node@v2 + with: + node-version: '14' + cache: 'npm' + + - name: script/update + run: script/update + + - name: script/build + run: script/build + + - name: Create Pull Request + uses: peter-evans/create-pull-request@v3 + with: + commit-message: Update dependencies + title: Update dependencies \ No newline at end of file diff --git a/.github/workflows/reserves/validate.yml b/.github/workflows/reserves/validate.yml new file mode 100644 index 000000000000..98035af24c90 --- /dev/null +++ b/.github/workflows/reserves/validate.yml @@ -0,0 +1,35 @@ +name: validate + +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: true + +on: + push: + branches: + - 'main' + - 'release/*' + tags: + - 'v*' + pull_request: + +jobs: + validate: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + target: + - lint + - validate-vendor + - validate-git + steps: + - + name: Checkout + uses: actions/checkout@v3 + - + name: Run + run: | + make ${{ matrix.target }} + env: + COMMIT_RANGE: ${{ format('{0}..{1}', github.sha, 'HEAD') }} diff --git a/.github/workflows/reserves/validate_links.yml b/.github/workflows/reserves/validate_links.yml new file mode 100644 index 000000000000..33fef0887640 --- /dev/null +++ b/.github/workflows/reserves/validate_links.yml @@ -0,0 +1,28 @@ +name: "Validate links" + +on: + workflow_dispatch: + schedule: + - cron: '0 0 * * *' + +env: + FILENAME: README.md + +jobs: + validate_links: + name: 'Check all links are working' + runs-on: ubuntu-latest + steps: + - name: Checkout repository + uses: actions/checkout@v2 + + - name: Set up Python + uses: actions/setup-python@v2 + with: + python-version: '3.8' + + - name: Install dependencies + run: python -m pip install -r scripts/requirements.txt + + - name: Validate all links from README.md + run: python scripts/validate/links.py ${FILENAME} diff --git a/.github/workflows/reserves/workflows-call/component-build.yml b/.github/workflows/reserves/workflows-call/component-build.yml new file mode 100644 index 000000000000..2f22fb09a19a --- /dev/null +++ b/.github/workflows/reserves/workflows-call/component-build.yml @@ -0,0 +1,88 @@ +name: "Eventide component workflow: build image and push" + +on: + workflow_call: + inputs: + component_name: + required: true + type: string + repository: + type: string + branch: + type: string + default: master + cache_image_tag: + type: boolean + default: true + + secrets: + GOOGLE_CREDENTIALS: + required: true + BUNDLE_RUBYGEMS__PKG__GITHUB__COM: + required: true + BUNDLE_GITHUB__COM: + required: true + GCLOUD_PROJECT_ID: + required: true + CHECKOUT_TOKEN: + required: false + GOOGLE_CREDENTIALS_GITHUB_ACTIONS: + required: true + +jobs: + build: + env: + IMAGEBASE: europe-docker.pkg.dev/nimble-lead-281209/eu.gcr.io/${{ inputs.component_name }} + + runs-on: self-hosted + + steps: + - uses: actions/checkout@v4 + with: + repository: ${{ inputs.repository || github.repository }} + token: ${{ secrets.CHECKOUT_TOKEN || github.token }} + ref: ${{ inputs.branch }} + + - id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + credentials_json: '${{ secrets.GOOGLE_CREDENTIALS_GITHUB_ACTIONS }}' + + - name: Set branch + run: echo "branch=${GITHUB_REF_NAME:-$(git branch | grep "\*" | cut -d\ -f2)}" >> $GITHUB_ENV + + - name: Set commit_sha + run: echo "commit_sha=${GITHUB_SHA:-$(git rev-parse HEAD)}" >> $GITHUB_ENV + + - name: Set imagetag + run: echo "imagetag=${IMAGETAG:-${{ env.branch }}-$(date +%Y-%m-%d-%H-%M-%z |sed -e 's/+//')-$(echo "${{ env.commit_sha }}"|cut -c1-8)}" >> $GITHUB_ENV + + - name: Build image and push to Docker (to GCP) + uses: docker/build-push-action@v5 + with: + context: . + load: false + push: true + cache-from: | + ${{ env.IMAGEBASE }}:latest + tags: | + ${{ env.IMAGEBASE }}:${{ env.imagetag }} + ${{ env.IMAGEBASE }}:latest + labels: | + ${{ inputs.component_name }} + build-args: | + POSTURE=operational + secrets: | + BUNDLE_RUBYGEMS__PKG__GITHUB__COM=${{ secrets.BUNDLE_RUBYGEMS__PKG__GITHUB__COM }} + BUNDLE_GITHUB__COM=${{ secrets.BUNDLE_GITHUB__COM }} + + - run: echo ${{ env.imagetag }} > imagetag + + - name: Cache imagetag + uses: actions/cache@v4 + if: ${{ inputs.cache_image_tag }} + env: + cache-name: cache-imagetag + with: + path: imagetag + key: image-cache-${{ runner.os }}-${{ github.repository }}-${{ github.ref_name }}-${{ github.sha }} diff --git a/.github/workflows/reserves/workflows-call/component-ci.yml b/.github/workflows/reserves/workflows-call/component-ci.yml new file mode 100644 index 000000000000..9a896b904a53 --- /dev/null +++ b/.github/workflows/reserves/workflows-call/component-ci.yml @@ -0,0 +1,92 @@ +name: "Eventide component workflow: CI Tests" + +on: + workflow_call: + inputs: + component_name: + required: true + type: string + + secrets: + GOOGLE_CREDENTIALS: + required: true + BUNDLE_RUBYGEMS__PKG__GITHUB__COM: + required: true + BUNDLE_GITHUB__COM: + required: false + GCLOUD_PROJECT_ID: + required: true + GOOGLE_CREDENTIALS_GITHUB_ACTIONS: + required: true + +jobs: + build: + env: + IMAGEBASE: europe-docker.pkg.dev/nimble-lead-281209/eu.gcr.io/${{ inputs.component_name }} + POSTGRES_HOST_AUTH_METHOD: trust + PGHOST: postgres + PGUSER: postgres + EVENTIDE_DATABASE_URL: postgres://message_store@postgres/message_store + DATABASE_URL: postgres://postgres@postgres:5432 + + runs-on: self-hosted + + services: + postgres: + image: postgres:alpine + ports: + - 5432:5432 + env: + POSTGRES_USER: postgres + POSTGRES_HOST_AUTH_METHOD: trust + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + steps: + - uses: actions/checkout@v4 + + - id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + credentials_json: '${{ secrets.GOOGLE_CREDENTIALS_GITHUB_ACTIONS }}' + + - name: Set branch + run: echo "branch=${GITHUB_REF_NAME:-$(git branch | grep "\*" | cut -d\ -f2)}" >> $GITHUB_ENV + + - name: Set commit_sha + run: echo "commit_sha=${GITHUB_SHA:-$(git rev-parse HEAD)}" >> $GITHUB_ENV + + - name: Set imagetag + run: echo "imagetag=${IMAGETAG:-${{ env.branch }}-$(date +%Y-%m-%d-%H-%M-%z |sed -e 's/+//')-$(echo "${{ env.commit_sha }}"|cut -c1-8)}" >> $GITHUB_ENV + + - name: Build image and load locally to Docker + uses: docker/build-push-action@v5 + with: + context: . + load: true + push: false + cache-from: | + ${{ env.IMAGEBASE }}:latest + tags: | + ${{ env.IMAGEBASE }}:${{ env.imagetag }} + labels: | + ${{ inputs.component_name }} + build-args: | + POSTURE=dev + secrets: | + BUNDLE_RUBYGEMS__PKG__GITHUB__COM=${{ secrets.BUNDLE_RUBYGEMS__PKG__GITHUB__COM }} + BUNDLE_GITHUB__COM=${{ secrets.BUNDLE_GITHUB__COM }} + + - name: Tests + run: | + docker run --network "${{ job.container.network }}" \ + --env POSTGRES_HOST_AUTH_METHOD=$POSTGRES_HOST_AUTH_METHOD \ + --env PGHOST=$PGHOST \ + --env PGUSER=$PGUSER \ + --env EVENTIDE_DATABASE_URL=$EVENTIDE_DATABASE_URL \ + --env DATABASE_URL=$DATABASE_URL \ + --rm ${{ env.IMAGEBASE }}:${{ env.imagetag }} \ + /bin/bash -c "bin/ci_tests" diff --git a/.github/workflows/reserves/workflows-call/component-ci_no-docker.yml b/.github/workflows/reserves/workflows-call/component-ci_no-docker.yml new file mode 100644 index 000000000000..ca26ee79ef25 --- /dev/null +++ b/.github/workflows/reserves/workflows-call/component-ci_no-docker.yml @@ -0,0 +1,46 @@ +name: "Eventide component workflow: no docker" + +on: + workflow_call: + secrets: + BUNDLE_RUBYGEMS__PKG__GITHUB__COM: + required: true + BUNDLE_GITHUB__COM: + required: false + +jobs: + build: + runs-on: self-hosted + + env: + ImageOS: ubuntu20 + BUNDLE_RUBYGEMS__PKG__GITHUB__COM: ${{ secrets.BUNDLE_RUBYGEMS__PKG__GITHUB__COM }} + BUNDLE_GITHUB__COM: ${{ secrets.BUNDLE_GITHUB__COM }} + POSTGRES_HOST_AUTH_METHOD: trust + PGHOST: localhost + PGUSER: postgres + EVENTIDE_DATABASE_URL: postgres://message_store@localhost/message_store + + services: + postgres: + image: postgres:alpine + ports: + - 5432:5432 + env: + POSTGRES_USER: postgres + POSTGRES_HOST_AUTH_METHOD: trust + options: >- + --health-cmd pg_isready + --health-interval 10s + --health-timeout 5s + --health-retries 5 + + steps: + - uses: actions/checkout@v4 + + - uses: ruby/setup-ruby@v1 + with: + bundler-cache: true + + - name: Tests + run: bin/ci_tests diff --git a/.github/workflows/reserves/workflows-call/component-deploy.yml b/.github/workflows/reserves/workflows-call/component-deploy.yml new file mode 100644 index 000000000000..e5099d4c35a6 --- /dev/null +++ b/.github/workflows/reserves/workflows-call/component-deploy.yml @@ -0,0 +1,70 @@ +name: "Eventide component workflow: deploy" + +on: + workflow_call: + inputs: + component_name: + required: true + type: string + namespace: + required: true + type: string + environment: + required: true + type: string + + secrets: + CLUSTER_NAME: + required: true + CLUSTER_REGION: + required: true + GOOGLE_CREDENTIALS: + required: true + GCLOUD_PROJECT_ID: + required: true + GOOGLE_CREDENTIALS_GITHUB_ACTIONS: + required: true + +jobs: + deploy: + env: + IMAGEBASE: europe-docker.pkg.dev/nimble-lead-281209/eu.gcr.io/${{ inputs.component_name }} + + runs-on: self-hosted + + steps: + - uses: actions/checkout@v4 + + - id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + credentials_json: '${{ secrets.GOOGLE_CREDENTIALS_GITHUB_ACTIONS }}' + + - id: 'get-credentials' + uses: 'google-github-actions/get-gke-credentials@v2' + with: + cluster_name: ${{ secrets.CLUSTER_NAME }} + location: ${{ secrets.CLUSTER_REGION }} + project_id: ${{ secrets.GCLOUD_PROJECT_ID }} + + - name: Cache imagetag + uses: actions/cache@v4 + env: + cache-name: cache-imagetag + with: + path: imagetag + key: image-cache-${{ runner.os }}-${{ github.repository }}-${{ github.ref_name }}-${{ github.sha }} + restore-keys: | + image-cache-${{ runner.os }}-${{ github.repository }}-${{ github.ref_name }}- + + - name: Build image and deploy + run: | + imagetag=`cat imagetag` + cd kustomize/${{ inputs.environment }} + kustomize edit set image ${{ inputs.component_name }}-image=${{ env.IMAGEBASE }}:$imagetag + kustomize build | kubectl apply --namespace=${{ inputs.namespace }} -f- + + - name: Check status + run: | + source /home/runner/status_deployment.sh + check_status_and_auto_rollback_if_necessary "${{ inputs.environment }}-${{ inputs.component_name }}" "${{ inputs.namespace }}" diff --git a/.github/workflows/reserves/workflows-call/component-package.yml b/.github/workflows/reserves/workflows-call/component-package.yml new file mode 100644 index 000000000000..430a488d146f --- /dev/null +++ b/.github/workflows/reserves/workflows-call/component-package.yml @@ -0,0 +1,30 @@ +name: "Eventide component workflow: Publish a new package version" + +on: + workflow_call: + inputs: + package: + required: true + type: string + + secrets: + PUBLISH_PACKAGE_WORKFLOW: + required: true + +jobs: + build: + runs-on: self-hosted + + steps: + - uses: actions/checkout@v4 + + - uses: ruby/setup-ruby@v1 + + - name: Build gem + run: gem build ${{ inputs.package }}.gemspec + + - name: Store version + run: echo "version=$( ls -a | grep -P ".gem$" | grep -oP "\d+\.\d+\.\d+")" >> $GITHUB_ENV + + - name: Publish gem + run: GEM_HOST_API_KEY=${{ secrets.PUBLISH_PACKAGE_WORKFLOW }} gem push --host https://rubygems.pkg.github.com/hubbado ${{ inputs.package }}-${{ env.version}}.gem diff --git a/.github/workflows/retry.yml b/.github/workflows/retry.yml new file mode 100644 index 000000000000..98c1cf778054 --- /dev/null +++ b/.github/workflows/retry.yml @@ -0,0 +1,18 @@ +name: Retry workflow + +on: + workflow_dispatch: + inputs: + run_id: + required: true +jobs: + rerun: + runs-on: ubuntu-latest + steps: + - name: rerun ${{ inputs.run_id }} + env: + GH_REPO: ${{ github.repository }} + GH_TOKEN: ${{ github.token }} + run: | + gh run watch ${{ inputs.run_id }} > /dev/null 2>&1 + gh run rerun ${{ inputs.run_id }} --debug From 80b0c8c9339e72d42747170c5dbca4bb01e28e88 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 23 Feb 2025 04:11:00 +0000 Subject: [PATCH 0002/1030] update workflows --- .github/actions/2-multiplication/2-feed/action.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 062adb50a36f..26a5a898224b 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -164,7 +164,7 @@ runs: Environment="HOME=/home/runner" ExecStop=/usr/bin/docker exec mydb -s quit ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt -v /mnt/disks/deeplearning/home/runner:/home/runner --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} + ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} - path: /etc/systemd/system/runner.service permissions: "0644" @@ -191,7 +191,7 @@ runs: runcmd: - systemctl stop docker - - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos /mnt/disks/deeplearning/home + - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos - mv /var/lib/docker/* /mnt/disks/deeplearning/var/lib/docker/cos/ || true - find /var/lib/docker -mindepth 1 -delete - rmdir /var/lib/docker || true @@ -199,7 +199,6 @@ runs: - mount --bind /mnt/disks/deeplearning/var/lib/docker/cos /var/lib/docker - systemctl daemon-reload - systemctl start docker - - docker run --rm --pull=always --entrypoint "" -v /mnt/disks/deeplearning/home:/mnt/disks/deeplearning/home eq19/feed:spin-${{ env.IMAGE_TAG }} sh -c "cp -r /home/runner /mnt/disks/deeplearning/home/" - systemctl start runner.service EOL From 9b296c01b346cde018021057ad69be063f33ec20 Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Sun, 23 Feb 2025 11:24:28 +0700 Subject: [PATCH 0003/1030] Update main.yml --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 532bad526be0..81430df8946f 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -44,7 +44,7 @@ jobs: with: submodules: recursive token: ${{ secrets.ACCESS_TOKEN }} - repository: ${{ env.ID != 1 && env.DOC || '' }} + repository: ${{ env.ID == 1 && env.DOC || '' }} - name: ⚙️ Build Maps uses: eq19/maps@v8 From d2e6df5960123704e05964ec39fd0eddb0025c9d Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Sun, 23 Feb 2025 11:58:22 +0700 Subject: [PATCH 0004/1030] Update main.yml --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 81430df8946f..ca3059a4cacb 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -32,7 +32,7 @@ jobs: - { os: macos-latest, id: 2 } env: ID: ${{ matrix.config.id }} - DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' + DOC: '${{ GITHUB_REPOSITORY }}.wiki' # 🔧 Commutation relations runs-on: ${{ matrix.config.os }} From 46317077ad2801768f7b0d540399353eea9976ca Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Sun, 23 Feb 2025 12:04:09 +0700 Subject: [PATCH 0005/1030] Update main.yml --- .github/workflows/main.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ca3059a4cacb..e0704eb25fc0 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - #push: + push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] @@ -32,7 +32,7 @@ jobs: - { os: macos-latest, id: 2 } env: ID: ${{ matrix.config.id }} - DOC: '${{ GITHUB_REPOSITORY }}.wiki' + DOC: '${GITHUB_REPOSITORY}.wiki' # 🔧 Commutation relations runs-on: ${{ matrix.config.os }} From 790adc8e13f51131afd04ebe6c0ce7429c240a0c Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Sun, 23 Feb 2025 12:09:56 +0700 Subject: [PATCH 0006/1030] Update main.yml --- .github/workflows/main.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index e0704eb25fc0..532bad526be0 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - push: + #push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] @@ -32,7 +32,7 @@ jobs: - { os: macos-latest, id: 2 } env: ID: ${{ matrix.config.id }} - DOC: '${GITHUB_REPOSITORY}.wiki' + DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' # 🔧 Commutation relations runs-on: ${{ matrix.config.os }} @@ -44,7 +44,7 @@ jobs: with: submodules: recursive token: ${{ secrets.ACCESS_TOKEN }} - repository: ${{ env.ID == 1 && env.DOC || '' }} + repository: ${{ env.ID != 1 && env.DOC || '' }} - name: ⚙️ Build Maps uses: eq19/maps@v8 From 0b502866f1e8776f92d3b0f6589ea6b9d59452d9 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 25 Feb 2025 01:21:26 +0000 Subject: [PATCH 0007/1030] update workflows --- .../2-multiplication/2-feed/action.yml | 37 ++++++++++++++++++- 1 file changed, 36 insertions(+), 1 deletion(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 26a5a898224b..689cddda0de5 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -122,6 +122,17 @@ runs: groups: admin,adm,video,docker,google-sudoers write_files: + - path: /etc/sysctl.d/99-swappiness.conf + permissions: '0644' + content: | + vm.swappiness=10 + + - path: /etc/security/limits.conf + append: true + content: | + * soft as 250000 # Soft limit of ~244MiB + * hard as 500000 # Hard limit of ~488MiB + - path: /etc/systemd/system/firewall.service permissions: "0644" owner: root @@ -184,6 +195,22 @@ runs: ExecStartPre=/usr/bin/docker-credential-gcr configure-docker ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=Google-optimized-instance -e RUNNER_REPOSITORY_URL=https://github.com/${{ inputs.repository }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name myrunner --pull=always eq19/lexer:id-${{ env.IMAGE_TAG }} + - path: /etc/systemd/system/docker-limit.service + permissions: '0644' + content: | + [Unit] + Description=Limit Docker Memory Usage + After=docker.service + Requires=docker.service + + [Service] + Type=oneshot + ExecStart=/usr/bin/docker update --memory=200M --memory-swap=300M $(docker ps -q) + RemainAfterExit=yes + + [Install] + WantedBy=multi-user.target + bootcmd: - fsck.ext4 -tvy /dev/sdb1 - mkdir -p /mnt/disks/deeplearning @@ -191,6 +218,12 @@ runs: runcmd: - systemctl stop docker + - fallocate -l 500M /swapfile + - chmod 600 /swapfile + - mkswap /swapfile + - swapon /swapfile + - echo '/swapfile none swap sw 0 0' >> /etc/fstab + - sysctl --system - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos - mv /var/lib/docker/* /mnt/disks/deeplearning/var/lib/docker/cos/ || true - find /var/lib/docker -mindepth 1 -delete @@ -200,8 +233,10 @@ runs: - systemctl daemon-reload - systemctl start docker - systemctl start runner.service + - systemctl enable docker-limit.service + - systemctl start docker-limit.service EOL - + # Ref: https://stackoverflow.com/a/47018417/4058484 - name: 🪂 Setup Runner by Gcloud #if: runner.os == 'Windows' From 30cb99ec545a381f660fb271fd8b6b54c6ad6448 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 25 Feb 2025 03:30:15 +0000 Subject: [PATCH 0008/1030] update workflows --- .../2-multiplication/2-feed/action.yml | 37 +------------------ 1 file changed, 1 insertion(+), 36 deletions(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 689cddda0de5..26a5a898224b 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -122,17 +122,6 @@ runs: groups: admin,adm,video,docker,google-sudoers write_files: - - path: /etc/sysctl.d/99-swappiness.conf - permissions: '0644' - content: | - vm.swappiness=10 - - - path: /etc/security/limits.conf - append: true - content: | - * soft as 250000 # Soft limit of ~244MiB - * hard as 500000 # Hard limit of ~488MiB - - path: /etc/systemd/system/firewall.service permissions: "0644" owner: root @@ -195,22 +184,6 @@ runs: ExecStartPre=/usr/bin/docker-credential-gcr configure-docker ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=Google-optimized-instance -e RUNNER_REPOSITORY_URL=https://github.com/${{ inputs.repository }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name myrunner --pull=always eq19/lexer:id-${{ env.IMAGE_TAG }} - - path: /etc/systemd/system/docker-limit.service - permissions: '0644' - content: | - [Unit] - Description=Limit Docker Memory Usage - After=docker.service - Requires=docker.service - - [Service] - Type=oneshot - ExecStart=/usr/bin/docker update --memory=200M --memory-swap=300M $(docker ps -q) - RemainAfterExit=yes - - [Install] - WantedBy=multi-user.target - bootcmd: - fsck.ext4 -tvy /dev/sdb1 - mkdir -p /mnt/disks/deeplearning @@ -218,12 +191,6 @@ runs: runcmd: - systemctl stop docker - - fallocate -l 500M /swapfile - - chmod 600 /swapfile - - mkswap /swapfile - - swapon /swapfile - - echo '/swapfile none swap sw 0 0' >> /etc/fstab - - sysctl --system - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos - mv /var/lib/docker/* /mnt/disks/deeplearning/var/lib/docker/cos/ || true - find /var/lib/docker -mindepth 1 -delete @@ -233,10 +200,8 @@ runs: - systemctl daemon-reload - systemctl start docker - systemctl start runner.service - - systemctl enable docker-limit.service - - systemctl start docker-limit.service EOL - + # Ref: https://stackoverflow.com/a/47018417/4058484 - name: 🪂 Setup Runner by Gcloud #if: runner.os == 'Windows' From ccb51696f3964529318ac1c25e86f2ec5f3493d8 Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 27 Feb 2025 09:43:10 +0000 Subject: [PATCH 0009/1030] update workflows --- .github/actions/2-multiplication/2-feed/action.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 26a5a898224b..db3d06a53c15 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -164,7 +164,7 @@ runs: Environment="HOME=/home/runner" ExecStop=/usr/bin/docker exec mydb -s quit ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} + ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --memory-swap=2G --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} - path: /etc/systemd/system/runner.service permissions: "0644" @@ -200,6 +200,7 @@ runs: - systemctl daemon-reload - systemctl start docker - systemctl start runner.service + - /usr/bin/docker exec mydb sh -c 'if [ ! -f /swapfile ]; then fallocate -l 2G /swapfile || dd if=/dev/zero of=/swapfile bs=1M count=2048; chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile; sysctl vm.swappiness=10; fi' EOL # Ref: https://stackoverflow.com/a/47018417/4058484 From 21868780d9dc3c3264736c10c764fe1b000a6980 Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 27 Feb 2025 10:37:52 +0000 Subject: [PATCH 0010/1030] update workflows --- .github/actions/2-multiplication/2-feed/action.yml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index db3d06a53c15..26a5a898224b 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -164,7 +164,7 @@ runs: Environment="HOME=/home/runner" ExecStop=/usr/bin/docker exec mydb -s quit ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --memory-swap=2G --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} + ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} - path: /etc/systemd/system/runner.service permissions: "0644" @@ -200,7 +200,6 @@ runs: - systemctl daemon-reload - systemctl start docker - systemctl start runner.service - - /usr/bin/docker exec mydb sh -c 'if [ ! -f /swapfile ]; then fallocate -l 2G /swapfile || dd if=/dev/zero of=/swapfile bs=1M count=2048; chmod 600 /swapfile; mkswap /swapfile; swapon /swapfile; sysctl vm.swappiness=10; fi' EOL # Ref: https://stackoverflow.com/a/47018417/4058484 From 1b0d7123f7c212bc0e6b8e373bc92377409af37f Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 27 Feb 2025 14:27:09 +0000 Subject: [PATCH 0011/1030] update workflows --- .github/actions/2-multiplication/2-feed/action.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 26a5a898224b..5d5991b690de 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -164,7 +164,7 @@ runs: Environment="HOME=/home/runner" ExecStop=/usr/bin/docker exec mydb -s quit ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} + ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --memory=512M --memory-swap=1G --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} - path: /etc/systemd/system/runner.service permissions: "0644" @@ -191,6 +191,11 @@ runs: runcmd: - systemctl stop docker + - fallocate -l 2G /mnt/disks/deeplearning/swapfile || dd if=/dev/zero of=/mnt/disks/deeplearning/swapfile bs=1M count=2048 + - chmod 600 /mnt/disks/deeplearning/swapfile + - mkswap /mnt/disks/deeplearning/swapfile + - swapon /mnt/disks/deeplearning/swapfile + - echo '/mnt/disks/deeplearning/swapfile none swap sw 0 0' >> /etc/fstab - mkdir -p /mnt/disks/deeplearning/var/lib/docker/cos - mv /var/lib/docker/* /mnt/disks/deeplearning/var/lib/docker/cos/ || true - find /var/lib/docker -mindepth 1 -delete From bff46969969bb5e582a7781162e96d2d22d6f2dc Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Fri, 28 Feb 2025 09:46:22 +0700 Subject: [PATCH 0012/1030] Update main.yml --- .github/workflows/main.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 532bad526be0..bc226b701414 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - #push: + push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] @@ -35,6 +35,7 @@ jobs: DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' # 🔧 Commutation relations + needs: parsering runs-on: ${{ matrix.config.os }} continue-on-error: ${{ matrix.config.id != 1 && false || true }} @@ -105,7 +106,7 @@ jobs: strategy: matrix: config: - - { os: self-hosted, id: 3 } + #- { os: self-hosted, id: 3 } - { os: windows-latest, id: 4 } # 💢 Completeness relations @@ -117,7 +118,7 @@ jobs: PASS: ${{ contains(needs.lexering.result, 'success') }} ## ✔️ Completeness status - needs: lexering + #needs: lexering runs-on: ${{ matrix.config.os }} #if: ${{ (needs.lexering.result == 'success') }} continue-on-error: ${{ matrix.config.id == 3 && true || false }} From 5b9bfd3221b2b7e3335cb7b470f57e6038c83c77 Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Fri, 28 Feb 2025 09:50:44 +0700 Subject: [PATCH 0013/1030] Update main.yml --- .github/workflows/main.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bc226b701414..95ca86b76dbb 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -28,7 +28,7 @@ jobs: fail-fast: true matrix: config: - - { os: ubuntu-latest, id: 1 } + #- { os: ubuntu-latest, id: 1 } - { os: macos-latest, id: 2 } env: ID: ${{ matrix.config.id }} @@ -133,7 +133,7 @@ jobs: ref: ${{ matrix.config.id == 3 && 'gh-source' || 'gh-pages' }} - name: 🚀 Setup Parser - if: ${{ env.PASS == 'true' }} + #if: ${{ env.PASS == 'true' }} uses: eq19/parser@v2 id: set-parser with: @@ -145,7 +145,7 @@ jobs: (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} - name: 🌀 Evaluate Syntax - if: ${{ env.PASS == 'true' }} + #if: ${{ env.PASS == 'true' }} uses: eq19/syntax@v2 id: eval-syntax with: From 34ef072aeb8d897558d517f957fae2a5634e9f3f Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Tue, 4 Mar 2025 00:40:47 +0700 Subject: [PATCH 0014/1030] Update main.yml --- .github/workflows/main.yml | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 95ca86b76dbb..532bad526be0 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - push: + #push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] @@ -28,14 +28,13 @@ jobs: fail-fast: true matrix: config: - #- { os: ubuntu-latest, id: 1 } + - { os: ubuntu-latest, id: 1 } - { os: macos-latest, id: 2 } env: ID: ${{ matrix.config.id }} DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' # 🔧 Commutation relations - needs: parsering runs-on: ${{ matrix.config.os }} continue-on-error: ${{ matrix.config.id != 1 && false || true }} @@ -106,7 +105,7 @@ jobs: strategy: matrix: config: - #- { os: self-hosted, id: 3 } + - { os: self-hosted, id: 3 } - { os: windows-latest, id: 4 } # 💢 Completeness relations @@ -118,7 +117,7 @@ jobs: PASS: ${{ contains(needs.lexering.result, 'success') }} ## ✔️ Completeness status - #needs: lexering + needs: lexering runs-on: ${{ matrix.config.os }} #if: ${{ (needs.lexering.result == 'success') }} continue-on-error: ${{ matrix.config.id == 3 && true || false }} @@ -133,7 +132,7 @@ jobs: ref: ${{ matrix.config.id == 3 && 'gh-source' || 'gh-pages' }} - name: 🚀 Setup Parser - #if: ${{ env.PASS == 'true' }} + if: ${{ env.PASS == 'true' }} uses: eq19/parser@v2 id: set-parser with: @@ -145,7 +144,7 @@ jobs: (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} - name: 🌀 Evaluate Syntax - #if: ${{ env.PASS == 'true' }} + if: ${{ env.PASS == 'true' }} uses: eq19/syntax@v2 id: eval-syntax with: From 945449467a4721d65ef5ec8eb1330b9fab5a3e08 Mon Sep 17 00:00:00 2001 From: eq19 Date: Mon, 3 Mar 2025 21:30:33 +0000 Subject: [PATCH 0015/1030] update workflows --- .../actions/4-identition/5-syntax/action.yml | 6 +- .../actions/4-identition/6-grammar/action.yml | 14 +- .github/entrypoint/entrypoint_.sh | 202 ------------------ .github/entrypoint/github_pages.sh | 10 +- .github/entrypoint/init.sh | 3 +- .github/entrypoint/wiki.sh | 14 +- 6 files changed, 23 insertions(+), 226 deletions(-) delete mode 100755 .github/entrypoint/entrypoint_.sh diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index 15c8f27eca46..04ad910942fe 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -81,6 +81,7 @@ runs: using: composite steps: + # See: https://github.com/svopper/munaiah-analyser/blob/18cb09be3a343a051239d5b6378ed7524dad3147/out/groonga/grntest/.github_workflows_windows.yml#L50 - name: 💎 Setup Ruby uses: ruby/setup-ruby@v1 if: runner.os == 'Windows' @@ -111,12 +112,7 @@ runs: JOB_ID: ${{ inputs.id }} GH_TOKEN: ${{ inputs.token }} GCP_TOKEN: ${{ steps.auth.outputs.id_token }} - BASE_GIT: "/d/a/_actions/eq19/eq19/v2/.github" - BASE: "https://github.com/eq19/eq19.github.io.wiki.git" - WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" run: | curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ -H "Authorization: Bearer ${GCP_TOKEN}" -H "Content-Type: application/json" \ --data @${{ inputs.workspace }}/docs/data.json | jq '.' - export PATH=$BASE_GIT/entrypoint:$PATH && github_pages.sh - diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index e566e16d9359..e6de41483bdd 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -109,17 +109,21 @@ runs: cat _config.yml echo -e "\n$hr\nSource Pages\n$hr" - ls -al ${GITHUB_WORKSPACE//\\//} + ls -al . + + FILE=$(find . -name "plugin_manager.rb") + sed -i 's|unless site.safe|if site.safe|g' $FILE + sed -i 's|plugins_path.each|Array(["./_plugins"]).each|g' $FILE echo -e "\n$hr\nBuild Pages\n$hr" #Jekyll Quick Reference https://gist.github.com/DrOctogon/bfb6e392aa5654c63d12 - JEKYLL_GITHUB_TOKEN=${{ inputs.token }} DISABLE_WHITELIST=true bundle exec jekyll build --profile -t -p ${GITHUB_WORKSPACE//\\//}/_plugins -d ${GITHUB_WORKSPACE//\\//}/docs + JEKYLL_GITHUB_TOKEN=${{ inputs.token }} DISABLE_WHITELIST=true bundle exec jekyll build --profile --trace -t -d ./docs - if [[ "${TARGET_REPOSITORY}" == "eq19/eq19.github.io" ]]; then echo "www.eq19.com" > ${GITHUB_WORKSPACE//\\//}/docs/CNAME; fi - rm -rf ${GITHUB_WORKSPACE//\\//}/docs/.nojekyll && touch ${GITHUB_WORKSPACE//\\//}/docs/.nojekyll + if [[ "${TARGET_REPOSITORY}" == "eq19/eq19.github.io" ]]; then echo "www.eq19.com" > ./docs/CNAME; fi + rm -rf ./docs/.nojekyll && touch ./docs/.nojekyll echo -e "\n$hr\nFinal Pages\n$hr" - ls -al ${GITHUB_WORKSPACE//\\//}/docs + ls -al ./docs # Assign the environment variable correctly COMMIT="${{ env.LATEST_COMMIT }}" diff --git a/.github/entrypoint/entrypoint_.sh b/.github/entrypoint/entrypoint_.sh deleted file mode 100755 index 4e062fd313ed..000000000000 --- a/.github/entrypoint/entrypoint_.sh +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/env bash - -set -e -export WORKING_DIR=${PWD} -chown -R $(whoami) ${WORKING_DIR} -export hr=$(printf '=%.0s' {1..80}) - -# Initial default value -export TOKEN=${INPUT_TOKEN} -export ACTOR=${GITHUB_ACTOR} -export REPOSITORY=${INPUT_REPOSITORY} -export OWNER=${GITHUB_REPOSITORY_OWNER} -export BRANCH=${INPUT_BRANCH:=gh-pages} -export PROVIDER=${INPUT_PROVIDER:=github} -export BUNDLER_VER=${INPUT_BUNDLER_VER:=>=0} -export JEKYLL_BASEURL=${INPUT_JEKYLL_BASEURL:=} -export PRE_BUILD_COMMANDS=${INPUT_PRE_BUILD_COMMANDS:=} -export SCRIPT_DIR=/maps/Scripts && chmod -R +x ${SCRIPT_DIR} -export PREFIX=${WORKING_DIR}/docs/_data/ta-lib/linux/ta-lib - -# https://stackoverflow.com/a/42137273/4058484 -export JEKYLL_SRC=${WORKING_DIR} -export JEKYLL_GITHUB_TOKEN=${TOKEN} - -if [[ "${OWNER}" == "chetabahana" ]]; then - export JEKYLL_SRC=${WORKING_DIR}/docs -fi -export JEKYLL_CFG=${JEKYLL_SRC}/_config.yml -sed -i -e "s/eq19/${OWNER}/g" ${JEKYLL_CFG} - -if [[ -z "${TOKEN}" ]]; then - echo -e "Please set the TOKEN environment variable." - exit 1 -fi - -# Pre-handle Jekyll baseurl -if [[ -n "${JEKYLL_BASEURL-}" ]]; then - JEKYLL_BASEURL="--baseurl ${JEKYLL_BASEURL}" -fi - -# Initialize environment -export RUBYOPT=W0 -export RUBY_VERSION=2.7.0 -export RAILS_VERSION=5.0.1 -export BUNDLER_VER=${BUNDLER_VER} -export BUNDLE_GEMFILE=/maps/Gemfile -export BUNDLE_SILENCE_ROOT_WARNING=1 -#export TA_LIBRARY_PATH=${PREFIX}/lib -export NOKOGIRI_USE_SYSTEM_LIBRARIES=1 -#export TA_INCLUDE_PATH=${PREFIX}/include -export PAGES_REPO_NWO=$GITHUB_REPOSITORY -export VENDOR_BUNDLE=${WORKING_DIR}/vendor/bundle -export SSL_CERT_FILE=$(realpath .github/hook-scripts/cacert.pem) - -# identity -echo -e "\n$hr\nWHOAMI\n$hr" -whoami -pwd -id - -# os version -echo -e "\n$hr\nOS VERSION\n$hr" -cat /etc/os-release -uname -r - -# file system -echo -e "\n$hr\nFILE SYSTEM\n$hr" -df -h - -# rootdir -echo -e "$hr\nROOT DIR\n$hr" -cd / && pwd && ls -al - -# packages -echo -e "$hr\nVENV DIR\n$hr" -cd /maps && pwd && ls -al - -# current -echo -e "$hr\nWORK DIR\n$hr" -cd ${WORKING_DIR} && pwd && ls -al - -#echo -e "$hr\nPRIOR INSTALLATION\n$hr" -#chown -R root:root ${HOME} && dpkg -l - -# cloning default repository -# https://stackoverflow.com/a/74439875/4058484 -git config --global user.name "${ACTOR}" -git config --global user.email "${ACTOR}@users.noreply.github.com" -git config --global credential.helper store &>/dev/null -echo "https://{ACTOR}:${TOKEN}@github.com" > /root/.git-credentials -git clone --quiet --recurse-submodules -j8 ${REPOSITORY} /maps/feed/default - -export NPM_CACHE_DIR=${VENDOR_BUNDLE}/npm -apt-get install -qq npm &>/dev/null && apt-get install -qq yarn &>/dev/null -npm install --prefix /maps --cache ${NPM_CACHE_DIR} &>/dev/null - -export PATH=${HOME}/.local/bin:$PATH -export PIP_CACHE_DIR=${VENDOR_BUNDLE}/pip - -#wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz &>/dev/null -cd ${PREFIX} #tar -xzf ta-lib-0.4.0-src.tar.gz &>/dev/null && cd ta-lib/ -./configure --prefix=/usr &>/dev/null && make &>/dev/null && make install &>/dev/null - -cd ${WORKING_DIR} -# https://pypi.org/project/pipx/ -python -m pip install --upgrade pip setuptools six wheel &>/dev/null -python -m pip install pytest-cov -r /maps/requirements.txt &>/dev/null -python -m pip install jax[cuda11_cudnn82] \ - -f https://storage.googleapis.com/jax-releases/jax_releases.html &>/dev/null - -export GEM_PATH=${VENDOR_BUNDLE}/gem -export GEM_HOME=${GEM_PATH}/ruby/${RUBY_VERSION} -export PATH=$PATH:${GEM_HOME}/bin - -# https://stackoverflow.com/a/30369485/4058484 -apt-get install -qq ruby ruby-dev ruby-bundler build-essential &>/dev/null -gem install rails --version "$RAILS_VERSION" --quiet --silent &>/dev/null - -# installed packages -#echo -e "\n$hr\nUPON INSTALLATION\n$hr" -#dpkg -l - -# Setting default ruby version -echo -e "$hr\nTENSORFLOW VERSION\n$hr" -pip show tensorflow-gpu && pip -V - -# https://stackoverflow.com/a/60945404/4058484 -ruby -v && bundler version && python -V -node -v && npm -v - -# Restore modification time (mtime) of git files -echo -e "$hr\nEPOCH TEST\n$hr" -${SCRIPT_DIR}/restore.sh -${SCRIPT_DIR}/prime_list.sh -${SCRIPT_DIR}/init_environment.sh - -# Clean up bundler cache -CLEANUP_BUNDLER_CACHE_DONE=false -bundle config path ${GEM_PATH} -bundle config cache_all true - -cleanup_bundler_cache() { - ${SCRIPT_DIR}/cleanup_bundler.sh - rm -rf ${GEM_HOME} && mkdir -p ${GEM_HOME} - gem install bundler -v "${BUNDLER_VER}" &>/dev/null - echo -e "\nCLEANUP BUNDLE\n$hr" && bundle install - CLEANUP_BUNDLER_CACHE_DONE=true -} - -# If the vendor/bundle folder is cached in a differnt OS (e.g. Ubuntu), -# it would cause `jekyll build` failed, we should clean up the cache firstly. -OS_NAME_FILE=${GEM_PATH}/os-name -os_name=$(cat /etc/os-release | grep '^NAME=') -os_name=${os_name:6:-1} - -if [[ "$os_name" != "$(cat $OS_NAME_FILE 2>/dev/null)" ]]; then - cleanup_bundler_cache - echo -e $os_name > $OS_NAME_FILE -fi - -# Check and execute pre_build_commands commands -cd ${JEKYLL_SRC} - -if [[ ${PRE_BUILD_COMMANDS} ]]; then - eval "${PRE_BUILD_COMMANDS}" -fi - -build_jekyll() { - echo -e "\n$hr\nJEKYLL BUILD\n$hr" && pwd - # https://gist.github.com/DrOctogon/bfb6e392aa5654c63d12 - JEKYLL_GITHUB_TOKEN=${TOKEN} bundle exec jekyll build --trace --profile \ - ${JEKYLL_BASEURL} -c ${JEKYLL_CFG} -d ${WORKING_DIR}/build - - # vendor/bundle - echo -e "\n$hr\nVENDOR BUNDLE\n$hr" && chown -R root:root ${HOME} - cd ${HOME} && mv -f .gem gem && mv -f gem/* ${VENDOR_BUNDLE}/gem/ - echo ${VENDOR_BUNDLE} && ls -al ${VENDOR_BUNDLE} && echo -e "\n" - echo ${NPM_CACHE_DIR} && ls -al ${NPM_CACHE_DIR} && echo -e "\n" - echo ${PIP_CACHE_DIR} && ls -al ${PIP_CACHE_DIR} && echo -e "\n" - echo ${GEM_PATH} && ls -al ${GEM_PATH} && echo -e "\n" - echo ${GEM_HOME} && ls -al ${GEM_HOME} && echo -e "\n" - chmod -R a+rwx,o-w ${PIP_CACHE_DIR} - echo -e "Source cleaning..\n" - rm -rf gem && ls -al -} - -build_jekyll || { - $CLEANUP_BUNDLER_CACHE_DONE && exit -1 - echo -e "\nRebuild all gems and try to build again\n" - cleanup_bundler_cache - build_jekyll -} - - -cd ${WORKING_DIR} -echo -e "$hr\nDEPLOYMENT\n$hr" -# https://unix.stackexchange.com/a/83895/158462 -git submodule foreach -q ${SCRIPT_DIR}/github_pages.sh - - -apt-get clean && rm -rf /var/lib/apt/lists/* ${RUNNER_TEMP}/* /var${RUNNER_TEMP}/* -exit $? diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh index 1dee0d06e8d9..cf5f46807724 100755 --- a/.github/entrypoint/github_pages.sh +++ b/.github/entrypoint/github_pages.sh @@ -4,17 +4,17 @@ hr='------------------------------------------------------------------------------------' if [[ "${WIKI}" != "${BASE}" ]]; then - rm -rf ${RUNNER_TEMP//\\//}/wikidir + rm -rf ${RUNNER_TEMP}/wikidir - git clone $WIKI ${RUNNER_TEMP//\\//}/wikidir - cd ${RUNNER_TEMP//\\//}/wikidir && mv -f Home.md README.md + git clone $WIKI ${RUNNER_TEMP}/wikidir + cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP}/README.md >> $1/README.md' sh {} \; find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; fi #echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cd ${GITHUB_WORKSPACE//\\//} && pwd && ls -alR . +cd ${GITHUB_WORKSPACE} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 3437b845ba47..524438f45cdd 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -82,7 +82,7 @@ elif [[ "${JOB_ID}" == "3" ]]; then rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && ls -lR . + cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . else @@ -90,7 +90,6 @@ else git clone --single-branch --branch gh-source $TARGET_REPO gh-source cd ${GITHUB_WORKSPACE//\\//} - #find -not -path "./.git/*" -not -name ".git" | grep git find -not -path "./.git/*" -not -name ".git" -delete rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh index ef6077e4eb96..2237ab0f5f3e 100755 --- a/.github/entrypoint/wiki.sh +++ b/.github/entrypoint/wiki.sh @@ -23,18 +23,18 @@ gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 find $1 -type f -name "README.md" -exec rm -rf {} \; -rm -rf ${RUNNER_TEMP//\\//}/README.md +rm -rf ${RUNNER_TEMP}/README.md -mv -f ${RUNNER_TEMP//\\//}/wikidir/README.md ${RUNNER_TEMP//\\//}/README.md -cp -R ${RUNNER_TEMP//\\//}/wikidir/* $1 +mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md +cp -R ${RUNNER_TEMP}/wikidir/* $1 -rm -rf ${RUNNER_TEMP//\\//}/spin.txt && touch ${RUNNER_TEMP//\\//}/spin.txt +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done -rm -rf ${RUNNER_TEMP//\\//}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP//\\//}/Sidebar.md -sed -i 's/0. \[\[//g' ${RUNNER_TEMP//\\//}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP//\\//}/Sidebar.md +rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md +sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' echo -e "\n$hr\nSPIN\n$hr" -cat ${RUNNER_TEMP//\\//}/spin.txt +cat ${RUNNER_TEMP}/spin.txt From 69cdb992c41764c649b899caa31cc35a1062aaff Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 4 Mar 2025 05:31:24 +0000 Subject: [PATCH 0016/1030] update workflows --- .../actions/4-identition/5-syntax/action.yml | 5 + .github/entrypoint/entrypoint_.sh | 202 ++++++++++++++++++ .github/entrypoint/github_pages.sh | 10 +- .github/entrypoint/init.sh | 3 +- .github/entrypoint/wiki.sh | 14 +- 5 files changed, 221 insertions(+), 13 deletions(-) create mode 100755 .github/entrypoint/entrypoint_.sh diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index 04ad910942fe..386b8b5d4a8c 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -112,7 +112,12 @@ runs: JOB_ID: ${{ inputs.id }} GH_TOKEN: ${{ inputs.token }} GCP_TOKEN: ${{ steps.auth.outputs.id_token }} + BASE_GIT: "/d/a/_actions/eq19/eq19/v2/.github" + BASE: "https://github.com/eq19/eq19.github.io.wiki.git" + WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" run: | curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ -H "Authorization: Bearer ${GCP_TOKEN}" -H "Content-Type: application/json" \ --data @${{ inputs.workspace }}/docs/data.json | jq '.' + export PATH=$BASE_GIT/entrypoint:$PATH && github_pages.sh + diff --git a/.github/entrypoint/entrypoint_.sh b/.github/entrypoint/entrypoint_.sh new file mode 100755 index 000000000000..4e062fd313ed --- /dev/null +++ b/.github/entrypoint/entrypoint_.sh @@ -0,0 +1,202 @@ +#!/usr/bin/env bash + +set -e +export WORKING_DIR=${PWD} +chown -R $(whoami) ${WORKING_DIR} +export hr=$(printf '=%.0s' {1..80}) + +# Initial default value +export TOKEN=${INPUT_TOKEN} +export ACTOR=${GITHUB_ACTOR} +export REPOSITORY=${INPUT_REPOSITORY} +export OWNER=${GITHUB_REPOSITORY_OWNER} +export BRANCH=${INPUT_BRANCH:=gh-pages} +export PROVIDER=${INPUT_PROVIDER:=github} +export BUNDLER_VER=${INPUT_BUNDLER_VER:=>=0} +export JEKYLL_BASEURL=${INPUT_JEKYLL_BASEURL:=} +export PRE_BUILD_COMMANDS=${INPUT_PRE_BUILD_COMMANDS:=} +export SCRIPT_DIR=/maps/Scripts && chmod -R +x ${SCRIPT_DIR} +export PREFIX=${WORKING_DIR}/docs/_data/ta-lib/linux/ta-lib + +# https://stackoverflow.com/a/42137273/4058484 +export JEKYLL_SRC=${WORKING_DIR} +export JEKYLL_GITHUB_TOKEN=${TOKEN} + +if [[ "${OWNER}" == "chetabahana" ]]; then + export JEKYLL_SRC=${WORKING_DIR}/docs +fi +export JEKYLL_CFG=${JEKYLL_SRC}/_config.yml +sed -i -e "s/eq19/${OWNER}/g" ${JEKYLL_CFG} + +if [[ -z "${TOKEN}" ]]; then + echo -e "Please set the TOKEN environment variable." + exit 1 +fi + +# Pre-handle Jekyll baseurl +if [[ -n "${JEKYLL_BASEURL-}" ]]; then + JEKYLL_BASEURL="--baseurl ${JEKYLL_BASEURL}" +fi + +# Initialize environment +export RUBYOPT=W0 +export RUBY_VERSION=2.7.0 +export RAILS_VERSION=5.0.1 +export BUNDLER_VER=${BUNDLER_VER} +export BUNDLE_GEMFILE=/maps/Gemfile +export BUNDLE_SILENCE_ROOT_WARNING=1 +#export TA_LIBRARY_PATH=${PREFIX}/lib +export NOKOGIRI_USE_SYSTEM_LIBRARIES=1 +#export TA_INCLUDE_PATH=${PREFIX}/include +export PAGES_REPO_NWO=$GITHUB_REPOSITORY +export VENDOR_BUNDLE=${WORKING_DIR}/vendor/bundle +export SSL_CERT_FILE=$(realpath .github/hook-scripts/cacert.pem) + +# identity +echo -e "\n$hr\nWHOAMI\n$hr" +whoami +pwd +id + +# os version +echo -e "\n$hr\nOS VERSION\n$hr" +cat /etc/os-release +uname -r + +# file system +echo -e "\n$hr\nFILE SYSTEM\n$hr" +df -h + +# rootdir +echo -e "$hr\nROOT DIR\n$hr" +cd / && pwd && ls -al + +# packages +echo -e "$hr\nVENV DIR\n$hr" +cd /maps && pwd && ls -al + +# current +echo -e "$hr\nWORK DIR\n$hr" +cd ${WORKING_DIR} && pwd && ls -al + +#echo -e "$hr\nPRIOR INSTALLATION\n$hr" +#chown -R root:root ${HOME} && dpkg -l + +# cloning default repository +# https://stackoverflow.com/a/74439875/4058484 +git config --global user.name "${ACTOR}" +git config --global user.email "${ACTOR}@users.noreply.github.com" +git config --global credential.helper store &>/dev/null +echo "https://{ACTOR}:${TOKEN}@github.com" > /root/.git-credentials +git clone --quiet --recurse-submodules -j8 ${REPOSITORY} /maps/feed/default + +export NPM_CACHE_DIR=${VENDOR_BUNDLE}/npm +apt-get install -qq npm &>/dev/null && apt-get install -qq yarn &>/dev/null +npm install --prefix /maps --cache ${NPM_CACHE_DIR} &>/dev/null + +export PATH=${HOME}/.local/bin:$PATH +export PIP_CACHE_DIR=${VENDOR_BUNDLE}/pip + +#wget http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz &>/dev/null +cd ${PREFIX} #tar -xzf ta-lib-0.4.0-src.tar.gz &>/dev/null && cd ta-lib/ +./configure --prefix=/usr &>/dev/null && make &>/dev/null && make install &>/dev/null + +cd ${WORKING_DIR} +# https://pypi.org/project/pipx/ +python -m pip install --upgrade pip setuptools six wheel &>/dev/null +python -m pip install pytest-cov -r /maps/requirements.txt &>/dev/null +python -m pip install jax[cuda11_cudnn82] \ + -f https://storage.googleapis.com/jax-releases/jax_releases.html &>/dev/null + +export GEM_PATH=${VENDOR_BUNDLE}/gem +export GEM_HOME=${GEM_PATH}/ruby/${RUBY_VERSION} +export PATH=$PATH:${GEM_HOME}/bin + +# https://stackoverflow.com/a/30369485/4058484 +apt-get install -qq ruby ruby-dev ruby-bundler build-essential &>/dev/null +gem install rails --version "$RAILS_VERSION" --quiet --silent &>/dev/null + +# installed packages +#echo -e "\n$hr\nUPON INSTALLATION\n$hr" +#dpkg -l + +# Setting default ruby version +echo -e "$hr\nTENSORFLOW VERSION\n$hr" +pip show tensorflow-gpu && pip -V + +# https://stackoverflow.com/a/60945404/4058484 +ruby -v && bundler version && python -V +node -v && npm -v + +# Restore modification time (mtime) of git files +echo -e "$hr\nEPOCH TEST\n$hr" +${SCRIPT_DIR}/restore.sh +${SCRIPT_DIR}/prime_list.sh +${SCRIPT_DIR}/init_environment.sh + +# Clean up bundler cache +CLEANUP_BUNDLER_CACHE_DONE=false +bundle config path ${GEM_PATH} +bundle config cache_all true + +cleanup_bundler_cache() { + ${SCRIPT_DIR}/cleanup_bundler.sh + rm -rf ${GEM_HOME} && mkdir -p ${GEM_HOME} + gem install bundler -v "${BUNDLER_VER}" &>/dev/null + echo -e "\nCLEANUP BUNDLE\n$hr" && bundle install + CLEANUP_BUNDLER_CACHE_DONE=true +} + +# If the vendor/bundle folder is cached in a differnt OS (e.g. Ubuntu), +# it would cause `jekyll build` failed, we should clean up the cache firstly. +OS_NAME_FILE=${GEM_PATH}/os-name +os_name=$(cat /etc/os-release | grep '^NAME=') +os_name=${os_name:6:-1} + +if [[ "$os_name" != "$(cat $OS_NAME_FILE 2>/dev/null)" ]]; then + cleanup_bundler_cache + echo -e $os_name > $OS_NAME_FILE +fi + +# Check and execute pre_build_commands commands +cd ${JEKYLL_SRC} + +if [[ ${PRE_BUILD_COMMANDS} ]]; then + eval "${PRE_BUILD_COMMANDS}" +fi + +build_jekyll() { + echo -e "\n$hr\nJEKYLL BUILD\n$hr" && pwd + # https://gist.github.com/DrOctogon/bfb6e392aa5654c63d12 + JEKYLL_GITHUB_TOKEN=${TOKEN} bundle exec jekyll build --trace --profile \ + ${JEKYLL_BASEURL} -c ${JEKYLL_CFG} -d ${WORKING_DIR}/build + + # vendor/bundle + echo -e "\n$hr\nVENDOR BUNDLE\n$hr" && chown -R root:root ${HOME} + cd ${HOME} && mv -f .gem gem && mv -f gem/* ${VENDOR_BUNDLE}/gem/ + echo ${VENDOR_BUNDLE} && ls -al ${VENDOR_BUNDLE} && echo -e "\n" + echo ${NPM_CACHE_DIR} && ls -al ${NPM_CACHE_DIR} && echo -e "\n" + echo ${PIP_CACHE_DIR} && ls -al ${PIP_CACHE_DIR} && echo -e "\n" + echo ${GEM_PATH} && ls -al ${GEM_PATH} && echo -e "\n" + echo ${GEM_HOME} && ls -al ${GEM_HOME} && echo -e "\n" + chmod -R a+rwx,o-w ${PIP_CACHE_DIR} + echo -e "Source cleaning..\n" + rm -rf gem && ls -al +} + +build_jekyll || { + $CLEANUP_BUNDLER_CACHE_DONE && exit -1 + echo -e "\nRebuild all gems and try to build again\n" + cleanup_bundler_cache + build_jekyll +} + + +cd ${WORKING_DIR} +echo -e "$hr\nDEPLOYMENT\n$hr" +# https://unix.stackexchange.com/a/83895/158462 +git submodule foreach -q ${SCRIPT_DIR}/github_pages.sh + + +apt-get clean && rm -rf /var/lib/apt/lists/* ${RUNNER_TEMP}/* /var${RUNNER_TEMP}/* +exit $? diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh index cf5f46807724..1dee0d06e8d9 100755 --- a/.github/entrypoint/github_pages.sh +++ b/.github/entrypoint/github_pages.sh @@ -4,17 +4,17 @@ hr='------------------------------------------------------------------------------------' if [[ "${WIKI}" != "${BASE}" ]]; then - rm -rf ${RUNNER_TEMP}/wikidir + rm -rf ${RUNNER_TEMP//\\//}/wikidir - git clone $WIKI ${RUNNER_TEMP}/wikidir - cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md + git clone $WIKI ${RUNNER_TEMP//\\//}/wikidir + cd ${RUNNER_TEMP//\\//}/wikidir && mv -f Home.md README.md find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP}/README.md >> $1/README.md' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; fi #echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cd ${GITHUB_WORKSPACE} && pwd && ls -alR . +cd ${GITHUB_WORKSPACE//\\//} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 524438f45cdd..3437b845ba47 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -82,7 +82,7 @@ elif [[ "${JOB_ID}" == "3" ]]; then rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . + cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && ls -lR . else @@ -90,6 +90,7 @@ else git clone --single-branch --branch gh-source $TARGET_REPO gh-source cd ${GITHUB_WORKSPACE//\\//} + #find -not -path "./.git/*" -not -name ".git" | grep git find -not -path "./.git/*" -not -name ".git" -delete rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh index 2237ab0f5f3e..ef6077e4eb96 100755 --- a/.github/entrypoint/wiki.sh +++ b/.github/entrypoint/wiki.sh @@ -23,18 +23,18 @@ gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 find $1 -type f -name "README.md" -exec rm -rf {} \; -rm -rf ${RUNNER_TEMP}/README.md +rm -rf ${RUNNER_TEMP//\\//}/README.md -mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md -cp -R ${RUNNER_TEMP}/wikidir/* $1 +mv -f ${RUNNER_TEMP//\\//}/wikidir/README.md ${RUNNER_TEMP//\\//}/README.md +cp -R ${RUNNER_TEMP//\\//}/wikidir/* $1 -rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +rm -rf ${RUNNER_TEMP//\\//}/spin.txt && touch ${RUNNER_TEMP//\\//}/spin.txt find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done -rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md -sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md +rm -rf ${RUNNER_TEMP//\\//}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP//\\//}/Sidebar.md +sed -i 's/0. \[\[//g' ${RUNNER_TEMP//\\//}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP//\\//}/Sidebar.md find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' echo -e "\n$hr\nSPIN\n$hr" -cat ${RUNNER_TEMP}/spin.txt +cat ${RUNNER_TEMP//\\//}/spin.txt From aeda553204cb56c1d06cee9f6fec2132624068a9 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 4 Mar 2025 08:10:21 +0000 Subject: [PATCH 0017/1030] update workflows --- .github/actions/4-identition/5-syntax/action.yml | 7 +------ .github/entrypoint/front.sh | 4 ++-- .github/entrypoint/github_pages.sh | 8 ++++---- .github/entrypoint/init.sh | 2 +- .github/entrypoint/wiki.sh | 14 +++++++------- 5 files changed, 15 insertions(+), 20 deletions(-) diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index 386b8b5d4a8c..7578a976a04e 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -112,12 +112,7 @@ runs: JOB_ID: ${{ inputs.id }} GH_TOKEN: ${{ inputs.token }} GCP_TOKEN: ${{ steps.auth.outputs.id_token }} - BASE_GIT: "/d/a/_actions/eq19/eq19/v2/.github" - BASE: "https://github.com/eq19/eq19.github.io.wiki.git" - WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" run: | curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ -H "Authorization: Bearer ${GCP_TOKEN}" -H "Content-Type: application/json" \ - --data @${{ inputs.workspace }}/docs/data.json | jq '.' - export PATH=$BASE_GIT/entrypoint:$PATH && github_pages.sh - + --data @${{ inputs.workspace }}/docs/data.json | jq '.' diff --git a/.github/entrypoint/front.sh b/.github/entrypoint/front.sh index b6230c651082..9e7967ac2855 100755 --- a/.github/entrypoint/front.sh +++ b/.github/entrypoint/front.sh @@ -12,9 +12,9 @@ edit_file () { I+=("${SPIN[5]}") N+=("${SPIN[6]}") G+=("${SPIN[7]}") - done < ${RUNNER_TEMP//\\//}/spin.txt + done < ${RUNNER_TEMP}/spin.txt - IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP//\\//}/Sidebar.md; unset IFS; + IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/Sidebar.md; unset IFS; for ((i=0; i < ${#LINE[@]}; i++)); do TEXT="${LINE[$i]}"; IFS='|'; array=($TEXT); unset IFS; diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh index 1dee0d06e8d9..adc391f9728d 100755 --- a/.github/entrypoint/github_pages.sh +++ b/.github/entrypoint/github_pages.sh @@ -4,10 +4,10 @@ hr='------------------------------------------------------------------------------------' if [[ "${WIKI}" != "${BASE}" ]]; then - rm -rf ${RUNNER_TEMP//\\//}/wikidir + rm -rf ${RUNNER_TEMP}/wikidir - git clone $WIKI ${RUNNER_TEMP//\\//}/wikidir - cd ${RUNNER_TEMP//\\//}/wikidir && mv -f Home.md README.md + git clone $WIKI ${RUNNER_TEMP}/wikidir + cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; @@ -17,4 +17,4 @@ fi #echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cd ${GITHUB_WORKSPACE//\\//} && pwd && ls -alR . +cd ${GITHUB_WORKSPACE} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 3437b845ba47..a73040e4d9d0 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -82,7 +82,7 @@ elif [[ "${JOB_ID}" == "3" ]]; then rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && ls -lR . + cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . else diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh index ef6077e4eb96..2237ab0f5f3e 100755 --- a/.github/entrypoint/wiki.sh +++ b/.github/entrypoint/wiki.sh @@ -23,18 +23,18 @@ gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 find $1 -type f -name "README.md" -exec rm -rf {} \; -rm -rf ${RUNNER_TEMP//\\//}/README.md +rm -rf ${RUNNER_TEMP}/README.md -mv -f ${RUNNER_TEMP//\\//}/wikidir/README.md ${RUNNER_TEMP//\\//}/README.md -cp -R ${RUNNER_TEMP//\\//}/wikidir/* $1 +mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md +cp -R ${RUNNER_TEMP}/wikidir/* $1 -rm -rf ${RUNNER_TEMP//\\//}/spin.txt && touch ${RUNNER_TEMP//\\//}/spin.txt +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done -rm -rf ${RUNNER_TEMP//\\//}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP//\\//}/Sidebar.md -sed -i 's/0. \[\[//g' ${RUNNER_TEMP//\\//}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP//\\//}/Sidebar.md +rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md +sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' echo -e "\n$hr\nSPIN\n$hr" -cat ${RUNNER_TEMP//\\//}/spin.txt +cat ${RUNNER_TEMP}/spin.txt From 7702fc9eb880df8b0d0e1f25c6c49b576c96e7e8 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 5 Mar 2025 04:21:25 +0000 Subject: [PATCH 0018/1030] update workflows --- .../actions/4-identition/6-grammar/action.yml | 2 +- .github/entrypoint/front.sh | 2 +- .github/entrypoint/gist.sh | 25 ++++++++++++++++--- .github/entrypoint/github_pages.sh | 20 --------------- .github/entrypoint/init.sh | 16 ++++++------ .github/entrypoint/wiki.sh | 1 + 6 files changed, 31 insertions(+), 35 deletions(-) delete mode 100755 .github/entrypoint/github_pages.sh diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index e6de41483bdd..e6f1d7d6a505 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -90,7 +90,7 @@ runs: GH_TOKEN: ${{ inputs.token }} BASE: "D:/a/_actions/eq19/eq19/v2/.github" run: | - gh variable set RERUN_RUNNER --body "false" + #gh variable set RERUN_RUNNER --body "false" if [[ '${{ inputs.failure_status }}' == 'true' ]]; then git config --global user.name "$GITHUB_ACTOR" diff --git a/.github/entrypoint/front.sh b/.github/entrypoint/front.sh index 9e7967ac2855..08bafb2fd423 100755 --- a/.github/entrypoint/front.sh +++ b/.github/entrypoint/front.sh @@ -14,7 +14,7 @@ edit_file () { G+=("${SPIN[7]}") done < ${RUNNER_TEMP}/spin.txt - IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/Sidebar.md; unset IFS; + IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/wikidir/_Sidebar.md; unset IFS; for ((i=0; i < ${#LINE[@]}; i++)); do TEXT="${LINE[$i]}"; IFS='|'; array=($TEXT); unset IFS; diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh index d2ed098c4768..65896b9c56ce 100755 --- a/.github/entrypoint/gist.sh +++ b/.github/entrypoint/gist.sh @@ -1,7 +1,17 @@ #!/usr/bin/env bash +# Clean up rm -rf ${RUNNER_TEMP}/*dir +# Clone wiki +git clone $1 ${RUNNER_TEMP}/wikidir +mv -f ${RUNNER_TEMP}/wikidir/Home.md ${RUNNER_TEMP}/wikidir/README.md + +# Take titles from _Sidebar.md of wiki +sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md +sed -i 's/\]\]//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md + +# Clone gist to be put later on wiki's exponentiation and identition folders gh gist clone 0ce5848f7ad62dc46dedfaa430069857 ${RUNNER_TEMP}/gistdir/identition/span1 gh gist clone b32915925d9d365e2e9351f0c4ed786e ${RUNNER_TEMP}/gistdir/identition/span2 gh gist clone 88d09204b2e5986237bd66d062406fde ${RUNNER_TEMP}/gistdir/identition/span3 @@ -22,9 +32,16 @@ gh gist clone dc30497160f3389546d177da901537d9 ${RUNNER_TEMP}/gistdir/exponentia gh gist clone e84a0961dc7636c01d5953d19d65e30a ${RUNNER_TEMP}/gistdir/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef ${RUNNER_TEMP}/gistdir/exponentiation/span18 -rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +# Clean up the gists collection +find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; find ${RUNNER_TEMP}/gistdir -type f -name "README.md" -exec rm -rf {} \; -git clone ${BASE} ${RUNNER_TEMP}/workdir -mv -f ${RUNNER_TEMP}/workdir/Home.md ${RUNNER_TEMP}/workdir/README.md -find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done +# Collect and sort all spins on gistdir +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | \ + while ((i++)); IFS= read -r f; do sort.sh $f $i; done +cat ${RUNNER_TEMP}/spin.txt + +# Copy the gist to wiki +cp -R ${RUNNER_TEMP}/gistdir/* ${RUNNER_TEMP}/wikidir/ +find ${RUNNER_TEMP}/wikidir -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh deleted file mode 100755 index adc391f9728d..000000000000 --- a/.github/entrypoint/github_pages.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash -# Structure: Cell Types – Modulo 6 - -hr='------------------------------------------------------------------------------------' - -if [[ "${WIKI}" != "${BASE}" ]]; then - rm -rf ${RUNNER_TEMP}/wikidir - - git clone $WIKI ${RUNNER_TEMP}/wikidir - cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md - - find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; - -fi - -#echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 -find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cd ${GITHUB_WORKSPACE} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index a73040e4d9d0..b4cd86845184 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -74,15 +74,14 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cat /home/runner/_site/_config.yml - gist.sh ${TARGET_REPOSITORY} ${FOLDER} + cd /home/runner/_site && cat _config.yml + gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + cp -R ${RUNNER_TEMP}/wikidir/* . - find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; - mv -f ${RUNNER_TEMP}/workdir/* /home/runner/_site/ - - rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md - sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . + if [[ "${WIKI}" != "${BASE}" ]]; then + gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cp -R ${RUNNER_TEMP}/wikidir/* "$1/"' sh {} \; + fi else @@ -90,7 +89,6 @@ else git clone --single-branch --branch gh-source $TARGET_REPO gh-source cd ${GITHUB_WORKSPACE//\\//} - #find -not -path "./.git/*" -not -name ".git" | grep git find -not -path "./.git/*" -not -name ".git" -delete rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh index 2237ab0f5f3e..9edd307616fc 100755 --- a/.github/entrypoint/wiki.sh +++ b/.github/entrypoint/wiki.sh @@ -35,6 +35,7 @@ rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md echo -e "\n$hr\nSPIN\n$hr" cat ${RUNNER_TEMP}/spin.txt From ad06fa48c7c9a8c64f9e9ed3d24dc7d10274726e Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 5 Mar 2025 09:23:39 +0000 Subject: [PATCH 0019/1030] update workflows --- .github/entrypoint/init.sh | 6 +++--- .github/entrypoint/wiki.sh | 41 -------------------------------------- 2 files changed, 3 insertions(+), 44 deletions(-) delete mode 100755 .github/entrypoint/wiki.sh diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index b4cd86845184..04e66e4fcd5c 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -75,12 +75,12 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then cd /home/runner/_site && cat _config.yml - gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt - cp -R ${RUNNER_TEMP}/wikidir/* . + gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt && cp -R ${RUNNER_TEMP}/wikidir/* . if [[ "${WIKI}" != "${BASE}" ]]; then - gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + gist.sh ${WIKI} > /dev/nul && echo -e "\n$hr\n${FOLDER}\n$hr" && cat ${RUNNER_TEMP}/spin.txt find . -type d -name "${FOLDER}" -prune -exec sh -c 'cp -R ${RUNNER_TEMP}/wikidir/* "$1/"' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md' sh {} \; fi else diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh deleted file mode 100755 index 9edd307616fc..000000000000 --- a/.github/entrypoint/wiki.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -hr='------------------------------------------------------------------------------------' - -gh gist clone 0ce5848f7ad62dc46dedfaa430069857 $1/identition/span1 -gh gist clone b32915925d9d365e2e9351f0c4ed786e $1/identition/span2 -gh gist clone 88d09204b2e5986237bd66d062406fde $1/identition/span3 -gh gist clone 8cab5e72d52ecb338a2f2187082a1699 $1/identition/span4 -gh gist clone 54600a56d20163c2da8910dd804ec406 $1/identition/span5 -gh gist clone f1af4317b619154719546e615aaa2155 $1/identition/span6 -gh gist clone 6c89c3b0f109e0ead561a452720d1ebf $1/identition/span7 -gh gist clone f21abd90f8d471390aad23d6ecc90d6d $1/identition/span8 -gh gist clone 6e2fcc2138be6fb68839a3ede32f0525 $1/identition/span9 -gh gist clone b541275ab7deda356feef32d600e44d8 $1/identition/span10 -gh gist clone 80c8098f16f3e6ca06893b17a02d910e $1/identition/span11 -gh gist clone 4ffc4d02579d5cfd336a553c6da2f267 $1/identition/span12 - -gh gist clone f78d4470250720fb18111165564d555f $1/exponentiation/span13 -gh gist clone 765ddc69e339079a5a64b56c1d46e00f $1/exponentiation/span14 -gh gist clone b9f901cda16e8a11dd24ee6b677ca288 $1/exponentiation/span15 -gh gist clone dc30497160f3389546d177da901537d9 $1/exponentiation/span16 -gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 -gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 - -find $1 -type f -name "README.md" -exec rm -rf {} \; -rm -rf ${RUNNER_TEMP}/README.md - -mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md -cp -R ${RUNNER_TEMP}/wikidir/* $1 - -rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt -find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done - -rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md -sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - -find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md - -echo -e "\n$hr\nSPIN\n$hr" -cat ${RUNNER_TEMP}/spin.txt From 68c4b15b77b9390d2b84ae74008073150539d863 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 5 Mar 2025 10:56:51 +0000 Subject: [PATCH 0020/1030] update workflows --- .../actions/4-identition/6-grammar/action.yml | 2 +- .github/entrypoint/front.sh | 2 +- .github/entrypoint/gist.sh | 25 ++---------- .github/entrypoint/github_pages.sh | 20 ++++++++++ .github/entrypoint/init.sh | 16 ++++---- .github/entrypoint/wiki.sh | 40 +++++++++++++++++++ 6 files changed, 75 insertions(+), 30 deletions(-) create mode 100755 .github/entrypoint/github_pages.sh create mode 100755 .github/entrypoint/wiki.sh diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index e6f1d7d6a505..e6de41483bdd 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -90,7 +90,7 @@ runs: GH_TOKEN: ${{ inputs.token }} BASE: "D:/a/_actions/eq19/eq19/v2/.github" run: | - #gh variable set RERUN_RUNNER --body "false" + gh variable set RERUN_RUNNER --body "false" if [[ '${{ inputs.failure_status }}' == 'true' ]]; then git config --global user.name "$GITHUB_ACTOR" diff --git a/.github/entrypoint/front.sh b/.github/entrypoint/front.sh index 08bafb2fd423..9e7967ac2855 100755 --- a/.github/entrypoint/front.sh +++ b/.github/entrypoint/front.sh @@ -14,7 +14,7 @@ edit_file () { G+=("${SPIN[7]}") done < ${RUNNER_TEMP}/spin.txt - IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/wikidir/_Sidebar.md; unset IFS; + IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/Sidebar.md; unset IFS; for ((i=0; i < ${#LINE[@]}; i++)); do TEXT="${LINE[$i]}"; IFS='|'; array=($TEXT); unset IFS; diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh index 65896b9c56ce..d2ed098c4768 100755 --- a/.github/entrypoint/gist.sh +++ b/.github/entrypoint/gist.sh @@ -1,17 +1,7 @@ #!/usr/bin/env bash -# Clean up rm -rf ${RUNNER_TEMP}/*dir -# Clone wiki -git clone $1 ${RUNNER_TEMP}/wikidir -mv -f ${RUNNER_TEMP}/wikidir/Home.md ${RUNNER_TEMP}/wikidir/README.md - -# Take titles from _Sidebar.md of wiki -sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md -sed -i 's/\]\]//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md - -# Clone gist to be put later on wiki's exponentiation and identition folders gh gist clone 0ce5848f7ad62dc46dedfaa430069857 ${RUNNER_TEMP}/gistdir/identition/span1 gh gist clone b32915925d9d365e2e9351f0c4ed786e ${RUNNER_TEMP}/gistdir/identition/span2 gh gist clone 88d09204b2e5986237bd66d062406fde ${RUNNER_TEMP}/gistdir/identition/span3 @@ -32,16 +22,9 @@ gh gist clone dc30497160f3389546d177da901537d9 ${RUNNER_TEMP}/gistdir/exponentia gh gist clone e84a0961dc7636c01d5953d19d65e30a ${RUNNER_TEMP}/gistdir/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef ${RUNNER_TEMP}/gistdir/exponentiation/span18 -# Clean up the gists collection -find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; -find ${RUNNER_TEMP}/gistdir -type f -name "README.md" -exec rm -rf {} \; - -# Collect and sort all spins on gistdir rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt -find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | \ - while ((i++)); IFS= read -r f; do sort.sh $f $i; done -cat ${RUNNER_TEMP}/spin.txt +find ${RUNNER_TEMP}/gistdir -type f -name "README.md" -exec rm -rf {} \; -# Copy the gist to wiki -cp -R ${RUNNER_TEMP}/gistdir/* ${RUNNER_TEMP}/wikidir/ -find ${RUNNER_TEMP}/wikidir -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +git clone ${BASE} ${RUNNER_TEMP}/workdir +mv -f ${RUNNER_TEMP}/workdir/Home.md ${RUNNER_TEMP}/workdir/README.md +find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh new file mode 100755 index 000000000000..adc391f9728d --- /dev/null +++ b/.github/entrypoint/github_pages.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Structure: Cell Types – Modulo 6 + +hr='------------------------------------------------------------------------------------' + +if [[ "${WIKI}" != "${BASE}" ]]; then + rm -rf ${RUNNER_TEMP}/wikidir + + git clone $WIKI ${RUNNER_TEMP}/wikidir + cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md + + find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; + +fi + +#echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 +find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +cd ${GITHUB_WORKSPACE} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 04e66e4fcd5c..a73040e4d9d0 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -74,14 +74,15 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cd /home/runner/_site && cat _config.yml - gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt && cp -R ${RUNNER_TEMP}/wikidir/* . + cat /home/runner/_site/_config.yml + gist.sh ${TARGET_REPOSITORY} ${FOLDER} - if [[ "${WIKI}" != "${BASE}" ]]; then - gist.sh ${WIKI} > /dev/nul && echo -e "\n$hr\n${FOLDER}\n$hr" && cat ${RUNNER_TEMP}/spin.txt - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cp -R ${RUNNER_TEMP}/wikidir/* "$1/"' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md' sh {} \; - fi + find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; + mv -f ${RUNNER_TEMP}/workdir/* /home/runner/_site/ + + rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md + sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md + cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . else @@ -89,6 +90,7 @@ else git clone --single-branch --branch gh-source $TARGET_REPO gh-source cd ${GITHUB_WORKSPACE//\\//} + #find -not -path "./.git/*" -not -name ".git" | grep git find -not -path "./.git/*" -not -name ".git" -delete rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh new file mode 100755 index 000000000000..2237ab0f5f3e --- /dev/null +++ b/.github/entrypoint/wiki.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash + +hr='------------------------------------------------------------------------------------' + +gh gist clone 0ce5848f7ad62dc46dedfaa430069857 $1/identition/span1 +gh gist clone b32915925d9d365e2e9351f0c4ed786e $1/identition/span2 +gh gist clone 88d09204b2e5986237bd66d062406fde $1/identition/span3 +gh gist clone 8cab5e72d52ecb338a2f2187082a1699 $1/identition/span4 +gh gist clone 54600a56d20163c2da8910dd804ec406 $1/identition/span5 +gh gist clone f1af4317b619154719546e615aaa2155 $1/identition/span6 +gh gist clone 6c89c3b0f109e0ead561a452720d1ebf $1/identition/span7 +gh gist clone f21abd90f8d471390aad23d6ecc90d6d $1/identition/span8 +gh gist clone 6e2fcc2138be6fb68839a3ede32f0525 $1/identition/span9 +gh gist clone b541275ab7deda356feef32d600e44d8 $1/identition/span10 +gh gist clone 80c8098f16f3e6ca06893b17a02d910e $1/identition/span11 +gh gist clone 4ffc4d02579d5cfd336a553c6da2f267 $1/identition/span12 + +gh gist clone f78d4470250720fb18111165564d555f $1/exponentiation/span13 +gh gist clone 765ddc69e339079a5a64b56c1d46e00f $1/exponentiation/span14 +gh gist clone b9f901cda16e8a11dd24ee6b677ca288 $1/exponentiation/span15 +gh gist clone dc30497160f3389546d177da901537d9 $1/exponentiation/span16 +gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 +gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 + +find $1 -type f -name "README.md" -exec rm -rf {} \; +rm -rf ${RUNNER_TEMP}/README.md + +mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md +cp -R ${RUNNER_TEMP}/wikidir/* $1 + +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done + +rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md +sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md + +find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' + +echo -e "\n$hr\nSPIN\n$hr" +cat ${RUNNER_TEMP}/spin.txt From d1d8c2756bb6e83f69eab8920e61da400c245c49 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 5 Mar 2025 12:13:39 +0000 Subject: [PATCH 0021/1030] update workflows --- .../actions/4-identition/6-grammar/action.yml | 2 +- .github/entrypoint/front.sh | 2 +- .github/entrypoint/gist.sh | 25 ++++++++++++++++--- .github/entrypoint/github_pages.sh | 20 --------------- .github/entrypoint/init.sh | 16 ++++++------ .github/entrypoint/wiki.sh | 1 + 6 files changed, 31 insertions(+), 35 deletions(-) delete mode 100755 .github/entrypoint/github_pages.sh diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index e6de41483bdd..e6f1d7d6a505 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -90,7 +90,7 @@ runs: GH_TOKEN: ${{ inputs.token }} BASE: "D:/a/_actions/eq19/eq19/v2/.github" run: | - gh variable set RERUN_RUNNER --body "false" + #gh variable set RERUN_RUNNER --body "false" if [[ '${{ inputs.failure_status }}' == 'true' ]]; then git config --global user.name "$GITHUB_ACTOR" diff --git a/.github/entrypoint/front.sh b/.github/entrypoint/front.sh index 9e7967ac2855..08bafb2fd423 100755 --- a/.github/entrypoint/front.sh +++ b/.github/entrypoint/front.sh @@ -14,7 +14,7 @@ edit_file () { G+=("${SPIN[7]}") done < ${RUNNER_TEMP}/spin.txt - IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/Sidebar.md; unset IFS; + IFS=$'\n' read -d '' -r -a LINE < ${RUNNER_TEMP}/wikidir/_Sidebar.md; unset IFS; for ((i=0; i < ${#LINE[@]}; i++)); do TEXT="${LINE[$i]}"; IFS='|'; array=($TEXT); unset IFS; diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh index d2ed098c4768..65896b9c56ce 100755 --- a/.github/entrypoint/gist.sh +++ b/.github/entrypoint/gist.sh @@ -1,7 +1,17 @@ #!/usr/bin/env bash +# Clean up rm -rf ${RUNNER_TEMP}/*dir +# Clone wiki +git clone $1 ${RUNNER_TEMP}/wikidir +mv -f ${RUNNER_TEMP}/wikidir/Home.md ${RUNNER_TEMP}/wikidir/README.md + +# Take titles from _Sidebar.md of wiki +sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md +sed -i 's/\]\]//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md + +# Clone gist to be put later on wiki's exponentiation and identition folders gh gist clone 0ce5848f7ad62dc46dedfaa430069857 ${RUNNER_TEMP}/gistdir/identition/span1 gh gist clone b32915925d9d365e2e9351f0c4ed786e ${RUNNER_TEMP}/gistdir/identition/span2 gh gist clone 88d09204b2e5986237bd66d062406fde ${RUNNER_TEMP}/gistdir/identition/span3 @@ -22,9 +32,16 @@ gh gist clone dc30497160f3389546d177da901537d9 ${RUNNER_TEMP}/gistdir/exponentia gh gist clone e84a0961dc7636c01d5953d19d65e30a ${RUNNER_TEMP}/gistdir/exponentiation/span17 gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef ${RUNNER_TEMP}/gistdir/exponentiation/span18 -rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +# Clean up the gists collection +find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; find ${RUNNER_TEMP}/gistdir -type f -name "README.md" -exec rm -rf {} \; -git clone ${BASE} ${RUNNER_TEMP}/workdir -mv -f ${RUNNER_TEMP}/workdir/Home.md ${RUNNER_TEMP}/workdir/README.md -find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done +# Collect and sort all spins on gistdir +rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt +find ${RUNNER_TEMP}/gistdir -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | \ + while ((i++)); IFS= read -r f; do sort.sh $f $i; done +cat ${RUNNER_TEMP}/spin.txt + +# Copy the gist to wiki +cp -R ${RUNNER_TEMP}/gistdir/* ${RUNNER_TEMP}/wikidir/ +find ${RUNNER_TEMP}/wikidir -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' diff --git a/.github/entrypoint/github_pages.sh b/.github/entrypoint/github_pages.sh deleted file mode 100755 index adc391f9728d..000000000000 --- a/.github/entrypoint/github_pages.sh +++ /dev/null @@ -1,20 +0,0 @@ -#!/usr/bin/env bash -# Structure: Cell Types – Modulo 6 - -hr='------------------------------------------------------------------------------------' - -if [[ "${WIKI}" != "${BASE}" ]]; then - rm -rf ${RUNNER_TEMP}/wikidir - - git clone $WIKI ${RUNNER_TEMP}/wikidir - cd ${RUNNER_TEMP}/wikidir && mv -f Home.md README.md - - find . -type d -name "${FOLDER}" -prune -exec sh -c 'wiki.sh "$1"' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cat ${RUNNER_TEMP//\\//}/README.md >> $1/README.md' sh {} \; - find . -type d -name "${FOLDER}" -prune -exec sh -c 'ls -alR' sh {} \; - -fi - -#echo "action_state=yellow" | Out-File -FilePath $env:GITHUB_ENV -Append # no need for -Encoding utf8 -find . -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cd ${GITHUB_WORKSPACE} && pwd && ls -alR . diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index a73040e4d9d0..b4cd86845184 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -74,15 +74,14 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cat /home/runner/_site/_config.yml - gist.sh ${TARGET_REPOSITORY} ${FOLDER} + cd /home/runner/_site && cat _config.yml + gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + cp -R ${RUNNER_TEMP}/wikidir/* . - find ${RUNNER_TEMP}/gistdir -type d -name .git -prune -exec rm -rf {} \; - mv -f ${RUNNER_TEMP}/workdir/* /home/runner/_site/ - - rm -rf ${RUNNER_TEMP}/Sidebar.md && cp _Sidebar.md ${RUNNER_TEMP}/Sidebar.md - sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - cd /home/runner/_site && cp -R ${RUNNER_TEMP}/gistdir/* . && github_pages.sh && ls -lR . + if [[ "${WIKI}" != "${BASE}" ]]; then + gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + find . -type d -name "${FOLDER}" -prune -exec sh -c 'cp -R ${RUNNER_TEMP}/wikidir/* "$1/"' sh {} \; + fi else @@ -90,7 +89,6 @@ else git clone --single-branch --branch gh-source $TARGET_REPO gh-source cd ${GITHUB_WORKSPACE//\\//} - #find -not -path "./.git/*" -not -name ".git" | grep git find -not -path "./.git/*" -not -name ".git" -delete rm -rf ${RUNNER_TEMP//\\//}/gh-source/.git diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh index 2237ab0f5f3e..9edd307616fc 100755 --- a/.github/entrypoint/wiki.sh +++ b/.github/entrypoint/wiki.sh @@ -35,6 +35,7 @@ rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md echo -e "\n$hr\nSPIN\n$hr" cat ${RUNNER_TEMP}/spin.txt From 524350230033e4a12ba0b2997c14c4f4ef64e6bb Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Thu, 6 Mar 2025 05:12:10 +0700 Subject: [PATCH 0022/1030] Update main.yml Signed-off-by: eQuantum <8466209+eq19@users.noreply.github.com> --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 532bad526be0..b63ebc203012 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - #push: + push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] From c42e5bf45b65ca88c286e3084001ec21e4b2b368 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 5 Mar 2025 22:12:42 +0000 Subject: [PATCH 0023/1030] update workflows --- .github/entrypoint/init.sh | 5 ++--- .github/workflows/main.yml | 2 +- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index b4cd86845184..2a6e3f711aef 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -45,7 +45,7 @@ if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then fi -echo -e "\n$hr\nWORKSPACE\n$hr" +echo -e "\n$hr\nWORKSPACE: $FOLDER\n$hr" if [[ "${JOB_ID}" == "1" ]]; then #[[ "${LATEST_COMMIT}" == *"RERUN_RUNNER"* ]] && gh variable set RERUN_RUNNER --body "true" @@ -75,8 +75,7 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then cd /home/runner/_site && cat _config.yml - gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt - cp -R ${RUNNER_TEMP}/wikidir/* . + gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt && cp -R ${RUNNER_TEMP}/wikidir/* . if [[ "${WIKI}" != "${BASE}" ]]; then gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index b63ebc203012..532bad526be0 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - push: + #push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] From 73abb0abae3bf4449b87ee38f67e8c8060b720aa Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 6 Mar 2025 09:50:40 +0000 Subject: [PATCH 0024/1030] update workflows --- .github/entrypoint/init.sh | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 2a6e3f711aef..8d7f893b20cd 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -35,8 +35,8 @@ if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then echo 'TARGET_REPO='${TARGET_REPO} >> ${GITHUB_ENV} echo 'REMOTE_REPO='${REMOTE_REPO} >> ${GITHUB_ENV} - if [[ -f _config.yml ]]; then - FOLDER=$(yq '.span' _config.yml) + if [[ -f /home/runner/_site/_config.yml ]]; then + FOLDER=$(yq '.span' /home/runner/_site/_config.yml) export FOLDER=$(eval echo $FOLDER) elif [[ -f /home/runner/_site/.env ]]; then set -a && . /home/runner/_site/.env && set +a @@ -75,7 +75,8 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then cd /home/runner/_site && cat _config.yml - gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt && cp -R ${RUNNER_TEMP}/wikidir/* . + gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt + cp -R ${RUNNER_TEMP}/wikidir/* . && pwd && ls -al . if [[ "${WIKI}" != "${BASE}" ]]; then gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt From c7aa9557526bfdf7a930e7d2a4389ca6eaa5b49b Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Thu, 6 Mar 2025 21:12:55 +0700 Subject: [PATCH 0025/1030] Update main.yml Signed-off-by: eQuantum <8466209+eq19@users.noreply.github.com> --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 532bad526be0..b63ebc203012 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - #push: + push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] From 507fc9ac2a43c6c52226bec280e290f4e14babf4 Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 6 Mar 2025 14:13:34 +0000 Subject: [PATCH 0026/1030] update workflows --- .github/entrypoint/gist.sh | 3 +++ .github/entrypoint/init.sh | 8 +++----- .github/workflows/main.yml | 2 +- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh index 65896b9c56ce..8bf7234b4ea8 100755 --- a/.github/entrypoint/gist.sh +++ b/.github/entrypoint/gist.sh @@ -45,3 +45,6 @@ cat ${RUNNER_TEMP}/spin.txt # Copy the gist to wiki cp -R ${RUNNER_TEMP}/gistdir/* ${RUNNER_TEMP}/wikidir/ find ${RUNNER_TEMP}/wikidir -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' + +cp -R ${RUNNER_TEMP}/wikidir/* $2/ +ls -al $2 diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 8d7f893b20cd..36d9a84f5955 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -36,6 +36,7 @@ if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then echo 'REMOTE_REPO='${REMOTE_REPO} >> ${GITHUB_ENV} if [[ -f /home/runner/_site/_config.yml ]]; then + cat /home/runner/_site/_config.yml FOLDER=$(yq '.span' /home/runner/_site/_config.yml) export FOLDER=$(eval echo $FOLDER) elif [[ -f /home/runner/_site/.env ]]; then @@ -74,13 +75,10 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cd /home/runner/_site && cat _config.yml - gist.sh ${BASE} > /dev/null && cat ${RUNNER_TEMP}/spin.txt - cp -R ${RUNNER_TEMP}/wikidir/* . && pwd && ls -al . + cd /home/runner/_site && gist.sh ${BASE} $(pwd) if [[ "${WIKI}" != "${BASE}" ]]; then - gist.sh ${WIKI} > /dev/null && cat ${RUNNER_TEMP}/spin.txt - find . -type d -name "${FOLDER}" -prune -exec sh -c 'cp -R ${RUNNER_TEMP}/wikidir/* "$1/"' sh {} \; + find . -type d -name "${FOLDER}" -prune -exec sh -c 'gist.sh ${WIKI} "$1"' sh {} \; fi else diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index b63ebc203012..532bad526be0 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - push: + #push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] From 26244ee528c728eaa3894dce1c401a07f119d83d Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 7 Mar 2025 12:10:25 +0000 Subject: [PATCH 0027/1030] update workflows --- .github/actions/4-identition/5-syntax/action.yml | 2 +- .github/entrypoint/gist.sh | 8 ++++++-- .github/entrypoint/init.sh | 2 +- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index 7578a976a04e..e052c1169c48 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -115,4 +115,4 @@ runs: run: | curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ -H "Authorization: Bearer ${GCP_TOKEN}" -H "Content-Type: application/json" \ - --data @${{ inputs.workspace }}/docs/data.json | jq '.' + --data @${{ inputs.workspace }}/_data/orgs.json | jq '.' diff --git a/.github/entrypoint/gist.sh b/.github/entrypoint/gist.sh index 8bf7234b4ea8..d1a3119b46fe 100755 --- a/.github/entrypoint/gist.sh +++ b/.github/entrypoint/gist.sh @@ -1,11 +1,14 @@ #!/usr/bin/env bash +hr='------------------------------------------------------------------------------------' + # Clean up rm -rf ${RUNNER_TEMP}/*dir +rm -rf ${RUNNER_TEMP}/README.md # Clone wiki git clone $1 ${RUNNER_TEMP}/wikidir -mv -f ${RUNNER_TEMP}/wikidir/Home.md ${RUNNER_TEMP}/wikidir/README.md +mv -f ${RUNNER_TEMP}/wikidir/Home.md ${RUNNER_TEMP}/README.md # Take titles from _Sidebar.md of wiki sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/wikidir/_Sidebar.md @@ -46,5 +49,6 @@ cat ${RUNNER_TEMP}/spin.txt cp -R ${RUNNER_TEMP}/gistdir/* ${RUNNER_TEMP}/wikidir/ find ${RUNNER_TEMP}/wikidir -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' +echo -e "\n$hr\nWiki Dir: $2\n$hr" cp -R ${RUNNER_TEMP}/wikidir/* $2/ -ls -al $2 +cat ${RUNNER_TEMP}/README.md >> $2/README.md && ls -al $2 diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 36d9a84f5955..d12880691497 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -75,7 +75,7 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cd /home/runner/_site && gist.sh ${BASE} $(pwd) + cd /home/runner/_site && rm -rf docs && gist.sh ${BASE} $(pwd) if [[ "${WIKI}" != "${BASE}" ]]; then find . -type d -name "${FOLDER}" -prune -exec sh -c 'gist.sh ${WIKI} "$1"' sh {} \; From 544325b584d290e2b3b3dedc7e53696311d8ca09 Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Sun, 9 Mar 2025 06:44:27 +0700 Subject: [PATCH 0028/1030] Update main.yml Signed-off-by: eQuantum <8466209+eq19@users.noreply.github.com> --- .github/workflows/main.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 532bad526be0..bdebe35a5aa6 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -65,7 +65,10 @@ jobs: id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} chat_id: ${{ secrets.MESSAGE_API }} + api_key: ${{ secrets.ACCESS_API }} + api_secret: ${{ secrets.ACCESS_KEY }} rerun_runner: ${{ vars.RERUN_RUNNER }} + pypi_token: ${{ secrets.PYPI_TOKEN }} bot_token: ${{ secrets.MONITOR_TOKEN }} logs_token: ${{ secrets.WARNING_TOKEN }} credentials: ${{ secrets.GCP_CREDENTIALS }} From a1f2bbc1f7c2951a4b115143d88d711a628a3b01 Mon Sep 17 00:00:00 2001 From: eq19 Date: Mon, 10 Mar 2025 18:05:12 +0000 Subject: [PATCH 0029/1030] update workflows --- .github/actions/1-addition/2-feed/action.yml | 29 +++++- .github/entrypoint/artifact.sh | 9 +- .github/entrypoint/init.sh | 2 +- .github/entrypoint/startup.sh | 0 .github/entrypoint/sync.sh | 95 ++++++++++++++++++++ .github/entrypoint/wiki.sh | 41 --------- 6 files changed, 130 insertions(+), 46 deletions(-) mode change 100644 => 100755 .github/entrypoint/startup.sh create mode 100755 .github/entrypoint/sync.sh delete mode 100755 .github/entrypoint/wiki.sh diff --git a/.github/actions/1-addition/2-feed/action.yml b/.github/actions/1-addition/2-feed/action.yml index da588b6b0863..ef49be6872c5 100644 --- a/.github/actions/1-addition/2-feed/action.yml +++ b/.github/actions/1-addition/2-feed/action.yml @@ -67,6 +67,24 @@ inputs: description: 'The deploy repository' default: ${{ github.repository }} required: false + api_key: + description: The key of api application + required: true + api_secret: + description: The secret of api application + required: true + pypi_token: + description: The pypi token + required: true + bot_token: + description: The monitor token + required: true + logs_token: + description: The warning token + required: true + chat_id: + description: The telegram chat id + required: true outputs: artifact: @@ -95,11 +113,20 @@ runs: TMP: ${{ inputs.tmp }} RUN: ${{ github.run_id }} USER: ${{ github.actor }} - #TOKEN: ${{ inputs.token }} REPO: ${{ github.repository }} + ACCESS_API: ${{ inputs.api_key }} + ACCESS_KEY: ${{ inputs.api_secret }} + ACCESS_TOKEN: ${{ inputs.token }} + MESSAGE_API: ${{ inputs.chat_id }} + MONITOR_TOKEN: ${{ inputs.bot_token }} + WARNING_TOKEN: ${{ inputs.logs_token }} + PYPI_TOKEN: ${{ inputs.pypi_token }} ACTOR: ${{ github.triggering_actor }} OWNER: ${{ github.repository_owner }} BEARER: ${{ steps.auth.outputs.id_token }} + DOCKER_HUB_TOKEN: ${{ inputs.hub_token }} + DOCKER_HUB_PASSWORD: ${{ inputs.hub_password }} + GCP_CREDENTIALS: ${{ inputs.credentials }} run: | shopt -s dotglob rm -rf ${{ inputs.action_path }}/.git diff --git a/.github/entrypoint/artifact.sh b/.github/entrypoint/artifact.sh index eee4bc3aa438..aed2e74643d4 100755 --- a/.github/entrypoint/artifact.sh +++ b/.github/entrypoint/artifact.sh @@ -76,8 +76,6 @@ set_target() { } jekyll_build() { - - echo -e "\n$hr\nCONFIG\n$hr" [[ $1 == *"github.io"* ]] && OWNER=$2 if [[ $1 != "eq19.github.io" ]]; then SITEID=$(( $3 + 2 )); else SITEID=1; fi @@ -109,8 +107,13 @@ jekyll_build() { sed -i "1s|^|id: ${SITEID}\n|" ${RUNNER_TEMP}/_config.yml echo 'ID='${SITEID} >> ${GITHUB_ENV} + + echo -e "\n$hr\nCONFIG\n$hr" cat ${RUNNER_TEMP}/_config.yml - + + echo -e "\n$hr\nSET TOKEN\n$hr" + sync.sh ${REPO} ${TARGET_REPOSITORY} ${GH_TOKEN} + } # Get structure on gist files diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index d12880691497..5fb021253acb 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -75,7 +75,7 @@ elif [[ "${JOB_ID}" == "2" ]]; then elif [[ "${JOB_ID}" == "3" ]]; then - cd /home/runner/_site && rm -rf docs && gist.sh ${BASE} $(pwd) + cd /home/runner/_site && rm -rf README.md docs && gist.sh ${BASE} $(pwd) if [[ "${WIKI}" != "${BASE}" ]]; then find . -type d -name "${FOLDER}" -prune -exec sh -c 'gist.sh ${WIKI} "$1"' sh {} \; diff --git a/.github/entrypoint/startup.sh b/.github/entrypoint/startup.sh old mode 100644 new mode 100755 diff --git a/.github/entrypoint/sync.sh b/.github/entrypoint/sync.sh new file mode 100755 index 000000000000..b1646ced17f2 --- /dev/null +++ b/.github/entrypoint/sync.sh @@ -0,0 +1,95 @@ +#!/bin/bash + +# GitHub Personal Access Token (PAT) - Store securely +GITHUB_PAT="$3" + +# GitHub API Base URL +GITHUB_API="https://api.github.com" + +# Function to check if a secret exists in a repository +check_secret_exists() { + local repo=$1 + local secret_name=$2 + curl -s -o /dev/null -w "%{http_code}" -H "Authorization: token $GITHUB_PAT" \ + -H "Accept: application/vnd.github.v3+json" "$GITHUB_API/repos/$repo/actions/secrets/$secret_name" +} + +# Function to set a secret in a repository +set_secret() { + local repo=$1 + local secret_name=$2 + local secret_value=$3 + + # Set the secret + gh secret set $secret_name --repo $repo --body "$secret_value" +} + +# Function to check if a variable exists in a repository +check_variable_exists() { + local repo=$1 + local var_name=$2 + response=$(curl -s -H "Authorization: token $GITHUB_PAT" \ + -H "Accept: application/vnd.github.v3+json" "$GITHUB_API/repos/$repo/actions/variables") + + echo "$response" | jq -e ".variables[] | select(.name == \"$var_name\")" > /dev/null 2>&1 + return $? +} + +# Function to set a variable in a repository +set_variable() { + local repo=$1 + local var_name=$2 + local var_value=$3 + + # Set the variable + gh variable set $var_name --repo $repo --body "$var_value" +} + +# Main function to sync secrets and variables from one repo to another +sync_secrets_and_variables() { + local source_repo=$1 + local target_repo=$2 + + echo "🔄 Syncing secrets from $source_repo to $target_repo..." + + # Get all secrets from source repository + secrets=$(curl -s -H "Authorization: token $GITHUB_PAT" \ + -H "Accept: application/vnd.github.v3+json" "$GITHUB_API/repos/$source_repo/actions/secrets" | jq -r '.secrets[].name') + + for secret in $secrets; do + if [[ "$(check_secret_exists "$target_repo" "$secret")" == "404" ]]; then + echo "➕ Secret '$secret' does not exist in $target_repo. Copying..." + set_secret "$target_repo" "$secret" "${!secret}" + else + echo "✅ Secret '$secret' already exists in $target_repo." + fi + done + + echo "🔄 Syncing variables from $source_repo to $target_repo..." + + # Get all variables from source repository + variables=$(curl -s -H "Authorization: token $GITHUB_PAT" \ + -H "Accept: application/vnd.github.v3+json" "$GITHUB_API/repos/$source_repo/actions/variables" | jq -r '.variables[].name') + + for var in $variables; do + if ! check_variable_exists "$target_repo" "$var"; then + echo "➕ Variable '$var' does not exist in $target_repo. Copying..." + var_value=$(curl -s -H "Authorization: token $GITHUB_PAT" \ + -H "Accept: application/vnd.github.v3+json" "$GITHUB_API/repos/$source_repo/actions/variables/$var" | jq -r '.value') + set_variable "$target_repo" "$var" "$var_value" + else + echo "✅ Variable '$var' already exists in $target_repo." + fi + done + + echo "✅ Sync complete!" +} + +# Check for required arguments +if [[ $# -ne 3 ]]; then + echo "Usage: $0 " + exit 1 +fi + +# Run the sync function +sync_secrets_and_variables "$1" "$2" diff --git a/.github/entrypoint/wiki.sh b/.github/entrypoint/wiki.sh deleted file mode 100755 index 9edd307616fc..000000000000 --- a/.github/entrypoint/wiki.sh +++ /dev/null @@ -1,41 +0,0 @@ -#!/usr/bin/env bash - -hr='------------------------------------------------------------------------------------' - -gh gist clone 0ce5848f7ad62dc46dedfaa430069857 $1/identition/span1 -gh gist clone b32915925d9d365e2e9351f0c4ed786e $1/identition/span2 -gh gist clone 88d09204b2e5986237bd66d062406fde $1/identition/span3 -gh gist clone 8cab5e72d52ecb338a2f2187082a1699 $1/identition/span4 -gh gist clone 54600a56d20163c2da8910dd804ec406 $1/identition/span5 -gh gist clone f1af4317b619154719546e615aaa2155 $1/identition/span6 -gh gist clone 6c89c3b0f109e0ead561a452720d1ebf $1/identition/span7 -gh gist clone f21abd90f8d471390aad23d6ecc90d6d $1/identition/span8 -gh gist clone 6e2fcc2138be6fb68839a3ede32f0525 $1/identition/span9 -gh gist clone b541275ab7deda356feef32d600e44d8 $1/identition/span10 -gh gist clone 80c8098f16f3e6ca06893b17a02d910e $1/identition/span11 -gh gist clone 4ffc4d02579d5cfd336a553c6da2f267 $1/identition/span12 - -gh gist clone f78d4470250720fb18111165564d555f $1/exponentiation/span13 -gh gist clone 765ddc69e339079a5a64b56c1d46e00f $1/exponentiation/span14 -gh gist clone b9f901cda16e8a11dd24ee6b677ca288 $1/exponentiation/span15 -gh gist clone dc30497160f3389546d177da901537d9 $1/exponentiation/span16 -gh gist clone e84a0961dc7636c01d5953d19d65e30a $1/exponentiation/span17 -gh gist clone e9832026b5b78f694e4ad22c3eb6c3ef $1/exponentiation/span18 - -find $1 -type f -name "README.md" -exec rm -rf {} \; -rm -rf ${RUNNER_TEMP}/README.md - -mv -f ${RUNNER_TEMP}/wikidir/README.md ${RUNNER_TEMP}/README.md -cp -R ${RUNNER_TEMP}/wikidir/* $1 - -rm -rf ${RUNNER_TEMP}/spin.txt && touch ${RUNNER_TEMP}/spin.txt -find $1 -type f -name 'spin_*.txt' | sort -n -t _ -k 2 | while ((i++)); IFS= read -r f; do sort.sh $f $i; done - -rm -rf ${RUNNER_TEMP}/Sidebar.md && cp $1/_Sidebar.md ${RUNNER_TEMP}/Sidebar.md -sed -i 's/0. \[\[//g' ${RUNNER_TEMP}/Sidebar.md && sed -i 's/\]\]//g' ${RUNNER_TEMP}/Sidebar.md - -find $1 -iname '*.md' -print0 | sort -zn | xargs -0 -I '{}' front.sh '{}' -cat ${RUNNER_TEMP}/wikidir/README.md >> $1/README.md - -echo -e "\n$hr\nSPIN\n$hr" -cat ${RUNNER_TEMP}/spin.txt From 80aa70bf8f3ffa49e4c4541045474b7c6f879e8b Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 11 Mar 2025 02:57:14 +0000 Subject: [PATCH 0030/1030] update workflows --- .github/entrypoint/init.sh | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 5fb021253acb..8fa58d5eac1a 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -62,9 +62,19 @@ if [[ "${JOB_ID}" == "1" ]]; then git add . && git commit --allow-empty -m "rerun due to job update" && git push exit 1 else - PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json - mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json - + #PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json + #mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json + curl -s -H "Authorization: token $GH_TOKEN" \ + -H "Accept: application/vnd.github.v3+json" \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" \ + | jq -r '.value' > $1/dataFile/user_data/strategies/fibbo.json + + if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then + echo "Valid JSON" && cat $1/dataFile/user_data/strategies/fibbo.json + else + echo "Invalid JSON" + fi + mv -f $1/dataFile/user_data ${GITHUB_WORKSPACE}/ cd ${GITHUB_WORKSPACE} && ls -al ${GITHUB_WORKSPACE} fi From 19821dffd8a4ec1a8fb1237a41c415e02f334787 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 11 Mar 2025 11:56:03 +0000 Subject: [PATCH 0031/1030] update workflows --- .github/entrypoint/init.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 8fa58d5eac1a..834362be9d8a 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -46,7 +46,10 @@ if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then fi -echo -e "\n$hr\nWORKSPACE: $FOLDER\n$hr" +echo -e "\n$hr\nWORKSPACE\n$hr" +PARAMS_JSON=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" | jq -r '.value') + if [[ "${JOB_ID}" == "1" ]]; then #[[ "${LATEST_COMMIT}" == *"RERUN_RUNNER"* ]] && gh variable set RERUN_RUNNER --body "true" @@ -64,10 +67,7 @@ if [[ "${JOB_ID}" == "1" ]]; then else #PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json #mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json - curl -s -H "Authorization: token $GH_TOKEN" \ - -H "Accept: application/vnd.github.v3+json" \ - "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" \ - | jq -r '.value' > $1/dataFile/user_data/strategies/fibbo.json + echo "${PARAMS_JSON}" | jq '.' > output.json> $1/dataFile/user_data/strategies/fibbo.json if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then echo "Valid JSON" && cat $1/dataFile/user_data/strategies/fibbo.json From 37fa3bc80e89546e9e8fb5a097614a3e00b164c0 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 11 Mar 2025 12:02:14 +0000 Subject: [PATCH 0032/1030] update workflows --- .github/entrypoint/init.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 834362be9d8a..1f1ce4a2420e 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -67,7 +67,7 @@ if [[ "${JOB_ID}" == "1" ]]; then else #PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json #mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json - echo "${PARAMS_JSON}" | jq '.' > output.json> $1/dataFile/user_data/strategies/fibbo.json + echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then echo "Valid JSON" && cat $1/dataFile/user_data/strategies/fibbo.json From 59c2007a015263589aa196d25125ff1274598aab Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 14 Mar 2025 02:36:16 +0000 Subject: [PATCH 0033/1030] update workflows --- .github/entrypoint/init.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 1f1ce4a2420e..56a74c8fb65b 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -70,7 +70,7 @@ if [[ "${JOB_ID}" == "1" ]]; then echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then - echo "Valid JSON" && cat $1/dataFile/user_data/strategies/fibbo.json + cat $1/dataFile/user_data/strategies/fibbo.json else echo "Invalid JSON" fi From 3e43e5abcaaafefc09577e5b40b622a02882811a Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 14 Mar 2025 04:27:43 +0000 Subject: [PATCH 0034/1030] update workflows --- .github/actions/4-identition/6-grammar/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index e6f1d7d6a505..5473ab5a7b82 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -109,7 +109,7 @@ runs: cat _config.yml echo -e "\n$hr\nSource Pages\n$hr" - ls -al . + rm -rf user_data && ls -al . FILE=$(find . -name "plugin_manager.rb") sed -i 's|unless site.safe|if site.safe|g' $FILE From bfeb309c2052c938f0cf9b8d64710dfeed13b377 Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 14 Mar 2025 04:39:30 +0000 Subject: [PATCH 0035/1030] update workflows --- .github/actions/4-identition/6-grammar/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index 5473ab5a7b82..0e4cb59e50e4 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -109,7 +109,7 @@ runs: cat _config.yml echo -e "\n$hr\nSource Pages\n$hr" - rm -rf user_data && ls -al . + rm -rf _user && ls -al . FILE=$(find . -name "plugin_manager.rb") sed -i 's|unless site.safe|if site.safe|g' $FILE From d5c5de2f9f4431faa9315bc9b2911c9c5dbde9b8 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sat, 22 Mar 2025 02:18:07 +0000 Subject: [PATCH 0036/1030] update workflows --- .github/entrypoint/init.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 56a74c8fb65b..916daae0b1a0 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -52,7 +52,7 @@ PARAMS_JSON=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: applicatio if [[ "${JOB_ID}" == "1" ]]; then - #[[ "${LATEST_COMMIT}" == *"RERUN_RUNNER"* ]] && gh variable set RERUN_RUNNER --body "true" + gh variable set RERUN_RUNNER --body "false" cd ${GITHUB_WORKSPACE} && rm -rf .github cp -r /home/runner/work/_actions/eq19/eq19/v2/.github . @@ -67,8 +67,8 @@ if [[ "${JOB_ID}" == "1" ]]; then else #PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json #mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json - echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json + echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then cat $1/dataFile/user_data/strategies/fibbo.json else From a770a61d6e8af12641e4c25c7a013f51a213735d Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 23 Mar 2025 10:18:44 +0000 Subject: [PATCH 0037/1030] update workflows --- .github/actions/1-addition/3-lexer/action.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/actions/1-addition/3-lexer/action.yml b/.github/actions/1-addition/3-lexer/action.yml index 57357e65c2f9..6681327aeedb 100644 --- a/.github/actions/1-addition/3-lexer/action.yml +++ b/.github/actions/1-addition/3-lexer/action.yml @@ -122,6 +122,7 @@ runs: id: ${{ inputs.id }} TARGET_REPOSITORY: ${{ env.TARGET_REPOSITORY }} run: | + gh variable set RERUN_RUNNER --body "true" BASE_NAME=$(basename ${{ inputs.image_name }}) echo 'output_${id}='${TARGET_REPOSITORY} >> ${GITHUB_OUTPUT} IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) From 1e05a66f5c8e03e2275f4ec204d344ba0dae5fac Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 23 Mar 2025 16:07:51 +0000 Subject: [PATCH 0038/1030] update workflows --- .../2-multiplication/1-maps/action.yml | 2 +- .github/entrypoint/init.sh | 21 +++++++++++-------- 2 files changed, 13 insertions(+), 10 deletions(-) diff --git a/.github/actions/2-multiplication/1-maps/action.yml b/.github/actions/2-multiplication/1-maps/action.yml index 4d2cc0a3bdad..fd288d0af1aa 100644 --- a/.github/actions/2-multiplication/1-maps/action.yml +++ b/.github/actions/2-multiplication/1-maps/action.yml @@ -125,7 +125,7 @@ runs: # Ref: https://cloud.google.com/compute/docs/instances/create-start-instance - name: 💎 Remove Existing Self-Hosted Runner - if: inputs.rerun_runner == 'true' + #if: inputs.rerun_runner == 'true' shell: bash run: | for instance_uri in $(gcloud compute instances list --uri 2> /dev/null); do diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 916daae0b1a0..2c87d49012b0 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -47,36 +47,39 @@ if [[ -z ${PASS} ]] || [[ "${PASS}" == "true" ]]; then fi echo -e "\n$hr\nWORKSPACE\n$hr" -PARAMS_JSON=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ - "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" | jq -r '.value') +RERUN_RUNNER=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/RERUN_RUNNER" | jq -r '.value') if [[ "${JOB_ID}" == "1" ]]; then - gh variable set RERUN_RUNNER --body "false" - cd ${GITHUB_WORKSPACE} && rm -rf .github cp -r /home/runner/work/_actions/eq19/eq19/v2/.github . chown -R "$(whoami)" .github git remote set-url origin ${REMOTE_REPO} git add . && git commit -m "update workflows" --quiet && git push --quiet - if [ $? -eq 0 ]; then + + if [[ $? -eq 0 ]]; then + git clone --single-branch --branch gh-pages $REMOTE_REPO gh-pages && cd gh-pages git add . && git commit --allow-empty -m "rerun due to job update" && git push exit 1 + else - #PARAMS=.github/entrypoint/artifact/python/src/params/spaces.json - #mv -f ${GITHUB_WORKSPACE}/$PARAMS $1/dataFile/user_data/strategies/fibbo.json + PARAMS_JSON=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ + "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" | jq -r '.value') echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json + if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then cat $1/dataFile/user_data/strategies/fibbo.json else echo "Invalid JSON" fi - mv -f $1/dataFile/user_data ${GITHUB_WORKSPACE}/ - cd ${GITHUB_WORKSPACE} && ls -al ${GITHUB_WORKSPACE} + cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . + if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi + fi elif [[ "${JOB_ID}" == "2" ]]; then From 72f7165de0bd5a6861dcf246585e7028e0075782 Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Mon, 24 Mar 2025 18:46:51 +0700 Subject: [PATCH 0039/1030] Update main.yml Signed-off-by: eQuantum <8466209+eq19@users.noreply.github.com> --- .github/workflows/main.yml | 38 ++++++++++++++++---------------------- 1 file changed, 16 insertions(+), 22 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index bdebe35a5aa6..ff74af4ef997 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - #push: + push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] @@ -18,7 +18,6 @@ permissions: write-all # 🕸️ Global Environment env: GH_TOKEN: ${{ secrets.ACCESS_TOKEN }} - PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} TARGET_REPOSITORY: ${{ vars.TARGET_REPOSITORY }} jobs: @@ -28,15 +27,16 @@ jobs: fail-fast: true matrix: config: - - { os: ubuntu-latest, id: 1 } - - { os: macos-latest, id: 2 } + - { os: ubuntu-latest, jobs_id: 1 } + - { os: macos-latest, jobs_id: 2 } env: - ID: ${{ matrix.config.id }} + JOBS_ID: ${{ matrix.config.jobs_id }} + PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }} DOC: '${{ vars.TARGET_REPOSITORY }}.wiki' # 🔧 Commutation relations runs-on: ${{ matrix.config.os }} - continue-on-error: ${{ matrix.config.id != 1 && false || true }} + continue-on-error: ${{ matrix.config.jobs_id != 1 && false || true }} steps: - name: 📂 Checkout @@ -44,13 +44,12 @@ jobs: with: submodules: recursive token: ${{ secrets.ACCESS_TOKEN }} - repository: ${{ env.ID != 1 && env.DOC || '' }} + repository: ${{ env.JOBS_ID != 1 && env.DOC || '' }} - name: ⚙️ Build Maps uses: eq19/maps@v8 id: build-parser with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} rerun_runner: ${{ vars.RERUN_RUNNER }} credentials: ${{ secrets.GCP_CREDENTIALS }} @@ -62,7 +61,6 @@ jobs: uses: eq19/feed@v6 id: feed-mapping with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} chat_id: ${{ secrets.MESSAGE_API }} api_key: ${{ secrets.ACCESS_API }} @@ -80,7 +78,6 @@ jobs: uses: eq19/lexer@v4 id: set-lexering with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} config: ${{ secrets.CONFIG_JSON }} credentials: ${{ secrets.GCP_CREDENTIALS }} @@ -92,8 +89,7 @@ jobs: id: gen_outputs shell: bash run: | - id=${{ matrix.config.id }} - output=${{ matrix.config.id == 1 && + output=${{ env.JOBS_ID == 1 && steps.feed-mapping.outputs.addition || steps.set-lexering.outputs.multiplication }} echo "output_${id}=${output}" >> "${GITHUB_OUTPUT}" @@ -108,14 +104,15 @@ jobs: strategy: matrix: config: - - { os: self-hosted, id: 3 } - - { os: windows-latest, id: 4 } + - { os: self-hosted, jobs_id: 3 } + - { os: windows-latest, jobs_id: 4 } # 💢 Completeness relations max-parallel: 1 fail-fast: true env: + JOBS_ID: ${{ matrix.config.jobs_id }} IP_ADDRESS: '${{ needs.lexering.outputs.address }}' PASS: ${{ contains(needs.lexering.result, 'success') }} @@ -123,7 +120,7 @@ jobs: needs: lexering runs-on: ${{ matrix.config.os }} #if: ${{ (needs.lexering.result == 'success') }} - continue-on-error: ${{ matrix.config.id == 3 && true || false }} + continue-on-error: ${{ env.JOBS_ID == 3 && true || false }} steps: - name: 📂 Checkout @@ -132,42 +129,39 @@ jobs: submodules: recursive token: ${{ secrets.ACCESS_TOKEN }} repository: ${{ env.TARGET_REPOSITORY }} - ref: ${{ matrix.config.id == 3 && 'gh-source' || 'gh-pages' }} + ref: ${{ env.JOBS_ID == 3 && 'gh-source' || 'gh-pages' }} - name: 🚀 Setup Parser if: ${{ env.PASS == 'true' }} uses: eq19/parser@v2 id: set-parser with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} credentials: ${{ secrets.GCP_CREDENTIALS }} pauli_sum: ${{ needs.lexering.outputs.function }} failure_status: ${{ contains(steps.*.conclusion, 'failure') || - (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} + (env.JOBS_ID == 4 && contains(needs.*.result, 'failure')) }} - name: 🌀 Evaluate Syntax if: ${{ env.PASS == 'true' }} uses: eq19/syntax@v2 id: eval-syntax with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} api_key: ${{ secrets.ACCESS_API }} api_secret: ${{ secrets.ACCESS_KEY }} credentials: ${{ secrets.GCP_CREDENTIALS }} pauli_sum: ${{ toJSON(steps.set-parser.outputs) }} failure_status: ${{ contains(steps.*.conclusion, 'failure') || - (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} + (env.JOBS_ID == 4 && contains(needs.*.result, 'failure')) }} - name: ⚧️ Set Grammar Rules uses: eq19/grammar@v1 id: grammar-rules if: always() with: - id: ${{ matrix.config.id }} token: ${{ secrets.ACCESS_TOKEN }} credentials: ${{ secrets.GCP_CREDENTIALS }} pauli_sum: ${{ toJSON(steps.eval-syntax.outputs) }} failure_status: ${{ contains(steps.*.conclusion, 'failure') || - (matrix.config.id == 4 && contains(needs.*.result, 'failure')) }} + (env.JOBS_ID == 4 && contains(needs.*.result, 'failure')) }} From db9668b21eba62fa6497ad3024898d2672adc15b Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Mon, 24 Mar 2025 18:52:43 +0700 Subject: [PATCH 0040/1030] Update main.yml Signed-off-by: eQuantum <8466209+eq19@users.noreply.github.com> --- .github/workflows/main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index ff74af4ef997..0f492c739131 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -120,7 +120,7 @@ jobs: needs: lexering runs-on: ${{ matrix.config.os }} #if: ${{ (needs.lexering.result == 'success') }} - continue-on-error: ${{ env.JOBS_ID == 3 && true || false }} + continue-on-error: ${{ matrix.config.jobs_id == 3 && true || false }} steps: - name: 📂 Checkout From e0e7fa9666bca6c3bcfc77ab8433a09d8d37fd63 Mon Sep 17 00:00:00 2001 From: eq19 Date: Mon, 24 Mar 2025 11:53:17 +0000 Subject: [PATCH 0041/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 7 ------- .github/actions/2-multiplication/1-maps/action.yml | 9 --------- .github/actions/3-exponentiation/4-parser/action.yml | 8 -------- .github/actions/3-exponentiation/6-grammar/action.yml | 11 +++-------- .github/actions/4-identition/4-parser/action.yml | 9 --------- .github/actions/4-identition/5-syntax/action.yml | 5 ----- .github/actions/4-identition/6-grammar/action.yml | 6 +++--- .github/entrypoint/init.sh | 6 +++--- .github/workflows/main.yml | 2 +- 9 files changed, 10 insertions(+), 53 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 4c1dcbf117be..ae236b5bcd7e 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -8,9 +8,6 @@ branding: color: 'red' inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -79,8 +76,6 @@ runs: steps: - name: Remove Existing Connection shell: bash - env: - JOB_ID: ${{ inputs.id }} run: | HEADER="Accept: application/vnd.github+json" /home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh ${{ inputs.action_path }} @@ -92,8 +87,6 @@ runs: - name: View context attributes uses: actions/github-script@v7 - env: - JOB_ID: ${{ inputs.id }} with: script: console.log(context) diff --git a/.github/actions/2-multiplication/1-maps/action.yml b/.github/actions/2-multiplication/1-maps/action.yml index fd288d0af1aa..dbb14e78e6c4 100644 --- a/.github/actions/2-multiplication/1-maps/action.yml +++ b/.github/actions/2-multiplication/1-maps/action.yml @@ -2,9 +2,6 @@ name: 'Docker Deploy Action' description: > 🪂 A Github Action to deploy pages conveniently inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -88,17 +85,11 @@ runs: # Ref: https://cloud.google.com/compute/docs/instances/create-start-instance - name: 💎 Remove Existing Self-Hosted Runner shell: bash - env: - JOB_ID: ${{ inputs.id }} - TOKEN: ${{ inputs.token }} run: | /Users/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh - name: View context attributes uses: actions/github-script@v7 - env: - JOB_ID: ${{ inputs.id }} - GH_TOKEN: ${{ inputs.token }} with: script: console.log(context) diff --git a/.github/actions/3-exponentiation/4-parser/action.yml b/.github/actions/3-exponentiation/4-parser/action.yml index 4a8cf9e1281f..f8a006810014 100644 --- a/.github/actions/3-exponentiation/4-parser/action.yml +++ b/.github/actions/3-exponentiation/4-parser/action.yml @@ -8,9 +8,6 @@ branding: color: 'red' inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -77,8 +74,6 @@ runs: - name: 💎 Reinvented shell: bash env: - JOB_ID: ${{ inputs.id }} - TOKEN: ${{ inputs.token }} BASE: "https://github.com/eq19/eq19.github.io.wiki.git" WIKI: "https://github.com/${{ env.TARGET_REPOSITORY }}.wiki.git" run: | @@ -91,9 +86,6 @@ runs: - name: View context attributes uses: actions/github-script@v7 - env: - JOB_ID: ${{ inputs.id }} - GH_TOKEN: ${{ inputs.token }} with: script: console.log(context) diff --git a/.github/actions/3-exponentiation/6-grammar/action.yml b/.github/actions/3-exponentiation/6-grammar/action.yml index 7bc9107ba4d3..a5a75c7d1a4d 100644 --- a/.github/actions/3-exponentiation/6-grammar/action.yml +++ b/.github/actions/3-exponentiation/6-grammar/action.yml @@ -8,9 +8,6 @@ branding: color: 'red' inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -100,8 +97,6 @@ runs: - name: ⏪ Rewind all jobs on failure shell: bash - env: - GH_TOKEN: ${{ inputs.token }} run: | if [[ '${{ inputs.failure_status }}' == 'true' ]]; then @@ -123,13 +118,13 @@ runs: shopt -s dotglob && mv -f /home/runner/_site/* . # Assign the environment variable correctly - COMMIT="${{ env.LATEST_COMMIT }}" + #COMMIT="${{ env.LATEST_COMMIT }}" # Remove double quotes - MESSAGE=$(echo "$COMMIT" | tr -d '"') + MESSAGE=$(echo "$LATEST_COMMIT" | tr -d '"') # Debug output to verify - echo "Original: $COMMIT" + echo "Original: $LATEST_COMMIT" echo "Processed: $MESSAGE" # Commit and push changes diff --git a/.github/actions/4-identition/4-parser/action.yml b/.github/actions/4-identition/4-parser/action.yml index 3767836dbeab..4d6feb0a9102 100644 --- a/.github/actions/4-identition/4-parser/action.yml +++ b/.github/actions/4-identition/4-parser/action.yml @@ -8,9 +8,6 @@ branding: color: 'red' inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -83,16 +80,10 @@ runs: steps: - name: Run identity shell: bash - env: - JOB_ID: ${{ inputs.id }} - TOKEN: ${{ inputs.token }} run: | /d/a/_actions/eq19/eq19/v2/.github/entrypoint/init.sh - name: View context attributes uses: actions/github-script@v7 - env: - JOB_ID: ${{ inputs.id }} - GH_TOKEN: ${{ inputs.token }} with: script: console.log(context) diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index e052c1169c48..526a7089529d 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -8,9 +8,6 @@ branding: color: 'red' inputs: - id: - description: The deploy id - required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -109,8 +106,6 @@ runs: - name: Run identity shell: bash env: - JOB_ID: ${{ inputs.id }} - GH_TOKEN: ${{ inputs.token }} GCP_TOKEN: ${{ steps.auth.outputs.id_token }} run: | curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function \ diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index 0e4cb59e50e4..6517000a1367 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -126,13 +126,13 @@ runs: ls -al ./docs # Assign the environment variable correctly - COMMIT="${{ env.LATEST_COMMIT }}" + #COMMIT="${{ env.LATEST_COMMIT }}" # Remove double quotes - MESSAGE=$(echo "$COMMIT" | tr -d '"') + MESSAGE=$(echo "$LATEST_COMMIT" | tr -d '"') # Debug output to verify - echo "Original: $COMMIT" + echo "Original: $LATEST_COMMIT" echo "Processed: $MESSAGE" # Commit and push changes diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 2c87d49012b0..e3412ceb49fe 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -50,7 +50,7 @@ echo -e "\n$hr\nWORKSPACE\n$hr" RERUN_RUNNER=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/RERUN_RUNNER" | jq -r '.value') -if [[ "${JOB_ID}" == "1" ]]; then +if [[ "${JOBS_ID}" == "1" ]]; then cd ${GITHUB_WORKSPACE} && rm -rf .github cp -r /home/runner/work/_actions/eq19/eq19/v2/.github . @@ -82,11 +82,11 @@ if [[ "${JOB_ID}" == "1" ]]; then fi -elif [[ "${JOB_ID}" == "2" ]]; then +elif [[ "${JOBS_ID}" == "2" ]]; then ls -alR ${GITHUB_WORKSPACE} -elif [[ "${JOB_ID}" == "3" ]]; then +elif [[ "${JOBS_ID}" == "3" ]]; then cd /home/runner/_site && rm -rf README.md docs && gist.sh ${BASE} $(pwd) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index 0f492c739131..da492856bfe9 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -2,7 +2,7 @@ name: COmpendium of RElations – Modulo 6 on: - push: + #push: workflow_run: types: [completed] workflows: ["pages-build-deployment"] From 09967148691422170d39cfc4839232a5b5cdc393 Mon Sep 17 00:00:00 2001 From: eq19 Date: Tue, 25 Mar 2025 06:41:25 +0000 Subject: [PATCH 0042/1030] update workflows --- .github/actions/1-addition/2-feed/action.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/actions/1-addition/2-feed/action.yml b/.github/actions/1-addition/2-feed/action.yml index ef49be6872c5..00be0be2fe9d 100644 --- a/.github/actions/1-addition/2-feed/action.yml +++ b/.github/actions/1-addition/2-feed/action.yml @@ -187,7 +187,8 @@ runs: imageName: ${{ inputs.image_name }} configFile: _site/.devcontainer.json env: | - TOKEN=${{ env.GH_TOKEN }} + ID=${{ env.ID }} + GH_TOKEN=${{ env.GH_TOKEN }} REMOTE_REPO=${{ env.REMOTE_REPO }} RERUN_RUNNER=${{ inputs.rerun_runner }} BEARER=${{ steps.auth.outputs.id_token }} From f52981e17560649eb91ddc5c18011e97a13bd0b7 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 26 Mar 2025 21:38:11 +0000 Subject: [PATCH 0043/1030] update workflows --- .../artifact/python/src/params/spaces.json | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/.github/entrypoint/artifact/python/src/params/spaces.json b/.github/entrypoint/artifact/python/src/params/spaces.json index 858ee348e189..eb82ec5334df 100644 --- a/.github/entrypoint/artifact/python/src/params/spaces.json +++ b/.github/entrypoint/artifact/python/src/params/spaces.json @@ -7,9 +7,6 @@ "11": 0.006, "19": 0 }, - "stoploss": { - "stoploss": -0.137 - }, "trailing": { "trailing_stop": true, "trailing_stop_positive": 0.099, @@ -20,15 +17,15 @@ "max_open_trades": Infinity }, "buy": { + "SmoothK": 2, "buy_additional_indicator": "NONE", "buy_fast_dema": "13", "buy_fib_level": "0.618", "buy_rsi": 45, "buy_slow_ema": "34", + "buy_stoch_osc": 29, "buy_swing_period": 50, "period": 14, - "SmoothK": 2, - "buy_stoch_osc": 29, "smoothD": 3 }, "sell": { @@ -39,16 +36,19 @@ "sell_stoch_osc": 85 }, "protection": { - "cooldown_lookback": 7, - "low_profit_trade_limit": 6, - "max_drawdown_trade_limit": 4, - "stop_duration": 156, - "trade_limit": 9, + "cooldown_lookback": 4, + "low_profit_trade_limit": 7, + "max_drawdown_trade_limit": 6, + "stop_duration": 36, + "trade_limit": 3, "use_low_profit": true, "use_max_drawdown_protection": false, - "use_stop_protection": true + "use_stop_protection": false + }, + "stoploss": { + "stoploss": -0.104 } }, "ft_stratparam_v": 1, - "export_time": "2025-02-20 13:28:29.884316+00:00" -} + "export_time": "2025-03-26 11:31:32.125452+00:00" +} \ No newline at end of file From 455b5e47c92c5554433daaf2ec45141c6f6b39bb Mon Sep 17 00:00:00 2001 From: eq19 Date: Thu, 27 Mar 2025 16:15:39 +0000 Subject: [PATCH 0044/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index ae236b5bcd7e..d4ade95c996e 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -119,7 +119,7 @@ runs: ##python ${{ inputs.action_path }}/pythonCode/testIREE.py ##Ref: https://github.com/tsoding/JelloVM - cd ${{ inputs.action_path }} && javac javaCode/Main.java + cd ${{ inputs.action_path }} && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java python app.py output.txt && cat dataFile/results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode From 6462caa3cc7ba0cc2b6c7bf6c07f5c3fb705e732 Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 28 Mar 2025 10:13:50 +0000 Subject: [PATCH 0045/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index d4ade95c996e..bc4d8e0ed342 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -120,7 +120,7 @@ runs: ##Ref: https://github.com/tsoding/JelloVM cd ${{ inputs.action_path }} && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java - python app.py output.txt && cat dataFile/results/output.txt + cd dataFile/user_data/ft_client/test_client && python app.py output.txt && cat results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py From 094333b3b3f245b560ace4946177b6ee58fab0cc Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 28 Mar 2025 10:25:24 +0000 Subject: [PATCH 0046/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index bc4d8e0ed342..1f95547355cd 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -120,7 +120,7 @@ runs: ##Ref: https://github.com/tsoding/JelloVM cd ${{ inputs.action_path }} && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java - cd dataFile/user_data/ft_client/test_client && python app.py output.txt && cat results/output.txt + cd dataFile/user_data/ft_client/test_client && ls -al . && python app.py output.txt && cat results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py From 7d6fcc0708ba4145c1726e481d94e45cf90780c7 Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 28 Mar 2025 19:54:59 +0000 Subject: [PATCH 0047/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 4 ++-- .github/entrypoint/init.sh | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 1f95547355cd..5fcc8e612163 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -119,8 +119,8 @@ runs: ##python ${{ inputs.action_path }}/pythonCode/testIREE.py ##Ref: https://github.com/tsoding/JelloVM - cd ${{ inputs.action_path }} && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java - cd dataFile/user_data/ft_client/test_client && ls -al . && python app.py output.txt && cat results/output.txt + #cd ${{ inputs.action_path }} && javac -d user_data/ft_client/test_client javaCode/Main.java + cd user_data/ft_client/test_client && ls -al . && python app.py output.txt && cat results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index e3412ceb49fe..8fd21b8068bd 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -77,8 +77,9 @@ if [[ "${JOBS_ID}" == "1" ]]; then echo "Invalid JSON" fi - cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi + cd $1 && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java + cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . fi From ed4e84a1210dc38d4153bb6c9055c880483da115 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sat, 29 Mar 2025 15:25:04 +0000 Subject: [PATCH 0048/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 4 ++-- .github/entrypoint/init.sh | 3 +-- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 5fcc8e612163..ae236b5bcd7e 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -119,8 +119,8 @@ runs: ##python ${{ inputs.action_path }}/pythonCode/testIREE.py ##Ref: https://github.com/tsoding/JelloVM - #cd ${{ inputs.action_path }} && javac -d user_data/ft_client/test_client javaCode/Main.java - cd user_data/ft_client/test_client && ls -al . && python app.py output.txt && cat results/output.txt + cd ${{ inputs.action_path }} && javac javaCode/Main.java + python app.py output.txt && cat dataFile/results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 8fd21b8068bd..e3412ceb49fe 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -77,9 +77,8 @@ if [[ "${JOBS_ID}" == "1" ]]; then echo "Invalid JSON" fi - if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi - cd $1 && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . + if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi fi From 725e693227b21f1281572860ce2fa992daf4aaa8 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 30 Mar 2025 04:05:12 +0000 Subject: [PATCH 0049/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 2 +- .github/entrypoint/init.sh | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index ae236b5bcd7e..0f5c9a26bb50 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -119,7 +119,7 @@ runs: ##python ${{ inputs.action_path }}/pythonCode/testIREE.py ##Ref: https://github.com/tsoding/JelloVM - cd ${{ inputs.action_path }} && javac javaCode/Main.java + cd ${{ inputs.action_path }} #&& javac javaCode/Main.java python app.py output.txt && cat dataFile/results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index e3412ceb49fe..81c0d51fa848 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -77,6 +77,7 @@ if [[ "${JOBS_ID}" == "1" ]]; then echo "Invalid JSON" fi + cd $1 && javac javaCode/Main.java cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi From 428f7d51eaaa7d7fab8d977946293460f4f4ebf3 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 30 Mar 2025 04:44:05 +0000 Subject: [PATCH 0050/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 6 ++++-- .github/entrypoint/init.sh | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 0f5c9a26bb50..5be9bbaf8ca3 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -119,8 +119,10 @@ runs: ##python ${{ inputs.action_path }}/pythonCode/testIREE.py ##Ref: https://github.com/tsoding/JelloVM - cd ${{ inputs.action_path }} #&& javac javaCode/Main.java - python app.py output.txt && cat dataFile/results/output.txt + #cd ${{ inputs.action_path }} && javac javaCode/Main.java + + cd user_data/ft_client/test_client && ls -al . + python app.py output.txt && cat results/output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 81c0d51fa848..8fd21b8068bd 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -77,9 +77,9 @@ if [[ "${JOBS_ID}" == "1" ]]; then echo "Invalid JSON" fi - cd $1 && javac javaCode/Main.java - cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi + cd $1 && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java + cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . fi From 771f74b3d343017bf680f0c870e1039335f81c03 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 30 Mar 2025 06:37:55 +0000 Subject: [PATCH 0051/1030] update workflows --- .github/entrypoint/init.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 8fd21b8068bd..d07b306b1d86 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -69,17 +69,17 @@ if [[ "${JOBS_ID}" == "1" ]]; then PARAMS_JSON=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" \ "https://api.github.com/repos/${GITHUB_REPOSITORY}/actions/variables/PARAMS_JSON" | jq -r '.value') - echo "${PARAMS_JSON}" | jq '.' > $1/dataFile/user_data/strategies/fibbo.json + echo "${PARAMS_JSON}" | jq '.' > $1/user_data/strategies/fibbo.json - if jq empty < $1/dataFile/user_data/strategies/fibbo.json; then - cat $1/dataFile/user_data/strategies/fibbo.json + if jq empty < $1/user_data/strategies/fibbo.json; then + cat $1/user_data/strategies/fibbo.json else echo "Invalid JSON" fi if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi - cd $1 && javac -d dataFile/user_data/ft_client/test_client javaCode/Main.java - cd ${GITHUB_WORKSPACE} && mv -f $1/dataFile/user_data . && ls -al . + cd $1 && javac -d user_data/ft_client/test_client javaCode/Main.java + cd ${GITHUB_WORKSPACE} && mv -f $1/user_data . && ls -al . fi From 7a62fb3c29f9833ae055648e313a2d09f292477b Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 30 Mar 2025 10:13:42 +0000 Subject: [PATCH 0052/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 5be9bbaf8ca3..5aaea365856c 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -109,8 +109,8 @@ runs: - name: Install dependencies shell: bash run: | - python -m pip -q install --upgrade pip - python -m pip -q install -r ${{ inputs.action_path }}/dataFile/requirements.txt + #python -m pip -q install --upgrade pip + #python -m pip -q install -r ${{ inputs.action_path }}/dataFile/requirements.txt ##python -m pip -q install -e ./.github/entrypoint/artifact/python ##iree-compile --iree-llvmcpu-list-targets && iree-run-module --list_drivers From 7fb5179778e716c750b9b00df520fcdf96dc26f1 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 30 Mar 2025 10:58:29 +0000 Subject: [PATCH 0053/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 25 +++++++++----------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 5aaea365856c..9c4712909b58 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -96,19 +96,19 @@ runs: username: ${{ inputs.hub_username }} password: ${{ inputs.hub_token }} - - uses: actions/setup-python@v5 - with: - python-version: 3.12 - allow-prereleases: true + #- uses: actions/setup-python@v5 + #with: + #python-version: 3.12 + #allow-prereleases: true - - uses: actions/cache@v4 - with: - path: ~/.cache/pip - key: ${{ runner.os }}-pip + #- uses: actions/cache@v4 + #with: + #path: ~/.cache/pip + #key: ${{ runner.os }}-pip - - name: Install dependencies - shell: bash - run: | + #- name: Install dependencies + #shell: bash + #run: | #python -m pip -q install --upgrade pip #python -m pip -q install -r ${{ inputs.action_path }}/dataFile/requirements.txt @@ -121,9 +121,6 @@ runs: ##Ref: https://github.com/tsoding/JelloVM #cd ${{ inputs.action_path }} && javac javaCode/Main.java - cd user_data/ft_client/test_client && ls -al . - python app.py output.txt && cat results/output.txt - ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py ##cd ${{ inputs.action_path }}/pythonCode && python npreader.py -h && python map.py -h From 42ea79fdc682882bcac26c633455930e57a3a341 Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 2 Apr 2025 06:11:49 +0000 Subject: [PATCH 0054/1030] update workflows --- .github/actions/1-addition/2-feed/action.yml | 1 - .github/entrypoint/artifact.sh | 1 - 2 files changed, 2 deletions(-) diff --git a/.github/actions/1-addition/2-feed/action.yml b/.github/actions/1-addition/2-feed/action.yml index 00be0be2fe9d..b9120d83a9b9 100644 --- a/.github/actions/1-addition/2-feed/action.yml +++ b/.github/actions/1-addition/2-feed/action.yml @@ -123,7 +123,6 @@ runs: PYPI_TOKEN: ${{ inputs.pypi_token }} ACTOR: ${{ github.triggering_actor }} OWNER: ${{ github.repository_owner }} - BEARER: ${{ steps.auth.outputs.id_token }} DOCKER_HUB_TOKEN: ${{ inputs.hub_token }} DOCKER_HUB_PASSWORD: ${{ inputs.hub_password }} GCP_CREDENTIALS: ${{ inputs.credentials }} diff --git a/.github/entrypoint/artifact.sh b/.github/entrypoint/artifact.sh index aed2e74643d4..d2964ec8e2d6 100755 --- a/.github/entrypoint/artifact.sh +++ b/.github/entrypoint/artifact.sh @@ -28,7 +28,6 @@ set_target() { if [[ "$i" -lt "${#array[@]}-1" ]]; then echo "," >> ${RUNNER_TEMP}/orgs.json; fi done echo "]" >> ${RUNNER_TEMP}/orgs.json - curl -s -X POST https://us-central1-feedmapping.cloudfunctions.net/function -H "Authorization: Bearer ${BEARER}" -H "Content-Type: application/json" --data @${RUNNER_TEMP}/orgs.json | jq '.' > ${RUNNER_TEMP}/gist.json fi # Iterate the Structure From 6ee300831c7c664ce52eb9ca77d4d79e3f961051 Mon Sep 17 00:00:00 2001 From: eq19 Date: Fri, 4 Apr 2025 06:20:20 +0000 Subject: [PATCH 0055/1030] update workflows --- .github/entrypoint/init.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index d07b306b1d86..80145c8ee35b 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -78,8 +78,8 @@ if [[ "${JOBS_ID}" == "1" ]]; then fi if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi - cd $1 && javac -d user_data/ft_client/test_client javaCode/Main.java - cd ${GITHUB_WORKSPACE} && mv -f $1/user_data . && ls -al . + cd $1 && javac -d user_data/ft_client/test_client javaCode/Main.java && cd ${GITHUB_WORKSPACE} + if [[ ! -d user_data ]]; then mv -f $1/user_data . && ls -al; fi fi From 9a49c37c27a6f3bd48382fe870c7c855a6be9dd8 Mon Sep 17 00:00:00 2001 From: eq19 Date: Sun, 6 Apr 2025 04:24:44 +0000 Subject: [PATCH 0056/1030] update workflows --- .github/entrypoint/init.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/entrypoint/init.sh b/.github/entrypoint/init.sh index 80145c8ee35b..e39ef815a7d5 100755 --- a/.github/entrypoint/init.sh +++ b/.github/entrypoint/init.sh @@ -77,9 +77,9 @@ if [[ "${JOBS_ID}" == "1" ]]; then echo "Invalid JSON" fi + cd $1 && javac -d user_data/ft_client/test_client javaCode/Main.java + cd ${GITHUB_WORKSPACE} && rm -rf user_data && mv -f $1/user_data . && ls -al . if [[ "${RERUN_RUNNER}" != "false" ]]; then gh variable set RERUN_RUNNER --body "false"; fi - cd $1 && javac -d user_data/ft_client/test_client javaCode/Main.java && cd ${GITHUB_WORKSPACE} - if [[ ! -d user_data ]]; then mv -f $1/user_data . && ls -al; fi fi From adc42b5c875ac0c641b1f5d98a5622d004b2ba2d Mon Sep 17 00:00:00 2001 From: eQuantum <8466209+eq19@users.noreply.github.com> Date: Wed, 7 May 2025 13:13:12 +0000 Subject: [PATCH 0057/1030] Update file --- .github/workflows/main.yml | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml index da492856bfe9..d282d39b977d 100644 --- a/.github/workflows/main.yml +++ b/.github/workflows/main.yml @@ -6,6 +6,8 @@ on: workflow_run: types: [completed] workflows: ["pages-build-deployment"] + repository_dispatch: + types: [trigger_action, retry_workflow] # 🕸️ Allow concurrent concurrency: @@ -18,6 +20,7 @@ permissions: write-all # 🕸️ Global Environment env: GH_TOKEN: ${{ secrets.ACCESS_TOKEN }} + RUNNER_TITLE: "Google-optimized-instance" TARGET_REPOSITORY: ${{ vars.TARGET_REPOSITORY }} jobs: @@ -42,20 +45,25 @@ jobs: - name: 📂 Checkout uses: actions/checkout@v4 with: - submodules: recursive token: ${{ secrets.ACCESS_TOKEN }} repository: ${{ env.JOBS_ID != 1 && env.DOC || '' }} - + submodules: ${{ env.JOBS_ID != 1 && 'recursive' || false }} + - name: ⚙️ Build Maps uses: eq19/maps@v8 id: build-parser with: token: ${{ secrets.ACCESS_TOKEN }} - rerun_runner: ${{ vars.RERUN_RUNNER }} + chat_id: ${{ secrets.MESSAGE_API }} + api_key: ${{ secrets.ACCESS_API }} + api_secret: ${{ secrets.ACCESS_KEY }} + pypi_token: ${{ secrets.PYPI_TOKEN }} + bot_token: ${{ secrets.MONITOR_TOKEN }} + logs_token: ${{ secrets.WARNING_TOKEN }} credentials: ${{ secrets.GCP_CREDENTIALS }} docker_hub_token: ${{ secrets.DOCKER_HUB_TOKEN }} docker_hub_password: ${{ secrets.DOCKER_HUB_PASSWORD }} - initiate_pauli_sum: ${{ toJSON(steps.set-lexering.outputs) }} + initiate_pauli_sum: ${{ toJSON(steps.build-parser.outputs) }} - name: 🪂 Feed Mapping uses: eq19/feed@v6 @@ -65,7 +73,6 @@ jobs: chat_id: ${{ secrets.MESSAGE_API }} api_key: ${{ secrets.ACCESS_API }} api_secret: ${{ secrets.ACCESS_KEY }} - rerun_runner: ${{ vars.RERUN_RUNNER }} pypi_token: ${{ secrets.PYPI_TOKEN }} bot_token: ${{ secrets.MONITOR_TOKEN }} logs_token: ${{ secrets.WARNING_TOKEN }} From 891d56f42f88dc8009832723e0af8b2250e8d13e Mon Sep 17 00:00:00 2001 From: eq19 Date: Wed, 7 May 2025 14:11:53 +0000 Subject: [PATCH 0058/1030] update workflows --- .github/actions/1-addition/1-maps/action.yml | 131 +- .github/actions/1-addition/2-feed/action.yml | 44 +- .github/actions/1-addition/3-lexer/action.yml | 38 +- .../2-multiplication/1-maps/action.yml | 31 +- .../2-multiplication/2-feed/action.yml | 39 +- .../2-multiplication/3-lexer/action.yml | 2 +- .../3-exponentiation/5-syntax/action.yml | 3 + .../3-exponentiation/6-grammar/action.yml | 19 +- .../actions/4-identition/4-parser/action.yml | 3 +- .../actions/4-identition/5-syntax/action.yml | 3 +- .../actions/4-identition/6-grammar/action.yml | 70 +- .github/entrypoint/artifact.sh | 70 +- .github/entrypoint/dockerfile/Doxyfile | 2473 +++++++ .github/entrypoint/dockerfile/conf/Dockerfile | 226 + .github/entrypoint/dockerfile/conf/README.md | 5 + .../00-copy-config-file.sh | 3 + .../10-create-tables.sh | 53 + .../11-fill-data.sh | 39 + .../entrypoint/dockerfile/conf/init-db.sql | 17 + .../entrypoint/dockerfile/conf/package.json | 109 + .../entrypoint/dockerfile/conf/pg_hba.conf | 10 + .../dockerfile/conf/postgresql.conf | 800 +++ .../dockerfile/conf/postgresql.conf.init | 11 + .../entrypoint/dockerfile/conf/query-db.sql | 1 + .../entrypoint/dockerfile/conf/run_future.sh | 21 + .../dockerfile/conf/supervisord.conf | 24 + .../dockerfile/conf/supervisord.conf.base | 19 + .github/entrypoint/dockerfile/conf/t.sh | 12 + .github/entrypoint/dockerfile/hooks/index.js | 6318 +++++++++++++++++ .../dockerfile/hooks/job_completed.sh | 41 + .../dockerfile/hooks/job_started.sh | 102 + .github/entrypoint/dockerfile/index.js | 139 + .github/entrypoint/dockerfile/package.json | 33 + .../entrypoint/dockerfile/requirements.apt | 45 + .../entrypoint/dockerfile/scripts/compound.js | 109 + .../entrypoint/dockerfile/scripts/doxygen.sh | 44 + .../dockerfile/scripts/entrypoint.sh | 75 + .../dockerfile/scripts/exitpoint.sh | 83 + .../entrypoint/dockerfile/scripts/helpers.js | 72 + .../dockerfile/scripts/jekyll_site.sh | 45 + .../entrypoint/dockerfile/scripts/logger.js | 62 + .github/entrypoint/dockerfile/scripts/main.js | 81 + .../entrypoint/dockerfile/scripts/markdown.js | 30 + .../entrypoint/dockerfile/scripts/parser.js | 497 ++ .github/entrypoint/dockerfile/scripts/post.js | 39 + .../entrypoint/dockerfile/scripts/runner.sh | 16 + .../dockerfile/scripts/templates.js | 101 + .../dockerfile/templates/cpp/class.md | 48 + .../dockerfile/templates/cpp/index.md | 27 + .../dockerfile/templates/cpp/namespace.md | 33 + .../dockerfile/templates/cpp/page.md | 34 + .../dockerfile/templates/doxygen/doc/.gitkeep | 0 .../templates/doxygen/doc/api-bicycle.md | 40 + .../templates/doxygen/doc/api-mountainbike.md | 48 + .../templates/doxygen/doc/api-racingbike.md | 40 + .../templates/doxygen/doc/page-changelog.md | 10 + .../templates/doxygen/doc/page-overview.md | 8 + .../templates/doxygen/src/CMakeLists.txt | 12 + .../templates/doxygen/src/gettingStarted.cpp | 127 + .../templates/doxygen/src/sample2HDM.cpp | 46 + .../templates/doxygen/src/samplePMSSM.cpp | 20 + .../templates/doxygen/src/sampleQCDSextet.cpp | 83 + .../templates/doxygen/src/sampleQED.cpp | 40 + .../templates/doxygen/src/sampleSQED.cpp | 46 + .../templates/doxygen/src/sampleScalar.cpp | 43 + .../templates/doxygen/xml/bicycle_8cpp.xml | 43 + .../templates/doxygen/xml/bicycle_8h.xml | 33 + .../templates/doxygen/xml/changelog.xml | 17 + .../templates/doxygen/xml/changelog_8md.xml | 22 + .../doxygen/xml/classtransport_1_1Bicycle.xml | 80 + .../xml/classtransport_1_1MountainBike.xml | 87 + .../xml/classtransport_1_1RacingBike.xml | 72 + .../templates/doxygen/xml/combine.xslt | 15 + .../templates/doxygen/xml/compound.xsd | 1131 +++ .../dir_68267d1309a1af8e8297ef4c3efbcdba.xml | 18 + .../templates/doxygen/xml/group__bicycle.xml | 12 + .../doxygen/xml/group__mountainbike.xml | 12 + .../doxygen/xml/group__racingbike.xml | 12 + .../templates/doxygen/xml/index.xml | 53 + .../templates/doxygen/xml/index.xsd | 66 + .../doxygen/xml/mountainbike_8cpp.xml | 40 + .../templates/doxygen/xml/mountainbike_8h.xml | 54 + .../doxygen/xml/namespacetransport.xml | 14 + .../templates/doxygen/xml/overview.xml | 12 + .../templates/doxygen/xml/overview_8md.xml | 20 + .../templates/doxygen/xml/racingbike_8cpp.xml | 44 + .../templates/doxygen/xml/racingbike_8h.xml | 47 + .../templates/doxygen/xml/transport_8h.xml | 102 + .github/entrypoint/init.sh | 154 +- .github/entrypoint/remote.sh | 55 +- .github/entrypoint/sync.sh | 57 +- .../templates/{config.yml => blank_issue.yml} | 0 .github/templates/jekyll_config.yml | 41 + 93 files changed, 15000 insertions(+), 226 deletions(-) create mode 100644 .github/entrypoint/dockerfile/Doxyfile create mode 100644 .github/entrypoint/dockerfile/conf/Dockerfile create mode 100644 .github/entrypoint/dockerfile/conf/README.md create mode 100755 .github/entrypoint/dockerfile/conf/docker-entrypoint-initdb.d/00-copy-config-file.sh create mode 100755 .github/entrypoint/dockerfile/conf/docker-entrypoint-initdb.d/10-create-tables.sh create mode 100755 .github/entrypoint/dockerfile/conf/docker-entrypoint-initdb.d/11-fill-data.sh create mode 100644 .github/entrypoint/dockerfile/conf/init-db.sql create mode 100644 .github/entrypoint/dockerfile/conf/package.json create mode 100644 .github/entrypoint/dockerfile/conf/pg_hba.conf create mode 100644 .github/entrypoint/dockerfile/conf/postgresql.conf create mode 100644 .github/entrypoint/dockerfile/conf/postgresql.conf.init create mode 100644 .github/entrypoint/dockerfile/conf/query-db.sql create mode 100644 .github/entrypoint/dockerfile/conf/run_future.sh create mode 100644 .github/entrypoint/dockerfile/conf/supervisord.conf create mode 100644 .github/entrypoint/dockerfile/conf/supervisord.conf.base create mode 100644 .github/entrypoint/dockerfile/conf/t.sh create mode 100644 .github/entrypoint/dockerfile/hooks/index.js create mode 100755 .github/entrypoint/dockerfile/hooks/job_completed.sh create mode 100755 .github/entrypoint/dockerfile/hooks/job_started.sh create mode 100644 .github/entrypoint/dockerfile/index.js create mode 100644 .github/entrypoint/dockerfile/package.json create mode 100644 .github/entrypoint/dockerfile/requirements.apt create mode 100644 .github/entrypoint/dockerfile/scripts/compound.js create mode 100755 .github/entrypoint/dockerfile/scripts/doxygen.sh create mode 100755 .github/entrypoint/dockerfile/scripts/entrypoint.sh create mode 100755 .github/entrypoint/dockerfile/scripts/exitpoint.sh create mode 100644 .github/entrypoint/dockerfile/scripts/helpers.js create mode 100755 .github/entrypoint/dockerfile/scripts/jekyll_site.sh create mode 100644 .github/entrypoint/dockerfile/scripts/logger.js create mode 100644 .github/entrypoint/dockerfile/scripts/main.js create mode 100644 .github/entrypoint/dockerfile/scripts/markdown.js create mode 100644 .github/entrypoint/dockerfile/scripts/parser.js create mode 100644 .github/entrypoint/dockerfile/scripts/post.js create mode 100755 .github/entrypoint/dockerfile/scripts/runner.sh create mode 100644 .github/entrypoint/dockerfile/scripts/templates.js create mode 100644 .github/entrypoint/dockerfile/templates/cpp/class.md create mode 100644 .github/entrypoint/dockerfile/templates/cpp/index.md create mode 100644 .github/entrypoint/dockerfile/templates/cpp/namespace.md create mode 100644 .github/entrypoint/dockerfile/templates/cpp/page.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/.gitkeep create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/api-bicycle.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/api-mountainbike.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/api-racingbike.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/page-changelog.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/doc/page-overview.md create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/CMakeLists.txt create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/gettingStarted.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/sample2HDM.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/samplePMSSM.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/sampleQCDSextet.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/sampleQED.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/sampleSQED.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/src/sampleScalar.cpp create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/bicycle_8cpp.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/bicycle_8h.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/changelog.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/changelog_8md.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/classtransport_1_1Bicycle.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/classtransport_1_1MountainBike.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/classtransport_1_1RacingBike.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/combine.xslt create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/compound.xsd create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/dir_68267d1309a1af8e8297ef4c3efbcdba.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/group__bicycle.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/group__mountainbike.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/group__racingbike.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/index.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/index.xsd create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/mountainbike_8cpp.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/mountainbike_8h.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/namespacetransport.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/overview.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/overview_8md.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/racingbike_8cpp.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/racingbike_8h.xml create mode 100644 .github/entrypoint/dockerfile/templates/doxygen/xml/transport_8h.xml rename .github/templates/{config.yml => blank_issue.yml} (100%) create mode 100644 .github/templates/jekyll_config.yml diff --git a/.github/actions/1-addition/1-maps/action.yml b/.github/actions/1-addition/1-maps/action.yml index 9c4712909b58..7cbf41dd7b90 100644 --- a/.github/actions/1-addition/1-maps/action.yml +++ b/.github/actions/1-addition/1-maps/action.yml @@ -8,6 +8,9 @@ branding: color: 'red' inputs: + id: + description: The deploy id + required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -26,6 +29,7 @@ inputs: required: true image_name: description: Tag to use for the new image + default: ${{ github.action_repository }} required: true dockerfile_path: description: Path to the dockerfile from which to build the image @@ -49,9 +53,6 @@ inputs: description: Path for the docker cache destination required: false default: /tmp/.buildx-cache-new - provider: - description: 'The deploy provider' - required: true owner: description: 'The deploy owner' required: false @@ -62,6 +63,24 @@ inputs: description: 'The deploy repository' default: ${{ github.repository }} required: false + api_key: + description: The key of api application + required: true + api_secret: + description: The secret of api application + required: true + pypi_token: + description: The pypi token + required: true + bot_token: + description: The monitor token + required: true + logs_token: + description: The warning token + required: true + chat_id: + description: The telegram chat id + required: true outputs: artifact: @@ -76,26 +95,38 @@ runs: steps: - name: Remove Existing Connection shell: bash + env: + RUN: ${{ github.run_id }} + USER: ${{ github.actor }} + REPO: ${{ github.repository }} + ACTOR: ${{ github.triggering_actor }} + OWNER: ${{ github.repository_owner }} run: | - HEADER="Accept: application/vnd.github+json" - /home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh ${{ inputs.action_path }} + gh variable set RERUN_RUNNER --body "false" - #See: https://docs.github.com/en/rest/actions/self-hosted-runners + HEADER="Accept: application/vnd.github+json" + RESPONSE=$(gh api -H "${HEADER}" repos/${{ github.repository }}/actions/runners) TOTAL_COUNT=$(gh api -H "${HEADER}" /repos/${{ github.repository }}/actions/runners --jq '.total_count') - RUNNER_ID=$(gh api -H "${HEADER}" /repos/${{ github.repository }}/actions/runners --jq '.runners.[].id') - if (( $TOTAL_COUNT != 0 )); then gh api --method DELETE -H "${HEADER}" /repos/${{ github.repository }}/actions/runners/${RUNNER_ID}; fi + STATUS=$(echo "$RESPONSE" | jq -r --arg NAME "${{ env.RUNNER_TITLE }}" '.runners[] | select(.name == $NAME).status') + + if [[ "$TOTAL_COUNT" -eq 0 ]]; then + export "RUNNER_COUNT=$TOTAL_COUNT" + gh variable set RERUN_RUNNER --body "true" + elif [[ "$STATUS" == "offline" ]]; then + export "RUNNER_STATUS=$STATUS" + gh variable set RERUN_RUNNER --body "true" + RUNNER_ID=$(gh api -H "${HEADER}" /repos/${{ github.repository }}/actions/runners --jq '.runners.[].id') + gh api --method DELETE -H "${HEADER}" /repos/${{ github.repository }}/actions/runners/${RUNNER_ID} + fi + + /home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/init.sh ${{ inputs.action_path }} + mv -f /home/runner/work/_actions/eq19/eq19/v2/.github/entrypoint/dockerfile/* ${{ inputs.action_path }}/ - name: View context attributes uses: actions/github-script@v7 with: script: console.log(context) - - name: Login to DockerHub - uses: docker/login-action@v3 - with: - username: ${{ inputs.hub_username }} - password: ${{ inputs.hub_token }} - #- uses: actions/setup-python@v5 #with: #python-version: 3.12 @@ -106,9 +137,33 @@ runs: #path: ~/.cache/pip #key: ${{ runner.os }}-pip - #- name: Install dependencies - #shell: bash - #run: | + - name: 💎 Set ID Token + id: 'auth' + uses: 'google-github-actions/auth@v2' + with: + token_format: 'id_token' + create_credentials_file: false + credentials_json: ${{ inputs.credentials }} + id_token_audience: 'https://us-central1-feedmapping.cloudfunctions.net/function' + + - name: Install dependencies + shell: bash + env: + ACCESS_API: ${{ inputs.api_key }} + ACCESS_KEY: ${{ inputs.api_secret }} + ACCESS_TOKEN: ${{ inputs.token }} + MESSAGE_API: ${{ inputs.chat_id }} + MONITOR_TOKEN: ${{ inputs.bot_token }} + WARNING_TOKEN: ${{ inputs.logs_token }} + PYPI_TOKEN: ${{ inputs.pypi_token }} + DOCKER_HUB_TOKEN: ${{ inputs.hub_token }} + DOCKER_HUB_PASSWORD: ${{ inputs.hub_password }} + GCP_CREDENTIALS: ${{ inputs.credentials }} + run: | + # Get all secrets from source repository + secrets=$(curl -s -H "Authorization: token $GH_TOKEN" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/${{ github.repository }}/actions/secrets" | jq -r '.secrets[].name') + for secret in $secrets; do gh secret set $secret --repo ${{ env.TARGET_REPOSITORY }} --body "${!secret}"; done + #python -m pip -q install --upgrade pip #python -m pip -q install -r ${{ inputs.action_path }}/dataFile/requirements.txt @@ -120,31 +175,45 @@ runs: ##Ref: https://github.com/tsoding/JelloVM #cd ${{ inputs.action_path }} && javac javaCode/Main.java + python user_data/ft_client/test_client/app.py output.txt ##Ref: https://github.com/eq19/maps/tree/v3/pythonCode ##python ${{ github.workspace }}/.github/entrypoint/artifact/python/gellmann.py ##cd ${{ inputs.action_path }}/pythonCode && python npreader.py -h && python map.py -h - + + - name: Login to DockerHub + uses: docker/login-action@v3 + with: + username: ${{ inputs.hub_username }} + password: ${{ inputs.hub_token }} + - name: 🚀 Initiate Lexer uses: devcontainers/ci@v0.3 with: - push: always skipContainerUserIdUpdate: true imageName: ${{ inputs.image_name }} imageTag: target-${{ github.run_number }} configFile: ${{ inputs.action_path }}/.devcontainer.json - runCmd: ls -al / + push: ${{ env.RERUN_RUNNER == 'true' && 'always' || 'never' }} + runCmd: bash user_data/ft_client/test_client/maps.sh + env: | + ID=${{ env.ID }} + GH_TOKEN=${{ env.GH_TOKEN }} + REMOTE_REPO=${{ env.REMOTE_REPO }} + BEARER=${{ steps.auth.outputs.id_token }} + GITHUB_REPOSITORY=${{ github.repository }} + TARGET_REPOSITORY=${{ env.TARGET_REPOSITORY }} + WORKSPACE=/workspaces/${{ github.event.repository.name }} + OUTPUT=user_data/ft_client/test_client/results/output.txt + ARTIFACT=user_data/ft_client/test_client/results/orgs.json - - name: 💎 Variables - id: variables + - name: 🗑️ Delete Docker Tags AND Images + id: delete_images shell: bash + env: + IMAGE_NAME: "${{ inputs.image_name }}" + HUB_USERNAME: "${{ inputs.hub_username }}" + HUB_PASSWORD: "${{ inputs.hub_password }}" + MAX_DELETIONS: 15 run: | - BASE_NAME=$(basename ${{ inputs.image_name }}) - IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) - for ((i=0; i < ${#array[@]}; i++)); do - if [ "${array[$i]}" != "latest" ]; then - HUB_TOKEN=$(curl -s -H "Content-Type: application/json" -X POST -d "{\"username\": \"${{ inputs.hub_username }}\", \"password\": \"${{ inputs.hub_password }}\"}" https://hub.docker.com/v2/users/login/ | jq -r .token) - curl -s -i -X DELETE -H "Accept: application/json" -H "Authorization: JWT $HUB_TOKEN" https://hub.docker.com/v2/namespaces/${{ inputs.hub_username }}/repositories/$BASE_NAME/tags/${array[$i]} - echo "deleted: ${{ inputs.image_name }}:${array[$i]}" - fi - done + ./.github/entrypoint/remote.sh diff --git a/.github/actions/1-addition/2-feed/action.yml b/.github/actions/1-addition/2-feed/action.yml index b9120d83a9b9..bbc3ccace831 100644 --- a/.github/actions/1-addition/2-feed/action.yml +++ b/.github/actions/1-addition/2-feed/action.yml @@ -53,10 +53,6 @@ inputs: description: Path for the docker cache destination required: false default: /tmp/.buildx-cache-new - rerun_runner: - description: Redeploy hosted runner - default: 'false' - required: true owner: description: 'The deploy owner' required: false @@ -97,48 +93,13 @@ outputs: runs: using: composite steps: - - name: 💎 Set ID Token - id: 'auth' - uses: 'google-github-actions/auth@v2' - with: - token_format: 'id_token' - create_credentials_file: false - credentials_json: ${{ inputs.credentials }} - id_token_audience: 'https://us-central1-feedmapping.cloudfunctions.net/function' - - name: 💎 Set Artifact id: artifact shell: bash - env: - TMP: ${{ inputs.tmp }} - RUN: ${{ github.run_id }} - USER: ${{ github.actor }} - REPO: ${{ github.repository }} - ACCESS_API: ${{ inputs.api_key }} - ACCESS_KEY: ${{ inputs.api_secret }} - ACCESS_TOKEN: ${{ inputs.token }} - MESSAGE_API: ${{ inputs.chat_id }} - MONITOR_TOKEN: ${{ inputs.bot_token }} - WARNING_TOKEN: ${{ inputs.logs_token }} - PYPI_TOKEN: ${{ inputs.pypi_token }} - ACTOR: ${{ github.triggering_actor }} - OWNER: ${{ github.repository_owner }} - DOCKER_HUB_TOKEN: ${{ inputs.hub_token }} - DOCKER_HUB_PASSWORD: ${{ inputs.hub_password }} - GCP_CREDENTIALS: ${{ inputs.credentials }} run: | shopt -s dotglob rm -rf ${{ inputs.action_path }}/.git - mv -f ${{ inputs.action_path }}/_config.yml ${RUNNER_TEMP}/ - export PATH=$GITHUB_WORKSPACE/.github/entrypoint:$PATH && artifact.sh - - mv -f ${RUNNER_TEMP}/.env ${{ inputs.action_path }}/ - mv -f ${RUNNER_TEMP}/_config.yml ${{ inputs.action_path }}/ - - mkdir -p ${{ inputs.action_path }}/_data - mv -f ${RUNNER_TEMP}/*.json ${{ inputs.action_path }}/_data/ - mv -f ${{ inputs.action_path }} ${{ github.workspace }}/_site mkdir -p ${{ inputs.action_path }} && cp -R ${{ github.workspace }}/_site/. ${{ inputs.action_path }}/ @@ -180,19 +141,18 @@ runs: uses: devcontainers/ci@v0.3 id: set_output with: - push: always imageTag: spin-${{ env.ID }} skipContainerUserIdUpdate: true imageName: ${{ inputs.image_name }} configFile: _site/.devcontainer.json + runCmd: bash user_data/ft_client/test_client/feed.sh + push: ${{ env.RERUN_RUNNER == 'true' && 'always' || 'never' }} env: | ID=${{ env.ID }} GH_TOKEN=${{ env.GH_TOKEN }} REMOTE_REPO=${{ env.REMOTE_REPO }} - RERUN_RUNNER=${{ inputs.rerun_runner }} BEARER=${{ steps.auth.outputs.id_token }} TARGET_REPOSITORY=${{ env.TARGET_REPOSITORY }} - runCmd: bash user_data/ft_client/test_client/test_rest_client.sh - name: 🪂 Remove existing images id: variables diff --git a/.github/actions/1-addition/3-lexer/action.yml b/.github/actions/1-addition/3-lexer/action.yml index 6681327aeedb..44f33d91324c 100644 --- a/.github/actions/1-addition/3-lexer/action.yml +++ b/.github/actions/1-addition/3-lexer/action.yml @@ -81,39 +81,42 @@ runs: - name: 🪂 Adapt feed mapping shell: bash run: | - TARGET_REPO='https://github.com/${{ env.TARGET_REPOSITORY }}' - cd ${{ github.workspace }}/_site && rm -rf .git .devcontainer.json Dockerfile - git clone --single-branch -b gh-pages ${TARGET_REPO} ${{ inputs.action_path }}/_site &>/dev/null + gh variable set ORGS_JSON --body "$(cat $RUNNER_TEMP/orgs.json)" + mv -f .github/entrypoint/dockerfile/* ${{ inputs.action_path }}/ + cd ${{ github.workspace }}/_site && rm -rf .git .devcontainer.json Dockerfile* + #git clone --single-branch -b gh-pages ${REMOTE_REPO} ${{ inputs.action_path }}/_site &>/dev/null - cd ${{ inputs.action_path }}/_site - shopt -s dotglob && rm -rf ${RUNNER_TEMP}/* - [[ -d docs ]] && mv -f docs ${RUNNER_TEMP}/ - [[ -d .sass-cache ]] && mv -f .sass-cache ${RUNNER_TEMP}/ - [[ -f .jekyll-metadata ]] && mv -f .jekyll-metadata ${RUNNER_TEMP}/ + #cd ${{ inputs.action_path }}/_site + #shopt -s dotglob && rm -rf ${RUNNER_TEMP}/* + #[[ -d docs ]] && mv -f docs ${RUNNER_TEMP}/ + #[[ -d .sass-cache ]] && mv -f .sass-cache ${RUNNER_TEMP}/ + #[[ -f .jekyll-metadata ]] && mv -f .jekyll-metadata ${RUNNER_TEMP}/ - rm -rf * + #rm -rf * #git rm -rfq . #git clean -fxd - mv -f ${RUNNER_TEMP}/* . - mv -f ${{ github.workspace }}/_site/* . + #mv -f ${RUNNER_TEMP}/* . + #mv -f ${{ github.workspace }}/_site/* . + + mv -f ${{ github.workspace }}/_site ${{ inputs.action_path }}/ cat ${{ inputs.action_path }}/Dockerfile - name: 🚀 Initiate Lexer uses: devcontainers/ci@v0.3 id: artifact with: - push: always imageTag: id-${{ env.ID }} skipContainerUserIdUpdate: true - runCmd: ls -alR /home/runner/_site imageName: ${{ inputs.image_name }} + runCmd: bash user_data/ft_client/test_client/lexer.sh configFile: ${{ inputs.action_path }}/.devcontainer.json + push: ${{ env.RERUN_RUNNER == 'true' && 'always' || 'never' }} env: | - CXX=clang++ - CC=clang - FC=gfortran + ID=${{ env.ID }} + GH_TOKEN=${{ env.GH_TOKEN }} + REMOTE_REPO=${{ env.REMOTE_REPO }} + BEARER=${{ steps.auth.outputs.id_token }} TARGET_REPOSITORY=${{ env.TARGET_REPOSITORY }} - LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/lib - name: 💎 Variables id: variables @@ -122,7 +125,6 @@ runs: id: ${{ inputs.id }} TARGET_REPOSITORY: ${{ env.TARGET_REPOSITORY }} run: | - gh variable set RERUN_RUNNER --body "true" BASE_NAME=$(basename ${{ inputs.image_name }}) echo 'output_${id}='${TARGET_REPOSITORY} >> ${GITHUB_OUTPUT} IFS=', '; array=($(curl -L -s 'https://registry.hub.docker.com/v2/repositories/${{ inputs.image_name }}/tags?page_size=1024' | jq -rc '."results"[]["name"]' | yq eval -P | sed "s/ /, /g")) diff --git a/.github/actions/2-multiplication/1-maps/action.yml b/.github/actions/2-multiplication/1-maps/action.yml index dbb14e78e6c4..59cfec0a3427 100644 --- a/.github/actions/2-multiplication/1-maps/action.yml +++ b/.github/actions/2-multiplication/1-maps/action.yml @@ -1,7 +1,12 @@ name: 'Docker Deploy Action' + description: > 🪂 A Github Action to deploy pages conveniently + inputs: + id: + description: The deploy id + required: false hub_username: description: Username for Docker Hub default: ${{ github.actor }} @@ -28,6 +33,13 @@ inputs: description: Build args required: false default: "" + docker_hub_repo_short_description: + description: Short description text for the docker hub repo + required: true + docker_hub_repo_readme_file_path: + description: | + Path of the md file which will be used as the docker hub repo readme + required: true cache_src: description: Path for the docker cache source required: false @@ -36,8 +48,8 @@ inputs: description: Path for the docker cache destination required: false default: /tmp/.buildx-cache-new - rerun_runner: - description: 'Redeploy of runner' + provider: + description: 'The deploy provider' required: true owner: description: 'The deploy owner' @@ -45,6 +57,15 @@ inputs: token: description: 'The deploy token' required: false + chat_id: + description: 'The telegram chat id' + required: false + bot_token: + description: 'The monitor token' + required: false + logs_token: + description: 'The warning token' + required: false action_path: description: 'The action path' required: false @@ -52,6 +73,10 @@ inputs: description: 'The deploy repository' default: ${{ github.repository }} required: false + initiate_pauli_sum: + description: Build args + required: false + default: "" branding: icon: 'command' @@ -116,7 +141,7 @@ runs: # Ref: https://cloud.google.com/compute/docs/instances/create-start-instance - name: 💎 Remove Existing Self-Hosted Runner - #if: inputs.rerun_runner == 'true' + if: env.RERUN_RUNNER == 'true' shell: bash run: | for instance_uri in $(gcloud compute instances list --uri 2> /dev/null); do diff --git a/.github/actions/2-multiplication/2-feed/action.yml b/.github/actions/2-multiplication/2-feed/action.yml index 5d5991b690de..3b7a8131714f 100644 --- a/.github/actions/2-multiplication/2-feed/action.yml +++ b/.github/actions/2-multiplication/2-feed/action.yml @@ -110,6 +110,7 @@ runs: # See https://github.com/thesofproject/sof/blob/main/zephyr/docker-run.sh - name: 🪂 Cloud Config + if: env.RERUN_RUNNER == 'true' id: metadata shell: bash run: | @@ -132,32 +133,34 @@ runs: [Service] Type=oneshot RemainAfterExit=true - ExecStart=/sbin/iptables -A INPUT -p tcp --match multiport --dports 80,5432,6379 -j ACCEPT + ExecStart=/sbin/iptables -A INPUT -p tcp --match multiport --dports 80,5432,6379,8080 -j ACCEPT - - path: /etc/systemd/system/redis.service + - path: /etc/systemd/system/runner1.service permissions: "0644" owner: root content: | [Unit] - Description=Redis service + Description=Runner service After=docker.service firewall.service Wants=docker.service firewall.service [Service] - Restart=always + RestartSec=5s + Restart=on-failure Environment="HOME=/home/runner" - ExecStop=/usr/bin/docker exec myredis -s quit + ExecStop=/usr/bin/docker stop runner1 + ExecStopPost=/usr/bin/docker rm runner1 ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -v /mnt:/mnt --name=myredis -p 6379:6379 --pull=always eq19/maps:target-${{ github.run_number }} + ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=${{ env.RUNNER_TITLE }} -e RUNNER_REPOSITORY_URL=https://github.com/${{ env.TARGET_REPOSITORY }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -e CONTAINER_NAME=runner1 -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name runner1 --pull=always eq19/maps:target-${{ github.run_number }} - - path: /etc/systemd/system/postgres.service + - path: /etc/systemd/system/mydb.service permissions: "0644" owner: root content: | [Unit] - Description=Postgres service - After=redis.service - Wants=redis.service + Description=MyDB service + After=runner1.service + Wants=runner1.service [Service] Restart=always @@ -166,23 +169,23 @@ runs: ExecStartPre=/usr/bin/docker-credential-gcr configure-docker ExecStart=/usr/bin/docker run --rm -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -e TELEGRAM_CHAT_ID=${{ inputs.chat_id }} -e MONITOR_BOT_TOKEN=${{ inputs.bot_token }} -e WARNING_BOT_TOKEN=${{ inputs.logs_token }} -v /mnt:/mnt --memory=512M --memory-swap=1G --name=mydb -p 5432:5432 --pull=always eq19/feed:spin-${{ env.IMAGE_TAG }} - - path: /etc/systemd/system/runner.service + - path: /etc/systemd/system/runner2.service permissions: "0644" owner: root content: | [Unit] Description=gcloud compute - Wants=postgres.service - After=postgres.service + Wants=mydb.service + After=mydb.service [Service] RestartSec=5s Restart=on-failure Environment="HOME=/home/runner" - ExecStop=/usr/bin/docker stop myrunner - ExecStopPost=/usr/bin/docker rm myrunner + ExecStop=/usr/bin/docker stop runner2 + ExecStopPost=/usr/bin/docker rm runner2 ExecStartPre=/usr/bin/docker-credential-gcr configure-docker - ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=Google-optimized-instance -e RUNNER_REPOSITORY_URL=https://github.com/${{ inputs.repository }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name myrunner --pull=always eq19/lexer:id-${{ env.IMAGE_TAG }} + ExecStart=/usr/bin/docker run --rm -e RUNNER_REPLACE_EXISTING=true -e RUNNER_NAME=${{ env.RUNNER_TITLE }} -e RUNNER_REPOSITORY_URL=https://github.com/${{ inputs.repository }} -e GITHUB_ACCESS_TOKEN=${{ inputs.token }} -e CONTAINER_NAME=runner2 -v /home/runner/.docker:/home/runner/.docker -v /mnt:/mnt -v /var/log:/host/var/log -v /etc/systemd:/host/etc/systemd -v /var/run/docker.sock:/var/run/docker.sock --name runner2 --pull=always eq19/lexer:id-${{ env.IMAGE_TAG }} bootcmd: - fsck.ext4 -tvy /dev/sdb1 @@ -204,12 +207,12 @@ runs: - mount --bind /mnt/disks/deeplearning/var/lib/docker/cos /var/lib/docker - systemctl daemon-reload - systemctl start docker - - systemctl start runner.service + - systemctl start runner2.service EOL # Ref: https://stackoverflow.com/a/47018417/4058484 - name: 🪂 Setup Runner by Gcloud - #if: runner.os == 'Windows' + if: env.RERUN_RUNNER == 'true' id: instances shell: bash run: > diff --git a/.github/actions/2-multiplication/3-lexer/action.yml b/.github/actions/2-multiplication/3-lexer/action.yml index 78017617e3f1..349ff9bb87de 100644 --- a/.github/actions/2-multiplication/3-lexer/action.yml +++ b/.github/actions/2-multiplication/3-lexer/action.yml @@ -82,7 +82,7 @@ runs: using: composite steps: - name: configure - #if: runner.os == 'Windows' + if: env.RERUN_RUNNER == 'true' shell: bash env: CXX: g++-12 diff --git a/.github/actions/3-exponentiation/5-syntax/action.yml b/.github/actions/3-exponentiation/5-syntax/action.yml index cc2aa02b3bba..3014cac3e231 100644 --- a/.github/actions/3-exponentiation/5-syntax/action.yml +++ b/.github/actions/3-exponentiation/5-syntax/action.yml @@ -83,6 +83,7 @@ runs: steps: #Fill in metadata with ${{ steps.artifact.outputs.orgs_json }} - name: Run identity + if: runner.os == 'Redis' shell: bash run: | TOKEN=$(/mnt/disks/deeplearning/usr/bin/gcloud auth print-identity-token) @@ -97,6 +98,7 @@ runs: - name: 💎 Connect to Redis # Runs a script that creates a Redis client, populates # the client with data, and retrieves data + if: runner.os == 'Redis' env: # The hostname used to communicate with the Redis client service container # Ref: https://gitlab.com/NamingThingsIsHard/crypto/freqtrade/clients @@ -109,6 +111,7 @@ runs: echo '${{ inputs.pauli_sum }}' - name: 💎 Connect to postgres + if: env.RERUN_RUNNER == 'true' env: # use the internal host here because we have specified a container for the job. # If we were running the job on the VM this would be localhost diff --git a/.github/actions/3-exponentiation/6-grammar/action.yml b/.github/actions/3-exponentiation/6-grammar/action.yml index a5a75c7d1a4d..4edff2873b42 100644 --- a/.github/actions/3-exponentiation/6-grammar/action.yml +++ b/.github/actions/3-exponentiation/6-grammar/action.yml @@ -85,13 +85,9 @@ runs: run: | if [[ '${{ inputs.failure_status }}' == 'false' ]]; then - TARGET_WIKI="https://${{ github.actor }}:${{ inputs.token }}@github.com/$TARGET_REPOSITORY.wiki.git" - git clone $TARGET_WIKI ${{ inputs.runner_temp }}/wiki - cd ${{ inputs.runner_temp }}/wiki - cat _Sidebar.md - - #ls -alR /home/runner/xml - #ls -alR /home/runner/templates/doxygen/doc + TARGET_WIKI="https://${{ github.actor }}:${{ inputs.token }}@github.com/$TARGET_REPOSITORY.wiki.git" + rm -rf $RUNNER_TEMP/wiki && git clone $TARGET_WIKI $RUNNER_TEMP/wiki + cat $RUNNER_TEMP/wiki/_Sidebar.md fi @@ -110,14 +106,7 @@ runs: else - cd /home/runner/_site && rm -rf .git .github .env .ssh .cache node-modules - - cd ${{ github.workspace }} - find -not -path "./.git/*" -not -name ".git" | grep git - find -not -path "./.git/*" -not -name ".git" -delete - shopt -s dotglob && mv -f /home/runner/_site/* . - - # Assign the environment variable correctly + # Assign the environment variable correctly #COMMIT="${{ env.LATEST_COMMIT }}" # Remove double quotes diff --git a/.github/actions/4-identition/4-parser/action.yml b/.github/actions/4-identition/4-parser/action.yml index 4d6feb0a9102..2bcf1b74e06d 100644 --- a/.github/actions/4-identition/4-parser/action.yml +++ b/.github/actions/4-identition/4-parser/action.yml @@ -81,7 +81,8 @@ runs: - name: Run identity shell: bash run: | - /d/a/_actions/eq19/eq19/v2/.github/entrypoint/init.sh + drive=/$(echo "$GITHUB_WORKSPACE" | cut -c1 | tr '[:upper:]' '[:lower:]') + $drive/a/_actions/eq19/eq19/v2/.github/entrypoint/init.sh - name: View context attributes uses: actions/github-script@v7 diff --git a/.github/actions/4-identition/5-syntax/action.yml b/.github/actions/4-identition/5-syntax/action.yml index 526a7089529d..82c7545c9adc 100644 --- a/.github/actions/4-identition/5-syntax/action.yml +++ b/.github/actions/4-identition/5-syntax/action.yml @@ -103,7 +103,8 @@ runs: id_token_audience: 'https://us-central1-feedmapping.cloudfunctions.net/function' #Fill in metadata with ${{ steps.artifact.outputs.orgs_json }} - - name: Run identity + - name: Run identity + if: runner.os != 'Windows' shell: bash env: GCP_TOKEN: ${{ steps.auth.outputs.id_token }} diff --git a/.github/actions/4-identition/6-grammar/action.yml b/.github/actions/4-identition/6-grammar/action.yml index 6517000a1367..b9438038ea9c 100644 --- a/.github/actions/4-identition/6-grammar/action.yml +++ b/.github/actions/4-identition/6-grammar/action.yml @@ -90,7 +90,6 @@ runs: GH_TOKEN: ${{ inputs.token }} BASE: "D:/a/_actions/eq19/eq19/v2/.github" run: | - #gh variable set RERUN_RUNNER --body "false" if [[ '${{ inputs.failure_status }}' == 'true' ]]; then git config --global user.name "$GITHUB_ACTOR" @@ -103,11 +102,6 @@ runs: else - cd ${GITHUB_WORKSPACE//\\//} - - echo -e "\n$hr\nConfig\n$hr" - cat _config.yml - echo -e "\n$hr\nSource Pages\n$hr" rm -rf _user && ls -al . @@ -136,8 +130,66 @@ runs: echo "Processed: $MESSAGE" # Commit and push changes - git add . - git commit --allow-empty -m "$MESSAGE" - git push + git add . && git commit --allow-empty -m "$MESSAGE" && git push + + echo -e "\n$hr\nFinal Status\n$hr" + # Fetch all branches + git fetch --all + + # Check if any branches (exclude tags) with 'v' suffix exist + if ! git branch -a | grep -E 'remotes/origin/v[0-9]*$' | grep -v 'HEAD'; then + # Get default branch name (usually main or master) + set -e # Exit on error + + DEFAULT_BRANCH=$(git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@') + + git checkout $DEFAULT_BRANCH + git pull origin $DEFAULT_BRANCH + + # Delete local v0 branch if it exists + git branch -D v0 || true + + # Create new v0 branch + git checkout -b v0 + + # Update .github directory + rm -rf .github + cp -r /d/a/_actions/eq19/eq19/v2/.github . + + git add . + git commit --allow-empty -m "Update GitHub workflows and configurations" + git push origin v0 + + #Change the setting of default brance + curl -s -X PATCH \ + -H "Authorization: token $GH_TOKEN" \ + -H "Accept: application/vnd.github.v3+json" \ + https://api.github.com/repos/$TARGET_REPOSITORY \ + -d '{"default_branch":"v0"}' > /dev/null + echo "Created v0 branch and set it as default" + fi + + if [[ "$TARGET_REPOSITORY" != *"eq19/"* ]]; then + # Unpublish existing GitHub Pages + curl -s -X DELETE \ + -H "Authorization: token $GH_TOKEN" \ + -H "Accept: application/vnd.github.v3+json" \ + https://api.github.com/repos/$TARGET_REPOSITORY/pages > /dev/null + + # Disable GitHub Pages + curl -s -X PATCH \ + -H "Authorization: token $GH_TOKEN" \ + -H "Accept: application/vnd.github.v3+json" \ + https://api.github.com/repos/$TARGET_REPOSITORY/pages \ + -d '{"build_type":"legacy","source":null}' > /dev/null + + # Trigger a dispaches action + curl -s -X POST \ + -H "Authorization: token $GH_TOKEN" \ + -H "Accept: application/vnd.github.v3+json" \ + https://api.github.com/repos/$TARGET_REPOSITORY/dispatches \ + -d '{"event_type": "trigger_action", "client_payload": {"env": "prod", "version": "1.0"}}' + echo -e "\nDispatched to https://github.com/${{ env.TARGET_REPOSITORY }}/actions" + fi fi diff --git a/.github/entrypoint/artifact.sh b/.github/entrypoint/artifact.sh index d2964ec8e2d6..65f0f7819053 100755 --- a/.github/entrypoint/artifact.sh +++ b/.github/entrypoint/artifact.sh @@ -1,4 +1,5 @@ #!/usr/bin/env bash + # Structure: Cell Types – Modulo 6 # https://www.hexspin.com/proof-of-confinement/ @@ -72,6 +73,7 @@ set_target() { echo " organization: [$(cat ${RUNNER_TEMP}/user_orgs)]" >> ${RUNNER_TEMP}/_config.yml fi return $(( $SPAN + $SPIN )) + } jekyll_build() { @@ -93,11 +95,12 @@ jekyll_build() { [[ $1 != *"github.io"* ]] && sed -i "1s|^|baseurl: /$1\n|" ${RUNNER_TEMP}/_config.yml FOLDER="span$(( 19 - $SITEID ))" - TARGET_REPOSITORY=${OWNER}/$1 + export TARGET_REPOSITORY=${OWNER}/$1 gh variable set FOLDER --body "$FOLDER" + gh variable set TARGET_REPOSITORY --body "$TARGET_REPOSITORY" + echo 'FOLDER='${FOLDER} >> ${RUNNER_TEMP}/.env echo 'repo='${TARGET_REPOSITORY} >> ${GITHUB_OUTPUT} - gh variable set TARGET_REPOSITORY --body "$TARGET_REPOSITORY" echo 'TARGET_REPOSITORY='${TARGET_REPOSITORY} >> ${GITHUB_ENV} sed -i "1s|^|title: eQuantum\n|" ${RUNNER_TEMP}/_config.yml @@ -106,13 +109,66 @@ jekyll_build() { sed -i "1s|^|id: ${SITEID}\n|" ${RUNNER_TEMP}/_config.yml echo 'ID='${SITEID} >> ${GITHUB_ENV} - - echo -e "\n$hr\nCONFIG\n$hr" cat ${RUNNER_TEMP}/_config.yml - echo -e "\n$hr\nSET TOKEN\n$hr" - sync.sh ${REPO} ${TARGET_REPOSITORY} ${GH_TOKEN} - + if [[ "$(next_repo "${TARGET_REPOSITORY}")" != *"eq19/"* ]]; then + + echo -e "\nTest Module Structure:" + echo "1. ${OWNER}/maps → $(next_repo "${OWNER}/maps")" + echo "2. ${OWNER}/grammar → $(next_repo "${OWNER}/grammar")" + NEXT_REPOSITORY=$(next_repo "${OWNER}/$(yq -r '.track.pinned | .[-1]' ${RUNNER_TEMP}/_config.yml)") + echo "3. ${OWNER}/$(yq -r '.track.pinned | .[-1]' ${RUNNER_TEMP}/_config.yml) → ${NEXT_REPOSITORY}" + echo "4. ${NEXT_REPOSITORY} → $(next_repo "${NEXT_REPOSITORY}")" + + NEXT_REPOSITORY=$(next_repo "${TARGET_REPOSITORY}") + gh variable set TARGET_REPOSITORY --repo $TARGET_REPOSITORY --body "$NEXT_REPOSITORY" + + fi + +} + +# Define the next repository function using jq +next_repo() { + + local target_repo="$1" + jq -r --arg target "$target_repo" ' + ($target | split("/")) as $parts | + $parts[0] as $org | + $parts[1] as $repo | + + (map(.login) | index($org)) as $org_index | + if $org_index == null then + "Organization not found: \($org)" | halt_error(1) + else + .[$org_index] as $current_org | + (.[($org_index + 1) % length].login) as $next_org | + + if $repo == "\($org).github.io" then + "\($org)/\($current_org.key1[0])" + else + ($current_org.key1 | index($repo)) as $key1_index | + if $key1_index != null then + if ($key1_index + 1) < ($current_org.key1 | length) then + "\($org)/\($current_org.key1[$key1_index + 1])" + else + "\($org)/\($current_org.key2[0])" + end + else + ($current_org.key2 | index($repo)) as $key2_index | + if $key2_index != null then + if ($key2_index + 1) < ($current_org.key2 | length) then + "\($org)/\($current_org.key2[$key2_index + 1])" + else + "\($next_org)/\($next_org).github.io" + end + else + "Repository not found: \($repo)" | halt_error(1) + end + end + end + end + ' ${RUNNER_TEMP}/orgs.json + } # Get structure on gist files diff --git a/.github/entrypoint/dockerfile/Doxyfile b/.github/entrypoint/dockerfile/Doxyfile new file mode 100644 index 000000000000..425f17c1a37e --- /dev/null +++ b/.github/entrypoint/dockerfile/Doxyfile @@ -0,0 +1,2473 @@ +# Doxyfile 1.8.13 + +# This file describes the settings to be used by the documentation system +# doxygen (www.doxygen.org) for a project. +# +# All text after a double hash (##) is considered a comment and is placed in +# front of the TAG it is preceding. +# +# All text after a single hash (#) is considered a comment and will be ignored. +# The format is: +# TAG = value [value, ...] +# For lists, items can also be appended using: +# TAG += value [value, ...] +# Values that contain spaces should be placed between quotes (\" \"). + +#--------------------------------------------------------------------------- +# Project related configuration options +#--------------------------------------------------------------------------- + +# This tag specifies the encoding used for all characters in the config file +# that follow. The default is UTF-8 which is also the encoding used for all text +# before the first occurrence of this tag. Doxygen uses libiconv (or the iconv +# built into libc) for the transcoding. See http://www.gnu.org/software/libiconv +# for the list of possible encodings. +# The default value is: UTF-8. + +DOXYFILE_ENCODING = UTF-8 + +# The PROJECT_NAME tag is a single word (or a sequence of words surrounded by +# double-quotes, unless you are using Doxywizard) that should identify the +# project for which the documentation is generated. This name is used in the +# title of most generated pages and in a few other places. +# The default value is: My Project. + +PROJECT_NAME = "Moxygen Example" + +# The PROJECT_NUMBER tag can be used to enter a project or revision number. This +# could be handy for archiving the generated documentation or if some version +# control system is used. + +PROJECT_NUMBER = + +# Using the PROJECT_BRIEF tag one can provide an optional one line description +# for a project that appears at the top of each page and should give viewer a +# quick idea about the purpose of the project. Keep the description short. + +PROJECT_BRIEF = + +# With the PROJECT_LOGO tag one can specify a logo or an icon that is included +# in the documentation. The maximum height of the logo should not exceed 55 +# pixels and the maximum width should not exceed 200 pixels. Doxygen will copy +# the logo to the output directory. + +PROJECT_LOGO = + +# The OUTPUT_DIRECTORY tag is used to specify the (relative or absolute) path +# into which the generated documentation will be written. If a relative path is +# entered, it will be relative to the location where doxygen was started. If +# left blank the current directory will be used. + +OUTPUT_DIRECTORY = + +# If the CREATE_SUBDIRS tag is set to YES then doxygen will create 4096 sub- +# directories (in 2 levels) under the output directory of each output format and +# will distribute the generated files over these directories. Enabling this +# option can be useful when feeding doxygen a huge amount of source files, where +# putting all generated files in the same directory would otherwise causes +# performance problems for the file system. +# The default value is: NO. + +CREATE_SUBDIRS = NO + +# If the ALLOW_UNICODE_NAMES tag is set to YES, doxygen will allow non-ASCII +# characters to appear in the names of generated files. If set to NO, non-ASCII +# characters will be escaped, for example _xE3_x81_x84 will be used for Unicode +# U+3044. +# The default value is: NO. + +ALLOW_UNICODE_NAMES = NO + +# The OUTPUT_LANGUAGE tag is used to specify the language in which all +# documentation generated by doxygen is written. Doxygen will use this +# information to generate all constant output in the proper language. +# Possible values are: Afrikaans, Arabic, Armenian, Brazilian, Catalan, Chinese, +# Chinese-Traditional, Croatian, Czech, Danish, Dutch, English (United States), +# Esperanto, Farsi (Persian), Finnish, French, German, Greek, Hungarian, +# Indonesian, Italian, Japanese, Japanese-en (Japanese with English messages), +# Korean, Korean-en (Korean with English messages), Latvian, Lithuanian, +# Macedonian, Norwegian, Persian (Farsi), Polish, Portuguese, Romanian, Russian, +# Serbian, Serbian-Cyrillic, Slovak, Slovene, Spanish, Swedish, Turkish, +# Ukrainian and Vietnamese. +# The default value is: English. + +OUTPUT_LANGUAGE = English + +# If the BRIEF_MEMBER_DESC tag is set to YES, doxygen will include brief member +# descriptions after the members that are listed in the file and class +# documentation (similar to Javadoc). Set to NO to disable this. +# The default value is: YES. + +BRIEF_MEMBER_DESC = YES + +# If the REPEAT_BRIEF tag is set to YES, doxygen will prepend the brief +# description of a member or function before the detailed description +# +# Note: If both HIDE_UNDOC_MEMBERS and BRIEF_MEMBER_DESC are set to NO, the +# brief descriptions will be completely suppressed. +# The default value is: YES. + +REPEAT_BRIEF = YES + +# This tag implements a quasi-intelligent brief description abbreviator that is +# used to form the text in various listings. Each string in this list, if found +# as the leading text of the brief description, will be stripped from the text +# and the result, after processing the whole list, is used as the annotated +# text. Otherwise, the brief description is used as-is. If left blank, the +# following values are used ($name is automatically replaced with the name of +# the entity):The $name class, The $name widget, The $name file, is, provides, +# specifies, contains, represents, a, an and the. + +ABBREVIATE_BRIEF = "The $name class" \ + "The $name widget" \ + "The $name file" \ + is \ + provides \ + specifies \ + contains \ + represents \ + a \ + an \ + the + +# If the ALWAYS_DETAILED_SEC and REPEAT_BRIEF tags are both set to YES then +# doxygen will generate a detailed section even if there is only a brief +# description. +# The default value is: NO. + +ALWAYS_DETAILED_SEC = NO + +# If the INLINE_INHERITED_MEMB tag is set to YES, doxygen will show all +# inherited members of a class in the documentation of that class as if those +# members were ordinary class members. Constructors, destructors and assignment +# operators of the base classes will not be shown. +# The default value is: NO. + +INLINE_INHERITED_MEMB = NO + +# If the FULL_PATH_NAMES tag is set to YES, doxygen will prepend the full path +# before files name in the file list and in the header files. If set to NO the +# shortest path that makes the file name unique will be used +# The default value is: YES. + +FULL_PATH_NAMES = YES + +# The STRIP_FROM_PATH tag can be used to strip a user-defined part of the path. +# Stripping is only done if one of the specified strings matches the left-hand +# part of the path. The tag can be used to show relative paths in the file list. +# If left blank the directory from which doxygen is run is used as the path to +# strip. +# +# Note that you can specify absolute paths here, but also relative paths, which +# will be relative from the directory where doxygen is started. +# This tag requires that the tag FULL_PATH_NAMES is set to YES. + +STRIP_FROM_PATH = + +# The STRIP_FROM_INC_PATH tag can be used to strip a user-defined part of the +# path mentioned in the documentation of a class, which tells the reader which +# header file to include in order to use a class. If left blank only the name of +# the header file containing the class definition is used. Otherwise one should +# specify the list of include paths that are normally passed to the compiler +# using the -I flag. + +STRIP_FROM_INC_PATH = + +# If the SHORT_NAMES tag is set to YES, doxygen will generate much shorter (but +# less readable) file names. This can be useful is your file systems doesn't +# support long names like on DOS, Mac, or CD-ROM. +# The default value is: NO. + +SHORT_NAMES = NO + +# If the JAVADOC_AUTOBRIEF tag is set to YES then doxygen will interpret the +# first line (until the first dot) of a Javadoc-style comment as the brief +# description. If set to NO, the Javadoc-style will behave just like regular Qt- +# style comments (thus requiring an explicit @brief command for a brief +# description.) +# The default value is: NO. + +JAVADOC_AUTOBRIEF = NO + +# If the QT_AUTOBRIEF tag is set to YES then doxygen will interpret the first +# line (until the first dot) of a Qt-style comment as the brief description. If +# set to NO, the Qt-style will behave just like regular Qt-style comments (thus +# requiring an explicit \brief command for a brief description.) +# The default value is: NO. + +QT_AUTOBRIEF = NO + +# The MULTILINE_CPP_IS_BRIEF tag can be set to YES to make doxygen treat a +# multi-line C++ special comment block (i.e. a block of //! or /// comments) as +# a brief description. This used to be the default behavior. The new default is +# to treat a multi-line C++ comment block as a detailed description. Set this +# tag to YES if you prefer the old behavior instead. +# +# Note that setting this tag to YES also means that rational rose comments are +# not recognized any more. +# The default value is: NO. + +MULTILINE_CPP_IS_BRIEF = NO + +# If the INHERIT_DOCS tag is set to YES then an undocumented member inherits the +# documentation from any documented member that it re-implements. +# The default value is: YES. + +INHERIT_DOCS = YES + +# If the SEPARATE_MEMBER_PAGES tag is set to YES then doxygen will produce a new +# page for each member. If set to NO, the documentation of a member will be part +# of the file/class/namespace that contains it. +# The default value is: NO. + +SEPARATE_MEMBER_PAGES = NO + +# The TAB_SIZE tag can be used to set the number of spaces in a tab. Doxygen +# uses this value to replace tabs by spaces in code fragments. +# Minimum value: 1, maximum value: 16, default value: 4. + +TAB_SIZE = 4 + +# This tag can be used to specify a number of aliases that act as commands in +# the documentation. An alias has the form: +# name=value +# For example adding +# "sideeffect=@par Side Effects:\n" +# will allow you to put the command \sideeffect (or @sideeffect) in the +# documentation, which will result in a user-defined paragraph with heading +# "Side Effects:". You can put \n's in the value part of an alias to insert +# newlines. + +ALIASES = + +# This tag can be used to specify a number of word-keyword mappings (TCL only). +# A mapping has the form "name=value". For example adding "class=itcl::class" +# will allow you to use the command class in the itcl::class meaning. + +TCL_SUBST = + +# Set the OPTIMIZE_OUTPUT_FOR_C tag to YES if your project consists of C sources +# only. Doxygen will then generate output that is more tailored for C. For +# instance, some of the names that are used will be different. The list of all +# members will be omitted, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_FOR_C = NO + +# Set the OPTIMIZE_OUTPUT_JAVA tag to YES if your project consists of Java or +# Python sources only. Doxygen will then generate output that is more tailored +# for that language. For instance, namespaces will be presented as packages, +# qualified scopes will look different, etc. +# The default value is: NO. + +OPTIMIZE_OUTPUT_JAVA = NO + +# Set the OPTIMIZE_FOR_FORTRAN tag to YES if your project consists of Fortran +# sources. Doxygen will then generate output that is tailored for Fortran. +# The default value is: NO. + +OPTIMIZE_FOR_FORTRAN = NO + +# Set the OPTIMIZE_OUTPUT_VHDL tag to YES if your project consists of VHDL +# sources. Doxygen will then generate output that is tailored for VHDL. +# The default value is: NO. + +OPTIMIZE_OUTPUT_VHDL = NO + +# Doxygen selects the parser to use depending on the extension of the files it +# parses. With this tag you can assign which parser to use for a given +# extension. Doxygen has a built-in mapping, but you can override or extend it +# using this tag. The format is ext=language, where ext is a file extension, and +# language is one of the parsers supported by doxygen: IDL, Java, Javascript, +# C#, C, C++, D, PHP, Objective-C, Python, Fortran (fixed format Fortran: +# FortranFixed, free formatted Fortran: FortranFree, unknown formatted Fortran: +# Fortran. In the later case the parser tries to guess whether the code is fixed +# or free formatted code, this is the default for Fortran type files), VHDL. For +# instance to make doxygen treat .inc files as Fortran files (default is PHP), +# and .f files as C (default is Fortran), use: inc=Fortran f=C. +# +# Note: For files without extension you can use no_extension as a placeholder. +# +# Note that for custom extensions you also need to set FILE_PATTERNS otherwise +# the files are not read by doxygen. + +EXTENSION_MAPPING = + +# If the MARKDOWN_SUPPORT tag is enabled then doxygen pre-processes all comments +# according to the Markdown format, which allows for more readable +# documentation. See http://daringfireball.net/projects/markdown/ for details. +# The output of markdown processing is further processed by doxygen, so you can +# mix doxygen, HTML, and XML commands with Markdown formatting. Disable only in +# case of backward compatibilities issues. +# The default value is: YES. + +MARKDOWN_SUPPORT = YES + +# When the TOC_INCLUDE_HEADINGS tag is set to a non-zero value, all headings up +# to that level are automatically included in the table of contents, even if +# they do not have an id attribute. +# Note: This feature currently applies only to Markdown headings. +# Minimum value: 0, maximum value: 99, default value: 0. +# This tag requires that the tag MARKDOWN_SUPPORT is set to YES. + +TOC_INCLUDE_HEADINGS = 0 + +# When enabled doxygen tries to link words that correspond to documented +# classes, or namespaces to their corresponding documentation. Such a link can +# be prevented in individual cases by putting a % sign in front of the word or +# globally by setting AUTOLINK_SUPPORT to NO. +# The default value is: YES. + +AUTOLINK_SUPPORT = YES + +# If you use STL classes (i.e. std::string, std::vector, etc.) but do not want +# to include (a tag file for) the STL sources as input, then you should set this +# tag to YES in order to let doxygen match functions declarations and +# definitions whose arguments contain STL classes (e.g. func(std::string); +# versus func(std::string) {}). This also make the inheritance and collaboration +# diagrams that involve STL classes more complete and accurate. +# The default value is: NO. + +BUILTIN_STL_SUPPORT = NO + +# If you use Microsoft's C++/CLI language, you should set this option to YES to +# enable parsing support. +# The default value is: NO. + +CPP_CLI_SUPPORT = NO + +# Set the SIP_SUPPORT tag to YES if your project consists of sip (see: +# http://www.riverbankcomputing.co.uk/software/sip/intro) sources only. Doxygen +# will parse them like normal C++ but will assume all classes use public instead +# of private inheritance when no explicit protection keyword is present. +# The default value is: NO. + +SIP_SUPPORT = NO + +# For Microsoft's IDL there are propget and propput attributes to indicate +# getter and setter methods for a property. Setting this option to YES will make +# doxygen to replace the get and set methods by a property in the documentation. +# This will only work if the methods are indeed getting or setting a simple +# type. If this is not the case, or you want to show the methods anyway, you +# should set this option to NO. +# The default value is: YES. + +IDL_PROPERTY_SUPPORT = YES + +# If member grouping is used in the documentation and the DISTRIBUTE_GROUP_DOC +# tag is set to YES then doxygen will reuse the documentation of the first +# member in the group (if any) for the other members of the group. By default +# all members of a group must be documented explicitly. +# The default value is: NO. + +DISTRIBUTE_GROUP_DOC = NO + +# If one adds a struct or class to a group and this option is enabled, then also +# any nested class or struct is added to the same group. By default this option +# is disabled and one has to add nested compounds explicitly via \ingroup. +# The default value is: NO. + +GROUP_NESTED_COMPOUNDS = YES + +# Set the SUBGROUPING tag to YES to allow class member groups of the same type +# (for instance a group of public functions) to be put as a subgroup of that +# type (e.g. under the Public Functions section). Set it to NO to prevent +# subgrouping. Alternatively, this can be done per class using the +# \nosubgrouping command. +# The default value is: YES. + +SUBGROUPING = YES + +# When the INLINE_GROUPED_CLASSES tag is set to YES, classes, structs and unions +# are shown inside the group in which they are included (e.g. using \ingroup) +# instead of on a separate page (for HTML and Man pages) or section (for LaTeX +# and RTF). +# +# Note that this feature does not work in combination with +# SEPARATE_MEMBER_PAGES. +# The default value is: NO. + +INLINE_GROUPED_CLASSES = NO + +# When the INLINE_SIMPLE_STRUCTS tag is set to YES, structs, classes, and unions +# with only public data fields or simple typedef fields will be shown inline in +# the documentation of the scope in which they are defined (i.e. file, +# namespace, or group documentation), provided this scope is documented. If set +# to NO, structs, classes, and unions are shown on a separate page (for HTML and +# Man pages) or section (for LaTeX and RTF). +# The default value is: NO. + +INLINE_SIMPLE_STRUCTS = NO + +# When TYPEDEF_HIDES_STRUCT tag is enabled, a typedef of a struct, union, or +# enum is documented as struct, union, or enum with the name of the typedef. So +# typedef struct TypeS {} TypeT, will appear in the documentation as a struct +# with name TypeT. When disabled the typedef will appear as a member of a file, +# namespace, or class. And the struct will be named TypeS. This can typically be +# useful for C code in case the coding convention dictates that all compound +# types are typedef'ed and only the typedef is referenced, never the tag name. +# The default value is: NO. + +TYPEDEF_HIDES_STRUCT = NO + +# The size of the symbol lookup cache can be set using LOOKUP_CACHE_SIZE. This +# cache is used to resolve symbols given their name and scope. Since this can be +# an expensive process and often the same symbol appears multiple times in the +# code, doxygen keeps a cache of pre-resolved symbols. If the cache is too small +# doxygen will become slower. If the cache is too large, memory is wasted. The +# cache size is given by this formula: 2^(16+LOOKUP_CACHE_SIZE). The valid range +# is 0..9, the default is 0, corresponding to a cache size of 2^16=65536 +# symbols. At the end of a run doxygen will report the cache usage and suggest +# the optimal cache size from a speed point of view. +# Minimum value: 0, maximum value: 9, default value: 0. + +LOOKUP_CACHE_SIZE = 0 + +#--------------------------------------------------------------------------- +# Build related configuration options +#--------------------------------------------------------------------------- + +# If the EXTRACT_ALL tag is set to YES, doxygen will assume all entities in +# documentation are documented, even if no documentation was available. Private +# class members and static file members will be hidden unless the +# EXTRACT_PRIVATE respectively EXTRACT_STATIC tags are set to YES. +# Note: This will also disable the warnings about undocumented members that are +# normally produced when WARNINGS is set to YES. +# The default value is: NO. + +EXTRACT_ALL = NO + +# If the EXTRACT_PRIVATE tag is set to YES, all private members of a class will +# be included in the documentation. +# The default value is: NO. + +EXTRACT_PRIVATE = NO + +# If the EXTRACT_PACKAGE tag is set to YES, all members with package or internal +# scope will be included in the documentation. +# The default value is: NO. + +EXTRACT_PACKAGE = NO + +# If the EXTRACT_STATIC tag is set to YES, all static members of a file will be +# included in the documentation. +# The default value is: NO. + +EXTRACT_STATIC = NO + +# If the EXTRACT_LOCAL_CLASSES tag is set to YES, classes (and structs) defined +# locally in source files will be included in the documentation. If set to NO, +# only classes defined in header files are included. Does not have any effect +# for Java sources. +# The default value is: YES. + +EXTRACT_LOCAL_CLASSES = YES + +# This flag is only useful for Objective-C code. If set to YES, local methods, +# which are defined in the implementation section but not in the interface are +# included in the documentation. If set to NO, only methods in the interface are +# included. +# The default value is: NO. + +EXTRACT_LOCAL_METHODS = NO + +# If this flag is set to YES, the members of anonymous namespaces will be +# extracted and appear in the documentation as a namespace called +# 'anonymous_namespace{file}', where file will be replaced with the base name of +# the file that contains the anonymous namespace. By default anonymous namespace +# are hidden. +# The default value is: NO. + +EXTRACT_ANON_NSPACES = NO + +# If the HIDE_UNDOC_MEMBERS tag is set to YES, doxygen will hide all +# undocumented members inside documented classes or files. If set to NO these +# members will be included in the various overviews, but no documentation +# section is generated. This option has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_MEMBERS = NO + +# If the HIDE_UNDOC_CLASSES tag is set to YES, doxygen will hide all +# undocumented classes that are normally visible in the class hierarchy. If set +# to NO, these classes will be included in the various overviews. This option +# has no effect if EXTRACT_ALL is enabled. +# The default value is: NO. + +HIDE_UNDOC_CLASSES = NO + +# If the HIDE_FRIEND_COMPOUNDS tag is set to YES, doxygen will hide all friend +# (class|struct|union) declarations. If set to NO, these declarations will be +# included in the documentation. +# The default value is: NO. + +HIDE_FRIEND_COMPOUNDS = NO + +# If the HIDE_IN_BODY_DOCS tag is set to YES, doxygen will hide any +# documentation blocks found inside the body of a function. If set to NO, these +# blocks will be appended to the function's detailed documentation block. +# The default value is: NO. + +HIDE_IN_BODY_DOCS = NO + +# The INTERNAL_DOCS tag determines if documentation that is typed after a +# \internal command is included. If the tag is set to NO then the documentation +# will be excluded. Set it to YES to include the internal documentation. +# The default value is: NO. + +INTERNAL_DOCS = NO + +# If the CASE_SENSE_NAMES tag is set to NO then doxygen will only generate file +# names in lower-case letters. If set to YES, upper-case letters are also +# allowed. This is useful if you have classes or files whose names only differ +# in case and if your file system supports case sensitive file names. Windows +# and Mac users are advised to set this option to NO. +# The default value is: system dependent. + +CASE_SENSE_NAMES = YES + +# If the HIDE_SCOPE_NAMES tag is set to NO then doxygen will show members with +# their full class and namespace scopes in the documentation. If set to YES, the +# scope will be hidden. +# The default value is: NO. + +HIDE_SCOPE_NAMES = NO + +# If the HIDE_COMPOUND_REFERENCE tag is set to NO (default) then doxygen will +# append additional text to a page's title, such as Class Reference. If set to +# YES the compound reference will be hidden. +# The default value is: NO. + +HIDE_COMPOUND_REFERENCE= NO + +# If the SHOW_INCLUDE_FILES tag is set to YES then doxygen will put a list of +# the files that are included by a file in the documentation of that file. +# The default value is: YES. + +SHOW_INCLUDE_FILES = YES + +# If the SHOW_GROUPED_MEMB_INC tag is set to YES then Doxygen will add for each +# grouped member an include statement to the documentation, telling the reader +# which file to include in order to use the member. +# The default value is: NO. + +SHOW_GROUPED_MEMB_INC = NO + +# If the FORCE_LOCAL_INCLUDES tag is set to YES then doxygen will list include +# files with double quotes in the documentation rather than with sharp brackets. +# The default value is: NO. + +FORCE_LOCAL_INCLUDES = NO + +# If the INLINE_INFO tag is set to YES then a tag [inline] is inserted in the +# documentation for inline members. +# The default value is: YES. + +INLINE_INFO = YES + +# If the SORT_MEMBER_DOCS tag is set to YES then doxygen will sort the +# (detailed) documentation of file and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. +# The default value is: YES. + +SORT_MEMBER_DOCS = YES + +# If the SORT_BRIEF_DOCS tag is set to YES then doxygen will sort the brief +# descriptions of file, namespace and class members alphabetically by member +# name. If set to NO, the members will appear in declaration order. Note that +# this will also influence the order of the classes in the class list. +# The default value is: NO. + +SORT_BRIEF_DOCS = NO + +# If the SORT_MEMBERS_CTORS_1ST tag is set to YES then doxygen will sort the +# (brief and detailed) documentation of class members so that constructors and +# destructors are listed first. If set to NO the constructors will appear in the +# respective orders defined by SORT_BRIEF_DOCS and SORT_MEMBER_DOCS. +# Note: If SORT_BRIEF_DOCS is set to NO this option is ignored for sorting brief +# member documentation. +# Note: If SORT_MEMBER_DOCS is set to NO this option is ignored for sorting +# detailed member documentation. +# The default value is: NO. + +SORT_MEMBERS_CTORS_1ST = NO + +# If the SORT_GROUP_NAMES tag is set to YES then doxygen will sort the hierarchy +# of group names into alphabetical order. If set to NO the group names will +# appear in their defined order. +# The default value is: NO. + +SORT_GROUP_NAMES = NO + +# If the SORT_BY_SCOPE_NAME tag is set to YES, the class list will be sorted by +# fully-qualified names, including namespaces. If set to NO, the class list will +# be sorted only by class name, not including the namespace part. +# Note: This option is not very useful if HIDE_SCOPE_NAMES is set to YES. +# Note: This option applies only to the class list, not to the alphabetical +# list. +# The default value is: NO. + +SORT_BY_SCOPE_NAME = NO + +# If the STRICT_PROTO_MATCHING option is enabled and doxygen fails to do proper +# type resolution of all parameters of a function it will reject a match between +# the prototype and the implementation of a member function even if there is +# only one candidate or it is obvious which candidate to choose by doing a +# simple string match. By disabling STRICT_PROTO_MATCHING doxygen will still +# accept a match between prototype and implementation in such cases. +# The default value is: NO. + +STRICT_PROTO_MATCHING = NO + +# The GENERATE_TODOLIST tag can be used to enable (YES) or disable (NO) the todo +# list. This list is created by putting \todo commands in the documentation. +# The default value is: YES. + +GENERATE_TODOLIST = YES + +# The GENERATE_TESTLIST tag can be used to enable (YES) or disable (NO) the test +# list. This list is created by putting \test commands in the documentation. +# The default value is: YES. + +GENERATE_TESTLIST = YES + +# The GENERATE_BUGLIST tag can be used to enable (YES) or disable (NO) the bug +# list. This list is created by putting \bug commands in the documentation. +# The default value is: YES. + +GENERATE_BUGLIST = YES + +# The GENERATE_DEPRECATEDLIST tag can be used to enable (YES) or disable (NO) +# the deprecated list. This list is created by putting \deprecated commands in +# the documentation. +# The default value is: YES. + +GENERATE_DEPRECATEDLIST= YES + +# The ENABLED_SECTIONS tag can be used to enable conditional documentation +# sections, marked by \if ... \endif and \cond +# ... \endcond blocks. + +ENABLED_SECTIONS = + +# The MAX_INITIALIZER_LINES tag determines the maximum number of lines that the +# initial value of a variable or macro / define can have for it to appear in the +# documentation. If the initializer consists of more lines than specified here +# it will be hidden. Use a value of 0 to hide initializers completely. The +# appearance of the value of individual variables and macros / defines can be +# controlled using \showinitializer or \hideinitializer command in the +# documentation regardless of this setting. +# Minimum value: 0, maximum value: 10000, default value: 30. + +MAX_INITIALIZER_LINES = 30 + +# Set the SHOW_USED_FILES tag to NO to disable the list of files generated at +# the bottom of the documentation of classes and structs. If set to YES, the +# list will mention the files that were used to generate the documentation. +# The default value is: YES. + +SHOW_USED_FILES = YES + +# Set the SHOW_FILES tag to NO to disable the generation of the Files page. This +# will remove the Files entry from the Quick Index and from the Folder Tree View +# (if specified). +# The default value is: YES. + +SHOW_FILES = YES + +# Set the SHOW_NAMESPACES tag to NO to disable the generation of the Namespaces +# page. This will remove the Namespaces entry from the Quick Index and from the +# Folder Tree View (if specified). +# The default value is: YES. + +SHOW_NAMESPACES = YES + +# The FILE_VERSION_FILTER tag can be used to specify a program or script that +# doxygen should invoke to get the current version for each file (typically from +# the version control system). Doxygen will invoke the program by executing (via +# popen()) the command command input-file, where command is the value of the +# FILE_VERSION_FILTER tag, and input-file is the name of an input file provided +# by doxygen. Whatever the program writes to standard output is used as the file +# version. For an example see the documentation. + +FILE_VERSION_FILTER = + +# The LAYOUT_FILE tag can be used to specify a layout file which will be parsed +# by doxygen. The layout file controls the global structure of the generated +# output files in an output format independent way. To create the layout file +# that represents doxygen's defaults, run doxygen with the -l option. You can +# optionally specify a file name after the option, if omitted DoxygenLayout.xml +# will be used as the name of the layout file. +# +# Note that if you run doxygen from a directory containing a file called +# DoxygenLayout.xml, doxygen will parse it automatically even if the LAYOUT_FILE +# tag is left empty. + +LAYOUT_FILE = + +# The CITE_BIB_FILES tag can be used to specify one or more bib files containing +# the reference definitions. This must be a list of .bib files. The .bib +# extension is automatically appended if omitted. This requires the bibtex tool +# to be installed. See also http://en.wikipedia.org/wiki/BibTeX for more info. +# For LaTeX the style of the bibliography can be controlled using +# LATEX_BIB_STYLE. To use this feature you need bibtex and perl available in the +# search path. See also \cite for info how to create references. + +CITE_BIB_FILES = + +#--------------------------------------------------------------------------- +# Configuration options related to warning and progress messages +#--------------------------------------------------------------------------- + +# The QUIET tag can be used to turn on/off the messages that are generated to +# standard output by doxygen. If QUIET is set to YES this implies that the +# messages are off. +# The default value is: NO. + +QUIET = NO + +# The WARNINGS tag can be used to turn on/off the warning messages that are +# generated to standard error (stderr) by doxygen. If WARNINGS is set to YES +# this implies that the warnings are on. +# +# Tip: Turn warnings on while writing the documentation. +# The default value is: YES. + +WARNINGS = YES + +# If the WARN_IF_UNDOCUMENTED tag is set to YES then doxygen will generate +# warnings for undocumented members. If EXTRACT_ALL is set to YES then this flag +# will automatically be disabled. +# The default value is: YES. + +WARN_IF_UNDOCUMENTED = YES + +# If the WARN_IF_DOC_ERROR tag is set to YES, doxygen will generate warnings for +# potential errors in the documentation, such as not documenting some parameters +# in a documented function, or documenting parameters that don't exist or using +# markup commands wrongly. +# The default value is: YES. + +WARN_IF_DOC_ERROR = YES + +# This WARN_NO_PARAMDOC option can be enabled to get warnings for functions that +# are documented, but have no documentation for their parameters or return +# value. If set to NO, doxygen will only warn about wrong or incomplete +# parameter documentation, but not about the absence of documentation. +# The default value is: NO. + +WARN_NO_PARAMDOC = NO + +# If the WARN_AS_ERROR tag is set to YES then doxygen will immediately stop when +# a warning is encountered. +# The default value is: NO. + +WARN_AS_ERROR = NO + +# The WARN_FORMAT tag determines the format of the warning messages that doxygen +# can produce. The string should contain the $file, $line, and $text tags, which +# will be replaced by the file and line number from which the warning originated +# and the warning text. Optionally the format may contain $version, which will +# be replaced by the version of the file (if it could be obtained via +# FILE_VERSION_FILTER) +# The default value is: $file:$line: $text. + +WARN_FORMAT = "$file:$line: $text" + +# The WARN_LOGFILE tag can be used to specify a file to which warning and error +# messages should be written. If left blank the output is written to standard +# error (stderr). + +WARN_LOGFILE = + +#--------------------------------------------------------------------------- +# Configuration options related to the input files +#--------------------------------------------------------------------------- + +# The INPUT tag is used to specify the files and/or directories that contain +# documented source files. You may enter file names like myfile.cpp or +# directories like /usr/src/myproject. Separate the files or directories with +# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING +# Note: If this tag is empty the current directory is searched. + +INPUT = ./src + +# This tag can be used to specify the character encoding of the source files +# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses +# libiconv (or the iconv built into libc) for the transcoding. See the libiconv +# documentation (see: http://www.gnu.org/software/libiconv) for the list of +# possible encodings. +# The default value is: UTF-8. + +INPUT_ENCODING = UTF-8 + +# If the value of the INPUT tag contains directories, you can use the +# FILE_PATTERNS tag to specify one or more wildcard patterns (like *.cpp and +# *.h) to filter out the source-files in the directories. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# read by doxygen. +# +# If left blank the following patterns are tested:*.c, *.cc, *.cxx, *.cpp, +# *.c++, *.java, *.ii, *.ixx, *.ipp, *.i++, *.inl, *.idl, *.ddl, *.odl, *.h, +# *.hh, *.hxx, *.hpp, *.h++, *.cs, *.d, *.php, *.php4, *.php5, *.phtml, *.inc, +# *.m, *.markdown, *.md, *.mm, *.dox, *.py, *.pyw, *.f90, *.f95, *.f03, *.f08, +# *.f, *.for, *.tcl, *.vhd, *.vhdl, *.ucf and *.qsf. + +FILE_PATTERNS = *.c \ + *.cc \ + *.cxx \ + *.cpp \ + *.c++ \ + *.java \ + *.ii \ + *.ixx \ + *.ipp \ + *.i++ \ + *.inl \ + *.idl \ + *.ddl \ + *.odl \ + *.h \ + *.hh \ + *.hxx \ + *.hpp \ + *.h++ \ + *.cs \ + *.d \ + *.php \ + *.php4 \ + *.php5 \ + *.phtml \ + *.inc \ + *.m \ + *.markdown \ + *.md \ + *.mm \ + *.dox \ + *.py \ + *.pyw \ + *.f90 \ + *.f95 \ + *.f03 \ + *.f08 \ + *.f \ + *.for \ + *.tcl \ + *.vhd \ + *.vhdl \ + *.ucf \ + *.qsf + +# The RECURSIVE tag can be used to specify whether or not subdirectories should +# be searched for input files as well. +# The default value is: NO. + +RECURSIVE = YES + +# The EXCLUDE tag can be used to specify files and/or directories that should be +# excluded from the INPUT source files. This way you can easily exclude a +# subdirectory from a directory tree whose root is specified with the INPUT tag. +# +# Note that relative paths are relative to the directory from which doxygen is +# run. + +EXCLUDE = + +# The EXCLUDE_SYMLINKS tag can be used to select whether or not files or +# directories that are symbolic links (a Unix file system feature) are excluded +# from the input. +# The default value is: NO. + +EXCLUDE_SYMLINKS = NO + +# If the value of the INPUT tag contains directories, you can use the +# EXCLUDE_PATTERNS tag to specify one or more wildcard patterns to exclude +# certain files from those directories. +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories for example use the pattern */test/* + +EXCLUDE_PATTERNS = */test/* + +# The EXCLUDE_SYMBOLS tag can be used to specify one or more symbol names +# (namespaces, classes, functions, etc.) that should be excluded from the +# output. The symbol name can be a fully qualified name, a word, or if the +# wildcard * is used, a substring. Examples: ANamespace, AClass, +# AClass::ANamespace, ANamespace::*Test +# +# Note that the wildcards are matched against the file with absolute path, so to +# exclude all test directories use the pattern */test/* + +EXCLUDE_SYMBOLS = *internal* + +# The EXAMPLE_PATH tag can be used to specify one or more files or directories +# that contain example code fragments that are included (see the \include +# command). + +EXAMPLE_PATH = + +# If the value of the EXAMPLE_PATH tag contains directories, you can use the +# EXAMPLE_PATTERNS tag to specify one or more wildcard pattern (like *.cpp and +# *.h) to filter out the source-files in the directories. If left blank all +# files are included. + +EXAMPLE_PATTERNS = * + +# If the EXAMPLE_RECURSIVE tag is set to YES then subdirectories will be +# searched for input files to be used with the \include or \dontinclude commands +# irrespective of the value of the RECURSIVE tag. +# The default value is: NO. + +EXAMPLE_RECURSIVE = NO + +# The IMAGE_PATH tag can be used to specify one or more files or directories +# that contain images that are to be included in the documentation (see the +# \image command). + +IMAGE_PATH = + +# The INPUT_FILTER tag can be used to specify a program that doxygen should +# invoke to filter for each input file. Doxygen will invoke the filter program +# by executing (via popen()) the command: +# +# +# +# where is the value of the INPUT_FILTER tag, and is the +# name of an input file. Doxygen will then use the output that the filter +# program writes to standard output. If FILTER_PATTERNS is specified, this tag +# will be ignored. +# +# Note that the filter must not add or remove lines; it is applied before the +# code is scanned, but not when the output code is generated. If lines are added +# or removed, the anchors will not be placed correctly. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +INPUT_FILTER = + +# The FILTER_PATTERNS tag can be used to specify filters on a per file pattern +# basis. Doxygen will compare the file name with each pattern and apply the +# filter if there is a match. The filters are a list of the form: pattern=filter +# (like *.cpp=my_cpp_filter). See INPUT_FILTER for further information on how +# filters are used. If the FILTER_PATTERNS tag is empty or if none of the +# patterns match the file name, INPUT_FILTER is applied. +# +# Note that for custom extensions or not directly supported extensions you also +# need to set EXTENSION_MAPPING for the extension otherwise the files are not +# properly processed by doxygen. + +FILTER_PATTERNS = + +# If the FILTER_SOURCE_FILES tag is set to YES, the input filter (if set using +# INPUT_FILTER) will also be used to filter the input files that are used for +# producing the source files to browse (i.e. when SOURCE_BROWSER is set to YES). +# The default value is: NO. + +FILTER_SOURCE_FILES = NO + +# The FILTER_SOURCE_PATTERNS tag can be used to specify source filters per file +# pattern. A pattern will override the setting for FILTER_PATTERN (if any) and +# it is also possible to disable source filtering for a specific pattern using +# *.ext= (so without naming a filter). +# This tag requires that the tag FILTER_SOURCE_FILES is set to YES. + +FILTER_SOURCE_PATTERNS = + +# If the USE_MDFILE_AS_MAINPAGE tag refers to the name of a markdown file that +# is part of the input, its contents will be placed on the main page +# (index.html). This can be useful if you have a project on for instance GitHub +# and want to reuse the introduction page also for the doxygen output. + +USE_MDFILE_AS_MAINPAGE = + +#--------------------------------------------------------------------------- +# Configuration options related to source browsing +#--------------------------------------------------------------------------- + +# If the SOURCE_BROWSER tag is set to YES then a list of source files will be +# generated. Documented entities will be cross-referenced with these sources. +# +# Note: To get rid of all source code in the generated output, make sure that +# also VERBATIM_HEADERS is set to NO. +# The default value is: NO. + +SOURCE_BROWSER = NO + +# Setting the INLINE_SOURCES tag to YES will include the body of functions, +# classes and enums directly into the documentation. +# The default value is: NO. + +INLINE_SOURCES = NO + +# Setting the STRIP_CODE_COMMENTS tag to YES will instruct doxygen to hide any +# special comment blocks from generated source code fragments. Normal C, C++ and +# Fortran comments will always remain visible. +# The default value is: YES. + +STRIP_CODE_COMMENTS = YES + +# If the REFERENCED_BY_RELATION tag is set to YES then for each documented +# function all documented functions referencing it will be listed. +# The default value is: NO. + +REFERENCED_BY_RELATION = NO + +# If the REFERENCES_RELATION tag is set to YES then for each documented function +# all documented entities called/used by that function will be listed. +# The default value is: NO. + +REFERENCES_RELATION = NO + +# If the REFERENCES_LINK_SOURCE tag is set to YES and SOURCE_BROWSER tag is set +# to YES then the hyperlinks from functions in REFERENCES_RELATION and +# REFERENCED_BY_RELATION lists will link to the source code. Otherwise they will +# link to the documentation. +# The default value is: YES. + +REFERENCES_LINK_SOURCE = YES + +# If SOURCE_TOOLTIPS is enabled (the default) then hovering a hyperlink in the +# source code will show a tooltip with additional information such as prototype, +# brief description and links to the definition and documentation. Since this +# will make the HTML file larger and loading of large files a bit slower, you +# can opt to disable this feature. +# The default value is: YES. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +SOURCE_TOOLTIPS = YES + +# If the USE_HTAGS tag is set to YES then the references to source code will +# point to the HTML generated by the htags(1) tool instead of doxygen built-in +# source browser. The htags tool is part of GNU's global source tagging system +# (see http://www.gnu.org/software/global/global.html). You will need version +# 4.8.6 or higher. +# +# To use it do the following: +# - Install the latest version of global +# - Enable SOURCE_BROWSER and USE_HTAGS in the config file +# - Make sure the INPUT points to the root of the source tree +# - Run doxygen as normal +# +# Doxygen will invoke htags (and that will in turn invoke gtags), so these +# tools must be available from the command line (i.e. in the search path). +# +# The result: instead of the source browser generated by doxygen, the links to +# source code will now point to the output of htags. +# The default value is: NO. +# This tag requires that the tag SOURCE_BROWSER is set to YES. + +USE_HTAGS = NO + +# If the VERBATIM_HEADERS tag is set the YES then doxygen will generate a +# verbatim copy of the header file for each class for which an include is +# specified. Set to NO to disable this. +# See also: Section \class. +# The default value is: YES. + +VERBATIM_HEADERS = YES + +#--------------------------------------------------------------------------- +# Configuration options related to the alphabetical class index +#--------------------------------------------------------------------------- + +# If the ALPHABETICAL_INDEX tag is set to YES, an alphabetical index of all +# compounds will be generated. Enable this if the project contains a lot of +# classes, structs, unions or interfaces. +# The default value is: YES. + +ALPHABETICAL_INDEX = YES + +# The COLS_IN_ALPHA_INDEX tag can be used to specify the number of columns in +# which the alphabetical index list will be split. +# Minimum value: 1, maximum value: 20, default value: 5. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +COLS_IN_ALPHA_INDEX = 5 + +# In case all classes in a project start with a common prefix, all classes will +# be put under the same header in the alphabetical index. The IGNORE_PREFIX tag +# can be used to specify a prefix (or a list of prefixes) that should be ignored +# while generating the index headers. +# This tag requires that the tag ALPHABETICAL_INDEX is set to YES. + +IGNORE_PREFIX = + +#--------------------------------------------------------------------------- +# Configuration options related to the HTML output +#--------------------------------------------------------------------------- + +# If the GENERATE_HTML tag is set to YES, doxygen will generate HTML output +# The default value is: YES. + +GENERATE_HTML = NO + +# The HTML_OUTPUT tag is used to specify where the HTML docs will be put. If a +# relative path is entered the value of OUTPUT_DIRECTORY will be put in front of +# it. +# The default directory is: html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_OUTPUT = html + +# The HTML_FILE_EXTENSION tag can be used to specify the file extension for each +# generated HTML page (for example: .htm, .php, .asp). +# The default value is: .html. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FILE_EXTENSION = .html + +# The HTML_HEADER tag can be used to specify a user-defined HTML header file for +# each generated HTML page. If the tag is left blank doxygen will generate a +# standard header. +# +# To get valid HTML the header file that includes any scripts and style sheets +# that doxygen needs, which is dependent on the configuration options used (e.g. +# the setting GENERATE_TREEVIEW). It is highly recommended to start with a +# default header using +# doxygen -w html new_header.html new_footer.html new_stylesheet.css +# YourConfigFile +# and then modify the file new_header.html. See also section "Doxygen usage" +# for information on how to generate the default header that doxygen normally +# uses. +# Note: The header is subject to change so you typically have to regenerate the +# default header when upgrading to a newer version of doxygen. For a description +# of the possible markers and block names see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_HEADER = + +# The HTML_FOOTER tag can be used to specify a user-defined HTML footer for each +# generated HTML page. If the tag is left blank doxygen will generate a standard +# footer. See HTML_HEADER for more information on how to generate a default +# footer and what special commands can be used inside the footer. See also +# section "Doxygen usage" for information on how to generate the default footer +# that doxygen normally uses. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_FOOTER = + +# The HTML_STYLESHEET tag can be used to specify a user-defined cascading style +# sheet that is used by each HTML page. It can be used to fine-tune the look of +# the HTML output. If left blank doxygen will generate a default style sheet. +# See also section "Doxygen usage" for information on how to generate the style +# sheet that doxygen normally uses. +# Note: It is recommended to use HTML_EXTRA_STYLESHEET instead of this tag, as +# it is more robust and this tag (HTML_STYLESHEET) will in the future become +# obsolete. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_STYLESHEET = + +# The HTML_EXTRA_STYLESHEET tag can be used to specify additional user-defined +# cascading style sheets that are included after the standard style sheets +# created by doxygen. Using this option one can overrule certain style aspects. +# This is preferred over using HTML_STYLESHEET since it does not replace the +# standard style sheet and is therefore more robust against future updates. +# Doxygen will copy the style sheet files to the output directory. +# Note: The order of the extra style sheet files is of importance (e.g. the last +# style sheet in the list overrules the setting of the previous ones in the +# list). For an example see the documentation. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_STYLESHEET = + +# The HTML_EXTRA_FILES tag can be used to specify one or more extra images or +# other source files which should be copied to the HTML output directory. Note +# that these files will be copied to the base HTML output directory. Use the +# $relpath^ marker in the HTML_HEADER and/or HTML_FOOTER files to load these +# files. In the HTML_STYLESHEET file, use the file name only. Also note that the +# files will be copied as-is; there are no commands or markers available. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_EXTRA_FILES = + +# The HTML_COLORSTYLE_HUE tag controls the color of the HTML output. Doxygen +# will adjust the colors in the style sheet and background images according to +# this color. Hue is specified as an angle on a colorwheel, see +# http://en.wikipedia.org/wiki/Hue for more information. For instance the value +# 0 represents red, 60 is yellow, 120 is green, 180 is cyan, 240 is blue, 300 +# purple, and 360 is red again. +# Minimum value: 0, maximum value: 359, default value: 220. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_HUE = 220 + +# The HTML_COLORSTYLE_SAT tag controls the purity (or saturation) of the colors +# in the HTML output. For a value of 0 the output will use grayscales only. A +# value of 255 will produce the most vivid colors. +# Minimum value: 0, maximum value: 255, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_SAT = 100 + +# The HTML_COLORSTYLE_GAMMA tag controls the gamma correction applied to the +# luminance component of the colors in the HTML output. Values below 100 +# gradually make the output lighter, whereas values above 100 make the output +# darker. The value divided by 100 is the actual gamma applied, so 80 represents +# a gamma of 0.8, The value 220 represents a gamma of 2.2, and 100 does not +# change the gamma. +# Minimum value: 40, maximum value: 240, default value: 80. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_COLORSTYLE_GAMMA = 80 + +# If the HTML_TIMESTAMP tag is set to YES then the footer of each generated HTML +# page will contain the date and time when the page was generated. Setting this +# to YES can help to show when doxygen was last run and thus if the +# documentation is up to date. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_TIMESTAMP = NO + +# If the HTML_DYNAMIC_SECTIONS tag is set to YES then the generated HTML +# documentation will contain sections that can be hidden and shown after the +# page has loaded. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_DYNAMIC_SECTIONS = NO + +# With HTML_INDEX_NUM_ENTRIES one can control the preferred number of entries +# shown in the various tree structured indices initially; the user can expand +# and collapse entries dynamically later on. Doxygen will expand the tree to +# such a level that at most the specified number of entries are visible (unless +# a fully collapsed tree already exceeds this amount). So setting the number of +# entries 1 will produce a full collapsed tree by default. 0 is a special value +# representing an infinite number of entries and will result in a full expanded +# tree by default. +# Minimum value: 0, maximum value: 9999, default value: 100. +# This tag requires that the tag GENERATE_HTML is set to YES. + +HTML_INDEX_NUM_ENTRIES = 100 + +# If the GENERATE_DOCSET tag is set to YES, additional index files will be +# generated that can be used as input for Apple's Xcode 3 integrated development +# environment (see: http://developer.apple.com/tools/xcode/), introduced with +# OSX 10.5 (Leopard). To create a documentation set, doxygen will generate a +# Makefile in the HTML output directory. Running make will produce the docset in +# that directory and running make install will install the docset in +# ~/Library/Developer/Shared/Documentation/DocSets so that Xcode will find it at +# startup. See http://developer.apple.com/tools/creatingdocsetswithdoxygen.html +# for more information. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_DOCSET = NO + +# This tag determines the name of the docset feed. A documentation feed provides +# an umbrella under which multiple documentation sets from a single provider +# (such as a company or product suite) can be grouped. +# The default value is: Doxygen generated docs. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_FEEDNAME = "Doxygen generated docs" + +# This tag specifies a string that should uniquely identify the documentation +# set bundle. This should be a reverse domain-name style string, e.g. +# com.mycompany.MyDocSet. Doxygen will append .docset to the name. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_BUNDLE_ID = org.doxygen.Project + +# The DOCSET_PUBLISHER_ID tag specifies a string that should uniquely identify +# the documentation publisher. This should be a reverse domain-name style +# string, e.g. com.mycompany.MyDocSet.documentation. +# The default value is: org.doxygen.Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_ID = org.doxygen.Publisher + +# The DOCSET_PUBLISHER_NAME tag identifies the documentation publisher. +# The default value is: Publisher. +# This tag requires that the tag GENERATE_DOCSET is set to YES. + +DOCSET_PUBLISHER_NAME = Publisher + +# If the GENERATE_HTMLHELP tag is set to YES then doxygen generates three +# additional HTML index files: index.hhp, index.hhc, and index.hhk. The +# index.hhp is a project file that can be read by Microsoft's HTML Help Workshop +# (see: http://www.microsoft.com/en-us/download/details.aspx?id=21138) on +# Windows. +# +# The HTML Help Workshop contains a compiler that can convert all HTML output +# generated by doxygen into a single compiled HTML file (.chm). Compiled HTML +# files are now used as the Windows 98 help format, and will replace the old +# Windows help format (.hlp) on all Windows platforms in the future. Compressed +# HTML files also contain an index, a table of contents, and you can search for +# words in the documentation. The HTML workshop also contains a viewer for +# compressed HTML files. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_HTMLHELP = NO + +# The CHM_FILE tag can be used to specify the file name of the resulting .chm +# file. You can add a path in front of the file if the result should not be +# written to the html output directory. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_FILE = + +# The HHC_LOCATION tag can be used to specify the location (absolute path +# including file name) of the HTML help compiler (hhc.exe). If non-empty, +# doxygen will try to run the HTML help compiler on the generated index.hhp. +# The file has to be specified with full path. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +HHC_LOCATION = + +# The GENERATE_CHI flag controls if a separate .chi index file is generated +# (YES) or that it should be included in the master .chm file (NO). +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +GENERATE_CHI = NO + +# The CHM_INDEX_ENCODING is used to encode HtmlHelp index (hhk), content (hhc) +# and project file content. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +CHM_INDEX_ENCODING = + +# The BINARY_TOC flag controls whether a binary table of contents is generated +# (YES) or a normal table of contents (NO) in the .chm file. Furthermore it +# enables the Previous and Next buttons. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +BINARY_TOC = NO + +# The TOC_EXPAND flag can be set to YES to add extra items for group members to +# the table of contents of the HTML help documentation and to the tree view. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTMLHELP is set to YES. + +TOC_EXPAND = NO + +# If the GENERATE_QHP tag is set to YES and both QHP_NAMESPACE and +# QHP_VIRTUAL_FOLDER are set, an additional index file will be generated that +# can be used as input for Qt's qhelpgenerator to generate a Qt Compressed Help +# (.qch) of the generated HTML documentation. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_QHP = NO + +# If the QHG_LOCATION tag is specified, the QCH_FILE tag can be used to specify +# the file name of the resulting .qch file. The path specified is relative to +# the HTML output folder. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QCH_FILE = + +# The QHP_NAMESPACE tag specifies the namespace to use when generating Qt Help +# Project output. For more information please see Qt Help Project / Namespace +# (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#namespace). +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_NAMESPACE = org.doxygen.Project + +# The QHP_VIRTUAL_FOLDER tag specifies the namespace to use when generating Qt +# Help Project output. For more information please see Qt Help Project / Virtual +# Folders (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#virtual- +# folders). +# The default value is: doc. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_VIRTUAL_FOLDER = doc + +# If the QHP_CUST_FILTER_NAME tag is set, it specifies the name of a custom +# filter to add. For more information please see Qt Help Project / Custom +# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_NAME = + +# The QHP_CUST_FILTER_ATTRS tag specifies the list of the attributes of the +# custom filter to add. For more information please see Qt Help Project / Custom +# Filters (see: http://qt-project.org/doc/qt-4.8/qthelpproject.html#custom- +# filters). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_CUST_FILTER_ATTRS = + +# The QHP_SECT_FILTER_ATTRS tag specifies the list of the attributes this +# project's filter section matches. Qt Help Project / Filter Attributes (see: +# http://qt-project.org/doc/qt-4.8/qthelpproject.html#filter-attributes). +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHP_SECT_FILTER_ATTRS = + +# The QHG_LOCATION tag can be used to specify the location of Qt's +# qhelpgenerator. If non-empty doxygen will try to run qhelpgenerator on the +# generated .qhp file. +# This tag requires that the tag GENERATE_QHP is set to YES. + +QHG_LOCATION = + +# If the GENERATE_ECLIPSEHELP tag is set to YES, additional index files will be +# generated, together with the HTML files, they form an Eclipse help plugin. To +# install this plugin and make it available under the help contents menu in +# Eclipse, the contents of the directory containing the HTML and XML files needs +# to be copied into the plugins directory of eclipse. The name of the directory +# within the plugins directory should be the same as the ECLIPSE_DOC_ID value. +# After copying Eclipse needs to be restarted before the help appears. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_ECLIPSEHELP = NO + +# A unique identifier for the Eclipse help plugin. When installing the plugin +# the directory name containing the HTML and XML files should also have this +# name. Each documentation set should have its own identifier. +# The default value is: org.doxygen.Project. +# This tag requires that the tag GENERATE_ECLIPSEHELP is set to YES. + +ECLIPSE_DOC_ID = org.doxygen.Project + +# If you want full control over the layout of the generated HTML pages it might +# be necessary to disable the index and replace it with your own. The +# DISABLE_INDEX tag can be used to turn on/off the condensed index (tabs) at top +# of each HTML page. A value of NO enables the index and the value YES disables +# it. Since the tabs in the index contain the same information as the navigation +# tree, you can set this option to YES if you also set GENERATE_TREEVIEW to YES. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +DISABLE_INDEX = NO + +# The GENERATE_TREEVIEW tag is used to specify whether a tree-like index +# structure should be generated to display hierarchical information. If the tag +# value is set to YES, a side panel will be generated containing a tree-like +# index structure (just like the one that is generated for HTML Help). For this +# to work a browser that supports JavaScript, DHTML, CSS and frames is required +# (i.e. any modern browser). Windows users are probably better off using the +# HTML help feature. Via custom style sheets (see HTML_EXTRA_STYLESHEET) one can +# further fine-tune the look of the index. As an example, the default style +# sheet generated by doxygen has an example that shows how to put an image at +# the root of the tree instead of the PROJECT_NAME. Since the tree basically has +# the same information as the tab index, you could consider setting +# DISABLE_INDEX to YES when enabling this option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +GENERATE_TREEVIEW = NO + +# The ENUM_VALUES_PER_LINE tag can be used to set the number of enum values that +# doxygen will group on one line in the generated HTML documentation. +# +# Note that a value of 0 will completely suppress the enum values from appearing +# in the overview section. +# Minimum value: 0, maximum value: 20, default value: 4. +# This tag requires that the tag GENERATE_HTML is set to YES. + +ENUM_VALUES_PER_LINE = 4 + +# If the treeview is enabled (see GENERATE_TREEVIEW) then this tag can be used +# to set the initial width (in pixels) of the frame in which the tree is shown. +# Minimum value: 0, maximum value: 1500, default value: 250. +# This tag requires that the tag GENERATE_HTML is set to YES. + +TREEVIEW_WIDTH = 250 + +# If the EXT_LINKS_IN_WINDOW option is set to YES, doxygen will open links to +# external symbols imported via tag files in a separate window. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +EXT_LINKS_IN_WINDOW = NO + +# Use this tag to change the font size of LaTeX formulas included as images in +# the HTML documentation. When you change the font size after a successful +# doxygen run you need to manually remove any form_*.png images from the HTML +# output directory to force them to be regenerated. +# Minimum value: 8, maximum value: 50, default value: 10. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_FONTSIZE = 10 + +# Use the FORMULA_TRANPARENT tag to determine whether or not the images +# generated for formulas are transparent PNGs. Transparent PNGs are not +# supported properly for IE 6.0, but are supported on all modern browsers. +# +# Note that when changing this option you need to delete any form_*.png files in +# the HTML output directory before the changes have effect. +# The default value is: YES. +# This tag requires that the tag GENERATE_HTML is set to YES. + +FORMULA_TRANSPARENT = YES + +# Enable the USE_MATHJAX option to render LaTeX formulas using MathJax (see +# http://www.mathjax.org) which uses client side Javascript for the rendering +# instead of using pre-rendered bitmaps. Use this if you do not have LaTeX +# installed or if you want to formulas look prettier in the HTML output. When +# enabled you may also need to install MathJax separately and configure the path +# to it using the MATHJAX_RELPATH option. +# The default value is: NO. +# This tag requires that the tag GENERATE_HTML is set to YES. + +USE_MATHJAX = NO + +# When MathJax is enabled you can set the default output format to be used for +# the MathJax output. See the MathJax site (see: +# http://docs.mathjax.org/en/latest/output.html) for more details. +# Possible values are: HTML-CSS (which is slower, but has the best +# compatibility), NativeMML (i.e. MathML) and SVG. +# The default value is: HTML-CSS. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_FORMAT = HTML-CSS + +# When MathJax is enabled you need to specify the location relative to the HTML +# output directory using the MATHJAX_RELPATH option. The destination directory +# should contain the MathJax.js script. For instance, if the mathjax directory +# is located at the same level as the HTML output directory, then +# MATHJAX_RELPATH should be ../mathjax. The default value points to the MathJax +# Content Delivery Network so you can quickly see the result without installing +# MathJax. However, it is strongly recommended to install a local copy of +# MathJax from http://www.mathjax.org before deployment. +# The default value is: http://cdn.mathjax.org/mathjax/latest. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest + +# The MATHJAX_EXTENSIONS tag can be used to specify one or more MathJax +# extension names that should be enabled during MathJax rendering. For example +# MATHJAX_EXTENSIONS = TeX/AMSmath TeX/AMSsymbols +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_EXTENSIONS = + +# The MATHJAX_CODEFILE tag can be used to specify a file with javascript pieces +# of code that will be used on startup of the MathJax code. See the MathJax site +# (see: http://docs.mathjax.org/en/latest/output.html) for more details. For an +# example see the documentation. +# This tag requires that the tag USE_MATHJAX is set to YES. + +MATHJAX_CODEFILE = + +# When the SEARCHENGINE tag is enabled doxygen will generate a search box for +# the HTML output. The underlying search engine uses javascript and DHTML and +# should work on any modern browser. Note that when using HTML help +# (GENERATE_HTMLHELP), Qt help (GENERATE_QHP), or docsets (GENERATE_DOCSET) +# there is already a search function so this one should typically be disabled. +# For large projects the javascript based search engine can be slow, then +# enabling SERVER_BASED_SEARCH may provide a better solution. It is possible to +# search using the keyboard; to jump to the search box use + S +# (what the is depends on the OS and browser, but it is typically +# , /