Skip to content

Moving from circleci to gha WRT benchmark flows - #3209

Open
filipecosta90 wants to merge 4 commits into
masterfrom
benchmarks.gha
Open

Moving from circleci to gha WRT benchmark flows#3209
filipecosta90 wants to merge 4 commits into
masterfrom
benchmarks.gha

Conversation

@filipecosta90

@filipecosta90 filipecosta90 commented Oct 11, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator

This PR removes the usage of circleci for benchmark workflows. It introduces 3 github actions:

  • One for merges to version and master/main branches
  • One for triggering benchmarks during PR work
  • One for triggering nightly benchmarks once a week

It requires the following secrets to be present among the repo action secrets:

  • PERFORMANCE_RTS_AUTH
  • PERFORMANCE_RTS_HOST
  • PERFORMANCE_RTS_PORT
  • PERFORMANCE_WH_TOKEN
  • PERFORMANCE_GH_TOKEN
  • PERFORMANCE_EC2_ACCESS_KEY
  • PERFORMANCE_EC2_SECRET_KEY
  • PERFORMANCE_EC2_REGION
  • PERFORMANCE_EC2_PRIVATE_PEM

Checks:

@filipecosta90 filipecosta90 added the action:run-benchmark Trigger CI benchmarks label Oct 11, 2023
@filipecosta90
filipecosta90 requested a review from chayim October 11, 2023 05:43
@filipecosta90

filipecosta90 commented Oct 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 16 stable tests between versions.
  • Detected a total of 3 highly unstable benchmarks.
  • Detected a total of 1 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and benchmarks.gha.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison benchmarks.gha (median obs. +- std.dev) % change (higher-better) Note
ENTITY_COUNT 49998 +- 0.0% (7 datapoints) 49998 +- nan% (1 datapoints) -0.0% -- no change --
EXPLICIT-EDGE-DELETION 50 +- 37.8% UNSTABLE (7 datapoints) 50 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP 33323 +- 18.9% UNSTABLE (7 datapoints) 33323 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_65perc_WRITE_25perc_DEL_10perc 50 +- 2.6% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 2.4% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP 11111 +- 0.0% (7 datapoints) 11111 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 0.0% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP 893 +- 0.5% (7 datapoints) 893 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP_MIXED_READ_75perc_WRITE_25perc 56 +- 0.0% (7 datapoints) 56 +- nan% (1 datapoints) 0.0% -- no change --
IMPLICIT-EDGE-DELETION 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
INSPECT-EDGES 2500 +- 0.0% (7 datapoints) 2500 +- nan% (1 datapoints) -0.0% -- no change --
MIX_READ_WRITE 10000 +- 0.0% (7 datapoints) 10000 +- nan% (1 datapoints) -0.0% -- no change --
NODE-BATCH-DELETE 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) 0.0% -- no change --
NODE-INDEX-LOOKUP 33 +- 0.0% (7 datapoints) 33 +- nan% (1 datapoints) -0.0% -- no change --
SORT_ENTITIES 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
UPDATE-BASELINE 24999 +- 0.0% (7 datapoints) 24998 +- nan% (1 datapoints) -0.0% -- no change --
VARIABLE-LENGTH-FILTER 33323 +- 18.9% UNSTABLE (7 datapoints) 33324 +- nan% (1 datapoints) 0.0% UNSTABLE (very high variance)
VARIABLE_LENGTH_EXPAND_INTO 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
allShortestPaths-10hop-100Kpaths 30 +- 5.4% (7 datapoints) 30 +- nan% (1 datapoints) 0.0% waterline=5.4%. -- no change --
allShortestPaths-4hop-10Kpaths 357 +- 3.3% (7 datapoints) 333 +- nan% (1 datapoints) -6.7% REGRESSION

swilly22
swilly22 suggested changes Oct 16, 2023
View reviewed changes
Comment thread .github/workflows/event-weekly.yml
Comment thread .github/workflows/event-push-to-integ.yml Outdated
Comment thread .github/workflows/benchmark-trigger.yml
Comment thread .github/workflows/benchmark-runner.yml
Comment thread .github/workflows/benchmark-flow.yml
Comment thread .github/workflows/benchmark-flow.yml Outdated
Comment thread .github/workflows/benchmark-flow.yml Outdated
Comment thread .github/workflows/benchmark-flow.yml
Comment thread .github/workflows/benchmark-flow.yml Outdated
@filipecosta90
filipecosta90 requested review from swilly22 and removed request for chayim October 20, 2023 17:51
@codecov

codecov Bot commented Sep 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.95%. Comparing base (8f6a553) to head (3cc29c4).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #3209   +/-   ##
=======================================
  Coverage   90.95%   90.95%           
=======================================
  Files         295      295           
  Lines       30427    30427           
=======================================
+ Hits        27675    27676    +1     
+ Misses       2752     2751    -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@swilly22 swilly22 Awaiting requested review from swilly22

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

action:run-benchmark Trigger CI benchmarks

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@filipecosta90 @swilly22