Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving from circleci to gha WRT benchmark flows #3209

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

filipecosta90
Copy link
Collaborator

@filipecosta90 filipecosta90 commented Oct 11, 2023

This PR removes the usage of circleci for benchmark workflows. It introduces 3 github actions:

  • One for merges to version and master/main branches
  • One for triggering benchmarks during PR work
  • One for triggering nightly benchmarks once a week

It requires the following secrets to be present among the repo action secrets:

  • PERFORMANCE_RTS_AUTH
  • PERFORMANCE_RTS_HOST
  • PERFORMANCE_RTS_PORT
  • PERFORMANCE_WH_TOKEN
  • PERFORMANCE_GH_TOKEN
  • PERFORMANCE_EC2_ACCESS_KEY
  • PERFORMANCE_EC2_SECRET_KEY
  • PERFORMANCE_EC2_REGION
  • PERFORMANCE_EC2_PRIVATE_PEM

Checks:

@filipecosta90 filipecosta90 added the action:run-benchmark Trigger CI benchmarks label Oct 11, 2023
@filipecosta90
Copy link
Collaborator Author

filipecosta90 commented Oct 12, 2023

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

  • Detected a total of 16 stable tests between versions.
  • Detected a total of 3 highly unstable benchmarks.
  • Detected a total of 1 regressions bellow the regression water line 5.0.

You can check a comparison in detail via the grafana link

Comparison between master and benchmarks.gha.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case Baseline master (median obs. +- std.dev) Comparison benchmarks.gha (median obs. +- std.dev) % change (higher-better) Note
ENTITY_COUNT 49998 +- 0.0% (7 datapoints) 49998 +- nan% (1 datapoints) -0.0% -- no change --
EXPLICIT-EDGE-DELETION 50 +- 37.8% UNSTABLE (7 datapoints) 50 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP 33323 +- 18.9% UNSTABLE (7 datapoints) 33323 +- nan% (1 datapoints) -0.0% UNSTABLE (very high variance)
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_65perc_WRITE_25perc_DEL_10perc 50 +- 2.6% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-1_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 2.4% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP 11111 +- 0.0% (7 datapoints) 11111 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-2_HOP_MIXED_READ_75perc_WRITE_25perc 59 +- 0.0% (7 datapoints) 59 +- nan% (1 datapoints) 0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP 893 +- 0.5% (7 datapoints) 893 +- nan% (1 datapoints) -0.0% -- no change --
GRAPH500-SCALE_18-EF_16-3_HOP_MIXED_READ_75perc_WRITE_25perc 56 +- 0.0% (7 datapoints) 56 +- nan% (1 datapoints) 0.0% -- no change --
IMPLICIT-EDGE-DELETION 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) -0.0% -- no change --
INSPECT-EDGES 2500 +- 0.0% (7 datapoints) 2500 +- nan% (1 datapoints) -0.0% -- no change --
MIX_READ_WRITE 10000 +- 0.0% (7 datapoints) 10000 +- nan% (1 datapoints) -0.0% -- no change --
NODE-BATCH-DELETE 50 +- 0.0% (7 datapoints) 50 +- nan% (1 datapoints) 0.0% -- no change --
NODE-INDEX-LOOKUP 33 +- 0.0% (7 datapoints) 33 +- nan% (1 datapoints) -0.0% -- no change --
SORT_ENTITIES 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
UPDATE-BASELINE 24999 +- 0.0% (7 datapoints) 24998 +- nan% (1 datapoints) -0.0% -- no change --
VARIABLE-LENGTH-FILTER 33323 +- 18.9% UNSTABLE (7 datapoints) 33324 +- nan% (1 datapoints) 0.0% UNSTABLE (very high variance)
VARIABLE_LENGTH_EXPAND_INTO 100 +- 0.0% (7 datapoints) 100 +- nan% (1 datapoints) 0.0% -- no change --
allShortestPaths-10hop-100Kpaths 30 +- 5.4% (7 datapoints) 30 +- nan% (1 datapoints) 0.0% waterline=5.4%. -- no change --
allShortestPaths-4hop-10Kpaths 357 +- 3.3% (7 datapoints) 333 +- nan% (1 datapoints) -6.7% REGRESSION

.github/workflows/event-weekly.yml Show resolved Hide resolved
.github/workflows/event-push-to-integ.yml Outdated Show resolved Hide resolved
.github/workflows/benchmark-trigger.yml Outdated Show resolved Hide resolved
.github/workflows/benchmark-runner.yml Show resolved Hide resolved
.github/workflows/benchmark-flow.yml Show resolved Hide resolved
.github/workflows/benchmark-flow.yml Outdated Show resolved Hide resolved
.github/workflows/benchmark-flow.yml Outdated Show resolved Hide resolved
.github/workflows/benchmark-flow.yml Show resolved Hide resolved
.github/workflows/benchmark-flow.yml Outdated Show resolved Hide resolved
@filipecosta90 filipecosta90 requested review from swilly22 and removed request for chayim October 20, 2023 17:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
action:run-benchmark Trigger CI benchmarks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants