Add index performance script #2042
Conversation
|
Example output: https://gist.github.com/cmurphy/71683066aa94084e6795cd406fd1eab4 I've not added any workflows to run this script regularly, but it could be useful in the future to run this automatically to track performance trends. It would not be appropriate to gate on this, since an individual performance run could be better or worse from one to another based on a wide variety of factors. I'm using this to do work on the public instance and it was suggested that it could be useful to include in this repo. I'm also fine with keeping it separate if this seems too niche. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #2042 +/- ##
===========================================
- Coverage 66.46% 48.93% -17.53%
===========================================
Files 92 80 -12
Lines 9258 6641 -2617
===========================================
- Hits 6153 3250 -2903
- Misses 2359 2987 +628
+ Partials 746 404 -342
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
cmurphy
force-pushed
the
performance-scripts
branch
from
c1b82a4
to
26c27a9
Compare
| setup_bastion() { | ||
| echo "Configuring the bastion..." | ||
| sudo apt install kubernetes-client google-cloud-sdk-gke-gcloud-auth-plugin git redis-tools gnuplot prometheus minisign -y | ||
| which hyperfine >/dev/null || ( wget -O /tmp/hyperfine_1.16.1_amd64.deb https://github.com/sharkdp/hyperfine/releases/download/v1.16.1/hyperfine_1.16.1_amd64.deb && sudo dpkg -i /tmp/hyperfine_1.16.1_amd64.deb ) |
There was a problem hiding this comment.
Can we download latest for each of these? There's some risk of breaking changes to the script, but that should be detectable at runtime.
| project_id = var.project | ||
| cluster_name = "rekor" | ||
|
|
||
| attestation_bucket = "cmurphy-sigstore-attestations" |
There was a problem hiding this comment.
Can you remove attestation bucket, since that isn't required until intoto types are in use?
| instance_name = module.bastion.name | ||
| zone = module.bastion.zone | ||
| members = [ | ||
| "serviceAccount:ga-206@colleenmurphy-testing-410318.iam.gserviceaccount.com", |
There was a problem hiding this comment.
Should make these variables, or at least mention in docs they'd need to be updated.
There was a problem hiding this comment.
Oops, did not mean to leave this in here
cmurphy
force-pushed
the
performance-scripts
branch
from
26c27a9
to
2d4829c
Compare
| @@ -0,0 +1,374 @@ | |||
| #!/bin/bash -e | |||
There was a problem hiding this comment.
can we use
#!/usr/bin/env bash
set -o errexit
2d4829c
to
22d62e8
Compare
Add terraform configuration and scripts to set up rekor standalone on GCP, perform a series of insert and search operations, use Prometheus to gather metrics, and plot the results with gnuplot. The scripts added here are for comparing mysql and redis as index storage backends. Other types of performance measurement scripts could be added here in the future. To get a realistic sense of query speed for searches, a large data set is needed. Rather than using the rekor API to insert real data, fake data is generated and uploaded directly to the backend before searching it. Different types of searches are performed: searches where there should be many results, searches where there should be few results, and searches where there should be no results. The goal is not to compare the latency of these different searches, but to take the overall average to compare across backends. Signed-off-by: Colleen Murphy <colleenmurphy@google.com>
cmurphy
force-pushed
the
performance-scripts
branch
from
22d62e8
to
73135f3
Compare
Add terraform configuration and scripts to set up rekor standalone on GCP, perform a series of insert and search operations, use Prometheus to gather metrics, and plot the results with gnuplot.
The scripts added here are for comparing mysql and redis as index storage backends. Other types of performance measurement scripts could be added here in the future.
To get a realistic sense of query speed for searches, a large data set is needed. Rather than using the rekor API to insert real data, fake data is generated and uploaded directly to the backend before searching it.
Different types of searches are performed: searches where there should be many results, searches where there should be few results, and searches where there should be no results. The goal is not to compare the latency of these different searches, but to take the overall average to compare across backends.
Depends on sigstore/scaffolding#1036
Summary
Release Note
Documentation