This repository has been archived by the owner on Apr 11, 2024. It is now read-only.
feat: add endAtRME option to end when rme drops below threshold #223
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use this config option to end the benchmark earlier than normal in case the relative margin of error (rme) drops below the specified amount, e.g. 1%.
watson
changed the title
4 tasks
watson
added a commit
to elastic/apm-agent-nodejs
that referenced
this pull request
Sep 13, 2019
This commit enables benchmarks for the Node.js agent. For every commit to master, Jenkins will the benchmarks located in the `test/benchmarks` directory. Currently, those benchmarks are: - `test/benchmarks/001-transaction-and-span-no-stack-trace.js`, Measures the overhead of starting and ending a transaction and a span without a stack trace - `test/benchmarks/002-transaction-and-span-overhead-realistic-size.js`, Measures the overhead of starting and ending a transaction and a span with a realistic size stack trace - `test/benchmarks/003-transaction-and-span-with-stack-trace.js`, Measures the overhead of starting and ending a transaction and a span with a simple uniform stack trace - `test/benchmarks/004-transaction.js`, Measures the overhead of starting and ending a transaction only - `test/benchmarks/005-transaction-reading-file.js`, Measures the overhead of starting and ending a transaction only while reading a file The benchmarks are using the benchmark.js benchmarking tool. For each of the benchmarks, it would be best to get the relative margin of error down below 1%, but benchmark.js doesn't support running the benchmark until a given relative margin of error threshold. So for now this commit just run each of them for 2x60 seconds (60 seconds for the benchmark it self and 60 seconds for the control), which in most cases gets them below the magic 1%. A PR has been opened to add this feature to benchmark.js: bestiejs/benchmark.js#223 Each of the benchmarks a new document is added to our Elasticsearch benchmarking cluster when running on Jenkins. To add a new benchmark, simply add a new file to the `test/benchmarks` directory. To get more info about how to run the benchmarks locally, run: npm run bench -- --help Closes #293 Closes #306
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This new config option allows to end the benchmark earlier than normal in case the relative margin of error drops below the specified amount, e.g. 1%.
Without this option, you'd have to set a high enough
maxTimeto ensure the benchmark ran long enough to get a lowrme. With this option, the benchmark can exit as early as possible to save time.Example