This repository contains a set of performance tests used to properly measure how changes in the compiler affect performance. A big part of this suite consists of the nofib benchmark suite along with benchmarks that are relevant for Eta.
The Java Micobenchmarking Harness is used to conduct the benchmarks. The follow process occurs when you run a benchmark:
- The benchmark is compiled into a JAR file.
- The
eta.main.mainmethod (the entry point into Eta programs) is invoked from the JMH framework. - The JMH framework will fork a single JVM process and run the program a number of times for warmup and a number of times for measurement and the average/error is calculated. At the end of each iteration, GC is run and all static thunks are reset to unevaluated state.
- Gradle
- JDK 1.7+
- etlas
First, get setup:
$ ./scripts/init.sh
A standalone script will run the fast part of the suite, also run by CircleCI.
$ ./scripts/fast-benchmarks.sh
A standalone script will run the slow part of the suite.
$ ./scripts/slow-benchmarks.sh
A single benchmark can be run:
./scripts/single-bench.sh [test-name] [measurement-iterations] [warmup-iterations]
Example:
./scripts/single-bench.sh imaginary/bernouilli 10 5
This indicates 10 measurement iterations and 5 warmup iterations.
Note: You can also pass --skip-check as a parameter to skip the checking of the test output.
If you want to do more fine-grained performance benchmarking, please see the remaining sections.
All runs of the runner should be at the root of the repository.
$ eta-bench [test-name] --run
Example:
$ eta-bench scs --run
$ eta-bench [suite-name] --run
Example:
$ eta-bench imaginary --run
$ eta-bench clean
If you had trouble using this project, you can give us feedback by: