kotlinx-benchmark

kotlinx-benchmark is a toolkit for running benchmarks for multiplatform code written in Kotlin. It is designed to work with Kotlin/JVM, Kotlin/JS, Kotlin/Native, and Kotlin/Wasm (experimental) targets.

To get started, ensure you're using Kotlin 1.9.20 or newer and Gradle 7.4 or newer.

Features

• Low noise and reliable results
• Statistical analysis
• Detailed performance reports

Table of contents

• Setting Up a Kotlin Multiplatform Project for Benchmarking
  • Target-specific configurations
    • Kotlin/JVM
    • Kotlin/JS
    • Kotlin/Native
    • Kotlin/Wasm
  • Writing Benchmarks
  • Running Benchmarks
  • Benchmark Configuration Profiles
  • Separate source set for benchmarks
• Examples
• Contributing

• Useful guides
  • Writing Benchmarks
  • Configuring Benchmarks Execution
  • Setting Up Kotlin/JVM and Java Projects for Benchmarking
  • Setting Up a Separate Source Set for Benchmarks
  • Overview of Tasks Provided by kotlinx-benchmark Gradle Plugin

Setting Up a Kotlin Multiplatform Project for Benchmarking

To configure a Kotlin Multiplatform project for benchmarking, follow the steps below. If you want to benchmark only Kotlin/JVM and Java code, you may refer to our comprehensive guide dedicated to setting up benchmarking in those specific project types.

Kotlin DSL

1. Applying Benchmark Plugin: Apply the benchmark plugin.

   // build.gradle.kts
   plugins {
       id("org.jetbrains.kotlinx.benchmark") version "0.4.10"
   }

2. Specifying Plugin Repository: Ensure you have the Gradle Plugin Portal for plugin lookup in the list of repositories:

   // settings.gradle.kts
   pluginManagement {
       repositories {
           gradlePluginPortal()
       }
   }

3. Adding Runtime Dependency: Next, add the kotlinx-benchmark-runtime dependency to the common source set:

   // build.gradle.kts
   kotlin {
       sourceSets {
           commonMain {
               dependencies {
                   implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.10")
               }
           }
       }
   }

4. Specifying Runtime Repository: Ensure you have mavenCentral() for dependencies lookup in the list of repositories:

   // build.gradle.kts
   repositories {
       mavenCentral()
   }

Groovy DSL

1. Applying Benchmark Plugin: Apply the benchmark plugin.

   // build.gradle
   plugins {
       id 'org.jetbrains.kotlinx.benchmark' version '0.4.10'
   }

2. Specifying Plugin Repository: Ensure you have the Gradle Plugin Portal for plugin lookup in the list of repositories:

   // settings.gradle
   pluginManagement {
       repositories {
           gradlePluginPortal()
       }
   }

3. Adding Runtime Dependency: Next, add the kotlinx-benchmark-runtime dependency to the common source set:

   // build.gradle
   kotlin {
       sourceSets {
           commonMain {
               dependencies {
                   implementation 'org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.10'
               }
           }
       }
   }

4. Specifying Runtime Repository: Ensure you have mavenCentral() for dependencies lookup in the list of repositories:

   // build.gradle
   repositories {
       mavenCentral()
   }

Target-specific configurations

To run benchmarks on a platform ensure your Kotlin Multiplatform project targets that platform. For different platforms, there may be distinct requirements and settings that need to be configured. The guide below contains the steps needed to configure each supported platform for benchmarking.

Kotlin/JVM

To run benchmarks in Kotlin/JVM:

1. Create a JVM target:

   // build.gradle.kts
   kotlin {
       jvm()
   }

2. Register jvm as a benchmark target:

   // build.gradle.kts
   benchmark {
       targets { 
           register("jvm")
       }
   }

3. Apply allopen plugin to ensure your benchmark classes and methods are open.

   // build.gradle.kts
   plugins {
       kotlin("plugin.allopen") version "1.9.20"
   }
   
   allOpen {
       annotation("org.openjdk.jmh.annotations.State")
   }

   Explanation

   Assume that you've annotated each of your benchmark classes with @State(Scope.Benchmark):

   // MyBenchmark.kt
   @State(Scope.Benchmark)
   class MyBenchmark {
       // Benchmarking-related methods and variables
       @Benchmark
       fun benchmarkMethod() {
           // benchmarking logic
       }
   }

   In Kotlin, classes are final by default, which means they can't be overridden. This conflicts with the Java Microbenchmark Harness (JMH) operation, which kotlinx-benchmark uses under the hood for running benchmarks on JVM. JMH requires benchmark classes and methods to be open to be able to generate subclasses and conduct the benchmark.

   This is where the allopen plugin comes into play. With the plugin applied, any class annotated with @State is treated as open, which allows JMH to work as intended:

   // build.gradle.kts
   plugins {
       kotlin("plugin.allopen") version "1.9.20"
   }
   
   allOpen {
       annotation("org.openjdk.jmh.annotations.State")
   }

   This configuration ensures that your MyBenchmark class and its benchmarkMethod function are treated as open.

   You can alternatively mark your benchmark classes and methods open manually, but using the allopen plugin enhances code maintainability.

Kotlin/JS

To run benchmarks in Kotlin/JS:

1. Create a JS target with Node.js execution environment:

   // build.gradle.kts
   kotlin {
       js { 
           nodejs() 
       }
   }

2. Register js as a benchmark target:

   // build.gradle.kts
   benchmark {
       targets {
           register("js")
       }
   }

Kotlin/Native

To run benchmarks in Kotlin/Native:

1. Create a Native target:

   // build.gradle.kts
   kotlin {
       linuxX64()
   }

2. Register linuxX64 as a benchmark target:

   // build.gradle.kts
   benchmark {
       targets {
           register("linuxX64")
       }
   }

It is possible to register multiple native targets. However, benchmarks can be executed only for the host target. This library supports all targets supported by the Kotlin/Native compiler.

Kotlin/Wasm

To run benchmarks in Kotlin/Wasm:

1. Create a Wasm target with Node.js execution environment:

   // build.gradle.kts
   kotlin {
       wasm { 
           nodejs() 
       }
   }

2. Register wasm as a benchmark target:

   // build.gradle.kts
   benchmark {
       targets {
           register("wasm")
       }
   }

Note: Kotlin/Wasm is an experimental compilation target for Kotlin. It may be dropped or changed at any time. Refer to Kotlin/Wasm documentation for up-to-date information about the target stability.

Writing Benchmarks

After setting up your project and configuring targets, you can start writing benchmarks. As an example, let's write a simplified benchmark that tests how fast we can add up numbers in an ArrayList:

1. Create Benchmark Class: Create a class in your source set where you'd like to add the benchmark. Annotate this class with @State(Scope.Benchmark).

   @State(Scope.Benchmark)
   class MyBenchmark {
   
   }

2. Set Up Variables: Define variables needed for the benchmark.

   private val size = 10
   private val list = ArrayList<Int>()

3. Initialize Resources: Within the class, you can define any setup or teardown methods using @Setup and @TearDown annotations respectively. These methods will be executed before and after the entire benchmark run.

   @Setup
   fun prepare() {
       for (i in 0..<size) {
           list.add(i)
       }
   }
   
   @TearDown
   fun cleanup() {
       list.clear()
   }

4. Define Benchmark Methods: Next, create methods that you would like to be benchmarked within this class and annotate them with @Benchmark.

   @Benchmark
   fun benchmarkMethod(): Int {
       return list.sum()
   }

Your final benchmark class will look something like this:

import kotlinx.benchmark.*

@State(Scope.Benchmark)
class MyBenchmark {
    private val size = 10
    private val list = ArrayList<Int>()

    @Setup
    fun prepare() {
        for (i in 0..<size) {
            list.add(i)
        }
    }

    @TearDown
    fun cleanup() {
        list.clear()
    }

    @Benchmark
    fun benchmarkMethod(): Int {
        return list.sum()
    }
}

Note: Benchmark classes located in the common source set will be run in all platforms, while those located in a platform-specific source set will be run only in the corresponding platform.

See writing benchmarks for a complete guide for writing benchmarks.

Running Benchmarks

To run your benchmarks in all registered platforms, run benchmark Gradle task in your project. To run only on a specific platform, run <target-name>Benchmark, e.g., jvmBenchmark.

For more details about the tasks created by the kotlinx-benchmark plugin, refer to this guide.

Benchmark Configuration Profiles

The kotlinx-benchmark library provides the ability to create multiple configuration profiles. The main configuration is already created by the toolkit. Additional profiles can be created as needed in the configurations section of the benchmark block:

// build.gradle.kts
benchmark {
    configurations {
        named("main") {
            warmups = 20
            iterations = 10
            iterationTime = 3
            iterationTimeUnit = "s"
        }
        register("smoke") {
            include("<pattern of fully qualified name>")
            warmups = 5
            iterations = 3
            iterationTime = 500
            iterationTimeUnit = "ms"
        }
    }
}

Refer to our comprehensive guide to learn about configuration options and how they affect benchmark execution.

Separate source set for benchmarks

Often you want to have benchmarks in the same project, but separated from main code, much like tests. Refer to our detailed documentation on configuring your project to set up a separate source set for benchmarks.

Examples

To help you better understand how to use the kotlinx-benchmark library, we've provided an examples subproject. These examples showcase various use cases and offer practical insights into the library's functionality.

Contributing

We welcome contributions to kotlinx-benchmark! If you want to contribute, please refer to our Contribution Guidelines.