Native benchmark fails on 0.4.10

[CharlieTap]

I'm running:

Kotlin 1.9.2
OS: Sonoma
CPU: Apple M1 Max
Gradle: 8.5

You can recreate by running build on this branch:

https://github.com/CharlieTap/cachemap/tree/failing-benchmark

I get the following:

Execution failed for task ':benchmark:macosArm64BenchmarkGenerate'.
> There was a failure while executing work items
   > A failure occurred while executing kotlinx.benchmark.gradle.NativeSourceGeneratorWorker
      > 'void org.jetbrains.kotlin.serialization.deserialization.DeserializationComponents.<init>(org.jetbrains.kotlin.storage.StorageManager, org.jetbrains.kotlin.descriptors.ModuleDescriptor, org.jetbrains.kotlin.serialization.deserialization.DeserializationConfiguration, org.jetbrains.kotlin.serialization.deserialization.ClassDataFinder, org.jetbrains.kotlin.serialization.deserialization.AnnotationAndConstantLoader, org.jetbrains.kotlin.descriptors.PackageFragmentProvider, org.jetbrains.kotlin.serialization.deserialization.LocalClassifierTypeSettings, org.jetbrains.kotlin.serialization.deserialization.ErrorReporter, org.jetbrains.kotlin.incremental.components.LookupTracker, org.jetbrains.kotlin.serialization.deserialization.FlexibleTypeDeserializer, java.lang.Iterable, org.jetbrains.kotlin.descriptors.NotFoundClasses, org.jetbrains.kotlin.serialization.deserialization.ContractDeserializer, org.jetbrains.kotlin.descriptors.deserialization.AdditionalClassPartsProvider, org.jetbrains.kotlin.descriptors.deserialization.PlatformDependentDeclarationFilter, org.jetbrains.kotlin.protobuf.ExtensionRegistryLite, org.jetbrains.kotlin.types.checker.NewKotlinTypeChecker, org.jetbrains.kotlin.resolve.sam.SamConversionResolver, java.util.List, org.jetbrains.kotlin.serialization.deserialization.EnumEntriesDeserializationSupport, int, kotlin.jvm.internal.DefaultConstructorMarker)'

[fzhinkin]

@CharlieTap I tried to reproduce the problem and the build failed with two different issues none of which match the summary you posted above.

Could you please share your build log via --scan?

The two aforementioned issues I faced while running the benchmarks are:

• for me, benchmark tasks are running with kotlin-compiler-embeddable:1.8.20 instead of 1.9.20 and it just can't resolve the klib:

* What went wrong:
Execution failed for task ':benchmark:macosArm64BenchmarkGenerate'.
> There was a failure while executing work items
   > A failure occurred while executing kotlinx.benchmark.gradle.NativeSourceGeneratorWorker
      > e: Could not find "/Users/Filipp.Zhinkin/Development/cachemap/benchmark/build/classes/kotlin/macosArm64/main/klib/benchmark.klib" in [/Users/Filipp.Zhinkin/Development/cachemap]

• once I overcome the issue with the wrong compiler version on the class path, benchmarks generation failed because multiple benchmark modes are implicitly unsupported for native targets (https://github.com/Kotlin/kotlinx-benchmark/blob/3f32f2d9d9732a2a01d231134c52e6554428b519/plugin/main/src/kotlinx/benchmark/gradle/SuiteSourceGenerator.kt#L142C26-L142C26).

[CharlieTap]

I've posted a fresh branch here with it failing here, with the extra mode the removed (this should probably be documented as the annotations are copied from JMH so I won't be the last to run into this)

https://github.com/CharlieTap/cachemap/tree/failing-native-benchmark

And also heres a scan:

https://scans.gradle.com/s/lazrbivpp5g7s

With regards to kotlin compiler embeddable I'm not sure what you're referring to as 1.8.20 is not a dependency of my project at all?

[fzhinkin]

@CharlieTap thanks for posting the scan!

With regards to kotlin compiler embeddable I'm not sure what you're referring to as 1.8.20 is not a dependency of my project at all?

It's a transitive dependency for coroutines. In your case it seems to be the similar issue, but with serialization. I'm not quite sure why it's happening (Gradle reports that 1.9.* versions of these artifacts should be used instead of 1.8.*), I'll check what could be done there.

[fzhinkin]

Filed #174 for the issue with multiple benchmark modes.

[fzhinkin]

It's a transitive dependency for coroutines

Looking at the benchmark module, it's from the kotlinter:

$ ./gradlew :benchmark:buildEnvironment
...
+--- org.jmailen.kotlinter:org.jmailen.kotlinter.gradle.plugin:3.15.0
|    \--- org.jmailen.gradle:kotlinter-gradle:3.15.0
|         +--- com.pinterest.ktlint:ktlint-core:0.49.1
|         |    +--- org.jetbrains.kotlin:kotlin-compiler-embeddable:1.8.20
...
|         +--- com.pinterest.ktlint:ktlint-rule-engine:0.49.1
|         |    +--- com.pinterest.ktlint:ktlint-logger:0.49.1
...
|         |    +--- org.jetbrains.kotlin:kotlin-compiler-embeddable:1.8.20 (*)
...

[fzhinkin]

@CharlieTap could you please try placing the following block before plugins in benchmark's build.gradle.kts?

buildscript {
    configurations.all {
        resolutionStrategy {
            eachDependency {
                if (requested.group == "org.jetbrains.kotlin") {
                    val kv = libs.versions.kotlin.get()
                    logger.warn("${requested.group}:${requested.name}:${requested.version} --> $kv")
                    useVersion(kv)
                }
            }
        }
    }
}

[CharlieTap]

Ohh thats fixes it!

So both ktlint and this library use the embedded compiler and its picking the 1st one loaded on the classpath? Is there a fix for this other than forcing the version like you provided above?

[fzhinkin]

The problem is that benchmarks plugin declares embedded compiler as a compile-only dependency (and there's a good reason for doing that) and that may need to use of incompatible compiler version in runtime.
We'll check what could be done here to prevent the problem in general.

[qurbonzoda]

There is an ongoing effort from the Build Tools team to resolve this issue: https://youtrack.jetbrains.com/issue/KT-66764