Cannot run benchmarks

[toolforger]

Describe the bug
https://github.com/EsotericSoftware/kryo/tree/master/benchmarks#readme has two issues:

• It instructs the user to run the Kryo benchmarks using a run.sh shell script. The script fails (see below).
• It refers to JMH parameters but does not explain where to find explanations for them, or which of them are useful for Kryo.

To Reproduce

$ git clone git@github.com:EsotericSoftware/kryo.git  
Cloning into 'kryo'...
Enter passphrase for key '/home/jo/.ssh/id_rsa': 
remote: Enumerating objects: 13908, done.
remote: Counting objects: 100% (2061/2061), done.
remote: Compressing objects: 100% (372/372), done.
remote: Total 13908 (delta 1789), reused 1752 (delta 1648), pack-reused 11847
Receiving objects: 100% (13908/13908), 14.99 MiB | 8.96 MiB/s, done.
Resolving deltas: 100% (7692/7692), done.
$ cd kryo
$ mvn clean install
[INFO] Scanning for projects...
...
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:01 min
[INFO] Finished at: 2024-04-13T18:06:32+02:00
[INFO] ------------------------------------------------------------------------
$ cd benchmarks
$ mvn clean install
[INFO] Scanning for projects...
...
[INFO] --- maven-dependency-plugin:3.6.1:build-classpath (build-classpath) @ kryo-benchmarks ---
[INFO] Dependencies classpath:
/home/jo/.m2/repository/com/esotericsoftware/kryo/5.6.1-SNAPSHOT/kryo-5.6.1-SNAPSHOT.jar:/home/jo/.m2/repository/com/esotericsoftware/reflectasm/1.11.9/reflectasm-1.11.9.jar:/home/jo/.m2/repository/org/objenesis/objenesis/3.3/objenesis-3.3.jar:/home/jo/.m2/repository/com/esotericsoftware/minlog/1.3.1/minlog-1.3.1.jar:/home/jo/.m2/repository/org/openjdk/jmh/jmh-core/1.37/jmh-core-1.37.jar:/home/jo/.m2/repository/net/sf/jopt-simple/jopt-simple/5.0.4/jopt-simple-5.0.4.jar:/home/jo/.m2/repository/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar:/home/jo/.m2/repository/net/bytebuddy/byte-buddy/1.14.13/byte-buddy-1.14.13.jar
...
[INFO] --- maven-install-plugin:3.1.1:install (default-install) @ kryo-benchmarks ---
[INFO] Installing /home/jo/Projekte/tmp/kryo/benchmarks/pom.xml to /home/jo/.m2/repository/com/esotericsoftware/kryo-benchmarks/5.6.1-SNAPSHOT/kryo-benchmarks-5.6.1-SNAPSHOT.pom
[INFO] Installing /home/jo/Projekte/tmp/kryo/benchmarks/target/kryo-benchmarks-5.6.1-SNAPSHOT.jar to /home/jo/.m2/repository/com/esotericsoftware/kryo-benchmarks/5.6.1-SNAPSHOT/kryo-benchmarks-5.6.1-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  9.621 s
[INFO] Finished at: 2024-04-13T18:08:10+02:00
[INFO] ------------------------------------------------------------------------
$ chmod +x run.sh
+ mkdir -p charts/results
+ /bin/java -cp '../eclipse/bin;../eclipse/.apt_generated;../lib/*;lib/*' com.esotericsoftware.kryo.benchmarks.KryoBenchmarks -f 4 -wi 5 -i 3 -t 2 -w 2s -r 2s -rf csv -rff charts/results/fieldSerializer.csv FieldSerializerBenchmark
Error: Could not find or load main class com.esotericsoftware.kryo.benchmarks.KryoBenchmarks
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.benchmarks.KryoBenchmarks

Well, sure, -cp uses semicolon, I'm on Linux and need to use a colon.
Edited run.sh, replaced ';' with ':', tried again:

$ ./run.sh
+ mkdir -p charts/results
+ /bin/java -cp '../eclipse/bin:../eclipse/.apt_generated:../lib/*:lib/*' com.esotericsoftware.kryo.benchmarks.KryoBenchmarks -f 4 -wi 5 -i 3 -t 2 -w 2s -r 2s -rf csv -rff charts/results/fieldSerializer.csv FieldSerializerBenchmark
Error: Could not find or load main class com.esotericsoftware.kryo.benchmarks.KryoBenchmarks
Caused by: java.lang.ClassNotFoundException: com.esotericsoftware.kryo.benchmarks.KryoBenchmarks

Indeed there's no ../eclipse.bin or ../eclipse/.apt_generated directory.

Environment:

• OS: Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux
• JDK Version: OpenJDK 64-Bit Server VM (build 17.0.10+7-Debian-1deb12u1, mixed mode, sharing)
• Kryo Version: git main

Additional context

Let me know if you need any.

[theigl]

It instructs the user to run the Kryo benchmarks using a run.sh shell script. The script fails (see below).

I just reviewed the readme for the benchmarks module. Where does it recommend running via run.sh?

It refers to JMH parameters but does not explain where to find explanations for them, or which of them are useful for Kryo.

The readme suggests reasonable defaults for the parameters and explains how to get a full list of parameters. What else do you need?

[toolforger]

• You're right, I just double-checked: Nothing recommends using run.sh.
• I'm not 100% sure why I was so focused on run.sh. I think I was trying to make sense of that, and since it is undocumented in what it does and its purpose, I got even more confused because I was looking for the wrong kind of information in README.md (if that makes any sense to you).
• There's one bit in the README that is unhelpful: The "digging through the JMH code" link. It is failing with "504 Gateway Time-out", and I recall it didn't work for me, either.

I guess I just lost track all the leads and got sidetracked chasing the dead ends, so nothing made sense to me because I was focusing on those parts that were dead ends.
I think that's what's happening a lot when you're new to a project, and you get overwhelmed by information because you do not yet have an intuition for what information is important for the task at hand and what isn't.

Actually I'm running benchmarks just fine now, it has started to "click".
The only thing that I'm missing is a documentation how the benchmark graphs in the release are done, as I'll want that to quickly check whether I damaged performance or not.

[NathanSweet]

JMH is a bit of a pain to setup because it generates classes.

[toolforger]

Answering to @NathanSweet, just for the record:

JMH is a bit of a pain to setup because it generates classes.

If it's a pain, then not due to class generation. Not anymore.
The final pieces fell into place maybe two years ago, when IDEs got full annotation processor integration, including the ability to configure their own incremental build machinery from a Maven or Gradle build.