Support for oTel agent in shared library format (.so file) ?

[santoshkashyap]

Hi,
When running the otelcol with Java Spring boot application on CloudFoundry (via build pack) I get a following error:

The agent is passed via environment variable: JBP_CONFIG_JAVA_OPTS: 'java_opts: "-agentpath:$PWD/collector/otelcol"'

The oTel binary agent I used : x86_86(or amd64) - otelcol_0.59.0_linux_amd64.tar.gz.

For binary oTel agents, it would be nice to have .so format (eg: otelcol_0.59.0_linux_amd64.so) for use with java_opts.

[jpkrohling]

That's new to me, and I'm intrigued. I thought JVM agents were only for Java applications. This probably shows my ignorance on the subject, but shouldn't you be using the Java auto-instrumentation agent?

https://github.com/open-telemetry/opentelemetry-java-instrumentation

[santoshkashyap]

https://github.com/open-telemetry/opentelemetry-java-instrumentation

I check it already. It works fine. With this I had to use a slightly different option:
JBP_CONFIG_JAVA_OPTS: 'java_opts: "-javaagent:BOOT-INF/lib/opentelemetry-javaagent-1.17.0.jar"'

Loosely it translates to following start command(Omitted few options for simplicity):

META-INF/.java_buildpack/jvm/bin/java -javaagent:BOOT-INF/lib/opentelemetry-javaagent-1.17.0.jar -Djava.io.tmpdir=/home/vcap/tmp -Dlog4j2.formatMsgNoLookups=true -agentpath:META-INF/.java_buildpack/jvm_kill/jvmkill-1.16.0.RELEASE-trusty.so=printHeapHistogram=1 org.springframework.boot.loader.JarLauncher

As you would know -javaagent:BOOT-INF/lib/opentelemetry-javaagent-1.17.0.jar does the trick.

Notice the different between the two in the java_opts argument. In case of a non-java agent binary, it would be
-agentpath and for java agent it would be -javaagent. The concern with using javaagent is that many supported frameworks are still in alpha and subject to broken changes (as noted here).
For example:

Similarly, other framework support is also in alpha :io.opentelemetry.jdbc, io.opentelemetry.apache-httpclient-4.0, io.opentelemetry.spring-webmvc-3.1.

Besides this, it also a popular option to use native agent based on the platform. Eg: For Cloud Foundry, this is Linux x86_64(AMD64). Vendors like Dynatrace also support it.

Dynatrace, for example uses this approach to inject its OneAgent binary via the build pack, see framework-dynatrace_one_agent. In this approach, the OneAgent is automatically supplied when the application is run in CloudFoundry with a service binding to Dynatrace as following:

Final start command:

JAVA_OPTS="-agentpath:$PWD/.java-buildpack/jre/bin/jvmkill-1.16.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR -XX:ActiveProcessorCount=$(nproc)
                 -agentpath:$PWD/.java-buildpack/dynatrace_one_agent/agent/bin/1.243.189.20220715-135319/linux-x86-64/liboneagentloader.so -Xshare:off -Djava.ext.dirs=
                 -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" &&
                 CALCULATED_MEMORY=$($PWD/.java-buildpack/jre/bin/java-buildpack-memory-calculator-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=20063 -poolType=metaspace -stackThreads=250
                 -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY" && MALLOC_ARENA_MAX=2 DT_TENANT=<tenant_id>
                 DT_TENANTTOKEN=<dt_token>
                 DT_CONNECTION_POINT=<dt-URL>
                 DT_APPLICATIONID=service-a DT_LOGSTREAM=stdout SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/jre/bin/java $JAVA_OPTS -cp
                 $PWD/.:$PWD/.java-buildpack/container_security_provider/container_security_provider-1.19.0_RELEASE.jar org.springframework.boot.loader.JarLauncher

Especially, notice the option: -agentpath:$PWD/.java-buildpack/dynatrace_one_agent/agent/bin/1.243.189.20220715-135319/linux-x86-64/liboneagentloader.so for the Dynatrace OneAgent. It would be nice to have something similar with OpenTelemetry releases as well. From our project stand point, I'm doing an evaluation on various approaches to use a Telemetry agent when the application runs on different runtimes like CloudFoundry and K8S.