Application Startup Profiling

[franz1981]

While profiling an application startup by using async-profiling as an agent there's still the missing bit of info re measuring the runtime bootstrap till agent loading complete; I see this one as an interesting feature to be implemented together with the "classify" one i.e. marking as "bootstraps" the native-only stack-traces of the JVM before "jattach" is available.
I have no idea if it's possible to implement something like that given the current architecture of the profiler, but it would be of great help for applications that need to be restarted/scale-up frequently, providing additional useful informations re hidden and still important costs (compiler threads startup cost, GC threads startup cost, initialization of GC data-structures and/or pre-touch costs, etc etc).

Obviously this means that it cannot be used as an agent anymore :)

[parttimenerd]

One idea would be to run perf record for the first 10 or so seconds on the JVM (till the async-profiler is surely loaded) and then combine the results later with the async-profiler results.

It should alternatively be possible to use the stack walking mechanisms of async-profiler to create a small tool that offers similarity functionality to perf record, recording only till it sees the first calls to the async-profiler sampling methods in its traces.

[franz1981]

Yep, my point is that AP knows when the JVM is available and running so can classify samples with more accuracy then what I would do myself using 2 separate tools.
AFAIK AP already provide native traces that doesn't have AGCT stacks in (GC/Compiler ones) meaning that have a way to correctly walk their stack traces (thanks to the libjvm mapping probably, I didn't checked)

[parttimenerd]

Yes, but I think implementing it directly in the Agent is impossible. I rather envision something like

ap-record java -agentlib:libasyncprofiler.so=... 

Where ap-record is able to set environment variables, so that the agent knows where to look for the recording of the initial few seconds.

[apangin]

I have made a PoC recently of starting async-profiler as LD_PRELOAD'ed library (spoiler: for profiling native apps), so it's definitely possible to use it for JVM startup, too.
While not promising anything and not committing myself, I can tell this is one of the features that may appear in near future.

[parttimenerd]

I'm happy to help in the effort.

[franz1981]

And can schedule some free cycles to help as well, in case (although I know that @apangin is a fast train when he decides to do it :D )

spoiler: for profiling native apps

considering how many different tools are required to do the same in the native world, async profiler looks way more appealing to me

[parttimenerd]

Any progress? I'm happy to work on your PoC.

[apangin]

Profiling of non-Java processes has been added in #751.
It can also be used for profiling java startup:

LD_PRELOAD=libasyncProfiler.so ASPROF_COMMAND=start,event=cpu,interval=100us,threads,file=profile.html java -version

The downside of such a profile is that it does not name any Java frames.

Java frame information becomes available after JVM bootstrap has completed, i.e. after JVM TI VMStart even has been sent.
A recently added cstack=vm mode is also helpful for analyzing JVM startup from that moment.

Merging two above profiles is possible, but adds redundant complexity/ugliness to the code. I don't think it's worth productizing the feature right now, assuming the existing capabilities are enough to troubleshoot startup issues.