How to utilize the output of profile:hz:99 with stack traces and counts to plot graphs

[kode54]

Is there any utility to plot graphs and analyze logs? Someone asked me to help them identify a possible kernel issue with sched-ext and asked for this log output while running a particular game, and I actually want to analyze the data by means other than scrolling through it in a text editor.

[jordalgo]

@kode54 Thanks for your question. Generating bpftrace data formatted for visualization tools is on my todo list (especially as it relates to sched_ext). At the moment, you'd have to do something manually yourself e.g. generate json output and transform it into something like chrome trace format.

The small internal sched_ext community, AFAICT, is using these two tracing tools for sched_ext analysis:
https://gitlab.inria.fr/schedgraph/schedgraph (which doesn't seem to be loading for me at the moment)
https://github.com/skyrmi/scheduler-tracing-tools

Hopefully, I have some additional utilities for this usecase in a few weeks.

[danobi]

FWIW I took a brief look at that almost 4 years ago (wow, time flies). At the time I was thinking something along the lines of custom web app using react+D3 as frontend and some rust binary as back-of-frontend that interacts with bpftrace.

These days, having used grafana for other purposes (see https://github.com/facebookincubator/below/tree/main/contrib/grafana), I would probably take a closer look at a grafana datasource integration: https://grafana.com/docs/grafana/latest/datasources/ . Chrome tracing tools might be ok for some things, but for visualization flexibility it would be very difficult to beat grafana.

[danobi]

@jordalgo I added you to my private bpftrace-viz repo. It's pretty junky code with faked data. But it might still run and could give you some inspiration.

[jordalgo]

Thanks @danobi !

[kode54]

I was also directed to https://github.com/brendangregg/FlameGraph/ by the developers I was supposed to be helping out.

[dxuuu]

Yeah flamegraph is awesome. A bunch of visualization tools supoprt that. Grafana, chrome tracing, firefox profiler, etc.

Firefox profiler is another great option I've used in the past: https://profiler.firefox.com/