Continuous Profile-Guided Optimization (PGO) platform based on Grafana Pyroscope

[zamazan4ik]

Hi!

I evaluate multiple approaches to applying Profile-Guided Optimization (PGO) to the IT ecosystem (if you are interested you can check the results here). According to my tests, PGO helps with achieving better performance for many existing applications.

There are two major kinds of PGO: Instrumentation and Sampling (about both of them you can read the Clang compiler documentation). Despite Instrumentation-based PGO is much more well-known in the community (from my experience), it has major drawbacks. One of the biggest drawbacks is the performance overhead during the profiling phase.

To resolve this issue, Google invented Sampling PGO (also sometimes called AutoFDO (https://github.com/google/autofdo)). This approach uses perf-based profiling to collect the PGO profiles directly from the production environment, and then use them during the optimization phase. More details about the whole ecosystem around AutoFDO in Google can be found in their paper.

Unfortunately, right now there is no open-source ecosystem to support the AutoFDO approach at scale as it's done in Google. But Grafana Pyroscope looks like an important piece in this pipeline (we can say it could replace Google Wide Profiler (GWP) in the AutoFDO setup in Google).

My idea is to think about building the same system as Google has internally but based on Pyroscope. It could be an interesting opportunity for Pyroscope to become a unique solution in the continuous optimization area. As I see in the Pyroscope users activity already there is some interest in PGO from users.

Would be great to hear thoughts about the idea from the Pyroscope devs.

Thank you for your attention.

[Rperry2174]

hey @zamazan4ik !

Thanks for the very interesting discussion topic and for sharing your thoughts on PGO in your IT ecosystem context.

I would say I'm definitely very intrigued by your sugggestion of leveraging Pyroscope for implementing open source ecosystem similar to Google's autofdo.

We've previously explored the integration of PGO within the Go context, particularly through our profile-cli tool. This tool allows querying specific applications over user-defined time periods to generate pprof files, which can then be utilized for PGO in Golang.

What you're proposing seems to align closely with these capabilities but I guess with a slightly different profile format.

Moving forward, here are some potential next steps and considerations for implementing your idea:

• Format Compatibility: We need to investigate the compatibility of the profile format used in AutoFDO with our existing profile-cli tool. Understanding the conversion process, if required, is vital to ensure seamless integration
• Performance Impact Assessment: Assessing the performance overhead during the profiling phase, especially in a production environment, is crucial -- we always try to make sure to use low-overhead profilers so that the improvement isn't outweighted by the overhead

Would love to work together to do a POC of this if you're interested!

[zamazan4ik]

Thank you for your interest!

Format Compatibility: We need to investigate the compatibility of the profile format used in AutoFDO with our existing profile-cli tool

AutoFDO tool from Google converts Linux perf record profiles to LLVM format (.profdata) or GCC format (GCOV). So from the format compatibility perspective, I think you need to check the integration with LLVM/GCC profiles. Or you can decide to directly support Linux perf profiles format - in this case, you need to re-implement logic from AutoFDO somewhere in the eprofile-cli tool.

Also, there is a tool to convert Linux perf profiles into the pprof format (the format used by the Go PGO implementation) - https://github.com/google/perf_data_converter . But how well these profiles are suitable for the Go PGO approach still unknown: golang/go#64489

Performance Impact Assessment: Assessing the performance overhead during the profiling phase, especially in a production environment, is crucial -- we always try to make sure to use low-overhead profilers so that the improvement isn't outweighted by the overhead

One of the main goals of AutoFDO is performance impact reduction. With Linux perf you can tweak the "performance overhead <-> PGO profile quality" tradeoff via the perf sample rate configuration. According to the Google AutoFDO paper, AutoFDO overhead is almost invisible for their services in production.

[bboreham]

Marco Ferror gave a talk at GopherCon about how they automate PGO in their CI pipeline.

Unfortunately the videos are not out yet and I can't find an paper that they published.

[zamazan4ik]

I believe we can kindly ping @marcoferrer here for more details :)

[zamazan4ik]

@simonswine at FOSDEM 2024 we discussed this topic a bit and you said to ping you here :)

What can be our next steps in trying to push forward the PGO profiling infrastructure for native languages in Pyroscope? I suppose some kind of (hopefully) helpful information is already here. If you need some more information - feel free to ask me! At least I will try to find an answer or find the right person with an answer :)

[simonswine]

Yes nice to meet you @zamazan4ik and thank you for following up on this topic. I agree that Pyroscope is a good fit for a PGO platform. I think the things we would need make sure is that we can convert the measured profiles without loosing necessary information.

I suggest the next steps should be, ensuring that if we can convert profiles from the target format into pprof and back.

Before any work on Pyroscope itself I think we should do a proof-of-concept, with format converters.

For perf I could only find the perf -> pprof conversion:

https://github.com/google/perf_data_converter
https://pkg.go.dev/rsc.io/perf2pprof@v0.0.0-20160720185217-3aa1cdec6387
https://github.com/grafana/pyroscope/tree/main/pkg/og/convert/perf

Are you aware of anything that converts pprof back to the perf format?

[zamazan4ik]

Unfortunately not, I am not aware of converters from pprof format to perf format (or any other compatible with GCC/Clang/Rustc formats).

By the way, why do we need to have such a conversion? Different conversions between file formats can cause losing some information during the conversion stage; an additional conversion step - for native compilers, in this case, the whole pipeline will be "pprof -> (via perf_data_converter) -> perf -> (via AutoFDO tooling) -> LLVM/GCC formats". Is it possible to support collecting perf profiles directly with Pyroscope?

[simonswine]

Pyroscope will take each profile sent and store it in its database in a deduplicated way.

So each and every values is processed and then at query time reconstructed. The amount of data sent is too big to keep it around individually for each profile.

The actual model of the database used is fairly close to pprof, so for pprof it won't loose information.

[zamazan4ik]

Got it - now it makes sense why we want to convert to/from pprof. However, seems like in this case, we will need to implement "pprof -> perf" converter at first. Probably we do not need even "fully" support perf format - only the subset which is used by AutoFDO but it'll require a bit of research since the only source for AutoFDO required fields in perf profiles is its source code.