Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO) improvements on multiple projects. The results are available here. According to the tests, these optimizations can help with achieving better performance in many cases for many applications: networking apps like HAproxy and Envoy, many databases, parsing utilities like Vector, etc. Since this, I think trying to optimize pgbouncer with PGO and PLO can be a good idea. Having a faster pgbouncer can help with serving more users per CPU core with pgbouncer and reduce the Total Cost of Ownership (TCO) for the solution. According to the graphics in the issue pgbouncer's CPU consumption can be quite huge sometimes.

I can suggest the following action points:

• Perform PGO benchmarks on pgbouncer. If it shows improvements - add a note to the documentation about possible improvements in pgbouncer performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize pgbouncer according to their workloads.
• Optimize pre-built pgbouncer binaries

Testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too (Clang and Rustc already use BOLT as an addition to PGO) but I recommend starting from the usual PGO.

Examples of how PGO optimization is integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I have some examples of how PGO information looks in the documentation:

• ClickHouse: https://clickhouse.com/docs/en/operations/optimizing-performance/profile-guided-optimization
• Databend: https://databend.rs/doc/contributing/pgo
• Vector: https://vector.dev/docs/administration/tuning/pgo/
• Nebula: https://docs.nebula-graph.io/3.5.0/8.service-tuning/enable_autofdo_for_nebulagraph/
• GCC: Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
• Clang:
  • https://llvm.org/docs/HowToBuildWithPGO.html
  • https://llvm.org/docs/AdvancedBuilds.html
• tsv-utils: https://github.com/eBay/tsv-utils/blob/master/docs/BuildingWithLTO.md

Regarding LLVM BOLT integration, I have the following examples:

• Rustc:
  • Rustc itself (GitHub PR)
  • LLVM in Rustc (Reddit)
• CPython: GitHub PR
• YDB: GitHub comment
• Clang:
  • Slides
  • Results on building Clang
  • Linaro results
  • on AMD 7950X3D
  • GitHub comment from LLVM
• LDC: GitHub comment
• HHVM, Proxygen and others: Facebook paper
• NodeJS: Blog
• Chromium: Blog
• MySQL, MongoDB, memcached, Verilator: Paper

I hope these ideas could be interesting for someone from the pgbouncer community.