30-40% CPU usage increase with 8.2.8-bookworm

[tlamy]

We recently updated from 8.2.7-fpm to 8.2.8-fpm, where base system changed from bullseye to bookworm.
After a bit fiddling to re-enable blowfish encryption in OpenSSL 3, I noticed heavily increased cpu usage in all our deployments.
After basing our images on 8.2-fpm-bullseye (plus build and deploy), CPU usage is down to normal again.

What could have caused that behavior?
How long do you plan to provide bullseye based images?
What can I do to help pin down the problem?

[tlamy]

We deployed on 11:55.

Neither Dockerfiles nor code have changed significantly, and CPU usage was comparable with as we used 8.2.7 (bullseye)

[yosifkit]

How long do you plan to provide bullseye based images?

We'll have bullseye based images until the next Debian stable major release (trixie). It doesn't have a release date but, judging by past releases, it'll likely be around mid 2025 (https://wiki.debian.org/DebianReleases).

What could have caused that behavior?

Not sure. We've had issues in the past that our images are slower than the equivalent Debian packages but have been unable to pinpoint exactly why (#493).

What can I do to help pin down the problem?

You could test to see if the Debian PHP 8.2 packages in Bookworm exhibit the same amount of CPU usage. If they do have similar usage, then maybe a package update in Bookworm is the problem, otherwise it might be something in the way we compile PHP 😢.

[tlamy]

Just a small heads-up, I'm still at it trying to pinpoint the original cause.
I managed to create an image using Ondrey Sury's PHP packages for Debian, but still struggling to create some test code, since I don't want to test in production.

[magnetik]

I face the same problem. Not sure on how to pinpoint the origin.

[pereorga]

@magnetik @tlamy can you produce a simple example to try to reproduce the issue?

[particleflux]

Similar issue, but not restricted to this docker image.
We run an image based on debian:bookworm-20230919-slim directly. In general, switched from 8.0 to 8.2 (and upgraded from debian buster to bookworm base image).

So I guess it's either bookworm related, or php-fpm 8.2 related, or the packaging by deb.sury.org (which disabled JIT between those: oerdnj/deb.sury.org#1924)

Unfortunately, can't easily reproduce it either, it only happens on our prod env, with like 300 requests/min

[GreenReaper]

Seems like the thing to do is try re-enabling the tracing JIT and giving it a suitable buffer depending on the codebase, assuming this can be done without too much trouble.

[yuqi]

I have encountered the same issue. The same problem occurred after we upgraded from php:7.4-fpm-bullseye to php:8.1-fpm (bookworm), where P95 and AVG response time increased. Finally, it returned to normal after using php:8.1-fpm-bullseye.