[Bug]: datadog-ipc-helper as a service and needrestart

[moerkey]

Bug report

Hello,

I have the pleasure to administrate some web servers where I have to update the Datadog PHP tracer from time to time.

Since version 0.90.0 I have trouble with the added process datadog-ipc-helper and needrestart. After installing the new version, some services want to get restarted all the time. After the update of Datadog PHP Tracer the tool needrestart complains about salt-minion.

root@SERVER:~# needrestart -qbn
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-88-generic
NEEDRESTART-KEXP: 5.15.0-88-generic
NEEDRESTART-KSTA: 1
NEEDRESTART-SVC: salt-minion.service

root@SERVER:~# needrestart -b -m a -vvv
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.5
[main] running in root mode
[main] systemd detected
[main] vm detected
NEEDRESTART-VER: 3.5
[Core] #615 is a NeedRestart::Interp::Python
[Python] #615: source=/usr/bin/networkd-dispatcher
[main] #132576 uses obsolete binary /memfd:spawn_worker_trampoline
[main] #132576 is not a child
[main] #132576 exe => /memfd:spawn_worker_trampoline
[main] trying systemctl status
[main] #132576 is salt-minion.service
[main] inside container or vm, skipping microcode checks
[Kernel] Linux: kernel release 5.15.0-88-generic, kernel version #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023
Failed to load NeedRestart::Kernel::kFreeBSD: [Kernel/kFreeBSD] Not running on GNU/kFreeBSD!
[Kernel/Linux] /boot/vmlinuz-5.15.0-88-generic => 5.15.0-88-generic (buildd@lcy02-amd64-058) #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 [5.15.0-88-generic]*
[Kernel/Linux] /boot/vmlinuz-5.15.0-87-generic => 5.15.0-87-generic (buildd@lcy02-amd64-011) #97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023 [5.15.0-87-generic]
[Kernel/Linux] Expected linux version: 5.15.0-88-generic
NEEDRESTART-KCUR: 5.15.0-88-generic
NEEDRESTART-KEXP: 5.15.0-88-generic
NEEDRESTART-KSTA: 1
NEEDRESTART-SVC: salt-minion.service

But why salt-minion do you ask, because the systemd thinks datadog-ipc-helper is correlated to the salt-minion daemon.

root@SERVER:~# systemctl status salt-minion.service 
● salt-minion.service - The Salt Minion
     Loaded: loaded (/lib/systemd/system/salt-minion.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2023-11-14 09:19:13 CET; 13min ago
       Docs: man:salt-minion(1)
             file:///usr/share/doc/salt/html/contents.html
             https://docs.saltproject.io/en/latest/contents.html
   Main PID: 145819 (python3.10)
      Tasks: 9 (limit: 9437)
     Memory: 148.3M
        CPU: 2.907s
     CGroup: /system.slice/salt-minion.service
             ├─132576 datadog-ipc-helper "" /usr/lib/php/20200930/ddtrace.so /proc/self/fd/4 ddog_daemon_entry_point
             ├─145819 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
             └─145826 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager"

root@SERVER:~# systemctl status
...
             ├─salt-minion.service 
             │ ├─132576 datadog-ipc-helper  /usr/lib/php/20200930/ddtrace.so /proc/self/fd/4 ddog_daemon_entry_point
             │ ├─145819 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion
             │ └─145826 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager
...

root@SERVER:~# ps aux | grep datadog-ipc-helpe[r]
root      132576  1.1  0.3 112340 24840 ?        Ssl  09:06   0:23 datadog-ipc-helper  /usr/lib/php/20200930/ddtrace.so /proc/self/fd/4 ddog_daemon_entry_point

root@SERVER:~# pstree
systemd-+-VGAuthService
        |-agent---10*[{agent}]
        |-agetty
        |-atd
        |-beam.smp-+-erl_child_setup-+-inet_gethost---inet_gethost
        |          |                 `-sh
        |          `-23*[{beam.smp}]
        |-check_mk_agent---sleep
        |-cmk-agent-ctl---3*[{cmk-agent-ctl}]
        |-cron
        |-dbus-daemon
        |-dd-ipc-helper
        |-ddappsec-helper---12*[{ddappsec-helper}]
...

But this behaviour changes after a reboot. Then a PHP process should be restarted over and over again.

root@SERVER:~# needrestart -qbn
NEEDRESTART-VER: 3.5
NEEDRESTART-KCUR: 5.15.0-88-generic
NEEDRESTART-KEXP: 5.15.0-88-generic
NEEDRESTART-KSTA: 1
NEEDRESTART-SVC: php8.0-fpm.service

root@SERVER:~# needrestart -b -m a -vvv
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.5
[main] running in root mode
[main] systemd detected
[main] vm detected
NEEDRESTART-VER: 3.5
[Core] #615 is a NeedRestart::Interp::Python
[Python] #615: source=/usr/bin/networkd-dispatcher
[main] #137881 uses obsolete binary /memfd:spawn_worker_trampoline
[main] #137881 is not a child
[main] #137881 exe => /memfd:spawn_worker_trampoline
[main] trying systemctl status
[main] #137881 is php8.0-fpm.service
[main] inside container or vm, skipping microcode checks
[Kernel] Linux: kernel release 5.15.0-88-generic, kernel version #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023
[Kernel/Linux] /boot/vmlinuz-5.15.0-88-generic => 5.15.0-88-generic (buildd@lcy02-amd64-058) #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 [5.15.0-88-generic]*
[Kernel/Linux] /boot/vmlinuz-5.15.0-87-generic => 5.15.0-87-generic (buildd@lcy02-amd64-011) #97-Ubuntu SMP Mon Oct 2 21:09:21 UTC 2023 [5.15.0-87-generic]
[Kernel/Linux] Expected linux version: 5.15.0-88-generic
NEEDRESTART-KCUR: 5.15.0-88-generic
NEEDRESTART-KEXP: 5.15.0-88-generic
NEEDRESTART-KSTA: 1
NEEDRESTART-SVC: php8.0-fpm.service

root@SERVER:~# systemctl status
...
             ├─php8.0-fpm.service 
             │ ├─137869 php-fpm: master process (/etc/php/8.0/fpm/php-fpm.conf)
             │ ├─137881 datadog-ipc-helper  /usr/lib/php/20200930/ddtrace.so /proc/self/fd/4 ddog_daemon_entry_point
             │ ├─137885 /opt/datadog/dd-library/0.93.2/bin/ddappsec-helper --lock_path - --socket_path fd:11
             │ ├─288524 php-fpm: pool www
...


root@SERVER:~# ps aux | grep datadog-ipc-helpe[r]
www-data  137881  2.9  0.5 132936 45404 ?        Ss   09:07   4:44 datadog-ipc-helper  /usr/lib/php/20200930/ddtrace.so /proc/self/fd/4 ddog_daemon_entry_point

root@SERVER:~# pstree
systemd-+-VGAuthService
        |-agent---10*[{agent}]
        |-agetty
        |-atd
        |-beam.smp-+-erl_child_setup-+-inet_gethost---inet_gethost
        |          |                 `-sh
        |          `-23*[{beam.smp}]
        |-check_mk_agent---sleep
        |-cmk-agent-ctl---3*[{cmk-agent-ctl}]
        |-cron
        |-dbus-daemon
        |-dd-ipc-helper
        |-ddappsec-helper---45*[{ddappsec-helper}]
...

Some version infos.

root@SERVER:~# cat /etc/issue
Ubuntu 22.04.3 LTS \n \l

root@SERVER:~# dpkg -l | grep fpm
ii  php7.3-fpm                            7.3.33-14+ubuntu22.04.1+deb.sury.org+1       amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php7.4-fpm                            1:7.4.33-8+ubuntu22.04.1+deb.sury.org+1      amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php8.0-fpm                            1:8.0.30-2+ubuntu22.04.1+deb.sury.org+1      amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)

root@SERVER:~# ls -lh /opt/datadog/dd-library/
total 20K
drwxr-xr-x 5 root root 4,0K Jul  6 06:19 0.89.0
drwxr-xr-x 5 root root 4,0K Jul 19 06:07 0.90.0
drwxr-xr-x 5 root root 4,0K Sep 25 12:07 0.91.2
drwxr-xr-x 5 root root 4,0K Nov  2 15:35 0.93.1
drwxr-xr-x 5 root root 4,0K Nov 14 09:06 0.93.2

root@SERVER:~# needrestart --version

needrestart 3.5 - Restart daemons after library updates.
...
Yes, there is needrestart 3.6 already but the error is still the same.

So I would question how data datadog-ipc-helper will be started.
Can it be deactivated?
Why can systemd not get the correct parent of datadog-ipc-helper?
Should there be a parent?
And last but not least, why is needrestart thinking that datadog-ipc-helper needs to get restarted?

PHP version

8.0.30

Tracer or profiler version

0.93.2

Installed extensions

[PHP Modules]
amqp
bcmath
calendar
Core
ctype
curl
date
ddappsec
ddtrace
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
hash
iconv
igbinary
imagick
intl
json
libxml
mbstring
mongodb
openssl
pcntl
pcre
PDO
pdo_sqlsrv
Phar
posix
readline
redis
Reflection
session
shmop
SimpleXML
sockets
sodium
SPL
sqlsrv
standard
sysvmsg
sysvsem
sysvshm
tidy
tokenizer
xml
xmlreader
xmlrpc
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
ddappsec
ddtrace

Output of phpinfo()

No response

Upgrading from

0.89.0 > 0.90.0 > 0.91.2 > 0.93.1 > 0.93.2

[bwoebi]

Hey @moerkey,

the correct parent process of datadog-ipc-helper would be systemd itself, being pid 1; i.e. that's what happens when a process is daemonized (forks itself twice and kills the first fork).

Systemd thinks the datadog-ipc-helper belongs to php8.0-fpm.service as both share the same cgroup, given that a php-fpm process is starting it.

I'm not sure why systemd puts the datadog-ipc-helper into /system.slice/salt-minion.service for you though, after the original cgroup terminates.

As far as I have researched, it doesn't seem to be possible to put the datadog-ipc-helper into another cgroup (generally) as the fpm and apache processes are running as www-data, but nearly everything needs root and it's not like there's an user session active for www-data either, which could inherit it.

I am not sure what or whether we're able to do about this.

The process can currently be disabled by disabling telemetry, but for example ASM also relies on that telemetry, and in future more things will be sent through it.

[moerkey]

Hey @bwoebi,

thank you for the explanation. At first, I would fix that the process does not run as root (user who installed the app) after an installation or update of Datadog PHP tracer. It should run as www-data right? Currently, I need to reboot the system that it runs as this user.

Out of curiosity. How does the process start itself, or who starts it?
And where is the binary? I could not find it on disk.

Cheers

[bwoebi]

It does not matter as which user the daemon runs, and can be communicated with by any user over its shared memory unix socket.
It simply is spawned by the first php process, i.e. the first time php with the extension code is executed.