Debugging messages from spawners lost when restarting hub

[Frank-Steiner]

Bug description

As we had problems with the jupyterlab processes spawned from jupyterhub, we added
c.Spawner.args = ["--debug"]
to jupyterhub_config.py to get debugging messages from the singleuser processes in the systemd journal.

This does work when we set this option, then restart jupyterhub, then stop and start the servers. Then we see /usr/bin/python3.10 /opt/jhub/bin/jupyterhub-singleuser --debug in the linux process list and get all debugging messages in the journal.

But if we restart jupyterhub again afterwards, the debugging output gets lost. The jupyterhub-singleuser process keeps running and can be used in the browser, but its debugging messages no longer are caught by the jupyterhub process and thus no longer sent to the systemd journal.

Not sure if this is a bug or if it's just not possible to reestablish the connection to the output channel of the singleuser processes.

To workaround, I changed our sudospawner-singleuser script so that it calls
exec "$(dirname "$0")/jupyterhub-singleuser" $@ 2>&1 | /usr/bin/logger -t jupyterlab

This way the singleuser processes do the logging to the journal themselves and that keeps working after restarting jupyterhub.

So I've two questions:
a) Is the described behaviour a bug or is just impossible to further collect output of the singleuser processes after restarting jupyterhub (because of broken pipes or sth...)?
b) Could my workaround cause problems because I redirect stdout/stderr on my own?

Your personal set up

• OS: SuSE Linux Enterprise 15 SP4

• Version(s): jupyterhub 3.1.1, python 3.10