Zulip botserver in k8s does not forward bot logs

[gndrmnn]

We have successfully deployed the Zulip botserver using kubernetes and created a first bot with it. This bot and the botserver work nicely and communicate without a problem with our Zulip instance.

We are using the Python builtin logger inside the bot with self.logger = logging.getLogger(__name__) and then calls to self.logger.info etc.
However, these logging messages are somehow suppressed by the botserver. If we do a kubectl logs <botserver-podname> we simply get something like this:

 * Serving Flask app 'zulip_botserver.server'
 * Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on <redacted>
 * Running on <redacted>
Press CTRL+C to quit
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: <redacted>
<redacted> - - [23/Jan/2024 14:56:26] "POST / HTTP/1.1" 200 -

which contains all the expected log messages from flask/werkzeug but none of the ones we are logging in the bot code. Kubernets should forward anything that happens on stdout/stderr automatically, so I am really surprised by this result.

Note: Logging works like a charm when using the single zulip-run-bot command.

I would also expect to find bot logs in the botserver log.

[0xUgochukwu]

I've experienced this issue before while working with Kubernetes and Python, this is because Kubernetes is buffering the output of the application.

As with all buffers this will the output will be released/displayed as soon as a flush is invoked on the buffer or its limit is reached. You can prevent this default buffering by setting the environment variable PYTHONUNBUFFERED=1

This should solve the problem, but if it doesn't there are other things to look at like the encoding of the output etc.

To understand more of how this works: PYTHONUNBUFFERED Explained