GESIS BinderHub server was accumulating Running pods that were more than 1 day old

[rgaiacs]

Around 2023-06-21 17:15 CEST, we launch a cron job to avoid this problem.

Need further investigation to discover the source of the problem.

[minrk]

OVH is seeing this, too. I suspect it's a recent update to jupyterhub/zero-to-jupyterhub that's causing something to get missed.

Two categories of problem to track down:

1. JupyterHub/KubeSpawner is leaving orphan pods (i.e. the pod is running but jupyterhub doesn't have a record of it). symptom: the pods older than 6 hours do not have an associated user.
2. max-age culling is not working properly. This could be because of a bug in start-time reporting from jupyterhub, or a bug in jupyterhub-idle-culler not actually performing the max age culling for some reason

[rgaiacs]

Thanks for the information covering OVH.

[minrk]

I've looked through some logs, and OVH definitely has quite a few orphan pods. So I think this is a change in kubespawner that's making it possible to leave orphaned pods, likely failing to clean up after a failed start (hard to say precisely, because OVH has no log retention, so we can only look back into the very recent past). OVH is showing occasional reflector failure events, which may well be related, because deleting a pod not in the reflector will skip the deletion.

Unfortunately, JupyterHub doesn't give Spawners a hook to look for orphaned resources.

Here's a notebook to collect and view (and clean up, if you want) orphaned pods on a cluster.