Error: No space left on device

[veenstrajelmer]

Bug description

Found built image, launching...
Launching server...
Server requested
2023-07-26T07:41:31.961145Z [Normal] Successfully assigned ovh2/jupyter-deltares-2ddfm-5ftools-2dwwkth8hu to user-202211a-node-7cb28d
2023-07-26T07:41:33Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef52" already present on machine
2023-07-26T07:41:33Z [Normal] Created container tc-init
2023-07-26T07:41:34Z [Warning] Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/5836d2ab87a4df94658270fc3f27d2a445ffbc1dfe7946ffef17c527a7b00158/resolv.conf" to rootfs at "/etc/resolv.conf": open /run/containerd/io.containerd.runtime.v2.task/k8s.io/tc-init/rootfs/etc/resolv.conf: no space left on device: unknown
2023-07-26T07:41:35Z [Warning] Error: write /var/lib/kubelet/pods/129e7996-a61c-483a-ba70-5dadf255bc81/etc-hosts: no space left on device
2023-07-26T07:41:49Z [Normal] Started container tc-init
2023-07-26T07:41:50Z [Normal] Pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:13738de3a180168e9d216910e0c9fb350c766a88"

If you leave it going (different session):

Found built image, launching...
Launching server...
Server requested
2023-07-26T08:58:26.703344Z [Normal] Successfully assigned ovh2/jupyter-deltares-2ddfm-5ftools-2dz158yt5f to user-202211a-node-7cb28d
2023-07-26T08:58:26Z [Warning] error making pod data directories: mkdir /var/lib/kubelet/pods/6ceefea3-00cd-4c0c-8989-a59a0075939c: no space left on device
2023-07-26T08:58:27Z [Warning] MountVolume.SetUp failed for volume "etc-jupyter" : mkdir /var/lib/kubelet/pods/6ceefea3-00cd-4c0c-8989-a59a0075939c: no space left on device
2023-07-26T08:58:27Z [Warning] MountVolume.SetUp failed for volume "etc-jupyter-templates" : mkdir /var/lib/kubelet/pods/6ceefea3-00cd-4c0c-8989-a59a0075939c: no space left on device
2023-07-26T08:58:38Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef52" already present on machine
2023-07-26T08:58:38Z [Normal] Created container tc-init
2023-07-26T08:58:39Z [Normal] Started container tc-init
2023-07-26T08:58:40Z [Normal] Pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1"
2023-07-26T09:02:50Z [Warning] Failed to pull image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": rpc error: code = Unknown desc = failed to pull and unpack image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": mkdir /var/lib/containerd/io.containerd.content.v1.content/ingest/3a8a8b89fac5a50b800e045e34bbbfb33e3c225b46ba3394c2d2a2c0922eea34: no space left on device
2023-07-26T09:02:50Z [Warning] Error: ErrImagePull
2023-07-26T09:02:50Z [Normal] Back-off pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1"
2023-07-26T09:02:50Z [Warning] Error: ImagePullBackOff
Spawn failed: Timeout
Launch attempt 1 failed, retrying...
Server requested
2023-07-26T09:03:31.890127Z [Normal] Successfully assigned ovh2/jupyter-deltares-2ddfm-5ftools-2d084i2yxa to user-202211a-node-7cb28d
2023-07-26T09:03:33Z [Normal] Container image "jupyterhub/mybinder.org-tc-init:2020.12.4-0.dev.git.4289.h140cef52" already present on machine
2023-07-26T09:03:33Z [Normal] Created container tc-init
2023-07-26T09:03:33Z [Normal] Started container tc-init
2023-07-26T09:03:33Z [Normal] Pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1"
2023-07-26T09:03:51Z [Warning] Failed to pull image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": rpc error: code = Unknown desc = failed to pull and unpack image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": write /var/lib/containerd/io.containerd.content.v1.content/ingest/3a8a8b89fac5a50b800e045e34bbbfb33e3c225b46ba3394c2d2a2c0922eea34/total: no space left on device
2023-07-26T09:03:51Z [Warning] Error: ErrImagePull
2023-07-26T09:03:52Z [Normal] Back-off pulling image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1"
2023-07-26T09:03:52Z [Warning] Error: ImagePullBackOff
2023-07-26T09:04:13Z [Warning] Failed to pull image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": rpc error: code = Unknown desc = failed to pull and unpack image "2lmrrh8f.gra7.container-registry.ovh.net/mybinder-builds/r2d-g5b5b759deltares-2ddfm-5ftools-eff2ec:1b57095685a23c3ccf102ade4d9b541a4ba6edd1": failed to copy: write /var/lib/containerd/io.containerd.content.v1.content/ingest/71d139a184e5d56e6da79e7a22aeccc5c8fc625d0f1a4d1239b1666ff23e7c25/data: no space left on device

Expected behaviour

Crash or automatically assigning a new pod.

Actual behaviour

The launching process continues but it hangs.

How to reproduce

This happens regularly when opening a binder for my dfm_tools repos: https://mybinder.org/v2/gh/Deltares/dfm_tools/v0.13.1?urlpath=/tree/docs/notebooks
However, since I expect this to be caused by limited disk space on the assinged pod, the error is a bit arbitrary and therefore not super reproducible.

Full environment

https://github.com/Deltares/dfm_tools/blob/v0.13.1/setup.cfg

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉