nvproxy: unknown control command 0x3d05 #10413
Comments
|
The reproduction program is almost identical to the one in #9827, which is why I revisited that issue's test. |
|
This seems to be running fine for me on an A100-40GB machine in GCE on driver version Please note:
So maybe you are using a different driver version? Or maybe something to do with the Oracle Cloud environment? |
We have On H100 worker: We use the same driver version across all GPU workers. |
|
Updated driver version and still can not repro the failure on my GCE VM: Surprisingly, this workload gets stuck without gVisor. I will add |
Interesting. This may be the same problem as in #9827 where the test got stuck on The program doesn't get stuck on
🙏 |
copybara-service bot
pushed a commit
that referenced
this issue
May 9, 2024
Updates #10413 PiperOrigin-RevId: 632173012
copybara-service bot
pushed a commit
that referenced
this issue
May 9, 2024
Updates #10413 PiperOrigin-RevId: 632173012
copybara-service bot
pushed a commit
that referenced
this issue
May 9, 2024
Updates #10413 PiperOrigin-RevId: 632173012
copybara-service bot
pushed a commit
that referenced
this issue
May 9, 2024
Updates #10413 PiperOrigin-RevId: 632173012
copybara-service bot
pushed a commit
that referenced
this issue
May 9, 2024
Updates #10413 PiperOrigin-RevId: 632277477
|
@thundergolfer Let me know if e9b3218 fixes the issue. If so, please close this. |
copybara-service bot
pushed a commit
that referenced
this issue
May 13, 2024
This is helpful for handling parameter types that have one field for frontend FD that needs to be translated (and are simple apart from that). Avoids repetitive code. Rename HasRMCtrlFD->HasFrontendFD so it can have a broader meaning. Implement generic handlers for frontend ioctl and control commands. Updates #10413. PiperOrigin-RevId: 633238248
copybara-service bot
pushed a commit
that referenced
this issue
May 14, 2024
This is helpful for handling parameter types that have one field for frontend FD that needs to be translated (and are simple apart from that). Avoids repetitive code. Rename HasRMCtrlFD->HasFrontendFD so it can have a broader meaning. Implement generic handlers for frontend ioctl and control commands. Updates #10413. PiperOrigin-RevId: 633238248
copybara-service bot
pushed a commit
that referenced
this issue
May 14, 2024
This is helpful for handling parameter types that have one field for frontend FD that needs to be translated (and are simple apart from that). Avoids repetitive code. Rename HasRMCtrlFD->HasFrontendFD so it can have a broader meaning. Implement generic handlers for frontend ioctl and control commands. Updates #10413. PiperOrigin-RevId: 633238248
copybara-service bot
pushed a commit
that referenced
this issue
May 14, 2024
This is helpful for handling parameter types that have one field for frontend FD that needs to be translated (and are simple apart from that). Avoids repetitive code. Rename HasRMCtrlFD->HasFrontendFD so it can have a broader meaning. Implement generic handlers for frontend ioctl and control commands. Updates #10413. PiperOrigin-RevId: 633700330
copybara-service bot
pushed a commit
that referenced
this issue
May 21, 2024
…EXPORT_OBJECT_INFO, NV0000_CTRL_CMD_OS_UNIX_IMPORT_OBJECT_FROM_FD, NV0041_CTRL_CMD_GET_SURFACE_INFO Following up on #10413 (comment). Ayush's fix revealed more missing commands. With these changes, the reproduction in #10413 _still does not work._ Here's an updated reproduction Dockerfile that crashes because of the SIGCHILD handler. Without the SIGCHILD handler the program hangs. ```Dockerfile FROM python:3.11-slim-bookworm RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim RUN wget https://bootstrap.pypa.io/get-pip.py RUN python3 get-pip.py RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above. RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler COPY <<EOF repro.py print("Hello from inside container.") import psutil current_process = psutil.Process() parent_process = current_process.parent() print(f"Processes: {current_process=} {parent_process=}") import time import torch import torch.nn as nn import torch.nn.functional as F import lightning as L from memory_profiler import profile from torchvision.datasets import CIFAR100 from torchvision import transforms from torchvision import models from torch.utils.data import DataLoader import os import signal import pathlib def handler(signum, frame): print('Signal handler called with signal', signum) os.waitpid(-1, 0) raise KeyboardInterrupt() # gVisor is ignoring the SIGCHILD 'Discarding ignored signal 17' signal.signal(signal.SIGCHLD, handler) class MagixNet(L.LightningModule): def __init__(self, nbr_cat): super().__init__() module = models.resnet50(weights=models.ResNet50_Weights.DEFAULT) module.fc = nn.Linear(2048, nbr_cat) self.module = module def forward(self, x): return self.module(x) def training_step(self, batch, batch_idx): x, y = batch y_hat = self(x) loss = F.cross_entropy(y_hat, y) return loss def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.02) def prepare_data(): pipeline = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), ]) train_ds = CIFAR100('data', train=True, download=True, transform=pipeline) train_dl = DataLoader(train_ds, batch_size=128, num_workers=4) val_ds = CIFAR100('data', train=False, download=True, transform=pipeline) val_dl = DataLoader(val_ds, batch_size=128, num_workers=4) return train_dl, val_dl if __name__ == "__main__": torch.set_float32_matmul_precision('medium') train_dl, val_dl = prepare_data() model = MagixNet(100) trainer = L.Trainer(max_epochs=1, strategy="ddp_notebook") start = time.time() trainer.fit(model, train_dl, val_dl) print(f"Training duration (seconds): {time.time() - start:.2f}") nccl_debug_file = pathlib.Path("/tmp/runsc-nccl.txt") if nccl_debug_file.exists(): print("NCCL Debugging") print(nccl_debug_file.read_text()) EOF ENTRYPOINT ["python3", "repro.py"] ``` Run like this: ``` sudo docker run --runtime=runsc-2 --shm-size=1000GB --gpus '"device=GPU-48070a35-b2ea-643c-eebe-0c55d2a541a4,GPU-8061048a-aa0f-76bd-457b-71c6be60386e"' -e NCCL_DEBUG=INFO -e NCCL_DEBUG_FILE="/tmp/runsc-nccl.txt" sha256:1c1fc535214ec1111b46a87fe20558e7c078185e4158c3ce253dc56a5a9be628 ``` **`/etc/docker/daemon.json`** ``` { "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }, "runsc-2": { "path": "/home/modal/runsc2", "runtimeArgs": ["--nvproxy", "--nvproxy-docker", "-debug-log=/tmp/runsc-2/", "-debug", "-strace"] }, } } ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10434 from thundergolfer:master 76bf495 PiperOrigin-RevId: 635812044
copybara-service bot
pushed a commit
that referenced
this issue
May 21, 2024
…EXPORT_OBJECT_INFO, NV0000_CTRL_CMD_OS_UNIX_IMPORT_OBJECT_FROM_FD, NV0041_CTRL_CMD_GET_SURFACE_INFO Following up on #10413 (comment). Ayush's fix revealed more missing commands. With these changes, the reproduction in #10413 _still does not work._ Here's an updated reproduction Dockerfile that crashes because of the SIGCHILD handler. Without the SIGCHILD handler the program hangs. ```Dockerfile FROM python:3.11-slim-bookworm RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim RUN wget https://bootstrap.pypa.io/get-pip.py RUN python3 get-pip.py RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above. RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler COPY <<EOF repro.py print("Hello from inside container.") import psutil current_process = psutil.Process() parent_process = current_process.parent() print(f"Processes: {current_process=} {parent_process=}") import time import torch import torch.nn as nn import torch.nn.functional as F import lightning as L from memory_profiler import profile from torchvision.datasets import CIFAR100 from torchvision import transforms from torchvision import models from torch.utils.data import DataLoader import os import signal import pathlib def handler(signum, frame): print('Signal handler called with signal', signum) os.waitpid(-1, 0) raise KeyboardInterrupt() # gVisor is ignoring the SIGCHILD 'Discarding ignored signal 17' signal.signal(signal.SIGCHLD, handler) class MagixNet(L.LightningModule): def __init__(self, nbr_cat): super().__init__() module = models.resnet50(weights=models.ResNet50_Weights.DEFAULT) module.fc = nn.Linear(2048, nbr_cat) self.module = module def forward(self, x): return self.module(x) def training_step(self, batch, batch_idx): x, y = batch y_hat = self(x) loss = F.cross_entropy(y_hat, y) return loss def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.02) def prepare_data(): pipeline = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), ]) train_ds = CIFAR100('data', train=True, download=True, transform=pipeline) train_dl = DataLoader(train_ds, batch_size=128, num_workers=4) val_ds = CIFAR100('data', train=False, download=True, transform=pipeline) val_dl = DataLoader(val_ds, batch_size=128, num_workers=4) return train_dl, val_dl if __name__ == "__main__": torch.set_float32_matmul_precision('medium') train_dl, val_dl = prepare_data() model = MagixNet(100) trainer = L.Trainer(max_epochs=1, strategy="ddp_notebook") start = time.time() trainer.fit(model, train_dl, val_dl) print(f"Training duration (seconds): {time.time() - start:.2f}") nccl_debug_file = pathlib.Path("/tmp/runsc-nccl.txt") if nccl_debug_file.exists(): print("NCCL Debugging") print(nccl_debug_file.read_text()) EOF ENTRYPOINT ["python3", "repro.py"] ``` Run like this: ``` sudo docker run --runtime=runsc-2 --shm-size=1000GB --gpus '"device=GPU-48070a35-b2ea-643c-eebe-0c55d2a541a4,GPU-8061048a-aa0f-76bd-457b-71c6be60386e"' -e NCCL_DEBUG=INFO -e NCCL_DEBUG_FILE="/tmp/runsc-nccl.txt" sha256:1c1fc535214ec1111b46a87fe20558e7c078185e4158c3ce253dc56a5a9be628 ``` **`/etc/docker/daemon.json`** ``` { "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }, "runsc-2": { "path": "/home/modal/runsc2", "runtimeArgs": ["--nvproxy", "--nvproxy-docker", "-debug-log=/tmp/runsc-2/", "-debug", "-strace"] }, } } ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10434 from thundergolfer:master bf18079 PiperOrigin-RevId: 635812044
copybara-service bot
pushed a commit
that referenced
this issue
May 21, 2024
…EXPORT_OBJECT_INFO, NV0000_CTRL_CMD_OS_UNIX_IMPORT_OBJECT_FROM_FD, NV0041_CTRL_CMD_GET_SURFACE_INFO Following up on #10413 (comment). Ayush's fix revealed more missing commands. With these changes, the reproduction in #10413 _still does not work._ Here's an updated reproduction Dockerfile that crashes because of the SIGCHILD handler. Without the SIGCHILD handler the program hangs. ```Dockerfile FROM python:3.11-slim-bookworm RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim RUN wget https://bootstrap.pypa.io/get-pip.py RUN python3 get-pip.py RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above. RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler COPY <<EOF repro.py print("Hello from inside container.") import psutil current_process = psutil.Process() parent_process = current_process.parent() print(f"Processes: {current_process=} {parent_process=}") import time import torch import torch.nn as nn import torch.nn.functional as F import lightning as L from memory_profiler import profile from torchvision.datasets import CIFAR100 from torchvision import transforms from torchvision import models from torch.utils.data import DataLoader import os import signal import pathlib def handler(signum, frame): print('Signal handler called with signal', signum) os.waitpid(-1, 0) raise KeyboardInterrupt() # gVisor is ignoring the SIGCHILD 'Discarding ignored signal 17' signal.signal(signal.SIGCHLD, handler) class MagixNet(L.LightningModule): def __init__(self, nbr_cat): super().__init__() module = models.resnet50(weights=models.ResNet50_Weights.DEFAULT) module.fc = nn.Linear(2048, nbr_cat) self.module = module def forward(self, x): return self.module(x) def training_step(self, batch, batch_idx): x, y = batch y_hat = self(x) loss = F.cross_entropy(y_hat, y) return loss def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.02) def prepare_data(): pipeline = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), ]) train_ds = CIFAR100('data', train=True, download=True, transform=pipeline) train_dl = DataLoader(train_ds, batch_size=128, num_workers=4) val_ds = CIFAR100('data', train=False, download=True, transform=pipeline) val_dl = DataLoader(val_ds, batch_size=128, num_workers=4) return train_dl, val_dl if __name__ == "__main__": torch.set_float32_matmul_precision('medium') train_dl, val_dl = prepare_data() model = MagixNet(100) trainer = L.Trainer(max_epochs=1, strategy="ddp_notebook") start = time.time() trainer.fit(model, train_dl, val_dl) print(f"Training duration (seconds): {time.time() - start:.2f}") nccl_debug_file = pathlib.Path("/tmp/runsc-nccl.txt") if nccl_debug_file.exists(): print("NCCL Debugging") print(nccl_debug_file.read_text()) EOF ENTRYPOINT ["python3", "repro.py"] ``` Run like this: ``` sudo docker run --runtime=runsc-2 --shm-size=1000GB --gpus '"device=GPU-48070a35-b2ea-643c-eebe-0c55d2a541a4,GPU-8061048a-aa0f-76bd-457b-71c6be60386e"' -e NCCL_DEBUG=INFO -e NCCL_DEBUG_FILE="/tmp/runsc-nccl.txt" sha256:1c1fc535214ec1111b46a87fe20558e7c078185e4158c3ce253dc56a5a9be628 ``` **`/etc/docker/daemon.json`** ``` { "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }, "runsc-2": { "path": "/home/modal/runsc2", "runtimeArgs": ["--nvproxy", "--nvproxy-docker", "-debug-log=/tmp/runsc-2/", "-debug", "-strace"] }, } } ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10434 from thundergolfer:master bf18079 PiperOrigin-RevId: 635812044
copybara-service bot
pushed a commit
that referenced
this issue
May 21, 2024
…EXPORT_OBJECT_INFO, NV0000_CTRL_CMD_OS_UNIX_IMPORT_OBJECT_FROM_FD, NV0041_CTRL_CMD_GET_SURFACE_INFO Following up on #10413 (comment). Ayush's fix revealed more missing commands. With these changes, the reproduction in #10413 _still does not work._ Here's an updated reproduction Dockerfile that crashes because of the SIGCHILD handler. Without the SIGCHILD handler the program hangs. ```Dockerfile FROM python:3.11-slim-bookworm RUN apt-get update && apt-get install --yes python3 python3-distutils clang wget vim RUN wget https://bootstrap.pypa.io/get-pip.py RUN python3 get-pip.py RUN python3 -m pip install clang~=10.0.1 # must match version of `clang` installed above. RUN python3 -m pip install --ignore-installed torch torchvision lightning numpy memory_profiler COPY <<EOF repro.py print("Hello from inside container.") import psutil current_process = psutil.Process() parent_process = current_process.parent() print(f"Processes: {current_process=} {parent_process=}") import time import torch import torch.nn as nn import torch.nn.functional as F import lightning as L from memory_profiler import profile from torchvision.datasets import CIFAR100 from torchvision import transforms from torchvision import models from torch.utils.data import DataLoader import os import signal import pathlib def handler(signum, frame): print('Signal handler called with signal', signum) os.waitpid(-1, 0) raise KeyboardInterrupt() # gVisor is ignoring the SIGCHILD 'Discarding ignored signal 17' signal.signal(signal.SIGCHLD, handler) class MagixNet(L.LightningModule): def __init__(self, nbr_cat): super().__init__() module = models.resnet50(weights=models.ResNet50_Weights.DEFAULT) module.fc = nn.Linear(2048, nbr_cat) self.module = module def forward(self, x): return self.module(x) def training_step(self, batch, batch_idx): x, y = batch y_hat = self(x) loss = F.cross_entropy(y_hat, y) return loss def configure_optimizers(self): return torch.optim.Adam(self.parameters(), lr=0.02) def prepare_data(): pipeline = transforms.Compose([ transforms.Resize((224, 224)), transforms.ToTensor(), ]) train_ds = CIFAR100('data', train=True, download=True, transform=pipeline) train_dl = DataLoader(train_ds, batch_size=128, num_workers=4) val_ds = CIFAR100('data', train=False, download=True, transform=pipeline) val_dl = DataLoader(val_ds, batch_size=128, num_workers=4) return train_dl, val_dl if __name__ == "__main__": torch.set_float32_matmul_precision('medium') train_dl, val_dl = prepare_data() model = MagixNet(100) trainer = L.Trainer(max_epochs=1, strategy="ddp_notebook") start = time.time() trainer.fit(model, train_dl, val_dl) print(f"Training duration (seconds): {time.time() - start:.2f}") nccl_debug_file = pathlib.Path("/tmp/runsc-nccl.txt") if nccl_debug_file.exists(): print("NCCL Debugging") print(nccl_debug_file.read_text()) EOF ENTRYPOINT ["python3", "repro.py"] ``` Run like this: ``` sudo docker run --runtime=runsc-2 --shm-size=1000GB --gpus '"device=GPU-48070a35-b2ea-643c-eebe-0c55d2a541a4,GPU-8061048a-aa0f-76bd-457b-71c6be60386e"' -e NCCL_DEBUG=INFO -e NCCL_DEBUG_FILE="/tmp/runsc-nccl.txt" sha256:1c1fc535214ec1111b46a87fe20558e7c078185e4158c3ce253dc56a5a9be628 ``` **`/etc/docker/daemon.json`** ``` { "runtimes": { "nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }, "runsc-2": { "path": "/home/modal/runsc2", "runtimeArgs": ["--nvproxy", "--nvproxy-docker", "-debug-log=/tmp/runsc-2/", "-debug", "-strace"] }, } } ``` FUTURE_COPYBARA_INTEGRATE_REVIEW=#10434 from thundergolfer:master bf18079 PiperOrigin-RevId: 635812044
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Doing multi-GPU training on A100s and seeing that on gVisor it gets stuck. Tried the below program on the following GPUs within Modal:
Both the H100 and A100 run into these unknown control commands:
Which is
NV0000_CTRL_CMD_OS_UNIX_EXPORT_OBJECT_TO_FD-> https://github.com/NVIDIA/open-gpu-kernel-modules/blob/083cd9cf17ab95cd6f9fb50a5349c21eaa2f7d4b/src/common/sdk/nvidia/inc/ctrl/ctrl0000/ctrl0000unix.h#L146-L147Steps to reproduce
runsc version
docker version (if using docker)
uname
No response
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
The text was updated successfully, but these errors were encountered: