How to run colossalAI in kubernetes? #2542

tswangdi · 2023-02-02T02:57:11Z

tswangdi
Feb 2, 2023

I have tried to run colossalAI in docker container:

sudo docker run --gpus all --shm-size=1g --ulimit memlock=-1 -it -p 5000:5000 hpcaitech/colossalai:0.1.10-torch1.11-cu11.3 bash

It works well:

root@25dcc4349b6c:/workspace# colossalai benchmark -g 2
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
=== Benchmarking Parameters ===
gpus: 2
batch_size: 8
seq_len: 512
dimension: 1024
warmup_steps: 10
profile_steps: 50
layers: 2
model: mlp

_meta_registrations seems to be incompatible with PyTorch 1.11.0.
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
=== size: 2, mode: None ===
Average forward time: 0.0005432605743408203
Average backward time: 0.0012273168563842773
Max allocated GPU memory: 0.26564550399780273
Max cached GPU memory: 0.287109375

=== size: 2, mode: 1d ===
Average forward time: 0.002881293296813965
Average backward time: 0.0010549449920654297
Max allocated GPU memory: 0.2500166893005371
Max cached GPU memory: 0.287109375

root@25dcc4349b6c:/workspace#  colossalai run --master_port 5000 --nproc_per_node 4 train.py
...
# It works well

When I run colossalAI in Kubernetes pod, I get a SIGKILL:

kubectl exec -it test-colossal -- bash

root@test-colossal:/workspace# nvidia-smi
Thu Feb  2 02:46:25 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.60.13    Driver Version: 525.60.13    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:02:00.0 Off |                  N/A |
|  0%   29C    P8     9W / 270W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:83:00.0 Off |                  N/A |
| 25%   26C    P8     8W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

root@test-colossal:/workspace# colossalai check -i
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
CUDA Version: 11.3
PyTorch Version: 1.11.0
CUDA Version in PyTorch Build: 11.3
PyTorch CUDA Version Match: ✓
CUDA Extension: ✓

root@test-colossal:/workspace# colossalai benchmark -g 2
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
=== Benchmarking Parameters ===
gpus: 2
batch_size: 8
seq_len: 512
dimension: 1024
warmup_steps: 10
profile_steps: 50
layers: 2
model: mlp

_meta_registrations seems to be incompatible with PyTorch 1.11.0.
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/bin/colossalai", line 8, in <module>
    sys.exit(cli())
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/colossalai/cli/benchmark/__init__.py", line 27, in benchmark
    run_benchmark(args)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/colossalai/cli/benchmark/benchmark.py", line 42, in run_benchmark
    mp.spawn(run_func, nprocs=args.gpus)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGKILL

Sometimes, the following message appears before Trackback:

[W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:57137 (errno: 99 - Cannot assign requested address).

My pod manifest:

apiVersion: v1
kind: Pod
metadata:
  ...
spec:
  containers:
  - command:
    - bash
    - -c
    - mkdir -p /run/sshd && /usr/sbin/sshd -De    # I want to test multiple pods in the next step, so i start sshd.
    image: hpcaitech/colossalai:0.1.10-torch1.11-cu11.3
    imagePullPolicy: IfNotPresent
    name: colossal
    resources:
      limits:
        cpu: "1"
        memory: 1Gi
        nvidia.com/gpu: "2"
      requests:
        cpu: "1"
        memory: 1Gi
        nvidia.com/gpu: "2"
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    ...

So, how can i debug pod, how to config kubernetes cluster and pod?

Answered by tswangdi

Feb 3, 2023

root@test-colossal:/workspace# colossalai benchmark -g 2
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
=== Benchmarking Parameters ===
gpus: 2
batch_size: 8
seq_len: 512
dimension: 1024
warmup_steps: 10
profile_steps: 50
layers: 2
model: mlp

_meta_registrations seems to be incompatible with PyTorch 1.11.0.
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
[W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:43404 (errno: 99 - Cannot assign requested address).
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/bin/colossalai", line 8, in <module>
    sys.exit(cli())
  File "/opt/conda/envs/pytorch/lib/python3.…

View full answer

tswangdi · 2023-02-03T02:22:18Z

tswangdi
Feb 3, 2023
Author

root@test-colossal:/workspace# colossalai benchmark -g 2
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
=== Benchmarking Parameters ===
gpus: 2
batch_size: 8
seq_len: 512
dimension: 1024
warmup_steps: 10
profile_steps: 50
layers: 2
model: mlp

_meta_registrations seems to be incompatible with PyTorch 1.11.0.
_meta_registrations seems to be incompatible with PyTorch 1.11.0.
[W socket.cpp:558] [c10d] The client socket has failed to connect to [localhost]:43404 (errno: 99 - Cannot assign requested address).
Traceback (most recent call last):
  File "/opt/conda/envs/pytorch/bin/colossalai", line 8, in <module>
    sys.exit(cli())
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/colossalai/cli/benchmark/__init__.py", line 27, in benchmark
    run_benchmark(args)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/colossalai/cli/benchmark/benchmark.py", line 42, in run_benchmark
    mp.spawn(run_func, nprocs=args.gpus)
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/opt/conda/envs/pytorch/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 140, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with signal SIGKILL


root@test-colossal:/workspace# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 test-colossal:57948    test-colossal:34641 TIME_WAIT
tcp        0      0 test-colossal:48218    test-colossal:43297 TIME_WAIT
tcp        0      0 test-colossal:43839    test-colossal:51256 TIME_WAIT
tcp6       0      0 localhost:43404         localhost:46458         TIME_WAIT
tcp6       0      0 localhost:43404         localhost:46442         TIME_WAIT
tcp6       0      0 127.0.0.1:43404         127.0.0.1:46460         TIME_WAIT
tcp6       0      0 127.0.0.1:43404         127.0.0.1:46430         TIME_WAIT
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node   Path

1 reply

tswangdi Feb 3, 2023
Author

Memory is too small, set the Memory to 10 Gi and benchmark runs normally

Besides, shared memory is necessary

FrankLeeeee · 2023-02-05T08:33:42Z

FrankLeeeee
Feb 5, 2023
Maintainer

Great, you like run Colossal-AI on kubernetes just like how you run with PyTorch distrubuted code.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to run colossalAI in kubernetes? #2542

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

How to run colossalAI in kubernetes? #2542

tswangdi Feb 2, 2023

Replies: 2 comments · 1 reply

tswangdi Feb 3, 2023 Author

tswangdi Feb 3, 2023 Author

FrankLeeeee Feb 5, 2023 Maintainer

tswangdi
Feb 2, 2023

Replies: 2 comments 1 reply

tswangdi
Feb 3, 2023
Author

tswangdi Feb 3, 2023
Author

FrankLeeeee
Feb 5, 2023
Maintainer