Containerlab failed to create some nodes and links in a large topology deployment: context deadline exceeded

[liuyuefengcn]

Describe the bug
[ X] Did you check if this is a duplicate issue?
[ Y] Did you test it on the latest version?
containerlab version v0.49.0
FRRouting version stable/9.0

To Reproduce
containerlab topofile:
***@route:~/frrlab$ cat frrlab30x30.yml

name: frrlab
mgmt:
  network: custom_mgmt
  ipv4-subnet: 172.100.0.0/16
topology:
  nodes:
    router1.1:
      kind: linux
      image: frr:ubi1.0
      binds:
        - router1.1/daemons:/etc/frr/daemons
        - router1.1/frr.conf:/etc/frr/frr.conf
    router1.2:
      kind: linux
      image: frr:ubi1.0
      binds:
        - router1.2/daemons:/etc/frr/daemons
        - router1.2/frr.conf:/etc/frr/frr.conf
	......
    router30.29:
      kind: linux
      image: frr:ubi1.0
      binds:
        - router30.29/daemons:/etc/frr/daemons
        - router30.29/frr.conf:/etc/frr/frr.conf
    router30.30:
      kind: linux
      image: frr:ubi1.0
      binds:
        - router30.30/daemons:/etc/frr/daemons
        - router30.30/frr.conf:/etc/frr/frr.conf
  links:
    - endpoints: ["router1.1:eth-east", "router2.1:eth-west"]
    - endpoints: ["router1.1:eth-south", "router1.30:eth-north"]
    - endpoints: ["router1.2:eth-east", "router2.2:eth-west"]
	......
    - endpoints: ["router30.30:eth-east", "router1.30:eth-west"]
    - endpoints: ["router30.30:eth-south", "router30.29:eth-north"]
 
sudo clab deploy --topo frrlab30x30.yml

Expected behavior:
All nodes and links can be created successfully.

Actual behavior:
Ubuntu 9.4.0-1ubuntu1~20.04.2
Linux version 5.4.0-169-generic
After running ContainerLab topfile, most nodes and links are created successfully, but a few fail, as shown in the figure below：

Error info：
ERRO[0374] failed deploy phase for node "router9.7": Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.43/containers/0ff5aff30d0c0c0f74f65040c57890d06f21610a00edcfc7e26996379f6d8301/start": context deadline exceeded
ERRO[0378] failed deploy links for node "router12.21": failed to Statfs "": no such file or directory

any solutions please.. 😢

[steiler]

Alright, so the container nodes, that make it into the link creation phase should all be deployed and running. Usually the failed to Statfs message indicates that the referenced container is not running any more. If you check the state via docker ps you should be able to verify that. Further you should check the logs, to figure out why the container is not running any more.
Are you sure it is not a matter of resource shortage? Your're maybe running low on memory...

[steiler]

Further I see you get context deadline exceeded messages.
To prevent these you might need to increate the timeout.

Global Flags:
      ...
      --timeout duration   timeout for external API requests (e.g. container runtimes), e.g: 30s, 1m, 2m30s (default 2m0s)
      ...

[hellt]

Limiting number of workers can help as well
I think the flag is --workers

[liuyuefengcn]

@steiler Thank you very much for your timely reply. The problem has been solved according to your method.

[liuyuefengcn]

@hellt I have also tried your method and it also solves the problem with --max-workers flag. Thank you very much.

[liuyuefengcn]

Using the --max-workers option can solve this problem, thank you very much.

…

------------------ 原始邮件 ------------------ 发件人: "Markus ***@***.***>; 发送时间: 2023年12月28日(星期四) 下午4:05 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [srl-labs/containerlab] Containerlab failed to create some nodes and links in a large topology deployment: context deadline exceeded (Discussion #1803) Alright, so the container nodes, that make it into the link creation phase should all be deployed and running. Usually the failed to Statfs message indicates that the referenced container is not running any more. If you check the state via docker ps you should be able to verify that. Further you should check the logs, to figure out why the container is not running any more. Are you sure it is not a matter of resource shortage? Your're maybe running low on memory... — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[liuyuefengcn]

Yes, using the --timeout or --max-workers option can solve this problem, thank you very much.

…

------------------ 原始邮件 ------------------ 发件人: "Markus ***@***.***>; 发送时间: 2023年12月28日(星期四) 下午4:14 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [srl-labs/containerlab] Containerlab failed to create some nodes and links in a large topology deployment: context deadline exceeded (Discussion #1803) Further I see you get context deadline exceeded messages. To prevent these you might need to increate the timeout. Global Flags: ... --timeout duration timeout for external API requests (e.g. container runtimes), e.g: 30s, 1m, 2m30s (default 2m0s) ... — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>