Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

network-mode: none - clab_intfs calculation logic #1851

Open
kafo2205 opened this issue Feb 1, 2024 · 12 comments · May be fixed by #1864
Open

network-mode: none - clab_intfs calculation logic #1851

kafo2205 opened this issue Feb 1, 2024 · 12 comments · May be fixed by #1864

Comments

@kafo2205
Copy link

kafo2205 commented Feb 1, 2024 •
edited

Hello,

I am trying to use network-mode: none with VM-based node, but node does not start:

name: vsrx-cluster
topology:
  nodes:
    node0:
      kind: juniper_vsrx
      image: vrnetlab/vr-vsrx:23.2R1.13
      network-mode: none
      env:
        CLAB_INTFS: 0


| # |           Name           | Container ID |           Image            |     Kind     |  State  |   IPv4 Address   |     IPv6 Address     |
+---+--------------------------+--------------+----------------------------+--------------+---------+------------------+----------------------+
| 1 | clab-vsrx-cluster-node0  | 001d7344ec31 | vrnetlab/vr-vsrx:23.2R1.13 | juniper_vsrx | running | N/A              | N/A                  |


$ sudo docker logs -f 001d7344ec31
2024-02-01 12:46:49,838: vrnetlab   DEBUG    Creating overlay disk image
2024-02-01 12:46:49,854: vrnetlab   DEBUG    Starting vrnetlab VSRX
2024-02-01 12:46:49,854: vrnetlab   DEBUG    VMs: [<__main__.VSRX_vm object at 0x7fb4aac6e250>]
2024-02-01 12:46:49,860: vrnetlab   DEBUG    VM not started; starting!
2024-02-01 12:46:49,860: vrnetlab   INFO     Starting VSRX_vm
2024-02-01 12:46:49,861: vrnetlab   DEBUG    number of provisioned data plane interfaces is 1
2024-02-01 12:46:49,861: vrnetlab   DEBUG    waiting for provisioned interfaces to appear...

looks like clab_intfs calculation logic should be taken into account with network-mode:none

sudo docker exec -it 001d7344ec31 bash
root@node0:/# env | grep INTF
CLAB_INTFS=1

pls also look at this check ethX, where X is >0**, there was an issue:

sudo containerlab deploy -t ~/clab/vsrx-cluster/vsrx-cluster.yaml
INFO[0000] Containerlab v0.50.0 started
INFO[0000] Parsing & checking topology file: vsrx-cluster.yaml
**Error: "node0" interface name "eth0" doesn't match the required pattern. It should be named as ethX, where X is >0**

yaml:

  links:
    - endpoints: ["node0:eth0", "switch:eth1"]
    - endpoints: ["node1:eth0", "switch:eth2"]

Thank you for your support!
Br,
D
@kafo2205 kafo2205 changed the title network-mode:none - clab_intfs calculation logic network-node:none - clab_intfs calculation logic Feb 1, 2024
@kafo2205 kafo2205 changed the title network-node:none - clab_intfs calculation logic network-node:mone - clab_intfs calculation logic Feb 1, 2024
@kafo2205 kafo2205 changed the title network-node:mone - clab_intfs calculation logic network-node:mode - clab_intfs calculation logic Feb 1, 2024
@kafo2205 kafo2205 changed the title network-node:mode - clab_intfs calculation logic network-mode:none - clab_intfs calculation logic Feb 1, 2024
@kafo2205 kafo2205 changed the title network-mode:none - clab_intfs calculation logic network-mode: none - clab_intfs calculation logic Feb 1, 2024
@hellt hellt linked a pull request Feb 5, 2024 that will close this issue
Draft
@hellt
Copy link
Member

hellt commented Feb 5, 2024

@kafo2205

can you try the beta containerlab build

sudo docker run --rm -v $(pwd):/workspace ghcr.io/oras-project/oras:v1.1.0 pull ghcr.io/srl-labs/clab-oci:3f2f6e66 && chmod +x ./containerlab

and you will have to build the vsrx again using this branch https://github.com/hellt/vrnetlab/tree/count-eth0

Then you should be able to use eth0 in the vsrx and see it booting. Let me know if something is not clear

@kafo2205
Copy link
Author

kafo2205 commented Feb 5, 2024 •
edited

hi @hellt
I tried but not working:

~/clab/vsrx-cluster $ sudo ~/containerlab deploy -t vsrx-cluster.yaml
INFO[0000] Containerlab v0.0.0 started
INFO[0000] Parsing & checking topology file: vsrx-cluster.yaml
Error: "node0" interface name "eth0" doesn't match the required pattern: eth*$

@hellt
Copy link
Member

hellt commented Feb 5, 2024

@kafo2205
I think I screwed the regexp...

Here is a fix

sudo docker run --rm -v $(pwd):/workspace ghcr.io/oras-project/oras:v1.1.0 pull ghcr.io/srl-labs/clab-oci:0beff1c8

@kafo2205
Copy link
Author

kafo2205 commented Feb 5, 2024 •
edited

Cool, now it boots up :)
..hm but the question is how it should work with eth0 interfaces

this is my topology file srx-node0 <-> n9k-switch <-> srx-node1:

name: vsrx-cluster
topology:
  nodes:
    node0:
      kind: juniper_vsrx
      image: vrnetlab/vr-vsrx:23.2R1.13
      network-mode: none
    node1:
      kind: juniper_vsrx
      image: vrnetlab/vr-vsrx:23.2R1.13
      network-mode: none
    switch:
      kind: cisco_n9kv
      image: vrnetlab/vr-n9kv:10.2.6
  links:
    - endpoints: ["node0:eth0", "switch:eth1"]
    - endpoints: ["node1:eth0", "switch:eth2"]

lab is running:

+---+--------------------------+--------------+----------------------------+--------------+---------+----------------+----------------------+
| # |           Name           | Container ID |           Image            |     Kind     |  State  |  IPv4 Address  |     IPv6 Address     |
+---+--------------------------+--------------+----------------------------+--------------+---------+----------------+----------------------+
| 1 | clab-vsrx-cluster-node0  | 27a39f0afd30 | vrnetlab/vr-vsrx:23.2R1.13 | juniper_vsrx | running | N/A            | N/A                  |
| 2 | clab-vsrx-cluster-node1  | 4f5fb6364e50 | vrnetlab/vr-vsrx:23.2R1.13 | juniper_vsrx | running | N/A            | N/A                  |
| 3 | clab-vsrx-cluster-switch | bd20cb25513a | vrnetlab/vr-n9kv:10.2.6    | cisco_n9kv   | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+--------------------------+--------------+----------------------------+--------------+---------+----------------+----------------------+

srx succesfully started, ip set to fxp interface (based on the logs)

$ sudo docker logs -f 27a39f0afd30
2024-02-05 21:15:51,413: vrnetlab   DEBUG    Creating overlay disk image
2024-02-05 21:15:51,427: vrnetlab   DEBUG    Starting vrnetlab VSRX
2024-02-05 21:15:51,427: vrnetlab   DEBUG    VMs: [<__main__.VSRX_vm object at 0x7f5501a101f0>]
2024-02-05 21:15:51,432: vrnetlab   DEBUG    VM not started; starting!
2024-02-05 21:15:51,433: vrnetlab   INFO     Starting VSRX_vm
2024-02-05 21:15:51,433: vrnetlab   DEBUG    number of provisioned data and control plane interfaces is 1
2024-02-05 21:15:51,433: vrnetlab   DEBUG    waiting for provisioned interfaces to appear...
2024-02-05 21:15:56,439: vrnetlab   DEBUG    highest allocated interface id determined to be: 0
2024-02-05 21:15:56,439: vrnetlab   DEBUG    control and datap plane interfaces detected, continuing...
2024-02-05 21:15:56,440: vrnetlab   DEBUG    qemu cmd: qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -m 4096 -serial telnet:0.0.0.0:5000,server,nowait -drive if=ide,file=/junos-vsrx3-x86-64-23.2R1.13-overlay.qcow2 -smp 2 -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:b0:11:8f:00 -netdev user,id=p00,net=10.0.0.0/24,tftp=/tftpboot,hostfwd=tcp::2022-10.0.0.15:22,hostfwd=udp::2161-10.0.0.15:161,hostfwd=tcp::2830-10.0.0.15:830,hostfwd=tcp::2080-10.0.0.15:80,hostfwd=tcp::2443-10.0.0.15:443

...


[edit]
root#'
2024-02-05 21:21:11,117: vrnetlab   DEBUG    writing to serial console: 'set interfaces fxp0 unit 0 family inet address 10.0.0.15/24'
2024-02-05 21:21:11,117: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:11,165: vrnetlab   TRACE    read from serial console: ' set interfaces fxp0 unit 0 family inet address 10.0.0.15/24

[edit]
root#'
2024-02-05 21:21:11,165: vrnetlab   DEBUG    writing to serial console: 'set system management-instance'
2024-02-05 21:21:11,165: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:11,212: vrnetlab   TRACE    read from serial console: ' set system management-instance

[edit]
root#'
2024-02-05 21:21:11,213: vrnetlab   DEBUG    writing to serial console: 'set routing-instances mgmt_junos description management-instance'
2024-02-05 21:21:11,213: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:11,257: vrnetlab   TRACE    read from serial console: ' set routing-instances mgmt_junos description management-instance

[edit]
root#'
2024-02-05 21:21:11,257: vrnetlab   DEBUG    writing to serial console: 'set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 next-hop 10.0.0.2'
2024-02-05 21:21:11,257: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:11,301: vrnetlab   TRACE    read from serial console: ' set routing-instances mgmt_junos routing-options static route 0.0.0.0/0 next-hop 10.0.0.2

[edit]
root#'
2024-02-05 21:21:11,301: vrnetlab   DEBUG    writing to serial console: 'commit'
2024-02-05 21:21:11,301: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:36,884: vrnetlab   TRACE    read from serial console: ' commit
commit complete

[edit]
root#'
2024-02-05 21:21:36,884: vrnetlab   DEBUG    writing to serial console: 'commit'
2024-02-05 21:21:36,884: vrnetlab   TRACE    waiting for '#' on serial console
2024-02-05 21:21:42,487: vrnetlab   TRACE    read from serial console: ' commit
commit complete

[edit]
root#'
2024-02-05 21:21:42,487: vrnetlab   DEBUG    writing to serial console: 'exit'
2024-02-05 21:21:42,487: vrnetlab   TRACE    waiting for '>' on serial console
2024-02-05 21:21:42,532: vrnetlab   TRACE    read from serial console: ' exit
Exiting configuration mode

root>'
2024-02-05 21:21:42,533: vrnetlab   DEBUG    writing to serial console: 'exit'
2024-02-05 21:21:42,533: launch     INFO     completed bootstrap configuration
2024-02-05 21:21:42,533: launch     INFO     Startup complete in: 0:05:51.100304

how to connect to srx console?

$ sudo docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED          STATUS                    PORTS                                                                                       NAMES
bd20cb25513a   vrnetlab/vr-n9kv:10.2.6      "/launch.py --userna…"   27 minutes ago   Up 27 minutes (healthy)   22/tcp, 80/tcp, 443/tcp, 830/tcp, 5000/tcp, 6030/tcp, 10000-10099/tcp, 57400/tcp, 161/udp   clab-vsrx-cluster-switch
4f5fb6364e50   vrnetlab/vr-vsrx:23.2R1.13   "/launch.py --userna…"   27 minutes ago   Up 27 minutes (healthy)                                                                                               clab-vsrx-cluster-node1
27a39f0afd30   vrnetlab/vr-vsrx:23.2R1.13   "/launch.py --userna…"   27 minutes ago   Up 27 minutes (healthy)                                                                                               clab-vsrx-cluster-node0

$ sudo docker exec -it 27a39f0afd30 bash
root@node0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
263: eth0@if262: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP group default
    link/ether aa:c1:ab:aa:0d:46 brd ff:ff:ff:ff:ff:ff link-netnsid 1
root@node0:/# telnet 127.0.0.1 5000
bash: telnet: command not found
root@node0:/#

Also tried to set up switch in the srx-fxp subnet but looks like there is no connectivity...

switch# show interface e1/1-2 status
--------------------------------------------------------------------------------
Port          Name               Status    Vlan      Duplex  Speed   Type
--------------------------------------------------------------------------------
Eth1/1        --                 connected 1         full    1000    10g
Eth1/2        --                 connected 1         full    1000    10g

switch# show interface e1/1-2 switchport | i Acc
  Access Mode VLAN: 1 (default)
  Access Mode VLAN: 1 (default)

switch# show mac address-table interface e1/1
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
switch# show mac address-table interface e1/2
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
        (NA)- Not Applicable
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
switch#

interface Vlan1
  no shutdown
  ip address 10.0.0.1/24

PING 10.0.0.15 (10.0.0.15): 56 data bytes
36 bytes from 10.0.0.1: Destination Host Unreachable
^C
--- 10.0.0.15 ping statistics ---
1 packets transmitted, 0 packets received, 100.00% packet loss
switch# ping 10.0.0.16
PING 10.0.0.16 (10.0.0.16): 56 data bytes
36 bytes from 10.0.0.1: Destination Host Unreachable
^C
--- 10.0.0.16 ping statistics ---
1 packets transmitted, 0 packets received, 100.00% packet loss
switch# sh ip arp | i Vlan
10.0.0.15       00:00:07  INCOMPLETE      Vlan1
10.0.0.16       00:00:03  INCOMPLETE      Vlan1
switch#

@hellt
Copy link
Member

hellt commented Feb 5, 2024

you should be ablet to install telnet on your containerlab host and from there do telnet <node name> 5000

@kafo2205
Copy link
Author

kafo2205 commented Feb 5, 2024 •
edited

actually on the host there isn't ip address assigned for network-mode: none containers

| # |           Name           | Container ID |           Image            |     Kind     |  State  |  IPv4 Address  |     IPv6 Address     |
+---+--------------------------+--------------+----------------------------+--------------+---------+----------------+----------------------+
| 1 | clab-vsrx-cluster-node0  | 2820bb80469e | vrnetlab/vr-vsrx:23.2R1.13 | juniper_vsrx | running | N/A            | N/A                  |
| 2 | clab-vsrx-cluster-node1  | 484973aadba6 | vrnetlab/vr-vsrx:23.2R1.13 | juniper_vsrx | running | N/A            | N/A                  |

so not able to telnet 5000

$ telnet clab-vsrx-cluster-node0 5000
telnet: could not resolve clab-vsrx-cluster-node0/5000: Name or service not known

but I installed telnet in the container itself

$ sudo docker exec -it 2820bb80469e bash
root@node0:/# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

FreeBSD/amd64 (Amnesiac) (ttyu0)

login: admin
Password:
Last login: Mon Feb  5 22:40:34 on ttyu0

--- JUNOS 23.2R1.13 Kernel 64-bit XEN JNPR-12.1-20230613.7723847_buil
admin> show configuration | display set | match fxp
set interfaces fxp0 unit 0 family inet address 10.0.0.16/24


 $ sudo docker exec -it 484973aadba6 bash
root@node1:/# telnet localhost 5000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

login: admin
Password:

--- JUNOS 23.2R1.13 Kernel 64-bit XEN JNPR-12.1-20230613.7723847_buil
admin> show configuration | display set | match fxp
set interfaces fxp0 unit 0 family inet address 10.0.0.15/24

admin>


admin> ping routing-instance mgmt_junos 10.0.0.15
PING 10.0.0.15 (10.0.0.15): 56 data bytes
64 bytes from 10.0.0.15: icmp_seq=0 ttl=64 time=2.706 ms
64 bytes from 10.0.0.15: icmp_seq=1 ttl=64 time=0.063 ms
^C
--- 10.0.0.15 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.063/1.385/2.706/1.321 ms

admin> ping routing-instance mgmt_junos 10.0.0.16
PING 10.0.0.16 (10.0.0.16): 56 data bytes
^C
--- 10.0.0.16 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

admin> ping routing-instance mgmt_junos 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 56 data bytes
^C
--- 10.0.0.1 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

admin> show arp
MAC Address       Address         Name                      Interface               Flags
52:55:0a:00:00:02 10.0.0.2        10.0.0.2                  fxp0.0                  none

admin> ping routing-instance mgmt_junos 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=255 time=0.366 ms
^C
--- 10.0.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.366/0.366/0.366/0.000 ms

admin>

srx nodes are not able to ping thru the switch, I will try to interconnect them directly, but anyway do you know what is 52:55:0a:00:00:02 10.0.0.2 ??

@hellt
Copy link
Member

hellt commented Feb 6, 2024

52:55:0a:00:00:02 10.0.0.2 is the qemu user interface that is used to represent the VM eth0 interface and it is indeed addressed with 10.0.0.2 -- https://github.com/hellt/vrnetlab/blob/b680560073c71b3d4926f9a99426fa6c6094c794/common/vrnetlab.py#L408

it seems that to support network-mode:none for vrnetlab-based VMs some more tuning would be needed. Like not using qemu user mode and adding the tap interface.

Unfortunately at this point I won't have time to look at this. Maybe someone from the community can try that

@kafo2205
Copy link
Author

kafo2205 commented Feb 6, 2024

@hellt thank you,

it seems that to support network-mode:none for vrnetlab-based VMs some more tuning would be needed. Like not using qemu user mode and adding the tap interface.

maybe I can try at least some manual w/a if you can share with me some examples how to do it :)

@kafo2205
Copy link
Author

kafo2205 commented Feb 6, 2024 •
edited

for example how to modify this qemu cmd:

qemu-system-x86_64 -enable-kvm -display none -machine pc -monitor tcp:0.0.0.0:4000,server,nowait -m 4096 -serial telnet:0.0.0.0:5000,server,nowait -drive if=ide,file=/junos-vsrx3-x86-64-23.2R1.13-overlay.qcow2 -smp 2 -device pci-bridge,chassis_nr=1,id=pci.1 -device virtio-net-pci,netdev=p00,mac=0C:00:12:02:b2:00 -netdev user,id=p00,net=10.0.0.0/24,tftp=/tftpboot,hostfwd=tcp::2022-10.0.0.15:22,hostfwd=udp::2161-10.0.0.15:161,hostfwd=tcp::2830-10.0.0.15:830,hostfwd=tcp::2080-10.0.0.15:80,hostfwd=tcp::2443-10.0.0.15:443

to bypass qemu user interface that is used to represent the VM eth0 and allow to interconnect eth0 with tc backend

@hellt
Copy link
Member

hellt commented Feb 6, 2024

there is no easy answer to that. It is done in hellt/vrnetlab project via different functions

@hellt
Copy link
Member

hellt commented Feb 12, 2024

Unfortunately the change is more involved than I anticipated; We have to also change the provisioning logic for this case of a self-managed eth0 interface.

And it drags a lot of changes that are not easy to merge in just now.

So I think for now there is no quick way to have eth0 be managed by a user (and not docker) without doing some manual work...

I will keep this issue open so that I remember about it when doing some rework in vrnetlab code base

@kafo2205
Copy link
Author

Sure I understand, I hope once it will be available I see several usecases like oob mgmt infra, ztp/dhcp, clustering,
Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Count management interface in CLAB_INTFS_WITH_MGMT env var srl-labs/containerlab
2 participants
@hellt @kafo2205