How to send the traffic to vale bridge based vm

[zhangwei1984]

Hi all,

After we start the vm with vale bridge, we want to send the traffic to this vm. Any idea?
The vm screen

vm list

root@nimbnode21:~# xl list
Name                                        ID   Mem VCPUs  State   Time(s)
Domain-0                                     0 32271    24     r-----     378.1
clickos                                     10     7     1     r-----     405.1

vif10.0   Link encap:Ethernet  HWaddr fe:ff:ff:ff:ff:ff  
          inet addr:10.0.0.11  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

The click example

FromDevice->Print("Hi")->ToDevice

We want to send the traffic to vm. But when I run the pktgen from domain 0
It shows this:

root@nimbnode21:~# vale-ctl 
bdg_ctl [98] bridge:0 port:1 vale0:vif10.0

root@nimbnode21:~# pkt-gen -f tx -i vale0:vif10.0
052.426593 main [1624] interface is vale0:vif10.0
052.427249 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0
052.427267 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0
052.427323 nm_open [542] NIOCREGIF failed vale0:vif10.0
052.427335 main [1803] Unable to open vale0:vif10.0: Invalid argument
052.427369 main [1851] aborting
Usage:

[fmanco]

Hi

valeo:vif10.0 is the port vif10.0 on switch vale0. You can't send traffic directly to a port attached to the switch, you need to create a new port on that switch by doing pkt-gen -f tx -i vale0:txp, where txp is a name of your choice.

[zhangwei1984]

I have tried this. It also does not work. Along the compiling process, all of the operations get succeed. I can install the module (netmap_lin.ko, ixgbe.ko, xen-netback.ko successfully), start the vm and launch the click app. But why my vm click can not receive anything. This is what I did:

root@nimbnode21:/home/wei# pkt-gen -f tx -i vale0:tx
063.556810 main [1624] interface is vale0:tx
063.557681 extract_ip_range [275] range is 10.0.0.1:0 to 10.0.0.1:0
063.557717 extract_ip_range [275] range is 10.1.0.1:0 to 10.1.0.1:0
063.575386 main [1807] mapped 41376KB at 0x7f462f09c000
Sending on vale0:tx: 1 queues, 1 threads and 1 cpus.
10.0.0.1 -> 10.1.0.1 (00:00:00:00:00:00 -> ff:ff:ff:ff:ff:ff)
063.575479 main [1885] Sending 512 packets every  0.000000000 s
063.575490 main [1887] Wait 2 secs for phy reset
065.575666 main [1889] Ready...
065.575727 nm_open [457] overriding ifname vale0:tx ringid 0x0 flags 0x1
065.575887 sender_body [996] start
065.588478 sender_body [1065] drop copy
066.576880 main_thread [1421] 16009494 pps (16026112 pkts in 1001038 usec)
067.577932 main_thread [1421] 16395408 pps (16412672 pkts in 1001053 usec)
068.578999 main_thread [1421] 16395706 pps (16413184 pkts in 1001066 usec)
069.580071 main_thread [1421] 16384867 pps (16402432 pkts in 1001072 usec)

root@nimbnode21:~# vale-ctl 
bdg_ctl [98] bridge:0 port:0 vale0:vif14.0
bdg_ctl [98] bridge:0 port:1 vale0:tx

My VM can not receive the packets also

xenbus initialised on irq 1 mfn 0x176c30
Thread "shutdown": pointer: 0x2000802fb0, stack: 0x270000
Dummy main: start_info=0x208b20
Thread "main": pointer: 0x2000803760, stack: 0x280000
sparsing 0MB at 1e7000
"main" 
[on:80] * 18 for clickos/0/elements*
[on:80] * 17 for clickos/0/control*
[on_status:246] status change to Running
Thread "click": pointer: 0x2000804a40, stack: 0x290000
[router_thread:204] Starting driver...

vm configuration

name   = 'clickos'
    kernel = '/root/click_large'
    vcpus  = '1'
    # pinning your VCPU helps performance
    #cpus   = '3'
    memory = '8'

    # uncoment this line if you wish to use our backend
    vif    = ['ip=10.0.0.1,bridge=vale0,script=vif-vale']
    #vif    = ['mac=00:15:17:15:5d:74,bridge=xenbr0']

    on_poweroff = 'destroy'
    on_reboot   = 'restart'
    on_crash    = 'preserve'                        
    click       = 'mirror.click'

    # uncomment this line to enable PVH mode for ClickOS
    #pvh=1

click configuration

FromDevice(0)->Print("Hi")->ToDevice(0)

[fmanco]

What commands are you using to create the VM and start the click configuration? Also what's the output of xenstore-ls after the configuration is started?

[zhangwei1984]

Now it does not crash. The VM can receive the packets. But this process is very very slow. I fell it takes several mins. At some point, it will stuck at Mapping TX ring long time and I have to restart the VM.
Do you know how to make it quickly start?

Thread "click": pointer: 0x2040010a00, stack: 0x750000
backend dom 0
Mapping TX rings
    0 map errors
Mapping RX rings
    0 map errors
Mapping TX buffers
    0 map errors
Mapping RX buffers
    0 map errors
init_netfront_netmap device/vif/0
Waiting for /local/domain/0/backend/vif/10/0/state change to connected
unmasking event-channel-tx 4
unmasking event-channel-rx 5
[router_thread:157] Starting driver...

[zhangwei1984]

It takes about 2 mins start the click app. Is something wrong with my setup? How can I reduce the startup time? Now I can do the service chain. Thanks, @fmanco !

root@nimbnode21:~/vms# xl list

Name                                        ID   Mem VCPUs  State   Time(s)

Domain-0                                     0 30340    24     r-----    2886.2

vm1                                         12  1024     1     -b----     164.0

vm2                                         13  1024     1     -b----     160.8

[fmanco]

That's weird it's taking so much time. Which version of xen are you running?

[jpemartins]

Could you try booting up your guest on one of the CPUs being used for Domain-0 ? Say if your Domain-0 is pinned to 0,1,2,3. You boot ClickOS on CPU 3, and then after booted switch to CPU 4.

[zhangwei1984]

I am using xen 4.6.0. Thanks so much for your help along the way! I can run 1,2,3VM. But after 4VMs, after I start click example, the VM still gets crashed. I gave 1G memory. It does not help to improve the startup time even pin the core like @jpemartins said. I saw it spends a very long time map the rx and tx ring. What could be the possible reason?

[fmanco]

That's indeed weird. We never tested ClickOS with Xen 4.6, could you possible try an older version, like Xen 4.2? Also try to reduce the amount of memory you give to the VM (you need to figure the minimum by trial and error).