Multiple VMs on the same subnet

[alexellis]

I've been trying to create multiple VMs on the same subnet, with the same, stable gateway IP.

Whenever a tap name is specified on more than one VM, the request gets rejected, whether pre-created or created by cloud-hypervisor at start-up.

There's some rather complex and undocumented code in Firecracker's Go SDK which sets up new network namespaces, and allows for this configuration using a host bridge and CNI along with tc-redirect-tap - which has zero documentation.

Is there a way that a stable IP can be set up i.e. 172.16.0.1/24 for the host, and each cloud-hypervisor VM can obtain a working IP within that subnet i.e. 172.16.0.2, 172.16.0.3, 172.16.0.4 and so forth?

[rbradford]

I've been trying to create multiple VMs on the same subnet, with the same, stable gateway IP.

Whenever a tap name is specified on more than one VM, the request gets rejected, whether pre-created or created by cloud-hypervisor at start-up.

There's some rather complex and undocumented code in Firecracker's Go SDK which sets up new network namespaces, and allows for this configuration using a host bridge and CNI along with tc-redirect-tap - which has zero documentation.

Is there a way that a stable IP can be set up i.e. 172.16.0.1/24 for the host, and each cloud-hypervisor VM can obtain a working IP within that subnet i.e. 172.16.0.2, 172.16.0.3, 172.16.0.4 and so forth?

You can't reuse tap names on the host (or maybe you can inside a network namespace, IDK) but if you want those machines to be able to reach other then create a bridge, create taps in advance, add them to the bridge and give them to each VM without specifying any IP configuration. You can then give the bridge interface an IP on the host and then you have host to guest and guest to guest communication. If you only care about guest to host communication I think you can reuse the same host IP/guest IP (or change it per guest) as each network will be isolated.

[alexellis]

Thanks for the suggestion, having the gateway IP on the bridge might solve this.

Each tap device seems to get a 192.x IP, when none is given. Is it possible to have no IP attached on the host-end of the tap?

[rbradford]

Are you precreating the tap device or relying on CH to create it? If you precreate a TAP device it shouldn't try and add an IP - if it is then that is a regression.

[alexellis]

Hi, I was not pre-creating the tap, but I've just tried that and it works as you describe. Thank you 👍

Is there a way to suppress the warning though?

cloud-hypervisor: 85.763595ms: <vmm> WARN:net_util/src/open_tap.rs:103 -- Tap tap6 already exists. IP configuration will not be overwritten.

[peng6662001]

Hi @alexellis :
You can use tap ids for different vms like this:

vm1
--net id=net_1,tap=,mac=12:34:56:78:90:02,ip=192.168.2.1,mask=255.255.255.0
vm2
--net id=net_2,tap=,mac=12:34:56:78:90:03,ip=192.168.3.1,mask=255.255.255.0
vm32
--net id=net_33,tap=,mac=12:34:56:78:90:34,ip=192.168.34.1,mask=255.255.255.0

I created 32 vms and no warning is printed.