Propagate host interface link status to VM interfaces

[DolceTriade]

Use case

EVE allows you to bind VM interfaces to a linux bridge. However, if the hosts's physical interface in that bridge doesn't have link, there is no way for the VM's interface to reflect that. There should be a way for a VM to query the link status of the host interfaces.

Describe the solution you'd like

There should be a way for a VM to query the link status of the host interfaces.

This could be in the form of an API on the metadata server that you can post and get back some JSON blob, or even better, have the link status of the interfaces inside the VM change based on the link status of the physical interface they are attached to on the host.

[eriknordmark]

A network instance isn't required to have an external port, which is useful to create various host-only connectivity (can be a local network instance with DNS+DHCP service, or a switch network instance which can pass any Ethernet types).
Furthermore, a local network instance can have more than one external port (currently specified using "uplink" to mean all of the EVE management ports). In that case the current external port will change if there is a failure.

For those reasons it isn't clear what the link status of the host interface would mean in the general case.
Perhaps you can explain what problem you are trying to solve from the perspective of the guest VM?

[DolceTriade]

The issue we are trying to solve for is detecting when a host network port loses link so we can take action in the VM (for failover, clean up, alerting, etc).

This isn't for connection to the Internet but rather for connection to other devices on the network which we don't know ahead of time so we can't actively probe.

There is no guarantee that they will support IPv6 so we can't do broadcast ping either.

There might be other heuristics we can use to potentially guess this status, but the clearest indication is whether the link is up or not. (Passthrough solves this for us at the cost of losing a full NIC to the workload. SR-IOV solves this better on hardware that supports it since we get better performance AND we can still share the NIC with the host and other workloads. This last option is for hardware that cannot afford to give a full NIC away to a VM and also does not support SR-IOV).

While EVE does support a variety of network configurations, EVE also knows which host adapters are bound to a particular VM either using NAT in a local instance or bridged together using the switch instance, so perhaps it can expose the host NIC link status when queried.

I don't think we can actually affect the link state of the interfaces in the VM without modifying QEMU, so I suppose an API on the metadata server is my next best idea...

[eriknordmark]

@DolceTriade The issue is "link to whom" is up.

Suppose I have three app instances deployed using a switch network instance across two devices.
AppA and AppB on device1 and AppC in device2.

The Ethernet cable (for the port used by the switch network instance) on device1 is unplugged, hence that physical port is marked as NO-CARRIER in ip link.
In that case if the AppA also sees a NO-CARRIER on its virtual Ethernet interface it might assume that it can reach neither AppB nor AppC when in fact AppB is still reachable since it is on the same device.

Thus I think propagation of this only makes sense when only one app instance can possibly use a particular Ethernet port. And for this case you can configure that port as using PCI passthrough (app-direct in the EVE API) in which case the driver is running in the Linux kernel and sees the status already.

(FWIW I don't know how SR-IOV with local loopback in the NIC chip handles this issue.)

So fundamentally if an app wants to know whether it can reach a given IP address it needs to test that reachability (called BFD for routers, and we called it IPMP in Solaris.)

[DolceTriade]

RIght, this makes the most sense in a "single app" use case (though arguably, even in a shared app use case, the apps being aware that the link is down seems useful so that they can take some action). Passthrough isn't always viable option (if you need to share the port with mgmt for example) and having an external party needing to configure BFD or some other protocol is also undesirable if you want to allow people to just "plug in" have it work.

Even if we don't directly propagate the link status on the Ethernet NIC, wouldn't some mechanism like an API in zedrouter's metadata service similar to how we already expose WWAN information be desirable?