Restoring VM fails when using explicitly passed FDs #6286

pupacha · 2024-03-11T13:53:04Z

When a cloud-hypervisor VM, that is created with tapFDs explicitly passed via socket as CMSG (SCM_RIGHTS), is snapshot and restored, it fails.
This is because the config that vm.snapshot generates (config.json) contains the FDs list in it. And when vm.restore is called (with the same directory path used during for vm.snapshot), CH reads this config.json and fails with following error message as those FDs no longer exist.

Error restoring VM: VmRestore(DeviceManager(CreateVirtioNet(TapError(IoctlError(2147767506, Os { code: 22, kind: InvalidInput, message: "Invalid argument" })))))

Snapshot's config.json ("net" content)

[
  {
    "tap": null,
    "ip": "192.168.249.1",
    "mask": "255.255.255.0",
    "mac": "52:54:00:de:d9:65",
    "host_mac": null,
    "mtu": null,
    "iommu": true,
    "num_queues": 8,
    "queue_size": 256,
    "vhost_user": false,
    "vhost_socket": null,
    "vhost_mode": "Client",
    "id": "_net2",
    "fds": [
      26,
      27,
      28,
      29
    ],
    "rate_limiter_config": null,
    "pci_segment": 0,
    "offload_tso": true,
    "offload_ufo": true,
    "offload_csum": true
  }
]

This was discovered when I was implementing libvirt's save/restore features for ch driver at https://github.com/pupacha/libvirt/tree/ch/save_restore_basic

The text was updated successfully, but these errors were encountered:

liuw · 2024-03-11T22:40:45Z

Most likely the FD will be invalid when you restore the guest, right?

pupacha · 2024-03-12T04:16:24Z

Most likely the FD will be invalid when you restore the guest, right?

Yes, the FDs would be invalid while restore. So, we might have to enlighten CH to take "FDs" during restore operation as well.

liuw · 2024-03-13T01:14:51Z

It doesn't quite make sense to me yet. How is the guest created? Did you pass a list of fds while it was created?

pupacha · 2024-03-13T02:01:30Z

It doesn't quite make sense to me yet. How is the guest created? Did you pass a list of fds while it was created?

Yes, a list of FDs are passed during guest creation. Following is the sequence of steps.

vm.create - Config passed without any "netconfig" attributes
vm.add-net - netconfig passed along with a list of FDs
vm.boot to start the guest
vm.snapshot - saves the guest state (CH saves FDs list as well in the config)
vm.restore - fails because of the invalidated FDs in the snapshot config

pupacha · 2024-03-19T13:44:50Z

Based on discussion with @jinankjain and inputs from him, here's a solution proposal to fix the issue.

Idea is to introduce 2 new things

Add a new flag to vmm.restore api say "--restore-net"
Introduce new API called vm.restore-net

1. Adding --restore-net Flag

The --restore-net flag will be designed to modify the behavior of the vm.restore API. When set to False, this flag instructs vm.restore to disregard the NetConfig obtained from the snapshot's configuration file. Consequently, the VM will be restored into a PAUSED state without any instantiated Net objects. This parameter should be utilized exclusively in scenarios where file descriptors (FDs) were explicitly provided during VM creation. By default, the --restore-net flag is set to True, aligning with the expected behavior of the vm.restore API.

2. Introducing the vm.restore-net API

The vm.restore-net API will be introduced to facilitate the restoration of Net(s) from a snapshotted state. This API requires two parameters:

id: Represents the identifier of the NetConfig.
source_url: Denotes the directory path of the snapshot.

This API should be invoked after a vm.restore API call with False passed to --restore-net in order to restore Net devices from the saved device state. Tap FDs can also be passed via SCM_RIGHTS to this API. The actions performed by the vm.restore-net API are as follows:

Read the NetConfigs from the snapshot directory and the Net device state from state.json.
Filter NetConfig and State using the id passed to the API.
Perform similar operations as vm.add-net but with the device State passed.

Following sequence of operations can be performed to achieve VM Restore with FDs passed explicitly from user.

 $ ./cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
 $ #vm.create - Config passed without any "netconfig" attributes
 $ #vm.add-net - netconfig passed along with a list of FDs (id=net1)
 $ #vm.boot to start the guest
 $ #vm.snapshot - saves the guest state to /home/foo/snapshot

 $ ./cloud-hypervisor --api-socket /tmp/cloud-hypervisor.sock
 $ ./ch-remote --api-socket=/tmp/cloud-hypervisor.sock restore source_url=file:///home/foo/snapshot --restore-net <False>

 $# call below API with new FDs passed via SCM_RIGHTS 
 $ ./ch-remote --api-socket=/tmp/cloud-hypervisor.sock vm.restore-net source_url=file:///home/foo/snapshot id=net1

@jinankjain @liuw @rbradford @likebreath Could you please review this and provide your inputs ?

likebreath · 2024-03-19T22:02:12Z

@pupacha Thank you for reporting the issue and looking into potential solutions. I agree with the general direction of the proposal, but I don't think we need so many changes of the APIs.

I think it would be enough to extend the vm.restore API with an optional parameter that takes pairs of <net ID, net FDs>. On the VM restoring path (e.g. from the Vmm::vm_restore(), the VmConfig is received at the very beginning, where you can check if net FDs are being used (via VmConfig.net.fds). If the net FDs are being used, you can replace these invalid FDs with the net FDs from vm.restore (or report VM restore failure if net FDs are not provided). The result of the VM restoring path should not need additional changes.

With that, using invalid FDs is wrong and bug. It impacts both snapshot and live-upgrade/migration, and needs to be fixed.

pupacha · 2024-03-20T06:15:56Z

Thanks @likebreath for reviewing the proposal. I have explored solution on the same lines as you suggested i.e., to pass FDs somehow to vm.restore API itself. But here are few drawbacks I figured out.

The support for passing FDs as input field to the API call, has been recently removed (for security reasons?). Currently, FDs can also be passed via SCM_RIGHTS and only vm.add-net API supports it.
Even if we support taking FDs via SCM_RIGHTS to vm.restore, the mapping of FDs to NetId cannot be maintained. There could be scenarios where user can let CH to create the Taps for some NetConfigs and also pass explicit FDs for others. This needs mapping of Net Id to FD.
One way to achieve this could be by doing something like following
- Pass pairs of <Net Id, FDs count> to vm.restore and a series of FDs (list) via SCM_RIGHTS
- CH will sequentially associate the FDs to the Net Id based on the fd count passed during API call.
- Example input [<net1, 4>, <net2, 0>, <net3, 4>] and FDs list = [23, 24, 25, 26, 27, 28, 29, 30]
  CH will associate FDs 23, 24, 25, 26 to net1, no FDs to net2 and 27, 28, 29, 30 to net3
While this approach looks feasible, I feel the indirect association of the API input fields to FDs (passed via SCM_RIGHTS) should not be there. (In this case, count to FDs)

rbradford · 2024-03-20T09:49:13Z

Can we look into if it's viable to pause the VM, remove the network devices, snapshot and then in reverse restore, hotplug with new FDs and then resume - provided e.g the ordering is preserved along with the MAC address this should result in correct behaviour.

If this is viable in libvirt then we can disable restoring VMs that have FD backed TAPs.

pupacha · 2024-03-20T11:48:57Z

@rbradford From what I understand, if we remove the network devices, the devices' state is lost. And hotplug later, would make the VM treat them as new devices. The network devices' state should also be preserved and restored, but with the new FDs.

rbradford · 2024-03-20T12:36:24Z

@rbradford From what I understand, if we remove the network devices, the devices' state is lost. And hotplug later, would make the VM treat them as new devices. The network devices' state should also be preserved and restored, but with the new FDs.

Yes, the kernel won't be happy with that because it won't reinitialise the devices - I think adding FDs to vm.restore is the way to go.

pupacha · 2024-03-20T12:46:36Z

@rbradford Adding FDs directly to vm.restore has some drawbacks which I mentioned in my previous comment #6286 (comment)

rbradford · 2024-03-20T12:57:40Z

@rbradford Adding FDs directly to vm.restore has some drawbacks which I mentioned in my previous comment #6286 (comment)

I think this a fine approach:

One way to achieve this could be by doing something like following

Pass pairs of <Net Id, FDs count> to vm.restore and a series of FDs (list) via SCM_RIGHTS
CH will sequentially associate the FDs to the Net Id based on the fd count passed during API call.
Example input [<net1, 4>, <net2, 0>, <net3, 4>] and FDs list = [23, 24, 25, 26, 27, 28, 29, 30]
CH will associate FDs 23, 24, 25, 26 to net1, no FDs to net2 and 27, 28, 29, 30 to net3

praveen-pk · 2024-03-21T14:52:04Z

One way to achieve this could be by doing something like following

Pass pairs of <Net Id, FDs count> to vm.restore and a series of FDs (list) via SCM_RIGHTS
CH will sequentially associate the FDs to the Net Id based on the fd count passed during API call.
Example input [<net1, 4>, <net2, 0>, <net3, 4>] and FDs list = [23, 24, 25, 26, 27, 28, 29, 30]
CH will associate FDs 23, 24, 25, 26 to net1, no FDs to net2 and 27, 28, 29, 30 to net3

Unfortunately, this may not work. passing FDs using SCM_RIGHTS will not preserve the FD #. Following above example if the restored cloud-hypervisor already has an FD with ID 23, the FD will be received in cloud-hypervisor with a different FD # (24 or something different). Reliance on preserving FDs from libvirt to cloud-hypervisor is not a viable solution.

We can use net-add after vm.restore before vm.resume? Incoming net-add requests can replace FDs in the Network objects in Cloud-hypervisor. To ensure, correct network interfaces are being updated net-add could receive an optional NetID (_net2) argument. Once all the FDs are received by cloud-hypervisor, it can proceed with vm.restore without any changes. The same sequence would also work in live migration case.

For this to work, we need to maintain 1-to-1 mapping between interfaces in Libvirt and cloud-hypervisor. This should be possible, if we preserve some private information per domain in Libvirt.

likebreath · 2024-03-21T22:27:41Z

One way to achieve this could be by doing something like following

Pass pairs of <Net Id, FDs count> to vm.restore and a series of FDs (list) via SCM_RIGHTS
CH will sequentially associate the FDs to the Net Id based on the fd count passed during API call.
Example input [<net1, 4>, <net2, 0>, <net3, 4>] and FDs list = [23, 24, 25, 26, 27, 28, 29, 30]
CH will associate FDs 23, 24, 25, 26 to net1, no FDs to net2 and 27, 28, 29, 30 to net3

@pupacha What you propose is what I mean by extending vm.restore, e.g. making it taking valid FDs that are donated via SCM_RIGHTS. You clarified the exact syntax of the extended vm.restore, which will be something like --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30].

Note that you won't need to specific any net devices that are not backed directly by donated FDs (e.g. the TAP devices associated with these net devices will be opened based on the TAP name).

Unfortunately, this may not work. passing FDs using SCM_RIGHTS will not preserve the FD #. Following above example if the restored cloud-hypervisor already has an FD with ID 23, the FD will be received in cloud-hypervisor with a different FD # (24 or something different). Reliance on preserving FDs from libvirt to cloud-hypervisor is not a viable solution.

We can use net-add after vm.restore before vm.resume? Incoming net-add requests can replace FDs in the Network objects in Cloud-hypervisor. To ensure, correct network interfaces are being updated net-add could receive an optional NetID (_net2) argument. Once all the FDs are received by cloud-hypervisor, it can proceed with vm.restore without any changes. The same sequence would also work in live migration case.

For this to work, we need to maintain 1-to-1 mapping between interfaces in Libvirt and cloud-hypervisor. This should be possible, if we preserve some private information per domain in Libvirt.

@praveen-pk We don't need to preserve FD #. What we need is to replace the invalid FDs (deserialized from the VmConfig file) with valid FDs (donated via SCM_RIGHTS).

A single vm.restore command should be enough when it is extended in the way that was described above. I think the majority of the changes will be from vmm::config::RestoreConfig and its parsing, along with changes from Vmm::vm_restore to validate and replace invalid FDs with valid FDs (from the updated vmm::config::RestoreConfig). Of course, ch-remote will also need to be updated to support the new vm.restore command.

praveen-pk · 2024-03-21T22:58:34Z

@likebreath, thanks for the clarification. I was misled by references to FD#s above.

Even in the proposed update:
--restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30]

is there a need to pass netfds? We can just pass:
--restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>]

following the above example, of all the FDs received by cloud-hypervisor, first 4 will be assigned to net1 and next 4 will be assigned to net3.

pupacha · 2024-03-22T11:56:06Z

@pupacha What you propose is what I mean by extending vm.restore, e.g. making it taking valid FDs that are donated via SCM_RIGHTS. You clarified the exact syntax of the extended vm.restore, which will be something like --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30].

Note that you won't need to specific any net devices that are not backed directly by donated FDs (e.g. the TAP devices associated with these net devices will be opened based on the TAP name).

@likebreath Right, we can have validation and enforce from CH such that only (and all of) the net devs having backing FDs are passed again to restore.

likebreath · 2024-03-22T15:57:58Z

@likebreath, thanks for the clarification. I was misled by references to FD#s above.

Even in the proposed update: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30]

is there a need to pass netfds? We can just pass: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>]

following the above example, of all the FDs received by cloud-hypervisor, first 4 will be assigned to net1 and next 4 will be assigned to net3.

I believe you do. Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or ch-remote? This will be the same as how we support sending FDs via CLI for --net.

Maybe you are thinking of using an utility program to send the serialization data and FDs directly with SCM_RIGHTS. That would work without an explicit netfds option, but using CLI requires the option.

pupacha · 2024-03-22T17:34:14Z

@likebreath, thanks for the clarification. I was misled by references to FD#s above.
Even in the proposed update: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30]
is there a need to pass netfds? We can just pass: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>]
following the above example, of all the FDs received by cloud-hypervisor, first 4 will be assigned to net1 and next 4 will be assigned to net3.

I believe you do. Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or ch-remote? This will be the same as how we support sending FDs via CLI for --net.

@likebreath The CLI for --net does not support taking FDs. Net FDs can only be passed via SCM_RIGHTS to vm.add-net api.

Maybe you are thinking of using an utility program to send the serialization data and FDs directly with SCM_RIGHTS. That would work without an explicit netfds option, but using CLI requires the option.

Yes, for this fix, plan is to support FDs directly with SC_RIGHTS without the netfds option.

likebreath · 2024-03-25T22:00:16Z

@likebreath, thanks for the clarification. I was misled by references to FD#s above.
Even in the proposed update: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30]
is there a need to pass netfds? We can just pass: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>]
following the above example, of all the FDs received by cloud-hypervisor, first 4 will be assigned to net1 and next 4 will be assigned to net3.

I believe you do. Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or ch-remote? This will be the same as how we support sending FDs via CLI for --net.

@likebreath The CLI for --net does not support taking FDs. Net FDs can only be passed via SCM_RIGHTS to vm.add-net api.

I think you are mistaken. Please check the CLI parser here [1]. Also our doc has an example of how it is being used here [2].

[1] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/vmm/src/config.rs#L1312-L1315
[2] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md

Maybe you are thinking of using an utility program to send the serialization data and FDs directly with SCM_RIGHTS. That would work without an explicit netfds option, but using CLI requires the option.

Yes, for this fix, plan is to support FDs directly with SC_RIGHTS without the netfds option.

I still believe you will need the netfds option, and my reasoning remains the same: "Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or (the CLI of) ch-remote?" We need to support both.

The 'NetConfig' may contain FDs which can't be serialized correctly, as FDs can only be donated from another process via a Unix domain socket with `SCM_RIGHTS`. To avoid false use of the serialized FDs, this patch explicitly set 'NetConfig' FDs as invalid for (de)serialization. See: cloud-hypervisor#6286 Signed-off-by: Bo Chen <chen.bo@intel.com>

pupacha · 2024-03-26T05:50:20Z

@likebreath, thanks for the clarification. I was misled by references to FD#s above.
Even in the proposed update: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>],netfds=[23, 24, 25, 26, 27, 28, 29, 30]
is there a need to pass netfds? We can just pass: --restore source_url=file:///foo/bar,netdevs=[<net1, 4>, <net3, 4>]
following the above example, of all the FDs received by cloud-hypervisor, first 4 will be assigned to net1 and next 4 will be assigned to net3.

I believe you do. Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or ch-remote? This will be the same as how we support sending FDs via CLI for --net.

@likebreath The CLI for --net does not support taking FDs. Net FDs can only be passed via SCM_RIGHTS to vm.add-net api.

I think you are mistaken. Please check the CLI parser here [1]. Also our doc has an example of how it is being used here [2].

[1] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/vmm/src/config.rs#L1312-L1315 [2] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/macvtap-bridge.md

@likebreath Cloud-hypervisor has moved away from taking FDs as input to any APIs. Even when fds field is part of input request, it is ignored and set to None. Currently, vm.add-net is the only API that supports taking FDs but only via SCM_RIGHTS.

[1] #5522
[2] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/vmm/src/api/http/http_endpoint.rs#L207
[3] https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/vmm/src/api/http/http_endpoint.rs#L210

Maybe you are thinking of using an utility program to send the serialization data and FDs directly with SCM_RIGHTS. That would work without an explicit netfds option, but using CLI requires the option.

Yes, for this fix, plan is to support FDs directly with SC_RIGHTS without the netfds option.

I still believe you will need the netfds option, and my reasoning remains the same: "Otherwise how would you support to send FDs via CLI of --snapshot for Cloud Hypervisor or (the CLI of) ch-remote?" We need to support both.

From what I understand, ch-remote does not support fds parameter for any of the APIs. FDs can only be sent via SCM_RIGHTS.

rbradford · 2024-03-26T09:53:04Z

From what I understand, ch-remote does not support fds parameter for any of the APIs. FDs can only be sent via SCM_RIGHTS.

ch-remote does support this - it uses SCM_RIGHTS under the hood of course: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/src/bin/ch-remote.rs#L392-L402

I think some of the confusion comes from that we're thinking that this only would be supported with vm.restore as a HTTP end point - not by restoring using the --restore on the cloud-hypervisor binary itself (of course we can add restore to ch-remote too.)

pupacha · 2024-03-26T12:57:37Z

From what I understand, ch-remote does not support fds parameter for any of the APIs. FDs can only be sent via SCM_RIGHTS.

ch-remote does support this - it uses SCM_RIGHTS under the hood of course: https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/src/bin/ch-remote.rs#L392-L402

I think some of the confusion comes from that we're thinking that this only would be supported with vm.restore as a HTTP end point - not by restoring using the --restore on the cloud-hypervisor binary itself (of course we can add restore to ch-remote too.)

I wasn't aware of this under the hood SCM_RIGHTS usage from ch-remote. Thanks for the clarification @rbradford.
@likebreath as you suggested, I will add netfds as cli params for restore command from ch-remote.

likebreath · 2024-03-26T16:04:15Z

@pupacha Glad my message finally got across. To be clear, as I mentioned earlier, we need to the netfds as CLI option for both ch-remote and cloud-hypervisor.

The 'NetConfig' may contain FDs which can't be serialized correctly, as FDs can only be donated from another process via a Unix domain socket with `SCM_RIGHTS`. To avoid false use of the serialized FDs, this patch explicitly set 'NetConfig' FDs as invalid for (de)serialization. See: cloud-hypervisor#6286 Signed-off-by: Bo Chen <chen.bo@intel.com>

pupacha · 2024-03-26T16:46:14Z

Thanks @likebreath for the PR on setting FDs as invalid for serialization & deserialization.
Just FYI, I have started on implementing the solution we discussed for this issue.

The 'NetConfig' may contain FDs which can't be serialized correctly, as FDs can only be donated from another process via a Unix domain socket with `SCM_RIGHTS`. To avoid false use of the serialized FDs, this patch explicitly set 'NetConfig' FDs as invalid for (de)serialization. See: cloud-hypervisor#6286 Signed-off-by: Bo Chen <chen.bo@intel.com>

The 'NetConfig' may contain FDs which can't be serialized correctly, as FDs can only be donated from another process via a Unix domain socket with `SCM_RIGHTS`. To avoid false use of the serialized FDs, this patch explicitly set 'NetConfig' FDs as invalid for (de)serialization. See: #6286 Signed-off-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot operation. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. Hence, added 2 new parameters 1. net_ids 2. net_fds to 'RestoreConfig'. Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds two new fields to 'RestoreConfig' - 1.net_ids 2.net_fds. 'net_ids' is a list of NetConfig id. 'net_fds' is a list of FDs for required NetConfigs. These fds are replaced into the fds field of NetConfig appropriately. Implement 'validate' for RestoreConfig Implement exclusive HTTP PutHandler for VmRestore. Allow net FDs to be sent along with 'restore' in ch-remote Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra Aekkaladevi <paekkaladevi@linux.microsoft.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds two new fields to 'RestoreConfig' - 1.net_ids 2.net_fds. 'net_ids' is a list of NetConfig id. 'net_fds' is a list of FDs for required NetConfigs. These fds are replaced into the fds field of NetConfig appropriately. Implement 'validate' for RestoreConfig Implement exclusive HTTP PutHandler for VmRestore. Allow net FDs to be sent along with 'restore' in ch-remote Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds two new fields to 'RestoreConfig' - 1.net_ids 2.net_fds. 'net_ids' is a list of NetConfig id. 'net_fds' is a list of FDs for required NetConfigs. These fds are replaced into the fds field of NetConfig appropriately. Implement 'validate' for RestoreConfig Use vm_action_put_handler_body_with_fds for VmRestore http handler Allow net FDs to be sent along with 'restore' in ch-remote Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. Also, implement 'validate' fn for RestoreConfig Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: #6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes: #6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes: cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: #6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes: #6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: cloud-hypervisor#6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes cloud-hypervisor#6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

'NetConfig' FDs, when explicitly passed via SCM_RIGHTS during VM creation, are marked as invalid during snapshot. See: #6332. So, Restore should support input for the new net FDs. This patch adds new field 'net_fds' to 'RestoreConfig'. The FDs passed using this new field are replaced into the 'fds' field of NetConfig appropriately. The 'validate()' function ensures all net devices from 'VmConfig' backed by FDs have a corresponding 'RestoreNetConfig' with a matched 'id' and expected number of FDs. The unit tests provide different inputs to parse and validate functions to make sure parsing and error handling is as per expectation. Fixes #6286 Signed-off-by: Purna Pavan Chandra <paekkaladevi@linux.microsoft.com> Co-authored-by: Bo Chen <chen.bo@intel.com>

jinankjain assigned pupacha Mar 13, 2024

rbradford changed the title ~~RESOTRE FAILS, from snapshot of VM with TapFDs explicitly passed.~~ Restoring VM fails when using explicitly passed FDs Mar 20, 2024

likebreath mentioned this issue Mar 25, 2024

vmm: Explicitly set NetConfig FDs as invalid for (de)serialization #6332

Merged

pupacha mentioned this issue Apr 21, 2024

vmm: Support passing Net FDs to Restore #6402

Merged

likebreath added the bug Something isn't working label May 10, 2024

rbradford closed this as completed in #6402 May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restoring VM fails when using explicitly passed FDs #6286

Restoring VM fails when using explicitly passed FDs #6286

pupacha commented Mar 11, 2024 •

edited

liuw commented Mar 11, 2024

pupacha commented Mar 12, 2024

liuw commented Mar 13, 2024

pupacha commented Mar 13, 2024 •

edited

pupacha commented Mar 19, 2024

likebreath commented Mar 19, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

praveen-pk commented Mar 21, 2024

likebreath commented Mar 21, 2024

praveen-pk commented Mar 21, 2024

pupacha commented Mar 22, 2024

likebreath commented Mar 22, 2024

pupacha commented Mar 22, 2024

likebreath commented Mar 25, 2024

pupacha commented Mar 26, 2024

rbradford commented Mar 26, 2024

pupacha commented Mar 26, 2024

likebreath commented Mar 26, 2024

pupacha commented Mar 26, 2024

Restoring VM fails when using explicitly passed FDs #6286

Restoring VM fails when using explicitly passed FDs #6286

Comments

pupacha commented Mar 11, 2024 • edited

liuw commented Mar 11, 2024

pupacha commented Mar 12, 2024

liuw commented Mar 13, 2024

pupacha commented Mar 13, 2024 • edited

pupacha commented Mar 19, 2024

1. Adding --restore-net Flag

2. Introducing the vm.restore-net API

likebreath commented Mar 19, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

pupacha commented Mar 20, 2024

rbradford commented Mar 20, 2024

praveen-pk commented Mar 21, 2024

likebreath commented Mar 21, 2024

praveen-pk commented Mar 21, 2024

pupacha commented Mar 22, 2024

likebreath commented Mar 22, 2024

pupacha commented Mar 22, 2024

likebreath commented Mar 25, 2024

pupacha commented Mar 26, 2024

rbradford commented Mar 26, 2024

pupacha commented Mar 26, 2024

likebreath commented Mar 26, 2024

pupacha commented Mar 26, 2024

pupacha commented Mar 11, 2024 •

edited

pupacha commented Mar 13, 2024 •

edited