[BUG] system restore stuck because of the volume/PV/PVC restoration

[ChanYiLin]

Describe the bug

in our code https://github.com/longhorn/longhorn-manager/blob/ebace7b56389b017c97876f15b5298d07095b745/controller/system_rollout_controller.go#L1264-L1290
PV restoration will tried to find the Volume first
PVC restoration will tried to find the PV first
But I have moved the Volume restoration out of the parallel restoration
So all 3 resources won't be able to restored and because of the deadlock

To Reproduce

Create a Volume/PV/PVC and do the system backup and restore

Expected behavior

All 3 resources should be restored successfully

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

1. Create a BackingImage vol-0 , Created From Download From URL and a volume using it
2. Create the PV/PVC for the volume
3. Create a system-backup
4. kubectl get lhbbi, lhb -n longhorn-system will see the backup of the Volume and the Backingimage
5. Delete the Volume and the BackingImage and the PV and PVC
6. Restore the system-backup
7. The BackingImage and The BackingImage source type will be restored, and the Volume/PV/PVC will be restored correctly

PR: longhorn/longhorn-manager#2819

[c3y1huang]

@ChanYiLin , should we have an automation test case to cover this?

[ChanYiLin]

Hi @c3y1huang Yes, I will modify the origin test case to include PV and PVC after the fix merged
here: https://github.com/longhorn/longhorn-tests/blob/master/manager/integration/tests/test_system_backup_restore.py#L121

[roger-ryao]

Verified on master-head 20240530

• longhorn master-head f79dc9b
• longhorn-manager master-head longhorn/longhorn-manager@ad74200

The test steps
#8601 (comment)

Result Passed