[BACKPORT][v1.5.6][BUG] Volume failed to create healthy replica after data locality and replica count changed and got stuck in degraded state forever

[github-actions]

backport #8522

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at: [BUG] Volume failed to create healthy replica after data locality and replica count changed and got stuck in degraded state forever #8522 (comment).

• Is there a workaround for the issue? If so, where is it documented?
  Delete the unscheduled replica with the HardNodeAffinity.

• Does the PR include the explanation for the fix or the feature?

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at: Always clean up FailedToScheduleReplica with wrong HardNodeAffinity (backport #2792) longhorn-manager#2810,

[roger-ryao]

After executing the Jenkins job, we could observe the issue on the v1.5.x-head. https://ci.longhorn.io/job/private/job/longhorn-tests-regression/6975/
After the longhorn/longhorn-manager#2810 is merged, we can rerun the job again for verification.