fix: avoid initial 1s delay and use exponential backoff on spec / status checks for LogicalVolume

[jakobmoellerdev]

fix #904

[jakobmoellerdev]

I believe I spotted a side effect of this change which is kind of strange. There is currently a failure in the resize test while checking for failure events. However, when isolating this test to run alone the failure doesnt occur, making me believe this is actually a regression from a different test.

[jakobmoellerdev]

I think something is completely off here. The tests flake hard and im trying to understand why. This looks like a race condition somewhere, still trying to identify whats happening

[toshipp]

I'll check it next week, but responses may be delayed due to the holiday period.

[jakobmoellerdev]

I have noticed an interesting error message that is concerning to me as it leads me to believe that there might be an issue on the external-resizer patch:

│ Name:             topo-pvc.17c9d3a398b23494                                                                                                                                                                                                                                   │
│ Namespace:        e2etest-xgwhounmef                                                                                                                                                                                                                                          │
│ Labels:           <none>                                                                                                                                                                                                                                                      │
│ Annotations:      <none>                                                                                                                                                                                                                                                      │
│ API Version:      v1                                                                                                                                                                                                                                                          │
│ Count:            1                                                                                                                                                                                                                                                           │
│ Event Time:       <nil>                                                                                                                                                                                                                                                       │
│ First Timestamp:  2024-04-26T12:17:12Z                                                                                                                                                                                                                                        │
│ Involved Object:                                                                                                                                                                                                                                                              │
│   API Version:       v1                                                                                                                                                                                                                                                       │
│   Kind:              PersistentVolumeClaim                                                                                                                                                                                                                                    │
│   Name:              topo-pvc                                                                                                                                                                                                                                                 │
│   Namespace:         e2etest-xgwhounmef                                                                                                                                                                                                                                       │
│   Resource Version:  1509                                                                                                                                                                                                                                                     │
│   UID:               d447f77e-fcb4-49f7-acea-73d26b524755                                                                                                                                                                                                                     │
│ Kind:                Event                                                                                                                                                                                                                                                    │
│ Last Timestamp:      2024-04-26T12:17:12Z                                                                                                                                                                                                                                     │
│ Message:             Mark PVC "e2etest-xgwhounmef/topo-pvc" as file system resize required failed: can't patch status of  PVC e2etest-xgwhounmef/topo-pvc with Operation cannot be fulfilled on persistentvolumeclaims "topo-pvc": the object has been modified; please apply │
│  your changes to the latest version and try again                                                                                                                                                                                                                             │
│ Metadata:                                                                                                                                                                                                                                                                     │
│   Creation Timestamp:  2024-04-26T12:17:12Z                                                                                                                                                                                                                                   │
│   Resource Version:    1524                                                                                                                                                                                                                                                   │
│   UID:                 0199e840-7f9a-422e-b44e-c3014f8f5c0e                                                                                                                                                                                                                   │
│ Reason:                VolumeResizeFailed 

This comes from the external-resizer PVC update for a FileSystemResizeRequired condition failing with a collision and is an issue in our tests because we test for VolumeExpansionFailed events. In this case the PVC is updated quite fast so the Event is triggered but we have an immediate success after.

[toshipp]

LGTM!

[toshipp]

@jakobmoellerdev
If you wish to squash some commits, please tell us.

[jakobmoellerdev]

I will squash

[jakobmoellerdev]

I have the fix for external resizer not sending events in merged now, so once we update to the 1.30 release of resizer, the event test will no longer have to contain the special check.

[toshipp]

I'm glad to hear that. I assume this is yours. Great work! Thank you @jakobmoellerdev