qa: Non-deterministic behavior in walletnotify.py

[patricklodder]

There seems to be non-deterministic behavior in walletnotify.py (or in CWallet::AddToWallet) where 30-40% of the time, perhaps only under load, CI fails, always when testing notifications after reorgs.

  File "/home/runner/work/dogecoin/dogecoin/qa/rpc-tests/test_framework/test_framework.py", line 146, in main
    self.run_test()
  File "/home/runner/work/dogecoin/dogecoin/qa/rpc-tests/walletnotify.py", line 118, in run_test
    assert self.notifs[self.current_line + 1] == "{} {}".format(txid, 0)

[xanimo]

I've experienced this multiple times over the past few weeks 😿 most recently https://github.com/xanimo/dogecoin/actions/runs/8547684076/job/23422183133 so +1

[daank-c]

I picked through this a bit, and wanted to mention that the latest one-off CI failure for walletnotify.py actually appears further down at Line 175.

From #3514

Failed test log: https://github.com/dogecoin/dogecoin/actions/runs/8618664633/job/23621563836

Failed

stderr:
  File "/home/runner/work/dogecoin/dogecoin/qa/rpc-tests/test_framework/test_framework.py", line 146, in main
    self.run_test()
  File "/home/runner/work/dogecoin/dogecoin/qa/rpc-tests/walletnotify.py", line 175, in run_test
    assert self.notifs[self.current_line + i] == "{} {}".format(txid, 0)

Pass: False, Duration: 15 s

[patricklodder]

The 40% failure rate wasn't a joke - I'm manually restarting nearly half the CI runs.

Line 175 is the other place where we're testing notifications post-reorg, just like line 118.

The first fail condition happens in the sunny case where the same tx gets mined post-reorg and the second fail condition happens in the rainy case where we get doublespent after a reorg.

I'll put fixing this higher on my list.