test_vip_request test flake

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/logs/test_ztunnel_postsubmit/1778535127471624192#1:build-log.txt%3A1759

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/logs/test_ztunnel_postsubmit/1779908701956608000

[MorrisLaw]

Is there a way to easily see the frequency of this flake as it occurs in the CI? (e.g. how many flakes every N runs).

According to the stack trace, this failing line:

stream.read_exact(&mut buf).await.unwrap();

seems to be related to the chance of flake described here.

// run 15 requests so chance of flake here is 1/2^15 = ~0.003%
run_tcp_client_iters(lb_client, 15, manager.resolver(), &format!("{TEST_VIP}:80"))?;

Wonder if it makes sense to decrease the chance to ~0.00009% by running it 20 times. Or maybe we should just fix the flake altogether, but this comment leads to me believe it's not straightforward? Do you know any additional context about this comment?

[MorrisLaw]

Nvm, job history shows a significant amount of failures more than the chance described in that comment, 5 out of 20! https://prow.istio.io/job-history/gs/istio-prow/logs/test_ztunnel_postsubmit

[howardjohn]

This test was actually removed/replaced (now basically service_loadbalancing). As part of that replacement, the new test seems to be full reliable.

[howardjohn]

209 runs so far, 0 failures (100.00% pass rate). 3.10638293s avg, 3.939583567s max, 1.017441112s min