CI flake: Admin net pol TestNetworkPolicyV2Conformance test attempting to use port already in-use

[martinkennelly]

2024-05-09T17:45:20.1404158Z === RUN   TestNetworkPolicyV2Conformance
2024-05-09T17:45:20.1407058Z     network_policy_v2_test.go:31: Configuring environment for network policy V2 API conformance tests
2024-05-09T17:45:20.1421212Z     network_policy_v2_test.go:53: Starting the network policy V2 conformance test suite
2024-05-09T17:45:20.1422439Z     suite.go:125: Test Setup: Applying base manifests
2024-05-09T17:45:20.2616014Z     apply.go:124: Creating network-policy-conformance-gryffindor Namespace
2024-05-09T17:45:20.2686256Z     apply.go:124: Creating network-policy-conformance-slytherin Namespace
2024-05-09T17:45:20.2850881Z     apply.go:124: Creating network-policy-conformance-hufflepuff Namespace
2024-05-09T17:45:20.2958762Z     apply.go:124: Creating network-policy-conformance-ravenclaw Namespace
2024-05-09T17:45:20.3033351Z     apply.go:124: Creating network-policy-conformance-forbidden-forrest Namespace
2024-05-09T17:45:20.3185458Z     apply.go:124: Creating harry-potter StatefulSet
2024-05-09T17:45:20.3249033Z     apply.go:124: Creating draco-malfoy StatefulSet
2024-05-09T17:45:20.3312093Z     apply.go:124: Creating cedric-diggory StatefulSet
2024-05-09T17:45:20.3584068Z     apply.go:124: Creating luna-lovegood StatefulSet
2024-05-09T17:45:20.4050777Z     apply.go:124: Creating centaur StatefulSet
2024-05-09T17:45:20.4482467Z     suite.go:128: Test Setup: Ensuring Namespaces and Pods from base manifests are ready
2024-05-09T17:45:20.4551943Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
2024-05-09T17:45:21.4627860Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
....
....
2024-05-09T17:50:19.4577962Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-forbidden-forrest not rolled out yet. 1/2 replicas are available.
2024-05-09T17:50:20.4499365Z     helper.go:120: Error retrieving StatefulSet harry-potter from namespace network-policy-conformance-gryffindor: client rate limiter Wait returned an error: context deadline exceeded
2024-05-09T17:50:20.4501830Z     helper.go:123: StatefulSet replicas in namespace network-policy-conformance-gryffindor not rolled out yet. 0/2 replicas are available.
2024-05-09T17:50:20.4502682Z     suite.go:143: 
2024-05-09T17:50:20.4503713Z         	Error Trace:	/home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/kubernetes/helper.go:130
2024-05-09T17:50:20.4505387Z         	            				/home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/suite/suite.go:143
2024-05-09T17:50:20.4507231Z         	            				/home/runner/go/pkg/mod/sigs.k8s.io/network-policy-api@v0.1.5/conformance/utils/suite/conformance_suite.go:132
2024-05-09T17:50:20.4509264Z         	            				/home/runner/work/ovn-kubernetes/ovn-kubernetes/test/conformance/network_policy_v2_test.go:89
2024-05-09T17:50:20.4509992Z         	Error:      	Received unexpected error:
2024-05-09T17:50:20.4510860Z         	            	context deadline exceeded
2024-05-09T17:50:20.4511779Z         	Test:       	TestNetworkPolicyV2Conformance
2024-05-09T17:50:20.4514561Z         	Messages:   	error waiting for network-policy-conformance-gryffindor, network-policy-conformance-slytherin, network-policy-conformance-hufflepuff, network-policy-conformance-ravenclaw, network-policy-conformance-forbidden-forrest namespaces to be ready

I checked the logs for why one of the replicas pods isnt coming up and it complained about trying to bind to a port already in-use:

$cat 5.log 
2024-05-09T17:48:22.496335577Z stderr F I0509 17:48:22.496230       1 log.go:194] Started HTTP server on port 34358
2024-05-09T17:48:22.496458999Z stderr F I0509 17:48:22.496401       1 log.go:194] Started UDP server on port 172.18.0.3 34345
2024-05-09T17:48:22.4965566Z stderr F I0509 17:48:22.496512       1 log.go:194] listen tcp :34358: bind: address already in use

Seen in #4344 (https://github.com/ovn-org/ovn-kubernetes/actions/runs/9020680306/job/24814796912?pr=4344)

First time seeing it.

[martinkennelly]

cc @tssurya

[tssurya]

thanks @martinkennelly for catching this
/assign @tssurya