test_on_demand_dns_request flake

[howardjohn]

https://prow.istio.io/view/gs/istio-prow/pr-logs/pull/istio_ztunnel/900/test_ztunnel/1779885655640248320

[howardjohn]

This is because we use nip.io. Our tests should not depend on external networking

[MorrisLaw]

Is this resolved now since the merging of this PR which removes code that would add_nip_io_nameserver? @howardjohn

[howardjohn]

there is still some nip IO usage I think.. in test_custom_workload? but maybe

[MorrisLaw]

Yeah, this line:

true => format!("example.{}.nip.io.", ip_str),

Just wasn't sure if that was left on purpose or missed.

[MorrisLaw]

Have you had any success reproducing the flake locally; if so, what command? I've just been running make test, which is probably not going to surface it easily.

[MorrisLaw]

Ran a few variations of this so far:

for i in {1..1000}; do
    RUST_BACKTRACE=1 cargo test --test direct test_on_demand_dns_request || break
done

[howardjohn]

I recommend the stress tool (see https://github.com/istio/istio/wiki/Test-Flakes`.

Then you can do something like

stress out/rust/debug/deps/ztunnel-6ec905bca6646dc6 inpod::statemanager::tests::idemepotency_add_workload_fails

(Don't do stress cargo test or it will recompile (not really, since it will detect it is unchanged -- but even that takes a while))

[howardjohn]

I also made a small script, use like rust-flaker direct test_on_demand_dns_request

rust-flaker () {
        relpath () {
                python -c "import os.path; print(os.path.relpath('$1','${2:-$PWD}'))"
        }
        clr='\e[0m'
        blue='\e[0;34m'
        executables="$(cargo test --no-run --message-format json | jq -R 'fromjson? | select(type == "object") | select(.executable != null) | select(.profile.test == true) | select(.target.kind | contains(["bin"]) == false)| .executable' -r)"
        want_name="${1}"
        local bin=""
        while read -r line
        do
                name="$(basename $line | cut -d- -f1)"
                if [[ "${want_name}" == "${name}" ]]
                then
                        echo "${blue}Found test binary: $(relpath $line)${clr}"
                        bin=${line}
                        break
                fi
        done <<< "${executables}"
        if [[ "${bin}" == "" ]]
        then
                "Test binary '${want_name}' not found, have:"
                while read -r line
                do
                        echo "  $(basename $line | cut -d- -f1)"
                done <<< "${executables}"
                return 1
        fi
        echo "${blue}Running test once...${clr}"
        $bin "${2:-}" || return 1
        echo "${blue}Running test repeatedly...${clr}"
        filter="${2:-}"
        shift
        shift
        stress "$@" $bin "${filter}"
}