test_vhost_user_blk tests failed on aarch64 worker

[likebreath]

These tests have been failing multiple times on the main branch:
https://cloud-hypervisor-jenkins.westus.cloudapp.azure.com/blue/rest/organizations/jenkins/pipelines/cloud-hypervisor/branches/main/runs/675/nodes/151/log/?start=0
https://cloud-hypervisor-jenkins.westus.cloudapp.azure.com/blue/rest/organizations/jenkins/pipelines/cloud-hypervisor/branches/main/runs/676/nodes/47/log/?start=0
https://cloud-hypervisor-jenkins.westus.cloudapp.azure.com/blue/rest/organizations/jenkins/pipelines/cloud-hypervisor/branches/main/runs/677/nodes/47/steps/176/log/?start=0

Also errors being reported seem to be very consistent:

[2023-11-09T03:44:43.655Z] ==== Start ssh command output (FAILED) ====
[2023-11-09T03:44:43.655Z] 
[2023-11-09T03:44:43.655Z] command="sudo mount -o rw -t ext4 /dev/vdc mount_image/"
[2023-11-09T03:44:43.655Z] auth="PasswordAuth {
[2023-11-09T03:44:43.655Z]     username: "cloud",
[2023-11-09T03:44:43.655Z]     password: "cloud123",
[2023-11-09T03:44:43.655Z] }"
[2023-11-09T03:44:43.655Z] ip="192.168.48.2"
[2023-11-09T03:44:43.655Z] output=""
[2023-11-09T03:44:43.655Z] error="NonZeroExitStatus(32)"
[2023-11-09T03:44:43.655Z] 
[2023-11-09T03:44:43.655Z] ==== End ssh command outout ====

The first failure was from commit 815c7f4 while the corresponding PR #5925 passed the CI (and I confirmed the PR was running the tests with the latest container image e.g. 20231108-0).

[likebreath]

It is very strange that I can't reproduce the above errors offline when I ran the complete integration test suite with musl build on the aarch64 worker machine manually. I will need to look into it later.

@michael2012z @MrXinWang Please let me know if you have any tips. Thank you.

[michael2012z]

The problem seems tricky, I cannot reproduce the failure in my local machine.

[peng6662001]

I ran test_vhost_user_blk on Ampere Altra and Ampere One and all tests passed.

running 3 tests
[2023-11-30T08:05:43Z ERROR vhost_user_block] Error from the main thread: HandleRequest(Disconnected)      
[2023-11-30T08:05:45Z ERROR vhost_user_block] Error from the main thread: HandleRequest(Disconnected)
[2023-11-30T08:05:47Z ERROR vhost_user_block] Error from the main thread: HandleRequest(Disconnected)
test common_parallel::test_vhost_user_blk_default ... ok
test common_parallel::test_vhost_user_blk_direct ... ok
test common_parallel::test_vhost_user_blk_readonly ... ok

[likebreath]

Right. That's the tricky part. We can't reproduce the error manually, even though the CI is using the same machine...