rsync error: unexplained error (code 255) at io.c(228)

[muzammil-iftikhar]

Describe the bug
I am trying to move pvc from one cluster to another. When i run the pv-migrate command it fails with the error :

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]"

To Reproduce
Steps to reproduce the behavior:

pv-migrate migrate \
  --source-kubeconfig ~/.kube/config_dev \
  --source-context tkc-dev-wso2-ns001 \
  --source-namespace muz \
  --dest-kubeconfig ~/.kube/config_prod \
  --dest-context tkc-prod-wso2-ns001 \
  --dest-namespace muz \
  --dest-delete-extraneous-files \
  --ignore-mounted \
  --strategies "lbsvc" \
  --log-level debug \
  --log-format json \
  demo-pvc demo-new-pvc

Expected behavior
pv-migrate should migrate the pvc from source cluster to destination cluster successfully

Console output

{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"ssh: connect to host 10.50.49.34 port 22: Operation timed out","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync attempt 1/10 failed, waiting 5 seconds before trying again","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"ssh: connect to host 10.50.49.34 port 22: Operation timed out","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync attempt 2/10 failed, waiting 5 seconds before trying again","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"ssh: connect to host 10.50.49.34 port 22: Operation timed out","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync attempt 3/10 failed, waiting 5 seconds before trying again","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"ssh: connect to host 10.50.49.34 port 22: Operation timed out","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync attempt 4/10 failed, waiting 5 seconds before trying again","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"ssh: connect to host 10.50.49.34 port 22: Operation timed out","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}
{"dest":"demo-new-pvc","dest_ns":"muz","id":"dbdab","level":"debug","msg":"rsync error: unexplained error (code 255) at io.c(228) [Receiver=3.2.3]","source":"demo-pvc","source_ns":"muz","time":"2022-11-16T11:25:55+05:00"}

• Source and destination Kubernetes versions [e.g. v1.17.14-gke.1600, v1.21.1+k3s1]
  Client Version: v1.19.1+wcp.3
  Server Version: v1.22.9+vmware.1

• Source and destination container runtimes [e.g. containerd://1.4.4-k3s2, docker://19.3.6]
  containerd://1.5.11

• pv-migrate version and architecture [e.g. v0.5.5 - darwin_x86_64]
  pv-migrate version 1.0.1 (commit: 45224c5) (build date: 2022-11-03T23:35:05Z)
  linux_x86_64

• Installation method [e.g. homebrew, binary download]
  binary download

• Source and destination PVC type, size and accessModes [e.g. ReadWriteMany, 8G, kubernetes.io/gce-pd -> ReadWriteOnce, N/A, rancher.io/local-path ]
  Source:

Name:          demo-pvc
Namespace:     muz
StorageClass:  tkg-rwx-policy-dev
Status:        Bound
Volume:        pvc-c80ee98b-1fd8-44bb-ac65-dcdd77b927ab
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100Mi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    pv-migrate-dbcdd-src-sshd-66dc6f84b9-wt5c8
Events:        <none>

Destination:

Name:          demo-new-pvc
Namespace:     muz
StorageClass:  tkg-rwx-policy-prod
Status:        Bound
Volume:        pvc-d9b50a86-053c-425c-8dde-565ab476ebda
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      100Mi
Access Modes:  RWO
VolumeMode:    Filesystem
Mounted By:    pv-migrate-dbcdd-dest-sshd-6c6bf94d97-vsts7
Events:        <none>

• Laptop operating system:
  Distributor ID: RedHatEnterprise
  Description: Red Hat Enterprise Linux release 8.4 (Ootpa)
  Release: 8.4
  Codename: Ootpa

[syedsaadahmed]

Facing the same error while using the pv-migrate migrate with the same flags.

[Itdevopsnet]

Any update for this issue?

[utkuozdemir]

The unexplained error (code 255) is a generic error that can mean a lot of things. In the case of the original issue, the connectivity does not succeed:

ssh: connect to host 10.50.49.34 port 22: Operation timed out

Destination IP seems to be a private one (10.x...) - are you sure that that IP is supposed to be reachable at all from the other cluster? Over port 22?