Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression: "invalid flags combined with -t" when resuming with syncoid --sendoptions #918

Open
Deltik opened this issue Apr 24, 2024 · 0 comments · May be fixed by #919
Open

Regression: "invalid flags combined with -t" when resuming with syncoid --sendoptions #918

Deltik opened this issue Apr 24, 2024 · 0 comments · May be fixed by #919

Comments

@Deltik
Copy link
Contributor

Deltik commented Apr 24, 2024 •
edited

Bug Description

Syncoid with --sendoptions now fails with the error invalid flags combined with -t when resuming an interrupted transfer:

root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
INFO: Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
invalid flags combined with -t
usage:
        send [-DnPpRvLecwhb] [-[i|I] snapshot] <snapshot>
        send [-DnvPLecw] [-i snapshot|bookmark] <filesystem|volume|snapshot>
        send [-DnPpvLec] [-i bookmark|snapshot] --redact <bookmark> <snapshot>
        send [-nvPe] -t <receive_resume_token>
        send [-Pnv] --saved filesystem

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                                               ]  0%            
cannot receive: failed to read from stream
CRITICAL ERROR:  zfs send -p -w  -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1 | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 8536288 |  zfs receive  -s -F 'jimsalterjrs_sanoid_918/destination' 2>&1 failed: 256

This is a regression introduced by commit 09b42d6:

root@demo:~/sanoid# git bisect log
git bisect start
# bad: [b31ed6e325e4e145d112fab582e348f6ca90631b] Merge pull request #916 from 0xFelix/zstdmt
git bisect bad b31ed6e325e4e145d112fab582e348f6ca90631b
# good: [9564454d74d9b8bc2b924ee261a6e40c5c278662] Merge pull request #601 from phreaker0/prepare-2.1.0
git bisect good 9564454d74d9b8bc2b924ee261a6e40c5c278662
# good: [3bde229a617b2fcb9976e8837d09b6e136df07c2] Merge pull request #622 from rbike/syncoid-hold
git bisect good 3bde229a617b2fcb9976e8837d09b6e136df07c2
# bad: [790ea544ff82bb4945c269b3fe6e451076d6307b] Merge branch 'master' into zfs-get
git bisect bad 790ea544ff82bb4945c269b3fe6e451076d6307b
# good: [18ccb7df350d22bd02a4efd01e4fefd1db4bdc3f] Fix tiny typo in README.md
git bisect good 18ccb7df350d22bd02a4efd01e4fefd1db4bdc3f
# bad: [ca6e60b920994b5a61b295674121aa23f32dc0bb] Merge branch 'master' into update-send-recv-options
git bisect bad ca6e60b920994b5a61b295674121aa23f32dc0bb
# bad: [14ed85163a82c743d67c2b25fac1882f302ab0ea] Filter snapshots in getsnapsfallback()
git bisect bad 14ed85163a82c743d67c2b25fac1882f302ab0ea
# bad: [603c286b50128a3ceb061f0e2c03310a00220977] Don't iterate over snaps twice
git bisect bad 603c286b50128a3ceb061f0e2c03310a00220977
# bad: [09b42d6ade2843171be810db7df3b189954b7992] Refactor system calls
git bisect bad 09b42d6ade2843171be810db7df3b189954b7992
# good: [c4e70280225f7b3ec03cc075daf218dde468412f] Refactor terminal output
git bisect good c4e70280225f7b3ec03cc075daf218dde468412f
# first bad commit: [09b42d6ade2843171be810db7df3b189954b7992] Refactor system calls

How to Reproduce

I hacked together a git bisect run script that finds the regression:

#!/bin/bash

truncate -s128M /tmp/jimsalterjrs_sanoid_918.img
zpool create jimsalterjrs_sanoid_918 /tmp/jimsalterjrs_sanoid_918.img
zfs create jimsalterjrs_sanoid_918/source
zfs snap jimsalterjrs_sanoid_918/source@empty
dd if=/dev/urandom of=/jimsalterjrs_sanoid_918/source/garbage.bin bs=1M count=16
zfs snap jimsalterjrs_sanoid_918/source@something

zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination

./syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/source jimsalterjrs_sanoid_918/destination

if [ $? -ne 0 ]; then
    echo "Regression detected"
    zfs destroy -r jimsalterjrs_sanoid_918/destination
    zpool destroy jimsalterjrs_sanoid_918
    exit 1
else
    echo "No regression detected"
    zfs destroy -r jimsalterjrs_sanoid_918/destination
    zpool destroy jimsalterjrs_sanoid_918
    exit 0
fi

Manual run:

root@demo:~# truncate -s128M /tmp/jimsalterjrs_sanoid_918.img
root@demo:~# zpool create jimsalterjrs_sanoid_918 /tmp/jimsalterjrs_sanoid_918.img
root@demo:~# zfs create jimsalterjrs_sanoid_918/source
root@demo:~# zfs snap jimsalterjrs_sanoid_918/source@empty
root@demo:~# dd if=/dev/urandom of=/jimsalterjrs_sanoid_918/source/garbage.bin bs=1M count=16
16+0 records in
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.303385 s, 55.3 MB/s
root@demo:~# zfs snap jimsalterjrs_sanoid_918/source@something
root@demo:~# zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination
cannot receive incremental stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1
root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
8.28MiB 0:00:00 [ 118MiB/s] [=============================================================================================================================================================================================================================================] 101%            
Sending incremental jimsalterjrs_sanoid_918/source@something ... syncoid_demo_2024-04-24:17:39:27-GMT00:00 (~ 4 KB):
2.23KiB 0:00:00 [79.3KiB/s] [=================================================================================================================================>                                                                                                            ] 55%            
root@demo:~# zfs destroy -r jimsalterjrs_sanoid_918/destination
root@demo:~# zfs send -pwR jimsalterjrs_sanoid_918/source@something | head --bytes=8M | zfs recv -s jimsalterjrs_sanoid_918/destination
cannot receive incremental stream: checksum mismatch or incomplete stream.
Partially received snapshot is saved.
A resuming stream can be generated on the sending system by running:
    zfs send -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1
root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
INFO: Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
invalid flags combined with -t
usage:
        send [-DnPpRvLecwhb] [-[i|I] snapshot] <snapshot>
        send [-DnvPLecw] [-i snapshot|bookmark] <filesystem|volume|snapshot>
        send [-DnPpvLec] [-i bookmark|snapshot] --redact <bookmark> <snapshot>
        send [-nvPe] -t <receive_resume_token>
        send [-Pnv] --saved filesystem

For the property list, run: zfs set|get

For the delegated permission list, run: zfs allow|unallow
0.00 B 0:00:00 [0.00 B/s] [>                                                                                                                                                                                                                                               ]  0%            
cannot receive: failed to read from stream
CRITICAL ERROR:  zfs send -p -w  -t 1-1694168810-138-789c636064000310a501c49c50360710a715e5e7a69766a6304081c717d7729790471a0a40363b92bafca4acd4e412060626a83a0cf9b4b4e2d412904c0d5c9e0d493ea9b224b518484f08aec3aabf241fe28a7d1dcb2eddf7f9a9ef8124cf0996cf4bcc4d6560c8cacc2d4ecc29492dca2a2a8e2f4ecccbcf4c89b734b4d02fce2f2d4a4e7528cecf4d2dc9c8cc4b87f947026a3eccffa9b949a929f9d960fbb991c493f3730b8a528b8b81527000002ebc32a1 | mbuffer  -q -s 128k -m 16M | pv -p -t -e -r -b -s 8536288 |  zfs receive  -s -F 'jimsalterjrs_sanoid_918/destination' 2>&1 failed: 256
root@demo:~# echo $?
2

Expected Behavior

The unsupported syncoid --sendoptions are removed, allowing interrupted send/receive operations to complete.

In Sanoid v2.1.0, the output looked like this:

root@demo:~# syncoid --sendoptions="pw" jimsalterjrs_sanoid_918/{source,destination}
Resuming interrupted zfs send/receive from jimsalterjrs_sanoid_918/source to jimsalterjrs_sanoid_918/destination (~ 8.1 MB remaining):
8.28MiB 0:00:00 [ 118MiB/s] [=============================================================================================================================================================================================================================================] 101%            
Sending incremental jimsalterjrs_sanoid_918/source@something ... syncoid_demo_2024-04-24:17:39:27-GMT00:00 (~ 4 KB):
2.23KiB 0:00:00 [79.3KiB/s] [=================================================================================================================================>                                                                                                            ] 55%
root@demo:~# echo $?
0
Deltik added a commit to Deltik/sanoid that referenced this issue Apr 24, 2024
@Deltik
fix(syncoid): zfs send arg allowlist when sendsource is receivetoken
e9ad83a 
The `runsynccmd` subroutine was not matching the `$sendsource` when a
receive resume token is passed in. All usages that pass in the receive
resume token do not begin with a space; instead, they start with `-t `.

Fixes: jimsalterjrs#918
Deltik added a commit to Deltik/sanoid that referenced this issue Apr 24, 2024
@Deltik
fix(syncoid): zfs send arg allowlist when sendsource is receivetoken
afc2d7d 
The `runsynccmd` subroutine was not matching the `$sendsource` when a
receive resume token is passed in. All usages that pass in the receive
resume token do not begin with a space; instead, they start with `-t `.

Fixes: jimsalterjrs#918
Deltik added a commit to Deltik/sanoid that referenced this issue Apr 24, 2024
@Deltik
fix(syncoid): zfs send arg allowlist when sendsource is receivetoken
c5fed7c 
The `runsynccmd` subroutine was not matching the `$sendsource` when a
receive resume token is passed in. All usages that pass in the receive
resume token do not begin with a space; instead, they start with `-t `.

Fixes: jimsalterjrs#918
Deltik added a commit to Deltik/sanoid that referenced this issue Apr 24, 2024
@Deltik
fix(syncoid): zfs send arg allowlist when sendsource is receivetoken
a625b9c 
The `runsynccmd` subroutine was not matching the `$sendsource` when a
receive resume token is passed in. All usages that pass in the receive
resume token do not begin with a space; instead, they start with `-t `.

Fixes: jimsalterjrs#918
@Deltik Deltik linked a pull request Apr 24, 2024 that will close this issue
Open
6 tasks
Deltik added a commit to Deltik/sanoid that referenced this issue Apr 26, 2024
@Deltik
fix(syncoid): zfs send arg allowlist when sendsource is receivetoken
fab4b40 
The `runsynccmd` subroutine was not matching the `$sendsource` when a
receive resume token is passed in. All usages that pass in the receive
resume token do not begin with a space; instead, they start with `-t `.

Fixes: jimsalterjrs#918
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

syncoid: Fix typo preventing resumed transfer with --sendoptions Deltik/sanoid
1 participant
@Deltik