Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstream should be able to rechunk records. #16184

Open
rincebrain opened this issue May 10, 2024 · 3 comments
Open

zstream should be able to rechunk records. #16184

rincebrain opened this issue May 10, 2024 · 3 comments
Labels
Component: Send/Recv "zfs send/recv" feature Type: Feature Feature request or new feature

Comments

@rincebrain
Copy link
Contributor

Describe the feature would like to see added to OpenZFS

Similar to #16183, but instead of en/decrypting, I think we already understand the edge cases involved in doing something like zfs send -RLec mypool/ds1_128k@snap4 | zstream resize -o recordsize=4M | ....

Obviously you're not going to get away with this on something that's encrypted when it comes in, but I think at most you need a new partial write verb for send streams, or just handling in receive directly, to be able to say "I know this is a 4M record in your head, but please go rewrite this 128k region", since IIRC right now the verbs you have in send streams are just "delete" and "write entire record".

How will this feature improve OpenZFS?

A lot of time has been spent, at least by me, on trying to make tooling to automatically migrate things with rename awareness and similar to preserve snapshots when migrating recordsizes, since send|recv will only downsize it for you. The other direction is, I think, pretty feasible, and would make life for people, say, testing a bunch of different recordsizes on their existing snapshots over time work much more conveniently.

Additional context

N/A

@rincebrain rincebrain added Type: Feature Feature request or new feature Component: Send/Recv "zfs send/recv" feature labels May 10, 2024
@shodanshok
Copy link
Contributor

A lot of time has been spent, at least by me, on trying to make tooling to automatically migrate things with rename awareness and similar to preserve snapshots when migrating recordsizes, since send|recv will only downsize it for you.

Maybe I missed something, but how can zfs send | zfs recv reduce recordsize?

While it is true that zfs send tank/zsrc@snap1 | zfs recv tank/zdst -o recordsize=32k creates tank/zdst with 32K recordsize, the transferred files actually have the original recordsize (128K in this case). Are you talking about a large_record source (>1MB records) and a smaller, default recordsize (128K) destination?

Thanks.

@rincebrain
Copy link
Contributor Author

If you do zfs send foo/bar@snap1 and foo/bar has recordsize > 128k, and did not use zfs send -L, it will chunk them up into 128k records.

@shodanshok
Copy link
Contributor

Thanks for explaining. I agree that the ability to "rechunk" records would be a good thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Send/Recv "zfs send/recv" feature Type: Feature Feature request or new feature
Projects
None yet
Development

No branches or pull requests

2 participants