Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow pipe in data in CLI mode #45

Closed
deajan opened this issue Dec 19, 2023 · 3 comments
Closed

Allow pipe in data in CLI mode #45

deajan opened this issue Dec 19, 2023 · 3 comments
Labels
enhancement New feature or request
Milestone

Comments

@deajan
Copy link
Contributor

deajan commented Dec 19, 2023

See N.L

@deajan deajan added the enhancement New feature or request label Dec 20, 2023
@deajan
Copy link
Contributor Author

deajan commented Dec 20, 2023

So restic totally supports this via https://restic.readthedocs.io/en/latest/040_backup.html#reading-data-from-stdin

We need to find a way to redirect stdin to restic without the need of queues or additionnal python plumbing, to avoid memory copies.
command_runner allows giving a subprocess a stdin pipe. Let's see if we can pass a fd (sys.stdin). If so, we need to handle:

  • encoding to use none (bytes), which will affect restic stdout stderr which will need later stage encoding
  • buffering to use standard expected buffers, or even bigger ones since restic deals in generally bigger than 64k chunks (need performance tests)

@deajan
Copy link
Contributor Author

deajan commented May 9, 2024

Basically implemented, we still need to test this in compiled version in both windows and linux.

@deajan deajan added this to the 3.0 milestone May 9, 2024
@deajan deajan mentioned this issue May 10, 2024
20 tasks
@deajan
Copy link
Contributor Author

deajan commented May 15, 2024

Tested on Linux and Windows platforms, both compiled and interpreted versions.

@deajan deajan closed this as completed May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant