Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for streaming large files #128

Open
mykola-mokhnach opened this issue Sep 16, 2020 · 4 comments
Open

Add support for streaming large files #128

mykola-mokhnach opened this issue Sep 16, 2020 · 4 comments

Comments

@mykola-mokhnach
Copy link

Hello Adam,

Thanks for the great library, it's really useful.
I'd like to mention one thing about its usage: when trying to perform replacements in big files (like hundreds of megabytes) then OutOfMemory errors could easily pop up, because the lib loads the file content fully into the memory before, actually, making the replacement. I'm not quite sure whether this was done by design, but it, probably, makes sense to at least mention this fact in README.

An alternative solution, which is memory-friendly (uses streams), could be something like https://github.com/eugeneware/replacestream

@adamreisnz
Copy link
Owner

Hi there, yep some work on this was started in #40, but that has eventually been abandoned.

I'd be happy to review and accept PR's that implement some kind of streaming option or support for large files.

However, I'm not sure how the replace algorithm will work correctly if the file contents are read in chunks (for example, the regex may not find a match when a chunk is split exactly at a match).

@adamreisnz adamreisnz changed the title The replacement algorithm does not effectively use RAM Add support for streaming large files Sep 18, 2020
@Noeldesarrollo
Copy link

Hello @adamreisnz
I have a solution implemented and tested, but i don't have permissions to create a branch.

@adamreisnz
Copy link
Owner

Hi @Noeldesarrollo , please fork the repository and then open a pull request, and I'll be able to have a look.
Thanks!

@Noeldesarrollo
Copy link

PR: #139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants