Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unnecessary memory error #26

Open
valentincalomme opened this issue Nov 13, 2019 · 1 comment
Open

Unnecessary memory error #26

valentincalomme opened this issue Nov 13, 2019 · 1 comment
Labels
improvement Improvement of an existing feature question Interrogation on the functioning of the program

Comments

@valentincalomme
Copy link
Contributor

As far as I can tell, generated examples can be accessed via a generator. However, to write to a file, that generator is turned into a list, which, if that list can cause a memory error.

However, by looking at the source code, it seems that writing to file is done by batch anyway. So why keep all generated examples in memory?

I think this memory error can be easily avoided.

@SimGus
Copy link
Owner

SimGus commented Nov 13, 2019

Hi,

This is not an error but rather a preparation: I'd like to make the generation lazy and have the adapters ask for the examples only when they need them. As you point out, this will allow to use less memory. I just didn't have the time to refactor the adapters (and the command line interface) so that they work this way yet.

Feel free to make those changes if you want to.

@SimGus SimGus added improvement Improvement of an existing feature question Interrogation on the functioning of the program labels Nov 13, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement of an existing feature question Interrogation on the functioning of the program
Projects
None yet
Development

No branches or pull requests

2 participants