Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clean.py memory usage #44

Open
coolio51 opened this issue Jan 7, 2021 · 0 comments
Open

Clean.py memory usage #44

coolio51 opened this issue Jan 7, 2021 · 0 comments

Comments

@coolio51
Copy link

coolio51 commented Jan 7, 2021

Hi Seth,

In the clean.py, the memory usage (RAM) keeps going up. So, if a large dataset with many classes is being studied, eventually, the computer goes out of memory. I get this message from time to time:

MemoryError: Unable to allocate 1.22 MiB for an array with shape (160399,) and data type int64

This can easily be dealt with by increasing the page.sys file memory size, increasing computer spec, or applying clean.py one subset of the data at a time but I was wondering if it would not be better to dump 'garbage' between two classes. In your YT video, that would mean dumping memory between Cello and Clarinet for example.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant