Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Created read_tipsy_ascii to speed up reading of ascii auxiliary tipsy… #373

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ibackus
Copy link

@ibackus ibackus commented May 2, 2017

… arrays.

This is to address issue 371

Compared to before, I see about a 4x speedup in ASCII read-time, tested on arrays from 10^5 to 10^8 particles in length. However, more memory will be used since a full copy of the array needs to be generated. See below.

This method skips using the slow numpy.fromtxt in favor of readlines. Unfortunately, readlines reads an approximate amount of text in bytes and therefore we cannot specify the number of lines to read ahead of time. This means that to perform the mapping from the disk-indexing to the memory-indexing I read the entire array from disk then map its elements to a second array. This means that during read time double the memory will be required, which is not ideal. This could be avoid by a smarter interleaving of disk buffers: 1 buffer for readlines and 1 buffer for the memory indexing, but that is a little more involved to implement.

… arrays.

Removed unused ascii loadblock function from tipsy reader
@apontzen
Copy link
Member

apontzen commented May 3, 2017

Unfortunately I think this swaps CPU complexity for memory complexity and it's not obvious to me that we'll actually help people. Certainly I've been in situations where I'm basically memory-bound with large simulations. If we could have a solution that sticks within the "block" approach that'd be awesome!

@ibackus
Copy link
Author

ibackus commented May 3, 2017

Yeah, that's what I was thinking. It can be done, but it's a little more work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants