Created read_tipsy_ascii to speed up reading of ascii auxiliary tipsy… #373
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… arrays.
This is to address issue 371
Compared to before, I see about a 4x speedup in ASCII read-time, tested on arrays from 10^5 to 10^8 particles in length. However, more memory will be used since a full copy of the array needs to be generated. See below.
This method skips using the slow
numpy.fromtxt
in favor ofreadlines
. Unfortunately,readlines
reads an approximate amount of text in bytes and therefore we cannot specify the number of lines to read ahead of time. This means that to perform the mapping from the disk-indexing to the memory-indexing I read the entire array from disk then map its elements to a second array. This means that during read time double the memory will be required, which is not ideal. This could be avoid by a smarter interleaving of disk buffers: 1 buffer forreadlines
and 1 buffer for the memory indexing, but that is a little more involved to implement.