Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow reading of arbitrary dimension tipsy aux arrays (e.g. 1D or 3D) #403

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ibackus
Copy link

@ibackus ibackus commented Aug 11, 2017

This addresses issue #379.

This is unfortunately not straightforward, since the tipsy format does not include information about the number of columns in a tipsy array.

There are several different file types, all which have dimensions inferred
using different methods:

  • For binary arrays, the number of dimensions are inferred robustly from the
    file size, dtype, and number of particles.
  • For gzipped binary arrays, the uncompressed filesize cannot be directly
    read for files >4GB, but in general it is possible to correctly estimate the
    number of dimensions
  • For ascii arrays, the number of dimensions are estimated by estimating the
    average size (in bytes) of a single data point and dividing the file size
    by this and the number of particles. This should be fairly robust except in
    some edge cases. A warning is raised if the estimate appears bad.
  • Reading of gzipped ASCII arrays has also been implemented, but the number
    of dimensions cannot be estimated

This addresses issue pynbody#379.

There are several different file types, all which have dimensions inferred
using different methods:

 * For binary arrays, the number of dimensions are inferred robustly from the
file size, dtype, and number of particles.
 * For gzipped binary arrays, the uncompressed filesize cannot be directly
read for files >4GB, but in general it is possible to correctly estimate the
number of dimensions
 * For ascii arrays, the number of dimensions are estimated by estimating the
average size (in bytes) of a single data point and dividing the file size
by this and the number of particles.  This should be fairly robust except in
some edge cases.  A warning is raised if the estimate appears bad.
 * Reading of gzipped ASCII arrays has also been implemented, but the number
of dimensions cannot be estimated
@ibackus
Copy link
Author

ibackus commented Aug 11, 2017

I'm not sure why this has failed a test. It has failed the python 3.6 test, with the error message:

File "/home/travis/miniconda/envs/3.6_1.13.1/lib/python3.6/site-packages/pynbody-0.41-py3.6-linux-x86_64.egg/pynbody/snapshot/tipsy.py", line 899, in __read_array_from_disk
    array_shape = [n_read, ndim]
UnboundLocalError: local variable 'ndim' referenced before assignment

However, it should not be able to get to that point in the code without ndim being assigned a value. This is especially bizarre, because all the other test builds are passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant