Cannot allocate memory

[keflavich]

Trying to load a cube, I've encountered the following error:
error: [Errno 12] Cannot allocate memory

Unfortunately, this defeats the purpose of spectral-cube entirely. Any ideas for workarounds? In principle, we might be able to write up an interface that can load only part of the cube, but I don't know how to do that with the FITS io system.

[keflavich]

Example traceback:

In [19]: fits.open('full_W51_7m12m_spw0_hires_lines.fits')[0].data.shape
Traceback (most recent call last):
  File "<ipython-input-19-f94477ee2d18>", line 1, in <module>
    fits.open('full_W51_7m12m_spw0_hires_lines.fits')[0].data.shape
  File "/diskb/aginsbur/repos/astropy/astropy/utils/decorators.py", line 498, in __get__
    val = self.fget(obj)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 208, in data
    data = self._get_scaled_image_data(self._data_offset, self.shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 621, in _get_scaled_image_data
    raw_data = self._get_raw_data(shape, code, offset)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/base.py", line 568, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/file.py", line 268, in readarray
    dtype=np.uint8)
  File "/home/aginsbur/anaconda/lib/python2.7/site-packages/numpy/core/memmap.py", line 260, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
error: [Errno 12] Cannot allocate memory

[keflavich]

Solution is here: astropy/astropy#1380 (comment)

[keflavich]

This has led to a new error:

In [7]: fits.open('full_W51_7m12m_spw1_hires_lines.fits')[0].data.shape
Traceback (most recent call last):
  File "<ipython-input-7-0459cfea92b3>", line 1, in <module>
    fits.open('full_W51_7m12m_spw1_hires_lines.fits')[0].data.shape
  File "/diskb/aginsbur/repos/astropy/astropy/utils/decorators.py", line 498, in __get__
    val = self.fget(obj)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 208, in data
    data = self._get_scaled_image_data(self._data_offset, self.shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 621, in _get_scaled_image_data
    raw_data = self._get_raw_data(shape, code, offset)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/base.py", line 568, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/file.py", line 283, in readarray
    buffer=self._mmap)
TypeError: buffer is too small for requested array

In [8]: fits.open('full_W51_7m12m_spw1_hires_lines.fits', mode='denywrite')[0].data.shape
Traceback (most recent call last):
  File "<ipython-input-8-95c009e12b4b>", line 1, in <module>
    fits.open('full_W51_7m12m_spw1_hires_lines.fits', mode='denywrite')[0].data.shape
  File "/diskb/aginsbur/repos/astropy/astropy/utils/decorators.py", line 498, in __get__
    val = self.fget(obj)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 208, in data
    data = self._get_scaled_image_data(self._data_offset, self.shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/image.py", line 621, in _get_scaled_image_data
    raw_data = self._get_raw_data(shape, code, offset)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/hdu/base.py", line 568, in _get_raw_data
    return self._file.readarray(offset=offset, dtype=code, shape=shape)
  File "/diskb/aginsbur/repos/astropy/astropy/io/fits/file.py", line 283, in readarray
    buffer=self._mmap)
TypeError: buffer is too small for requested array

[keflavich]

The change broke the tests because the tests are using direct file modification, which isn't supported. Shouldn't be too hard to fix.

[coveralls]

Coverage remained the same at 72.695% when pulling e82b66a on keflavich:huge-file-issue299 into c09ac14 on radio-astro-tools:master.

[e-koch]

Isn't mode passed through kwargs from SpectralCube.read to here already? I regularly use mode='denywrite' and haven't run into memory issues when dealing with big cubes.

[keflavich]

Yes. But I think this should be the default, as we don't expect users to be modifying the cube data inplace.

[e-koch]

Ah ok. Agreed.