Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interpolation fails with single-precision data (at least sometimes?) #255

Open
pkgw opened this issue Sep 2, 2021 · 1 comment
Open

Comments

@pkgw
Copy link
Contributor

pkgw commented Sep 2, 2021

I'm trying to use reproject with some large images. Even though the FITS files are 12 GB and my system has 64 GB of RAM, I'm running out of memory, so I've been investigating ways to be more memory efficient. I'm currently trying to implement the memmap recommendation from the docs.

In so doing, reproject start rejected an innocuous-seeming transformation:

out = np.memmap(filename='temp.dat', mode='w+', shape=shape_out, dtype=input.dtype)
reproject_interp(
  (input, wcs),
  output_array=out,
  output_projection=wcs_out,
  shape_out=shape_out,
  return_footprint=False,
)

with:

ValueError: An output array of a different type than the input array was specified, which will create an undesired duplicate copy of the input array in memory.

I'm explicitly initializing my arrays to have the same dtype — how is this happening?

The culprit turns out to be:

array = np.asarray(array, dtype=float)

The input array is float32 data, and on my machine, at least (Linux x86_64 numpy 1.21.2), this line converts it to float64. This is counterintuitive at best and is also going to be doubling my memory consumption.

I'm pretty sure that this is a mistake — unless I'm missing something? I wanted to ask before working on a PR, although I'm pretty sure that there's a one-line way to ensure float-ness without unnecessarily converting.

@pkgw
Copy link
Contributor Author

pkgw commented Sep 2, 2021

Unrelated but while I'm at it: the big-mem docs use an independent_celestial_slices argument that's been deprecated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant