You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently when in parallel mode, if a user specifies a filename for the input file we load the data with astropy.io.fits, which even if memory mapped is then written out to a new memory-mapped file for the purposes of the parallel computation (to avoid copying the array in memory to all the processes).
We should find a way, whenever possible, to avoid writing out a new memmap if the original data is backed by a file on disk.
I'm not sure if there is a way to do this if HDU objects are passed in as hdu.data is a regular Numpy array and the details of the memmap are hidden in the buffer. However if a filename is passed, we should be able to set up a memmap ourselves using the BITPIX and NAXIS? in the header for the HDU. If parallel mode is specified, we could then warn if an HDU or HDUList is passed that this is not optimal and that a filename should be passed instead.
We should also make sure we support passing in np.memmap objects and properly handle these (again avoiding any re-writing out of the arrays).
The text was updated successfully, but these errors were encountered:
Currently when in parallel mode, if a user specifies a filename for the input file we load the data with astropy.io.fits, which even if memory mapped is then written out to a new memory-mapped file for the purposes of the parallel computation (to avoid copying the array in memory to all the processes).
We should find a way, whenever possible, to avoid writing out a new memmap if the original data is backed by a file on disk.
I'm not sure if there is a way to do this if HDU objects are passed in as
hdu.data
is a regular Numpy array and the details of the memmap are hidden in the buffer. However if a filename is passed, we should be able to set up a memmap ourselves using the BITPIX and NAXIS? in the header for the HDU. If parallel mode is specified, we could then warn if an HDU or HDUList is passed that this is not optimal and that a filename should be passed instead.We should also make sure we support passing in
np.memmap
objects and properly handle these (again avoiding any re-writing out of the arrays).The text was updated successfully, but these errors were encountered: