You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am trying to run binned images on 5000 wavelength points and the code throws an error when writing out the image file.
The error is thrown because the maximum chunk size of HDF5 is 4GB. Is there a way to set a lower chunk size?
The full error traceback is as follows:
[main] exiting raytracing iteration
[image_write] writing out images
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000: H5Pdcpl.c line 2019 in H5Pset_chunk(): number of elements in chunk must be < 4GB
major: Invalid arguments to routine
minor: Out of range
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000: H5D.c line 145 in H5Dcreate2(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#001: H5Dint.c line 490 in H5D__create_named(): unable to create and link to dataset
major: Dataset
minor: Unable to initialize object
#002: H5L.c line 1695 in H5L_link_object(): unable to create new link to object
major: Links
minor: Unable to initialize object
#003: H5L.c line 1939 in H5L_create_real(): can't insert link
major: Symbol table
minor: Unable to insert object
#004: H5Gtraverse.c line 867 in H5G_traverse(): internal path traversal failed
major: Symbol table
minor: Object not found
#005: H5Gtraverse.c line 639 in H5G_traverse_real(): traversal operator failed
major: Symbol table
minor: Callback failed
#006: H5L.c line 1742 in H5L_link_cb(): unable to create object
major: Object header
minor: Unable to initialize object
#007: H5O.c line 3178 in H5O_obj_create(): unable to open object
major: Object header
minor: Can't open object
#008: H5Doh.c line 291 in H5O__dset_create(): unable to create dataset
major: Dataset
minor: Unable to initialize object
#009: H5Dint.c line 1202 in H5D__create(): filters can only be used with chunked layout
major: Dataset
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000: H5Dio.c line 226 in H5Dwrite(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.10.1) thread 0:
#000: H5D.c line 332 in H5Dclose(): not a dataset
major: Invalid arguments to routine
minor: Inappropriate type
-------------------------------------------------------------------------------
HDF5 returned an error in hdf5_write_k_h5t_ieee_f64le [2]
See above for traceback
The text was updated successfully, but these errors were encountered:
I have figured out a workaround. I want to run it by you to make sure that this makes sense.
As an example, consider I want peeled images for a galaxy with 4500 wavelength points between 0.3 to 15 microns.
I first divide it up into chunks of 500 wavelength points. With chunk 1 going from 0.3 to 0.45, chunk 2 going from 0.45 to 0.7, and so on and so forth. Then I call add_peeled_images() for each chunk by setting the wavelength range accordingly.
lam_lim = [0.3, 0.47, 0.7....,15]
for k in range(len(lam_lim)-1):
image = m_imaging.add_peeled_images(sed=False, image=True)
image.set_wavelength_range(500, lam_lim[k], lam_lim[k+1])
m_imaging.write(model.inputfile+'.image', overwrite=True)
m_imaging.run(model.outputfile+'.image', mpi=True, n_processes=par.n_MPI_processes, overwrite=True)
By doing so Hyperion saves the info for each chunk in a different group in the rtout file. Which I can then just access by passing the group number to the get_image() function. Does this approach seem reasonable? I want to make sure it's not breaking anything in the background that I should be worried about.
I am trying to run binned images on 5000 wavelength points and the code throws an error when writing out the image file.
The error is thrown because the maximum chunk size of HDF5 is 4GB. Is there a way to set a lower chunk size?
The full error traceback is as follows:
The text was updated successfully, but these errors were encountered: