the exact meaning of different fields in remora dataset prepare output

[xieyy46]

Hi remora team,
Cheer for your good work! I want know what the data of different fields (shown in below figure) in remora dataset prepare output exactly mean?

[marcus1487]

This looks like the fields from a RemoraDataset version <3.0. I will look into adding more documentation on the exact meaning of the relevant fields in the updated version of the dataset. The definition of the majority of these fields can be found in the help section for the remora dataset prepare -h function to get you started though.

[xieyy46]

Hi marcus,
Thank you for your reply! I tried to used the latest version of remora and got the following output when running remora dataset prepare. However, I want know the correct way to load these npy file. I have tried numpy.load("xx.npy") and numpy.load("xx.pny", allow_pickle=True), but both got into error!

[marcus1487]

These files are memory mapped numpy arrays to allow for more efficient disk access. You can load them with the np.memmap function. You can see in the code the command to read in the files here. I would highly recommend loading the data in via the remora API as opposed to rolling your own readers for this data though. If you have questions about the data or require a specific access pattern that the API does not provide please raise an issue with these specific requests..