Loading a signal lazily

[magnunor]

Currently HyperSpy supports Lazy loading, however I don't think there is a way of specifying a file to be loaded lazily in HyperSpyUI.

This would be useful for loading very large datasets, which normally would exceed the available memory on most computers.

[vidartf]

In older versions of HyperSpyUI, this is possible by changing the HyperSpy preference to use lazy loading by default.

BTW: What does HyperSpy do for the navigation plot of lazy signals? The signals in the UI are always plotted, so if the plotting loads the entire signal into memory to calculate the navigator, this will not really be useful here...

[vidartf]

Also, I made a quick implementation locally, and trying to open a signal of 500MB seems to take forever and a lot of CPU (and memory usage of 500MB+ at some point). Any idea why?

[francisco-dlp]

The navigator is computed lazily i.e. the chunks are retrieved from the file, processed and deleted from memory. Still, it will obviously take time and CPU. For large signals it's usually better not to plot the navigator or to use another image as navigator if possible.

In hyperspy/hyperspy#1544 the traitsui loading widget now has a lazy checkbox.

[vidartf]

I also noticed that the UI's contrast controls call max/min every time a figure is activated, which will of course be expensive for navigator plots (it uses plot.data_function() to inspect min/max values). I guess this can at least be optimized to only run if the contrast controls are visible (and when being made visible), but that is only a tweak.

So lazy load in the UI should maybe only use navigator sliders then? Or put up an (optional?) dialog box asking what to do for navigator (e.g. sliders, another already loaded signal, another file, or just the standard calculated image).

[francisco-dlp]

The second option (dialog box) sounds good! The first one can be very limiting in some cases.