LiveViewer: Out of Synchronisation with Most Recent Image - Difference Increases with Dataset Size

[JackEAllen]

Summary

"We have been recording radiography images at an exposure time of 5s/image and find that there is a considerable lag for the Live Viewer images to get updated. It took ~3 minutes to display the new image this morning and there was a lag between the image that was displayed and the image being currently collected (the displayed image was more than 100 images behind the one currently being collected).
Once the user restarted Mantid, the lag was less, but still around 7 to 8 images behind."

"To add context, the current experiment produces images every 5 seconds, so 12 hours of scan will produce 8000+ images. I think we're targeting the parent RB folders so probably the live viewer struggled to scan through that many images to display the latest one."

Steps To Reproduce

• Create a dataset of ~8000 tiff files
• Create an output directory to copy images too
• Open Mantid Imaging Live Viewer and select output directory
• Run live viewer simulation script to copy files from fake dataset to output directory
• Watch images to see if they fall out of sync

Expected Behaviour

Live viewer should always show the newest file written within watched directory.

Current Behaviour

Live viewer fall out of sync with newest image written to directory after so many images within a large dataset.

Context

Discovered by IMAT when using live viewer to watch 12 hour of scan will produce 8000+ images. Acquisition time was ~5 seconds.

Failure Logs

Screenshot(s)

[JackEAllen]

I have managed to replicate with a dataset of 8000 tiffs. The live viewer fell out of sync at image 2250 where the most recent image written was 2278 using the simulate live viewer utility script.

[JackEAllen]

Can't say if it's the same issue with certainty, but second time round, running the same test, MI became unresponsive after falling out of sync when selecting the live viewer window.

2024-05-15 09:01:45,608 [mantidimaging.core.parallel.manager:L40] INFO: Creating process pool with 8 processes
2024-05-15 09:02:27,296 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
2024-05-15 09:02:45,515 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
2024-05-15 09:03:29,124 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
QFileSystemWatcher::removePaths: list is empty
QFileSystemWatcher::addPaths: list is empty
QFileSystemWatcher::removePaths: list is empty
QFileSystemWatcher::addPaths: list is empty
QFileSystemWatcher::removePaths: list is empty

When clearing the watched directory of files, the window becomes responsive again

[JackEAllen]

Can't say if it's the same issue with certainty, but second time round, running the same test, MI became unresponsive after falling out of sync when selecting the live viewer window.

2024-05-15 09:01:45,608 [mantidimaging.core.parallel.manager:L40] INFO: Creating process pool with 8 processes
2024-05-15 09:02:27,296 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
2024-05-15 09:02:45,515 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
2024-05-15 09:03:29,124 [mantidimaging.gui.mvp_base.view:L35] DEBUG: UI window closed
QFileSystemWatcher::removePaths: list is empty
QFileSystemWatcher::addPaths: list is empty
QFileSystemWatcher::removePaths: list is empty
QFileSystemWatcher::addPaths: list is empty
QFileSystemWatcher::removePaths: list is empty
When clearing the watched directory of files, the window becomes responsive again

So from testing a third time, i noticed that the python process switch's between responsive and unresponsive when slow down occurs so we are definitely reaching an upper threshhold somewhere. When the switching from unresponsive to responsive and back, this is when the live viewer updates to a newer image which has since become not the newest image written.

[JackEAllen]

When python becomes unresponsive, the live viewer still acknowledges new file changes coming in and sends calls to qt to update the image view with the newest file, despite the image not updating.