Memory problem

[harshk95]

When I try to preprocess videos bigger than my JAVA heap memory (34Gb, max allowed) I get the error below and I was wondering how can I analyse videos with size bigger than that.
Thank you very much in advance for your help

pre-allocating single ones matrix: 800 1280 38801
@@@@@@@
Error using ones
Requested 800x1280x38801 (148.0GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.
Error in loadMovieList (line 472)
outputMovie = ones(preallocSize,imgClass);
Error in ciatah/modelPreprocessMovieFunction (line 729)
thisMovie = loadMovieList(movieList,'convertToDouble',0,'frameList',thisFrameList,'inputDatasetName',options.datasetName,'treatMoviesAsContinuous',options.turboreg.treatMoviesAsContinuousSwitch,'loadSpecificImgClass','single');
Error in ciatah/modelPreprocessMovie (line 37)
obj.modelPreprocessMovieFunction('folderListPath',folderListInfo,'fileFilterRegexp',options.fileFilterRegexp,'datasetName',options.datasetName,'frameList',[]);
Error in ciatah/runPipeline (line 202)
obj.(thisFxn{1});
Error in ciatah/display (line 6)
obj.runPipeline;
@@@@@@@
pre-allocating movies to display...
+++++++
no movie!

MIJ 1.3.9: Matlab to ImageJ Interface

More Info: http://bigwww.epfl.ch/sage/soft/mij/
Help: MIJ.help
JVM> 1.3.9
JVM> Version: 1.8.0_181
JVM> Total amount of memory: 387584 Kb
JVM> Amount of free memory: 244514 Kb
ImageJ> Version:1.53k
ImageJ> Memory:141MB of 29964MB (<1%)
ImageJ> Directory plugins: Not specified
ImageJ> Directory macros: Not specified
ImageJ> Directory luts: C:\Users\realtime\Documents\MATLAB\luts
ImageJ> Directory image: Not specified
ImageJ> Directory imagej: C:\Users\realtime\Documents\MATLAB
ImageJ> Directory startup: C:\Users\realtime\Documents\MATLAB
ImageJ> Directory home: C:\Users\realtime\

Status> ImageJ is running.

Warning: The DrawMode property will be removed in a future release. Use the
SortMethod property instead.

In ciapkg.overloaded.msgbox (line 280)
In ciatah/modelPreprocessMovieFunction>playOutputMovies (line 2432)
In ciatah/modelPreprocessMovieFunction (line 1090)
In ciatah/modelPreprocessMovie (line 37)
In ciatah/runPipeline (line 202)
In ciatah/display (line 6)
+++++++
no movie!
ImageJ instance ended cleanly

[bahanonu]

Been working on improvements to get around RAM-limitation based memory issues when using modelPreprocessMovie (our newer cell extraction methods do not have this issue). Other work arounds can be done by directly using the ciatah command line api (e.g. https://bahanonu.github.io/ciatah/api_example_pipeline/), but I can get back once modelPreprocessMovie is updated.

A couple quick questions:

• What type of movies are these (e.g. miniscope, 2p, etc.) and how much RAM does this computer have?
• If you type memory on the command window, what is the output?

There is the possibility that you can downsample the raw movies (https://bahanonu.github.io/ciatah/pipeline_detailed_downsample_raw/) to reduce file/matrix size, as often the higher pixels/micron resolution is not needed for cell detection and activity trace extraction.

[harshk95]

Hi!
Thanks for the prompt reply and looking forward to the update.

• So, we have 1p imaging videos from Inscopix that need to be concatenated and then preprocessed.
  The PC running this has 128 Gb of RAM.

• When we run the memory command we get

Maximum possible array: 139597 MB (1.464e+11 bytes) *
Memory available for all arrays: 139597 MB (1.464e+11 bytes) *
Memory used by MATLAB: 3567 MB (3.740e+09 bytes)
Physical Memory (RAM): 130749 MB (1.371e+11 bytes)

Limited by System Memory (physical + swap file) available.

We tried downsampling in the inscopix software and that decreased the size of the videos and made it work.
Nonetheless, once we manage to see the snippet of the MC movie, the software opens a dialog box to continue to the next movie (but we are currently working on one only) and if we close that dialog box the MC movie ends up not being saved in the processing folder. How can we sort this out?

Thanks again for your help!!

[bahanonu]

Great to hear the downsampling works! For Inscopix miniscope movies often 2x or 4x downsampling in each dimension usually works fine.

Did a movie play in ImageJ or MATLAB? Can you send me the _preprocess.log file in the processing_info sub-folder that is in the same folder as the raw movie being processed? Should start with a date-time string for when that processing run started. That'll help me debug why you aren't seeing the output movie.

[harshk95]

20210910_113823_preprocess.log

Here the file! A snippet played in ImageJ, yes.
Thanks a lot for your help again

[bahanonu]

Looks like a turboreg error occurred during 2nd round of motion correction. Do you see 0000_00_00_p000_m1_NULL000_medFlt_detrend_1.h5 movie in the M1_4 folder?

Also, can you run with the below RegisType affine settings and let me know if it finishes fine? In general, unless you suspect significant warping in your Inscopix recordings, I would recommend against running with RegisType set to any projective options, as that can lead to sub-optimal results.

Else, if you can run with a single turboreg iteration and send me a copy of the movie (only a couple frames, e.g. see below), that can help me diagnose what happened with projective. e.g. the _snippet.h5 movie as below:

thisPath='full_path_to_output_movie';
[thisDir,filename1,ext1]=fileparts(thisPath);
saveMatrixToFile(loadMovieList(thisPath,'frameList',1:500),[thisDir filesep filename1 '_snippet' ext1]);

[harshk95]

We managed in the end to make it work: 5 iterations with a 4x downsampled video (size 5Gb approx.) managed to be preprocessed on a 512GB RAM computer and the results of motion correction are really great.

After preprocessing we need to use extract as an extraction method. For our application, people in our lab have worked in conjuction with the extract team to finetune the config and get results. Nonetheless, this finetuning was done on non-downsampled videos. So we went back and tried to run ciatah preprocessing on non downsampled files (approx. 80Gb) on a 512Gb RAM computer but this led to the out memory error again.

Is there any way we can improve RAM efficiency?

Very sorry for the trouble and thanks a lot for your help.

[bahanonu]

Thanks for the update on this and glad to hear the downsampled movies worked well for you-all.

re: non-downsampled movies. It is possible to improve the RAM efficiency by running ciatah pre-processing via the command-line (e.g. https://bahanonu.github.io/ciatah/api_example_pipeline/) so that certain steps (e.g. motion correction) are only done on chunks of the movie at a time (this is what we do for some of our cell extraction algorithms or other steps).

Could you let me know what steps you are running for ciatah pre-processing? I can get back with a script that can reduce RAM usage as still working to update CIAtah GUI for other users to handle this issue.

[harshk95]

Hey,
thanks a lot, that would be very helpful!

The preprocess steps we currently wish to use are:

• median filtering
• motion correction with spatial filters
• adjusting for cropped pixels

Thanks again!