Remove CNMF-E intermediate analysis folder

[bahanonu]

@zhoupc

Can you provide an option to either put the temporary folder created for the matfile in a user defined sub-folder or delete the folder after a successful run on CNMF-E? This folder can get large and take up unnecessary space after CNMF-E has been run.

e.g. in the below section of Sources2D, instead of [] for dir_output input, this should be a user modifiable parameter.
https://github.com/zhoupc/CNMF_E/blob/master/ca_source_extraction/%40Sources2D/Sources2D.m#L239

[zhoupc]

Hi, @bahanonu, thanks for pointing out the problem. there is an option for avoiding saving intermediate results.
you can set neuron.options.save_intermediate = false. I think by default this value is false.

Pengcheng

[bahanonu]

@zhoupc

Thanks. However, even with save_intermediate false (as it is by default https://github.com/zhoupc/CNMF_E/blob/master/ca_source_extraction/CNMFSetParms.m#L308), an intermediate folder is still created with the mat-file copy of the movie data and is not removed at the end. What does save_intermediate actually do?

Please re-open the issue until this is resolved.

[bahanonu]

Or do you have to run neuron.getReady_batch instead of neuron.getReady to prevent creation of an intermediate folder? e.g. will neuron.getReady always create a mat-file regardless of the save_intermediate flag?

[zhoupc]

there are following folders/files created for running CNMF-E:

1. data_x_x_x.mat
   this file is a copy of the video data. it saves data using block structure for fast accessing of patch video. This file uses a large space.

2. frames_x_y folder
   This folder saves the CNMF-E analysis for processing frames between x to y

3. within the folder frames_x_y, there are folders LOGS_DATE corresponding to different runs of CNMF-E. and within each folder, there are two files intermediate_results.mat and logs.txt to save intermediate results and the log.

even though you set neuron.save_intermediate = false, you can still have a file intermediate_results.mat, but it should be much smaller. It only saves the results of the initialization step.

This design is useful when you want to load the initialization results of an earlier run.

If you don't want to keep any intermediate results, even the data_x_x_x.mat file, probably a better solution is to create another function neuron.clean_workspace() to remove these extra files. what do you think? @bahanonu

[bahanonu]

@zhoupc

Awesome, thanks for the description, might be good to have that as it's own wiki page (e.g. CNMF-E files created) for future users.

A neuron.clean_workspace() method sounds like a good solution, so users can have control over clearing the workspace themselves and can do their own checks (in terms of CNMF-E output being close what they expect) before removing. If it could get ride of the entire ROOTFOLDER_source_extraction folder that is created, that would be helpful.

I ran with neuron.getReady_batch using a function built around https://github.com/zhoupc/CNMF_E/edit/master/demos/demo_batch_1p.m, but I am still getting the 1. data_x_x_x.mat file and frames_x_y folder.

[bahanonu]

Another solution is to allow the user access to dir_output in https://github.com/zhoupc/CNMF_E/blob/master/ca_source_extraction/endoscope/distribute_data.m or an intermediate_folder_path for anywhere else that an intermediate file is created so users can just put them all in a separate temporary folder or drive that can be cleared later.

[bahanonu]

@zhoupc

Can you update to allow the user to specify dir_output? There are cases in which I would like to manually enter the location else run into the 260 character path limit on Windows, which causes CNMF-E to crash.