multiplane_parallel in an HPC (Slum)

[nerf-common]

Hi,

I created this issue in another repo: cortex-lab/Suite2P#186. If you can provide help on the following, it would be appreciated.

"I provide support to researchers in a neuroscience institute. One of our users is trying to run suite2p using the multiplane_parallel option. We understand that we have to pass the host, username and pw.

I have some questions about it:

1. does this multiplane_parallel option speed up the computation?, if so, does this profit then of the resources in the local-computer PLUS an external server?

2. how would it work in a system as Slurm?
   I mean, in Slurm we can give several cpu cores and number of tasks, if the purpose of multiplane_parallel is to provide more resources, that should also be achieved using Slurm I believe."

Thanks,
Giuliano.

[generalciao]

Hi Giuliano,
I'm not a suite2p developer. I don't use multiplane_parallel (but see server.py for the implementation). I use SLURM and it works fine.

Suite2p will expect two settings files (db.npy and ops.npy) passed in as arguments. In my case, I keep a single standard ops.npy (saved from the GUI with my typical settings) that is used unchanged by all jobs, and I dynamically create a new db.npy for each SLURM run. To do that, I use srun echo -e "..." | python where the ... comprise a few lines of Python code to create and save a db.npy numpy array containing the ~3 settings that I need. So far those haven't really changed for me across jobs, so perhaps I could save db.npy once and re-use it, like I do for ops.npy? To be honest, I don't recall why I create db.npy dynamically for each job - probably in case I want to analyze multiple input file formats in the future.

Some additional suite2p configuration parameters may change from job to job (for example, frame rate, or the segmentation mode). These can be passed to the srun command as a space-separated list of double-hyphen inputs, and as I recall they will override parameters of the same name already defined in ops.npy. For example: --save_path0 "./" --fs 15

On a cluster it can be worth doing some benchmarking to figure out which storage options to use as the fast_disk, and how much RAM is needed (for me, the acquisition file size plus some fixed overhead, maybe 10-20GB, I forget what I measured).

Hope this helps.

[carsen-stringer]

you can also use multiplane_parallel (see code in run_s2p.py), which will run each plane as a single job. if you have a large cluster, then this will make your job as fast as running a single plane.

if you don't have multiplane or multi-ROI (mesoscope) recordings, then this option will not speed anything up