Unable to run spyking-circus without error

[h21ak9]

I used these instructions to install SpyKING CIRCUS (Link). Here is a quick summary of my exact steps.

1. I downloaded and installed msmpisetup.exe and msmpisdk.msi from here (Link)
2. conda create -n circus python=3.6
3. conda activate circus
4. conda install -c conda-forge mpi4py
5. conda install -c conda-forge -c spyking-circus spyking-circus <-- This hanged indefinitely, so I ran pip instead
   pip install spyking-circus. It looks like at least one other person has had a similar problem with installing it with conda (see Issue Installation hanging #385).

After successfully installing spyking circus with pip, I ran it on my data set (after configuring the .params file). Here is the output in my terminal window. The error you can see at the bottom. I have tried multiple runs but nothing works. Any understanding of how to fix this would be much appreciated. Thank you.

(circus) PS D:\data\raw> spyking-circus .\File5_sc.bin

##################################################################
#####             Welcome to the SpyKING CIRCUS              #####
#####                        (1.1.0)                         #####
#####             Written by P.Yger and O.Marre              #####
##################################################################


File          : D:\data\raw\File5_sc.bin
Steps         : filtering, whitening, clustering, fitting, merging
Number of CPU : 12/24
Parallel HDF5 : False
Shared memory : True
Hostfile      : C:\Users\USERNAME\spyking-circus\circus.hosts

##################################################################


-------------------------  Informations  -------------------------
| Number of recorded channels : 256
| Number of analyzed channels : 210
| File format                 : RAW_BINARY
| Data type                   : int16
| Sampling rate               : 40000 Hz
| Duration of the recording   : 20 min 37 s 175 ms
| Width of the templates      : 3 ms
| Spatial radius considered   : 10 um
| Threshold crossing          : negative
------------------------------------------------------------------
-------------------------  Informations  -------------------------
| Using only 12 out of 24 local CPUs available (-c to change)
------------------------------------------------------------------
-------------------------  Informations  -------------------------
| Filtering has already been done
------------------------------------------------------------------
Analyzing data to get whitening matrices and thresholds...
Found 8.43865s to compute the whitening matrix...
Because of whitening, need to recompute the thresholds...
Searching spikes to construct the PCA basis...
100%|####################################|[00:24<00:00,  1.90s/it]
Found 168000 waveforms over 168000 requested
-------------------------  Informations  -------------------------
| A basis with 5 dimensions has been built
------------------------------------------------------------------
Searching isolated random spikes to sample amplitudes...
100%|####################################|[01:26<00:00,  6.65s/it]
Found 1166929 spikes over 1680000 requested
Estimating amplitudes distributions...
Smart Search of good isolated spikes for the clustering (1/3)...
100%|####################################|[02:08<00:00,  9.89s/it]
Found 940139 isolated spikes over 1680000 requested (1051948 rejected)
Computing density estimations...
Searching random spikes to refine the clustering (2/3)...
100%|####################################|[01:21<00:00,  6.27s/it]
Found 534392 spikes over 1680000 requested
Refining density estimations...
Searching random spikes to refine the clustering (3/3)...
100%|####################################|[01:07<00:00,  5.23s/it]
Found 203218 spikes over 1680000 requested
Refining density estimations...
Running density-based clustering...
100%|####################################|[00:22<00:00,  1.28s/it]
-------------------------  Informations  -------------------------
| Number of clusters found : 828
| Number of local merges   : 0 (method nd-bhatta, param 2)
------------------------------------------------------------------
Estimating the templates with the median-raw procedure ...
100%|####################################|[00:15<00:00,  1.14it/s]
Removing 10 strongly shifted or noisy/mixture templates...
Merging similar templates...
100%|####################################|[01:07<00:00,  3.74s/it]
Removing mixtures of templates...
100%|####################################|[01:02<00:00,  3.49s/it]
100%|####################################|[00:01<00:00, 35.54it/s]
-------------------------  Informations  -------------------------
| Number of global merges    : 130
| Number of mixtures removed : 1
------------------------------------------------------------------
Computing optimal amplitudes for the templates...
100%|####################################|[01:48<00:00,  1.86s/it]
100%|####################################|[00:07<00:00,  2.35it/s]
Pre-computing the overlaps of templates...
100%|####################################|[02:24<00:00,  8.05s/it]
Traceback (most recent call last):
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\USERNAME\miniconda3\envs\circus\Scripts\spyking-circus-subtask.exe\__main__.py", line 7, in <module>
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\site-packages\circus\scripts\subtask.py", line 54, in main
    circus.launch(task, filename, nb_cpu, nb_gpu, use_gpu)
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\site-packages\circus\__init__.py", line 23, in launch
    module.main(params, nb_cpu, nb_gpu, use_gpu)
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\site-packages\circus\fitting.py", line 204, in main
    c_overs, mpi_memory_2 = io.load_data_memshared(params, 'overlaps')
  File "C:\Users\USERNAME\miniconda3\envs\circus\lib\site-packages\circus\shared\files.py", line 569, in load_data_memshared
    win_indices = MPI.Win.Allocate_shared(factor * intsize, intsize, comm=sub_comm)
  File "mpi4py/MPI/Win.pyx", line 116, in mpi4py.MPI.Win.Allocate_shared
mpi4py.MPI.Exception: Other MPI error, error stack:
MPI_Win_allocate_shared(size=1579082160, disp_unit=-1216941376, info=0x4, comm=0x1c000000, baseptr=0x0000016784000000, win=0x00000167BE6E0B98) failed
CreateFileMapping failed, error 1450

[yger]

I would suspect such an error is due to memory issue, but not entirely sure. What is the amount of RAM you are using? Please note that while spyking-circus will be maintained, a new version 2 is developed within spikeinterface, and should be easier to install/use!

[h21ak9]

Thank you for your reply. I am happy to report that I have fixed the problem. The issue was that the installation instructions for spyking circus link to MPI v10.0. I looked up the release notes and saw that the latest version today is v10.1.3. I uninstalled the old version and installed the new version (both the .exe and .msi files). I am now able to run the analysis to completion. Also, you asked how much RAM I am using -- 32gb. Another observation: Even with the new version of MPI, spyking circus breaks if I use too many cores (with the -c option). For example, one of the computers available to me has 24 logical cores. If I try spyking-circus <filename> -c 24, it breaks after about 30 minutes on an MPI error. If I just do spyking-circus <filename>, it defaults to 12 cores and runs to completion. Similarly, another 64-gb RAM machine I have access to has 32 logical cores, of which 16 support hyperthreading (I suppose that corresponds to 8 physical cores). If I try running spyking-circus on this computer with the -c option set to anything greater than 8, spyking-circus breaks with an MPI error after about 30 minutes. I am excited to hear about version 2 and its integration with spike interface. Do you know when it will release?

…

On Fri, Jul 14, 2023 at 4:08 AM Pierre Yger ***@***.***> wrote: I would suspect such an error is due to memory issue, but not entirely sure. What is the amount of RAM you are using? Please note that while spyking-circus will be maintained, a new version 2 is developed within spikeinterface, and should be easier to install/use! — Reply to this email directly, view it on GitHub <#388 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7YBIFKO35G6MZWCYM3HHELXQEEABANCNFSM6AAAAAA2IBVHOA> . You are receiving this because you authored the thread.Message ID: ***@***.***>