Spike sort in Docker containers with spiking-circus

[benkiisma]

Hello, I am trying to run the docker andI have an error when running the line :

sorting_SC = ss.run_spykingcircus(recording_loaded, output_folder="results_SC",
                                  docker_image="spikeinterface/spyking-circus-base:1.0.7", 
                                  verbose=True)

I installed docker and had docker python SDK installed too.
The error I had is the following:

Starting container
Installing spikeinterface==0.93.0 in spikeinterface/spyking-circus-base:1.0.7
Running spykingcircus sorter inside spikeinterface/spyking-circus-base:1.0.7
Stopping container

/Users/ismail/opt/anaconda3/envs/si090/lib/python3.8/pathlib.py:693: ResourceWarning: unclosed <socket.socket fd=131, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/ismail/.docker/run/docker.sock>
  self = object.__new__(cls)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/ismail/opt/anaconda3/envs/si090/lib/python3.8/pathlib.py:693: ResourceWarning: unclosed <socket.socket fd=133, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/ismail/.docker/run/docker.sock>
  self = object.__new__(cls)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/ismail/opt/anaconda3/envs/si090/lib/python3.8/pathlib.py:693: ResourceWarning: unclosed <socket.socket fd=135, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/ismail/.docker/run/docker.sock>
  self = object.__new__(cls)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/ismail/opt/anaconda3/envs/si090/lib/python3.8/pathlib.py:693: ResourceWarning: unclosed <socket.socket fd=137, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/ismail/.docker/run/docker.sock>
  self = object.__new__(cls)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/ismail/opt/anaconda3/envs/si090/lib/python3.8/pathlib.py:693: ResourceWarning: unclosed <socket.socket fd=139, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, raddr=/Users/ismail/.docker/run/docker.sock>
  self = object.__new__(cls)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Input In [93], in <cell line: 1>()
----> 1 sorting_SC = ss.run_spykingcircus(recording_loaded, output_folder="results_SC",
      2                                   docker_image="spikeinterface/spyking-circus-base:1.0.7", 
      3                                   verbose=True)

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py:461, in run_spykingcircus(*args, **kwargs)
    460 def run_spykingcircus(*args, **kwargs):
--> 461     return run_sorter('spykingcircus', *args, **kwargs)

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py:56, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params)
     49 """
     50 Generic function to run a sorter via function approach.
     51 
     52 >>> sorting = run_sorter('tridesclous', recording)
     53 """ + _common_param_doc
     55 if docker_image is not None:
---> 56     return run_sorter_container(sorter_name, recording, 'docker', docker_image,
     57                                 output_folder=output_folder,
     58                                 remove_existing_folder=remove_existing_folder,
     59                                 delete_output_folder=delete_output_folder, verbose=verbose,
     60                                 raise_error=raise_error, with_output=with_output, **sorter_params)
     61 if singularity_image is not None:
     62     return run_sorter_container(sorter_name, recording, 'singularity', singularity_image,
     63                                 output_folder=output_folder,
     64                                 remove_existing_folder=remove_existing_folder,
     65                                 delete_output_folder=delete_output_folder, verbose=verbose,
     66                                 raise_error=raise_error, with_output=with_output, **sorter_params)

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/sorters/runsorter.py:355, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    353 else:
    354     if with_output:
--> 355         sorting = SorterClass.get_result_from_folder(output_folder)
    357 if delete_output_folder:
    358     shutil.rmtree(output_folder)

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/sorters/basesorter.py:256, in BaseSorter.get_result_from_folder(cls, output_folder)
    252 if bool(log['error']):
    253     raise SpikeSortingError(
    254         "Spike sorting failed. You can inspect the runtime trace in spikeinterface_log.json")
--> 256 sorting = cls._get_result_from_folder(output_folder)
    258 recording = load_extractor(output_folder / 'spikeinterface_recording.json')
    259 if recording is not None:
    260     # can be None when not dumpable

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/sorters/spyking_circus/spyking_circus.py:169, in SpykingcircusSorter._get_result_from_folder(cls, output_folder)
    167 @classmethod
    168 def _get_result_from_folder(cls, output_folder):
--> 169     sorting = SpykingCircusSortingExtractor(folder_path=Path(output_folder) / 'recording')
    170     return sorting

File ~/opt/anaconda3/envs/si090/lib/python3.8/site-packages/spikeinterface/extractors/spykingcircusextractors.py:53, in SpykingCircusSortingExtractor.__init__(self, folder_path)
     51         break
     52 if results is None:
---> 53     raise Exception(spykingcircus_folder, " is not a spyking circus folder")
     55 # load params
     56 sample_rate = None

Exception: (PosixPath('/Users/ismail/Desktop/PDS/Official_Tutorial_SI_0.90_Aug2021/results_SC/recording'), ' is not a spyking circus folder')

The content of the log folder is :

---------------------  Informations  -------------------------
| Number of recorded channels : 32
| Number of analyzed channels : 32
| File format                 : NUMPY
| Data type                   : float32
| Sampling rate               : 30000 Hz
| Duration of the recording   : 2 min 27 s 618 ms
| Width of the templates      : 3 ms
| Spatial radius considered   : 100 um
| Threshold crossing          : negative
------------------------------------------------------------------
-------------------------  Informations  -------------------------
| Using only 2 out of 4 local CPUs available (-c to change)
------------------------------------------------------------------
�[37mFiltering with a Butterworth filter (order 3) in [300, 14250] Hz

  0%|                                            |[00:00<?, ?it/s]
 33%|████████████                        |[00:48<01:36, 48.08s/it]
 67%|████████████████████████            |[01:37<00:48, 48.65s/it]
100%|████████████████████████████████████|[02:09<00:00, 41.12s/it]
100%|████████████████████████████████████|[02:09<00:00, 43.10s/it]
�[37mAnalyzing data to get whitening matrices and thresholds...
�[37mFound 30s to compute the whitening matrix...
�[37mBecause of whitening, need to recompute the thresholds...
�[37mSearching spikes to construct the PCA basis...

  0%|                                            |[00:00<?, ?it/s]
100%|████████████████████████████████████|[01:29<00:00, 89.72s/it]
100%|████████████████████████████████████|[01:29<00:00, 89.72s/it]
�[37mFound 11648 waveforms over 25600 requested
�[33m-------------------------  Informations  -------------------------
�[33m| A basis with 5 dimensions has been built
�[33m------------------------------------------------------------------
�[37m
�[37mSearching isolated random spikes to sample amplitudes...

  0%|                                            |[00:00<?, ?it/s]
100%|███████████████████████████████████|[02:09<00:00, 129.20s/it]
100%|███████████████████████████████████|[02:09<00:00, 129.20s/it]
�[37mFound 14075 spikes over 256000 requested
�[37mEstimating amplitudes distributions...
�[37mSmart Search of good isolated spikes for the clustering (1/3)...

  0%|                                            |[00:00<?, ?it/s]
100%|███████████████████████████████████|[02:16<00:00, 136.79s/it]
100%|███████████████████████████████████|[02:16<00:00, 136.79s/it]
�[37mFound 14075 isolated spikes over 256000 requested (0 rejected)
�[37mComputing density estimations...
�[37mSearching random spikes to refine the clustering (2/3)...

  0%|                                            |[00:00<?, ?it/s]
100%|████████████████████████████████████|[00:48<00:00, 48.70s/it]
100%|████████████████████████████████████|[00:48<00:00, 48.71s/it]
�[37mFound 17 spikes over 256000 requested
�[37mRefining density estimations...
�[37mSearching random spikes to refine the clustering (3/3)...

  0%|                                            |[00:00<?, ?it/s]
100%|████████████████████████████████████|[00:50<00:00, 50.20s/it]
100%|████████████████████████████████████|[00:50<00:00, 50.20s/it]
�[37mFound 0 spikes over 256000 requested
�[33m-------------------------  Informations  -------------------------
�[33m| No more spikes in the recording, stop searching
�[33m------------------------------------------------------------------
�[37m
�[37mRefining density estimations...
�[37mRunning density-based clustering...

  0%|                                            |[00:00<?, ?it/s]
  6%|██▎                                 |[00:00<00:03,  4.07it/s]
 12%|████▌                               |[00:00<00:02,  6.19it/s]
 19%|██████▊                             |[00:02<00:13,  1.02s/it]
 25%|█████████                           |[00:02<00:07,  1.50it/s]
 31%|███████████▎                        |[00:09<00:33,  3.08s/it]
 38%|█████████████▌                      |[00:10<00:20,  2.10s/it]
 44%|███████████████▊                    |[00:10<00:13,  1.45s/it]
 50%|██████████████████                  |[00:10<00:08,  1.03s/it]
 56%|████████████████████▎               |[00:10<00:05,  1.30it/s]
 69%|████████████████████████▊           |[00:10<00:02,  2.22it/s]
 75%|███████████████████████████         |[00:10<00:01,  2.56it/s]
 81%|█████████████████████████████▎      |[00:11<00:01,  1.83it/s]
 88%|███████████████████████████████▌    |[00:12<00:01,  1.93it/s]
 94%|█████████████████████████████████▊  |[00:12<00:00,  2.39it/s]
100%|████████████████████████████████████|[00:12<00:00,  1.27it/s]
�[33m-------------------------  Informations  -------------------------
�[33m| Number of clusters found : 29
�[33m| Number of local merges   : 0 (method nd-bhatta, param 2)
�[33m| Not enough spikes gathered: -put safety_space=False?
�[33m|                             -remove smart_search?
�[33m------------------------------------------------------------------
�[37m
�[37mEstimating the templates with the median-raw procedure ...

  0%|                                            |[00:00<?, ?it/s]
  6%|██▎                                 |[00:01<00:24,  1.64s/it]
 19%|██████▊                             |[00:05<00:23,  1.77s/it]
 25%|█████████                           |[00:06<00:20,  1.75s/it]
 31%|███████████▎                        |[00:08<00:19,  1.78s/it]
 38%|█████████████▌                      |[00:10<00:17,  1.75s/it]
 56%|████████████████████▎               |[00:12<00:07,  1.07s/it]
 69%|████████████████████████▊           |[00:13<00:05,  1.00s/it]
 75%|███████████████████████████         |[00:18<00:06,  1.68s/it]
 81%|█████████████████████████████▎      |[00:23<00:07,  2.49s/it]
 88%|███████████████████████████████▌    |[00:25<00:04,  2.30s/it]
 94%|█████████████████████████████████▊  |[00:27<00:02,  2.14s/it]
100%|████████████████████████████████████|[00:27<00:00,  1.69s/it]
�[37mMerging similar templates...

  0%|                                            |[00:00<?, ?it/s]
  6%|██▎                                 |[00:00<00:11,  1.32it/s]
 19%|██████▊                             |[00:01<00:05,  2.29it/s]
 25%|█████████                           |[00:02<00:05,  2.00it/s]
 31%|███████████▎                        |[00:02<00:06,  1.80it/s]
 38%|█████████████▌                      |[00:03<00:05,  1.73it/s]
 56%|████████████████████▎               |[00:03<00:02,  2.76it/s]
 69%|████████████████████████▊           |[00:04<00:01,  2.93it/s]
 75%|███████████████████████████         |[00:05<00:01,  2.48it/s]
 81%|█████████████████████████████▎      |[00:05<00:01,  2.18it/s]
 88%|███████████████████████████████▌    |[00:06<00:00,  2.00it/s]
 94%|█████████████████████████████████▊  |[00:07<00:00,  1.89it/s]
100%|████████████████████████████████████|[00:07<00:00,  2.26it/s]
�[37mRemoving mixtures of templates...

  0%|                                            |[00:00<?, ?it/s]
  6%|██▎                                 |[00:00<00:10,  1.49it/s]
 19%|██████▊                             |[00:01<00:05,  2.42it/s]
 25%|█████████                           |[00:01<00:05,  2.08it/s]
 31%|███████████▎                        |[00:02<00:05,  1.87it/s]
 38%|█████████████▌                      |[00:03<00:05,  1.79it/s]
 56%|████████████████████▎               |[00:03<00:02,  2.83it/s]
 69%|████████████████████████▊           |[00:04<00:01,  2.97it/s]
 75%|███████████████████████████         |[00:05<00:01,  2.48it/s]
 81%|█████████████████████████████▎      |[00:05<00:01,  2.12it/s]
 88%|███████████████████████████████▌    |[00:06<00:01,  1.93it/s]
 94%|█████████████████████████████████▊  |[00:07<00:00,  1.84it/s]
100%|████████████████████████████████████|[00:07<00:00,  2.27it/s]

  0%|                                            |[00:00<?, ?it/s]
 23%|████████▎                           |[00:00<00:00, 24.23it/s]
 46%|████████████████▌                   |[00:00<00:00, 25.38it/s]
 69%|████████████████████████▉           |[00:00<00:00, 24.84it/s]
 92%|█████████████████████████████████▏  |[00:00<00:00, 25.16it/s]
100%|████████████████████████████████████|[00:00<00:00, 25.06it/s]
�[33m-------------------------  Informations  -------------------------
�[33m| Number of global merges    : 3
�[33m| Number of mixtures removed : 1
�[33m------------------------------------------------------------------
�[37m
�[37mComputing optimal amplitudes for the templates...

  0%|                                            |[00:00<?, ?it/s]
  8%|██▊                                 |[00:08<01:45,  8.77s/it]
===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 139 RUNNING AT 94486e4b2ebb
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9)
This typically refers to a problem with your application.
Please see the FAQ page for debugging suggestions
----------------------------  Error  -----------------------------
| Step "clustering" failed for reason Command '['mpiexec', '-np', '2', 'spyking-circus-subtask', 'clustering', '/Users/ismail/Desktop/PDS/Official_Tutorial_SI_0.90_Aug2021/results_SC/recording.npy', '2', '0', 'False', 'False']' returned non-zero exit status 9.!

Thank you very much for your help.

[yger]

Strange. Looks like the code is crashing, but this is rather small and not that challenging dataset, at a first glimpse. Could you send it to me?

[benkiisma]

Of course ! You can find the recording I used here : https://drive.google.com/file/d/1PzghSGv9EgcPU2uczahPXh2sozb8psgb/view?usp=sharing

[Aryo-Zare]

I posted a relatively similar issue in the spike-interface page :
SpikeInterface/spikeinterface#1034

[Aryo-Zare]

also check SpikeInterface/spikeinterface#895