BUG: Tetrode data is garbled in Kilosort4 but not Kilosort3

[afmcelroy]

Describe the issue:

My lab has been using Kilosort3 for spike sorting tetrode data for some time, and recently advanced to Kilosort4. Unfortunately, data that was able to be sorted normally in Kilosort3 doesn't seem to have worked in Kilosort4.
Here is a screenshot of the Kilosort4 data from phy. Note the strange shape of the waveforms. Every cluster that can be viewed looks like this:

Trying to view certain clusters also yields an out-of-bounds error:

Here is the same data (same animal, same recording session) sorted in Kilosort3, looking completely normal:

This may be some error with setting up Kilosort4, but I'm not sure what. It could also be an error with phy. Attached is a zipped folder containing channel_map.mat for these data and the python script (main5-1.py) I use to launch Kilosort4. 'n_chan_bin' on line 18 is 48 rather than 64 to exclude broken channels - this was a bug fix that I tried and which didn't work. Another possible issue is 'batch_size' on line 21, which was revised down from its default value because our GPU was running out of memory - our copy of Kilosort3 runs on the department computing cluster, which is less limited, so this may make a difference. Let me know what other files could be needed for debugging.
kilosort4 debugging.zip

Reproduce the bug:

No response

Error message:

The text of the out-of-bounds error from phy2:
13:02:31.019 [E] __init__:62          An error has occurred (IndexError): index 1018798 is out of bounds for axis 0 with size 1018768
Traceback (most recent call last):
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/widgets.py", line 269, in emitJS
    self._debouncer.submit(emit, *args)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/qt.py", line 620, in submit
    self.trigger()
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/qt.py", line 635, in trigger
    f(*args, **kwargs)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phylib/utils/event.py", line 141, in emit
    res.append(f(sender, *args, **kwargs))
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 820, in _clusters_selected
    self.similarity_view.reset(cluster_ids)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 379, in reset
    similar = emit('request_similar_clusters', self, cluster_ids[-1])
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phylib/utils/event.py", line 141, in emit
    res.append(f(sender, *args, **kwargs))
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 737, in _get_similar_clusters
    sim = self.similarity(cluster_id) or []
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/apps/template/gui.py", line 169, in template_similarity
    temp_i = np.nonzero(self.get_template_counts(cluster_id))[0]
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/utils/context.py", line 154, in memcached
    out = f(*args, **kwargs)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/apps/base.py", line 486, in get_template_counts
    st = self.model.spike_templates[spike_ids]
IndexError: index 1018798 is out of bounds for axis 0 with size 1018768

Version information:

OS: Linux Mint 20.3 Cinnamon version 5.2.7
Python: 3.9.18 (in the conda env for kilosort)
Kilosort: 4.0.2
Phy: 2.0b5

Context for the issue:

No response

Experiment information:

64-channel tetrode recording from primary motor cortex of mouse, using Whitematter wireless data logging system

[jacobpennington]

Hello,

Please try sorting this again after installing the latest version of Kilosort (v4.0.4), with the default batch size and dminx = 10. The memory issue is likely related to some bugs that have now been fixed. If you still encounter issues, please let us know.

[afmcelroy]

This seems to have worked, Kilosort4 produced well-sorted units for the original and a second recording after updating, using the default batch size and dminx = 10 - no memory issues or phy errors. Is there any more information you can give me about dminx as a parameter? My understanding is it represents the space in between contacts - is 10 a good value for tetrodes specifically or a general rule of thumb value? I know the default is based on the spacing of Neuropixels contacts.

[afmcelroy]

I did also get this UserWarning: NaNs and/or zeroes present in weights for spikedetect.run, may need to adjust 'min_template_size' and/or 'dminx' for best results. It seems to have not affected the sorting itself but I wanted to note it in case it is a bug or you can give any further context on it.

[jacobpennington]

@afmcelroy The UserWarning you mentioned is not a problem as long as you're getting good sorting results. It's just there as a debugging tool in case other problems pop up (which typically cause the sorting to crash).

dmin and dminx control how the universal templates used in the spike detection step are positioned, and should correspond to the vertical and horizontal spacing, respectively, of contacts on the physical probe. I suggested dminx = 10 because the probe file you shared had contacts spaced 10um apart, more-or-less.