Memory problem of Kilosort

[jnjnnjzch]

Describe the issue:

Hello,

I am currently using Kilosort 4 to analyze my data, which was recorded using a 1024-channel probe. However, I have encountered an abnormal memory requirement issue. It appears that Kilosort 4 requires over 1TB of memory.

To investigate the problem, I examined the code and noticed that the automatically calculated variable 'dminx' seems to be set too small. This has resulted in the creation of an excessive number of virtual electrodes.

To address the memory requirement problem, I manually adjusted the value of 'dminx' to a higher value, and it seems to have resolved the issue.

My main concern is how this change in the 'dminx' value might affect the spike sorting procedure. I have noticed that the excessive virtual electrodes could potentially impact the drift correction performed by Kilosort. Are there any other potential consequences or impacts that I should be aware of?

I would greatly appreciate your insights and advice regarding this matter.

Thank you in advance!

[jacobpennington]

Hello,

I'm assuming those 1024 channels are split over multiple shanks? This is a known issue, please see documentation here for a suggested workaround until we implement multi-shank support. You can also check this page for more information about updating dminx. Note that it refers to the most recent version of the code, which you can use by installing from source after pulling the latest changes, in which dminx defaults to 32 instead of attempting to determine it automatically.

If your probe is configured some other way (like a 2D MEA) this could also be related to #644.

[jnjnnjzch]

Hi Jacob,

Thank you for your response. I appreciate your willingness to help. I will proceed to verify the sorting result using dminx set to 32.

In our case, we have some concerns regarding the configuration of our probe. All 1024 channels are located on a single shank, and the signals are recorded simultaneously. While our probe is not a 2D MEA as mentioned in issue #644, it is a checkerboard needle probe. We believe that the behavior we are observing could be similar to the one described in that issue.

The automatic determination of the dminx algorithm seems to encounter difficulties when dealing with the checkerboard configuration. To illustrate this, here's a minimal example:

import numpy as np

ops = {}
ops['xc'] = np.concatenate([np.array([0, 50]) for i in range(5)])
print('xc is ', ops['xc'])
# Output: xc is: [ 0 50  0 50  0 50  0 50  0 50]

ops['yc'] = np.arange(10, 10+len(ops['xc'])*10, 10)
print('yc is ', ops['yc'])
# Output: yc is: [ 10  20  30  40  50  60  70  80  90 100]

Based on the code snippet you provided in the spikedetect.py file, specifically the template_centers(ops) function, the issue becomes evident:

yunq = np.unique(ops['yc'])
mxc = np.NaN * np.ones(len(yunq))
for j in range(len(yunq)):
    xc = ops['xc'][ops['yc']==yunq[j]]
    if len(xc)>1:
        mxc[j] = np.median(np.diff(np.sort(xc)))
    else:
        mxc[j] = 0
dminx = np.nanmedian(mxc)
dminx = np.maximum(1, dminx)

Since there is no other electrode on the same row in the checkerboard configuration, the length of xc is always 1. Consequently, mxc and dminx are always set to 0 as well. After the line dminx = np.maximum(1, dminx), dminx ends up being set to 1. This behavior is not what we expect and may result in dminx being too small.

Given these observations, I would appreciate your suggestions regarding the choice of dminx for a checkerboard probe configuration. Thank you again for your assistance!

[jacobpennington]

Hello, Yes, dminx for checkerboard patterns is also a known issue and has already been fixed (by getting rid of the automatic dminx behavior, it just defaults to 32 now). That will probably work okay for any single shank probe, but it you want to fine-tune it you should set it to the median lateral distance between contacts.

…

On Sat, Mar 30, 2024, 5:08 AM jnjnnj ***@***.***> wrote: Hi Jacob, Thank you for your response. I appreciate your willingness to help. I will proceed to verify the sorting result using dminx set to 32. In our case, we have some concerns regarding the configuration of our probe. All 1024 channels are located on a single shank, and the signals are recorded simultaneously. While our probe is not a 2D MEA as mentioned in issue #644 <#644>, it is a checkerboard needle probe. We believe that the behavior we are observing could be similar to the one described in that issue. The automatic determination of the dminx algorithm seems to encounter difficulties when dealing with the checkerboard configuration. To illustrate this, here's a minimal example: import numpy as np ops = {}ops['xc'] = np.concatenate([np.array([0, 50]) for i in range(5)])print('xc is ', ops['xc'])# Output: xc is: [ 0 50 0 50 0 50 0 50 0 50] ops['yc'] = np.arange(10, 10+len(ops['xc'])*10, 10)print('yc is ', ops['yc'])# Output: yc is: [ 10 20 30 40 50 60 70 80 90 100] Based on the code snippet you provided in the spikedetect.py file, specifically the template_centers(ops) function, the issue becomes evident: yunq = np.unique(ops['yc'])mxc = np.NaN * np.ones(len(yunq))for j in range(len(yunq)): xc = ops['xc'][ops['yc']==yunq[j]] if len(xc)>1: mxc[j] = np.median(np.diff(np.sort(xc))) else: mxc[j] = 0dminx = np.nanmedian(mxc)dminx = np.maximum(1, dminx) Since there is no other electrode on the same row in the checkerboard configuration, the length of mxc is always 0. Consequently, dminx is always set to 0 as well. After the line dminx = np.maximum(1, dminx), dminx ends up being set to 1. This behavior is not what we expect and may result in dminx being too small. Given these observations, I would appreciate your suggestions regarding the choice of dminx for a checkerboard probe configuration. Thank you again for your assistance! — Reply to this email directly, view it on GitHub <#647 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIQ6WYAVEKITU4NLNU6BSQDY2Z6KJAVCNFSM6AAAAABFOCK3CSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXHE4DINBWGU> . You are receiving this because you commented.Message ID: ***@***.***>

[jnjnnjzch]

Hello,

I've checked the newly updated codes and it indeed solved this problem. However, in my case, dminx should be set as the horizontal distance between electrodes to avoid CUDA out of memory error.

I'm closing this issue and thank you again for your patience!