warning Could not send read to Guppy

[jennieli421]

Hi, I noticed that the warning Could not send read to Guppy appears quite frequently, when I use the promethion exemplar bulkfast5 to run a simulation. Wonder if this is normal. If not, is there anything in my setting that is causing this.

Below is my caller setting and mapper setting (using latest readfish version):

[caller_settings.guppy]
config = "dna_r10.4.1_e8.2_400bps_5khz_fast_prom.cfg"
address = "ipc:///tmp/.guppy/5555"

[mapper_settings.no_op]

The log info:

2023-11-21 12:19:17,772 readfish 0265R/0.5843s; Avg: 0247R/0.5494s; Seq:2,619; Unb:3,571; Pro:0; Slow batches (>1.00s): 2/25
2023-11-21 12:19:18,401 readfish 0215R/0.6238s; Avg: 0246R/0.5523s; Seq:2,629; Unb:3,776; Pro:0; Slow batches (>1.00s): 2/26
WARNING:root:Could not send read 'RF-e0e0b000-305f-41c5-8113-24317883c2a0' to Guppy
WARNING:root:Could not send read 'RF-8e624fb5-21dd-4a2f-abb1-918395a691bc' to Guppy
WARNING:root:Could not send read 'RF-08507134-3d95-4596-ac3a-9526f617e315' to Guppy
WARNING:root:Could not send read 'RF-5d8d13fd-10a4-4619-b256-7d1d2055f74d' to Guppy
WARNING:root:Could not send read 'RF-0f9d2389-df02-4a38-a9bf-fd58a393caeb' to Guppy
WARNING:root:Could not send read 'RF-8964b721-8c21-4a76-96f4-81a7aa44d0bd' to Guppy
2023-11-21 12:19:19,610 readfish 0441R/1.1977s; Avg: 0253R/0.5762s; Seq:2,641; Unb:4,205; Pro:0; Slow batches (>1.00s): 3/27
WARNING:root:Could not send read 'RF-4bb63fa6-83ef-42f7-b4d4-04f8e9e87757' to Guppy
2023-11-21 12:19:20,775 readfish 0619R/1.1562s; Avg: 0266R/0.5969s; Seq:2,656; Unb:4,809; Pro:0; Slow batches (>1.00s): 4/28
WARNING:root:Could not send read 'RF-73a20d7e-a7e2-422d-962c-8b9f6a3690e4' to Guppy
WARNING:root:Could not send read 'RF-4f013487-60e9-41af-a6d4-e1441755bbeb' to Guppy
WARNING:root:Could not send read 'RF-a54b2a3f-3366-4666-9e9f-781c30841aad' to Guppy
2023-11-21 12:19:21,898 readfish 0546R/1.1088s; Avg: 0276R/0.6146s; Seq:2,667; Unb:5,344; Pro:0; Slow batches (>1.00s): 5/29
WARNING:root:Could not send read 'RF-b6ea220c-cfed-4a19-8401-bbb23305afd0' to Guppy
WARNING:root:Could not send read 'RF-cfa519ca-d4bb-4d33-aa50-261a35c3fcb5' to Guppy
2023-11-21 12:19:22,762 readfish 0428R/0.8563s; Avg: 0281R/0.6226s; Seq:2,682; Unb:5,757; Pro:0; Slow batches (>1.00s): 5/30
WARNING:root:Could not send read 'RF-4c79e062-463c-4aae-a28d-edb29773958f' to Guppy
WARNING:root:Could not send read 'RF-d06bf9b0-efe4-44e5-8ae2-4eb5e0b20629' to Guppy

[alexomics]

This is indicative that the basecaller server cannot accept any more reads at that point in time. What GPU device/devices are you using for this simulation? How is your basecaller configured? And are you running live basecalling at the same time? As the GPU is shared many users can all affect performance of readfish.

[jennieli421]

After turning off basecalling in MinKNOW there is no such warning anymore. Thanks for your help.

[alexomics]

The basecaller config is either how you have started the basecalling server or how MinKNOW has started it. There's some brief information on setting up the GPUs in the docs though this is more for those with multiple GPUs.

If MinKNOW is managing the basecalling server then one of these commands will show the service information, depending on whether it's Guppy or dorado.

systemctl status guppyd

or

systemctl status doradod

From there you maybe able to tweak the --chunks_per_caller value to allow basecalling live data and readfish data.

[jennieli421]

Follow up. I set the the log message when I run the experiment said "slow batches > 0.4s". I also observed the rejection peak was around 650bp, longer than what I expected (break_read_after_seconds=0.4). What might be the reason for this slowness and what should I try to improve it?

2023-11-22 14:29:37,120 readfish.targets 2565R/3.0045s; Avg: 2565R/3.0045s; Seq:1,633; Unb:422; Pro:510; Slow batches (>0.40s): 1/1

[jennieli421]

Hi, I would like to follow up on the issue related to GPU/CPU resource allocation. nvidia-smi command shows that both GPUs are used by dorado_basecall_server. However, I still would like to increase the speed of decision making by readfish and the speed of basecalling. Will the commands in this doc (https://looselab.github.io/readfish/guppy-params.html) help with this issue? Could you provide more details on the subsequent command to use after shutdown and restarting dorado? Is there anyway to change CPU resource allocation?

[mattloose]

Hi,

We suggest you look at the dorado github for optimising performance of dorado. see https://github.com/nanoporetech/dorado

We would configure a system with two GPUs such that one was dedicated to running readfish and the other dedicated to running basecalling for MinKNOW. This will mean you can optimise each and you can guarantee that they won't interfere with one another.

Thanks.