Question about Readfish basecalling performance for messy samples #343
Comments
|
Hi, You are running a slightly strange set up here. Firstly you have configured a second instance of the guppy basecall service on the same GPUs as the original service is running. This is going to possibly reduce your performance as guppy is not designed to work in this way. You have two options. One would be to point readfish at the exisiting guppy server that the promethION is already running on port 5555 and not setup your own dedicated instance of guppy. This should give reasonable performance and should not suffer from bottlenecks. If you want to ensure that you do not get bottlenecked then you should consider switching your system so that minKNOW only utilises a subset of the GPUs and 1 GPU is dedicated to the adaptive sampling pathway. In this example you would modify the standard guppy server to use cuda 1-3 and run your own guppy server completely independently on cuda 4. However, I also note that you are running an guppy server and not a more recent dorado basecaller server - you should consider updating your system. My final comment on the setup is have you considered using the ONT built in adaptive sampling as a comparator here to see if it is an issue with readfish or something else? I would definitely not run two guppy servers on the same physical GPUs. With respect to the later question about waiting for the basecalling batch to complete before aligning and unbocking. The answer to this is that the aligning step is not the bottleneck. The issue is the time it takes to basecall and get the data back. If that is slower than the time to generate the data then the whole process will lag regardless of the alignment and unblock times. I will make a note to update the documentation as to how best set up on these systems. But in short my advice would be:
|
|
Hello @mattloose. Thank you for the reply. We have some testing to do on our end to try to run this to ground. There are a few factors at play for why things are being run this way, specifically we have barcodes on these reads and are testing with barcode-dependent sequencing (though for these samples its trivially applied) which rules out using built in MinKNOW adaptive sampling in the long run as they still don't support that feature as far as I have been able to tell. Its worth testing it however. How do I configure the basecaller_server that MinKNOW spins up for its own purposes? I don't see anything in the UI for that. We certainly want to give Dorado a try, though based on ONT's own admission i'm not expecting anything too drastic in terms of performance between Guppy and Dorado. We have run successfully with these exact conditions for many samples in the past and its only with these recent lower quality samples that we are seeing a problem. I don't know for certain where the second basecalling server is from, I was under the impression that it was spun up by MinKNOW for its own purposes. In general we do not run these adaptive sampling runs with MinKNOW basecalling enabled in the first place as we didn't want to risk hitting performance. It should be noted, we ran 3 samples in paralell all pointed at the same base-caller server, only 2 of the samples (which happened to be of lower quality in the first place) were hitting more than a few slow batches with a third sample behaving normally. All three were pointed at the same base-caller server. It seems from looking at the Info logs that the block time is ~linear across all of them in the number of reads being fed to the basecaller per-batch. I don't know how to measure if they are interfering with eachother but its possible, thats why I commented on the relatively low GPU utilization. By that logic if I wanted to run this more efficiently I would divide my folowcells into multiple readfish instances to reduce that batching time which seams wrong. Would you recommend parallelizing these experiments across different basecall servers pointed at separate GPUs? I am surprised by your suggestion that we link the basecaller to only one GPU. I would have assumed that the guppy/dorado_basecall_servers are dynamically scheduling between the available GPUS which should cut down on potential processing or bus bottlenecks. Is it a known limitation that we should not be using the scheduler feature for the basecaller? We are willing to entertain and at least try this mode for running. |
|
Hi, Thanks for using the barcoding feature! So the preferred way of running this on your system is to use the onboard basecaller service which is started by MinKNOW. For this you would not need to start the basecaller service yourself - so the command below is not needed.
Instead, you would tell readfish to use the already running server on port 5555. You are correct that dorado/guppy should handle performance well across all the GPUs. When we want to guarantee that that is the case, we set up a dedicated GPU for readfish. If you want to do that you will need to change the configuration for the default guppy service setup by MinKNOW. Most likely this is availabel under the systemctl guppyd or doradod. However, from the additional information you have given above it is possible that you are running libraries that simply have too much short material - it would have to be a real spike of short material to have this strong an effect i think. One thing you could try is running the library for a few minutes without readfish and check the distributions. You could compare between the "good" and the "bad" library as well. |
|
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
|
This issue was closed because there has been no response for 5 days after becoming stale. |
We are trying to using readfish on some different sorts of library extractions on our PromethION instrument and we have noticed that for some our library preps we are seeing very poor performance for readfish/the basecaller that falls well outside of our expectations from testing on other libraries. Specifically we are seeing that for some of these libraries with many short fragments that readfish will be overwhelmed sending very large 1000+ batches of reads to the basecaller causing the round-trip time for reads to creep up. We have observed this will push the mean unblock time for reads into the 1500-2000bp range. An example of the slow performance:
I wanted to ask if you have any tips/advice for making the basecaller more performant? We observe that for the tower we have (which has 4 A800 GPUs that are all averaging ~20-40% utilization with several trials active) we still see this very long batch time when we have a lot of short reads. Here is the command we are using to to launch the basecaller:
In terms of versions:
Version Guppy Basecall Service Software, (C)Oxford Nanopore Technologies plc. Version 6.5.7+ca6d6af, client-server API version 15.0.0Have you seen behavior like this before for readfish? What is the bottleneck here? Passing the data to the GPUs? Clearly the actual processing on the GPUs is not enough to saturate the performance. I just want to ask if there is any advice that we are missing for how to run the basecaller/readfish to improve the response time so fewer of the batches in these messy samples are slow.
I also notice in the Readfish code that you wait for all reads to return from a batch before proceeding to align and unblock. Could this issue be alleviated by aligning/unblocking reads before the entire basecaller batch has finished? If it matters this is a barcoded run and as the trial keeps running and more of the pores start to degrade the unblock time becomes tolerable and few of the batches are slow to round trip.
The text was updated successfully, but these errors were encountered: