increasing the GPU usage #798
acn-reginald-casela
started this conversation in
General
Replies: 1 comment
-
No, you can increase num_workers with GPU utilization but it will not increase the throughput (#100 (comment)) |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Is there a way to fully utilize the GPU memory that I'm using? I've deployed an app in GKE that uses a NVIDIA Tesla T4 and it's only using roughly 3-4gb out of 16gb. I'm using the large-v2 model and the usual runtime for a 10-15s clip is around 2-5s.
I was wondering if using the num_workers parameter could also help with GPU utilization and decrease the runtime further to at least 0.5-1sec.
Beta Was this translation helpful? Give feedback.
All reactions