Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apparent locking issues when running across multiple GPUs #283

Open
gtebbutt opened this issue Dec 6, 2023 · 0 comments
Open

Apparent locking issues when running across multiple GPUs #283

gtebbutt opened this issue Dec 6, 2023 · 0 comments

Comments

@gtebbutt
Copy link

gtebbutt commented Dec 6, 2023

I've noticed an interesting issue when running on multi-GPU machines: although selecting gpu(N) as the decoding context initially works as expected, the overall throughput when running multiple processes drops off very rapidly until there's only one process showing activity on a single GPU, sometimes with occasional very short bursts of processing from others.

This happens even when the processes are totally independent (started separately from different screen sessions, operating on entirely different files, using separate GPUs, for example), which leads me to think there's probably a hardware- or system-level locking mechanism being used globally rather than per-process since it occurs even between separate python instances.

Working theory is that it could be falling through to a global lock of some kind due to setting decoder_info_.vidLock = nullptr;, but so far that hasn't brought us closer to a fix. Would be very helpful to hear if anyone else has (or hasn't!) run into similar issues?

Possibly related to #187 and/or #159?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant