You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've noticed an interesting issue when running on multi-GPU machines: although selecting gpu(N) as the decoding context initially works as expected, the overall throughput when running multiple processes drops off very rapidly until there's only one process showing activity on a single GPU, sometimes with occasional very short bursts of processing from others.
This happens even when the processes are totally independent (started separately from different screen sessions, operating on entirely different files, using separate GPUs, for example), which leads me to think there's probably a hardware- or system-level locking mechanism being used globally rather than per-process since it occurs even between separate python instances.
Working theory is that it could be falling through to a global lock of some kind due to setting decoder_info_.vidLock = nullptr;, but so far that hasn't brought us closer to a fix. Would be very helpful to hear if anyone else has (or hasn't!) run into similar issues?
I've noticed an interesting issue when running on multi-GPU machines: although selecting
gpu(N)
as the decoding context initially works as expected, the overall throughput when running multiple processes drops off very rapidly until there's only one process showing activity on a single GPU, sometimes with occasional very short bursts of processing from others.This happens even when the processes are totally independent (started separately from different
screen
sessions, operating on entirely different files, using separate GPUs, for example), which leads me to think there's probably a hardware- or system-level locking mechanism being used globally rather than per-process since it occurs even between separate python instances.Working theory is that it could be falling through to a global lock of some kind due to setting
decoder_info_.vidLock = nullptr;
, but so far that hasn't brought us closer to a fix. Would be very helpful to hear if anyone else has (or hasn't!) run into similar issues?Possibly related to #187 and/or #159?
The text was updated successfully, but these errors were encountered: