OpenCV video decoding too slow for vidgear - ffmpeg might be better suited for decoding #148
Comments
@golyalpha I Don't think that replacing OpenCV with FFmpeg is a good idea. Since the only leverage of FFmpeg over OpenCV is performance, While OpenCV is more advantageous as:
This is some of the reason why it's not a good idea. Being said that, we can still implement another Videocapture gear which works purely on FFmpeg. Thanks for this idea anyways. |
abhiTronix
added
ENHANCEMENT ⚡
|
Hmm, it is quite possible that merely using a different OpenCV backend - FFmpeg in this case - could potentially result in the lost performance to be gained, but I am quite skeptical to be honest. Gonna run some more benchmarks, and see. As for FFmpeg not being available on some linux distributions - afaik, most offer system or user packages using which you can install ffmpeg, that being said, certain features (like hwaccel support) might be missing. |
|
Alright, it seems that OpenCV is the part slowing down the decode process, because I just ran the same benchmark with ffmpeg as OpenCV backend, and the results are pretty much within the margin of error compared to the first one. |
|
I ended up using another python module(PyAv) to decode frames because opencv (and vidgear as well because of it) just gives me less frames, despite reporting a higher total frame count. FFmpeg both reports and gives me all frames. While less ideal, i used that for decoding and Writegear for encoding because it's so well made and easy to use. I should probably investigate if the ffmpeg backend for opencv works better though. |
|
@Thomasedv That frame count mismatch sounds like a bug, though I haven't had that issue. |
|
It likely is, probably related to the file format I was working with. But it's what pushed me to just swap to something other than opencv. |
|
@abhiTronix, how open are you to using PyAV instead of FFMpeg in a subprocess? |
@golyalpha PyAV directly provides Pythonic binding for the FFmpeg libraries, so there's no need of subprocess, thanks for bring this up @Thomasedv . |
|
Yeah, no, those were two separate things (PyAV OR FFMpeg in a subprocess), I didn't mean using PyAV in a subprocess. |
@Thomasedv Nope. It works worse according to my experience. |
|
@golyalpha Can you benchmark PyAV too? |
|
I can try, sure. |
This comment has been minimized.
This comment has been minimized.
|
Alright, so, I've ran two benchmarks. One for just pure PyAV, and two with conversion to ndarray (with conversion to bgr24 pixfmt) PyAV, no numpy: The framerates are much lower than what I expected, but at least all of them are above 60 With conversion to bgr24 ndarray: Unfortunately, these framerates are a bit too slow (really within a margin of error compared to OpenCV) for my tastes, though I am unsure what exactly is causing it - whether it's the yuv444p to bgr24 conversion, or the conversion to ndarray. |
|
@golyalpha Can you share some sample code? There are a few ways to improve performance. Here is a snippet from my loading: Some basic code that iterated all frames, that i previously used, take note that with or without FAST is negligible for the AUTO/FRAME, depending on run one is faster than the other. Timetable: Tested on ~6 sec 1080p 24fps video. The conversion to images is by far the bigger factor. Without that, it can be as low as 0.23 seconds. |
|
@golyalpha Are you using system FFmpeg or provide with PyAV? Try this to use system one (uninstall other one): https://pyav.org/docs/stable/overview/about.html#bring-your-own-ffmpeg |
|
@abhiTronix I'm on Windows, so there's only statically compiled ffmpeg. I'm gonna try out the suggestions provided by @Thomasedv though. |
|
@Thomasedv Take a look at this: https://pyav.org/docs/stable/overview/about.html#unsupported-features. Unfortunately, They don't support Hardware Decoding. |
abhiTronix
added
the
OPEN MIC 🎙️
|
Alright, here's the benchmarks for AUTO and FRAME thread modes: FRAME: As for Hardware Decoding, there is a valid point brought up by the FFMpeg team, regarding most CPUs being able to decode video just fine, and as the above benchmarks provide, it may be unnecessary. While I do feel like there's more performance to be had from ffmpeg SW decode, I believe the main goal of this issue - to get decode speeds up to a reasonable frame rate (i.e. at least in the ballpark of encode performance) - has been met, and I feel that any performance gains that are to be had, can also be achieved even once the video source gear with PyAV backend has been implemented. It should also be noted that all benchmarks above include the conversion to ndarray with bgr24 pixel format in their timings, meaning there shouldn't be any undue performance drops from my code, to VidGear's implementation. |
|
@golyalpha That's good (reasonably close i guess?), also thanks for the tip on ffmpeg backend for opencv. Also depending on CPU you might be limiting your decoding with encoding taking up the CPU. @abhiTronix I wouldn't mind a PyAV backend, but the reason i like VidGears WriteGear is that i can basically just supply my own arguments i use with ffmpeg for other applications and i'm good to go. No need to find out what i need to call to make sure it works as i want. So i hope it that won't get completely replaced. Not to mention this brings hardware support given the user has a ffmpeg build with support. |
|
As for the code that I currently use to run the decode benchmark, here it is: container = av.open(
path.join(
data,
f"{label}.mp4"
)
)
stream = container.streams.video[0]
stream.thread_type = 'FRAME'
start = perf_counter()
for frame in tqdm(
container.decode(stream),
desc=f"Decoding ({res['width']}x{res['height']})",
unit="f"
):
frame.to_ndarray(format="bgr24")
runs[label] = BENCHMARK_FRAMES/(perf_counter() - start)
container.close()
In my case, the data folder is geenrated from a single source file during the encode benchmarks. I'm gonna write some transcode benchmarks now to measure the throughput, but yeah. |
|
Alright, so, here's the full benchmark suite results. There's probably no surprise that the transcode benchmark did take a performance hit compared to both encode and decode benchmarks: Fortunately, this can be resolved by using two machines instead of one, and passing the frames using VidGear's NetGear. |
abhiTronix
changed the title
abhiTronix
added
WORK IN PROGRESS 🚧
🎉 A new library is here: https://github.com/abhiTronix/deffcode👍🏽 Any suggestions are most welcomed. |
|
@abhiTronix I'm using a Logitech Brio 4k webcam that needs MJPG to stream 4k. Using Videogear for this is very CPU intensive. On Windows, I get full FPS no problem. On Ubuntu, I get 100% CPU usage and lowered FPS. Is there a way to hardware accelerate webcams using OpenCV/VideoGear? I'm using this for a vehicle autonomy project (Openpilot Webcam) |
|
@zorrobyte use deffcode instead since Videogear works on OpenCV backend which itself is slower. |
Is it still possible to use cv options with deffcode? Is there a webcam code example for me to try? |
Unfortunately, Webcams are yet to be supported by deffcode. You can try CUDA backend for OpenCV Videocapture Decoding to speed up things while decoding. FYI Videogear or vidgear in general cannot accelerate beyond the speed of OpenCV itself. It can offload process to a different threads by multi-threading, but if the producer thread is already slow, it cannot do anything. |
|
can you update us with the time webcam will be supported? and it worked but the frames is not showing on WebGear_RTC website |
|
@AbdulrahmanSoliman1 It will happen eventually but I need to finish releasing Vidgear |
Detailed Description
I've ran some encode and decode benchmarks, and it's becoming fairly obvious, that VidGear (CamGear API) is currently unable to decode 1080p60 video on computers where, given certain settings, ffmpeg can encode 1080p60 in real time or slightly faster without hardware acceleration.
Context
Currently, it's impossible to pass 1080p60 H264 (only tested codec) video through VidGear on computers that should be able to decode 1080p60 video just fine. The idea would be to replace the OpenCV VideoStream API with something more performant, like ffmpeg, since ffmpeg is capable of outputting raw video into stdout.
The CamGear API should not need to change from the developer's standpoint.
Your Environment
Any Other Important Information
Encode/Decode benchmarks for VidGear (encode is compressed, so ffmpeg):
ffmpeg decode benchmark (4k60):
The text was updated successfully, but these errors were encountered: