Upgrade FFmpeg libraries to 5.0 #5051
Comments
|
Going to cc @Opelkuh as they seem to have the most knowledge around ffmpeg of those that have worked on it here. |
|
I got Index: osu.Framework/Graphics/Video/VideoDecoder.cs
<+>UTF-8
===================================================================
diff --git a/osu.Framework/Graphics/Video/VideoDecoder.cs b/osu.Framework/Graphics/Video/VideoDecoder.cs
--- a/osu.Framework/Graphics/Video/VideoDecoder.cs (revision abe410f850c600c98b0787bba4c3a8de16a32c67)
+++ b/osu.Framework/Graphics/Video/VideoDecoder.cs (date 1645826975338)
@@ -264,8 +264,9 @@
return 0;
var span = new Span<byte>(bufferPtr, bufferSize);
+ int read = decoder.videoStream.Read(span);
- return decoder.videoStream.Read(span);
+ return read == 0 ? AGffmpeg.AVERROR_EOF : read;
}
[MonoPInvokeCallback(typeof(avio_alloc_context_seek))]which resolves this part of the breaking changes:
As for |
|
@nekodex were the macOS failures only when running headless, or also via visual tests? |
|
visual tests |
|
After a bit of further digging, I came across this ffmpeg issue, which seems to confirm the issue lies with videotoolbox VP9 hardware acceleration (on intel igpus?)... and yeah, when testing the visual tests with hardware acceleration disabled, the remaining affected tests (webm) pass without issue. |
|
Does the fallback flow we have implemented mean this is a non-issue during actual user runtime? |
|
Not sure which flow you're referencing... this fallback logic from #4839? osu-framework/osu.Framework/Graphics/Video/VideoDecoder.cs Lines 353 to 356 in 1cd2ff3 If so, it doesn't seem to handle this case as the HW device initializes correctly... but then fails during decode (for VP9/webm at least) |
|
As @nekodex said, right now there's only a fallback in cases when a HW-accelerated codec can't be created or opened. This should prevent it from trying to use codecs that aren't supported by the device but any errors after the decoder starts don't disable HW decoding. Related: #4857 Could you please post some logs from the failing visual tests? Just so we know what to potentially look out for in #4857. Unless it's not even logging anything... I tried to repro it in GitHub CI but it passed all tests. |
|
mp4 (playback works, despite webm (shows first frame, but then doesn't play/decode beyond that): |
|
Also, to confirm I tested playback of the same files using ffmpeg+ffplay and received similar results - which is to say the .mp4 with hwaccel enabled throws the same mp4/h264: webm/vp9: Furthermore, checking playback with ffmpeg+ffplay from 4.3.3 and, well, h264 fails to decode at all with hwaccel whilst vp9 works fine. Go figure. 🤷 |
|
If the information in that ffmpeg issue is to be believed:
EDIT: This has been confirmed to not work. I might have another idea. Will post again if I figure it out |
|
cc @FreezyLemon , not sure if you're also looking to bring the libraries up to date as part of #5974. |
I'd like to, but the VP9 decoding bug is a bit of a dealbreaker. It could be worked around by disabling the hwaccel in this special case, but that's not great. Things seem to be moving in the ffmpeg ticket though, so maybe we can patch the issue soon-ish. If there's no good resolution, disabling hwaccel will always be an option too. In the meantime, the changes proposed in #5974 are basically version-independent. So I don't see a big reason to wait for this to be resolved. |
|
If we're going to update, can we add Opus support? It's currently the most efficient audio codec, and can allow us to drop bitrate to 128kbps (or 96 even), and still have good enough quality for mapping, which will help combat the ever-growing space requirement for beatmaps. |
|
@longnguyen2004 we do not use FFmpeg for audio decode, so i don't know what this has to do with this issue |
Making this to track issues that prevent upgrading of FFmpeg to 5.0.
As part of the osx-arm64/FFmpeg library replacement work in #5043, I also attempted to update FFmpeg to 5.0 but came across issues that caused test failures. I've pushed the work with the updated libraries for 5.0 to a branch here.
It all works with the exception of the following tests in
TestSceneVideo:TestJumpBackAfterEndOfPlayback(on osx-arm64, osx-x86_64, win-x86_64)Fixed thanks to @Opelkuh's patch below
TestJumpForward,TestJumpBack,TestJumpBackAfterEndOfPlayback) (on osx-x86_64)Seems related to videotoolbox VP9 hardware acceleration (as per this ffmpeg issue, limited to intel igpus?), tests pass with hardware acceleration disabled.
The text was updated successfully, but these errors were encountered: