Add Video Super Resolution for Windows (AMD, Intel and NVIDIA) and MacOS #1180
Conversation
Changes made: 1. Creation of a new class VideoEnhancement which check the liability to the feature. 2. Add the checkbox "Video AI-Enhancement" in the "Basic Settings" groupbox. 3. Disable VAE when fullscreen is selected 4. Add a registery record 5. On the Overlay and the mention "AI-Enhanced" when activated 6. Add a command line for the class VideoEnhancement
Adding VideoProcessor to D3D11VA to offline video processing from CPU to GPU, and leveraging additional GPU capabilities such as AI enhancement for the upscaling and some filtering. Changes made: 1. VideoProcessor is used to render the frame only when "Video AI-Enhancement" is enabled; when disabled, the whole process is unchanged. 2. Add methods to enable the Video Super Resolution for NVIDIA, and Intel. AMD method is currently empty, need to POC the solution with the AMF documentation. 3. Add methods to enable SDR to HDR. Currently only NVIDIA has such feature, but the code place is prepared if Intel and AMD will too. 4. Some existing variables local to a method (like BackBufferResource) changed to global scope to be consumed be also VideoProcessor methods. 5. In ::initialize(), the application checks if the system is capable of leveraging GPU AI enhancement, if yes, it inform the UI to display the feature. 6. ColorSpace setups (Source/Stream) for HDR are not optimal, further improvment might be possible. Issues observed are commented in the code at relevant places.
linckosz
changed the title
Changes made: 1. Creation of a new class VideoEnhancement which check the liability to the feature. 2. Add the checkbox "Video AI-Enhancement" in the "Basic Settings" groupbox. 3. Disable VAE when fullscreen is selected 4. Add a registery record 5. On the Overlay and the mention "AI-Enhanced" when activated 6. Add a command line for the class VideoEnhancement
Adding VideoProcessor to D3D11VA to offline video processing from CPU to GPU, and leveraging additional GPU capabilities such as AI enhancement for the upscaling and some filtering. Changes made: 1. VideoProcessor is used to render the frame only when "Video AI-Enhancement" is enabled; when disabled, the whole process is unchanged. 2. Add methods to enable the Video Super Resolution for NVIDIA, and Intel. AMD method is currently empty, need to POC the solution with the AMF documentation. 3. Add methods to enable SDR to HDR. Currently only NVIDIA has such feature, but the code place is prepared if Intel and AMD will too. 4. Some existing variables local to a method (like BackBufferResource) changed to global scope to be consumed be also VideoProcessor methods. 5. In ::initialize(), the application checks if the system is capable of leveraging GPU AI enhancement, if yes, it inform the UI to display the feature. 6. ColorSpace setups (Source/Stream) for HDR are not optimal, further improvment might be possible. Issues observed are commented in the code at relevant places.
|
I'm going to introduce usage of |
m_IsHDRenabled was a duplication of the existing condition "m_DecoderParams.videoFormat & VIDEO_FORMAT_MASK_10BIT". Replace the variable m_IsHDRenabled (2) m_IsHDRenabled was a duplication of the existing condition "m_DecoderParams.videoFormat & VIDEO_FORMAT_MASK_10BIT".
Remove VideoEnhancement::getVideoDriverInfo() method (which was based of Window Registry) and use the existing method CheckInterfaceSupport() from the Adapter.
No worry, I keep going with current ID3D11VideoProcessor implementation and will do the change once your code is available. |
- MaxCLL and MaxFALL at 0 as the source content is unknown in advance. - Output ColorSpace matched SwapChain
…oProcessor" "reset" was not used in latest code
- NVIDIA: After updating NVIDIA driver to 551.61, VSR works in Exclusive Fullscreen (Tested on a RTX 4070 Ti) - Intel: VSR works in Exclusive Fullscreen (Test on a Arc a380) - AMD: VSR is WIP
|
I did many tests about Color Space with HDR on, as part of the final result I found something interesting that I wanted to share. I have 2 Graphic cards on the same PC, a RTX 4070 Ti and a Arc a380. Conclusion: |
…nt is On - Simplification of the class VideoEnhancement as all properties will be set at D3D11va initialization - Since it never change during a whole session, only scan all GPU once at the application launch and keep track of the most suitable adapter index with VideoEnhancement->m_AdapterIndex. - Adapt setHDRoutput as the adapter might be different (not the one linked to the display). - In case Video Enhancement is Off, we keep using the previous behavior (=using the adapter linked to the display).
- Update setHDRoutput in case of multiple displays, to make sure we get the HDR information of the display where Moonlight is displayed
During the scan, it is useless to enable enhancement capabilities for all GPU as it will be done later right after for only the selected GPU.
@Kobain-Seo , With Video Enhancement: Without: |
|
@cgutman , |
linckosz
changed the title
When the Host has the HDR off, and if Moonlight has HDR on, the screen appears slightly grey.
|
Thank you for your great work. Screenshots are below:
|
|
@Kobain-Seo , |
The checkbox is greyed out instead of being hidden.
|
@linckosz (Medium bug) Using a RTX GPU, when I use the Window Mode, I can manually scale down randomly with no crash, but scaling up while keeping the ratio make the screen becoming black and oftenly crash Moonlight. To avoid the crashes, I just allow the picture to be stretch and bigger than the initial window size. does it mean that moonlight stretch screen when host stream aspect ratio of 4:3 with 16:9 resolution without black bars? I hope there is stretch option for moonlight. |
I able to fix the blackscreen bug later while scaling up the window and it now can keep the ratio adding the black bars. That’s why the box is checked. |
I played some old games which only support 1280x760 or 1024x768. But streaming resolution is 1920x1080. So basically moonlight only support fixed aspect ratio, there are black bars left and right. What i hope is to stretch(or zoom to fit) 4:3 ratio to 16:9. I wondered that 'Medium bug' may be relevant for stretching screen. Thank you for your kind reply. |
|
Hi there, I'm here to give feedback on the Legion Go (Z1 Extreme / 780M RDNA3). And it's a 16:10 1600p capable screen. As discussed on discord, I had a big doubt following a misunderstanding about the functionality implemented in Moonlight with AMD and its equivalent of VSR. But I find it hard to justify its use on a handleld PC. Today, with 5Ghz wifi, I'm able to stream in AV1 1600p=>1600p=>1600p 100mbps @120FPS without any packet loss or jitter. And the Legion Go is able to do this with a TDP of 6-7W and 4 cores deactivated. That's very handy for making the battery last a long time (~6h). 1600p=>1600p=>1600p + AI-Enhanced: impossible to maintain 120 FPS. And with 90-100FPS, I'm on a TDP of 12-13W. 1600p => 1050p/1200p => 1600p + AI-Enhanced: it's hard to go above 70 FPS. And even at 60 FPS and a TDP of 25W or 30W, I still get CPU throttling. Well, I've had some network problems since I recently changed my ISP box (Freebox Ultra), which has led to some jitter due to repeated latency peaks. And I was hoping it would support Jumbo Frames on wired networks, but it doesn't. I'm waiting for some hardware to fix it. Then I'll be able to resume my benchmarks more easily. |
- Force to use FSR 1.0 instead of FSR 1.1 as the second one force to convert YUV->RGB->YUV which is too heavy for iGPU, and we have to efficient way to identify a dedicated GPU from an APU. - Some memory optimization in the renderFrame method to accelerate the render.
- Add The library MetalFX - Check the Upscaling capability based on MacOS version (13+) - Add Spacial scaler for Luma and Chroma textures
linckosz
changed the title
|
Based on the work done from @TimmyOVO on the iOS version (metal branch), I was able to reuse the MetalFX method to apply it to the MacOS version.
|
|
Hey @linckosz, I've applied a few additional optimizations to the OS. And I'm glad it had a (minimal) impact on the Moonlight power reduction. AV1 / 120 Mbps / 120 FPS Here are some screenshots to show the difference in a static context :
As I was saying with the debug version, it really is night and day in terms of performance impact. I share other screenshots here: |
|
Hi @HakanFly ,
|
|
Hi, I have an off-topic question: Is it possible to implement Frame Generation on Moonlight? As far as I know, using Frame Generation on the host side can cause dropped frames on Moonlight. Therefore, I'm curious if it's possible to use Frame Generation on Moonlight instead. |
Hi @DeNT15T , |
|
Hi @DeNT15T, I have no problems with FSR3 FG on games that support it officially or with a mod. But I have an AMD GPU. I don't know about Nvidia. @linckosz I suppose FSR 3.1 should also bring its share of improvements. After all, I know that AFMF already supports Directx11. I thought Directx12 was just a limitation for DLSS FG. |
|
Few information about Frame generation methods with DX11. AFMF and RSR AMD Frame Interpolation (commonly called FG)
AMD AMF FRC |
|
Wow thank you for the feedback |
|
Linux Video Super Resolution result: Vulkan (plvk.cpp)
OpenGL ES2 (eglvid.cpp)
|
Despite the name of the renderer being "opengles2", I think we do generally get a GLES 3.0+ context from SDL. If you want to force it, I think you can do so with code like: |
Hi,
Context:
Video Super Resolution (VSR) is to Video as DLSS is to 3D Rendering.
Why not let Moonlight being one of the first game streaming solution leveraging such technology?
AI upscaling means significantly less bandwidth usage without compromising the video quality!
NVIDIA, Intel (link in French), and more recently AMD start to advertise their respective technologies to enhance video quality using AI.
Solution:
VSR was not a straight forward implementation, I needed to add the component Video Processor to D3D11VA in order to offload the frame processing from the CPU to the GPU, and leveraging their additional GPU capabilities.
I added a UI checkbox in SettingsView.qml, but the main process logic has been done in d3d11va.cpp.
NVIDIA is providing VSR and HDR enhancement, I could implement VSR perfectly on SDR content, but could not yet HDR (more detail below).
Intel is providing VSR, it has been implemented, but yet to be tested on Arc GPU (I don't have it).
AMD just released AMF Video Upscaling, I prepared the code but need a RX 7000 series (I don't have it) and apparently it might be a quite different approach of implementation.
Testings:
The solution works stable on my rig, I did try different configuration (size, bandwidth, V-Sync, HDR, AV1, etc.) during few days.
AMD Ryzen 5600x
32GB DDR4 3200
RTX 4070 Ti
Moonlight v5.0.1
Sunshine v0.21.0
(Update May 6th, 2024)
A complete report is available at the comment below.
I could test it with a wider range of GPUs:
Improvements:
Results (comparison):
Commits description
USER INTERFACE
Add a new UI feature called "Video AI-Enhancement" (VAE).
Changes made:
BACKEND PROCESSING
Adding VideoProcessor to D3D11VA to offload video processing from CPU to GPU, and leveraging additional GPU capabilities such as AI enhancement for the upscaling and some filtering.
Changes made: