What's the most efficient way to render a XAML visual tree onto a DXGI surface?

[AdamSzofran]

(I originally posted this question in the Win2D repo but this might be a more appropriate place for it.)

I'm using MediaCapture in a UWP app to get a video stream from a camera with the intent to simultaneously record it and live stream it. For sports, I'd like to overlay scoreboard information that I've laid out using XAML controls.

I plan to use either the D3D11 video APIs or a custom video effect (and Win2D if applicable) to do the overlay in real-time. However, both of those techniques require the overlay to be in a DXGI surface and I can't figure out how to efficiently render the XAML tree into one. The best approach I've seen is to use RenderTargetBitmap and then copy the bits to the CPU before pushing them back into DXGI. That seems horribly inefficient.

XamlUIPresenter looks like it would be perfect but

XamlUIPresenter is not intended for general Microsoft DirectX and Windows Runtime interoperation scenarios. You cannot use this API in a UWP app that you submit to the Microsoft Store, it will not pass certification.

Is there a way to do this with supported APIs entirely in video memory without the round trip to the CPU?

Thanks.

[daneuber]

Hi @AdamSzofran can you clarify whether you wanted the XAML scores overlay over BOTH the stream and recording or only one of them?

[AdamSzofran]

Both (live stream and recording). I have it working currently via RenderTargetBitmap to a SharpDX Texture2D through the CPU via GetPixelsAsync. Then into the video streams via IBasicVideoEffect with a VideoProcessorBlt.

Fortunately I only need to use the RenderTargetBitmap infrequently when the clock or score changes and even then I only have to update a small part of the overlay. Still, it would be nice to eliminate the path through the CPU because then I could use animations on the overlay that would update more frequently.

[AdamSzofran]

It would be great if there was a way to get the DXGI surface backing the RenderTargetBitmap (presumably there is one). Then you could use it on the GPU with D3D, D2D, etc. without having to copy it back to the CPU. Thanks.

[daneuber]

Thanks for the additional info @AdamSzofran! Given your scenario we'd suggest you build the UI using XAML, get the backing Visual (using ElementCompositionPreview) for the root element of the subtree you'd like to capture, and then capture the content to a D3D surface using the Capture API - var item = GraphicsCaptureItem.CreateFromVisual(visual); Let us know if that works for you!

[AdamSzofran]

Thank you @daneuber! I will try that. For some reason I thought the capture API required the user to select the visual manually and give permission but perhaps that's not required when capturing a Visual from your own app.

Edit: Ok, now I see. GraphicsCaptureItem.CreateFromVisual is new and requires one of the Insider Preview builds. Thanks. https://blogs.windows.com/buildingapps/2018/05/22/windows-10-sdk-preview-build-17672-available-now/

[kannan95]

Hi @daneuber @AdamSzofran , I am trying to build a screen capture UWP app to capture the whole display(Full screen). I don't need to provide a picker to the user to choose since I need to capture whole display always. The Capture API can be used to capture screen but it needs a picker to work.
How can I capture the whole screen using Capture API without a picker? If GraphicsCaptureItem.CreateFromVisual can be use to do that, how? Thanks in advance.

[daneuber]

Hi @kannan95 as of now the picker is required as part of the capture API, but providing the option to opt out of the picker is a request we're tracking internally as something we may support in the future.

[kannan95]

Thanks for the update @daneuber I'll definitely watch out for that.

[daneuber]

@AdamSzofran Yes sorry for the delay; CreateFromVisual was a new feature recently added to the Capture APIs available in Insider build. Let us know how trying that out for your scenario goes!

[Adam-VisualVocal]

Will do. I held off trying this while the feature was pre-release because I didn't have time to set up a VM to test it. Now that 17763 is out I plan to give it a go very soon. Thanks.

[AdamSzofran]

I did finally get this working, mostly. The issue I'm having now is that the Xaml is always rendered onto an opaque black background and alpha is not preserved in the captured image. How can I capture transparent or semi-transparent Xaml and have the resulting captured image not be fully opaque? Thanks.

[Zofware]

Looks like the transparency issue might be a bug: microsoft/Windows.UI.Composition-Win32-Samples#41

[Zofware]

I can confirm that the transparency issue is definitely fixed in Windows 10 build 19041 (aka May 2020, aka 2004). Yay!