Hit Testing on Visuals?

[legistek]

This article states "To do hit testing, you need to do bounds calculations by walking the visual tree yourself. " Traversing the visual tree is no problem. But how does one do hit testing on an individual Visual? Particularly with complex geometries, there seems to be no way to do it on Visuals that are not UIElements Am I missing something? Thanks!

[ajbennet]

Hit-testing at a Visual-level is not supported at the moment. Could you please share more information about your scenario and we can help figure out the right approach? It also helps us build the right set of features to make developers like you more successful. To your question on bounds calculations, yes it is not easy to do bounds calculations on complex visual trees or when the content is being animated/transformed. On simpler visual trees, since the developer builds the visual tree himself and positions them, they can do the math to figure out the bounds. It is not possible to use D2D for hit testing in this scenario.

[legistek]

Hi @ajbennet thanks for the info. Makes sense.

So I'm mostly experimenting right now out of curiosity but I was wondering if it would be possible to make a UI framework from scratch using Windows Composition but not be a UWP app. In particular, with WPF being open source I was curious if the WPF rendering backend could eventually be updated to use Windows Composition. Nothing urgent or mission critical but thanks for asking. However if I were to make a suggestion, it would be great if Visual had a hit testing method, and even better if it accounted for scale and rotation (or any transformation for that matter).

FWIW, I've been considering two approaches to hit testing in light of your answer, both using D2D. In one scenario I'd use a DXGI surface brush for every Visual, using D2D to render to the surface. Then I would have access to the D2D geometries and could use them for hit testing. However when I tried this with thousands of Visuals the performance really started to suffer. I'm certain I could improve the situation dramatically by caching surfaces based on rendering parameters, but this seems like a lot of wheel reinventing.

My other idea - which I haven't tried yet, but which I'm pretty sure will work - is to use Windows Composition shape visuals, but also make a "shadow" D2D geometry for each Visual with the same dimensions. Then I would likewise use the D2D geometry for hit testing but let Windows Composition continue to take care of all the rendering. The only thing that bothers me about this is that I feel like deep down you're holding the D2D geometries already somewhere and just not exposing them. :)

Finally I thought about just going back to DirectComposition. @ajbennet I think it was you or one of your colleagues who previously commented that Windows Composition is not built on DirectComposition. Is that strictly true even at the deepest levels? And do I really gain much - besides easier programming in C# - using Windows Composition vs. DirectComposition given that I'm not making UWP apps?

Thanks!

[legistek]

Hi, has there been any update on this in the last 4 years by chance?

I'm still a little confused how the Visual Layer could be considered a replacement for DirectComposition (see https://learn.microsoft.com/en-us/windows/win32/directcomp/directcomposition-portal) if it doesn't support something this basic.

I don't want to use WinUI3 and WPF is not a viable option for a new project iMO. If I'm not supposed to use DirectComposition anymore either I'm at a loss as to to what the "gold standard" for interactive UI in Windows even is right now.

[jeffstall]

Hi, can you clarify what feature that DirectComposition has that Windows.UI.Composition does not yet have?

[legistek]

@jeffstall sure I'm looking specifically for hit testing.

So if I use DirectComposition I have access to the Direct2D geometries and can use them for hit testing. Take an ellipse for example, or some complicated filled path.

As far as I can tell with Windows.UI.Conposition I can only do trivial hit testing based on the rectangular bounds.

I realize in both cases I need to traverse down my visual tree which is fine and good but my issue is determining if a point inside the rectangular bounds actually hits a filled pixel. But since I don't have access to DX geometries or surfaces it's not clear how I would do this.

[jeffstall]

Hi @legistek, can you clarify which specific DirectComposition APIs with parameters you're using that are not available with Windows.UI.Composition?

[legistek]

For example

https://learn.microsoft.com/en-us/windows/win32/api/d2d1/nf-d2d1-id2d1geometry-fillcontainspoint(d2d1_point_2f_constd2d1_matrix_3x2_f_bool)

[jeffstall]

Hi @legistek,

ID2D1Geometry::FillContainsPoint() is a D2D API that's related to immediate mode drawing and is different than DirectComposition. I am trying to clarify whether there a DirectComposition API (potentially that uses D2D geometries) that is not available in Windows.UI.Composition, or whether you are asking for a new hit-testing feature that works with D2D geometries, or something else.

With thanks!
~ Jeffrey

[legistek]

I'm not sure why this is confusing. DirectComposition allows you choose how you want to draw surfaces, including D2D or DXGI. So, if - for example - I were to use D2D to draw to a DirectComposition surface then I would be able to use ID2D1Geometry::FillContainsPoint() to hit test within the pixel contents of that surface.

See also https://learn.microsoft.com/en-us/archive/msdn-magazine/2014/august/windows-with-c-directcomposition-a-retained-mode-api-to-rule-them-all

With Windows.UI.Composition, I do not have access to the lower level data structures or surfaces. Therefore I appear to have no way to perform hit-testing inside the Visual if it uses a complex geometry. The only thing I can apparently do is create a DXGI surface brush which as I mentioned performs poorly at scale.

So back to my original question from 4 years ago (which had nothing to do with DirectComposition) - how do I do hit testing with Windows.UI.Composition on a complex geometry? And if the answer is, I can't (which is what @ajbennet said 4 years ago) then why is your documentation telling me I should be using Windows.UI.Composition to build my UI instead of lower level technologies such as DirectComposition in partnership with D2D?

[legistek]

@jeffstall ok so I know for a fact Windows.UI.Composition is built over DirectComposition on some level, which explains why you're surprised there's something I can do with DC that I can't do with W.U.C. But the problem is it's abstracting away the things I need.

Given this I see no reason to use W.U.C. The recommendation on the DC page that apps for Windows 10+ should use the "Visual Layer" (which just means W.U.C) really makes no sense. If I wanted something high level I'd just use WinUI3. I don't; I want something low level. Using W.U.C short of full blown UWP or WinUI3 doesn't appear to give me anything I can't do with DC and a lot of things I can't.

If you're looking for something to improve, what I'd say is that yes it would be nice to have all the DirectX scaffolding taken care of for me. But I still need to be able to draw my own primitives and have hit testing. Maybe if you added a virtual OnRender method to Visual, which provides the ID2D1DeviceContext I know you're already using, then I could hold onto all my geometries and use them for hit testing.