The API is designed to be Wayland-compatible and is designed against xdg_positioner object from xdg-shell protocol.
Any placement API that requires access to global screen coordinates won't be Wayland-compatible.

To position your popup relative to the top-left corner of a control use OffsetX/OffsetY properties with TopLeft anchor and BottomRight gravity.

We can, however, introduce a callback that would allow to modify offset/anchor/gravity properties right before the popup is shown

Just to clarify, the callback in WPF doesn't use screen coordinates. It is all coordinates relative to the target element. So a (0,0) placement value would mean upper left of the target control.

The WPF callback allows us to look at things like target element size, popup size, and horizontal/vertical offsets, and then return target element-relative placement options. I would think that should be Wayland-compatible since it's not global screen coordinates.

The WPF way would not make us alter the actual HorizontalOffset and VerticalOffset values since it expected the callback result's placements to specify a target element-relative location and axis. See the WPF CustomPopupPlacement for what the result there includes. Keep in mind that since it's using a PlacementMode.Custom setting for this, the Placement property is not even considered.

I admittedly don't know the nuts and bolts of how Avalonia implemented popup positioning very well but when searching around, I happened to see a PopupPositionerParameters class, which looks like a promising possible result for a callback.

If you guys are initializing a PopupPositionerParameters based on current props, what if we added a callback delegate property to Popup (or could be done via an event I suppose too) and if specified, it would pass in the PopupPositionerParameters you built but would allow us to tweak it further before it was actually used? The callback would need to have access to the target anchor element (to see it and its size), the popup (to see it and some of its properties).

If we could alter the PopupPositionerParameters before it was used, that would allow us to dynamically change popup positioning based on the current state of the target element. And if the callback wasn't specified, or it did nothing in its logic, your current default positioning parameters would be used. Does that sound reasonable?