Expose a "PassthroughEffect" to enable more flexibility for effect authors

[Sergio0694]

Overview

With the new infrastructure for custom effects, it's now possible for developers to build custom effects that can then also be "packaged" into ICanvasImage objects that can be easily passed around and reused. For instance, ComputeSharp exposes the CanvasEffect base type for this, which we're also using for all our custom effects in the Microsoft Store. One scenario that is relatively common though, especially in more complex effect graphs, is to be able to "switch" from different inputs. That is, consider this scenario:

flowchart LR
Source --> 1
1 --> 2
1 --> 3
2 --> 4
3 --> 4
4 --> Output
1 --> Output

We have an effect graph with an input node, then a bunch of effects, and then an output node. Suppose we want to be able to "switch" so that the output will either go through 4, or through 2 and bypass the other upstream effects entirely. To do this efficiently, we'd want the output node to be a passthrough effect, so that the internal effect implementation can just set the right source node for it in a way that's transparent to code consuming that output effect. This is crucial, because it means the effect can keep using the same ICanvasImage object (which can also be passed around), and will simply change its source.

Having a passthrough effect is also particularly useful to be able to have a uniform API surface that exposes ICanvasEffect no matter what the underlying source is. That is, you can have eg. a CanvasBitmap with a passthrough effect, and pass it around to code that only wants an ICanvasEffect object instead, and it'll work just fine.

API proposal

The proposal is to add a new PassthroughEffect, defined as follows:

namespace Microsoft.Graphics.Canvas.Effects;

public sealed class PassthroughEffect : ICanvasEffect
{
    public IGraphicsEffectSource Source { get; set; }

    // ICanvasEffect members here...
}

API use

The effect can be used as follows, eg. using CanvasEffect from ComputeSharp:

public sealed class BackgroundBlurEffect : CanvasEffect
{
    private GaussianBlurEffect? _effect1;
    private GaussianBlurEffect? _effect2;
    private GaussianBlurEffect? _effect3;
    private BlendEffect? _effect4;
    private PassthroughEffect? _output;

    private IGraphicsEffectSource? _source;
    private bool _bypassEffects;

    public IGraphicsEffectSource? Source
    {
        get => _source;
        set => SetAndInvalidateCanvasImage(ref _source, value);
    }

    public bool BypassEffects
    {
        get => _bypassEffects;
        set => SetAndInvalidateCanvasImage(ref _bypassEffects, value);
    }

    protected override ICanvasImage CreateCanvasImage()
    {
        // Create the effect graph nodes
        _effect1 = new GaussianBlurEffect();
        _effect2 = new GaussianBlurEffect();
        _effect3 = new GaussianBlurEffect();
        _effect4 = new BlendEffect();
        _output = new PassthroughEffect();

        // Connect the fixed nodes
        _effect2.Source = _effect1;
        _effect3.Source = _effect1;
        _effect4.Background = _effect2;
        _effect4.Foreground = _effect3;

        return _output;
    }

    protected override void ConfigureCanvasImage()
    {
        _effect1.Source = _source;
        _output.Source = _bypassEffects ? _effect1 : _effect4;
    }
}

In this case, the graph is only built once, and the passthrough effect is used as the output node. Then whenever the "bypass" property changes, there's no need to rebuild the whole graph, instead the output node can just "switch" the source node internally. This means that any downstream consumers of this output node also don't need to do anything else or otherwise be modified. They'd just get the correct graph when trying to build the realized D2D effect from that node, as it's always the same.

[rickbrew]

There are some special things to note about a passthrough effect that is implemented through the use of ID2D1TransformGraph::SetPassthroughNode(). These are based on my observations from using Direct2D for a lot of effect development.

1. Setting the D2D1_PROPERTY_PRECISION property on the passthrough effect will not be honored -- you truly are getting passthrough behavior in this case.
2. Setting the input to be an ID2D1CommandList will lose the command list's resolution independence. As reported through ID2D1DeviceContext::GetImageLocalBounds(), a command list can have non-integer values for its bounding box, while an effect cannot. (an example would be a command list with a DrawRectangle() call with non-integer coordinates)

Fixing 1 isn't really possible, as you don't get notified about changes to your own precision property. The fix here is to have a separate effect, which I call the PrecisionEffect. Instead of using SetPassthroughNode(), it uses a simple passthrough shader (e.g. return D2D.GetInput(0);). However, the utility of a "precision effect" is quite niche -- I'm not actually recommending you bother with this.

Fixing the second one? This one really depends on how your COM wrappers-and-interop system works. If you can have a single ICanvasImage instance that is able to reproject itself as a different ID2D1Image then you could do it. But generally these COM wrapper systems maintain a rigid mapping between RCWs and CCWs and their associated native or managed objects.

[Sergio0694]

"Fixing the second one? This one really depends on how your COM wrappers-and-interop system works. If you can have a single ICanvasImage instance that is able to reproject itself as a different ID2D1Image then you could do it. But generally these COM wrapper systems maintain a rigid mapping between RCWs and CCWs and their associated native or managed objects."

Can you clarify exactly what would that "fix" look like, in theory? I'm not sure I'm following what you mean 🤔

[rickbrew]

"Fixing the second one? This one really depends on how your COM wrappers-and-interop system works. If you can have a single ICanvasImage instance that is able to reproject itself as a different ID2D1Image then you could do it. But generally these COM wrapper systems maintain a rigid mapping between RCWs and CCWs and their associated native or managed objects."

Can you clarify exactly what would that "fix" look like, in theory? I'm not sure I'm following what you mean 🤔

Your managed object/wrapper needs a way to provide the native object it's creating/wrapping. Often this is fixed at the time of constructing the wrapper. But, if the wrapper is able to provide a different native object each time it's needed, then it can do so. However, this still comes with the risk that the native object is being held onto and won't be "updated."

For instance, in my code, I do stuff like this:

internal sealed class D2D1DeviceContext6 : ...
{
    public void DrawImage(IDeviceImage image, ...)
    {
        ...
        using ComPtr<ID2D1Image> spImage = default;
        HRESULT hr = ComObject.GetIUnknown(image, &spImage);
        hr.ThrowOnError();
        ...
        this.pD2D1DeviceContext6->DrawImage(spImage, ...);
    }
}

ComObject.GetIUnknown() will do various things to extract the native COM object from the managed wrapper. Because I control the whole interop system, I can make it possible to return a different object each time. However, I'd still need to recreate the effect graph to accomplish this; if the native object is embedded in an effect graph that is used after my wrapper changed its mind, it won't draw the new stuff unless the new native object is retrieved.

[Sergio0694]

Notes from API review

This is no longer a blocker for the Store, as we could work around this by expanding the effect graph APIs in ComputeSharp instead to allow toggling output nodes dynamically (done in Sergio0694/ComputeSharp#517). It might still be useful to expose a passthrough effect in the future (as it can be useful as part of building a more general effects system), but implementing this would require increasing the complexity in Win2D due to the handling of DPIs and DPI compensation in CanvasEffect and derived types. It's doable, but it might make sense to get back to this with a more extensive proposal to allow building a more generalized effect system as well (eg. through interfaces to allow connecting nodes together without knowing their concrete types, as well as querying and setting properties, etc.). Likely not worth just adding this passthrough as is for now.