VideoCommon: allow custom shaders to set the alpha value

[iwubcode]

When I initially was attempting to get custom shaders supported, there was a LOT to figure out and plenty of issues to resolve. Because of that, I effectively disabled any attempts at changing alpha values.

Recently, some users were wondering what Tales of Symphonia looked like with a stronger visual outline. I decided to see what this would look like and in order to do so, I needed to add in support for custom alpha values.

(0% outline and black) (20% outline and teal) (100% outline and black)

This won't allow you to set the alpha if the game doesn't already support it (that'd be another feature) but this does give users a little more capability!

[iwubcode]

@Pokechu22 - if you have time, would you mind reviewing this?

My only concern was destination alpha but I tested that on another game just now and it seems to operate appropriately.

[iwubcode]

@Pokechu22 - tested smg2 shadows. There was a code change needed as when destAlpha is set, the values are swapped, without it...the shadow never change intensity.

After the change, seems to work.

vec4 custom_main( in CustomShaderData data )
{
	float4 color = data.final_color;
	return color;
}

yields

vec4 custom_main( in CustomShaderData data )
{
	float4 color = data.final_color;
	color.a *= 0.5;
	return color;
}

yields

while

vec4 custom_main( in CustomShaderData data )
{
	float4 color = data.final_color;
	color.a *= 0.25;
	return color;
}

Re tested Xenoblade and Tales of Symphonia tests. Blending is controllable by the .a value as expected

[Pokechu22]

The main thing that would be useful to verify is if the first shader (which should leave the game's settings output unchanged) actually leaves output unchanged. MeltyMoltenGalaxy.dff is a good test case for this, as it has both a shadow and a small bit of of Luigi is behind a wall, both of which use constant alpha (and use different values for it in different situations).

... though, it doesn't look like they actually have blending set up in a way that matters; for the shadows/obscured luigi, it seems like they are just writing a constant value to the alpha channel and not doing any blending, and when actually applying it they use the blending configuration to set the alpha value in the framebuffer to 1 and use the output alpha for blending. So at least as far as I can tell, this isn't that interesting of a use of constant alpha.

Here's what libogc has to say about it, just for context:

 * \fn void GX_SetDstAlpha(u8 enable,u8 a)
 * \brief Sets a constant alpha value for writing to the Embedded Frame Buffer (EFB).
 *
 * \note To be effective, the EFB pixel type must have an alpha channel (see GX_SetPixelFmt()). The alpha compare operation (see
 * GX_SetAlphaCompare()) and blending operations (see GX_SetBlendMode()) still use source alpha (output from the last TEV stage) but when
 * writing the pixel color, the constant alpha will replace the pixel alpha in the EFB.

[iwubcode]

@Pokechu22 - thanks for your comment and for spending your time on this. After you commented, I added support to my editor to toggle all mods on/off and noticed that the shadows were indeed not matching vanilla output as you questioned.

I spent a while reading about blending after that and realized dual source is a very valid thing in graphics (not just some quirk of GC/Wii). I knew then that I was trying to fit a square peg into a round hole so to speak :)

So I've made a few changes:

• Custom shaders now apply after blending is finished, as they operate on the final color from Dolphin.
• Instead of returning a single output color, the shader is now intended to return a CustomShaderOutput structure. This structure provides output for the initial source and if dual source is enabled, it takes a second source which it uses for the final value. I want to eventually support multiple render targets, so I've named this main_rt_src0 and main_rt_src1 to convey that.
• I also pass in the second final output color (again if dual source is enabled) to the user

This gives the user complete control over the resulting output and yields correct behavior by simply passing the input values through.

One thing that needs to be improved in the future, is a way to re-apply the blend operation functionality. This might be useful in conditions where you are trying to replace a single color in the TEV chain but want to keep everything else the same. Switching the entire custom shader output to before WriteColor would give a user this capability but then they wouldn't be able to replace the blending if desired. Ideally all our Dolphin functionality would be put into callable functions that custom shaders could use but that currently isn't the case.

Anyway, let me know what you think!

[Pokechu22]

Custom shaders now apply after blending is finished, as they operate on the final color from Dolphin.

I don't think that's going to work super well. In most cases blending doesn't happen in the shaders in dolphin - it's only done there when using framebuffer fetch for blending, and that's not super well supported on desktop platforms (Vulkan doesn't support it at all, though it does work with MoltenVK, and the only desktop OpenGL GPU I know of that supports it is intel's integrated GPU). My understanding is that we mainly use it to work around cases where fixed-function blending doesn't work the way we want it to (e.g. bad driver behavior). You'd need to provide a way for custom shaders to change that fixed-function behavior too.

Note that uid_data->blend_enable has a comment saying "Only used with shader_framebuffer_fetch blend"; it's not a direct reflection of whether blending is enabled by the game. See the code around here.

I recommend testing this out on a Nvidia GPU and see if you get the expected behavior.

I added tracking for support of fbfetch to analytics in #11396, but I'm not sure where exactly to see that data.

[iwubcode]

I don't think that's going to work super well. In most cases blending doesn't happen in the shaders in dolphin

Not sure why not.

As for frame buffer fetch, I don't pretend to understand the hardware ramifications. I know Macs support frame buffer fetch but I do not have the hardware to test that.

I will see if someone with that hardware can take a look but I'm not expecting any issues as the code can hit either path (framebuffer fetch on or off) and provides the appropriate functionality regardless (return one or two colors).

Note that uid_data->blend_enable has a comment saying "Only used with shader_framebuffer_fetch blend"; it's not a direct reflection of whether blending is enabled by the game. See the code around here.

I handle both if blending is set and if it isn't, providing the same functionality regardless. This shader functionality isn't intended to force blending on for a draw call that doesn't provide blending, you can only use it on draw calls that already are doing some blending.

I recommend testing this out on a Nvidia GPU and see if you get the expected behavior.

All my testing is on a Nvidia GPU.

[TellowKrinkle]

Framebuffer fetch is used to emulate hardware blending, which is otherwise done in hardware after the pixel shader runs.

If you want results to match between fbfetch and non-fbfetch, you should run your custom shader before WriteLogicOp and WriteBlend, as those are what apply blending, which is required to happen after the pixel shader on non-fbfetch GPUs.

[Pokechu22]

Generally, Dolphin uses the graphics API's blending functionality for blending. For instance, if you look at the yellow triangle on the minimap in mario kart wii (mkwii-bluebox / dff), you can see the background through it. But, the shader outputs a consistent yellow value of (232, 212, 0) for the entire triangle, only changing the alpha value; the shader doesn't know what's in the background at all. Instead, the blending configuration is set up so that the new framebuffer content is alpha * shader output + (1 - alpha) * existing framebuffer content (i.e. src factor is src alpha and dst factor is 1 - src alpha). The benefit of this approach is that you can run the shader without needing to wait until you know the existing framebuffer content (and I think there are other hardware accelerations used too).

When framebuffer fetch is used, the graphics API's blending is disabled. Instead, the shader is passed the existing framebuffer content, does the alpha * shader output + (1 - alpha) * existing framebuffer math itself, and then outputs that new color that's written directly to the framebuffer. This is slower but gives a lot more flexibility, and can work around driver quirks.

You currently have custom shaders take the output of either of those two processes as input, but those two values have completely different meanings and behaviors, so custom shaders would behave differently depending on if framebuffer fetch is enabled or not. For instance, if your custom shader set the blue value to 0 but left red, green, and alpha unchanged, the yellow triangle in Mario Kart Wii would be unchanged with the normal blending path, but would look different with the framebuffer fetch path.

Another interesting test case is MarioSunshineWater.dff - the way the skyboxes in Super Mario Sunshine work is pretty weird (it renders in greyscale first and then uses an odd blending configuration to add color), and the water itself also uses blending.

If you also have an integrated intel GPU in addition to the Nvidia GPU and are on Windows, you can force the fbfetch path by searching for "graphics settings" in the new windows settings app, then adding a performance preference for dolphin.exe (make sure to select your development version), and then select "power saving" (this changes the GPU used by OpenGL, which you can't do directly within Dolphin). Then run Dolphin with OpenGL and exclusive ubershaders. Ubershaders use framebuffer fetch if it's available so that one shader can be used regardless of the blend mode.

[iwubcode]

Thank you both for your comments.

As mentioned on Discord, I think the current code seems to merge blending and non-blending in a way that makes it hard to do this change effectively (WriteColor does some blending logic, WriteLogicOp / WriteLogicOpBlend comes before WriteColor which I need to replace the color, etc).

However, I think I now have a somewhat better understanding of framebuffer fetch and why doing things in the current way will yield differing results depending on hardware support.

I will try and decouple and shift some code around to make it easier to apply my changes. The way to reproduce frame buffer fetch may also be helpful (thanks Pokehcu!).

[iwubcode]

After discussing more with Tellow, I realized I was going about it all wrong and that's why everything was complicated.

I think Pokechu hinted at it earlier as well but I should have been operating on prev (or TevResult in the ubershader) which is the color operation (including alpha!) after the tev stage is finished. That color is then passed to the rest of the functionality to split the color across the hardware outputs (if using dual source) and properly writes the blending result if hardware blending is unavailable.

I did keep the CustomShaderOutput struct, even though it's not strictly necessary anymore but figured the breaking change was good here as it was anywhere else.

Given the changes don't directly interface with the framebuffer fetch, I did not test that. But all previous tests (SMG2, Xenoblade, ToS outlines) seem to be working on my NVidia GPU. I did test some logic op behavior as well, though I didn't do anything too extensive (Opoona and FZero shadows)

[iwubcode]

@Pokechu22 / @TellowKrinkle - have you had a chance to look at the updated code? What do you think?