Add support for bindless textures from storage buffer on Vulkan

[gdkchan]

This has two changes. First, it has the change on #5936:

When the API or driver does not have native native support for quad primitives, we convert them to triangles. Because we convert each quad to 2 triangles, when we render quads, we have twice as many primitives than the game submitted. For this reason, the value for gl_PrimitiveID which is the index of the primitive on the shader is incorrect, and it is also double of what it should be.

To fix this issue, this change divides the ID by 2 if the primitive type is Quads or QuadStrip, the host does not support quads, and there is no geometry shader (as geometry shader must always generate new primitives, and they can't output quads).

And the second changes enables "full bindless" to also work when the handle comes from a storage buffer. Note that this still uses the code path where it loads all textures on the pool, rather than something that only loads the handles on the storage buffer.

This fixes some of the remaining issues that was originally fixed on the "full bindless" PR.

Castle Crashers Remastered.
Before:

After:

Mario Party Superstars.
Before:

After:

I tested this on the games that had massive performance regression when I enabled "full bindless" for everything, and none of them had performance regression with this change. Regardless, wider testing on UE4 games is welcome, since the number of games that I tested is low.

[github-actions]

Download the artifacts for this pull request:

• ryujinx-Release-1.1.0+c6df23b-linux_arm64
• ryujinx-Release-1.1.0+c6df23b-linux_x64
• ryujinx-Release-1.1.0+c6df23b-macos_universal
• ryujinx-Release-1.1.0+c6df23b-win_x64

Old GUI (GTK3)

• gtk-ryujinx-Release-1.1.0+c6df23b-linux_arm64
• gtk-ryujinx-Release-1.1.0+c6df23b-linux_x64
• gtk-ryujinx-Release-1.1.0+c6df23b-win_x64

GUI-less (SDL2)

• sdl2-ryujinx-headless-Release-1.1.0+c6df23b-linux_arm64
• sdl2-ryujinx-headless-Release-1.1.0+c6df23b-linux_x64
• sdl2-ryujinx-headless-Release-1.1.0+c6df23b-macos_universal
• sdl2-ryujinx-headless-Release-1.1.0+c6df23b-win_x64

Only for Developers

• ryujinx-Debug-1.1.0+c6df23b-linux_arm64
• sdl2-ryujinx-headless-Debug-1.1.0+c6df23b-linux_arm64
• gtk-ryujinx-Debug-1.1.0+c6df23b-linux_arm64
• ryujinx-Debug-1.1.0+c6df23b-linux_x64
• sdl2-ryujinx-headless-Debug-1.1.0+c6df23b-linux_x64
• gtk-ryujinx-Debug-1.1.0+c6df23b-linux_x64
• ryujinx-Debug-1.1.0+c6df23b-macos_universal
• sdl2-ryujinx-headless-Debug-1.1.0+c6df23b-macos_universal
• ryujinx-Debug-1.1.0+c6df23b-win_x64
• sdl2-ryujinx-headless-Debug-1.1.0+c6df23b-win_x64
• gtk-ryujinx-Debug-1.1.0+c6df23b-win_x64

[riperiperi]

Looks good, does fix those issues in superstars, and I imagine it'll help get crazier cases of using bindless textures working at a bare minimum.

This triggers every frame in clubhouse games (guest pass) with pool sizes of 4352 and 2304, but I can't see any visual difference. Same with Mario Party Superstars with similar pool sizes... I guess there's something that needs it, but I can't see it. Obviously superstars does need it when it's drawing shadows in some minigames, so maybe it's just something that runs all the time.

This could result in higher memory and CPU usage for these games, so it might be worth looking into reducing it to just use the handles on the storage buffer in future. Might not be feasible if they're written from GPU... I was a bit worried about resolution scale resets forcing the game to 1x resolution, but I didn't see any when playing a bunch of minigames.

MacOS in particular has huge performance issues in Mario Party Superstars now in most scenes. I imagine MoltenVK isn't happy with hundreds of potential bindings for each draw.

[legend800]

Just tried Super Mario Party on Vulkan but it crashes now post the Pad select screen. Log attached. Error:
"GUI.RenderThread Application : Unhandled exception caught: Ryujinx.Graphics.Vulkan.VulkanException: Unexpected API error "ErrorOutOfDeviceMemory".

Works fine on OGL but obviously missing shadows in many games. Tried vsync on/off and Memory mode=software/fast, clearing all caches but always crashes at that spot.

Config:
Windows 11 64-bit
GeForce GTX 3070
AMD Ryzen 5900x
32GB RAM

Log.txt

[MetrosexualGarbodor]

@legend800 this is completely unrelated to this PR, see #6478