(Vulkan) Implement HDR viewport readback

[CRefice]

Description

This PR adds HDR support for screenshots (aka viewport reading, aka readback) to the Vulkan video driver.
This is implemented on the GPU by running a pipeline which converts the HDR10 pixels in the backbuffer to BGRA8 format through a fragment shader which also applies tonemapping if necessary. The resulting image buffer is saved to an image using already-existing scaler logic for SDR screenshots.

If the currently active filter chain does not emit HDR10, the driver already applies SDR->HDR inverse tonemapping to the image as a final pass before submitting it to the swapchain. I thought it would be wasteful to then immediately tonemap the image back to the original result when taking a screenshot, so in these instances the original linear-space SDR image is used as a source for format conversion, and tonemapping is skipped.

Some light refactoring of HDR-relevant code is included in this PR for convenience of implementation. Embedded HDR shaders have also been refactored so that (inverse) tonemapping and related HDR utilities are all defined in one shader file, hdr_common.glsl, which is then #included by both the tonemapping and inverse tonemapping fragment shaders. I also explicitly enabled the VK_EXT_swapchain_colorspace extension, as the Vulkan validation layers were complaining about undefined constants on swapchain creation.

No explicit reference is made in the code to specific swapchain formats such as A2R10G10B10 or A2B10G10R10, so the implementation should be format-agnostic, but so far I only tested it on my machine which supports A2R10G10B10 only.

Screenshots

The following screenshots are of the SMPTE Color Bars from the SNES 240p test ROM. Click to zoom on both as these are 4k screenshots and (especially the shader one) will show artifacts when scaled.

No shaders active, reading directly from SDR buffer:

Using the crt-sony-megatron-default-hdr shader, which emits HDR10 colors:

Related Pull Requests

#16435: Original PR in which this feature was requested.

Reviewers

@warmenhoven @MajorPainTheCactus

[hizzlekizzle]

Sounds great!

I have an irrational fear of copyright lawyers

You can always use homebrew, like the 240p test suite, which we have available for many cores via the online updater's content downloader.

[LibretroAdmin]

I also explicitly enabled the VK_EXT_swapchain_colorspace extension, as the Vulkan validation layers were complaining about undefined constants on swapchain creation.

Cool, I noticed that this happened sometime ago and that this was missing from the initial Vulkan HDR implementation indeed.

[LibretroAdmin]

One quick question about VK_EXT_swapchain_colorspace, when this extension is not supported, I assume the current PR just disables HDR support instead of the driver not being able to be used right? As in it's an optional extension, we still want users to be able to use the Vulkan driver if they're on a GPU driver without this extension, we can just deal with it then by not enabling HDR codepath.

[warmenhoven]

Right now the PR unmodified crashes on macOS at startup in vkQueueSubmit just after printing this error:

-[MTLDebugRenderCommandEncoder setRenderPipelineState:]:1615: failed assertion `Set Render Pipeline State Validation
For color attachment 0, the render pipeline's pixelFormat (MTLPixelFormatBGRA8Unorm) does not match the framebuffer's pixelFormat (MTLPixelFormatRGB10A2Unorm).

I'll look into why. (Just to rule things out, VK_EXT_swapchain_colorspace is supported and the VkInstance gets created with it enabled.)

[CRefice]

You can always use homebrew, like the 240p test suite, which we have available for many cores via the online updater's content downloader.

Great point! I've updated the PR description

[CRefice]

One quick question about VK_EXT_swapchain_colorspace, when this extension is not supported, I assume the current PR just disables HDR support instead of the driver not being able to be used right? As in it's an optional extension, we still want users to be able to use the Vulkan driver if they're on a GPU driver without this extension, we can just deal with it then by not enabling HDR codepath.

That would be the correct behavior, yes. The current logic does not check if the extension is supported, but I'll make it work

[CRefice]

Right now the PR unmodified crashes on macOS at startup in vkQueueSubmit just after printing this error:

-[MTLDebugRenderCommandEncoder setRenderPipelineState:]:1615: failed assertion `Set Render Pipeline State Validation
For color attachment 0, the render pipeline's pixelFormat (MTLPixelFormatBGRA8Unorm) does not match the framebuffer's pixelFormat (MTLPixelFormatRGB10A2Unorm).

Maybe try ending the command buffer and starting a new one just for the readback pipeline? Could be there's some state that isn't flushed until queue submit.

[CRefice]

One quick question about VK_EXT_swapchain_colorspace, when this extension is not supported, I assume the current PR just disables HDR support instead of the driver not being able to be used right? As in it's an optional extension, we still want users to be able to use the Vulkan driver if they're on a GPU driver without this extension, we can just deal with it then by not enabling HDR codepath.

That would be the correct behavior, yes. The current logic does not check if the extension is supported, but I'll make it work

Done. I added an extra context flag for HDR support, and gated all HDR-related functionality behind it. As a bonus, the driver will no longer attempt to reinitialize HDR pipelines every frame if HDR is enabled in the settings but disabled in the OS, something which I noticed could cause a lot of slowdown.

[LibretroAdmin]

Thanks. So I think the current issue is the issues on Mac with MoltenVK, not sure how to really proceed there. I assume the other platforms are fine so far.

[CRefice]

I think I might have caught the issue with MoltenVK. When creating pipelines, the VkGraphicsPipelineCreateInfo pipe is reused to initialize all pipelines, but I forgot to restore pipe's render pass after creating the hdr readback pipeline. (Not sure how this wasn't caught by validation layers tbh.) @warmenhoven please try again 🤞🏼

[warmenhoven]

It works!