nv2a: Offset vertices to match HW rounding

[antangelo]

This is a continuation of the work done in #735 . Hardware will round vertices at 0.5625 instead of OpenGL's 0.5.

The approach taken in this PR shifts each vertex up and to the left by 0.0625, so that the HW rounding point 0.5625 is mapped to 0.5. A small bias is then added based on whether the subpixel was above or below the boundary, to correct for precision errors. This passes all but one of the test cases that abaire made (specifically, the projected adjacent geometry test with a bias of 0.5625 is not entirely matching HW), which is why I'm leaving it in draft for now.

This approach does not work at resolution scales higher than 1x. I'm not sure that there is a general solution that both works above 1x and does not introduce z-fighting or flickering.

[Fabxx]

tested on these titles at 1x, there are also no issues with the shadows being cut/clipped based on the camera angle:

SC:DA

SC:

[abaire]

As an FYI, the place where I eventually got stuck was in Tiger Woods where a single screen uses a mix of the fixed function and programmable pipelines. The difference in the way that vertexes are unprojected leads to twinkling seams between polys that were rendered via dissimilar paths. I think my test cases were SC:DA (and/or Spongebob) and Tiger Woods 2005 during the course flyover sequence.

I think the ProjAdjacentGeometry test is the important one: https://github.com/abaire/nxdk_pgraph_tests/blob/main/src/tests/vertex_shader_rounding_tests.cpp#L710

[antangelo]

As an FYI, the place where I eventually got stuck was in Tiger Woods where a single screen uses a mix of the fixed function and programmable pipelines. The difference in the way that vertexes are unprojected leads to twinkling seams between polys that were rendered via dissimilar paths. I think my test cases were SC:DA (and/or Spongebob) and Tiger Woods 2005 during the course flyover sequence.

I think the ProjAdjacentGeometry test is the important one: https://github.com/abaire/nxdk_pgraph_tests/blob/main/src/tests/vertex_shader_rounding_tests.cpp#L710

Thanks for the context, I'll see about picking up Tiger Woods as another test case. The only ProjAdjacentGeometry test case that this code fails is the one with 0.5625 bias. What's interesting is that HW rounds the leftmost dark square off one pixel to the right, but not the other three. The behavior in this PR rounds all four of them off to the right.

[abaire]

Hopefully it'll turn out that your fix is sufficient and I created the problem in Tiger Woods while chasing down other issues :)

I believe @Triticum0 found the issue w/ TW while testing my PR and can probably say definitively which version is problematic (though I suspect they used the same engine year after year so it might not matter). I'm away from my HW for an indeterminate amount of time and unfortunately I don't appear to have left a comment tracking the specific version I was testing, sorry!

[ajalberd]

Can confirm this works with Splinter Cell: Double Agent, removing the blue/green lines at the top and left.

[antangelo]

There seem to be a few small artifacts with this PR vs master in tiger woods 2005 in the form of white dots that flash in and out.

tigerwoods-comparison.mp4

[abaire]

There seem to be a few small artifacts with this PR vs master in tiger woods 2005 in the form of white dots that flash in and out.

Exactly, I believe that's the background color showing through the gaps caused by (presumably numerical precision induced) differences between the fixed and programmable pipelines.

Before I had to step back from this, I was beginning to consider simply merging the fixed and programmable paths by defining one or more static nv2a shaders to reproduce the fixed pipeline behavior. I hoped that would eliminate this issue and would reduce/remove the potential for other cases where behavior unintentionally ends up differing between the two paths.

[abaire]

Assuming you're able to fix the mismatch between fixed & programmable: I think it's worth bringing over some of the comments explaining why this is necessary, particularly the magic numbers involved. I don't think any of this would make sense to a future maintainer without understanding the context of the nv2a-specific rounding behavior, which differs from the generally well known/documented 0.5 delta between old DirectX and OpenGL.

I also don't know if it's actually possible to address higher render scales with a simple offsetting approach. 2x and higher will turn garbage that would've been truncated to 0 into a full pixel or more. I had started to look at using a viewport to handle the 9/16th rounding instead but I think I stopped on realizing that glViewportIndexedf requires OGL 4.1.

[NZJenkins]

FWIW I get different results on HW than the gold results.
Screenshot with only change being printing out the top-left x coordinate:

If you're comparing to golden results, did you check your build of the xbe behaves the same on HW?

Also, games tend to add 0.03125 (1/32) to positions
e.g. when Fable TLC draws bloom over the screen it generates positions like this:

v0 IN: (-1, -1), (1, -1), (-1, 1), ...
v0 after xbox VS program: (0.53125, 480.53125), (640.53125,  480.53125), (0.53125,  0.53125), ...

exactly halfway between 0.5 and the threshold 0.5625 (1/16).
If they are meant to be at the pixel centre 0.5, maybe coordinates get floored to the nearest 1/16, rather than needing 1/16 subtracted?

[antangelo]

That's odd, the hardware I had been testing on matches the golden results. Your output is interesting because at a glance, the upper programmable pipeline squares round up along the y-axis and all fixed squares round down, but there are no gaps in between the programmable and fixed squares like would be expected.

Flooring to the nearest 1/16 is an interesting idea, it's worth trying. Doing that alone doesn't fully address the issue though, because we would still need to handle differing rounding behavior at 8/16.
I'm not sure what exactly adding 1/32 to the positions is doing, as far as I understand 0.5 and 0.53125 would both round down on HW. If I recall right, Spongebob offsets positions similarly. Given that your HW and mine differ, there might be something more to it.