Hand Mesh Ext Issues

[maluoi]

Description

When trying to generate UVs for the hand mesh, StereoKit runs into a few problems that can cause the hand mesh to look "spotty" or completely transparent black, depending on the platform and situation.

There's a pair of problems here:

1. HoloLens 2 uses a reference mesh pose where the fingers extend in the Y+ direction, while Snapdragon Spaces uses a reference mesh pose that extends in the Z- direction. There may be some additional transform SK isn't using that will align these? Z- breaks SK's Y+ assumption, causing all hand verts to get a UV that ends up sampling a transparent black color.
2. HoloLens 2 may not actually be providing a fixed reference pose? If the hand is held as a fist when UV calculation occurs, the fingertips end up close to the origin, and end up in the transparent black part of the texture. It's also possible StereoKit may be incorrectly requesting the reference pose.

[LarryAultman]

@Maloui Does this affect the way HoloLens 2 detects the SK radius menu, the one in the palm of the hand? It is fairly difficult to activate reliably. At least for me it is.

[maluoi]

Hand mesh extension is purely a visual feature.

[austinbhale]

For the HL2, I'm having trouble activating the articulated hand mesh in DemoHands.cs the first time with a fairly specific repro, so I'm wondering if this is related:

1. As the demos initialize, I set hand visibility to false: Input.HandVisible(Handed.Max, false);
2. Start the camera tool to RenderTo a texture
3. When calling ColorizeFingers, set the hand visibility to true. But they don't appear while RenderTo is still active

8mb.video-Lmm-vHc686GY.mp4

[maluoi]

This is so remarkably specific in such a weird way, I imagine this has to be something interesting one way or another! :D I'm out at MIT Reality Hack for the rest of the week but this looks like a fun one to peek at when I get back!

[austinbhale]

Woa, if there are some people working with SK for their projects, I'd love to see them! Have fun :)

Thanks for looking at it! Here's a smaller script example that reproduces it:

namespace Sample.UWP;

using System;
using StereoKit;

internal class Program
{
    private static void Main(string[] _)
    {
        SKSettings settings = new()
        {
            appName = "Sample.UWP",
            assetsFolder = "Assets",
        };
        if (!SK.Initialize(settings))
        {
            Environment.Exit(1);
        }

        // Assemble the window pose
        Pose head = Input.Head;
        Vec3 dirForward = 0.4f * head.Forward.X0Z.Normalized;
        Vec3 windowPosition = head.position + dirForward;
        var windowRotation = Quat.LookAt(windowPosition.X0Z, head.position.X0Z);
        Pose windowPose = new(windowPosition, windowRotation);

        // Assemble the render target
        bool isRendering = true;
        Tex frameSurface = new(TexType.Rendertarget, TexFormat.Rgba32);
        frameSurface.SetSize(500, 500);
        frameSurface.AddZBuffer(TexFormat.Depth32);

        // Hand visibility
        bool handsVisible = false;
        Input.HandVisible(Handed.Max, false);

        SK.Run(() =>
        {
            UI.WindowBegin("Colorize Fingers", ref windowPose);

            if (UI.Button("Red"))
            {
                Input.HandVisible(Handed.Max, true);
                handsVisible = true;
                ColorizeFingers(Default.MaterialHand,
                    16,
                    true,
                    new Gradient(new GradientKey(new Color(1, 1, 1), 1)),
                    new Gradient(
                        new GradientKey(new Color(0, 0, 0, 0), 0),
                        new GradientKey(new Color(1, 0, 0), 0.4f),
                        new GradientKey(new Color(1, 0, 0), 0.6f),
                        new GradientKey(new Color(1, 0, 0), 0.9f)));
            }

            if (UI.Button("Normal"))
            {
                Input.HandVisible(Handed.Max, true);
                handsVisible = true;
                ColorizeFingers(Default.MaterialHand,
                    16,
                    true,
                    new Gradient(new GradientKey(new Color(1, 1, 1, 1), 1)),
                    new Gradient(
                        new GradientKey(new Color(.4f, .4f, .4f, 0), 0),
                        new GradientKey(new Color(.6f, .6f, .6f, 0), 0.4f),
                        new GradientKey(new Color(.8f, .8f, .8f, 1), 0.55f),
                        new GradientKey(new Color(1, 1, 1, 1), 1)));
            }

            UI.Toggle("RenderTo", ref isRendering);
            if (isRendering)
            {
                Renderer.RenderTo(frameSurface, Input.Head.ToMatrix(), Matrix.Perspective(45, 1, 0.01f, 100));
            }

            if (UI.Toggle("HandsVisible", ref handsVisible))
            {
                Input.HandVisible(Handed.Max, handsVisible);
            }

            if (UI.Button("Exit"))
            {
                SK.Quit();
            }

            UI.WindowEnd();
        });
    }

    private static void ColorizeFingers(Material handMaterial, int size, bool transparent, Gradient horizontal, Gradient vertical)
    {
        Tex tex = new(TexType.Image, TexFormat.Rgba32Linear)
        {
            AddressMode = TexAddress.Clamp,
        };

        var pixels = new Color32[size * size];
        for (int y = 0; y < size; y++)
        {
            Color v = vertical.Get(1 - y / (size - 1.0f));
            for (int x = 0; x < size; x++)
            {
                Color h = horizontal.Get(x / (size - 1.0f));
                pixels[x + y * size] = v * h;
            }
        }

        tex.SetColors(size, size, pixels);

        handMaterial[MatParamName.DiffuseTex] = tex;
        handMaterial.Transparency = transparent
            ? Transparency.Blend
            : Transparency.None;
    }
}

[austinbhale]

Hey nick! Just bumping this issue if you get a chance to try the script out. Thank you :)