Add networked hand tracking component

[avdynut]

Hi, thanks for the networked component.
We're adding it to the XR# CommunityToolkit. So multi-user experience would quickly be available for C# developers.

It would be really nice to have hand-tracking shared between users.
Here is a simple component that adds networked attribute for every hand joint. Only modelStyle: dots is supported.

Please check example project: https://glitch.com/edit/#!/naf-hand-tracking

Related issue #394

[vincentfretin]

Thank you for the PR. When we looked at how to network hands last time with @Utopiah I didn't pay attention that the dots model was creating aframe entities, so indeed we can networked those easily.

I left some comments. I didn't test that myself though. @Utopiah you may be interested to testing it and give feedback.

If we want it to be merged, we'll need some documentation in the README and an example.

But we probably want a component that works with the hand models as well.
Performance wise, the implementation here is using 25 networked entities for a hand, that may work good enough for two users or three in the room, but I guess we could have perf issues quickly with more users.
What would be far better is having a component syncing directly the two Float32Array jointPoses and jointRadii, naf doesn't support binary transfer, so it will still be needed to serialize those to an Array to serialize in json unfortunately.

[vincentfretin]

I don't see the hand-joint-template in your PR. What did you use for html template?
Answering this myself looking at your glitch:

<template id="hand-joint-template">
  <a-sphere radius="0.01"></a-sphere>
</template>

What about the networked schema? Default networked schema is only syncing position and rotation, did you add scale? I see hand-tracking-controls is setting scale on the joints https://github.com/aframevr/aframe/blob/e0c4ec6d2d07db426d27d76b32ec7aba7d9f5e60/src/components/hand-tracking-controls.js#L250
You probably want to create your own schema that sync position rotation scale with a correct requiresNetworkUpdate function to minimize networked updates, see
https://github.com/networked-aframe/networked-aframe#tracked-controllers-w-synced-gestures

          <a-entity id="left" hand-tracking-controls="hand:left; modelStyle:dots;" networked-hand-tracking="{options: 'template:#hand-joint-template; attachTemplateToLocal:false'}"></a-entity>

I didn't know we could use an object like that as attribute value, this is really working?

[vincentfretin]

I don't think this should be the responsibility of your component to empty the array of another component.
Why this line? It doesn't seem to be needed to me.

[vincentfretin]

Doing a setAttribute in tick for all joints are not efficient. You should really do it only once in an enter-vr listener I think.

[Utopiah]

@avdynut do you have a live Glitch to test?

Using the dots model isn't ideal and yes networking it all this way is too costly. Yet, I'd argue at this point something inefficient yet functional with known limits, e.g 2 participants only, is better than nothing, as is currently the case.

I'd suggest to accept the PR with minimal change and clear documentation and keep the improved version (hand model and Float32Array) for a subsequent PR.

[MusicStudioNYC]

How about the following considerations to make it easier/lighter to network so much data:

1. Quantize the data
2. Minimize which joints you network
3. Only send update if movement exceeds a threshold
4. Differential/delta encoding

Then perhaps once per second or two, send a full "keyframe" without the above optimizations to catch for any glitches from 2, 3 or 4.

Additionally, consider an adaptive approach in which the extent of optimization depends on the amount of hands being tracked. So a 2 person event will track every joint with full accuracy etc, while a 50 person event will be closer to just tracking spheres...