Add body tracking sample #178

cabanier · 2024-01-24T22:47:02Z

FYI @toji
Not ready for merging yet.
example video:
https://github.com/immersive-web/webxr-samples/assets/1513308/6e5227c6-d451-4846-a0a2-6607bfcce627

Maksims · 2024-01-25T07:04:23Z

This is amazing!
I had to implement upper body IK multiple times, and it is not an easy task, and falls short when estimating shoulder twist.
Also legs! Yes please!

toji

As you said, this is just an FYI for now but I've got some comments for you anyway!

toji · 2024-01-29T19:45:18Z

proposals/body-tracking.html

+
+    <meta http-equiv="origin-trial" content="Ahfj+MLeL6bh+LNmpnSdepftxoDHHwjUG2KWZ4jjCb1WoZxtBlzF3cDHuJNVqnhr3HXJwQ+kLaw57NO15S0mRwwAAABkeyJvcmlnaW4iOiJodHRwczovL2ltbWVyc2l2ZS13ZWIuZ2l0aHViLmlvOjQ0MyIsImZlYXR1cmUiOiJXZWJYUlBsYW5lRGV0ZWN0aW9uIiwiZXhwaXJ5IjoxNjI5ODQ5NTk5fQ==">
+
+    <title>AR Plane Detection</title>


Update title

toji · 2024-01-29T19:46:55Z

proposals/body-tracking.html

+				scene = new THREE.Scene();
+				scene.background = new THREE.Color( 0x505050 );
+
+				camera = new THREE.PerspectiveCamera( 50, window.innerWidth / window.innerHeight, 0.1, 50 );


Indentation is kind of all over the place in this sample. Seems like you've probably got your editor set to tabs and these files use spaces? It'd be appreciated if you could normalize it.

toji · 2024-01-29T19:50:03Z

proposals/body-tracking.html

+        document.querySelector('header').appendChild(xrButton.domElement);
+
+        if (navigator.xr) {
+          navigator.xr.isSessionSupported('immersive-ar')


Is this functionality only available in AR on Quest? If not, is there any way can we make this demo work in either AR or VR?

@toji I am very curious about this too.

@cabanier Are you using some sort of polyfill? Or the API is giving you that information?

@toji I am very curious about this too.

no, it can also be used in VR

@cabanier Are you using some sort of polyfill? Or the API is giving you that information?

What do you mean? a polyfill for???

@cabanier Sorry, let me rephrase. Is the WebXR implementation in the browser providing you all the information for the body tracking? Using the body-tracking feature? Or you are generating this information somewhere else and injecting in the browser? I am asking because body-tracking is not officially described/supported by the WebXR specification.

I am just very curious to understand where the bones positions/rotations comes from.

@cabanier Sorry, let me rephrase. Is the WebXR implementation in the browser providing you all the information for the body tracking? Using the body-tracking feature? Or you are generating this information somewhere else and injecting in the browser?

It's a new extension to WebXR that I'm proposing: https://cabanier.github.io/webxr-body-tracking/
It uses Meta's body tracking OpenXR implementation under the hood.

@cabanier Thanks for explaining. Nice work, I hope it's accepted! 🤞

toji · 2024-01-29T19:52:43Z

proposals/index.html

@@ -132,6 +132,9 @@ <h2 class='tagline'>Proposals</h2>
              description: 'Demonstrates use of the mesh detection API in an immersive-ar session. ' +
                           'Implements JavaScript-level hit-test on the meshes and leverages the Anchors API.' },

+            { title: 'Body tracking', category: 'AR',


Similar to my question above, doesn't seem like this is an AR-only feature? So I'd recommend using a category other than "AR" here.

Go ahead and make up something that feels appropriate. You can see from the Test Pages tab that we don't exactly stick to a rigid system here. I just don't want to give the impression that it's limited to a given session type unless it actually is.

Correct, just like hands this will also work in VR. I'll code something up

toji · 2024-01-29T19:57:43Z

proposals/body-tracking.html

+              const material = new THREE.MeshLambertMaterial();
+
+              spheres = new THREE.InstancedMesh( geometry, material, body.size );
+              spheres.translateZ( -1 ).setRotationFromMatrix (new THREE.Matrix4().makeRotationY(Math.PI));


I'm not clear on how this works, and it's probably worth a comment for developers sake.

My guess is that this is a transform applied to the entire collection of instances spheres as a sort of root transform? If so, how does this result in the behavior in the video you linked, which appears to mirror your movements rather than show them rotated 180? (Although to be fair you never really moved your hands one at a time, so it's hard to tell. Full body movement suggests a mirror instead of a rotation, though.)

In any case, more comments throughout would be welcome!

toji · 2024-01-29T19:59:52Z

proposals/body-tracking.html

+                matrix.identity();
+              }
+
+              matrix.setPosition( -position.x, position.y, position.z );


Oh, and I see that the positions are being inverted in X here too? Is THAT what causes the mirroring effect?

toji · 2024-01-29T20:00:29Z

proposals/body-tracking.html

+              const position = pose.transform.position;
+
+              if (!part.jointName.includes("hand")) {
+                matrix.copy(scalehand);


Why do hands need to be scaled up 3x? In your video it looks like the opposite, and the hands are scaled down?

Oh, wait, I see. This is checking to ensure that "hand" isn't part of the name. That's pretty confusing. Can you invert the logic here?

toji · 2024-01-29T20:03:25Z

proposals/body-tracking.html

+            let space = renderer.xr.getReferenceSpace();
+            body.forEach(part => {
+
+              const pose = frame.getPose(part, space);


Do we want to encourage people to use fillPoses() here? Or does that introduce too many complications for an otherwise simple sample.

yeah. I forgot about that call :-)
I will update the code

Add body tracking sample

bd95a77

cabanier requested a review from toji January 24, 2024 22:47

yuripourre mentioned this pull request Jan 25, 2024

Does WebXR support full body tracking? immersive-web/webxr#1247

Closed

toji reviewed Jan 29, 2024

View reviewed changes

rev three.js version

795d0cc

cabanier force-pushed the body-tracking branch from c591ea1 to 795d0cc Compare February 9, 2024 00:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add body tracking sample #178

Add body tracking sample #178

cabanier commented Jan 24, 2024

Maksims commented Jan 25, 2024

toji left a comment

toji Jan 29, 2024

toji Jan 29, 2024

toji Jan 29, 2024

yuripourre Jan 29, 2024

cabanier Jan 30, 2024

yuripourre Jan 30, 2024

cabanier Feb 12, 2024

yuripourre Feb 12, 2024

toji Jan 29, 2024

cabanier Jan 30, 2024

toji Jan 29, 2024

toji Jan 29, 2024

toji Jan 29, 2024

toji Jan 29, 2024

toji Jan 29, 2024

cabanier Jan 30, 2024


		<meta http-equiv="origin-trial" content="Ahfj+MLeL6bh+LNmpnSdepftxoDHHwjUG2KWZ4jjCb1WoZxtBlzF3cDHuJNVqnhr3HXJwQ+kLaw57NO15S0mRwwAAABkeyJvcmlnaW4iOiJodHRwczovL2ltbWVyc2l2ZS13ZWIuZ2l0aHViLmlvOjQ0MyIsImZlYXR1cmUiOiJXZWJYUlBsYW5lRGV0ZWN0aW9uIiwiZXhwaXJ5IjoxNjI5ODQ5NTk5fQ==">

		<title>AR Plane Detection</title>

Add body tracking sample #178

Are you sure you want to change the base?

Add body tracking sample #178

Conversation

cabanier commented Jan 24, 2024

Maksims commented Jan 25, 2024

toji left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment