Projection matrix is not updated in augmented face mode

[aartikov]

I am currently developing an augmented face application using SceneView. I've encountered an issue where the face mesh does not properly align with the actual face of the user.

Upon investigation, I found that the root cause of this issue seems to be related to the camera's projection matrix. Specifically, the projection matrix is intended to be updated in the onCameraUpdated method. However, this update does not occur in augmented face mode because the condition if(trackingState == TrackingState.TRACKING) is not satisfied.

Here is the problematic section of the code:

    open fun onCameraUpdated(camera: Camera) {
        val trackingState = camera.trackingState
        this.trackingState = trackingState
        if (trackingState == TrackingState.TRACKING) {
            // Update the node's transformation properties to match the tracked pose
            pose = camera.displayOrientedPose
            // Update the projection matrix.
            projectionTransform = camera.getProjectionTransform(near, far)
        }
    }

As a temporary solution, I have added the following code to force the calculation of the projection matrix:

onSessionUpdated = { _, frame ->
    // force projection matrix calculation because SceneView doesn't do it.
    cameraNode.projectionTransform = frame.camera.getProjectionTransform(cameraNode.near, cameraNode.far)
    ...
}

While this workaround resolves the issue temporarily, it would be beneficial for the SceneView library to handle this case natively.

[XuanDoQuy2000]

same issue

[hshapley]

How did you get Augmented Faces working in general? When I create an AugmentedFaceNode object, it crashes with the error vertexCount cannot be 0.

[aartikov]

How did you get Augmented Faces working in general? When I create an AugmentedFaceNode object, it crashes with the error vertexCount cannot be 0.

Yeah, I had the same issue. Just made my own implementation - AppAugmentedFaceNode.kt

[hshapley]

Ah, I did the same thing. Thank you for sending your code though! I'm curious, why did you take out the normals for the face mesh?
Also, I have to still pass in a builder that sets culling, castShadows, and receiveShadows to false to avoid another error from Filament.
I'm still not getting everything to work though - attaching a Node to the centerNode doesn't show up, and passing in a material for the faceMeshMaterial doesn't render anything on the face either.

[aartikov]

@hshapley

I'm curious, why did you take out the normals for the face mesh?

I don't use normals because I don't need lighting for face meshes. By the way, it's not trivial to add normals since Filament doesn't support the NORMAL vertex attribute. However, we could use TANGENTS, which are TBN (tangent, bitangent, normal) encoded as a quaternion.

Also, I have to still pass in a builder that sets culling, castShadows, and receiveShadows to false to avoid another error from Filament.

Right, I forgot to tell about it.

I'm still not getting everything to work though - attaching a Node to the centerNode doesn't show up, and passing in a material for the faceMeshMaterial doesn't render anything on the face either.

If you use my AppAugmentedFaceNode implementation don't forget to call AppAugmentedFaceNode::update whenever an AugmentedFace is tracked.

[hshapley]

I did not notice that update() needed to be called manually, but even doing so doesn't get it to work for me. Not sure what the disconnect is.
On a related note, have you gotten the occlusion material to work? If I simply apply the occlusion material to a sphere and track that with the face, it appears black with glitchy flickers. I'm referring to face_mesh_occluder.filamat.

[aartikov]

@hshapley
I didn't use the occlusion material yet.
This issue in Filament is related, probably.

[hshapley]

Wow, that's a blast from the past... I used that thread to solve issues with occlusion in an old version of Sceneform long ago. Lots of things have changed in SceneView since then though. I tried disabling depthCulling and it had an effect, but an undesirable one in making most models disappear. I will keep exploring a couple things but I don't see an easy fix as of yet. Thanks for all the pointers though.

[hshapley]

I got it working! I had to combine several things from the old implementation to get it to work, namely manipulating the render priority.
The camera stream render priority is set to LAST (7) in SceneView, and there is a comment for why that's done ("to avoid overdraw"). However, if the camera renders last with depthCulling disabled, it will render the camera over all models at default priority (4). So the camera needs to render first (0), then the occluding object (3 works fine), then the default objects. This gets everything to work, but unfortunately it will be difficult to incorporate into the library. If the camera stream should truly render last in most cases (to avoid overdraw), then we'd probably need to limit occlusion to AugmentedFaces for now, overriding the camera material and render priority in the AF sample (if we built one). Or we could add an option to the ARCameraStream, something like "isOcclusionEnabled", that would configure the stream appropriately based on the value.