Eyes blink and Smile in Augment Face Ar Android

[Amitdygon]

Hello @ThomasGorisse

Hope you are doing well.

I have implement Augment face Ar using sceneform-android and Able to get the All Morph traget from the 3D GLB model and i have do some research. I am able to get the Real time LEFT EYE , RIGHT EYE and SMILE effect using ML kit as right now ML kit provide us only 3 face probability and its work 100% accurate and real time. I will create demo project for this ASAP share the github repo. Thanks alot for your libarary its really help ful hope in some case if I am to find and animate all morph target like IOS .

Thanks

[ThomasGorisse]

Hi!
Awesome !!!
Do you only map the bones coming from your ML or also the center, right ear, left ear coming from ARCore? (So 6 bones if I'm right)
It might not be easy to find the right renderable entity within sceneform.
You should come to the discord server to discuss more about it.
We are currently finalizing the 1.0.0 version of SceneView that will be a lot easier for you to map bones since they are now directly accessible has nodes.
With SceneView 1.0.0, every entity (objects / bones / lights / cameras) in the glTF model file are extracted has nodes.
Meaning you only have to map the transform from ARCore and your ML to the bone node transform:

modelNode.childNodes.first { it.name == "left ear" }.worldPosition= leftEarPose.position

We have continuous discussion around 1.0.0 on discord contributors channel.
Join us and share what you have done so everyone can benefit.

[Amitdygon]

No No i just use frame capture for capture the frame of Ar view and process with ML kit its provide me real time blink and Smile effect let me share you few code its might help you also i have found a libaray which provide us almost 40 weight like blink,smile , talking , chick pull toungh out etc working on its how can i merge it with ar scene so i can use it with front camera.

Here Google ML kit with Ar -> https://github.com/Amitdygon/Face-Ar-with-ML-kit-
Here The project the provide wll face weight -> https://github.com/facemoji/mocap4face

Please check its might be helpful for others

[ThomasGorisse]

What you have made is completly awesome!!!
The only thing you should change is here:
https://github.com/Amitdygon/Face-Ar-with-ML-kit-/blob/6a84267b46fcb535387e6134f308e302ebf8fae9/app/src/main/java/com/example/mlkitwithlivecamera/ArCoreActivity.kt#L118
You are making an unecessary session update call instead of retrieving the frame corresponding to your augmented face.
Simply call this instead

val frame = arFragment?.arSceneView?.getArFrame()

Very great job around the ML face recognition applied to model morph weights.

[Amitdygon]

What you have made is completly awesome!!! The only thing you should change is here: https://github.com/Amitdygon/Face-Ar-with-ML-kit-/blob/6a84267b46fcb535387e6134f308e302ebf8fae9/app/src/main/java/com/example/mlkitwithlivecamera/ArCoreActivity.kt#L118 You are making an unecessary session update call instead of retrieving the frame corresponding to your augmented face. Simply call this instead

val frame = arFragment?.arSceneView?.getArFrame()

Very great job around the ML face recognition applied to model morph weights.

What about this SDK : https://github.com/facemoji/mocap4face
is there any possibility we can use our augmented face ar with this its provide all kind of weight from camera. Or any other solution where i can get all other weight like ArKit provide in IOS.