Missing keypoint recovery after convention

[Dipankar1997161]

Hello @Wei-Chen-hub,

I used the keypoint convention for smpl to coco.
There are certain missing joints after conversion, particularly the face keypoints.

Is it possible to recover "nose" keypoint after the conversion? That's the most important one for me.
here is the example result

I will wait for your response

[Wei-Chen-hub]

This is because SMPL only have limited joints for face and hand, you can see SMPL keypoints here

Nose is not included in smpl joint set (which only have 24 joints), convert_kps only converts keypoints with similiar definitions.

Considering the source of joints annotations, I suggest two ways:

• Regress full smpl joints, which supports nose (smpl_45 instead of smpl)
• Manually interpolate the nose keypoints, perhaps from head and neck, if original smpl parameters are not accessible

[Dipankar1997161]

Hello @Wei-Chen-hub,
Thanks for the clarification.

The smpl data I have generated has 24 joints, which is the standard used.
For the first method, is the full smpl regression possible through mmhuman3d? if yes, can you refer it to me.

I was thinking of the second method, but couldn't really find a proper source to read and try it out. I did try one directly with coco, but it turned out incorrectly.

Looking forward to hear from you

[Wei-Chen-hub]

For joints regression from SMPL parameters, you can see the official SMPL/SMPLX repo.

Alternatively, in MMHuman3D you can do this through several steps:

• Prepare body model as here
• Build body model (in your code)

from mmhuman3d.models.body_models.builder import build_body_model

smpl_model = build_body_model(dict(
    type='SMPL',
    keypoint_src='smpl_45',
    keypoint_dst='smpl_45',
    model_path='data/body_models/smpl', # modify this path to your SMPL model folder
    gender='neutral',
    num_betas=10,
    use_pca=False,
    batch_size=1)).to(self.device)

• Input parameters and get 3d joints

output = smpl_model(parameters_dict, return_joints=True) # smpl parameters
keypoints_3d = output['joints'].detach().cpu().numpy()

The input parameters are basically same as the official inplementation.

[ypilseong]

The data I need is a mediapipe key point. Could you tell me how to do a convention with a mediapipe keypoint?

[ypilseong]

@Wei-Chen-hub
Is it right to write the build_body_model in estimate_smpl.py?

[ypilseong]

@Dipankar1997161
Can I know how to convert keypoints to mediapipes?