Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

change the suface_model type from smplx to smpl #4

Open
minushuang opened this issue Jul 7, 2022 · 5 comments
Open

change the suface_model type from smplx to smpl #4

minushuang opened this issue Jul 7, 2022 · 5 comments

Comments

@minushuang
Copy link

minushuang commented Jul 7, 2022 •
edited

the code can work normally when the model type is smplx.
and now I try to fit the smpl params instead of the smplx following the smplx moshpp process.
at first I donwloaded the smpl basic models, and put them under support_files/smpl directory, then change the surface_model.type from smplx to smpl, and added opt_weights.smpl config which is a copy of smplx as below.

opt_weights:
  smpl:
    stagei_wt_poseH: 3.0
    stagei_wt_poseF: 3.
    stagei_wt_expr: 34.
    stagei_wt_pose: 3.
    stagei_wt_poseB: 3.
    stagei_wt_init_finger_left: 400.0
    stagei_wt_init_finger_right: 400.0
    stagei_wt_init_finger: 400.0
    stagei_wt_betas: 10.
    stagei_wt_init: 300
    stagei_wt_data: 75.
    stagei_wt_surf: 10000.
    stagei_wt_annealing: [ 1., .5, .25, .125 ]
    stageii_wt_data: 400
    stageii_wt_velo: 2.5
    stageii_wt_dmpl: 1.0
    stageii_wt_expr: 1.0
    stageii_wt_poseB: 1.6
    stageii_wt_poseH: 1.0
    stageii_wt_poseF: 1.0
    stageii_wt_annealing: 2.5

the error when I run the mosh job(task)

2022-07-07 08:11:49.227 | INFO     | soma.tools.parallel_tools:run_parallel_jobs:54 - #Job(s) submitted: 83
2022-07-07 08:11:49.227 | INFO     | soma.tools.parallel_tools:run_parallel_jobs:67 - Will run the jobs in random order.
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:95 -- mocap_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/custom_low_head2_yz_swap.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:97 -- stagei_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/male_stagei.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:98 -- stageii_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/custom_low_head2_yz_swap_stageii.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:103 -- surface_model: type: smpl; gender: male; fname:/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/smpl/male/model.pkl
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:__init__:107 -- optimize_fingers: False, optimize_face: False, optimize_toes: False, optimize_betas: True, optimize_dynamics: False
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:prepare_stagei_frames:157 -- Selecting 12 frames using method manual on frames with 100% least_avail_markers
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:prepare_stagei_frames:197 -- Using stagei_fnames for stage-i: ['/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_002.pkl_001091'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000137'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_001.pkl_001366'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000509'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/throw_001.pkl_000596'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_003.pkl_001488'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_001.pkl_000588'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/squat_002.pkl_001134'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/jump_002.pkl_000471'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/run_001.pkl_000032'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_001.pkl_001042'
 '/data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/soma_labeled_mocap_tracklet/SOMA_unlabeled_mpc/soma_subject1/dance_001.pkl_000289']
soma_subject1 -- custom_low_head2_yz_swap -- mosh_head:mosh_stagei:241 -- Attempting mosh stagei to create /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/soma_subject1/male_stagei.pkl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:120 -- using marker_layout_fname: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE//training_experiments/V48_02_SOMA/OC_05_G_03_real_000_synt_100/evaluations/mosh_results_tracklet/SOMA_unlabeled_mpc/SOMA_unlabeled_mpc_smpl.json
soma_subject1 -- custom_low_head2_yz_swap -- bodymodel_loader:load_moshpp_models:93 -- Loading model: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/smpl/male/model.pkl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:172 -- can_model.betas.shape: (300,)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:173 -- opt_models[0].betas.shape: (300,)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:192 -- Estimating for #latent markers: 53
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:229 -- Number of available markers in each stagei selected frames: (F00, 53), (F01, 53), (F02, 53), (F03, 53), (F04, 53), (F05, 53), (F06, 53), (F07, 53), (F08, 53), (F09, 53), (F10, 53), (F11, 53)
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:237 -- Rigidly aligning the body to the markers
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:262 -- MoSh stagei weights before annealing:
stagei_wt_poseH: 3.0
stagei_wt_poseF: 3.0
stagei_wt_expr: 34.0
stagei_wt_pose: 3.0
stagei_wt_poseB: 3.0
stagei_wt_init_finger_left: 400.0
stagei_wt_init_finger_right: 400.0
stagei_wt_init_finger: 400.0
stagei_wt_betas: 10.0
stagei_wt_init: 300
stagei_wt_data: 75.0
stagei_wt_surf: 10000.0
stagei_wt_annealing: [1.0, 0.5, 0.25, 0.125]
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:266 -- head_marker_corr_fname is provided and is being loaded: /data/hxh/project/soma/SOMA_FOLDER_TEMPLATE/support_files/ssm_head_marker_corr.npz
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:274 -- Successfully took into account the correlation of the head markers
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:280 -- Beginning mosh stagei with opt_settings.weights_type: smpl
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:354 -- Step 1/4 : Opt. wt_anneal_factor = 1.00, wt_data = 1.00, wt_poseB = 65.09
soma_subject1 -- custom_low_head2_yz_swap -- chmosh:mosh_stagei:356 -- stagei_wt_init for different marker types body = 300.00:
Traceback (most recent call last):
  File "test_soma.py", line 115, in <module>
    'randomly_run_jobs': True,
  File "/data/hxh/project/soma/src/soma/tools/run_soma_multiple.py", line 283, in run_soma_on_multiple_settings
    run_parallel_jobs(func=run_moshpp_once, jobs=mosh_jobs, parallel_cfg=moshpp_parallel_cfg)
  File "/data/hxh/project/soma/src/soma/tools/parallel_tools.py", line 79, in run_parallel_jobs
    func(job)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/mosh_head.py", line 598, in run_moshpp_once
    mp.mosh_stagei(mosh_stagei)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/mosh_head.py", line 245, in mosh_stagei
    v_template_fname=self.cfg.moshpp.v_template_fname)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/chmosh.py", line 421, in mosh_stagei
    [f'{k} = {np.sum(opt_objs[k].r ** 2):2.2e}' for k in sorted(opt_objs)])))
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/chmosh.py", line 421, in <listcomp>
    [f'{k} = {np.sum(opt_objs[k].r ** 2):2.2e}' for k in sorted(opt_objs)])))
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 708, in compute_r
    return self.a.r * self.b.r
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/reordering.py", line 376, in compute_r
    return np.concatenate([t.r for t in self.our_terms], axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/reordering.py", line 376, in <listcomp>
    return np.concatenate([t.r for t in self.our_terms], axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 594, in r
    self._call_on_changed()
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 589, in _call_on_changed
    self.on_changed(self._dirty_vars)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/prior/gmm_prior_ch.py", line 62, in on_changed
    for logl, w in zip(self.loglikelihoods, self.weights)])
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/moshpp-3.0-py3.7.egg/moshpp/prior/gmm_prior_ch.py", line 62, in <listcomp>
    for logl, w in zip(self.loglikelihoods, self.weights)])
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 319, in compute_r
    return np.sum(self.x.r, axis=self.axis)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 584, in compute_r
    return self.safe_power(self.x.r, self.pow.r)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 708, in compute_r
    return self.a.r * self.b.r
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 731, in compute_r
    return self.a.r.dot(self.b.r)
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch.py", line 596, in r
    self._cache['r'] = np.asarray(np.atleast_1d(self.compute_r()), dtype=np.float64, order='C')
  File "/root/anaconda3/envs/soma/lib/python3.7/site-packages/chumpy/ch_ops.py", line 566, in compute_r
    return self.a.r - self.b.r
ValueError: operands could not be broadcast together with shapes (69,) (63,)
@minushuang
Copy link
Author

minushuang commented Jul 11, 2022 •
edited

the length of pose_body_ids of smpl is 69 according to the code in chmosh.py, which is different from smplx 63

    if cfg.surface_model.type == 'smpl':
        pose_body_ids = all_pose_ids[3:] #69
    elif cfg.surface_model.type == 'smplh':
        pose_body_ids = all_pose_ids[3:66]
        if cfg.moshpp.optimize_fingers:  # dont chop chumpy variables two times
            pose_finger_ids = all_pose_ids[66:]
    elif cfg.surface_model.type == 'smplx':  # orient:3, body:63, jaw:3, eyel:3, eyer:3, handl, handr
        pose_body_ids = all_pose_ids[3:66]

modify the smpl pose_body_ids from
pose_body_ids = all_pose_ids[3:]
to
pose_body_ids = all_pose_ids[3:66]
is a temporary way for running the process but not the ultimate solution I think.

@minushuang
Copy link
Author

minushuang commented Jul 12, 2022 •
edited

the problem is at the pose_body_prior.pkl used in moshpp.prior.gmm_prior_ch.py, where the gmm['covars'] and gmm['means'] shapes are (8, 63, 63) and (8,63) respectively in the pkl file, so the npose=69 doesn't work when the model_surface_type is smpl.

npose = 63 if exclude_hands else 69

covars = gmm['covars'][:, :npose, :npose]
means = gmm['means'][:, :npose]
weights = gmm['weights']

@minushuang
Copy link
Author

the blender output mesh result is like #5 , the pose information is lost with correct shape and orientation.

@ritaank
Copy link

ritaank commented May 18, 2023

Hi,

This is super helpful, and I am trying to recreate this currently. To my understanding, you downloaded the SMPL python model files, and then restructured them into the support_files/smpl folder?

As part of this process, I renamed the SMPL pkl files from the original basicmodel_f_... to model.pkl to match the structure of smplx. I then copied the pose_body_prior.pkl file. My MoSh run is currently throwing an error since I don't have the corresponding model.npz file for each body type. Is this file just derived from the pkl file?

@ritaank
Copy link

ritaank commented May 18, 2023 •
edited

It seems the requisite .npz files stem directly from the .pkl files. The pkl files for SMPL can be opened as per the guidance in this comment. Special care must be taken for the scipy sparse matrices. Here's some boilerplate code:

import pickle
import numpy
import scipy

with open('./support_files/smpl/male/model.pkl', 'rb') as f:
    u = pickle._Unpickler(f)
    u.encoding = 'latin1'
    male_smpl_pkl = u.load()
    # print(male_smpl_pkl)

npz = {}
for k,v in male_smpl_pkl.items():
    if type(v) == np.ndarray:
        npz[k] = v
    elif type(v) == scipy.sparse.csc.csc_matrix:
        npz[k] = v.todense().astype(np.float32)
    else: # for strings and chumpy arrays
        npz[k] = np.array(v)

np.savez('./support_files/smpl/male/model.npz', **npz)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@minushuang @ritaank