Skip to content

[3DV 2022] A semi-supervised method for improving generalization for 3D hand reconstruction.

License

Notifications You must be signed in to change notification settings

eth-ait/tempclr

Repository files navigation

TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning [3DV 2022]

report report

TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning, Andrea Ziani*, Zicong Fan*, Muhammed Kocabas, Sammy Christen, Otmar Hilliges *Equal Contribution. International Conference on 3D Vision (3DV), 2022

Image

You can find our training code here.

Updates

  • Sept. 28, 2022: Inference code and pre-trained model are ready.
  • Sept. 1, 2022: Project page and arXiv paper are online.

Table of contents

In case you encounter any issue following this README, please Contact Us.

Download our release models

Our release models can be downloaded at this Dropbox.

Once you downloaded our release zip file, extract it in the project directory. The final directory structure should look like:

TempCLR/
├── ...
TempCLR_release/
├── FreiHAND_model/
│   └── checkpoints/
│       ├── latest_checkpoint
│       └── tempclr_freihand
├── HO3D_model/
│   └── checkpoints/
│       ├── latest_checkpoint
│       └── tempclr_ho3d
├── LICENSE
└── TempCLR_utils/
    ├── data/
    │   ├── all_means.pkl
    │   └── shape_mean.npy
    └── models/
        └── mano/
            └── README
environment.yml
LICENCE
mano_left_extra_joints.yaml
mano_right_extra_joints.yaml            

Setup inference code

Datasets

The repository provides the code to perform inference on the FreiHAND dataset and the HO-3D_v2 dataset.

In order to run the inference code, download the datasets first and extract them in the same directory. The resulting directory structure and directory names should look like:

datasets
├── freihand
└── HO3D_v2

Setup anaconda environment

This repository provides the recipe tempclr-env.yaml for a conda environment containing every dependency the code needs in order to work.

If you have not installed anaconda in your device, install Anaconda first and then come back to this guide.

Create the environment

To create the environment, open a shell in the TempCLR project directory and run this command:

conda env create -f environment.yml

Wait until the execution ends. If everything went well you should see a suggestion to activate tempclr-env .

Activate the environment

To activate the environment run:

conda activate tempclr-env

Great! The environment has been activated. Now that the environment has been set up proceed to the next section.

Setup environment variables

Download MANO models

Our project is based on MANO. Hence, MANO models are needed to correctly run this codebase. Before proceeding further, download MANO models (i.e. MANO_RIGHT.pkl and MANO_LEFT.pkl) from the official website and place them inside TempCLR_utils/models/mano directory.

Setup variables

In the shell where you will run the inference code, run the following commands:

export MODELS=<path_to_TempCLR_utils>/models
export MEAN_POSE_PATH=<path_to_TempCLR_utils>/data
export DATASETS=<path_to_the_datasets_directory>
export INFERENCE=<path_to_TempCLR_release>

Run inference code

Once followed all the previous steps, open a shell in the TempCLR project with the conda environment and environment variables set and use the following command to run the inference code:

python TempCLR/main.py --exp-cfg TempCLR/configs/evaluation_freihand.yaml

for inference on FreiHAND, and

python TempCLR/main.py --exp-cfg TempCLR/configs/evaluation_ho3d.yaml

for inference on HO-3D.

Since the ground-truth labels of the test sets are hidden in CodaLab competitions (see HO-3D and FreiHAND), the code will save a pred.json file in the pre-trained model directory. This file contains the predictions on the test set to be uploaded on CodaLab. Zip the json file and upload it in the respective CodaLab competition to obtain the results for the different metrics reported in the paper.

Config file utils

Save inference re-projection images

To save the re-projection images produced by the pre-trained models set these two flags in the config file:

save_reproj_images: True
create_image_summaries: True

The images will be saved in the pre-trained model directory under a new directory named summary .

Acknowledgements

Muhammed Kocabas is supported by the Max Planck ETH Center for Learning Systems. The authors would like to thank Vassilis Choutas for providing the code of the baseline model (ExPose) adopted for the project.

Citations

If you find this work useful, consider citing:

@inProceedings{ziani2022tempclr,
  title={TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning},
  author={Ziani, Andrea and Fan, Zicong and Kocabas, Muhammed and Christen, Sammy and Hilliges, Otmar},
  booktitle={International Conference on 3D Vision (3DV)},
  year={2022}
}

@inproceedings{hampali2020honnotate,
  title={Honnotate: A method for 3d annotation of hand and object poses},
  author={Hampali, Shreyas and Rad, Mahdi and Oberweger, Markus and Lepetit, Vincent},
  booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
  pages={3196--3206},
  year={2020}
}


@inproceedings{Freihand2019,
  author    = {Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus and Thomas Brox},
  title     = {FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images},
  booktitle = {IEEE International Conference on Computer Vision (ICCV)},
  year = {2019},
  url          = "https://lmb.informatik.uni-freiburg.de/projects/freihand/"
}

@inproceedings{ExPose:2020,
  title = {Monocular Expressive Body Regression through Body-Driven Attention},
  author = {Choutas, Vasileios and Pavlakos, Georgios and Bolkart, Timo and Tzionas, Dimitrios and Black, Michael J.},
  booktitle = {European Conference on Computer Vision (ECCV)},
  pages = {20--40},
  year = {2020},
  url = {https://expose.is.tue.mpg.de}
}

@article{MANO:SIGGRAPHASIA:2017,
    title = {Embodied Hands: Modeling and Capturing Hands and Bodies Together},
    author = {Romero, Javier and Tzionas, Dimitrios and Black, Michael J.},
    journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)},
    volume = {36},
    number = {6},
    series = {245:1--245:17},
    month = nov,
    year = {2017},
    month_numeric = {11}
}

About

[3DV 2022] A semi-supervised method for improving generalization for 3D hand reconstruction.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages