This repository contains the implementation of the following paper:
StyleGene: Crossover and Mutation of Region-level Facial Genes for Kinship Face Synthesis
Hao Li, Xianxu Hou, Zepeng Huang, Linlin Shen
From CVI@SZU, Shenzhen University
IEEE/CVF International Conference on Computer Vision (CVPR), 2023
[Paper] | [Project Page] | [Video] | [Slide] | [Huggingface]
The first row is the input grandparents, and the second and third rows are their descendants generated by our method.
High-fidelity kinship face synthesis has many potential applications, such as kinship verification, missing child identification, and social media analysis. However, it is challenging to synthesize high-quality descendant faces with genetic relations due to the lack of large-scale, high-quality annotated kinship data. This paper proposes RFG (Region-level Facial Gene) extraction framework to address this issue. We propose to use IGE (Image-based Gene Encoder), LGE (Latent-based Gene Encoder) and Gene Decoder to learn the RFGs of a given face image, and the relationships between RFGs and the latent space of StyleGAN2. As cycle-like losses are designed to measure the
$L_2$ distances between the output of Gene Decoder and image encoder, and that between the output of LGE and IGE, only face images are required to train our framework, i.e. no paired kinship face data is required. Based upon the proposed RFGs, a crossover and mutation module is further designed to inherit the facial parts of parents. A Gene Pool has also been used to introduce the variations into the mutation of RFGs. The diversity of the faces of descendants can thus be significantly increased. Qualitative, quantitative, and subjective experiments on FIW, TSKinFace, and FF-Databases clearly show that the quality and diversity of kinship faces generated by our approach are much better than the existing state-of-the-art methods.
git clone https://github.com/CVI-SZU/StyleGene.git
cd StyleGeneIt is suggested to create a Conda environment and install the following requirements
pip3 install -r requirements.txtRequirements:
- Python 3.9
- torch 1.12.1
- Pillow 9.2.0
- einops 0.3.2
- easydict 1.10
- gradio 3.32.0
- opencv-python 4.5.4.60
- pandas 1.4.4
- numpy 1.21.5
- ninja
We have uploaded all the necessary weights, please clone to the checkpoints/ directory.
mkdir checkpoints
cd checkpoints
git clone https://huggingface.co/wmpscc/StyleGenePut the pre-trained models under checkpoints/ as follows:
StyleGene
└── checkpoints
├── e4e_ffhq_encode.pt
├── geneFactorPool.pkl
├── res34_fair_align_multi_7_20190809.pt
├── shape_predictor_68_face_landmarks.dat.bz2
├── stylegan2-ffhq-config-f.pt
└── stylegene_N18.ckpt
Start the interactive interface and start playing.
cd StyleGene
python app.pyThe default URL is http://127.0.0.1:7860/. If you are running on the server, please pay attention to open port 7860.
If you are using our code, please consider citing our paper.
@inproceedings{li2023stylegene,
title={StyleGene: Crossover and Mutation of Region-Level Facial Genes for Kinship Face Synthesis},
author={Li, Hao and Hou, Xianxu and Huang, Zepeng and Shen, Linlin},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={20960--20969},
year={2023}
}
This method is intended for academic research purposes only and is strictly prohibited for commercial use. Users are required to comply with all local laws and regulations when using this method.
Part of the code comes from other repo, please abide by the original open source license for the relevant code.