This work is done when the author was interning at AILabs.tw
Please refer to rudychin/face_swap_docker for more detail
Inherit from YuvalNirkin/face_swap, this is a boosted version.
The pipeline of the framework is modified into a more efficient one. Specifically, we have the following contributions:
-
Merge expression regression into ResNet-101, which was used to regress shape and texture as illustrated in CNN3DMM. Hence, we completely throw away the computation overhead of expression approximation in the original pipeline. We fine-tuned the network with images from LFW and MegaFace.
-
Replace dlib face detection in two ways:
- Use YOLO instead (please refer to the
yolobranch) - Count on KCF tracking with dlib face detection (please refer to the
kcfbranch)
- We make it a shared library and develop a python wrapper for the ease-of-use.
Compare the original performance and the accelerated version (SmallerNet + Tracking + Downsampling):
| Library | Minimum Version | Notes |
|---|---|---|
| Boost | 1.47 | |
| OpenCV | 3.0 | |
| face_segmentation | 0.9 | |
| Caffe | 1.0 | ☕️ |
| Eigen | 3.0.0 | |
| GLEW | 2.0.0 | |
| Qt | 5.4.0 | |
| HDF5 | 1.8.18 | |
| KCFcpp | master |
mkdir build
cd build
cmake -DCMAKE_CXX_STANDARD=14 ..
make
make install
- Download the landmarks model file
- Download the face_seg_fcn8s.zip
- Download the dfm_cnn_resnet_101
Please modify the path within py_face_swap/pytest.py before you run it.
To swap an image, please modify from py_face_swap/pytest.py.
To swap a video, use tools/swapVideo.py.
- When face is moving to or from the screen, bounding box does not grow or shrink accordingly properly.
- We count on failure pose estimation to detect the failure of tracking passively. One possible solution is to use discriminative tracker, e.g. GOTURN
- Face Swap, the original version
- Deep face segmentation, used to segment face regions in the face swapping pipeline.
- CNN3DMM, used to estimate 3D face shapes from single images.