Matching in GAN-Space

Code for using GANs to aid in matching, accompanying the paper "Overcoming confounding in face datasets via GAN-based matching" (arXiv)

Projection and manipulation • Matching and benchmarking
Reproducibility • Reference
Quickstart demo (manipulate and interpolate your own face images!):

Projection and manipulation

This code allows one to project images into the GAN latent space, after which they can be modified for certain attributes (e.g. age, gender, hair-length) and mixed with other faces (e.g. other people, older/younger versions of the same person). All this code is handled by the projection_manipulation/project_and_manipulate.sh script - the easiest way to get started is to use the Colab notebook, where you can upload your own images, and they will be automatically cropped, aligned projected, manipulated, and interpolated

Start with 2 real images (higher-res photos work better, as well as photos where the face is front-facing and not obstructed by things like hats, scarves, etc.):

Interpolating between the images:

Manipulating an image along pre-specified attributes:

Can do a lot more, like blending together many faces or interpolating between different faces of the same person!

Matching and benchmarking

The matching code here finds images that match across a certain attribute (e.g. perceived gender). This is useful for removing confounding factors when doing downstream analyses of things like gender bias in facial recognition. Similarly, we can perform matching using other methods, such as propensity scores, using the GAN latent space as covariates. Some example matches:

Note: these annotations do not necessarily reflect the *gender identity* of the person, rather they refer to *binarized gender as perceived by a casual observer*

After performing matching, confounding is much lower on CelebA-HQ. This is illustrated by the fact that the mean values of several key (binary) attributes become much closer after matching:

Reproducibility

Dependencies

• uses tensorflow-gpu 1.14.0 (the gpu dependencies are only required for the projection / manipulation code which uses StyleGAN2)
• the required dependencies can be set up on AWS by selecting a deep learning AMI, running source activate python3, and then running pip install tensorflow-gpu==1.14.0

Data/cached outputs for reproducing pipeline in this gdrive folder

• data/celeba-hq/ims folder
  • unzip the images in celeba-hq dataset at 1024 x 1024 resolution into this folder
• data/processed folder
  • distances: dists_pairwise_gan.npy, dists_pairwise_vgg.npy, dists_pairwise_facial.npy, dists_pairwise_facial_facenet.npy, dists_pairwise_facial_facenet_casia.npy, dists_pairwise_facial_vgg2.npy - (30k x 30k) matrices storing the pairwise distances between all the images in celeba-hq using different distance measures
• data/processed/gen/generated_images_0.1
  • latents celeba_hq_latents_stylegan2.zip - these are used in downstream analysis and are required for the propensity score analysis
• (already present in repo) - annotations (e.g. gender, smiling, eyeglasses) + predicted metrics (e.g. predicted yaw, roll, pitch, quality, race) for each image + latent StyleGAN2 directions for different attributes + precomputed match numbers
• (optional) can download the raw annotations and annotated images as well
• (optional) all these paths can be changed in the config.py file

Scripts

Both the matching_benchmarking folder and the projection_manipulation folder contain two types of files:

• .py files in the scripts subdirectories - these scripts are used to calculate the cached outputs in the gdrive folder. They do not need to be rerun, but show how the cached outputs were generated and can be rerun on new datasets.
• .ipynb notebooks - these are used to reproduce the results from the cached outputs in the gdrive folde. Noteboks beginning with eda are for exploratory analysis, which can be useful but are note required to generate the final results in the paper

Reference

• this project builds on many wonderful open-source projects (see the readmes in the lib subfolders for more details) including
• stylegan: stylegan2 and stylegan2 encoder
• facial recogntion: dlib, python face_recognition, facenet
• gender/race prediction: fairface
• pose/background prediction: deep_head_pose, face_segmentation, and faceQnet

@article{singh2021matched,
    title={Matched sample selection with GANs for mitigating attribute confounding}, 
    author={Chandan Singh and Guha Balakrishnan and Pietro Perona},
    journal={arXiv preprint arXiv:2103.13455},    
    year={2021}
}