In this project, we assess a new framework called UMIN on a data-driven optimization problem. Such a problem happens recurrently in real life and can quickly become difficult to model when the input has a high dimensionality as images for instance. From the architecture of aircraft to the design of proteins, a great number of different techniques have already been explored. Based on former solutions, this work introduces a brand new Bayesian approach that updates previous frameworks. Former model architectures use generative adversarial networks on one side and a forward model on the other side to improve the accuracy of the results. However, employing a Bayesian forward network allows us to leverage its uncertainty estimates to enhance the accuracy of the results and also to reduce unrealistic samples output by the generator. By creating new experiments on a modern MNIST dataset and by reproducing former works taken as baseline, we show that the framework introduces in this work outperforms the previous method.
We used well-known metrics from the generative model community as the FID(Fréchet Inception Distance), KID(Kernel Inception Distance), and MSE(mean squared error) scores to justify our results.
├── README.md <- The top-level README for developers using this project.
├── data <- Data collected from MNIST
│
├── models <- incepetion models to compute fid/kid scores
│
├── MNIST_generation <- Scripts to augment original MNIST dataset.
│
├── fig <- Distribution figures from augmented MNIST dataset.
│
├── requirements.txt <- The requirements file for reproducing the analysis environment, e.g.
│ generated with `pip freeze > requirements.txt`
│
├── setup.py <- makes project pip installable (pip install -e .) so src can be imported
├── utils.py <- Plot functions repository
├── src <- Source code for use in this project.
│ ├── __init__.py <- Makes src a Python module
│ │
│ ├── data <- Scripts to download or generate data
│ │ └── make_dataset.py
│ │
│ ├── metrics <- Source of different metrics
│ │ └── metrics.py
│ │
│ ├── generative_model <- Generators training scripts
│ │ └── train_model.py
| |
│ ├── forward_model <- forward models training scripts
│ │ ├── inference_func_bayesian.py
| | |
| | └── inference_func_frequentist.py
| |
│ ├── uncertainty_policy <- Uncertainty selection function
│ │ └── policy.py
│ │
│ └── Inference <- Script to run the model inference
│ └── inference.py
│
└── tox.ini <- tox file with settings for running tox; see tox.testrun.org
If you have a google account by simple click on the following links you'll be able to run the experiments on Google Colab
Experiments :
-
The maximization experiment is presented here on Colab
https://colab.research.google.com/drive/1Gg8Iz8SEzYr6NcEqcZvwJgkh2Mm1UYqQ?usp=sharing
-
The minimization experiment is available here on Colab
Python 3.6.9 and older
Torch 1.4.0
-
If you are fine to use it with colab, you can open the link above and the various dependencies should be already satisfied.
-
If you want to be able to use the source code and deploy it on your local machine. You will need to clone the following github version (date : 08.04.2020)
git clone https://github.com/dccastro/Morpho-MNIST.git
cp -r Morpho-MNIST/morphomnist morphomnist
git clone https://github.com/abdulfatir/gan-metrics-pytorch.git
cp -r gan-metrics-pytorch/models models
git clone https://github.com/RomainGratier/Black-box_Optimization_via_Deep_Generative-Exploratory_Networks.git
cp -r Black-box_Optimization_via_Deep_Generative-Exploratory_Networks/src src
Explain how to run the automated tests for this system
You can load the morphomnist dataset as a pytorch dataset using the following commands
from src.data import MNISTDataset
dataset = MNISTDataset('train', 'thickness')
You can either load the models and play with them following the notebook or just load the training loop to train them.
from src.models import Generator, Discriminator, ForwardModel
from src.models import train_forward_model(), train_gan_model()
You can do inference experiments when the two models are trained
from src.inference import monte_carlo_inference
inf_best_5, mean_sample_measure_5, fid_gen_5 = monte_carlo_inference(5, generator, forward, trainset, testset, sample_number = 800)
A new augmented MNIST dataset was used to assess the framework quality. This dataset is build using the MNIST_generation scripts but one can also use the dataset created and available in the folder data. This dataset allows us to have an image of a digit with different characteristics labeled. The figure below shows those features and their distribution within the dataset original_thic_resample and the real MNIST.
Augmented MNIST | Original MNIST
We created two similar experiences to assess our model quality. Using our augmented MNIST dataset we can create a model that will try to optimize the stroke width of a digit. During the first experiment, the stroke width of a digit is increased compared to our previous knowledge about our data. Hence, the UMIN will be trained on digit with strokes width between 0 and 6 pixels [pxl] and it will try to create digits with higher stroke width. During the second experiment, the stroke width of the digits will be minimized. Hence, UMIN will be trained on digit with strokes width between 2.5 and 8 pixels [pxl] and it will try to create digits with lower stroke width.
The following figure displays samples from the true distribution from an augmented MNIST dataset.
Real samples:
Maximization experiment | Real samples | Minimization experiment
Here are displayed our results in terms of prediction precision with the mean squared error and mean relative error.
MSE/MRE
Real samples
The next table exhibit the quantitative assessment of the data quality using the frêchet inception distance (FID) and the kernel inception distance (KID).
FID/KID
To assess our results. It is interesting to compare the best sample selected by our inference function and the worst samples discarded by our inference function. The first two rows display the generated samples and the last two rows exhibit real samples with the targeted width stroke.
Best samples
Worst samples
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
- Romain Gratier de Saint-Louis PurpleBooth
- Yuejiang Liu - VITA Lab EPFL - PurpleBooth
- Alexandre Alahi - VITA Lab EPFL
This project is licensed under the MIT License - see the LICENSE.md file for details
Special thanks to the writers of the paper available on arxiv: https://arxiv.org/pdf/1912.13464.pdf. Their contribution motivates this project.
Another special thanks to writers of the paper available on arxiv ( https://arxiv.org/pdf/1901.02731.pdf ) and their github repository to allow us to use accurate Bayesian neural networks.
Bravo to the work done on https://github.com/dccastro/Morpho-MNIST that allows us to find an interesting dataset and established ground-truth measurements for our generated data
Bravo also to https://github.com/eriklindernoren/PyTorch-GAN to share the PyTorch implementation of useful GANs
And Bravo to https://github.com/abdulfatir/gan-metrics-pytorch.git that implemented a great FID score for MNIST dataset