Bayesian Methods for Neural Networks

**Contacts**:
Ekaterina Lobacheva - lobacheva.tjulja@gmail.com
Ilia Yakubovsky - ilia.yakubovskiy@yandex.ru

Please, add tag [BayesBootcamp] to all the emails.

Materials:

1. (russian) Ветров Д.П., Кропотов Д.А. Байесовские методы машинного обучения: часть 1, часть 2.
   
2. (russian) Wiki page of the Bayesian metods course at CS MSU.
   
3. Christopher M. Bishop. [Pattern Recognition and Machine Learning] (http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf).

Syllabus

Day 1.

• Bayesian methods: introduction, bayesian reasoning. Seminar: how to work with probabilities in bayesian paradigm.
• Analytical bayesian inference, conjugate distributions, exponential family. Seminar: inference in models with conjugate distributions.
• Bayesian linaer regression.

Additional materials (russian): notes, problems 1, problems 2

Day 2.

• Expectation–maximization (EM) algorithm. EM for Gaussian mixture.
• PCA and Bayesian PCA.
• Practice: ЕМ-algorithm for the investigation.

Additional materials (russian): notes

Day 3.

• Variational inference. Seminar: examples of usage for different models.
• How to use bayesian inference in real life and how to choose which method to use.
• Latent Dirichlet allocation

Additional materials (russian): notes 1, notes 2, notes 3.

Day 4.

• Stochastic variational inference.
• VAE
• IWAE – way to improve ELBO
• Normalizing flows – way to improve posterior distribution
• Practice: variational autoencoders.

Day 5.

• Bayesian neural network
• Variational dropout
• Soft Weight-Sharing for Neural Network Compression
• Variational Dropout Sparsifies Deep Neural Networks
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Summary of the course.