- Density estimation using Real NVP: https://arxiv.org/abs/1605.08803
A thought-provoking paper. Working with probabilistic generative models involve sampling (sample x given p(x)), inference (compute p(z|x) = p(z)p(x|z)/p(x) with latent (continuous) variables z) and exact likelihood computation (p(x) = int_{z}p(x|z)p(z)d(z). The paper proposes an interesting inversion models that address all the problems in a nice way. What I mean a nice way here is: exact sampling, easy to do inferenec and the likelihood can be computed easily. There seems no way of having such a model. But the author proves we can by exploiting bijective functions that map between each point of data (x) and a value of latent continous variable. The space of the variable must be easy to sample (e.g. a Gaussian: z ~ N(mean, variance)). Since we expoloiting bijective functions, it would be easy to find an inverse function of the bijective function so that given a point x, we can easily compute z (inference). Finally, the likelihood can be computed directly from the densitity function of z (it only involves additional computation regarding to the determinant of the Jocobian matrix with respect to the inverse function and x). Obviously, the bijective function must have a super rich representation to model arbitrary distribution, it must be easy and efficient to compute the Jacobian matrix (and the determinant of the matrix). The function they proposed relied on affine coupling layers. Relying on the idea everything can be done in a very neat way!
Question: how to train the model (maximize the likelhood of the data?) not sure I am missing something but they didn't describe that? This is no doubt a very cool work, and I was wondering how to apply the model to NLP applications as well! Finally, combining the model with VAE, GAN and autoagressive models could be a promising direction (see https://www.cs.toronto.edu/~duvenaud/courses/csc2541/slides/lec3-autoregressive.pdf)