Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is MonteCarlo elbo and Importance Weighted elbo computation mismatched? #52

Open
pkulwj1994 opened this issue Jun 1, 2021 · 0 comments

Comments

@pkulwj1994
Copy link

pkulwj1994 commented Jun 1, 2021

I notice the implement of MonteCarlo elbo and Importance Weighted elbo computation and find out things seems to be mismatched among two methods.

In elbo implementation from montecarlo.py I find code blocks below:

from numbers import Number
from torch.nn.functional import softmax

def elbo(q, p, sample_dim=None, batch_dim=None, alpha=0.1, beta=1.0, size_average=True, reduce=True):
    log_weights = q.log_joint(sample_dim, batch_dim, q.conditioned())    
    return (log_like(q, p, sample_dim, batch_dim, log_weights,size_average=size_average, reduce=reduce) -beta * kl(q, p, sample_dim, batch_dim, log_weights,size_average=size_average, reduce=reduce) +(beta + alpha) * ml(q, sample_dim, batch_dim, log_weights,size_average=size_average, reduce=reduce))

which seems to compute IW elbo involving computation of Importance Weight;

While in importance.py, I find shorter implementation below

from numbers import Number
from probtorch.util import log_mean_exp

def elbo(q, p, sample_dim=None, batch_dim=None, alpha=0.1,size_average=True, reduce=True):
    z = [n for n in q.sampled() if n in p]    log_pxyz = p.log_joint(sample_dim, batch_dim)    
    log_qz = q.log_joint(sample_dim, batch_dim, z)    
    log_qy = q.log_joint(sample_dim, batch_dim, q.conditioned())    
    log_pq = (log_pxyz - log_qz)    
    if sample_dim is None:        
        objective = log_pq + alpha * log_qy    
    else:        
        objective = log_mean_exp(log_pq, 0)        
        if not isinstance(log_qy, Number):            
            objective = objective + alpha * log_mean_exp(log_qy, 0)    
    if reduce:        
        objective = objective.mean() if size_average else objective.sum()    
    return objective

which seems like direct estimate of elbo based on samples from variational distribution q(z|x). So am I wrong or these two implenemtation are truely mismatched?

@pkulwj1994 pkulwj1994 reopened this Jun 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant