[bug] LinAlgError for laplace_approximation(), singular Hessian

[krademaker]

Originally filed as an issue in statistical_rethinking repo (textbook on Bayesian statistics with Pyro exercises).

Filed here instead upon owner's request, who replied: "The issue is hessian at MAP point is singular. Could you file an issue in Pyro instead?"

Environment

• Ubuntu 20.04.5 LTS, Python 3.10.8
• PyTorch 1.13.0
• Pyro 1.8.4+9ed468d

Code snippet

I used an in-built function for MAP estimation. Going through the code snippets 4.24-4.26 found here:

# 4.24
howell1 = pd.read_csv("../data/Howell1.csv", sep=";")
howell1.head()

Output:

	height	weight	age	male
0	151.765	47.825606	63.0	1
1	139.700	36.485807	63.0	0
2	136.525	31.864838	65.0	0
3	156.845	53.041914	41.0	1
4	145.415	41.276872	51.0	0

# 4.25
def flist(height):
    mu = pyro.sample("mu", dist.Normal(178, 20))
    sigma = pyro.sample("sigma", dist.Uniform(0, 50))
    with pyro.plate("plate"):
        pyro.sample("height", dist.Normal(mu, sigma), obs=height)

The error arises here:

#4.26
d2_height = torch.tensor(d2["height"], dtype=torch.float)
m4_1 = MAP(flist).run(d2_height)

---------------------------------------------------------------------------
_LinAlgError                              Traceback (most recent call last)
Cell In[38], line 2
      1 d2_height = torch.tensor(d2["height"], dtype=torch.float)
----> 2 m4_1 = MAP(flist).run(d2_height)

File ~/learning_pyro/notebooks/rethinking.py:101, in MAP.run(self, *args, **kwargs)
     99         except Exception as e:
    100             last_error = e
--> 101 raise last_error

File ~/learning_pyro/notebooks/rethinking.py:98, in MAP.run(self, *args, **kwargs)
     96 for i in range(10):
     97     try:
---> 98         return super(MAP, self).run(*args, **kwargs)
     99     except Exception as e:
    100         last_error = e

File ~/anaconda3/envs/PYRO/lib/python3.10/site-packages/pyro/infer/abstract_infer.py:251, in TracePosterior.run(self, *args, **kwargs)
    249 self._reset()
    250 with poutine.block():
--> 251     for i, vals in enumerate(self._traces(*args, **kwargs)):
    252         if len(vals) == 2:
    253             chain_id = 0

File ~/learning_pyro/notebooks/rethinking.py:85, in MAP._traces(self, *args, **kwargs)
     82     return loss
     84 optimizer.step(closure)
---> 85 guide = delta_guide.laplace_approximation(*args, **kwargs)
     87 # get posterior
     88 for i in range(self.num_samples):

File ~/anaconda3/envs/PYRO/lib/python3.10/site-packages/pyro/infer/autoguide/guides.py:1181, in AutoLaplaceApproximation.laplace_approximation(self, *args, **kwargs)
   1179     cov /= scale[:, None]
   1180     cov /= scale[None, :]
-> 1181     scale_tril = torch.linalg.cholesky(cov)
   1183 gaussian_guide = AutoMultivariateNormal(self.model)
   1184 gaussian_guide._setup_prototype(*args, **kwargs)

_LinAlgError: linalg.cholesky: The factorization could not be completed because the input is not positive-definite (the leading minor of order 1 is not positive-definite).

Note

I found that alternative values for mu = pyro.sample("mu", dist.Normal(178, 20)) did manage to get code snippet 4.29 to work:

#4.29 def model(height):
    mu = pyro.sample("mu", dist.Normal(178, 0.1))
    sigma = pyro.sample("sigma", dist.Uniform(0, 50))
    with pyro.plate("plate"):
        pyro.sample("height", dist.Normal(mu, sigma), obs=height)

m4_2 = MAP(model).run(d2_height)

The owner replied: "The issue is hessian at MAP point is singular. Could you file an issue in Pyro instead?".