MLE and MAP

Machine Learning: Maximum Likelihood Estimation (MLE) and Maximum a Posteri (MAP) Estimation

Subtleties

ML doesn't work good with sparse data because P(X | Y) might be zero. (for example, Xi = birthdate, Xi = Jan_25_1992)

P(Y=1 | X1...Xn) = (P(Y=1) * Mult P(Xi | Y=1) for i) / P (X1...Xn)

We can solve it by using prior with MAP estimation.

MLE

Pros

• invariant under reparameterization. So we can wrap in any function.

MAP

Pros

• avoid overfitting (regularization / shrinkage)
• tends to look like MLE asymptotically

Cons

• point estimation (no representation of uncertainty in θ). Because it could choose spike of θ because it has higher probability
• not invariant under reparameterization
• must assume prior on θ

Examples

Univariate Gaussian mean

in other words it is sample mean plus prior mean.

so when n->0 we get

but when n->∞ we get

• MLE symbolic example
• MLE statsmodel example

Related Topics

• The Cramer-Rao Lower Bound
  • Central Limit Theorem sum independent random variables are tend toward a normal distribution
• Likelihood Ratio Test (compare zero hypothesis with ml value)
• Wald Test
• etc

Videos

Jeff Miller (mathematicalmonk)

• (ML 6.1) Maximum a posteriori (MAP) estimation
• (ML 6.2) MAP for univariate Gaussian mean
• (ML 6.3) Interpretation of MAP as convex combination