Dominance analysis

[maxbiostat]

For all of the examples considered so far (Beta, Gamma, Normal) the entropy (and sometimes the KL too) optimisation problem is quite unstable, tending to one of the "corners" of the simplex. This happens because the entropy function is "dominated" by a parameter and thus any solution that assigns alpha_j = 1 for some j such that parameter_j = max(parameters) will be the best one.

To see this, consider the normal example. Since the entropy depends only on the variance, the distribution with larger variance will tend to dominate the optimisation problem, leading to the trivial solution (0, ..., 1, 0, ...). where the j-th position is a 1 and everybody else gets weight 0.

It would thus be interesting to analyse the entropy functions for the distributions considered to get insight into how much a parameter dominates the entropy.

The final solution, I think, is propose a constrained optimisation problem, where, for example, one would obtain the maximum entropy distribution with a given mean.