Is there a goodness-of-fit criterion to determine whether the model is appropriate?

[ellot]

When I use chi statistic in XSPEC, if the reduced chisq is approximately 1 then I will consider the model is appropriate.
I am wondering if there is such a convenient criterion in BXA? If there is no such criterion, so how to quantitively justify whether the model fits the data?

[JohannesBuchner]

You can do posterior predictive checks. It is helpful to apply some binning in the prediction (not the fit), for clarity. Here is how to make such a plot: https://johannesbuchner.github.io/BXA/xspec-analysis.html#model-checking

You can also do Q-Q plots, which avoid binning. Here is how to make such a plot: https://johannesbuchner.github.io/BXA/xspec-analysis.html#model-discovery

A further test is to leave some data out in the fit, and try to predict it.

Finally, you can do parametric bootrap: For each posterior sample, generate data and compute the likelihood there. If the loglikelihood distribution is very different than the loglikelihood distribution on the actual data, there may be a problem. This would give you a p-value (where does the most extreme loglikelihood of the data lie in the simulated distribution).

I should also say that "reduced chisq is approximately 1" is not quite correct. "chi²" follows a chi² distribution with some degrees of freedom, from which you can get p-values. However, it only follows such a distribution if the model is linear in the parameters, and if you have Gaussian distributed error bars.

Statisticians these days recommend to use visualisations and domain expertise more, rather than trying to shoehorn everything into a test.

[JohannesBuchner]

Another way to go about it is to look at some of the visualisations (e.g., Q-Q plot, posterior predictive check), and guess where the model could be improved, and fit a more complicated model. This can also be empirical, for example, adding a line somewhere where it seems needed.
Then, a model comparison between the initial and the more complicated model, e.g., with the AIC, can indicate whether the initial model was too simple.

[JohannesBuchner]

Please see https://johannesbuchner.github.io/BXA/tutorial_usage_plotbxa.html for a very nice new plotting class provided by David Homan.