Goodness of fit

[maxgalli]

Hi, first of all congratulation for this nice package. I recently started trying it out and I find it really handy, especially coming from a RooFit background.
I write here asking for something that I might have missed from the documentation: are there any methods that can be used to get useful values like e.g. the chi-square and the p-value (as probability of the reduced chi-square)?

Thanks in advance,

Massimiliano

[jonas-eschle]

Hi, many thanks, we're glad to hear that!

There is currently nothing like chi2 (but can be done manually simple), but there is hepstats, a library using the zfit interface which does the inference including different ways of computing p-values, limits etc.

Does that suit for your case?

[jonas-eschle]

Hi @maxgalli , just wondering if hepstats did what you wanted? Or did you encounter any problems?

[maxgalli]

hi @mayou36, sorry for the super late reply, I didn't see the notification for some reason.
I took a very quick look at hepstats but I didn't immediately find an example of what I was trying to achieve. To be fair, I didn't strictly need it in that moment, so I will for sure dig deeper when I will have the real necessity :)

[jonas-eschle]

Sure, just let me know in case it fits your need or also not, it is also good to know for future plans for the library

[angelcampoverde]

Hi,

I need the same and my suggestion is, implement something like:

import zfit
import numpy

arr = numpy.random.normal(0, 1, size=10000)

obs = zfit.Space('m', limits=(-10, 10))
mu  = zfit.Parameter("mu", 0.4, -5, 5)
sg  = zfit.Parameter("sg", 1.3,  0, 5)

pdf = zfit.pdf.Gauss(obs=obs, mu=mu, sigma=sg)

dat=zfit.Data.from_numpy(obs=obs, array=arr)
nll=zfit.loss.UnbinnedNLL(model=pdf, data=dat)
mnm=zfit.minimize.Minuit()

res=mnm.minimize(nll)

with zfit.run.set_graph_mode(False):
    res.hesse(name="hesse_np")

chi2, ndof, pvalue = res['chi2']

i.e make sure that the chi2 ends up, together with the number of degrees of freedom and the pvalue in a tuple that can be retrieved from the results object. We can do this ourselves, but it must be part of the library. Also this information should be printable along side the rest of the fit info:

I am writing a bunch of utility functions that fill this gap, as well as others, for now. But as you can imagine, that takes my time and I have many other stuff to do, beyond calculating chi2s.

Cheers.

[jonas-eschle]

Hi @angelcampoverde , thanks, glad to hear that this is a wanted feature and thanks for your efforts!

To calculate chi2, I would propose to use the builtin chi2 loss (maybe make sure that constant optimization is turned off, i.e. options takes an argument subtr_const or similar), that's easier. With get_params, you'd get the number of free parameters directly.

The point in adding this is a bit the scope: goodness of fit could easily be it's own repo. Chi2 is a rather trivial one, but there are many others and it can easily get out of hands. But I also see the convenience of having this information in the FitResult (and we maybe need more of an "inference result" that can contain a lot more information.

hepstats offers some of these, i.e. the p-value calculation with the likelihood profile, the asymptotic solution, or with actual toys. That is maybe the best place to add these functions.