Hello @mcmrc! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2024-05-25 14:34:36 UTC

about binomtest confidence intervals.

scipy stats has now also confint and alternative options (which it did not have when we added those in statsmodels)

AFAICS, the one sided binom test confints are the same as the beta. (although scipy does rootfinding also in the one-sided case)
unit tests for binomtest confidence interval in scipy with reference numbers from R
https://github.com/scipy/scipy/blob/main/scipy/stats/tests/test_morestats.py#L920

notation/definition
we use the "alternative" keyword also for confints, with larger or smaller for the outside/alternative region

boundary points
the one-sided intervals should have 0 or 1 for the "open" side instead of nan

Thanks for the PR

Thank you so much for all the comments.
I will do

• calculate reference numbers from R
• change the argument name to alternative and options to larger and smaller
• use 1 or 0 for open-side.

great

and add the one-sided intervals to binom test method.
Based on a quick check, I get the same results in scipy as using one of the sides of the beta intervals.
(AFAIR, those are exact in the one-sample case)

jeffreys looks fine to me, it could be replace by isf and ppf in the two one-sided cases. but your current code using interval just does a little bit of redundant work.
(scipy distribution interval method is "central", equal tail interval, so standard pattern with doubling alpha still holds.

It seems like the outputs from the current version of statsmodels and R 4.3.3 do not match.

statsmodels: proportion_confint(100, 200, method='binom_test') -> (0.42988371451156804, 0.5701162853923397)
R: binom.test(100, 200, conf.level=1-0.05)$conf.int -> [1] 0.4286584 0.5713416

Do you have any ideas?

The problem is base R,
they compute the central, equal tail confidence interval even though the hypothesis test is minlike

proportion_confint(100, 200, method='beta')
(0.42865843090683226, 0.5713415690931678)

There is an article that points this out and an R package that provides matching confidence interval
after looking for it: it's exactci, Fay, https://cran.r-project.org/web/packages/exactci/exactci.pdf
e.g. #8233

for method binomtest you need a unit test separate from the other methods, because you cannot check that the one-sided interval can be obtained by doubling alpha.

Thank you, Josef. Understood. scipy.stats.binomtest is not based on a distribution with equal tails.
The past issue is very educative. I'll keep on working on it.

minlike and central/equal-tail are based on the same distribution, loglikelihood
the equal-tail requires prob <= alpha / 2 in each tail
minlike provides the shortest interval (in uni-modal distributions) but only controls overall coverage and does not control coverage in each tail.

In symmetric distributions like normal, t, both methods are the same, but not in asymmetric distributions that we have in proportions and rates (Binomial and Poisson).

(Coverage in approximate methods can still be slightly offcenter because of the approximation, actual tail probabilites are only approximately alpha/2)

I committed new codes.

• calculated reference numbers from R (BinomCI, binom.test and exactci)
• changed the argument name to alternative and options to smaller and larger
• use 0 or 1 for open-side.
• updated binom_test method with alternative="two-sided"
• diverted method from beta to one-side binom_test
• simplified jeffreys method using ppf and isf
• refactored test file

Sorry, I've forgotten to remove redundant parts. Please review the latest one.

looks good overall
alternative has maybe larger and smaller switched
I need to go over some details later. (They require thinking which would distract me from my own PR)

Thank you so much for your time.
It is really confusing which side is "larger" or "smaller"; sorry for my mistake.
I read this docs and used "smaller" as equivalent as "less."

However, in your example:
alternative true probability is less than 0.5, but values less than 0.5 are inside the confidence interval, and not in the alternative rejection region.

I will have to think about this later again. I'm also always getting confused.

something like: at the upper limit of the left (lower) interval we reject the null hypothesis in favor of the alternative hypothesis that the proportion is larger than the value with a pvalue = alpha (in continuous case, and weak inequality in discrete case)
But I need to verify this.

AFAICS, my intuition was correct
for example https://www.itl.nist.gov/div898/handbook/eda/section3/eda352.htm

in our (my definition in statsmodels) the one-sided alternative also corresponds to the one-sided alternative in the hypothesis test.
If the true value is in/inside the confidence interval, then we reject the null hypothesis in favor of the alternative hypothesis with probability less than alpha in repeated sampling.

This means, the rejection region is in the direction of the alternative.
For the lower tail confidence interval, the rejection region is in the direction of the "larger" alternative.
For the upper tail confidence interval, the rejection region is in the direction of the "smaller"or "less" alternative.

I see that. So, the word 'alternative' corresponds to the reject region.

Can you squash this? earlier commits are not relevant

final version looks good to me and can be merged

Thanks for the PR

I've been always helped by statsmodels in my research.
Thank you so much for your time and effort, Josef.

unrelated unit test failure, merging

If you are interested, then you could add alternative in the same way to poisson rates
https://www.statsmodels.org/dev/generated/statsmodels.stats.rates.confint_poisson.html

Those are all central, so unit test could be to just compare with the two-sided interval.

Let me take that.
I created a new issue and will make a PR.
#9254