Why samples are not smoothed in ROPE but in Bayesfactor?

[zhengchencai]

Question and context
Hello,

I might be totally wrong here, but it seems that when calculating ROPE using rope() function, posterior samples are used directly and not smoothed. However, in bayesfactor_parameters(), logspline::logspline() is used to smooth the posterior. The computation of posterior odd seems very similar to that of rope, then why in one case smoothing is applied but not the other. Is the purpose of smoothing to make more stable BF calculation? For instance, a and b has a very tiny difference in this toy example. I understand it may not do any difference in practice, but just in case I miss something, is there any reason why not smooth posterior in ROPE? Thanks a lot.

set.seed("1231231231")
posterior <- rnorm(4000, 0.37, 0.37/5)
null <- c(-0.4, 0.4)
a <- 1 - diff(logspline::plogspline(null, f_posterior))
b <- 1 - bayestestR::rope(posterior, range = null, ci = 1, ci_method = "HDI")$ROPE_Percentage

[bwiernik]

I'm not sure the motivation behind the smoothing for BF, but in general, I suggest avoiding procedures with hidden auxiliary assumptions like smoothing, HDI, and density estimation. The samples themselves have asymptotic validity, so it makes sense to use them to compute ROPE percentages, particularly if you are also going to use them directly to estimate centroids and intervals

[zhengchencai]

Thanks @bwiernik and @mattansb, that's what I see from the source code of BF and ROPE and why I ask the question. diff(logspline::plogspline(null, f_posterior)) is copied from BF code. Following @mattansb explanation, do you think we should also then use the same way to calculate ROPE? Since ROPE is really the same as h1_posterior in BF calculation. Sure, ROPE doesn't care about being 0 or 1, but it might be more stable. For instance, I have two distributions to compare, ROPE gives me 1 for both of them, but BF will give me two distinct values. I agree with @bwiernik that we may want to use whatever samples we got from posterior and I certainly think smoothing is not a magic to improve the sampling accuracy, but as @mattansb mentioned, it brings stability when we want to extract a likelihood (point null)/probability (range null) value for comparison purpose, especially when null is around the very tail area we know 4000 sample even 10k is not enough. Thanks again.

[zhengchencai]

Sorry, I guess to be precise, I should say smoothing makes BF and ROPE more "comparable" for the purpose of comparing different distributions. "Stable" is not accurate.

[mattansb]

Sure, ROPE doesn't care about being 0 or 1, but it might be more stable.

If the null is around the tail of the posterior, and you have 10k samples, how unstable can the ROPE be? Somewhere between 0 and 1/10k? If you're ROPE estimate is actually substantially unstable, I would suggest examining the posterior (ESS, Rhat).

If it is stable, no need to add computation complexity.

I should say smoothing makes BF and ROPE more "comparable" for the purpose of comparing different distributions.

BF and ROPE (even when using the same null-region) are not comparable (even if computationally they use similar "bits" of math). One is a question of the posterior and one if a question of the data's compatibility with the prior.

[zhengchencai]

If the null is around the tail of the posterior, and you have 10k samples, how unstable can the ROPE be? Somewhere between 0 and 1/10k? If you're ROPE estimate is actually substantially unstable, I would suggest examining the posterior (ESS, Rhat).

To be honest, I don't have a comprehensive examination of this problem. My fitting diagnostic is fine without any pathology. However, ESS_tail is defiantly not enough accurate with 4000 samples, which means we do need smoothing to help stabilize the calculation, either is ROPE or BF. Otherwise, it may not be realistic to compute 10K samples from Stan. Maybe I should not have overestimated the utility of ROPE or BF, especially in the case of null is in the tail area. Or let's say I am not using it appropriately.

BF and ROPE (even when using the same null-region) are not comparable (even if computationally they use similar "bits" of math). One is a question of the posterior and one if a question of the data's compatibility with the prior.

Yes, it's true their interpretations are different. Sorry for my misleading words.

Thanks a lot for this discussion.

[zhengchencai]

Sorry for multiple replies, I also just realized I am asking a silly question, it seems I care numerical stability here for ROPE and BF, but actually, the questions itself is problematic: even if we do smoothing for ROPE and one calculation gives ROPE = 0.00003 and the other says 0.004, in practice, it really doesn't make a difference comparing to non-smoothing calculations saying ROPE is just 0.