Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add automatic convergence-checks #140

Open
1 of 3 tasks
AdamCSmithCWS opened this issue Oct 15, 2019 · 4 comments
Open
1 of 3 tasks

Add automatic convergence-checks #140

AdamCSmithCWS opened this issue Oct 15, 2019 · 4 comments
Assignees

Comments

@AdamCSmithCWS
Copy link
Collaborator

AdamCSmithCWS commented Oct 15, 2019

I think we should have the run_model function include some automatic checks on, at least, cross-chain convergence.
Initial thoughts on options:

  • 1. automatically summarize the parameters that fail some threshold Rhat value. .e.g, return a list of the parameter names and Rhat values, as well as throw a warning that says "n parameters failed to converge across chains, check your model results carefully and consider a longer burn-in".
  • 2. automatically extend the model run for another n_iterations and re-sample the posterior, if the number of parameters that fail the Rhat test is >~ %5 of the monitored parameters.
  • 3. add the extended model run as an option in the run_model function (e.g., auto_converge = TRUE)
@AdamCSmithCWS
Copy link
Collaborator Author

first suggestion is now integrated into the run-model function.
second is still a good idea
third might be dangerous (extremely long model runs for 1 or 2 extreme parameters)

@BrandonEdwards
Copy link
Collaborator

Just circling back on some outstanding issues. I think for (3), it might make sense to include an auto_converge = TRUE, but have an upper limit for the number of re-runs. We could set that by default, and/or allow the user to set it. I'm happy to take on that task, I'll assign myself here.

@BrandonEdwards BrandonEdwards self-assigned this Dec 15, 2020
@AdamCSmithCWS
Copy link
Collaborator Author

We should also incorporate the posterior package, which estimates an updated version of Rhat. The updated Rhat is much less sensitive to asymmetrical posteriors, such as those that are common for the n-values in these models.

Vehtari A., Gelman A., Simpson D., Carpenter B., & Bürkner P. C. (2020). Rank-normalization, folding, and localization: An improved Rhat for assessing convergence of MCMC. Bayesian Analysis.

@BrandonEdwards
Copy link
Collaborator

@AdamCSmithCWS Good idea re posterior package! Are you planning on adding that in on your end and submitting a PR, or would you like me to cover that? Doesn't matter to me!

Also, with that updated way of calculating Rhat, does it now make it less "dangerous" to have that 3rd option be implemented, where the model continues to run until fully converged (again,perhaps up until a certain number of total iterations)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants