Why is tar_stan_mcmc_rep not recommended for direct use?

[mike-lawrence]

I see there's separate tar_stan_mcmc_rep_summary and tar_stan_mcmc_rep_diagnostics functions, but when doing SBC it would make sense to use the diagnostics to filter/condition the summary info. This would be a case where it could make sense for the user to use tar_stan_mcmc_rep directly, right?

[wlandau]

tar_stan_mcmc_rep() can only return one kind of output at a time (the output argument must be a scalar if supplied). So in its current state I do not think it could accomplish this.

it would make sense to use the diagnostics to filter/condition the summary info

I had not considered this, and I still do not think I see the rationale. Would you elaborate?

when doing SBC

When I wrote the original docs, I defined SBC incorrectly. I am sorry if that was initially misleading. I do not see how to do SBC with the current summaries because it involves computing a rank statistic for each parameter, which requires matching .join_data to each parameter's draws before computing the actual summaries. I posted stan-dev/posterior#111 with hopes of handling SBC.

[mike-lawrence]

Ah, I misread the tar_stan_mcmc_rep docs; I thought it was providing as output the same multi-targets as tar_stan_mcmc. The idea is that two models may differ in both their rate of failing a given diagnostic and in their performance as captured by the summary, so it would be useful to have both pieces of information per combination of model and rep.

[wlandau]

I believe this is already planned for posterior: stan-dev/posterior#2 (comment). If/when posterior can handle HMC diagnostics, you would be able to select them in the summaries and summary_args arguments to tar_stan_mcmc_rep_summary(), both of which are already forwarded to posterior::summarize_draws().

[mattocci27]

So, if we want to see both summaries and diagnostics (e.g., divergent transitions) from multiple simulated datasets, do we need to run both tar_stan_mcmc_rep_summary and tar_stan_mcmc_rep_diagnostics?

[wlandau]

You could do it that way if you are okay with the summaries and diagnostics coming from the same datasets. Otherwise, you could use tarchetypes::tar_map_rep() directly and your own code that calls cmdstanr.

[mattocci27]

Thanks, tarchetypes::tar_map_rep() worked! I was able to add a column for num_divergent from cmdstanr::diagnostic_summary to a data frame of summaries. It was much easier to use tarchetypes::tar_map_rep() than I expected!