Setting target specific resources in tar_stan_mcmc with future.batchtools

[mrajeev08]

Is it possible to set target specific resources for the targets generated by tar_stan_mcmc?

I'm using future.batchtools on an sge cluster and I'd like to run the mcmc targets as jobs with four cores, but the data, draws, model file targets either as a job with one core or even better set these to deployment = "main".

I've also tried running these as one job with future.callr and tar_make_future(workers = {number_of_cores}). However, I'm a little confused as to how the mcmc parallelization will work then (i.e. using the parallel_chains argument in tar_stan_mcmc). Does stantargets know that the mcmc targets need n cores for the parallel chains, or will those chains end up running sequentially? If targets finish and there are available cores, will the mcmc targets start running chains that have not started yet or will it start other targets?

Apologies if I'm missing some targets specific knowledge here, very new to the targetopia:)

[wlandau]

Is it possible to set target specific resources for the targets generated by tar_stan_mcmc?

I'm using future.batchtools on an sge cluster and I'd like to run the mcmc targets as jobs with four cores, but the data, draws, model file targets either as a job with one core or even better set these to deployment = "main".

This is not currently possible in tar_stan_mcmc(). tar_stan_mcmc() tries to be an oversimplification to make most use cases convenient, and I am not sure how much it would be good to complicate the interface. For what it's worth, tar_make_clustermq() launches persistent workers, so there would be less total job submission/batchtools overhead moving from target to target.

I've also tried running these as one job with future.callr and tar_make_future(workers = {number_of_cores}). However, I'm a little confused as to how the mcmc parallelization will work then (i.e. using the parallel_chains argument in tar_stan_mcmc).

In that local case, targets creates a new R process for each worker, and then Stan launches a new process for each chain in parallel_chains. targets and Stan work independently to do this, and there can be more processes than physical cores.

worker is a new R process, and then the MCMC submits parallel_chains Stan processes to run the chains in parallel.

[mrajeev08]

Thanks for the quick response, makes sense!

Just to confirm, even if I were to use clustermq, all the targets would still have the same number of resources, correct? But there would just be lower overhead because it wouldn't take as many job submissions to get the targets set up? I couldn't quite figure out from the docs how the mcmc chain parallelization would work in this case with persistent workers.

I'm not sure whether I can convince our admin to get zeroMQ installed on our cluster, but we also get warnings when our jobs don't use CPU efficiently, so maybe I can make a case if it does resolve cores sitting idle:)

Thanks again so much!

[wlandau]

Just to confirm, even if I were to use clustermq, all the targets would still have the same number of resources, correct? But there would just be lower overhead because it wouldn't take as many job submissions to get the targets set up?

Yes, exactly.

I couldn't quite figure out from the docs how the mcmc chain parallelization would work in this case with persistent workers.

It's the same as with transient workers. Parallel chains happen across local processes within the target, and this all happens within the SGE job. The difference between persistent and transient workers is that a persistent worker does not shut down after the MCMC is complete, it waits for input from the main controlling process of the pipeline and then runs downstream summary targets as needed.

I'm not sure whether I can convince our admin to get zeroMQ installed on our cluster, but we also get warnings when our jobs don't use CPU efficiently, so maybe I can make a case if it does resolve cores sitting idle:)

Persistent workers do actually increase the risk of idle time on cluster jobs. The pipeline would get done faster because of decreased overhead in submitting jobs, but workers would unfortunately not be more efficient with CPU usage.

[wlandau]

To conserve resources, you could run different sections of the pipeline in stages.

tar_make(contains(c("data", "file")) # Run data and track Stan files locally.
tar_make_future(contains("mcmc")) # Skip the data/files and run the MCMCs on the cluster.
tar_make() # Skip the MCMCs and run the summary targets.

[mrajeev08]

Oh that would actually work perfectly! I guess the summary targets would not run until all the models are done (in this case with multiple models), but given that all the other targets are so lightweight compared to the mcmc runs that seems like a great solution.

Thanks for your quick responses, and the great package, Will!

[mrajeev08]

Marking Will's response below as the answer, as it gets you closest to the goal of target specific resources/conserving compute time on a cluster:

To conserve resources, you could run different sections of the pipeline in stages.

tar_make(contains(c("data", "file")) # Run data and track Stan files locally.
tar_make_future(contains("mcmc")) # Skip the data/files and run the MCMCs on the cluster.
tar_make() # Skip the MCMCs and run the summary targets.