[general] crew might fail when number of targets is large (roughly larger than 1000)

[psychelzh]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I am sorry I cannot provide a simole reprex for this. But for a large project, e.g., my pipeline script, if {crew} is used for parallel analysis, I constantly get the following error:

! Error running targets::tar_make()
Error messages: targets::tar_meta(fields = error, complete_only = TRUE)
Debugging guide: https://books.ropensci.org/targets/debugging.html
How to ask for help: https://books.ropensci.org/targets/help.html
Last error message:
    {crew} worker 15 launched 5 times in a row without completing any tasks. Either troubleshoot or raise launch_max asLast error traceback:

But if I use targets::tar_make_future(), it never fails. I guess it is because {crew} dispatches all the targets though pending.

[wlandau]

Unfortunately we can't say what exactly is happening. If you can provide a simpler runnable reprex, or if you can isolate the problem to crew without using targets, that would help a lot.

[psychelzh]

Thank you very much for your explanation! For my circumstances, it turns out to be the (3) case. After I add tasks_max = 1, it works without error.

[wlandau]

Fantastic! That all makes sense, and I am glad it works for you now.

[wlandau]

For what it's worth, reset_globals = TRUE might be an alternative to tasks_max = 1 for reducing memory costs in some cases, although the latter is still perfectly reasonable and should be safer in some circumstances.

[philiporlando]

Would the same "{crew} worker 15 launched 5 times in a row" error apply when using crew::crew_controller_local()?

[wlandau]

It might, depending on what else is going on in your pipeline. At any rate, it is straightforward to try reset_globals = TRUE and see if it works for you.

[psychelzh]

Recently, I found that if I have roughly 10k targets in a project, and {crew} scheduler will consume a crazy 40 GiB memory. But that is not an issue for the superseded tar_make_future() function. I know there are many advantages using {crew}, for example, better schedule planning, but given superseding means it will cover what the superseded function can do, I am wondering if the similar behavior like tar_make_future() will be possible?

Although it is probably not good practice for having so many targets in a project, the fact is that data analysis goes into many branches very quickly! Therefore, the behavior like tar_make_future() could sometimes save our life a lot. Or can there be more efficient memory compressing for {crew}.

[psychelzh]

Or following the logic that superseded functions will never go away, I should just switch to tar_make_future() for such circumstances?

[wlandau]

Either should be fine. The memory issue could be due to shikokuchuo/mirai#55 (comment), in which case updating to development crew may fix it.

[wlandau]

storage = "worker", retrieval = "worker", garbage_collection = TRUE, and memory = "transient" should also reduce memory usage.