Question: is coro thread safe?

[rkrug]

I have the following code, where the generator and the looping over the generated function is executed in parallel for different values.

now the lockfile (indicated in the code) is sometimes not deleted after the completion.

Therefore my question: is the usage of coro in a parallel processin=g environment safe, i.e. is cor thread safe?

Thanks,

Rainer

pbmcapply::pbmclapply(
            years,
            function(y) {

                    output_path <- file.path(SOME_BASEDIR, y).    ######### <<<<<------- EDIT HERE
                    oar <- openalexR::oa_generate(
                        ...
                    )

                # set <- vector("list", set_size)
                set <- NULL
                set_no <- 0

                file.create(file.path(output_path, "00_in_progress_00")). ## <- process lock file
                coro::loop(
                    for (x in oar) {
                        set <- c(set, list(x))
                        if ((length(set) >= set_size) | isTRUE(x == coro::exhausted())) {
                            saveRDS(set, file.path(output_path, paste0("set_", set_no, ".rds")))
                            # set <- vector("list", set_size) # reset recs
                            set <- list()
                            set_no <- set_no + 1
                        }
                    }
                )
                ### and save the last one
                saveRDS(set, file.path(output_path, paste0("set_", set_no, ".rds")))
                file.create(file.path(output_path, "00_complete_00"))
                file.rename(
                    file.path(output_path, "00_in_progress_00"),
                    file.path(output_path, "00_complete_00")
                )
            },
            mc.cores = mc_cores,
            mc.preschedule = FALSE
        )

[lionel-]

From a cursory look, if pbmclapply() runs the callback in parallel, it seems to me you have race conditions involved in the creation and deletion of the lock file? I mean in the parts that don't involve coro at all.

(of course coro is threadsafe but I'm not sure what you mean by this)

[rkrug]

Sorry - one point is missing in the example, i.e. that output_path is different for each y (see EDIT HERE in the edited example). If there is still a race condition, I am missing it.

[lionel-]

I'm not sure how this code works or what it tries to achieve and I can't help you without an actual reprex. (Even with one I'm a bit swamped currently.) But I don't think coro is the culprit here.

[rkrug]

OK - thanks. I will look at it and come back to you when needed.