Safely handle parallel/concurrent invocations

[everythingfunctional]

Description

At present, if you invoke fpm on the same project multiple times in parallel, the compiler will trip over itself as it tries to compile the same files at the same time, and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild when they should. If possible, fpm should avoid trying to build a project multiple times in parallel.

Possible Solution

A quick and dirty solution would be to create a "lock file" on fpm startup and delete it on program shutdown. If the "lock file" is already present, just wait until it's removed. It's not flawless, but could be sufficient.

Additional Information

No response

[milancurcic]

I can reproduce. Lock file sounds reasonable to me. I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind. Is there a way to handle that?

[perazz]

Interesting enhancement @everythingfunctional.

I think the proposed solution looks very reasonable. Could that be done on a per-dependency basis? (think something similar to cache.toml). We now have the option for a local registry, so, that has to be dealt with as well.

I think we also want to understand what causes this:

and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild

because we could figure out a fix and force a full rebuild if something went corrupt.

Edit: also see rust-lang/cargo#354

[everythingfunctional]

I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind.

Yep, that's the main danger. If you wanted to get fancy the lock file could contain the PID of the process that created, and then check if that process is still executing, if not delete it and create a new one. Basically, there are solutions.

we also want to understand what causes this

The compiler stepping on its own toes, but updating the timestamp of files anyways. I.e. it looks like the object/mod file got produced after the source code was updated, but it didn't get produced correctly.

[everythingfunctional]

FYI, my use case is trying to get ctest to run my tests in parallel, but there's no way to ensure they're actually compiled without invoking fpm test.