-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is mvnd "safe" for parallel builds where "mvn" alone is not? #896
Comments
First, please forget the "built in" parallel builder, use instead this https://github.com/takari/takari-smart-builder (same is used by mvnd). Second, did you try maven 3.9.x (preferably latest) version? You could use file locking for start (if local repository is on local FS) |
Thanks for suggestions, checked that the workers used maven 3.8.6... So change to 3.9.x could just fix the situation while keeping all those build calls independent as they are now? |
Maven 3.9 introduced "locks" for local repository, trying to solve exactly that: shared access to local repository from multiple processes... So best would be to try it out (hopefully using some sound FS like ext4 or alike, no windows in picture). |
Oh that funny moment when the huge internet looks like a small village: https://www.mail-archive.com/users@maven.apache.org/msg144072.html
So looking for some best way among several possibilities, to pass |
create (and check in into SCM) a file in project like this:
with contents
PS: plz double check this above, is from top of my head https://maven.apache.org/configure.html#mvn-maven-config-file |
Thanks for the options, my first shot missed the "file-gav" part ;) Putting them into SCM as part of the components' source seems a bit like overkill... they should be buildable anywhere (and maybe with different environmental settings), right? For posterity and my own back-tracking, I'll be exploring the |
While at it, I am trying to wrap my head around the https://maven.apache.org/resolver/local-repository.html#split-local-repository feature. Is there some trick that would allow several CI builds to share downloaded third-party artifacts but let them store separately and use without conflict some artifacts from designated "our" namespaces? I saw suggestions about e.g. |
And circling back to this repository's topic, with some new knowledge in mind - does current A large part of my team members' question boils down to whether we can replace the original maven by Originally this idea to use |
Split local repository is new feature, but you have to be aware that in Maven 3 land not all plugins "play nice" with it, see https://issues.apache.org/jira/browse/MNG-7706 In short, if using Maven 3.9.x and you do not see "plugin validation warnings" (see https://maven.apache.org/guides/plugins/validation/index.html), then you should be pretty much okay (but still no 100% guarantee). Best is to try locally and see first (so "lab testing" is what I'd recommend). Moreover, as split local repository does know to be a bit "mind boggling", I'd really even more recommend to play with it locally (on dev workstation), and when all in place, and if all OK, apply that to CI. mvnd := Split repo for that use case is next on my roadmap, but gonna happen only in Resolver 2.0 (so Maven 4 final), not in Maven 3.9.x lifecycle. Currently, "split" can only split based as documented: origin remote repo and cached vs install. Current goal of it was "branched development" (one local repo shared with several maven processes building same project but different branches of it). |
Related context:
Originally posted by @ppalaga in #498 (comment)
My question is rather this: we have CI jobs running
mvn
tests under the same user account on the same agent - whether for fully independent jobs, or for different parallel stages of the same job (spawning a burst of tests for different components etc.)This approach did hiccup with unexpected errors which we assume to be the cross-maven corruption of the same local repository, when downloads happened to run simultaneously and fetched the same files. We theorized of a few possible solutions, no idea if any are viable though (hence the question):
mvnd
, hoping it would take care of parallelly-requested downloads to same local repo in a sane manner (even if sequentializing them internally, etc.) and transparently formvn
"client" calls;mvn validate
before each longer build/test (assuming this would fetch all needed files) as the sequential operation, then do actualmvn test/compile/package/etc.
as the parallel operations which rely on files fetched safely before.Any insights would be most welcome :)
The text was updated successfully, but these errors were encountered: