mvn package followed by mvn test with GIB enabled is not able to resolve dependencies #599

debraj-manna · 2022-11-05T19:38:17Z

debraj-manna
Nov 5, 2022

Our project structure

parent/pom.xml

<properties>
	<gib.disable>true</gib.disable>
</properties/>
<modules>
	<module>anlytcs</module>
	<module>gossiper</module>
	<module>proto</module>
	<module>payment</module>
	...
</modules>

anlytcs/pom.xml

<dependencies>
    <dependency>
      <groupId>com.spotnana</groupId>
      <artifactId>gossiper</artifactId>
      <version>0.0.1-SNAPSHOT</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>com.spotnana</groupId>
      <artifactId>proto</artifactId>
      <version>0.0.1-SNAPSHOT</version>
      <scope>compile</scope>
    </dependency>
    ...
</dependencies>

I am first doing mvn clean package -DskipTests -T 8 -Pdev-local -Dmaven.repo.local='/tmp/m2_dir'

and then mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -Dgib.disable=false -o which is throwing the below error

debrajmanna@debrajmanna-DX6QR261G3 java % mvn test -Dmaven.repo.local='/tmp/m2_dir' -Dgib.disable=false
[INFO] Scanning for projects...
[INFO] gitflow-incremental-builder 4.2.0 starting...
[INFO] Reference commit of local base branch 'HEAD' has id: commit 0f5747dfc4119f63455911eca850a305a098fac8 1667674410 ------p
[INFO] Reference commit of local reference branch 'refs/heads/master' has id: commit 62f168e2181f4050ea4da900fb12275edc8ff406 1667653367 ------p
[INFO] Using merge base of id: commit 62f168e2181f4050ea4da900fb12275edc8ff406 1667653367 --tr-sp
[INFO] ------------------------------------------------------------------------
[INFO] Changed Artifacts:
[INFO]
[INFO] anlytcs
[INFO] supplier
[INFO]
[INFO] gitflow-incremental-builder exiting...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 13.0
[INFO] os.detected.version.major: 13
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] anlytcs                                                            [jar]
[INFO] supplier                                                           [jar]
[INFO] coverage                                                           [jar]
[INFO]
[INFO] ------------------------< com.spotnana:anlytcs >------------------------
[INFO] Building anlytcs 0.0.1-SNAPSHOT                                    [1/3]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for anlytcs 0.0.1-SNAPSHOT:
[INFO]
[INFO] anlytcs ............................................ FAILURE [  0.432 s]
[INFO] supplier ........................................... SKIPPED
[INFO] coverage ........................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.641 s
[INFO] Finished at: 2022-11-06T00:48:33+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project anlytcs: Could not resolve dependencies for project com.spotnana:anlytcs:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: com.spotnana:gossiper:jar:0.0.1-SNAPSHOT, com.spotnana:proto:jar:0.0.1-SNAPSHOT, com.spotnana:config:jar:0.0.1-SNAPSHOT, com.spotnana:common:jar:0.0.1-SNAPSHOT, com.spotnana:testing-common:jar:0.0.1-SNAPSHOT, com.spotnana:db-common:jar:0.0.1-SNAPSHOT, com.spotnana.obt:obt-common:jar:0.0.1-SNAPSHOT, com.spotnana:athena:jar:0.0.1-SNAPSHOT: com.spotnana:gossiper:jar:0.0.1-SNAPSHOT was not found in https://maven.pkg.github.com/Spotnana-Tech/codegen during a previous attempt. This failure was cached in the local repository and resolution is not reattempted until the update interval of github has elapsed or updates are forced -> [Help 1]

If I do mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -o it works. So some problem when I am enabling GIB.

Can someone suggest what is going wrong?

debraj-manna · 2022-11-05T20:26:43Z

debraj-manna
Nov 5, 2022
Author

One way I get around this is to not do package and use mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -Dgib.disable=false -Dgib.buildUpstream=true but this is executing all tests of the upstream modules. I tried setting -Dgib.skipTestsForUpstreamModules=true so that it only compiles but not execute the tests of upstream modules. It is not working.

@famod Any suggestions?

1 reply

famod Nov 5, 2022
Maintainer

So when anlytcs is changed and you enable buildUpstream and skipTestsForUpstreamModules, tests in e.g. gossiper are executed?
This shoudn't happen!

What happens when you build gossiper with -DskipTests (without GIB)?

famod · 2022-11-05T23:31:59Z

famod
Nov 5, 2022
Maintainer

If I do mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -o it works. So some problem when I am enabling GIB.

This is to be expected: You are not installing the artifacts into the local repo so they are not available in the second run which is only a partial run (thanks to GIB).
You need to use install in the first run, not just package.
The same issue would happen if you'd run mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir -pl anlytcs (after just running package).

Alternatively, -am or buildUpstream=true (as you've already tried) should work too, or even buildAll.

0 replies

debraj-manna · 2022-11-06T05:28:58Z

debraj-manna
Nov 6, 2022
Author

@famod Regarding buildUpstream and skipTestsForUpstreamModules what is happening is if I specify everything via system variable in command line like below. It is working as expected and skipping the tests for upstream module

mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -Dgib.disable=false -Dgib.buildUpstream=true -Dgib.skipTestsForUpstreamModules=true -o

But if I specify -Dgib.disable=false as system variable in command line and buildUpstream and skipTestsForUpstreamModules in pom like below then it is running the tests for upstream modules.

<properties>
     <gib.disable>true</gib.disable>
    <gib.skipTestsForUpstreamModules>true</gib.skipTestsForUpstreamModules>
    <gib.buildUpstream>true</gib.buildUpstream>
</properties>

Below are some of the relevant build logs showing the same.

debrajmanna@debrajmanna-DX6QR261G3 java % mvn -T 5 test -Dmaven.repo.local='/tmp/m2_dir' -Dgib.disable=false -o
[INFO] Scanning for projects...
[INFO] gitflow-incremental-builder 4.2.0 starting...
[INFO] Reference commit of local base branch 'HEAD' has id: commit e98fbab65f7dce08fa31ed4ea7c7ab5b05dd1bc5 1667676222 ------p
[INFO] Reference commit of local reference branch 'refs/heads/master' has id: commit 62f168e2181f4050ea4da900fb12275edc8ff406 1667653367 ------p
[INFO] Using merge base of id: commit 62f168e2181f4050ea4da900fb12275edc8ff406 1667653367 --tr-sp
[INFO] ------------------------------------------------------------------------
[INFO] Changed Artifacts:
[INFO]
[INFO] spotnana-parent
[INFO] anlytcs
[INFO] supplier
[INFO]
[INFO] gitflow-incremental-builder exiting...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 13.0
[INFO] os.detected.version.major: 13
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] spotnana-parent                                                    [pom]
[INFO] com.spotnana.proto                                                 [jar]
[INFO] config                                                             [jar]
[INFO] testing-common                                                     [jar]
[INFO] common                                                             [jar]
[INFO] db-common                                                          [jar]
[INFO] common-testing                                                     [jar]
[INFO] gossiper                                                           [jar]
[INFO] mock-servers                                                       [jar]
[INFO] obt-common                                                         [jar]
[INFO] athena                                                             [jar]
[INFO] anlytcs                                                            [jar]
[INFO] assets                                                             [jar]
[INFO] bckofc                                                             [jar]
[INFO] directory                                                          [jar]
[INFO] elasticsearch                                                      [jar]
[INFO] elasticsearch-testing                                              [jar]
[INFO] ens-common                                                         [jar]
[INFO] ens-channel-providers                                              [jar]
[INFO] ens-testing-common                                                 [jar]
[INFO] ens-channel-processors                                             [jar]
[INFO] ens-grpc                                                           [jar]
[INFO] profile-uploads                                                    [jar]
[INFO] nwatch                                                             [jar]
[INFO] objsearch                                                          [jar]
[INFO] objstore                                                           [jar]
[INFO] agent                                                              [jar]
[INFO] agent-client-lib                                                   [jar]
[INFO] autocmp                                                            [jar]
[INFO] brex-client                                                        [jar]
[INFO] bengine                                                            [jar]
[INFO] cengine                                                            [jar]
[INFO] intdbg                                                             [jar]
[INFO] sage-common                                                        [jar]
[INFO] master                                                             [jar]
[INFO] supplier                                                           [jar]
[INFO] stripe-client                                                      [jar]
[INFO] conferma-client                                                    [jar]
[INFO] profile                                                            [jar]
[INFO] sagev2                                                             [jar]
[INFO] payment                                                            [jar]
[INFO] coverage                                                           [jar]
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 5
....
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] --------------------< com.spotnana:testing-common >---------------------
[INFO] Building testing-common 0.0.1-SNAPSHOT                            [6/42]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
...
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running com.spotnana.testing.common.FieldConstructorTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.401 s - in com.spotnana.testing.common.FieldConstructorTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Is this expected?

2 replies

famod Nov 6, 2022
Maintainer

This caught my attention:

[INFO] Changed Artifacts:
[INFO]
[INFO] spotnana-parent

If your changes to parent pom.xml (the addition of the GIB properties) are not committed yet or the commit is not already contained in the reference branch, then this is expected because...well...you changed parent pom.xml und everything has the parent as upstream.

debraj-manna Nov 7, 2022
Author

Aah right. That was the issue. My changes for gib.buildUpstream and gib.skipTestsForUpstreamModules are not already committed in the reference branch. I was testing these parameters in the child branch.

debraj-manna · 2022-11-11T17:53:48Z

debraj-manna
Nov 11, 2022
Author

@famod - I am still facing some issues with dependency resolution or my understanding may be wrong.

Dependency is like this

payment
	stripe-client
	db-common
	athena

profile
	stripe-client
	db-common
	athena
	...

pom.xml has the following gib options

<gib.skipTestsForUpstreamModules>true</gib.skipTestsForUpstreamModules>
<gib.buildUpstream>true</gib.buildUpstream>

Below are the relevant logs from mvn test

debrajmanna@debrajmanna-DX6QR261G3 java % mvn -T 5 test -Dmaven.repo.local="/tmp/temp_m2" -Dgib.disable=false -Dgib.referenceBranch="refs/heads/master" 
[INFO] Scanning for projects...
[INFO] gitflow-incremental-builder 4.2.0 starting...
[INFO] Reference commit of local base branch 'HEAD' has id: commit ece1bb28eb7d2059be51ecd439c43ec1b8dc2082 1668185584 ------p
[INFO] Reference commit of local reference branch 'refs/heads/master' has id: commit b631e4e81b72cbf2e53aaf6e91dce7738def0a15 1668186592 ------p
[INFO] Using merge base of id: commit 9359e16f4394b9ba32d1039a121d5742c6b25af6 1668176766 --tr-sp
[INFO] ------------------------------------------------------------------------
[INFO] Changed Artifacts:
[INFO]
[INFO] stripe-client
[INFO]
[INFO] gitflow-incremental-builder exiting...
[INFO] ------------------------------------------------------------------------
[INFO] Detecting the operating system and CPU architecture
[INFO] ------------------------------------------------------------------------
[INFO] os.detected.name: osx
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 13.0
[INFO] os.detected.version.major: 13
[INFO] os.detected.version.minor: 0
[INFO] os.detected.classifier: osx-aarch_64
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] spotnana-parent                                                    [pom]
[INFO] com.spotnana.proto                                                 [jar]
[INFO] config                                                             [jar]
[INFO] testing-common                                                     [jar]
[INFO] common                                                             [jar]
[INFO] stripe-client                                                      [jar]
[INFO] profile                                                            [jar]
[INFO] payment                                                            [jar]
[INFO] coverage                                                           [jar]
[INFO]
[INFO] Using the MultiThreadedBuilder implementation with a thread count of 5
[INFO]
[INFO] --------------------< org.spotnana:spotnana-parent >--------------------
[INFO] Building spotnana-parent 1.0                                       [1/9]
[INFO] --------------------------------[ pom ]---------------------------------

GIB identified correctly ran the tests of stripe-client But on trying to build payment and profile it is failing with the below error

[ERROR] Failed to execute goal on project payment: Could not resolve dependencies for project com.spotnana:payment:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: com.spotnana:db-common:jar:0.0.1-SNAPSHOT, com.spotnana:athena:jar:0.0.1-SNAPSHOT: Cannot access github (https://maven.pkg.github.com/Spotnana-Tech/codegen) in offline mode and the artifact com.spotnana:db-common:jar:0.0.1-SNAPSHOT has not been downloaded from it before. -> [Help 1]
[ERROR] Failed to execute goal on project profile: Could not resolve dependencies for project com.spotnana:profile:jar:0.0.1-SNAPSHOT: The following artifacts could not be resolved: com.spotnana:db-common:jar:0.0.1-SNAPSHOT, com.spotnana:athena:jar:0.0.1-SNAPSHOT: Cannot access github (https://maven.pkg.github.com/Spotnana-Tech/codegen) in offline mode and the artifact com.spotnana:gossiper:jar:0.0.1-SNAPSHOT has not been downloaded from it before. -> [Help 1]

Should not the dependency of profile, and payment be compiled as well?

3 replies

famod Nov 12, 2022
Maintainer

It's complaining that it cannot find the jars of db-common and athena and in fact, those module are not part of the reactor (judging from the output you posted).
If those modules have not been installed before into the local repo then this error is to be expected.

If a pre-install is not an option (and I see good reasons for that being problematic), then I'd suggest using buildAll instead of buildUpstream and package instead of test. This way the reactor will have the jars of db-common and athena once they have been built.

Failed to execute goal on project ...

Can you check which goal excatly?

debraj-manna Nov 12, 2022
Author

Yes with buildAll it is working fine.

One query for my understanding db-common & athena are dependencies of profile & payment and not of stripe-client. So when profile & payment are identified to be built should not all the dependencies of profile & payment be built as well, with buildUpstream set to true and buildAll set to false?

famod Nov 12, 2022
Maintainer

You should actually be able to achieve that via https://github.com/gitflow-incremental-builder/gitflow-incremental-builder#gibbuildupstreammode set to impacted (and back to buildUpstream instead of buildAll).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mvn package followed by mvn test with GIB enabled is not able to resolve dependencies #599

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

mvn package followed by mvn test with GIB enabled is not able to resolve dependencies #599

debraj-manna Nov 5, 2022

Replies: 4 comments · 6 replies

debraj-manna Nov 5, 2022 Author

famod Nov 5, 2022 Maintainer

famod Nov 5, 2022 Maintainer

debraj-manna Nov 6, 2022 Author

famod Nov 6, 2022 Maintainer

debraj-manna Nov 7, 2022 Author

debraj-manna Nov 11, 2022 Author

famod Nov 12, 2022 Maintainer

debraj-manna Nov 12, 2022 Author

famod Nov 12, 2022 Maintainer

debraj-manna
Nov 5, 2022

Replies: 4 comments 6 replies

debraj-manna
Nov 5, 2022
Author

famod Nov 5, 2022
Maintainer

famod
Nov 5, 2022
Maintainer

debraj-manna
Nov 6, 2022
Author

famod Nov 6, 2022
Maintainer

debraj-manna Nov 7, 2022
Author

debraj-manna
Nov 11, 2022
Author

famod Nov 12, 2022
Maintainer

debraj-manna Nov 12, 2022
Author

famod Nov 12, 2022
Maintainer