Added system_lib for system prestage jars #5239

annaibm · 2024-04-18T13:01:57Z

Prestage system test prequisities to system_lib

related: #4912

annaibm · 2024-04-18T13:02:38Z

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39879/consoleFull
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39882/console
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/39906/consoleFull

llxia · 2024-04-18T17:28:52Z

pre-stage jars are not used. asm.jar still got downloaded to /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm/:

00:08:53.766  configure-asm:
00:08:53.766       [echo] Executing macro download-file
00:08:53.766       [echo] File to download: https://repository.ow2.org/nexus/content/repositories/releases/org/ow2/asm/asm/9.0/asm-9.0.jar
00:08:53.766       [echo] Destination: /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm/asm.jar
00:08:53.766       [echo] Download tool: curl
00:08:53.766      [mkdir] Created dir: /home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs/asm
00:08:53.766       [exec]   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
00:08:53.766       [exec]                                  Dload  Upload   Total   Spent    Left  Speed
00:08:53.766       [exec] 
00:08:54.639       [exec]   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0100  118k  100  118k    0     0   114k      0  0:00:01  0:00:01 --:--:--  114k
00:08:54.639

I suspect that we need to update the reference in system test repo.

llxia · 2024-04-18T18:12:27Z

Try change this line

aqa-tests/system/system.mk

Line 77 in 3ae4045

-systemtest-prereqs=$(Q)$(SYSTEMTEST_RESROOT)$(D)systemtest_prereqs$(Q) \

llxia · 2024-05-10T19:30:44Z

system/system.mk

@@ -71,15 +71,21 @@ ifdef JVM_OPTIONS
  APPLICATION_OPTIONS := $(APPLICATION_OPTIONS) -jvmArgs $(Q)$(JVM_OPTIONS)$(Q)
 endif

+ifndef SYSTEM_LIB_DIR
+  SYSTEM_LIB_DIR := "/home/jenkins/workspace/Grinder/aqa-tests/systemtest_prereqs"


Try to set SYSTEM_LIB_DIR :=$(SYSTEMTEST_RESROOT)/systemtest_prereqs

llxia · 2024-05-10T19:33:03Z

system/common.xml

@@ -337,7 +343,7 @@
 				</then>
 		</if>
 		<copy todir="${SYSTEMTEST_DEST}/systemtest_prereqs/">
-			<fileset dir="${TEST_ROOT}/systemtest_prereqs/" includes="**" />
+			<fileset dir="${system_lib_dir}" includes="**"/>


Why do we still need to copy to systemtest_prereqs dir?

No, we don't need to copy to the systemtest_prereqs directory. I had kept it to check if local tests might need it; apparently, it's not needed. I will remove it.

llxia · 2024-05-10T19:33:41Z

buildenv/jenkins/JenkinsfileBase

+					sh "perl ./aqa-tests/TKG/scripts/getDependencies.pl -path ${env.LIB_DIR} -task default -customUrl ${customUrl}"
+				} else {
+					sh "perl ./aqa-tests/TKG/scripts/getDependencies.pl -path ${env.LIB_DIR} -task default"
+				}


for system tests, we do not need customUrl?

Passing the customUrl for system tests was duplicating the process of downloading the jar files.

why having customUrl will duplicate the process? When running at openj9 Jenkins, we should try to use systemtest.getDependency from openj9 Jenkins, not Adoptium.

llxia · 2024-05-10T19:38:13Z

buildenv/jenkins/JenkinsfileBase

+				def defaultUrl = "https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/"
+				def systemUrl = "https://ci.adoptium.net/job/systemtest.getDependency/lastSuccessfulBuild/artifact/"
+				env.LIB_DIR = (env.BUILD_LIST == 'system') ? env.SYSTEM_LIB_DIR : env.LIB_DIR
+				def customUrl = (env.BUILD_LIST == 'system') ? systemUrl : defaultUrl


I would prefer to have if statement to make the logic clear:

if (env.BUILD_LIST == 'system') { env.LIB_DIR = env.SYSTEM_LIB_DIR customUrl = "https://ci.adoptium.net/job/systemtest.getDependency/lastSuccessfulBuild/artifact/" } else { customUrl = "https://ci.adoptium.net/job/test.getDependency/lastSuccessfulBuild/artifact/" }

Please note this change will conflict with #5289. Depends on which PR gets merged first, the other one will need to be updated.

annaibm · 2024-05-21T19:48:47Z

Grinder links:
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40745/ -linux
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40746/ -windows
https://hyc-runtimes-jenkins.swg-devops.com/job/Grinder/40768/ -mac

llxia · 2024-05-22T19:33:14Z

system/system.mk

 SYSTEMTEST_RESROOT=$(TEST_RESROOT)/../
+$(info SYSTEMTEST_RESROOT is $(SYSTEMTEST_RESROOT))


Please remove line 42 & Line 44. The value is set at a later stage.

llxia · 2024-05-22T19:33:26Z

system/system.mk

+  SYSTEM_LIB_DIR=$(SYSTEMTEST_RESROOT)/systemtest_prereqs
+endif
+
+$(info SYSTEM_LIB_DIR is $(SYSTEM_LIB_DIR))


Please remove line 80. The value is set at a later stage.

llxia · 2024-05-22T19:46:15Z

@annaibm can you point me to the code of creating systemtest_prereqs dir if it does not exist?

annaibm · 2024-05-22T19:55:12Z

@annaibm can you point me to the code of creating systemtest_prereqs dir if it does not exist?

@llxia I have reverted the code to include copy of systemtest_prereqs as in https://github.com/annaibm/aqa-tests/blob/test_systemtest/system/common.xml#L339C3-L341C10.

llxia · 2024-05-22T20:02:09Z

The current PR does not have The systemtest-prereqs directory could not be found. error locally?

annaibm · 2024-05-22T20:05:52Z

The current PR does not have The systemtest-prereqs directory could not be found. error locally?

No, it is getting processed as below

TESTING:
STF 08:58:26.042 - =========================   S T F   =========================
systemtest-prereqs has been processed, and set to: /home/jenkins/workspace/Grinder/jvmtest/system/systemtest_prereqsRetrieving amount of free space on drive containing /home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17163935052911/MiniMix_5m_0
There is 203270 Mb free

- Prestage system test prequisities to system_lib related: adoptium#4912 Signed-off-by: Anna Babu Palathingal <anna.bp@ibm.com>

annaibm force-pushed the test_systemtest branch from c9768f6 to ff8692e Compare April 18, 2024 15:43

annaibm force-pushed the test_systemtest branch 3 times, most recently from ed08e77 to 961dd88 Compare April 24, 2024 16:56

annaibm changed the title ~~Added system_prereq_lib for system prestage jars~~ Added system_lib for system prestage jars Apr 24, 2024

annaibm force-pushed the test_systemtest branch 7 times, most recently from 86bdafd to 8d35f5b Compare May 1, 2024 14:24

annaibm force-pushed the test_systemtest branch 2 times, most recently from cdb93d2 to 6876b4f Compare May 9, 2024 14:36

annaibm force-pushed the test_systemtest branch from 07284af to ee2d527 Compare May 9, 2024 18:29

llxia reviewed May 10, 2024

View reviewed changes

annaibm force-pushed the test_systemtest branch 6 times, most recently from 2149e1d to fb539fd Compare May 17, 2024 20:34

annaibm force-pushed the test_systemtest branch 3 times, most recently from 327b8e2 to 7722d60 Compare May 21, 2024 19:12

annaibm force-pushed the test_systemtest branch 5 times, most recently from d1707ab to 1987324 Compare May 22, 2024 19:20

llxia reviewed May 22, 2024

View reviewed changes

annaibm force-pushed the test_systemtest branch from 1987324 to bd90d5d Compare May 22, 2024 19:41

annaibm added 2 commits May 23, 2024 11:25

Added system_lib for system prestage jars

1fc9140

- Prestage system test prequisities to system_lib related: adoptium#4912 Signed-off-by: Anna Babu Palathingal <anna.bp@ibm.com>

added custom url

a5246bd

annaibm force-pushed the test_systemtest branch from 3f7024f to a5246bd Compare May 23, 2024 15:25

annaibm added 2 commits May 23, 2024 11:36

added customurl

7f2f379

added jenkinsfile

ba93cf2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added system_lib for system prestage jars #5239

Added system_lib for system prestage jars #5239

annaibm commented Apr 18, 2024 •

edited

annaibm commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia May 10, 2024

llxia May 10, 2024

annaibm May 13, 2024

llxia May 10, 2024

annaibm May 13, 2024

llxia May 22, 2024

llxia May 10, 2024

annaibm commented May 21, 2024 •

edited

llxia May 22, 2024

llxia May 22, 2024

llxia commented May 22, 2024

annaibm commented May 22, 2024

llxia commented May 22, 2024

annaibm commented May 22, 2024

		SYSTEMTEST_RESROOT=$(TEST_RESROOT)/../
		$(info SYSTEMTEST_RESROOT is $(SYSTEMTEST_RESROOT))

Added system_lib for system prestage jars #5239

Are you sure you want to change the base?

Added system_lib for system prestage jars #5239

Conversation

annaibm commented Apr 18, 2024 • edited

annaibm commented Apr 18, 2024

llxia commented Apr 18, 2024

llxia commented Apr 18, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annaibm commented May 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llxia commented May 22, 2024

annaibm commented May 22, 2024

llxia commented May 22, 2024

annaibm commented May 22, 2024

annaibm commented Apr 18, 2024 •

edited

annaibm commented May 21, 2024 •

edited