Update reproducible comparing tests on linux (not docker image) #3687

sophia-guo · 2024-03-05T20:49:49Z

Part of #3683

SCM_REF build issue Reproducible comparing using sbom json file got fatal: Not a valid object name jdk-21.0.3+6_adopt-266-ge01a50b2cac #3710
sbom with personal build branch and sha
Retrieving boot JDK 21.0.2+13-LTS not available -- BOOTJDK version is not abvailable by API https://ci.adoptium.net/view/Test_grinder/job/Grinder/9300/ -- remove suffix -LTS if bootjdk include -LTS
error: pathspec '"e95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b"' did not match any file(s) known to git. Update reproducible comparing tests on linux (not docker image) #3687 (comment)
Support running with upsteam jenkins jobs by mount test resources hit the user permission issue test-ibmcloud-ubuntu1604-x64-1 cannot clean workspace (Docker test issue) infrastructure#3490

The text was updated successfully, but these errors were encountered:

sophia-guo · 2024-03-05T20:56:30Z

Trying in adopitum jenkins with some necessary updates.
On test-ibmcloud-rhel7-x64-1 https://ci.adoptium.net/view/Test_system/job/Test_openjdk21_hs_dev.system_x86-64_linux/21/console

15:19:17  TESTING:
15:19:18  Loaded plugins: product-id, search-disabled-repos, subscription-manager
15:19:18  You need to be root to perform this command.

That is in linux_repro_build_compare.sh if it's rhel that some extra tools are needed to install https://github.com/adoptium/temurin-build/blob/master/tooling/reproducible/linux_repro_build_compare.sh#L27-L37. However jenkins doesn't have the permission.

sophia-guo · 2024-03-05T20:59:19Z

For non-rhel for now jq is need to install, which also has the permission issue. Can we ask the infra to install jq, which is also required for windows. @andrew-m-leonard @sxa

sophia-guo · 2024-03-18T14:30:12Z

SCM_REF build issue #3710

sophia-guo · 2024-03-22T20:30:44Z

Upstream build is triggered by personal temrurin-build ( could be the old repo openjdk-build) branch and sha, which will cause build failed -

   {
      "name" : "Temurin Build Ref",
      "value" : "https://github.com/andrew-m-leonard/openjdk-build/commit/532e5d1f4c39c121b42303c8f29d69e007ce96fe"
    },

15:32:44  Cloning into 'temurin-build'...
15:32:45  error: pathspec '"532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe"' did not match any file(s) known to git.

@andrew-m-leonard should personal branch be supported?

sophia-guo · 2024-03-25T12:56:15Z

@andrew-m-leonard see #3687 (comment)

sophia-guo · 2024-03-25T20:55:13Z

#3687 (comment)

This happened if Temurin Build Ref is referred develper's personal repo.

However I've tried the release version with main repo got the similar issue. - Using release jdk22 linux-x64 https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json/*view*/ I got similar error

16:36:01  error: pathspec '"e95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b"' did not match any file(s) known to git.
16:36:02  reproducibleCompare

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9302/consoleFull

sophia-guo · 2024-03-25T21:06:16Z

e95140b is the commit of branch https://github.com/adoptium/temurin-build/commits/v2024.03.01 , which is not in the master. Hence can't be found. Suppose reproducible comparing should support the release? @andrew-m-leonard

andrew-m-leonard · 2024-03-26T10:09:29Z

e95140b is the commit of branch https://github.com/adoptium/temurin-build/commits/v2024.03.01 , which is not in the master. Hence can't be found. Suppose reproducible comparing should support the release? @andrew-m-leonard

Should it matter where the commitSHA is from ?

sophia-guo · 2024-03-26T13:01:28Z

Shouldn't be. But, according to the test the sha can't be found.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/9302/consoleFull

andrew-m-leonard · 2024-03-26T14:29:34Z

I suspect this needs changing to not assume a git "branch" ? https://github.com/sophia-guo/openjdk-build/blob/33e187ebdab191756b3f6584ef3a24862e529d4c/test/system/reproducibleCompare/build.xml#L61
probably needs to clone, then do a checkout by SHA

sophia-guo · 2024-03-27T04:25:27Z

The build.xml only pre stages all shell scripts. https://github.com/sophia-guo/openjdk-build/blob/33e187ebdab191756b3f6584ef3a24862e529d4c/test/system/reproducibleCompare/build.xml#L61.
The reproducible build params are coming from sbom.json file.

The issue can be easily reproduced with master branch by following command

docker run -v "$(TEST_RESROOT):/home/jenkins/test" -w "/home/jenkins/test" --name reproducibleCompare centos:7 /bin/bash /home/jenkins/test/linux_repro_build_compare.sh  /home/jenkins/jdkbinary/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json /home/jenkins/jdkbinary/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

export $(TEST_RESROOT) to local temurin-build/tooling/reproducible. Download the jdkfile and sbom file to same directory. Update linux_repro_build_compare.sh to support local files.

andrew-m-leonard · 2024-03-27T08:09:29Z

The reproducible build params are coming from sbom.json file.

yeah, I am wondering if the SCM_REF for these examples in the SBOM should be empty ?
and in which case the git checkout should then checkout by openjdk_commit

sophia-guo · 2024-03-27T12:36:48Z

The files are release files
https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json

        {
          "name" : "SCM Ref",
          "value" : "jdk-22+36_adopt"
        },
        {
          "name" : "OpenJDK Source Commit",
          "value" : "https://github.com/adoptium/jdk22/commit/1aa79b171e3f8b58852a01e08d82230565a01989"
        },
        {
          "name" : "Temurin Build Ref",
          "value" : "https://github.com/adoptium/temurin-build/commit/e95140bf52915bda6c5ff271538789ed9a55162b"

With linux it should be easily reproduced by

linux_repo_build_compare.sh https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

andrew-m-leonard · 2024-03-27T14:46:31Z

@sophia-guo The jq command needs fixing, the git checkout command it is running is:

(cd temurin-build && git checkout e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b)

because jq is returning:

andrew@andrew:~/temurin-build/tooling/reproducible$ jq -r '.components[] | .properties[] | select (.name == "Temurin Build Ref") | .value' "OpenJDK22U-sbom_x64_linux_hotspot_22_36.json" | awk -F/ '{print $NF}'
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b

sophia-guo · 2024-03-27T16:39:44Z

#3687 (comment)

I see. Seems SBOM file content changed. For this specific sbomfile there are 4 components ( jdk, jre, static-lib, debug). All properties of those 4 components are duplicate except filename itself.

sophia-guo · 2024-04-02T18:16:35Z

The diff for reproducing release jdk22 is as following

diff -r jdk-22+36/release compare.1/jdk-22+36/release
11,13d10
< BUILD_SOURCE="git:e95140bf52915bda6c5ff271538789ed9a55162b"
< BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
< SOURCE_REPO="https://github.com/adoptium/jdk22.git"

That is for reproducing one the release file doesn't have those information, which is minor but not sure if it is expected. @andrew-m-leonard

andrew-m-leonard · 2024-04-03T14:46:40Z

The diff for reproducing release jdk22 is as following
diff -r jdk-22+36/release compare.1/jdk-22+36/release
11,13d10
< BUILD_SOURCE="git:e95140bf52915bda6c5ff271538789ed9a55162b"
< BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
< SOURCE_REPO="https://github.com/adoptium/jdk22.git"
That is for reproducing one the release file doesn't have those information, which is minor but not sure if it is expected. @andrew-m-leonard

I thought that should be clean'd by this

temurin-build/tooling/reproducible/repro_common.sh

Line 349 in 646a79f

sed -i '/^BUILD_SOURCE=.*$/d' "${DIR}/release"

sophia-guo · 2024-04-03T15:03:34Z

Linux_repro_build_compare.sh doesn't use the repro_common.sh

andrew-m-leonard · 2024-04-03T15:08:31Z

Linux_repro_build_compare.sh doesn't use the repro_common.sh

Ah yes, so we should re-use that clean method to make these compare if possible?

sophia-guo · 2024-04-03T15:11:10Z

We can do in separate issue or pr later.

sophia-guo self-assigned this Mar 5, 2024

github-actions bot added the x-linux Issues that affect or relate to the x64/x32 LINUX OS label Mar 5, 2024

sophia-guo mentioned this issue Mar 5, 2024

Add reproducible comparison tests for windows and update tests for linux running out of docker image #3683

Open

2 tasks

sophia-guo mentioned this issue Mar 6, 2024

Ansible request for jq for test machines adoptium/infrastructure#3453

Closed

github-actions bot added jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites labels Mar 28, 2024

sophia-guo mentioned this issue Mar 29, 2024

Update reproducible comparing tests with centos:7 #3739

Merged

sophia-guo closed this as completed in #3739 Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update reproducible comparing tests on linux (not docker image) #3687

Update reproducible comparing tests on linux (not docker image) #3687

sophia-guo commented Mar 5, 2024 •

edited

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 18, 2024

sophia-guo commented Mar 22, 2024 •

edited

sophia-guo commented Mar 25, 2024

sophia-guo commented Mar 25, 2024 •

edited

sophia-guo commented Mar 25, 2024

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 26, 2024 •

edited

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 27, 2024

andrew-m-leonard commented Mar 27, 2024 •

edited

sophia-guo commented Mar 27, 2024 •

edited

andrew-m-leonard commented Mar 27, 2024

sophia-guo commented Mar 27, 2024

sophia-guo commented Apr 2, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

Update reproducible comparing tests on linux (not docker image) #3687

Update reproducible comparing tests on linux (not docker image) #3687

Comments

sophia-guo commented Mar 5, 2024 • edited

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 18, 2024

sophia-guo commented Mar 22, 2024 • edited

sophia-guo commented Mar 25, 2024

sophia-guo commented Mar 25, 2024 • edited

sophia-guo commented Mar 25, 2024

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 26, 2024 • edited

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 27, 2024

andrew-m-leonard commented Mar 27, 2024 • edited

sophia-guo commented Mar 27, 2024 • edited

andrew-m-leonard commented Mar 27, 2024

sophia-guo commented Mar 27, 2024

sophia-guo commented Apr 2, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

sophia-guo commented Mar 5, 2024 •

edited

sophia-guo commented Mar 22, 2024 •

edited

sophia-guo commented Mar 25, 2024 •

edited

sophia-guo commented Mar 26, 2024 •

edited

andrew-m-leonard commented Mar 27, 2024 •

edited

sophia-guo commented Mar 27, 2024 •

edited