ssl-tests_0_FAILED but no clear error messages

[llxia]

internal link

[2024-05-04T03:23:26.571Z] TESTING:
[2024-05-04T03:23:26.571Z] ~/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests ~/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/TKG/output_17147930051846/ssl-tests_0
[2024-05-04T03:23:26.571Z] Non cygwin system!
[2024-05-04T03:23:26.571Z] /home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests
[2024-05-04T03:23:26.571Z] treating jdk as: 8
[2024-05-04T03:23:26.571Z] Running with /home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/jdkbinary/j2sdk-image...
[2024-05-04T03:23:30.854Z] XML output  to /home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests/test.1714793006/jdk/work
[2024-05-04T03:25:02.248Z] FAILED: ssl-tests.sh
[2024-05-04T03:29:44.265Z] Passed: ssl-tests-bc-2nd.sh
[2024-05-04T04:02:32.956Z] Passed: ssl-tests-bcfips.sh
[2024-05-04T04:07:16.793Z] Passed: ssl-tests-bcjsse.sh
[2024-05-04T04:07:34.344Z] FAILED: ssl-tests-nss-client.sh
[2024-05-04T04:07:46.263Z] FAILED: ssl-tests-openssl-client.sh
[2024-05-04T04:07:46.263Z] Test results: passed: 3; failed: 3
[2024-05-04T04:07:46.672Z] Error: Some tests failed or other problems occurred.
[2024-05-04T04:07:46.672Z] Report written to /home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests/test.1714793006/jdk/report/html/report.html
[2024-05-04T04:07:46.672Z] Results written to /home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests/test.1714793006/jdk/work
[2024-05-04T04:08:11.138Z] ~/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/TKG/output_17147930051846/ssl-tests_0
[2024-05-04T04:08:11.138Z] renamed '/home/jenkins/workspace/Test_openjdk8_j9_dev.functional_ppc64le_linux/aqa-tests/functional/security/ssl-tests/ssl-tests/test.1714793006.tar.gz' -> './test.1714793006.tar.gz'
[2024-05-04T04:08:11.138Z] -----------------------------------
[2024-05-04T04:08:11.138Z] ssl-tests_0_FAILED

[llxia]

junit report:

[llxia]

@judovana any insight?

[judovana]

hi. Yes. There the solution is muych more tricky. Because jtreg, by defaul, do not propagate output from shell scripts which is esential here.
We had contributed fix to jtregs: openjdk/jtreg@8c105f2

But it is not yet in any release - and will not be any time soon. Thus we have fork of jtregs, where we keep this change "released": https://github.com/andrlos/jtreg/releases ( I need to ping @andrlos to rebase and refresh, and most liekly to move under rh-openjdk). We keep 6.x brnach to make jdk8 happy and 7.y to make all other jdks happy.

The ssltests in Aqa vit are - for many good resons - using in-aqavit jtregs:

aqa-tests/functional/security/ssl-tests/playlist.xml

Line 26 in b2a0d5a

JTREG_HOME="$(TEST_RESROOT)$(D)jtreg" \

However, good news! Our default run.sh is using this "modified jar" : https://github.com/rh-openjdk/ssl-tests/blob/cfe583f4b379319e60519d34430e460f3599d6a5/run.sh#L113. So if you have posibility to run it locally, just drop that JTREG_HOME="$(TEST_RESROOT)$(D)jtreg" (or point it to the "correct" location with pathced jtreg, and you are done.

If you can not run it locally, you can use FORCE_TMP_JTREG feature of run.sh which will download rewuested jtregs tarball for you and use it no metter of JTREG_HOME. Eg I;m running aqavit's ssl tests with following preamble:

if [ "$OTOOL_JDK_VERSION" -le "8" ] ; then
    export FORCE_TMP_JTREG="https://github.com/andrlos/jtreg/releases/download/6.1-jtrfix-V01.0/jtreg-6+1-jtrfix.tar.gz"
  else
    export FORCE_TMP_JTREG="https://github.com/andrlos/jtreg/releases/download/7.3.1%2B1-jtrfix-V01.0/jtreg-7.3.1+1-jtrfix.tar.gz"
  fi

Which is moreover duplicating what is in run.sh, but thats how we agreed. Let me know if this was of any help for you. Will do my best to resolve it.

[llxia]

Thanks @judovana . Any outlook on when the fix will be the jtreg release?

[judovana]

No idea. They are releasing randomly by my view, sometimes aligned with future JDK needed features. Also note, that it will never go to 6.x :(
One of the ideas is to enable usage also of https://ci.adoptium.net/view/Dependencies/job/dependency_pipeline/lastSuccessfulBuild/artifact/jtreg/ 's jtregtip.tar.gz as one of the possible runners.

[llxia]

In my opinion, since the release of fixed jtreg is unknown, I believe it's best to hold off on running SSL tests in our weekly runs until the jtreg issue is resolved. Without a specific error message, running them would be a waste of both machine resources and triage efforts.

@pshipton @JasonFengJ9 @smlambert what do you think?

[judovana]

Maybe it would be better to be able to run them locally and debug?

[judovana]

Other possibility is to run that without jtregs, through makefiles.

But please do not rush on that, I do not want to have jtreg runners reverted in rush as it happened for CryptoTests.