Track runtimes of Windows pipelines with 5 releases

[sxa]

This week I have been performing dry runs for the April 2024 release cycle.
Since we have five releases in parallel, some platforms will take longer than usual to work through the list they are all initiated. This is the first time we'll have done five, with 4 being initiated previously (8, 11, 17, 21 - 22 is the new one with 21 in LTS alongside all of the others)

I have already removed win32 from the default triggers on 8, 11 and 17 (We do not build win32 for 21+) which should reduce contention (We will start a separate pipeline for those) and we are also in the process of switching off some of our windows machines and replacing them with more up to date versions as per #3238) Most of this issue will involve the gathering of data to show how things are going and determine any improvements we can make going forward.

[sxa]

Builds were kicked off in the order JDK22, 8, 11, 17, and 21 in order to determine the worst-case scenario for the current latest LTS release. In each case the latest ones are the important ones, with previous ones added for comparison where available.

For the "Win64 pipeline" tables, note that the queue+build+smoketest all have to complete before other test jobs are initiated, so the total time for the pipeline will be the sum of those plus the slowest of the others (usually extended.openjdk). Note also that the "sign_verification" step also needs to run on a test machine, and that is done after the others are complete so cn get delayed for extended amounts of time too (I'm seeing 8 hours for some of them - perhaps that could be targetted to build nodes in the future?)

release-openjdk22-pipeline

Win64 JDK22 pipeline:

release-openjdk8-pipeline

Win64 JDK8 pipeline

release-openjdk11-pipeline

Win64 JDK11 pipeline

release-openjdk17-pipeline

Win64 JDK17 pipeline

release-openjdk21-pipeline

Win64 JDK21 pipeline

Noting also that we've got some jobs that appear quite "stuck" as mentioned in slack:

• https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_x86-32_windows_rerun/10/consoleFull seems to have been stuck on the win2019 machine (running for over 6 hours and not progressing, despite only having two targets)
• https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_x86-64_windows_rerun/10/console is another re-run which seems to be generating quite a lot of exceptions on test-azure-win2022-x64-2
• https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_1/4/console may also be stuck as it started over four hours ago and the log shows it hasn't produced any more output after about the first half hour of execution.

[sxa]

System has mostly quiesced now. Regens done to remove Linux/s390x from JDK22u and win32 from all relevant releases (8, 11, 17). Those plus the usual jdk8/arm32 will be triggered manually in the April cycle. I have also added the test labels to build-azure-2 to increase capacity a little more.
The next dry-run (once regenerations are done) will also include the fix to allow the rhel6 test machine to be re-enabled, and the marking offline of all of the equinix_esxi systems which are due to be decommissioned.

Initiated new pipelines:

• 1731UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/13/
• 1826UTC :https://ci.adoptium.net/job/build-scripts/job/release-openjdk8-pipeline/48
  • Windows failed, rerunning it at https://ci.adoptium.net/job/build-scripts/job/jobs/job/release/job/jobs/job/jdk8u/job/jdk8u-release-windows-x64-temurin/18/
• 1850:UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk11-pipeline/44/
• 1914:UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk17-pipeline/61/
• 1918 UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/31/

JDK22

release-openjdk22-pipeline

jdk22u-release-windows-x64-temurin

Test_openjdk22_hs_extended.openjdk_x86-64_windows

JDK8

release-openjdk8-pipeline

jdk8u-release-windows-x64-temurin

Test_openjdk8_hs_extended.openjdk_x86-64_windows

JDK11

release-openjdk11-pipeline

jdk11u-release-windows-x64-temurin

Test_openjdk11_hs_extended.openjdk_x86-64_windows

JDK17

release-openjdk17-pipeline

jdk17u-release-windows-x64-temurin

Test_openjdk17_hs_extended.openjdk_x86-64_windows

JDK21

release-openjdk21-pipeline

jdk21u-release-windows-x64-temurin

Test_openjdk21_hs_extended.openjdk_x86-64_windows

[sxa]

From the release pipelines in April 2024:

JDK	64-bit	32-bit
JDK8	4h29 (0)	9h32 (3h)
JDK11	9h30 (0h)	12h16 (0h)
JDK17	25h	13h
JDK21	14h27	n/a
JDK22	25h	n/a