Look at moving macstadium machines to orka #2536
Comments
sxa
changed the title
|
As per discussion a few weeks ago that the action is on me to progress, George and I will look at this migration together. |
|
Related: adoptium/temurin-build#3354 |
|
Our orka systems have been deprovisioned due to inactivity - currently having negotiations to determine a way forward. |
|
Discussions with MacStadium have indicated that an orka-based solution (which would not be sponsored at present) would be approximately twice the cost of the static systems which we have at present so we are looking at alternative options. Here is a breakdown of the number of systems and their types we have at macstadium:
So that's a total of 4+9+5 = 18 systems. |
|
Looking at the performance of various systems, here are some runs of the JDK8/x64 extended.openjdk suite on the different machines:
[*] - These machines have not typically been used for running the openjdk suites in the past so these may be newly visible failures. The second G3B machine was one of the build machines rather than one tagged for test. So with the exception of the second line, the performance of these for running the full extended.openjdk suite looks reasonable. It should be noted that it is between 2x and 2.5x slower to run the same tests on JDK21 so around 8h for a G3B and 3h30 for a G4B. |
|
Some other pieces of note:
|
|
Noting that JDK8 will not build on macos12 with Xcode 13: If I try a cross-compile from macos11/aarch64 with Xcode 12 I need to make a couple of other changes Which seems to have been deprecated and then removed in earlier Xcode versions (Possible backport?) may be more problematic |
|
Bit of a bad idea to run all of them at the same time. Some of the test jobs have expired even after 1 day. Sifting through the tests that have finished and not expired, avoiding duplicates (ie if jdk_security1_0 and jdk_security1_1 have the same failed tests, only jdk_security1_0 is shown) test-macstadium-macos11-arm64-1 jdk_security1_0,jdk_security4_0,jdk_util_0,jdk_svc_sanity_0,jvm_compiler_0,jdk_io_0,jdk_other_0,jdk_net_0,jdk_net_0,jdk_time_0,jdk_tools_0,jdk_jfr_0,jdk_jdi_0,jdk_security_infra_0 test-macstadium-macos11-arm64-2 (same failures as -1) build-macstadium-macos11-arm64-2 build-macstadium-macos11-arm64-1 (same failures as -2) |
|
So the failures you've got are only from the arm64 ones? And are all those targets from the openjdk suite - where the others targets all good? |
Can only kick off one build job at a time and on one machine at a time 😅 , this will take a while |
|
A couple of other things to add to this list - see if we can build ok on clang13 on macos12 (The two macincloud machines) but also see if we can install the older version of xcode (The one used for JDK8) on a newer macos version. |
|
Notes from building x64 jdk8 on my m1 mac Install xcode11.7. I can do this on my own mac (with GUI), need to find a way to do this headless Switch to xcode 11.7 Install 'intel' homebrew into /usr/local/Homebrew, requires a new Rosetta bash shell Back to a non Rosetta shell: Command to run build If theres still errors with the freetype compilation, install intel freetype and rerun build |
|
I built another x64 jdk8 binary on build-macstadium-macos11-arm64-1 and uploaded it to jenkins here I kicked off the aqa test pipeline, https://ci.adoptium.net/job/AQA_Test_Pipeline/173/console. Only sanity openjdk failed |
|
In the interest of seeing how x64 mac tests run on arm64 mac, i kicked off https://ci.adoptium.net/job/AQA_Test_Pipeline/174/console (jdk11 aqa tests on test-macstadium-macos11-arm64-1 Most tests passed. Failing ones are: |
|
Ref #2536 (comment) com/sun/jdi/RedefineCrossEvent.java.RedefineCrossEvent is excluded on openj9, https://github.com/adoptium/aqa-tests/blob/80e978693163b65ce6d3caabeb823ba594766167/openjdk/excludes/ProblemList_openjdk8-openj9.txt#L333 Known issue adoptium/aqa-tests#227, it fails the same way Rerunning So a cross compiled x64 jdk8 binary passes the tests in the AQA pipeline. Excellent news |
|
JDK11 build completed using XCode command line tools (same as before) https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk11u/job/jdk11u-mac-x64-temurin/327/ |
|
Right now the main issues I'm seeing are with the VPN expiring after a certain amount of time, this should be resolved once the firewall is configured to allow Jenkins in |
|
@gdams Not sure it's been explicitly mentioned in here but since it came up int he PMC this week can you clarify the reason for moving to XCode 15? The openjdk build matrix lists 12 as the Oracle-supported compiler, with 13.1 as "known good" too. It seems possibly that this is the cause of a lot of warnings showing in the build: adoptium/temurin-build#3562 so we should consider how to handle this. |
|
Still seeing some Terminated failures, for example, from https://ci.adoptium.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_mac/938/console: |
Grinding away to see how reproducible this is and if there's any consistency: Noting that "Worked?" is going to indicate:
Noting that the macincloud x64 machines completed sanity.openjdk in about 40 minutes, the orka ones took about 1h30 None of them seemed to have any of the unexpected termination problems. Although since these were all kicked off in parallel there were unlikely to have re-used any existing machines... |
|
Looking at some recently failing jobs on macos: JDK8 extended.system#929 - Failed during the setup phase JDK8 extended.functional#572 - Failed a few minutes after the start JDK17 extended.functional - 2-3 minutes after the start JDK17 sanity.openjdk - very early failure This is looking like it might be the Orka system decommissioning the machine because it thinks it's no longer used after being provisioned in a previous run but it's not immediately clear. |
|
@gdams has raised the disconnect issues with MacStadium. Awaiting a response. |
|
Regularity of x64 mac test jobs being terminated / disconnected seems to have increased (4/9 of the dry run jobs fail to run). jdk17 dry run pipeline 
aarch64 mac test jobs seem not to suffer from this problem (as frequently, if at all) 
|
|
Unable jdk_net and jdk_nio 4 test cases related to multicasting do not pass on Orka machines, details here: jdk_net jdk_nio |
|
@gdams as discussed - here are some examples of the errors I'm seeing in the jenkins log as a result of Orka: Unable to make field private static final long java.nio.channels.ClosedChannelException.serialVersionUID accessible2024-04-09 22:00:28.171+0000 [id=2688702] WARNING jenkins.util.Listeners#lambda$notify$0 Deploying VM failed with: HTTP Code: 500, Error: Internal error occurred: Requested CPU is not available in the cluster_Note: I'm not sure if the exception underneath it in the log is directly related to the Orka message_2024-04-10 00:32:35.570+0000 [id=2701897] WARNING i.j.plugins.orka.AgentTemplate#provision: Deploying VM failed with: HTTP Code: 500, Error: Internal error occurred: Requested CPU is not available in the cluster There's also this: For the second one above, I guess it's possible that it's being generated as a result of us hitting capacity on the cluster but might be good to verify whether such a condition has happened today. Since we've been kicking off five release runs in parallel it's entirely possible this is a fairly unique condition :-) |
I need to request a new machine:
Please explain what this machine is needed for:
The text was updated successfully, but these errors were encountered: