EPIC: New Machine requirement: Replacement for Equinix x64 servers

[sxa]

I need to request a new machine:

• New machine operating system (e.g. linux/windows/macos/solaris/aix): Linux and Solaris
• New machine architecture (e.g. x64/aarch32/arm32/ppc64/ppc64le/sparc): x64
• Provider (leave blank if it does not matter): TBC
• Desired usage: Replacement for our x64 capacity at Equinix
• Any unusual specification/setup required: Potentially a hypervisor that can be used to run the Solaris systems
• How many of them are required: 3 (If equivalent size - around 48-64 core and capable of running multiple docker containers in parallel)

Please explain what this machine is needed for: Equinix have been sponsoring our infrastructure by providing a generous amount of capacity for the Adoptium infrastructure. This is now coming to and end and we need to make a plan for migrating our systems away from Equinix (Note: This does not affect the aarch64 Altras which are supplied as part of the Works On Arm project, but are hosted by Equinix)

This will involve migration of the following systems:

145.40.115.43 - VMware ESXi server (n3.large.x86 - London DC) and 147.28.133.218 - VMware ESXi server (m3.large.x86 - Paris DC)

These host our Solaris/x64 systems (including for Temurin Compliance) as well as other Linux VMs for performance test work on x64

London (145.40.x115.xx):

• https://ci.adoptium.net/computer/build%2Dequinix%5Fesxi%2Dsolaris10%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Dequinix%5Fesxi%2Dsolaris10%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Dequinix%5Fesxi%2Dubuntu2204%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Dequinix%5Fesxi%2Dubuntu2204%2Dx64%2D2/ (Offline)

Paris (147.28.133.2xx):

• jck-equinix_esxi-containerized-alpine317-x64-1
• jck-equinix_esxi-solaris10-x64-1
• jck-equinix_esxi-ubuntu2204-x64-1

dockerhost-equinix-ubuntu2004-x64-1 (AMD EPYC 7401P 24 core/48 thread - c2.medium.x86 - London DC)

Used for builds and hosting many containers:

• https://ci.adoptium.net/computer/test%2Ddocker%2Dcentos8%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Ddebian11%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dfedora35%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dfedora37%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dfedora37%2Dx64%2D3/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubi8%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2204%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2204%2Dx64%2D3/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dalpine314%2Dx64%2D1/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dalpine317%2Dx64%2D1/

dockerhost-equinix-ubuntu2204-x64-1 (Intel Xeon Gold 40 core - n3.xlarge.x86 - London DC)

Used for builds and hosting many containers

• https://ci.adoptium.net/computer/test%2Ddocker%2Dalpine314%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dalpine317%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dcentos8%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Ddebian11%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dfedora35%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dfedora37%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubi8%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2004%2Dx64%2D2/
• https://ci.adoptium.net/computer/test%2Ddocker%2Dubuntu2204%2Dx64%2D2/

C3AWX - (c3.small.x86 - Amsterdam DC)

Hosts our AWX instance and also the c3jenkins agent used for intermediate work on jenkins pipelines (Replacement for the "Built in Node")

Issues for individual systems:

• New Machine requirement: Ansible AWX #3339
• New Machine requirement: Replacement for C3jenkins #3340
• New Machine requirement: Solaris/x64 systems (Equinix replacement) #3347
• New Machine requirement: Linux/x64 equinix dockerhost replacement #3352
• New Machine requirement - Second Linux/x64 dockerhost/build host #3378
• https://github.com/temurin-compliance/temurin-compliance/issues/478 (Temurin Compliance)

[sxa]

Noting for C3awx:

• /var/lib/docker (AWX) is using 200Gb, although there seems to be only about 50Gb in use, mostly /Vendor_Files which has 43Gb - 33Gb is Visual Studio archives, 8Gb is Xcode 11.7, and about 3Gb for AIX. Going forward we sould probably try and use a mapped volume for this directory instead of hosting it within the container to make it easier to update. But overall about 100Gb and 1-2 cores would be fine for this system RAM uncertain
• Jenkins workspace is using 48Gb, but RAM and CPU requirements likely low. Maybe 2 core, 4Gb would cover it with a decent amount of disk (100-200G). Probably doesn't need to be on x64 ...

[sxa]

i'm going to drop the number of executors on the two dockerhost x64 machiens to 1 for now to see if that causes any additional delays. Some of the jobs do use multiple executors as per the screnshot below but those arne't time critical jobs (unless thye hold up others)

[sxa]

Current status:

Old host	New Host
build-equinix_esxi-solaris10-x64-1	build-skytap-solaris10-x64-1
test-equinix_esxi-solaris10-x64-1	test-azure-solaris10-x64-1
test-equinix_esxi-ubuntu2204-x64-1	Not replaced
test-equinix_esxi-ubuntu2204-x64-1	Not replaced
jck-equinix_esxi-ubuntu2204-x64-1	In progress
jck-equinix_esxi_containerized-alpine317-x64-1	To be hosted on esxi-ubuntu2204
jck-equinix_esxi-solaris10-x64-1	In progress
dockerhost-equinix-ubuntu2004-x64-1	dockerhost-skytap-ubuntu2204-x64-1
dockerhost-equinix-ubuntu2204-x64-1	dockerhost-azure-ubuntu2204-x64-1
C3jenkins	jenkins-worker

Also for decommission at Equinix:

• List of hosts on dockerhost-equinix-ubuntu2204-x64-1
• List of hosts on dockerhost-equinix-ubuntu2004-x64-1

[sxa]

Both equinix dockerhost x64 systems have now been shut down and the containers on them removed from jenkins.

[sxa]

c3-awx and vmware-esxi7 have been shut down and removed from the inventory (left for a few days just to ensure nothing is using them.

This only leaves vmware-esxi7-2 which hosts some of the Temurin Compliance systems and can go once @fredg02 confirms that we have the replacement Solaris boxes live and working on there

[steelhead31]

jck-equinix-solaris-x64-1 has now been shutdown and replaced.