v1.34.0: Slurm-GCP v6 Generally Available

What's Changed

In this release, we promote Slurm-GCP V6 to GA, making it the recommended version of Slurm-GCP. Find out more at:
Announcement

Key New Features 🎉

• Roll TGP to v5.28.0 by @alyssa-sm in #2535

Module Improvements 🔨

• Slurm6. Add support for nodeset network_storage by @mr0re1 in #2522
• Add documentation about how to use max-distance with Slurm v6 by @nick-stroud in #2551
• Remove login infix from instance name by @mr0re1 in #2540
• New output for private-service-access by @cdunbar13 in #2562
• Make batch job runnables a list instead of single string by @aaronegolden in #2516
• Implement recovery of HTCondor spool (job queue) by @tpdownes in #2500
• Slurm6. Stop using SlurmGCP remote modules for partition and nodeset_dyn by @mr0re1 in #2558
• Fix deployment of multiple Batch jobs by @aaronegolden in #2543
• Add debugging output to spack and ramble installations by @cdunbar13 in #2568

Improvements 🛠

• High level changes in Slurm GCP v6 by @nick-stroud in #2620
• OFE: new features and fixes. by @ek-nag in #2512
• Update A3 blueprint guidance for reservations by @tpdownes in #2573
• Proceed with destruction of other groups even if current failed by @mr0re1 in #2575
• Add blueprint to support Apptainer by @wkharold in #2565

Deprecations 💤

• Make Slurm-GCP v5 examples and references as legacy by @harshthakkar01 in #2567

Version Updates ⏫

• Update a3-highgpu-8g blueprint to use latest v5 tag by @tpdownes in #2572
• Update Slurm-GCP v5 modules and examples to 5.11.1 by @tpdownes in #2595
• Update Slurm-GCP v6 modules and examples to 6.5.2 by @tpdownes in #2594

Bug fixes 🐞

• OFE: fixing broken partitions update logic. by @ek-nag in #2563
• Allow only specific reservation for nodeset in slurm-gcp v6 nodeset by @harshthakkar01 in #2612
• Allow specific reservation for node-group in slurm-gcp v5 by @harshthakkar01 in #2614

Other changes

• Revert "Allow specific reservation for node-group in slurm-gcp v5" by @harshthakkar01 in #2621
• Revert "Revert "Allow specific reservation for node-group in slurm-gcp v5"" by @harshthakkar01 in #2622

Full Changelog: v1.33.0...v1.34.0

v1.33.0: "ghpc_stage" function; Slurm-GCP v6 improvements

What's Changed

Key New Features 🎉

• Add docs about ghpc_stage and other functions by @mr0re1 in #2485
• Add startup-script option to automatically install Docker at boot by @tpdownes in #2489

New Modules 🧱

• MultiVPC Module by @cdunbar13 in #2450

Module Improvements 🔨

• Address feature requests for HTCondor functionality in Windows by @tpdownes in #2469
• Slurm6. Replace service_account with service_account_email|scopes by @mr0re1 in #2495
• Slurm6. Replace vars disable_X -> enable_X by @mr0re1 in #2486
• Remove "hard" dependency between login instance and controller instance by @mr0re1 in #2413
• Allow the wait-for-startup module to take a list of instance names by @rohitramu in #2515
• Simplify "cleanup compute" by @mr0re1 in #2479
• Copy labels from the batch-job-template module to the actual Batch job spec by @aaronegolden in #2514
• Slurm6. Automatically set login intances name, don't put role into it by @mr0re1 in #2531
• Adopt Slurm-GCP 6.4.6 by @tpdownes in #2511

Improvements 🛠

• Update hpc-slurm-ramble-gromac example to use Slurm-GCP v6 modules by @harshthakkar01 in #2407
• Add Slurm-gcp v6 example for hpc-amd-slurm blueprint and references by @harshthakkar01 in #2411

Bug fixes 🐞

• Go binary downloads must include patch version by @nick-stroud in #2465
• Resolve Ansible crash in Spack installation by @tpdownes in #2462

Full Changelog: v1.32.1...v1.33.0

v1.32.1: Fix version number in modules

What's Changed

Fix version number in modules

Full Changelog: v1.32.0...v1.32.1

v1.32.0: Deployment files and Slurm-GCP v6 examples

What's Changed

Key New Features 🎉

• Deployment files allow merging generic blueprints with configurations specific to single deployments

New Modules 🧱

• Decoupling private access from Cloud SQL to allow multiple instances in same VPC by @cboneti in #2397

Improvements 🛠

• Possible breaking change to workflows that call ghpc create: Set default validation level to ERROR by @mr0re1 in #2383
• Add example using Slurm static compute nodes by @nick-stroud in #2393
• Add lustre slurm blueprint for v6 version and integration test by @harshthakkar01 in #2329
• Add v6 version of ml-slurm blueprint and integration test by @harshthakkar01 in #2337
• Add Slurm-GCP v6 version of zone policies blueprint by @alyssa-sm in #2390
• In place update hpc-slurm-qwiklab blueprint to use slurm-gcp v6 by @harshthakkar01 in #2402
• Add SlurmGCP v6 version of CAE blueprint by @harshthakkar01 in #2325
• Roll forward "Add example using Slurm static compute nodes" by @nick-stroud in #2408
• Update gcc to 13.1.0 on Rocky 8 Gromacs example by @tpdownes in #2445
• Add SlurmGCP v6 version of hcls blueprint and integration test by @harshthakkar01 in #2366
• Update MPI & Gromacs for cache hit and better compatibility w/ Rocky8 default image by @nick-stroud in #2456

Bug fixes 🐞

• Revert "Add example using Slurm static compute nodes" by @nick-stroud in #2404
• Fixes for workstation creation - new extension added for yaml by @cdunbar13 in #2421
• Updating garther startup script and integration test by @cdunbar13 in #2449

Full Changelog: v1.31.1...v1.32.0

v1.31.1: Updated provisioning guide for A3 VM family

What's Changed

The A3 provisioning guide was updated by @tpdownes to support 2 use cases:

• user-created reservations without compact placement policies that are automatically consumed by matching VMs
• Google Cloud-created reservations that must be specifically identified by Slurm cluster for consumption

See #2420 and reservation consumption documentation for details.

Full Changelog: v1.31.0...v1.31.1

v1.31.0: Improved Local File Management

What's Changed

Key New Features 🎉

• Implement ghpc_stage function to stage files into deployment by @mr0re1 in #2339

Module Improvements 🔨

• Support http_proxy in HTCondor Windows installation by @tpdownes in #2368
• Slurm6. Add support for dynamic nodeset. by @mr0re1 in #1986

Improvements 🛠

• Migrate config-ssh.yaml to Slurm-GCP v6 by @alyssa-sm in #2358
• Add SlurmGCP v6 version of node groups blueprint example by @harshthakkar01 in #2347
• Add Slurm-GCP v6 example for gpu.yaml by @alyssa-sm in #2376
• Add SlurmGCP v6 example of htc-slurm blueprint by @harshthakkar01 in #2348
• Script capture serial by @cboneti in #2380

Deprecations 💤

• Deprecate schedmd-slurm-gcp-v6-partition.network_storage by @mr0re1 in #2379
• Remove quota validator by @mr0re1 in #2382

Bug fixes 🐞

• Packer service account fix and alignment with Toolkit naming convention by @tpdownes in #2367

Full Changelog: v1.30.0...v1.31.0

v1.30.0 - Cloud HPC Toolkit A3 VM + NeMo Framework Solution

What's Changed

Key New Features 🎉

• Introduction of the Cloud HPC Toolkit A3 VM family blueprint featuring
  • A Slurm cluster composed of A3 VMs each with 8 NVIDIA H100 GPUs
  • An example for running the NVIDIA NeMo framework
  • An example for running the common nccl-tests benchmark

Module Improvements 🔨

• Add support for startup script per nodeset by @mr0re1 in #2296
• Allocate IP for vm-instance by @lemaitre-aneo in #2219
• Add random hex to batch job id and specify id when submitting by @nick-stroud in #2259
• HTCondor: support user-managed secret replication by @tpdownes in #2340
• Support setting system default web proxy in Windows startup by @tpdownes in #2351

Improvements 🛠

• Add TPU v4 blueprint and tutorial to demonstrate running TPU workload by @harshthakkar01 in #2287
• Update parameters for TPU nodeset module and add precondition checks and bump TPU to v3 by @harshthakkar01 in #2293
• Add Slurm v6 version for image builder blueprint by @harshthakkar01 in #2297
• Allow ghpc deploy blueprint.yaml by @mr0re1 in #2323
• Slurm GCP version update; will cooldown before deleting orphan nodes by @nick-stroud in #2322
• Add SlurmGCP v6 example of slurm compatible with startup scripts and integration test by @harshthakkar01 in #2346

Version Updates ⏫

• Adopt Slurm-GCP release 5.10.6 by @tpdownes in #2352

Bug fixes 🐞

• Added enable_devel for packer build to fix issue with bp by @cdunbar13 in #2334

New Contributors

• @lemaitre-aneo made their first contribution in #2219

Full Changelog: v1.29.0...v1.30.0

v1.29.0: New Firewall Rules module & Slurm-GCP v6 Improvements

What's Changed

Key New Features 🎉

• Initial commit of firewall-rules module by @tpdownes in #2244

New Modules 🧱

• Split service account creation from htcondor-setup by @tpdownes in #2250

Module Improvements 🔨

• Set http_proxy, https_proxy variables for user login and during startup-script by @tpdownes in #2237
• Update documentation for Packer to include minimum operational requirements by @tpdownes in #2241
• Modify cloud-storage-bucket to include ability to set bucket viewers by @tpdownes in #2247
• Add "submit" option to batch-job-template module by @aaronegolden in #2210
• Prevent usage of placement with static and auto-scale nodes in same nodeset by @nick-stroud in #2279

Improvements 🛠

• Add login node in spack wrfv3 tutorial example by @alyssa-sm in #2207

Version Updates ⏫

• Update to Slurm-GCP 5.10.4 by @tpdownes in #2251
• Bump cryptography from 42.0.0 to 42.0.2 in /community/front-end/ofe by @dependabot in #2257
• Slurm-GCP Release 6.4.1 by @tpdownes in #2230
• Updated Slurm v6 version, fix placement error by @nick-stroud in #2280
• Bump cryptography from 42.0.0 to 42.0.4 in /community/front-end/ofe by @dependabot in #2288
• Bump Golang version 1.20 -> 1.21 by @mr0re1 in #2291

Bug fixes 🐞

• Fix bug in validation of variable in PBS module by @tpdownes in #2255
• Fix for wait-for-startup.yml ansible task by @cdunbar13 in #2283

Full Changelog: v1.28.1...v1.29.0

v1.28.1: Slurm-GCP v4 reaches End-of-Life, improved Slurm-GCP v6 support

What's Changed

Key New Features 🎉

• Add support for string interpolation by @mr0re1 in #2076
• 🌈 Enable output colorization by default by @mr0re1 in #2145
• Add example of building Slurm on top of Rocky 8 by @nick-stroud in #2155

Module Improvements 🔨

• Slurm6. Make subnetwork_self_link required, don't pass subnetwork_project by @mr0re1 in #2067
• Slurm6. Automagicaly set nodeset.name from module id. by @mr0re1 in #2068
• Slurm6. Add support for additional_networks, access_config & reservation_name by @mr0re1 in #2062
• Reduce default maximum number of HTCondor execute points by @tpdownes in #2127
• Startup stackdriver option by @nick-stroud in #2120
• HTCondor: variable MIG behavior by @tpdownes in #2140
• Extending GKE Scheduler module by @ek-nag in #2137
• Copies python binaries instead of symlink for more isolated venv by @nick-stroud in #2151
• Increase dynamic node count to a more reasonable default value by @nick-stroud in #2153
• Update Chrome Remote Desktop to Debian 12 by default by @tpdownes in #2180
• Update startup-script module to latest release by @tpdownes in #2183
• Updates to HTCondor autoscaler by @tpdownes in #2204
• Change batch-job-base template from json to YAML by @aaronegolden in #2199
• Add Slurm configuration template for long Prolog/Epilog scripts by @tpdownes in #2218

Improvements 🛠

• Check if supplied value matches expected module variable type by @mr0re1 in #2089
• Update spack openfoam example to Slurm V6 by @harshthakkar01 in #2090
• Add has_to_be_used behaviour to some of modules by @mr0re1 in #2092
• Update spack wrf example and references to use Slurm V6 by @harshthakkar01 in #2132
• Update spack gromacs example and references to Slurm V6 by @harshthakkar01 in #2138
• Update hpc slurm gromac example and references to use Slurm V6 by @harshthakkar01 in #2149
• Update DAOS blueprints to use google-cloud-daos v0.5.0, slurm v6 by @mark-olson in #2147
• Bring $(...) functionality on par with ((...)) by @mr0re1 in #2053
• Add --force flag to ghpc create by @mr0re1 in #2162
• Update hpc-slurm-legacy-sharedvpc example and references to use Slurm V6 by @harshthakkar01 in #2143
• Add login node in the spack openfoam tutorial example by @harshthakkar01 in #2170
• Add login node to spack gromacs tutorial example by @alyssa-sm in #2227

Deprecations 💤

• Remove pre existing fs example and references by @harshthakkar01 in #2104
• Remove slurm-two-partitions-workstation example and references by @harshthakkar01 in #2106
• Remove use-resources example and references by @harshthakkar01 in #2107
• Update test outputs example and remove slurm partition and controller by @harshthakkar01 in #2109
• Remove test-gcs-fuse example and references by @harshthakkar01 in #2110
• Remove hpc-cluster-service-acct example and references by @harshthakkar01 in #2111
• Remove hpc-cluste-slurm-with-startup example and references by @harshthakkar01 in #2112
• Remove hpc-cluster-project example and references by @harshthakkar01 in #2113
• Remove hpc-cluster-high-io-remote-state example and references by @harshthakkar01 in #2114
• Remove lustre-new-vpc example and references by @harshthakkar01 in #2108
• Remove hpc-slurm-legacy example and references by @harshthakkar01 in #2103
• Remove intel-select blueprints and references by @harshthakkar01 in #2135
• Remove Slurm V4 modules and add note to use and reference V4 modules and examples by @harshthakkar01 in #2160
• Deprecate Dell Omnia module and example blueprint by @tpdownes in #2196

Version Updates ⏫

• Update Slurm Cloud SQL example to V6 by @alyssa-sm in #2168
• Update Slurm-GCP release to 5.10.2 by @tpdownes in #2189
• Bump django from 4.2.7 to 4.2.10 in /community/front-end/ofe by @dependabot in #2213

Bug fixes 🐞

• Update spack openfoam example to use /opt/apps directory by @harshthakkar01 in #2131
• Fix HTCondow Windows URI for latest 23.0 LTS release by @tpdownes in #2141
• Validation added to Slurm v5 login_startup_scripts_timeout by @cdunbar13 in #2148
• Ensure Windows VMs start HTCondor only after successful secret download by @tpdownes in #2174

New Contributors

• @stas00 made their first contribution in #2125
• @aaronegolden made their first contribution in #2199

Full Changelog: v1.27.0...v1.28.0

Submission Checklist

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cloud HPC Toolkit Contribution guidelines #

v1.27.0: Spack support for non-root users

What's Changed

Key New Features 🎉

• Perform spack and ramble actions with system user by @nick-stroud in #2052

New Modules 🧱

• Tutorial for FSI VaR MonteCarlo by @jrossthomson in #1874

Module Improvements 🔨

• Making CloudSQL to use internal IP address instead of external for Slurm Accounting DB. by @ek-nag in #1795
• OFE: Various new features and fixes. by @ek-nag in #2040
• Disable firewall rule logging by default by @tpdownes in #2057
• Slurm6. Add support for enable_slurm_gcp_plugins by @mr0re1 in #2066
• Support explicit reserved_ip_range for Filestore instances by @tpdownes in #2072
• Adopt gcloud storage over gsutil by default by @tpdownes in #2075
• Skip upgrade of wheel/setuptools if already installed by @tpdownes in #2074
• Support use of http/https proxy for pip/apt/yum package managers by @tpdownes in #2079

Improvements 🛠

• Creation of script to check upcoming maintenance on nodes by @cdunbar13 in #1977
• Add documentation how to use max_hops plugin by @nick-stroud in #1983
• Add Zonal with lower capacity band support for filestore instance by @harshthakkar01 in #1982
• Add codespell: workflow, by @yarikoptic in #1888
• Update slurm modules to point to fork by @mr0re1 in #2009
• Document permissions around manual usage of Spack installation by @nick-stroud in #2015
• Front End Deployment Changes by @max-nag in #1895
• vm-instance is now able to use a service account by @nick-stroud in #2051

Version Updates ⏫

• OFE: Spack upgrade to v0.21.0 by @ek-nag in #2004

Bug fixes 🐞

• Fix local SSD example by @tpdownes in #2026

New Contributors

• @yarikoptic made their first contribution in #1888
• @fustic made their first contribution in #2041

Full Changelog: v1.26.1...v1.27.0