Kestrel #405

nmerket · 2023-10-30T21:28:19Z

Fixes #313

A start at getting it to work on Kestrel.

Checklist

Not all may apply

nmerket

@rHorsey Here's my "working" Kestrel implementation. A couple notes:

pre-commit and black

I'm using pre-commit and black for auto formatting now. Recommend doing a

pre-commit install

after pip installing. If you don't, it's not the end of the world, CI will run black and fix things for you.

Avoid `/shared-projects` for now

They're still copying stuff over from Eagle and the permissions are all messed up. The top of my testing project file looks like

schema_version: '0.3'
os_version: 3.6.1
os_sha: bb9481519e
buildstock_directory: ../ # Relative to this file or absolute
project_directory: project_national # Relative to buildstock_directory
output_directory: /scratch/nmerket/national_baseline2
# weather_files_url: https://data.nrel.gov/system/files/156/BuildStock_TMY3_FIPS.zip
weather_files_path: /scratch/nmerket/weather/BuildStock_TMY3_FIPS.zip
sys_image_dir: /scratch/nmerket/images

You'll need to copy those files to /scratch or /projects as necessary. Also, create the environment on Kestrel not in /shared-projects (the default) at this point. I have instructions for that below.

nmerket · 2023-10-31T16:36:37Z

buildstockbatch/hpc.py

Renamed this file from eagle.py ➡️ hpc.py.

nmerket · 2023-10-31T16:37:42Z

buildstockbatch/hpc.py

@@ -54,11 +54,12 @@ def get_bool_env_var(varname):
    return os.environ.get(varname, "0").lower() in ("true", "t", "1", "y", "yes")


-class EagleBatch(BuildStockBatchBase):


Most of the code is the same between the Eagle and Kestrel implementation, so I separated it out into a base class I'm calling SlurmBatch that EagleBatch and KestrelBatch both inherit from.

nmerket · 2023-10-31T16:40:33Z

buildstockbatch/hpc.py

-        cores_per_node = 36
-        minutes_per_sim = eagle_cfg["minutes_per_sim"]
-        walltime = math.ceil(math.ceil(n_sims_per_job / cores_per_node) * minutes_per_sim)
+        minutes_per_sim = hpc_cfg["minutes_per_sim"]
+        walltime = math.ceil(math.ceil(n_sims_per_job / self.CORES_PER_NODE) * minutes_per_sim)


Rather than hardcode the number of cores and the like, it's a constant on each of the sub classes.

nmerket · 2023-10-31T16:42:50Z

buildstockbatch/hpc.py

@@ -677,6 +716,68 @@ def rerun_failed_jobs(self, hipri=False):
        self.queue_post_processing(job_ids, hipri=hipri)


+class EagleBatch(SlurmBatch):


This is the Eagle specific implementation. It's mostly constant, defaults, and validation that only apply to Eagle.

nmerket · 2023-10-31T16:44:16Z

buildstockbatch/hpc.py

+
+
+class KestrelBatch(SlurmBatch):
+    DEFAULT_SYS_IMAGE_DIR = "/kfs2/shared-projects/buildstock/singularity_images"


They're supposed to have this fixed by next week, but they're not ready with /shared-projects on Kestrel yet. They're still copying stuff over from Eagle and the permissions are all messed up. I'd recommend doing your testing from /scratch or /projects for now.

nmerket · 2023-10-31T16:49:10Z

buildstockbatch/kestrel_postprocessing.sh

+pdsh -w $SLURM_JOB_NODELIST_PACK_GROUP_1 "df -i; df -h"
+
+$MY_PYTHON_ENV/bin/dask scheduler --scheduler-file $SCHEDULER_FILE &> $OUT_DIR/dask_scheduler.out &
+pdsh -w $SLURM_JOB_NODELIST_PACK_GROUP_1 "$MY_PYTHON_ENV/bin/dask worker --scheduler-file $SCHEDULER_FILE --local-directory /tmp/scratch/dask --nworkers ${NPROCS} --nthreads 1 --memory-limit ${MEMORY}MB" &> $OUT_DIR/dask_workers.out &


This wasn't working for me when the python environment was on /kfs2/shared-projects/envs. There are some permissions things messed up there. Like the groups weren't being passed down to the compute nodes or something. Supposedly they're working on it. I recommend creating your virtualenv on /scratch or /projects for testing.

nmerket · 2023-10-31T16:50:12Z

buildstockbatch/schemas/v0.3.yaml

@@ -6,6 +6,7 @@ weather_files_url: str(required=False)
 sampler: include('sampler-spec', required=True)
 workflow_generator: include('workflow-generator-spec', required=True)
 eagle: include('hpc-spec', required=False)
+kestrel: include('hpc-spec', required=False)


Just add a kestrel key like you have the eagle one to your project file. Adjust the number of jobs and file locations and stuff. It's all the same structure and format, though.

nmerket · 2023-10-31T16:51:52Z

buildstockbatch/kestrel.sh

+module load python apptainer
+source "$MY_PYTHON_ENV/bin/activate"


You'll notice I abandoned conda as our python package and environment manager. There was too much trouble between it and pip when installing buildstockbatch. I opted to go with the system installed python (3.11) and use a venv.

I'm trying to figure out if we actually still used ruby native outside of the container but it looks like not...

nmerket · 2023-10-31T16:56:19Z

create_kestrel_env.sh

To install, it defaults to /shared-projects so you'll want to override that. Also we're using python venv now for environments instead of conda, so activating is a little different.

module load git # yes, really git clone git@github.com:NREL/buildstockbatch.git cd buildstockbatch git checkout kestrel mkdir -p /scratch/$USER/envs ./create_kestrel_env.sh -e /scratch/$USER/envs -d mybsb source /scratch/$USER/envs/mybsb/bin/activate buildstock_kestrel path/to/project_file.yml

nmerket · 2023-10-31T16:56:55Z

setup.py

+            "buildstock_eagle=buildstockbatch.hpc:eagle_cli",
+            "buildstock_kestrel=buildstockbatch.hpc:kestrel_cli",


Adding a separate buildstock_kestrel cli.

github-actions · 2023-10-31T18:18:02Z

File	Coverage
All files	`86%`	✅
base.py	`90%`	✅
exc.py	`57%`	✅
hpc.py	`78%`	✅
local.py	`70%`	✅
postprocessing.py	`84%`	✅
utils.py	`91%`	✅
cloud/docker_base.py	`93%`	✅
sampler/base.py	`79%`	✅
sampler/downselect.py	`33%`	✅
sampler/precomputed.py	`93%`	✅
sampler/residential_quota.py	`61%`	✅
test/shared_testing_stuff.py	`85%`	✅
test/test_docker.py	`33%`	✅
test/test_validation.py	`97%`	✅
workflow_generator/base.py	`90%`	✅
workflow_generator/commercial.py	`53%`	✅
workflow_generator/residential_hpxml.py	`86%`	✅

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against 3461316

nmerket · 2023-11-01T15:42:56Z

It just occurred to me that a venv created by one user might not be able to be used by another user (which was possible with conda). We should check that and ensure it works, and if not, switch back to conda.

afontani · 2023-11-02T15:07:21Z

This would be a nice to have feature: Issue 171

nmerket · 2023-11-08T23:23:13Z

This would be a nice to have feature: Issue 171

@afontani Let's add that once we have a better handle on what the appropriate limits are on Kestrel.

joseph-robertson

I successfully ran a test 100 datapoint project (timeseries_frequency=none). Annual results were postprocessed/uploaded as expected.

nmerket added 3 commits October 24, 2023 16:34

adding .venv to .gitignore

550237b

changing module eagle.py to hpc.py

db1c98a

saving my mostly working version

4e92040

nmerket self-assigned this Oct 31, 2023

Merge remote-tracking branch 'origin/develop' into kestrel

a7ed663

nmerket commented Oct 31, 2023

View reviewed changes

fixing eagle testing

a8a63a1

nmerket added 8 commits November 6, 2023 13:39

updating tests to hit the KestrelBatch class

0535eb6

changelog

30bebb7

stubbing out kestrel docs

b205ac0

singularity -> apptainer

041d2ee

Merge remote-tracking branch 'origin/develop' into kestrel

8bebd80

allowing .sif apptainer images

121b940

missed a singularity -> apptainer rename

cb32b7d

update docs

db96996

nmerket marked this pull request as ready for review November 8, 2023 23:21

nmerket requested a review from joseph-robertson November 8, 2023 23:22

nmerket requested a review from rHorsey November 8, 2023 23:24

nmerket added 3 commits November 9, 2023 07:13

Merge remote-tracking branch 'origin/develop' into kestrel

d9dbe3f

fixing environment variables sent to sbatch

e8d1ea6

Merge branch 'kestrel' of github.com:NREL/buildstockbatch into kestrel

9a3446e

nmerket mentioned this pull request Nov 10, 2023

AWS Service Account #414

Merged

9 tasks

joseph-robertson approved these changes Nov 14, 2023

View reviewed changes

nmerket added 3 commits November 17, 2023 16:06

Merge branch 'develop' into kestrel

d70f76e

fixing apptainer error message and expected filename

54740fe

Merge branch 'develop' into kestrel

7d853ef

nmerket added 2 commits November 21, 2023 10:16

Merge remote-tracking branch 'origin/develop' into kestrel

161edc3

fixing testing

3461316

nmerket merged commit 24a6fa3 into develop Nov 21, 2023
6 checks passed

nmerket deleted the kestrel branch November 21, 2023 20:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kestrel #405

Kestrel #405

nmerket commented Oct 30, 2023 •

edited

nmerket left a comment

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

rHorsey Nov 13, 2023

nmerket Oct 31, 2023

nmerket Oct 31, 2023

github-actions bot commented Oct 31, 2023 •

edited

nmerket commented Nov 1, 2023

afontani commented Nov 2, 2023

nmerket commented Nov 8, 2023

joseph-robertson left a comment

		@@ -54,11 +54,12 @@ def get_bool_env_var(varname):
		return os.environ.get(varname, "0").lower() in ("true", "t", "1", "y", "yes")


		class EagleBatch(BuildStockBatchBase):

		@@ -677,6 +716,68 @@ def rerun_failed_jobs(self, hipri=False):
		self.queue_post_processing(job_ids, hipri=hipri)


		class EagleBatch(SlurmBatch):



		class KestrelBatch(SlurmBatch):
		DEFAULT_SYS_IMAGE_DIR = "/kfs2/shared-projects/buildstock/singularity_images"

		module load python apptainer
		source "$MY_PYTHON_ENV/bin/activate"

		"buildstock_eagle=buildstockbatch.hpc:eagle_cli",
		"buildstock_kestrel=buildstockbatch.hpc:kestrel_cli",

Kestrel #405

Kestrel #405

Conversation

nmerket commented Oct 30, 2023 • edited

Checklist

nmerket left a comment

Choose a reason for hiding this comment

pre-commit and black

Avoid /shared-projects for now

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 31, 2023 • edited

nmerket commented Nov 1, 2023

afontani commented Nov 2, 2023

nmerket commented Nov 8, 2023

joseph-robertson left a comment

Choose a reason for hiding this comment

nmerket commented Oct 30, 2023 •

edited

Avoid `/shared-projects` for now

github-actions bot commented Oct 31, 2023 •

edited