Update to OS v3.2.1

[joseph-robertson]

Pull Request Description

Companion PR: NREL/resstock#661 (ServerDirectoryCleanup updates).

This PR is necessary to work with resstock develop.

By setting skip_zip_results=True in the workflow generator, each datapoint's run directory contains all simulation files and they aren't contained in a data_point.zip. Also, the size of simulations_job0.tar.gz actually decreases.

BEFORE

AFTER

Checklist

Not all may apply

• Code changes (must work)
• Tests exercising your feature/bug fix (check coverage report on CircleCI build -> Artifacts)
• All other unit tests passing
• Update validation for project config yaml file changes
• Update existing documentation
• Run a small batch run to make sure it all works (local is fine, unless an Eagle specific feature)
• Add to the changelog_dev.rst file and propose migration text in the pull request

[nmerket]

This is good. With the data_point.zip now gone, I was worried this code would break:

buildstockbatch/buildstockbatch/base.py

Lines 203 to 210 in 16cf0a8

	# Remove files already in data_point.zip
	zipfilename = os.path.join(sim_dir, 'run', 'data_point.zip')
	if os.path.isfile(zipfilename):
	with zipfile.ZipFile(zipfilename, 'r') as zf:
	for filename in zf.namelist():
	for filepath in (os.path.join(sim_dir, 'run', filename), os.path.join(sim_dir, filename)):
	if os.path.exists(filepath):
	os.remove(filepath)

It was looks in data_point.zip and removes things from the directory that are already in there. But if data_point.zip isn't there it skips that block, so I guess it's not hurting anything. I'm not sure what's going on in ComStock so maybe we leave it for now.

In general, I'd say just go through that cleanup function to make sure it's still relevant with the organization of the files being written with your update.

[joseph-robertson]

@nmerket I've updated the residential workflow generator to write argument values for ServerDirectoryCleanup. I'm not sure what else needs to be updated (e.g., pretty sure validation will now fail if server_directory_cleanup is found in a project yml file).

[nmerket]

@joseph-robertson This seems pretty straightforward and I like what you're doing with the server directory cleanup. You're right in that it will throw an error if that argument is there even though you're expecting it. That's easy enough to fix, though. Just add some yaml schema stuff here to tell it what to expect. It's even in the same file.

buildstockbatch/buildstockbatch/workflow_generator/residential.py

Lines 43 to 54 in 81b43f0

	schema_yml = """
	measures_to_ignore: list(str(), required=False)
	residential_simulation_controls: map(required=False)
	measures: list(include('measure-spec'), required=False)
	reporting_measures: list(include('measure-spec'), required=False)
	simulation_output: map(required=False)
	timeseries_csv_export: map(required=False)
	---
	measure-spec:
	measure_dir_name: str(required=True)
	arguments: map(required=False)
	"""

I haven't tried running it, but it sounds like @munank has and it worked for him, so that's pretty good.

A few things to update:

• The residential workflow generator schema as mentioned above.
• Documentation for the residential workflow generator to document the schema changes.
• Changelog entry

[joseph-robertson]

@nmerket Anything else that needs to be done here?

[asparke2]

@joseph-robertson I think this is fine as-is and we can fix up the ComStock server directory cleanup as necessary. One thing to be aware of is that it won't work if custom_gems = True. I don't think you use that for ResStock, so probably not a big deal. This branch has additional changes to accommodate custom_gems with OS 3.x, but not going to worry about that until FY22.

[nmerket]

Looks good to me. You okay if I merge then?

[joseph-robertson]

Let me get review on NREL/resstock#661 and then I'll let you know.