New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Integration Tests #326
Changes from 8 commits
446bdf4
bb8c526
fc668d1
e7d40f8
3aceed4
0febf50
1c8de64
fd5e22f
1735b9d
5474050
3d128c6
8ad08fd
563b9e7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,60 +1,82 @@ | ||
name: BuildStockBatch Tests | ||
on: [pull_request] | ||
on: | ||
push: | ||
branches: | ||
- develop | ||
pull_request: | ||
types: | ||
- synchronize | ||
- opened | ||
jobs: | ||
build: | ||
tests: | ||
runs-on: ubuntu-latest | ||
strategy: | ||
fail-fast: false | ||
matrix: | ||
python-version: ['3.8', '3.9', '3.10'] | ||
name: Tests - Python ${{ matrix.python-version }} | ||
steps: | ||
- uses: actions/checkout@v2 | ||
- uses: actions/setup-python@v2 | ||
- uses: actions/checkout@v3 | ||
with: | ||
path: buildstockbatch | ||
- uses: actions/checkout@v3 | ||
with: | ||
repository: NREL/resstock | ||
path: resstock | ||
ref: develop | ||
Comment on lines
+19
to
+26
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Checking out both repos |
||
- uses: actions/setup-python@v4 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I generally tried to update to the latest versions of all the actions. |
||
with: | ||
python-version: ${{ matrix.python-version }} | ||
- name: Download weather | ||
run: | | ||
mkdir weather | ||
cd weather | ||
wget --quiet https://data.nrel.gov/system/files/156/BuildStock_TMY3_FIPS.zip | ||
Comment on lines
+30
to
+34
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. All the testing files use the same batch of weather files. This keeps it from being downloaded 4 times. |
||
- name: Install buildstockbatch | ||
run: | | ||
cd buildstockbatch | ||
python -m pip install --progress-bar off --upgrade pip | ||
pip install .[dev] --progress-bar off | ||
- name: Linting | ||
run: | | ||
cd buildstockbatch | ||
# stop the build if there are Python syntax errors or undefined names | ||
flake8 buildstockbatch --count --select=E9,F63,F7,F82 --show-source --statistics | ||
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide | ||
flake8 buildstockbatch --count --max-line-length=127 --statistics | ||
- name: Run PyTest and Coverage | ||
run: | | ||
cd buildstockbatch | ||
pytest --junitxml=coverage/junit.xml --cov=buildstockbatch --cov-report=xml:coverage/coverage.xml --cov-report=html:coverage/htmlreport | ||
- name: Test Report | ||
uses: mikepenz/action-junit-report@v2.9.1 | ||
if: ${{ matrix.python-version == '3.9' }} | ||
uses: mikepenz/action-junit-report@v3.5.2 | ||
if: ${{ matrix.python-version == '3.10' }} | ||
with: | ||
report_paths: coverage/junit.xml | ||
report_paths: buildstockbatch/coverage/junit.xml | ||
check_name: Testing Report | ||
fail_on_failure: true | ||
- name: Save Coverage Report | ||
uses: actions/upload-artifact@v2 | ||
if: ${{ matrix.python-version == '3.9' }} | ||
uses: actions/upload-artifact@v3 | ||
if: ${{ matrix.python-version == '3.10' }} | ||
with: | ||
name: coverage-report-html | ||
path: coverage/htmlreport/ | ||
path: buildstockbatch/coverage/htmlreport/ | ||
- name: Report coverage to PR | ||
uses: 5monkeys/cobertura-action@v12 | ||
if: ${{ matrix.python-version == '3.9' }} | ||
uses: 5monkeys/cobertura-action@v13 | ||
if: ${{ matrix.python-version == '3.10' }} | ||
with: | ||
path: coverage/coverage.xml | ||
path: buildstockbatch/coverage/coverage.xml | ||
repo_token: ${{ secrets.GITHUB_TOKEN }} | ||
minimum_coverage: 24 | ||
minimum_coverage: 33 | ||
fail_below_threshold: true | ||
- name: Build documentation | ||
if: ${{ matrix.python-version == '3.9' }} | ||
if: ${{ matrix.python-version == '3.10' }} | ||
run: | | ||
cd docs | ||
cd buildstockbatch/docs | ||
make html SPHINXOPTS="-W --keep-going -n" | ||
- name: Save Docs | ||
uses: actions/upload-artifact@v2 | ||
if: ${{ matrix.python-version == '3.9' }} | ||
uses: actions/upload-artifact@v3 | ||
if: ${{ matrix.python-version == '3.10' }} | ||
with: | ||
name: documentation | ||
path: docs/_build/html/ |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,73 @@ | ||
import os | ||
import pathlib | ||
import pytest | ||
import shutil | ||
|
||
from buildstockbatch.localdocker import LocalDockerBatch | ||
|
||
resstock_directory = pathlib.Path( | ||
os.environ.get("RESSTOCK_DIR", pathlib.Path(__file__).resolve().parent.parent.parent.parent / "resstock") | ||
) | ||
resstock_required = pytest.mark.skipif( | ||
not resstock_directory.exists(), | ||
reason="ResStock checkout is not found" | ||
) | ||
|
||
|
||
@pytest.mark.parametrize("project_filename", [ | ||
resstock_directory / "project_national" / "national_baseline.yml", | ||
resstock_directory / "project_national" / "national_upgrades.yml", | ||
resstock_directory / "project_testing" / "testing_baseline.yml", | ||
resstock_directory / "project_testing" / "testing_upgrades.yml", | ||
], ids=lambda x: x.stem) | ||
Comment on lines
+18
to
+23
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm testing each of the project yaml files in resstock. |
||
@resstock_required | ||
def test_resstock_local_batch(project_filename, mocker): | ||
LocalDockerBatch.validate_project(str(project_filename)) | ||
batch = LocalDockerBatch(str(project_filename)) | ||
|
||
# Get the number of upgrades | ||
n_upgrades = len(batch.cfg.get("upgrades", [])) | ||
# Limit the number of upgrades to 2 to reduce simulation time | ||
if n_upgrades > 2: | ||
batch.cfg["upgrades"] = batch.cfg["upgrades"][0:2] | ||
n_upgrades = 2 | ||
|
||
# Modify the number of datapoints so we're not here all day. | ||
if n_upgrades == 0: | ||
n_datapoints = 4 | ||
else: | ||
n_datapoints = 2 | ||
batch.cfg["sampler"]["args"]["n_datapoints"] = n_datapoints | ||
Comment on lines
+29
to
+41
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't want to spend forever running these ResStock runs. I just want to make sure the machinery works, so I'm limiting the number of upgrades and datapoints to keep the CI running quickly. |
||
|
||
# FIXME: Find a better way to do this | ||
local_weather_file = resstock_directory.parent / "weather" / batch.cfg["weather_files_url"].split("/")[-1] | ||
if local_weather_file.exists(): | ||
del batch.cfg["weather_files_url"] | ||
batch.cfg["weather_files_path"] = str(local_weather_file) | ||
Comment on lines
+43
to
+46
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It'll check if the weather is found locally and swap that in instead. Since we downloaded it earlier to the appropriate place it should just be able to unzip it. |
||
|
||
batch.run_batch() | ||
|
||
# Make sure all the files are there | ||
out_path = pathlib.Path(batch.output_dir) | ||
simout_path = out_path / "simulation_output" | ||
assert (simout_path / "results_job0.json.gz").exists() | ||
assert (simout_path / "simulations_job0.tar.gz").exists() | ||
|
||
for upgrade_id in range(0, n_upgrades + 1): | ||
for bldg_id in range(1, n_datapoints + 1): | ||
assert (simout_path / "timeseries" / f"up{upgrade_id:02d}" / f"bldg{bldg_id:07d}.parquet").exists() | ||
|
||
batch.process_results() | ||
|
||
assert not (simout_path / "timeseries").exists() | ||
assert not (simout_path / "results_job0.json.gz").exists() | ||
assert (simout_path / "simulations_job0.tar.gz").exists() | ||
assert (out_path / "parquet" / "baseline" / "results_up00.parquet").exists() | ||
ts_pq_path = out_path / "parquet" / "timeseries" | ||
for upgrade_id in range(0, n_upgrades + 1): | ||
assert (ts_pq_path / f"upgrade={upgrade_id}" / "group0.parquet").exists() | ||
assert (out_path / "results_csvs" / f"results_up{upgrade_id:02d}.csv.gz").exists() | ||
assert (ts_pq_path / "_common_metadata").exists() | ||
assert (ts_pq_path / "_metadata").exists() | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do all these asserts check to see if all the simulations failed? The machinery might work, but there should be some real results. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point. It does check that timeseries files exist for each simulation, but I will also add something to open the results parquet file and ensure they all succeeded. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Checking to see "all" succeeded seems robust, but we might accidentally sample a building that will fail in OpenStudio-HPXML. The current failure rate seems low (~0.1%) enough. At least 1 successful run might be okay. |
||
shutil.rmtree(out_path) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doing these checks on merge into develop as well on @shorowit's recommendation.