Initial addition of validation capabilities

[rHorsey]

@nmerket @joseph-robertson Some initial thoughts for our discussion tomorrow.

Cheers,

[nmerket]

@rHorsey What do we need to do to get this ready to merge?

[rHorsey]

@nmerket A few more hours. Once I'm happy with my understanding of existing testing I'll extend it. I have a good example yml already.

[rHorsey]

@nmerket This is ready for review.

[rHorsey]

Also @nmerket I'm not sure why that col-names are causing issues in the test. I reran the master tests and buildstockbatch/test/test_base.py::test_combine_files fails with the same error as in this branch. See https://circleci.com/gh/NREL/buildstockbatch/156

[rajeee]

Also @nmerket I'm not sure why that col-names are causing issues in the test. I reran the master tests and buildstockbatch/test/test_base.py::test_combine_files fails with the same error as in this branch. See https://circleci.com/gh/NREL/buildstockbatch/156

@rHorsey Apparently, the order of the columns are different between the reference results.csv and the results.csv created during the test.

When I run the pytest on the latest master in my computer, it is working fine.

Two solutions that comes to mind (both of which involves some coding on my part):

1. Ensure that the column orders remain consistent when when creating the results.csv. My guess is the column are created by iterating over the keys of a dictionary or using dask's to_dataframe function , which doesn't always return things in the same order.

2. Make the testing column-order independent.

I think option 1 is probably a better idea for consistency, but option 2 would make the testing a bit more flexible to future (deliberate) order changes. Maybe a combination of both?

[nmerket]

I like where this is going. I think there's some stuff missing from the validation, which I commented on.

@joseph-robertson I assume we'll do extra validation against what's in NREL/OpenStudio-BuildStock as another PR?

[joseph-robertson]

I like where this is going. I think there's some stuff missing from the validation, which I commented on.

@joseph-robertson I assume we'll do extra validation against what's in NREL/OpenStudio-BuildStock as another PR?

@nmerket Sounds good to me.

[rHorsey]

@nmerket Ready for a second round of reviews. Thanks for the CLI test! Thoughts on allowing cfg['eagle']['n_jobs'] to not be set?

[nmerket]

@rHorsey A few more minor comments and suggestions.

As for cfg['eagle']['n_jobs'] If we have a good default, we could let it not be set. Do you have a recommendation for a default or a nice way to calculate the number of jobs we should use?

One last thought, this probably needs some testing with real project files that are in use "out in the wild". I'm on the fence of whether I 1.) collect a few project files from people and run them through the validation, or 2.) just merge this and deal with the backlash.

I should probably do 1. I'll go ask around for some project files.

[rHorsey]

@nmerket Please do. @ekpresent sent me what she's been using in the wild, which is what the complete test file is based off of. If you can find some more that'd be great. Otherwise I think we're ready to merge. Also I'm not opposed to 2.

[nmerket]

I got a few yaml files. I'll run them through the validator this morning as a sanity check. Otherwise this looks good to me.

[nmerket]

@rHorsey I made one change to the csv export measure arguments. Namely that I'm not validating them. We'll write a separate validator for that.

This is ready to merge. I ran CI locally because OS Standards was taking too long.