change default num_buildings_represented and downselect criteria #185

vtnate · 2020-09-24T17:19:47Z

Pull Request Description

Based off Slack conversation this changes the default num_buildings_represented and the default downselect criteria in some example yml files. This way we capture the single family detached housing stock in a manner consistent with the rest of BuildStock.

Checklist

All other unit tests passing
Update existing documentation
Run a small batch run to make sure it all works (local is fine, unless an Eagle specific feature)

nmerket

Seems reasonable. I often just want to run SFD when doing testing because they're faster. Do we want to be limiting the example files to SFD @afontani @ejhw?

vtnate · 2020-09-25T15:55:37Z

We could instead just comment that line selecting SFD but still leave it in as an example of how to structure the filtering logic.

Changing from ~80 million to ~130 million buildings represented feels like a pretty big change (is that true?). I'm glad our example file now matches the BuildStock data.

afontani · 2020-09-25T20:06:55Z

So the number of units represented is the correct number from ACS 5-yr 2016 (aggregates from census track dwelling unit counts). Since we now have multifamily and vacant units, I feel like the project_resstock_national.yml file should be renamed. The YAML file isn't really the full national project but occupied SFD only.

In general, I would support a more exhaustive set of example yaml instead of just commenting out rows.

ejhw · 2020-09-25T20:18:44Z

Thanks @vtnate ! We've been talking about moving all ResStock yamls to the OS-BS repo anyway (they'd have to reference the compatible bsb version). I think @TobiAdekanye was going to work on this.

I'd prefer to just keep example yamls in this repo. Does that sound okay @nmerket ?

I agree with @afontani that project_resstock_national.yml should not be downselected and we could have a separate yaml pre-downselected to SFD if we want that.

vtnate · 2020-09-29T15:45:35Z

Thanks guys. I removed the downselection to SFD in project_resstock_national.yml, though I kept it in aws_demo_project.yml for the slightly different use case @ejhw suggested.

@afontani Are you suggesting we host a bunch of yml files showing details of different operations, or a single yml with everything in it that can be edited by the user?

afontani · 2020-09-29T15:55:06Z

@vtnate: I am suggesting a set of YAML files showing different example use cases. However, where these YAML files live is still a question for the group.

ejhw · 2020-09-30T18:32:18Z

aws_demo_project.yml

  n_datapoints: 4  # Comment this line out if using a custom buildstock csv file
-  n_buildings_represented: 80000000
+  n_buildings_represented: 133172057  # Total number of residential dwelling units in CONUS. Downselected later as desired.


Add reference for this number here too please

ejhw · 2020-09-30T18:32:38Z

project_resstock_national.yml

-  n_datapoints: 4 
-  n_buildings_represented: 80000000
+  n_datapoints: 4
+  n_buildings_represented: 133172057  # Total number of residential dwelling units in CONUS. Downselected later as desired.


Add reference for this number here too please

ejhw · 2020-09-30T18:34:04Z

project_resstock_national.yml

-#   logic:
-#     - Geometry Building Type ACS|Single-Family Detached
+downselect:
+  resample: false


does resample: false mean that downselect is disabled? If so, add comment "# downselect is disabled"

ejhw · 2020-09-30T18:36:41Z

project_resstock_national.yml

+downselect:
+  resample: false
+  logic:
+    - Vacancy Status|Occupied


Is this just here as an example? If so, add comment saying so.

I guess I'd prefer to keep "Geometry Building Type ACS|Single-Family Detached" as the example logic; seems like it would be more common than "Vacancy Status|Occupied".

ejhw · 2020-10-02T18:03:09Z

aws_demo_project.yml

 baseline:
  n_datapoints: 4  # Comment this line out if using a custom buildstock csv file
-  n_buildings_represented: 133172057  # Total number of residential dwelling units in CONUS. Downselected later as desired.
+  n_buildings_represented: 133172057  # Total number of residential dwelling units in contiguous United States as defined by
+  # 2016 American Community Survey by census region. Data available at data.census.gov.


Is it 5-yr ACS (2012-2016)? Does the count include all dwelling units including high-rise residential and vacant units? cc @afontani

I believe both are true. @afontani can you confirm?

@vtnate, @ejhw: Correct.

project_resstock_national.yml

nmerket

Looks good to me. Let's wait until we hear from @ejhw before merging.

ejhw · 2020-10-09T18:56:45Z

Looks good to me. Let's wait until we hear from @ejhw before merging.

Waiting for @afontani to confirm the exact definition of the dwelling unit count so we can document it here: https://github.com/NREL/buildstockbatch/pull/185/files#r498970703

ACS or PUMS? Which year(s)?
Includes all residential dwelling units, including high-rise multifamily and unoccupied units
Also would be nice to document where the number came from (e.g., what tsv maker script?)

@vtnate did you test running the ymls to make sure they run as is?

afontani · 2020-10-09T19:20:41Z

Looks good to me. Let's wait until we hear from @ejhw before merging.

Waiting for @afontani to confirm the exact definition of the dwelling unit count so we can document it here: https://github.com/NREL/buildstockbatch/pull/185/files#r498970703

ACS or PUMS? Which year(s)?

Includes all residential dwelling units, including high-rise multifamily and unoccupied units

Also would be nice to document where the number came from (e.g., what tsv maker script?)

@vtnate did you test running the ymls to make sure they run as is?

@ejhw: The number comes from a census tract level query of ACS 5-yr 2016 (i.e. 2012-2016).

vtnate · 2020-10-12T14:19:08Z

Also would be nice to document where the number came from (e.g., what tsv maker script?)

@afontani Can you share this path or filename?

… into num-buildings

afontani · 2020-10-12T14:49:51Z

Also would be nice to document where the number came from (e.g., what tsv maker script?)

@afontani Can you share this path or filename?

@vtnate: https://github.com/NREL/resstock-estimation/blob/master/sources/spatial/tsv_maker.py

ejhw

👍

vtnate · 2020-10-13T17:51:53Z

@ejhw Yes, both eagle and aws have been tested successfully with these yml files as they are currently written. Note that (unrelated to this PR) sampling takes a loooooooong time when downselecting. @nmerket has been working on an improved sampling script that may be able to help reduce the sampling time.

change default num_buildings_represented and downselect criteria

4bddd70

vtnate added the documentation label Sep 24, 2020

vtnate requested a review from nmerket September 24, 2020 17:19

vtnate self-assigned this Sep 24, 2020

nmerket reviewed Sep 25, 2020

View reviewed changes

clean up comments and down-selecting

6fc479e

ejhw reviewed Sep 30, 2020

View reviewed changes

document num_buildings, clean up formatting and comments

c215d60

ejhw reviewed Oct 2, 2020

View reviewed changes

project_resstock_national.yml Outdated Show resolved Hide resolved

vtnate and others added 2 commits October 8, 2020 10:11

restore downselect, aws, and postprocessing blocks

c3bcb4e

fixing some indentation

f760675

nmerket requested a review from ejhw October 9, 2020 16:27

nmerket approved these changes Oct 9, 2020

View reviewed changes

vtnate added 3 commits October 12, 2020 09:20

more precise wording for source of n_buildings_represented

86a8615

Merge branch 'num-buildings' of https://github.com/NREL/buildstockbatch…

db5b450

… into num-buildings

update link to tmy3 weather file

64372f0

vtnate and others added 4 commits October 12, 2020 10:05

update aws_demo_project with n_buildings_represented source

77f46f6

update project_resstock_national with n_buildings_represented source

aec2307

comment out downselect lines in aws_demo_project.yml

0a22402

Tweak comment

fc04919

Tweak comment

0d162cd

ejhw approved these changes Oct 12, 2020

View reviewed changes

nmerket merged commit 6439a50 into develop Oct 13, 2020

nmerket deleted the num-buildings branch October 13, 2020 17:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change default num_buildings_represented and downselect criteria #185

change default num_buildings_represented and downselect criteria #185

vtnate commented Sep 24, 2020 •

edited

nmerket left a comment

vtnate commented Sep 25, 2020

afontani commented Sep 25, 2020 •

edited

ejhw commented Sep 25, 2020

vtnate commented Sep 29, 2020

afontani commented Sep 29, 2020

ejhw Sep 30, 2020

ejhw Sep 30, 2020

ejhw Sep 30, 2020

ejhw Sep 30, 2020

ejhw Oct 2, 2020

vtnate Oct 8, 2020

afontani Oct 9, 2020

nmerket left a comment

ejhw commented Oct 9, 2020

afontani commented Oct 9, 2020

vtnate commented Oct 12, 2020

afontani commented Oct 12, 2020

ejhw left a comment

vtnate commented Oct 13, 2020

change default num_buildings_represented and downselect criteria #185

change default num_buildings_represented and downselect criteria #185

Conversation

vtnate commented Sep 24, 2020 • edited

Pull Request Description

Checklist

nmerket left a comment

Choose a reason for hiding this comment

vtnate commented Sep 25, 2020

afontani commented Sep 25, 2020 • edited

ejhw commented Sep 25, 2020

vtnate commented Sep 29, 2020

afontani commented Sep 29, 2020

ejhw Sep 30, 2020

Choose a reason for hiding this comment

ejhw Sep 30, 2020

Choose a reason for hiding this comment

ejhw Sep 30, 2020

Choose a reason for hiding this comment

ejhw Sep 30, 2020

Choose a reason for hiding this comment

ejhw Oct 2, 2020

Choose a reason for hiding this comment

vtnate Oct 8, 2020

Choose a reason for hiding this comment

afontani Oct 9, 2020

Choose a reason for hiding this comment

nmerket left a comment

Choose a reason for hiding this comment

ejhw commented Oct 9, 2020

afontani commented Oct 9, 2020

vtnate commented Oct 12, 2020

afontani commented Oct 12, 2020

ejhw left a comment

Choose a reason for hiding this comment

vtnate commented Oct 13, 2020

vtnate commented Sep 24, 2020 •

edited

afontani commented Sep 25, 2020 •

edited