Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: racially insensitive language #205

Merged
merged 12 commits into from Mar 9, 2021
Merged

Fix: racially insensitive language #205

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6a69577
change notation for master and slave instances
aspeake Jan 13, 2021
517049d
revert changes to v0.1 and v0.2 schemas
aspeake Jan 19, 2021
568e74d
Merge branch 'develop' into issue-164
nmerket Jan 25, 2021
caf321e
Merge remote-tracking branch 'origin/develop' into issue-164
nmerket Feb 26, 2021
84c0794
updating v0.3.yaml schema for new keys
nmerket Feb 26, 2021
f932f51
uploading only unique weather files
nmerket Mar 4, 2021
7ae72de
getting to work on aws again...maybe
nmerket Mar 8, 2021
c3d526d
style fix and reverting some changes in aws example file
nmerket Mar 9, 2021
4de8760
updating numpy>=1.20 and pyarrow>=3.0 as those seem to work with each…
nmerket Feb 20, 2021
555cc0f
Merge remote-tracking branch 'origin/develop' into issue-164
nmerket Mar 9, 2021
5052701
switching back to the circleci python image and installing all deps w…
nmerket Mar 9, 2021
b486416
updating changelog and migration guide
nmerket Mar 9, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 4 additions & 2 deletions aws_demo_project.yml
View file
Open in desktop
@@ -1,6 +1,6 @@
schema_version: 0.2
stock_type: residential
buildstock_directory: ../OpenStudio-BuildStock # Relative to this file or absolute
buildstock_directory: ../resstock # Relative to this file or absolute
project_directory: project_national # Relative to buildstock_directory
output_directory: ../demo_test_outputs
weather_files_url: https://data.nrel.gov/system/files/128/ResStock_TMY3.zip
Expand Down Expand Up @@ -36,7 +36,9 @@ aws:
bucket: resbldg-datasets
prefix: testing/external_demo_project
emr:
slave_instance_count: 1
manager_instance_type: m5.xlarge
worker_instance_type: r5.4xlarge
worker_instance_count: 1
region: us-west-2
use_spot: true
batch_array_size: 4
Expand Down
10 changes: 5 additions & 5 deletions buildstockbatch/aws/aws.py
View file
Open in desktop
Expand Up @@ -1486,14 +1486,14 @@ def create_emr_cluster_function(self):
{
'Market': 'SPOT' if self.batch_use_spot else 'ON_DEMAND',
'InstanceRole': 'MASTER',
'InstanceType': self.emr_master_instance_type,
'InstanceType': self.emr_manager_instance_type,
'InstanceCount': 1
},
{
'Market': 'SPOT' if self.batch_use_spot else 'ON_DEMAND',
'InstanceRole': 'CORE',
'InstanceType': self.emr_slave_instance_type,
'InstanceCount': self.emr_slave_instance_count
'InstanceType': self.emr_worker_instance_type,
'InstanceCount': self.emr_worker_instance_count
},
],
'Ec2SubnetId': self.priv_vpc_subnet_id_1,
Expand Down Expand Up @@ -1667,8 +1667,8 @@ def validate_instance_types(project_file):
ec2 = boto3_session.client('ec2')
job_base = AwsJobBase('genericjobid', aws_config, boto3_session)
instance_types_requested = set()
instance_types_requested.add(job_base.emr_master_instance_type)
instance_types_requested.add(job_base.emr_slave_instance_type)
instance_types_requested.add(job_base.emr_manager_instance_type)
instance_types_requested.add(job_base.emr_worker_instance_type)
inst_type_resp = ec2.describe_instance_type_offerings(Filters=[{
'Name': 'instance-type',
'Values': list(instance_types_requested)
Expand Down
6 changes: 3 additions & 3 deletions buildstockbatch/aws/awsbase.py
View file
Open in desktop
Expand Up @@ -177,9 +177,9 @@ def __init__(self, job_identifier, aws_config, boto3_session):

# EMR
emr_config = aws_config.get('emr', {})
self.emr_master_instance_type = emr_config.get('master_instance_type', 'm5.4xlarge')
self.emr_slave_instance_type = emr_config.get('slave_instance_type', 'r5.4xlarge')
self.emr_slave_instance_count = emr_config.get('slave_instance_count', 4)
self.emr_manager_instance_type = emr_config.get('manager_instance_type', 'm5.4xlarge')
self.emr_worker_instance_type = emr_config.get('worker_instance_type', 'r5.4xlarge')
self.emr_worker_instance_count = emr_config.get('worker_instance_count', 4)
self.emr_cluster_security_group_name = f'{self.job_identifier}_emr_security_group'
self.emr_cluster_name = f'{self.job_identifier}_emr_dask_cluster'
self.emr_job_flow_role_name = f'{self.job_identifier}_emr_job_flow_role'
Expand Down
6 changes: 3 additions & 3 deletions buildstockbatch/schemas/v0.1.yaml
View file
Open in desktop
Expand Up @@ -33,9 +33,9 @@ aws-job-environment:
memory: int(min=1024, required=False)

aws-emr-spec:
master_instance_type: str(required=False)
slave_instance_type: str(required=False)
slave_instance_count: int(min=1, required=False)
manager_instance_type: str(required=False)
worker_instance_type: str(required=False)
worker_instance_count: int(min=1, required=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't be changing old schemas as the only function they serve now is to keep a record of previous input formats.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, addressed in 517049d

dask_worker_vcores: int(min=1, required=False)

hpc-spec:
Expand Down
6 changes: 3 additions & 3 deletions buildstockbatch/schemas/v0.2.yaml
View file
Open in desktop
Expand Up @@ -35,9 +35,9 @@ aws-job-environment:
memory: int(min=1024, required=False)

aws-emr-spec:
master_instance_type: str(required=False)
slave_instance_type: str(required=False)
slave_instance_count: int(min=1, required=False)
manager_instance_type: str(required=False)
worker_instance_type: str(required=False)
worker_instance_count: int(min=1, required=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here. Since we're making breaking changes to the input format, we should make a v0.3 schema and upgrade to that. I actually have a v0.3 schema going on in #187. Maybe we should wait for that to merge (soon) and make these changes there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 517049d. That sounds good, we can hold off for the next schema to get merged.

dask_worker_vcores: int(min=1, required=False)

hpc-spec:
Expand Down
6 changes: 3 additions & 3 deletions docs/project_defn.rst
View file
Open in desktop
Expand Up @@ -354,9 +354,9 @@ on the `AWS Batch <https://aws.amazon.com/batch/>`_ service.
subscription to receive further notification emails.
* ``emr``: Optional key to specify options for postprocessing using an EMR cluster. Generally the defaults should work fine.

* ``master_instance_type``: The `instance type`_ to use for the EMR master node. Default: ``m5.xlarge``.
* ``slave_instance_type``: The `instance type`_ to use for the EMR worker nodes. Default: ``r5.4xlarge``.
* ``slave_instance_count``: The number of worker nodes to use. Same as ``eagle.postprocessing.n_workers``.
* ``manager_instance_type``: The `instance type`_ to use for the EMR master node. Default: ``m5.xlarge``.
* ``worker_instance_type``: The `instance type`_ to use for the EMR worker nodes. Default: ``r5.4xlarge``.
* ``worker_instance_count``: The number of worker nodes to use. Same as ``eagle.postprocessing.n_workers``.
Increase this for a large dataset. Default: 2.
* ``dask_worker_vcores``: The number of cores for each dask worker. Increase this if your dask workers are running out of memory. Default: 2.
* ``job_environment``: Specifies the computing requirements for each simulation.
Expand Down
2 changes: 1 addition & 1 deletion project_resstock_national.yml
View file
Open in desktop
Expand Up @@ -48,7 +48,7 @@ eagle:
# bucket: resbldg-datasets
# prefix: testing/user_test
# emr:
# slave_instance_count: 1
# worker_instance_count: 1
# region: us-west-2
# use_spot: true
# batch_array_size: 10
Expand Down