You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
If the aws > s3 >prefix entry in the yml file has a trailing /, the s3 upload will work, and the glue crawler will be created, but no Athena table will be created because the s3 location will be invalid.
To Reproduce
Steps to reproduce the behavior:
Add a trailing / to the aws:s3:prefix entry in the yml file as shown below
postprocessing:
aws:
region_name: 'us-west-2'
s3:
bucket: eulp
prefix: simulation_output/regional_runs/comstock/region-1-fort-collins-3500-ts/
athena:
glue_service_role: service-role/AWSGlueServiceRole-default
database_name: testing
max_crawling_time: 300 #time to wait for the crawler to complete before aborting it
run a post-process only run
Expected behavior
First, sanitize the input to strip trailing /. Second, check that files exist inside the target s3 bucket before initializing the glue crawler.
Logs
Root issue was identified by noticing // in the following line of postprocessing.out:
INFO:2020-05-22 15:00:22:buildstockbatch.postprocessing:Upload to S3 completed. The files are uploaded to: eulp/simulation_output/regional_runs/comstock/region-1-fort-collins-3500-ts//region-1-fort-collins-3500-ts/
Platform (please complete the following information):
Eagle
BuildStockBatch 0.18
OpenStudio-BuildStock version, branch, or sha: comstockv1.8
Local Desktop OS: Windows
The text was updated successfully, but these errors were encountered:
If you'd prefer a PR in the current state I'll do it, but I figured I'd wait until I had fixed added/fixed unit tests for it first since there is an easy workaround. Right now the unit tests are failing on new code which checks the s3 location, and the test s3 location doesn't really exist.
Describe the bug
If the
aws > s3 >prefix
entry in the yml file has a trailing/
, the s3 upload will work, and the glue crawler will be created, but no Athena table will be created because the s3 location will be invalid.To Reproduce
Steps to reproduce the behavior:
/
to theaws:s3:prefix
entry in the yml file as shown belowExpected behavior
First, sanitize the input to strip trailing
/
. Second, check that files exist inside the target s3 bucket before initializing the glue crawler.Logs
Root issue was identified by noticing
//
in the following line ofpostprocessing.out
:Platform (please complete the following information):
The text was updated successfully, but these errors were encountered: