Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glue crawler Athena table creation fails silently on invalid yml input #159

Closed
asparke2 opened this issue May 27, 2020 · 3 comments · Fixed by #202
Closed

Glue crawler Athena table creation fails silently on invalid yml input #159

asparke2 opened this issue May 27, 2020 · 3 comments · Fixed by #202
Assignees
Labels
bug Something isn't working

Comments

@asparke2
Copy link
Member

Describe the bug
If the aws > s3 >prefix entry in the yml file has a trailing /, the s3 upload will work, and the glue crawler will be created, but no Athena table will be created because the s3 location will be invalid.

To Reproduce
Steps to reproduce the behavior:

  1. Add a trailing / to the aws:s3:prefix entry in the yml file as shown below
postprocessing:
  aws:
    region_name: 'us-west-2'
    s3:
      bucket: eulp    
      prefix: simulation_output/regional_runs/comstock/region-1-fort-collins-3500-ts/
    athena:
      glue_service_role: service-role/AWSGlueServiceRole-default
      database_name: testing
      max_crawling_time: 300 #time to wait for the crawler to complete before aborting it
  1. run a post-process only run

Expected behavior
First, sanitize the input to strip trailing /. Second, check that files exist inside the target s3 bucket before initializing the glue crawler.

Logs
Root issue was identified by noticing // in the following line of postprocessing.out:

INFO:2020-05-22 15:00:22:buildstockbatch.postprocessing:Upload to S3 completed. The files are uploaded to: eulp/simulation_output/regional_runs/comstock/region-1-fort-collins-3500-ts//region-1-fort-collins-3500-ts/

Platform (please complete the following information):

  • Eagle
  • BuildStockBatch 0.18
  • OpenStudio-BuildStock version, branch, or sha: comstockv1.8
  • Local Desktop OS: Windows
@asparke2 asparke2 added the bug Something isn't working label May 27, 2020
@asparke2 asparke2 self-assigned this May 27, 2020
@nmerket
Copy link
Member

nmerket commented Jun 9, 2020

You gonna open up a PR for this or what?

@asparke2
Copy link
Member Author

If you'd prefer a PR in the current state I'll do it, but I figured I'd wait until I had fixed added/fixed unit tests for it first since there is an easy workaround. Right now the unit tests are failing on new code which checks the s3 location, and the test s3 location doesn't really exist.

@nmerket
Copy link
Member

nmerket commented Jun 15, 2020

I'd be open to a draft PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants