Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/glue crawler trailing space failure #202

Merged
merged 11 commits into from Apr 14, 2021

Conversation

asparke2
Copy link
Member

Fixes #159

Pull Request Description

Removes any trailing slash from s3_prefix to avoid Athena table creation issues. Also warns if the target s3 directory is empty, which would result in an unexpectedly empty table in Athena.

Checklist

Not all may apply

  • Code changes (must work)
  • Tests exercising your feature/bug fix (check coverage report on CircleCI build -> Artifacts)
  • All other unit tests passing
  • Run a small batch run to make sure it all works (local is fine, unless an Eagle specific feature)

@nmerket
Copy link
Member

nmerket commented Apr 7, 2021

@rajeee @asparke2 The test_upload_files test is failing here. Can one of you look into it?

@asparke2
Copy link
Member Author

asparke2 commented Apr 7, 2021

@nmerket it fails because now the code checks the s3 directory for files before attempting to run the glue crawler. Because the CI isn't accessing a real s3 bucket, this fails. I assume the solution is to mock an s3 response, but I couldn't figure out how to do this and that's why the PR was sitting.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Glue crawler Athena table creation fails silently on invalid yml input
3 participants