Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Writing parquet files with newer datetime format #224

Merged
merged 4 commits into from Apr 27, 2021
Merged

Conversation

nmerket
Copy link
Member

@nmerket nmerket commented Apr 26, 2021

Fixes #221.

Pull Request Description

Removes the flavor='spark' from postprocessing.py as it no longer does what was intended and isn't necessary given Athena can understand the new datetime format.

I have run a small test with this and confirmed that Athena properly interprets the datetime fields.

Checklist

Not all may apply

  • Code changes (must work)
  • Tests exercising your feature/bug fix (check coverage report on CircleCI build -> Artifacts)
  • All other unit tests passing
  • Update validation for project config yaml file changes
  • Update existing documentation
  • Run a small batch run to make sure it all works (local is fine, unless an Eagle specific feature)
  • Add to the changelog_dev.rst file and propose migration text in the pull request

@nmerket nmerket changed the title removing flavor=spark from parquet writing Writing parquet files with newer datetime format Apr 26, 2021
@nmerket nmerket marked this pull request as ready for review April 26, 2021 22:17
@nmerket nmerket requested a review from T-Man-Stan April 26, 2021 22:18
@nmerket nmerket self-assigned this Apr 26, 2021
@nmerket nmerket added the bug Something isn't working label Apr 26, 2021
@nmerket nmerket merged commit b98ba46 into develop Apr 27, 2021
@nmerket nmerket deleted the athena-datetime branch April 27, 2021 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BSB to Athena Datetime Encoding Corrupted
1 participant