Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing timestamps in Athena #418

Merged
merged 7 commits into from Nov 21, 2023
Merged

Fixing timestamps in Athena #418

merged 7 commits into from Nov 21, 2023

Conversation

nmerket
Copy link
Member

@nmerket nmerket commented Nov 17, 2023

  • fixing timestamps
  • fixing crawler script

Fixes #411.

Pull Request Description

Changes the timestamps in the parquet files to be a second precision (which is all the more we measure). The default was nanoseconds in pandas 2. Glue/Athena will only work with milliseconds or larger.

Checklist

Not all may apply

  • Code changes (must work)
  • Tests exercising your feature/bug fix (check coverage report on Checks -> BuildStockBatch Tests -> Artifacts)
  • Coverage has increased or at least not decreased. Update minimum_coverage in .github/workflows/ci.yml as necessary.
  • All other unit and integration tests passing
  • Update validation for project config yaml file changes
  • Update existing documentation
  • Run a small batch run on Eagle to make sure it all works if you made changes that will affect Eagle
  • Add to the changelog_dev.rst file and propose migration text in the pull request

Copy link

github-actions bot commented Nov 20, 2023

File Coverage
All files 85%
base.py 90%
eagle.py 77%
exc.py 57%
local.py 70%
postprocessing.py 85%
utils.py 91%
cloud/docker_base.py 0%
sampler/base.py 79%
sampler/downselect.py 33%
sampler/precomputed.py 93%
sampler/residential_quota.py 61%
test/test_docker.py 33%
test/test_validation.py 97%
workflow_generator/base.py 90%
workflow_generator/commercial.py 53%
workflow_generator/residential_hpxml.py 86%

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against cfcd34d

@nmerket nmerket self-assigned this Nov 20, 2023
@nmerket nmerket merged commit a02603f into develop Nov 21, 2023
6 checks passed
@nmerket nmerket deleted the athena_timestamps branch November 21, 2023 16:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

glue crawler is casting time variables as bigints instead of timestamps
1 participant