Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Writing parquet files with newer datetime format #224

Merged
merged 4 commits into from Apr 27, 2021
Merged

Writing parquet files with newer datetime format #224

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7156f12
removing flavor=spark from parquet writing
nmerket Apr 26, 2021
4e39fa6
updating docs for athena engine version
nmerket Apr 26, 2021
e8211a4
updating changelog
nmerket Apr 26, 2021
d1600c9
Merge remote-tracking branch 'origin/develop' into athena-datetime
nmerket Apr 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion buildstockbatch/postprocessing.py
View file
Open in desktop
Expand Up @@ -147,7 +147,7 @@ def read_simulation_outputs(fs, reporting_measures, sim_dir, upgrade_id, buildin
def write_dataframe_as_parquet(df, fs, filename):
tbl = pa.Table.from_pandas(df, preserve_index=False)
with fs.open(filename, 'wb') as f:
parquet.write_table(tbl, f, flavor='spark')
parquet.write_table(tbl, f)


def clean_up_results_df(df, cfg, keep_upgrade_id=False):
Expand Down
14 changes: 11 additions & 3 deletions docs/changelog/changelog_dev.rst
View file
Open in desktop
Expand Up @@ -61,9 +61,17 @@ Development Changelog
This fixes that as well as checks to make sure files were uploaded
before running the crawler.

.. change::
:tags: bugfix
:pullreq: 224
:tickets: 221

Defaults to the newer datetime encoding in the parquet files now that
Athena can understand it.

.. change::
:tags: workflow
:pullreq:
:tickets:
:pullreq: 219
:tickets: 189

Adding measure arguments for reporting measures in the workflow generator.
Adding measure arguments for reporting measures in the workflow generator.
2 changes: 2 additions & 0 deletions docs/installation.rst
View file
Open in desktop
Expand Up @@ -66,9 +66,11 @@ configure your user account with your AWS credentials. This setup only needs to

1. `Install the AWS CLI`_ version 2 (select the version for your local OS; not for Docker).
2. `Configure the AWS CLI`_. (Don't type the ``$`` in the example.)
3. You may need to `change the Athena Engine version`_ for your query workgroup to v2.

.. _Install the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
.. _Configure the AWS CLI: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html#cli-quick-configuration
.. _change the Athena Engine version: https://docs.aws.amazon.com/athena/latest/ug/engine-versions-changing.html

.. _eagle_install:

Expand Down