Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed to use pivot tables in the download formatter #167

Merged
merged 4 commits into from Apr 30, 2024

Conversation

jcadam14
Copy link
Contributor

Closes #166

Changed the df_to_download to use pivot tables to generate the csv. This greatly improves the processing time of this function to just a few seconds for very large files, and consistently stays there as size increases.

Also used QUOTE_NONNUMERIC instead of specifically escaping just the description column.

@jcadam14 jcadam14 self-assigned this Apr 24, 2024
@jcadam14 jcadam14 linked an issue Apr 24, 2024 that may be closed by this pull request
Copy link

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/regtech_data_validator
  create_schemas.py
  data_formatters.py
Project Total  

This report was generated by python-coverage-comment-action

Copy link
Collaborator

@lchen-2101 lchen-2101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lchen-2101 lchen-2101 merged commit 5e312ea into main Apr 30, 2024
5 checks passed
@lchen-2101 lchen-2101 deleted the 166-refactor-df_to_download-post-mvp branch April 30, 2024 17:51
jcadam14 added a commit that referenced this pull request May 3, 2024
Closes #166 

Changed the df_to_download to use pivot tables to generate the csv. This
greatly improves the processing time of this function to just a few
seconds for very large files, and consistently stays there as size
increases.

Also used QUOTE_NONNUMERIC instead of specifically escaping just the
description column.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Refactor df_to_download
2 participants