Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor df_to_download #166

Closed
lchen-2101 opened this issue Apr 23, 2024 · 0 comments · Fixed by #167
Closed

Refactor df_to_download #166

lchen-2101 opened this issue Apr 23, 2024 · 0 comments · Fixed by #167
Assignees
@jcadam14

Comments

@lchen-2101
Copy link
Collaborator

refactor df_to_download is pandas can be leveraged to generate the csv, instead of manually constructing the csv string
@jcadam14 jcadam14 self-assigned this Apr 24, 2024
@jcadam14 jcadam14 changed the title Refactor df_to_download (POST MVP) Refactor df_to_download Apr 24, 2024
@jcadam14 jcadam14 mentioned this issue Apr 24, 2024
Merged
@jcadam14 jcadam14 linked a pull request Apr 24, 2024 that will close this issue
Changed to use pivot tables in the download formatter #167
Merged
lchen-2101 pushed a commit that referenced this issue Apr 30, 2024
@jcadam14
Changed to use pivot tables in the download formatter (#167)
5e312ea 
Closes #166 

Changed the df_to_download to use pivot tables to generate the csv. This
greatly improves the processing time of this function to just a few
seconds for very large files, and consistently stays there as size
increases.

Also used QUOTE_NONNUMERIC instead of specifically escaping just the
description column.
jcadam14 added a commit that referenced this issue May 3, 2024
@jcadam14
Changed to use pivot tables in the download formatter (#167)
7b0aeff 
Closes #166 

Changed the df_to_download to use pivot tables to generate the csv. This
greatly improves the processing time of this function to just a few
seconds for very large files, and consistently stays there as size
increases.

Also used QUOTE_NONNUMERIC instead of specifically escaping just the
description column.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcadam14 jcadam14

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Changed to use pivot tables in the download formatter cfpb/regtech-data-validator
2 participants
@jcadam14 @lchen-2101