Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

168 see if the df to json can be improved using pivot tables similar to the df to download #170

Conversation

jcadam14
Copy link
Contributor

Closes #168

Wasn't able to use pivot tables, but in looking at the df.to_json() output and understanding that structure, found a much quicker way to translate it to the json we want using just python dict references instead of the dataframes. Significant improvement in processing time, especially as the dataset grows.

Copy link

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  src/regtech_data_validator
  data_formatters.py 123
Project Total  

This report was generated by python-coverage-comment-action

…ing-pivot-tables-similar-to-the-df_to_download
Copy link
Collaborator

@lchen-2101 lchen-2101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tested fine, and gave out the same result as the previous implementation. at some post-mvp time, let's refactor this and the csv one to be more readable.

@lchen-2101 lchen-2101 merged commit a3c0f67 into main Apr 30, 2024
5 checks passed
@lchen-2101 lchen-2101 deleted the 168-see-if-the-df_to_json-can-be-improved-using-pivot-tables-similar-to-the-df_to_download branch April 30, 2024 18:00
jcadam14 added a commit that referenced this pull request May 3, 2024
…to the df to download (#170)

Closes #168 

Wasn't able to use pivot tables, but in looking at the df.to_json()
output and understanding that structure, found a much quicker way to
translate it to the json we want using just python dict references
instead of the dataframes. Significant improvement in processing time,
especially as the dataset grows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

See if the df_to_json can be improved using pivot tables similar to the df_to_download
2 participants