Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split web data into chunks #303

Open
2 tasks
Tracked by #304
ivan-aksamentov opened this issue Apr 29, 2022 · 0 comments
Open
2 tasks
Tracked by #304

Split web data into chunks #303

ivan-aksamentov opened this issue Apr 29, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@ivan-aksamentov
Copy link
Member

ivan-aksamentov commented Apr 29, 2022

The following JSON files contain data collections in array formats:

  • /web/data/clusters.json
  • /web/data/perClusterData.json
  • /web/data/perCountryData.json
  • /web/data/perCountryDataCaseCounts.json

Currently, the entire array needs to be downloaded every time, even if only some items are needed. In fact, the full download is always happening on page load, because the static import is used (see #302). This worked well in the beginning and we all hoped that pandemic is almost over back then. However, after more than a year, the data sizes have grown and mono-array is not an adequate solution anymore.

We could split the arrays into elements and allow the app to operate on individual items ("chunks"). Due to a sheer volume, we've already done this for acknowledgements data. Now it's time to transition the rest of data.

Steps:

  • adjust web data generation to produce separate JSON files instead of one large file with an array, plus an index file enumerating identifiers for every chunk
  • adjust app logic to handle chunked data, instead of iteration over an array

This makes the most sense after or together with #302, because it will revamp the whole web app logic. We don't want to do this twice.

@ivan-aksamentov ivan-aksamentov added the enhancement New feature or request label Apr 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant