Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Assessment importer: Optimize importer to run on ~5M rows #2646

Open
kevinrobinson opened this issue Sep 27, 2019 · 0 comments
Open

Assessment importer: Optimize importer to run on ~5M rows #2646

kevinrobinson opened this issue Sep 27, 2019 · 0 comments

Comments

@kevinrobinson
Copy link
Contributor

kevinrobinson commented Sep 27, 2019

Right now the X2AssessmentImporter runs on a recent window of data, even though districts typically create snapshots over longer time-period with each export.

This introduces edge cases with the automated import process, where some assessments like MCAS or ACCESS get added to district SIS systems after the assessment actually occurs (eg, it happens in May, but isn't entered until September). When this happens, someone in the district emails someone and we run the import manually over the whole time period.

It'd be simpler for maintenance if this was fully automated and we could run the import on the whole snapshot and not have to look at this. We added the 90 day window previously since the import process would run out of memory when processing files that were on the order of ~100mb and ~200k rows.

To make progress on this, let's try:

  1. Add a class that generates a large assessment CSV file (classes like FakeNextGenMcasElaResultGenerator might help). Add a rake task that runs it.
  2. Add a PerfTest that times the import process.
  3. In development mode locally, add a bash script that runs the PerfTest on the console and then greps the output to count the number of database queries (or do this in a better way :)).
  4. Optimize the importer to reduce the queries.

Alternately, measuring max memory usage would be great (but is harder), and other approaches to doing this are very welcome too!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant