You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now the X2AssessmentImporter runs on a recent window of data, even though districts typically create snapshots over longer time-period with each export.
This introduces edge cases with the automated import process, where some assessments like MCAS or ACCESS get added to district SIS systems after the assessment actually occurs (eg, it happens in May, but isn't entered until September). When this happens, someone in the district emails someone and we run the import manually over the whole time period.
It'd be simpler for maintenance if this was fully automated and we could run the import on the whole snapshot and not have to look at this. We added the 90 day window previously since the import process would run out of memory when processing files that were on the order of ~100mb and ~200k rows.
To make progress on this, let's try:
Add a class that generates a large assessment CSV file (classes like FakeNextGenMcasElaResultGenerator might help). Add a rake task that runs it.
Add a PerfTest that times the import process.
In development mode locally, add a bash script that runs the PerfTest on the console and then greps the output to count the number of database queries (or do this in a better way :)).
Optimize the importer to reduce the queries.
Alternately, measuring max memory usage would be great (but is harder), and other approaches to doing this are very welcome too!
The text was updated successfully, but these errors were encountered:
Right now the
X2AssessmentImporter
runs on a recent window of data, even though districts typically create snapshots over longer time-period with each export.This introduces edge cases with the automated import process, where some assessments like MCAS or ACCESS get added to district SIS systems after the assessment actually occurs (eg, it happens in May, but isn't entered until September). When this happens, someone in the district emails someone and we run the import manually over the whole time period.
It'd be simpler for maintenance if this was fully automated and we could run the import on the whole snapshot and not have to look at this. We added the 90 day window previously since the import process would run out of memory when processing files that were on the order of ~100mb and ~200k rows.
To make progress on this, let's try:
FakeNextGenMcasElaResultGenerator
might help). Add a rake task that runs it.PerfTest
that times the import process.PerfTest
on the console and then greps the output to count the number of database queries (or do this in a better way :)).Alternately, measuring max memory usage would be great (but is harder), and other approaches to doing this are very welcome too!
The text was updated successfully, but these errors were encountered: