JP-3603: reorder tweakreg to reduce iterations through models

[braingram]

This PR reorganizes operations within the tweakreg step to work towards improving image3 performance.

Changes include:

• Refactor how custom catalogs are resolved (the behavior should be the same although at the the moment there appear to be no tests for this feature, this PR adds tests). This will allow removal of special handling of tweakreg_catalog in ModelContainer (one of the reasons that class opens all models when constructed).
• use wcs_from_footprints on the in-memory wcs objects to compute the combined wcs for limiting the GAIA query (at the moment I also see no tests for this feature). This required a minor backwards-compatible change in assign_wcs.util.
• Refactor catalog/corrector and model linking to instead use the catalog/corrector index and model index. These previously were linked by reference and as align_wcs requires that all correctors be in-memory this results in all input image_models also required to be in memory. Breaking this link will (eventually) allow the image_models to be loaded one-by-one instead of having them all in memory at the same time.
• some minor changes to try and break out portions of TweakRegStep.process into functions to improve readability (and make unit-testing easier)
• minor change to _write_catalog to return a filename instead of modify the input model

This PR adds a parametrized test for the custom catalog handling in tweakreg (marked as slow as it has many combinations, each taking ~2 seconds on my machine, this will still be run in the CI as tox passes the --slow option to pytest).

Regression tests run:
https://plwishmaster.stsci.edu:8081/blue/organizations/jenkins/RT%2FJWST-Developers-Pull-Requests/detail/JWST-Developers-Pull-Requests/1443/pipeline/195
with one unrelated failure which also appears on main:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST/2885/testReport/jwst.regtest/test_miri_lrs_slit_spec3/_stable_deps__test_miri_lrs_slit_spec3_user_wcs_shape1_s2d_/

Resolves https://jira.stsci.edu/browse/JP-3603

Closes #8434

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Attention: Patch coverage is 66.38655% with 40 lines in your changes are missing coverage. Please review.

Project coverage is 58.01%. Comparing base (781e0e0) to head (80bb840).
Report is 3 commits behind head on master.

Files	Patch %	Lines
jwst/tweakreg/tweakreg_step.py	63.30%	40 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8424      +/-   ##
==========================================
+ Coverage   57.93%   58.01%   +0.08%     
==========================================
  Files         387      387              
  Lines       38839    38811      -28     
==========================================
+ Hits        22502    22517      +15     
+ Misses      16337    16294      -43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mcara]

These are my comments for now.

[braingram]

These are my comments for now.

Thanks for the quick response! I'll leave my responses to the other comments.

[mcara]

To the best of my abilities, this looks fine.

[braingram]

Thanks for the review! Your input was helpful in understanding the code and improving this PR.

[braingram]

To provide a small "teaser" to help illustrate the motivation for these changes the following shows memory used for a tweakreg run on an asn with 33 "members":

The memory quickly grows and reaches ~4.5 GB heap size (due to ModelContainer opening all models and ModelContainer and tweakreg keeping them open throughout the run).

With this PR (and some major modifications to ModelContainer to allow it to update models on disk) the following graph shows processing the same association:

with maximum ~1 GB heap size.

[nden]

I have a few minor comments about the tests.

[mairanteodoro]

Thanks, @braingram! Looks good to me.

I only have a couple of suggestions for improvement:

• we should add docstring with the purpose of the test;
• we should break down test_custom_catalog into smaller pieces.

[mairanteodoro]

Thanks, @braingram! Looks good to me.

[braingram]

@nden thanks for adding the regtest in #8477
I rebased this PR and ran the regtest with the added one passing.

@mcara I believe the changes in this PR should have little to no impact on #8476 but please let me know if this PR impacts that PR (or other planned work on tweakreg).

@hbushouse (or anyone else) any objection to merging this now that it's re-opened after rebasing following the other tweakreg changes?

[hbushouse]

@mcara @braingram @nden Is everyone agreed that this is ready to merge? No bad side effects with #8476 ?

[mcara]

Looks OK to me.