Large HAP catalogs still being rejected

[stscijgbot-hstdp]

Issue HLA-1187 was created on JIRA by Rick White:

In the latest version of the pipeline processing (using DrizzlePac 3.6.2), large segment catalogs are still being rejected for having too many sources. This was addressed by ticket HLA-1004 (see also HLA-1079) but apparently has not been fixed yet.

I believe there is a simple fix: set check_big_island_only=True in the final call to detect_and_eval_segments() on line 1773 of catalog_utils.py.

Examples and further discussion are in the comments below.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Example visits that show the problem:

• hst_11911_01_wfc3_uvis_ibc301
• hst_12339_02_wfc3_uvis_ibla02

The trailer file hst_11911_01_wfc3_uvis_total_ibc301_trl.txt for the first visit shows the typical pattern:

1. Tries Gaussian kernel: both biggest source (2.99%) and source fraction (25.24%) are too large.
2. Tries Ricker2D kernel: biggest source (2.61%) is too large. Note source fraction (25.35%) is still large, but is not flagged as too large (this is significant, see discussion below).
3. Tries Gaussian kernel with Background2D: both biggest source (2.48%) and source fraction (22.17%) are too large.
4. Tries Ricker2D kernel with Background2D: biggest source (2.49%) considered acceptable, but source fraction (24.52%) is flagged as too large.

The result is that the segment catalog is rejected.

The clue to the fix is in step 2. Note that the total source fraction there is large, but that is not identified as a reason to reject that version of the catalog. That is because line 1668 of catalog_utils.py specifies the parameter check_big_island_only=True for this call to the detect_and_eval_segments() method. That tells the function to use only the biggest source criterion, and not the source fraction criterion, to decide whether the catalog is acceptable.

Using check_big_island_only=True is a good idea for crowded field catalogs (which includes all Ricker2D kernel catalogs) because it is neither surprising nor incorrect that a large fraction of the image is covered by source segments. If that happens with the Gaussian kernel it indicates a problem, but it is not a problem for the Ricker2D kernel.

The fix for this issue is to use that same parameter in the call to detect_and_eval_segments() on line 1773 of catalog_utils.py. That is the call for attempt 4 (in the list above). Setting check_big_island_only to True is correct for the Ricker2D+Background2D case, just as it is for the Ricker2D case with a simple background.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

In a test branch, I had already modified some criteria potentially to allow for more segmentation catalogs to be generated.  Having tested this recommended change and found it to be better than one of the changes I was contemplating, I intend to include this update in the next release of drizzlepac.  Testing with hst_11911_01_wfc3_uvis_total_ibc301_trl.txt.  Output messages will be updated to reflect properly the state of the associated processing.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Sounds good!  Sorry I've been slow responding to comments, but this sounds like a good plan.  If there is a test version of the data you'd like me to look at, let me know.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Officially closing this ticket as additional testing and evaluation of this fix, as well as other fixes are handled by HLA-1079.